Method and system for displaying panoramic image in front of vehicle, storage medium and electronic equipment

Abstract

The invention belongs to the technical field of automobile intelligent driving, and provides an automobile front panoramic image display method and system, a storage medium and electronic equipment, and the method comprises the steps: obtaining an automobile front image, and generating a panoramic display image according to the automobile front image; acquiring real-time vehicle speed and steering wheel angle data of vehicle driving, calculating a predicted driving track of the vehicle according to the real-time vehicle speed and the steering wheel angle data, and generating a first panoramic image in combination with the panoramic display image; the information of the obstacle in front of the vehicle is obtained, the position of the obstacle in front of the vehicle is determined by combining the information of the obstacle in front of the vehicle with the first panoramic image, the position of the obstacle in front of the vehicle is marked in the first panoramic image, a second panoramic image is generated, and the PHUD display module displays the second panoramic image. The front view of a vehicle obtained by a driver is wider, images are displayed on the front windshield more clearly and are not prone to being influenced by external light and interference, and driving safety and convenience are improved.

Description

Technical Field

[0001] The present invention relates to the field of automobile intelligent driving technology, and in particular to a method and system for displaying a panoramic image in front of a vehicle. Background Art

[0002] In the prior art, the mainstream solution for displaying the front view of the vehicle is to use a 360-degree camera system to capture images of the vehicle's surroundings and project them on the central control. This system can obtain images in front of the vehicle through the front camera, and then project these images on the central control screen, allowing the driver to clearly observe the environment in front of the vehicle, including blind spots and obstacles. The disadvantage of this solution is that the image in front of the vehicle is displayed on the central control display screen, which will distract the driver from driving when he is distracted by the image on the central control display screen.

[0003] In the prior art, the invention patent application with application number CN202210438601.5 discloses a display system for a smart cockpit and a smart vehicle. The display system for the smart cockpit includes an immersive instrument, an A-pillar display screen, and a naked-eye 3D instrument; the immersive instrument is used to display the image information of the blind spot of the front of the vehicle, which is the image information in the driver's field of vision blocked by the front engine cover of the smart vehicle; the A-pillar display screen is used to display the image information of the A-pillar blind spot, which is the image information in the driver's field of vision blocked by the A-pillars on both sides of the smart vehicle; the naked-eye 3D instrument is set on the instrument panel of the smart vehicle, and displays the map information of the current location of the smart vehicle in a suspended manner. Although the image information of the blind spot of the front field of vision can be displayed, there is a physical frame, and the displayed field of vision in front of the vehicle is relatively narrow, and the picture displayed in a suspended manner cannot be better integrated with the real scene. Summary of the invention

[0004] To solve the above problems, the present disclosure provides a method and system for displaying a panoramic image in front of a vehicle, which collects images in front of the vehicle, processes the collected images in front of the vehicle, and generates a panoramic display image; calculates the predicted driving trajectory of the vehicle according to the real-time vehicle speed and steering wheel angle data, and generates a first panoramic image in combination with the panoramic display image; obtains obstacle information in front of the vehicle, combines the obstacle information in front of the vehicle with the first panoramic image, determines the position of the obstacle in front of the vehicle, and marks the position of the obstacle in front of the vehicle in the first panoramic image to generate a second panoramic image, and the PHUD display module displays the second panoramic image; realizes the display of the panoramic image in front of the vehicle including the predicted driving trajectory and the obstacle position to the driver.

[0005] The present invention is achieved through the following technical solutions:

[0006] In a first aspect, an embodiment of the present disclosure provides a method for displaying a panoramic image in front of a vehicle, the method comprising:

[0007] Acquire the image in front of the vehicle, and generate a panoramic display image based on the image in front of the vehicle;

[0008] Acquire real-time vehicle speed and steering wheel angle data of the vehicle, calculate the predicted driving trajectory of the vehicle according to the real-time vehicle speed and steering wheel angle data, and generate a first panoramic image in combination with the panoramic display image;

[0009] Obstacle information in front of the vehicle is obtained, and the obstacle information in front of the vehicle is combined with the first panoramic image to determine the position of the obstacle in front of the vehicle, and the position of the obstacle in front of the vehicle is marked in the first panoramic image to generate a second panoramic image, and the PHUD display module displays the second panoramic image.

[0010] Furthermore,

[0011] When the position of the obstacle in front of the vehicle overlaps with the predicted driving trajectory, a third panoramic image is generated based on the second panoramic image, and the PHUD display module displays the third panoramic image to send a warning signal to the driver.

[0012] Furthermore,

[0013] The image acquisition module is used to capture the image in front of the vehicle, and the image is converted into the image data in front of the vehicle, and the image data in front of the vehicle is sent to the image processing module;

[0014] The image processing module extracts the sub-images in front of the vehicle taken by each sub-camera of the image acquisition module according to the image data in front of the vehicle;

[0015] Extracting the plantago sub-images corresponding to the same frame of each plantago sub-image, performing image processing on each plantago sub-image, and determining a panoramic image of the front of the plantago;

[0016] The vehicle front sub-image corresponding to the same frame of each vehicle front sub-image is extracted frame by frame, and the vehicle front panoramic image is determined frame by frame to synthesize a panoramic display image.

[0017] Furthermore,

[0018] De-distort the Plantago image to correct the image distortion caused by camera distortion;

[0019] Adjust the brightness and contrast of each plantago image in the same frame to make the light of each plantago image consistent;

[0020] Using a feature detector, extracting feature points and descriptors of each Plantago image, and characterizing features in the Plantago image;

[0021] The matching device is used to match the feature points between different Plantago sub-images, and the similarity of the descriptors is compared to determine the corresponding relationship between the different Plantago sub-images;

[0022] According to the matching results of the feature points of the plantain sub-images, the homography matrix is ​​calculated to align the plantain sub-images, and the perspective transformation is performed using the homography matrix to map the plantain sub-images to the same coordinate system to correct the plantain sub-images;

[0023] The corrected Plantago sub-images of the same frame are spliced ​​together, and the overlapping areas after the splicing of the Plantago sub-images are processed to remove the obvious seams at the splicing of the Plantago sub-images.

[0024] Furthermore,

[0025] According to the real-time vehicle speed and steering wheel angle data, the vehicle's driving trajectory is calculated using a single track model to generate a predicted vehicle driving trajectory;

[0026] The predicted driving trajectory and the panoramic display image are fused to generate a second panoramic image.

[0027] Furthermore,

[0028] Collect obstacle information in front of the vehicle and send it to the obstacle warning module;

[0029] The obstacle warning module combines the obstacle information in front of the vehicle with the first panoramic image to determine the precise location and attributes of the obstacle; wherein the obstacle information in front of the vehicle includes distance, location and attributes;

[0030] The position and attributes of the obstacle in front of the vehicle are marked in the first panoramic image using a box and / or text to generate a second panoramic image.

[0031] Furthermore,

[0032] The obstacle information in front of the vehicle is converted into a data format, the spatial coordinates of the obstacle in front of the vehicle are represented in the form of a point cloud, and the point cloud of the obstacle in front of the vehicle is determined;

[0033] Determine whether there is a spatial overlap between the predicted driving trajectory of the vehicle and the point cloud of the obstacle in front of the vehicle;

[0034] If there is a spatial overlapping area between the predicted driving trajectory and the point cloud of the obstacle in front of the vehicle, a third panoramic image is generated based on the second panoramic image to send a warning signal to the driver.

[0035] In a second aspect, based on the same inventive concept, the disclosed embodiment further provides a vehicle front panoramic image display system, the system comprising: an image processing module, a trajectory prediction module and an obstacle warning module;

[0036] An image processing module is used to obtain the image in front of the vehicle and generate a panoramic display image based on the image in front of the vehicle;

[0037] A trajectory prediction module is used to obtain real-time vehicle speed and steering wheel angle data of the vehicle, calculate the predicted vehicle driving trajectory based on the real-time vehicle speed and steering wheel angle data, and generate a first panoramic image in combination with the panoramic display image;

[0038] The obstacle warning module is used to obtain information about obstacles in front of the vehicle, combine the information about obstacles in front of the vehicle with the first panoramic image, determine the position of the obstacles in front of the vehicle, and mark the position of the obstacles in front of the vehicle in the first panoramic image to generate a second panoramic image; it is also used to generate a third panoramic image based on the second panoramic image when the position of the obstacles in front of the vehicle overlaps with the predicted driving trajectory.

[0039] On the third aspect, based on the same inventive concept, the embodiment of the present disclosure also provides a computer-readable storage medium, storing one or more programs, which, when executed, can implement the aforementioned method for displaying a panoramic image in front of the vehicle.

[0040] In a fourth aspect, based on the same inventive concept, an embodiment of the present disclosure further provides an electronic device, including a processor, a communication interface, the aforementioned computer-readable storage medium, and a communication bus; wherein the processor, the communication interface, and the computer-readable storage medium communicate with each other via the communication bus;

[0041] The processor is used to execute the program stored in the aforementioned computer-readable storage medium.

[0042] Compared with the prior art, the present invention has the following advantages:

[0043] 1. It reduces the driver's visual blind spots during driving, allowing the driver to have a wider field of view in front of the car, and the image displayed to the driver can be better integrated with the real scene.

[0044] 2. The visible area completely covers the front cabin of the vehicle. The image displayed on the front windshield is clearer and less susceptible to external light and interference. The driver can obtain key road information without shifting his or her sight or distracting his or her attention, thereby improving driving safety and convenience.

[0045] Other features and advantages of the present disclosure will be described in the following description, and partly become apparent from the description, or be understood by implementing the present disclosure. The purpose and other advantages of the present disclosure can be realized and obtained by the structures pointed out in the description, claims and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the embodiments of the present disclosure or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0047] Figure 1 A flow chart of a method for displaying a panoramic image in front of a vehicle provided by an embodiment of the present disclosure;

[0048] Figure 2 A schematic diagram of a PHUD display provided in an embodiment of the present disclosure;

[0049] Figure 3 A block diagram of a front-view panoramic image display system provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0050] In order to make the purpose, technical solution and advantages of the embodiments of the present disclosure clearer, the technical solution in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present disclosure.

[0051] First, Figure 1 A flow chart of a method for displaying a panoramic image in front of a vehicle provided by an embodiment of the present disclosure, such as Figure 1 As shown, the embodiment of the present disclosure provides a method for displaying a panoramic image in front of a vehicle, comprising:

[0052] S1: Acquire the image in front of the vehicle, and generate a panoramic display image based on the image in front of the vehicle.

[0053] S2: Acquire the real-time vehicle speed and steering wheel angle data of the vehicle, calculate the predicted driving trajectory of the vehicle based on the real-time vehicle speed and steering wheel angle data, and generate a first panoramic image in combination with the panoramic display image.

[0054] S3: Obtain information about an obstacle in front of the vehicle, combine the information about the obstacle in front of the vehicle with the first panoramic image, determine the position of the obstacle in front of the vehicle, and mark the position of the obstacle in front of the vehicle in the first panoramic image to generate a second panoramic image.

[0055] In the disclosed embodiment, an image acquisition module is used to acquire images in front of the vehicle, and the acquired images in front of the vehicle are preprocessed to generate a panoramic display image; a predicted driving trajectory of the vehicle is calculated based on the real-time vehicle speed and steering wheel angle data, and a first panoramic image is generated in combination with the panoramic display image, wherein the first panoramic image includes the predicted driving trajectory on the basis of the panoramic display image; the position of an obstacle in front of the vehicle is determined, and the position of the obstacle in front of the vehicle is marked in the first panoramic image to generate a second panoramic image, wherein the second panoramic image also includes the position of the obstacle in front of the vehicle on the basis of the first panoramic image; and a PHUD display module is used to display the second panoramic image to the driver.

[0056] like Figure 2 As shown in the figure, the WHUD (Windshield Head-Up Display) display technology in the prior art is suspended above the front cabin of the vehicle and displayed at the level of the driver's normal line of sight. The image displayed by the WHUD is located in front of the driver's line of sight and may overlap with the actual road scene, interfering with the driver's normal field of vision, causing visual confusion and distraction of the driver. In addition, the information displayed by the WHUD may be affected by factors such as surrounding light and reflection, reducing image clarity and visibility.

[0057] Usually, the car's instrument panel is located in the center or front of the driver's cockpit. When the driver views the image in front of the car through the instrument panel, he needs to shift his sight from the road in front to the instrument panel. This sight shift will cause the driver's attention to be diverted from the road, reduce observation and vigilance of the road ahead, and increase driving risks. Especially when driving at high speeds or in complex road conditions, the driver's sight shift may cause the driver to miss key information or cause dangerous situations.

[0058] Therefore, this solution uses PHUD (Panoramic Head-Up Display) display technology to display the driver with a panoramic image in front of the vehicle. The projection position of the PHUD image is between the WHUD display position and the instrument position, and the PHUD display technology visible area completely covers the front cabin of the vehicle. The PHUD display module prints a high-reflectivity black ink area on the lower side of the imaging windshield. This design makes the image displayed on the front windshield clearer and less susceptible to external light and interference. By using PHUD display technology, the driver can obtain key road information without diverting his sight and distracting his attention, improving driving safety and convenience.

[0059] In some examples, obtaining a front image of a vehicle and generating a panoramic display image according to the front image of the vehicle specifically includes:

[0060] S11: Use the image acquisition module to capture the front image of the vehicle, convert it into the front image data of the vehicle, and send the front image data of the vehicle to the image processing module.

[0061] S12: The image processing module extracts the sub-images in front of the vehicle taken by each sub-camera of the image acquisition module according to the image data in front of the vehicle.

[0062] Specifically, the image acquisition module includes a plurality of sub-cameras for capturing images in front of the vehicle at different angles. Therefore, the sub-images in front of the vehicle captured by each sub-camera of the image acquisition module are extracted for subsequent determination of the panoramic display image.

[0063] S13: extracting the plantago sub-images corresponding to the same frame of each plantago sub-image, performing image processing on each plantago sub-image, and determining a front panoramic image of the plantago.

[0064] S14: extracting the vehicle front sub-image corresponding to the same frame of each vehicle front sub-image frame by frame, and determining the vehicle front panoramic image frame by frame, and synthesizing the panoramic display image.

[0065] Specifically, the frontal image is split frame by frame, and the split images are processed frame by frame. After processing each frame, a complete image is synthesized, that is, a panoramic display image. The panoramic display image includes visual blind spots that the driver cannot directly observe, such as the A-pillar of the car, thereby effectively reducing the driver's blind spots during driving.

[0066] In some examples, extracting the plantain sub-images corresponding to the same frame of each plantain sub-image, performing image processing on each plantain sub-image, and determining the front panoramic image of the vehicle specifically includes:

[0067] S131: Dedistort the Plantago image to correct image distortion caused by camera distortion.

[0068] S132: adjusting the brightness and contrast of each Plantago seed image in the same frame to make the light of each Plantago seed image consistent.

[0069] S133: Using a feature detector, extracting feature points and descriptors of each Plantago image, and characterizing features in the Plantago image.

[0070] S134: using a matcher to match feature points between different Plantago sub-images, and comparing similarities of descriptors to determine the corresponding relationship between different Plantago sub-images.

[0071] Specifically, the feature detector includes ORB (Oriented FAST and Rotated BRIEF) or SIFT (Scale-Invariant Feature Transform), and the matcher includes FLANN (Fast Library for Approximate Nearest Neighbors) or BFMatcher (Brute-Force Matcher).

[0072] S135: Calculate the homography matrix according to the matching results of the feature points of the Plantago sub-images, align the Plantago sub-images, and use the homography matrix to perform perspective transformation, map the Plantago sub-images to the same coordinate system, and correct the Plantago sub-images.

[0073] S136: stitching the corrected Plantago sub-images of the same frame, processing the overlapping areas after stitching the Plantago sub-images, and removing the obvious seams at the splicing of the Plantago sub-images.

[0074] In some examples, obtaining real-time vehicle speed and steering wheel angle data of the vehicle, calculating the predicted vehicle driving trajectory according to the real-time vehicle speed and steering wheel angle data, and generating a first panoramic image in combination with the panoramic display image specifically includes:

[0075] S21: According to the real-time vehicle speed and steering wheel angle data of the vehicle, a single-track model is used to calculate the vehicle's driving trajectory to generate a predicted vehicle driving trajectory.

[0076] S22: Fusing the predicted driving trajectory and the panoramic display image to generate a second panoramic image.

[0077] In some examples, obtaining information about an obstacle in front of the vehicle, combining the information about the obstacle in front of the vehicle with the first panoramic image, determining the position of the obstacle in front of the vehicle, and marking the position of the obstacle in front of the vehicle in the first panoramic image to generate a second panoramic image specifically includes:

[0078] S31: Collect obstacle information in front of the vehicle and send the obstacle information in front of the vehicle to the obstacle warning module.

[0079] S32: The obstacle warning module combines the obstacle information in front of the vehicle with the first panoramic image to determine the precise location and attributes of the obstacle.

[0080] Specifically, the obstacle information in front of the vehicle includes distance, position and attributes. Computer vision algorithms can be used to identify obstacles appearing in the first panoramic image. Computer vision algorithms can automatically analyze various objects and obstacles in the image in front of the vehicle, accurately identify their positions and features, and determine the attributes of the obstacle in front of the vehicle by identifying the obstacle features. The obstacle recognition module can also be used directly to collect the obstacle information in front of the vehicle. The attributes of the obstacle in front of the vehicle include: obstacle type, such as people, vehicles, cone barrels, etc., and corresponding height, width and other information.

[0081] S33: Marking the position and attributes of the obstacle in front of the vehicle in the first panoramic image using a box and / or text to generate a second panoramic image.

[0082] In some examples, the method for displaying a panoramic image in front of the vehicle further includes: when the position of the obstacle in front of the vehicle overlaps with the predicted driving trajectory, generating a third panoramic image based on the second panoramic image to send a warning signal to the driver. In addition to displaying the panoramic image in front of the vehicle more clearly and comprehensively, a specific warning needs to be given to the driver to remind the driver to slow down, stop or avoid.

[0083] Furthermore, when the position of the obstacle in front of the vehicle overlaps with the predicted driving trajectory, a third panoramic image is generated based on the second panoramic image to send a warning signal to the driver, specifically including:

[0084] S41: Convert the data format of the obstacle information in front of the vehicle, represent the spatial coordinates of the obstacle in front of the vehicle in the form of a point cloud, and determine the point cloud of the obstacle in front of the vehicle.

[0085] S42: Determine whether there is a spatial overlap area between the predicted driving trajectory of the vehicle and the point cloud of the obstacle in front of the vehicle.

[0086] S43: If there is a spatial overlap area between the predicted driving trajectory and the point cloud of the obstacle in front of the vehicle, a third panoramic image is generated based on the second panoramic image to send a warning signal to the driver.

[0087] Embodiment 1:

[0088] S101: Acquire a front image of the vehicle, and generate a panoramic display image based on the front image of the vehicle.

[0089] The image acquisition module in the present disclosure includes multiple sub-cameras for shooting in multiple directions and angles. The image acquisition module captures the image in front of the vehicle, converts it into image data in front of the vehicle, and transmits the image data in front of the vehicle to the image processing module for processing.

[0090] After receiving the front image data, the image processing module extracts the front sub-images corresponding to the same frame of each front sub-image according to the front image data, performs image processing on each front sub-image, and determines the front panoramic image. The following uses the same frame of front sub-images to determine a frame of front panoramic image as an example for explanation:

[0091] Dedistort the Plantago sub-image to correct the image distortion caused by camera distortion. Adjust the brightness and contrast of each Plantago sub-image in the same frame to make the light of each Plantago sub-image consistent to ensure that each Plantago sub-image has a consistent visual effect. Use a feature detector to extract the feature points and descriptors of each Plantago sub-image to characterize the features in the Plantago sub-image. Use a matcher to match the feature points between different Plantago sub-images, compare the similarity of the descriptors, and determine the correspondence between different Plantago sub-images. Among them, the feature detector includes ORB (Oriented FAST and RotatedBRIEF) or SIFT (Scale-Invariant Feature Transform), and the matcher includes FLANN (Fast Libraryfor Approximate Nearest Neighbors) or BFMatcher (Brute-Force Matcher).

[0092] According to the matching results of the feature points of the plantain sub-images, the homography matrix is ​​calculated, the plantain sub-images are aligned, and the homography matrix is ​​used to perform perspective transformation, and the plantain sub-images are mapped to the same coordinate system to correct the plantain sub-images. The corrected plantain sub-images of the same frame are spliced, and the overlapping areas after the plantain sub-images are processed, and the obvious seams at the splicing of the plantain sub-images are removed, and finally the plantain panoramic image is determined.

[0093] By using the above method, the vehicle front sub-images corresponding to the same frame of each vehicle front sub-image are extracted frame by frame, and the vehicle front panoramic image is determined frame by frame, and the vehicle front panoramic images of consecutive frames are synthesized into a panoramic display image.

[0094] S102: Calculate the predicted driving trajectory of the vehicle, and determine a first panoramic image in combination with the panoramic display image.

[0095] In the disclosed embodiment, the driving trajectory of the vehicle is calculated based on the real-time speed and steering wheel angle data of the vehicle. During the driving of the vehicle, the vehicle's travel trajectory is calculated using a single-track model using the real-time speed and steering wheel angle data. The single-track model is a kinematic model of the vehicle and can be used to accurately calculate the vehicle's forward trajectory. Using the vehicle's real-time speed, steering wheel angle, and wheelbase, the single-track model can calculate the vehicle's turning radius and update the vehicle's current position within a continuous time step, thereby generating a predicted driving trajectory for the vehicle. The predicted driving trajectory can be combined with a panoramic display image and clearly displayed in the driver's field of view through the PHUD display module, thereby improving the driver's visibility and understanding.

[0096] S103: Determine the position of the obstacle in front of the vehicle, and mark the position of the obstacle in front of the vehicle in the first panoramic image to generate a second panoramic image.

[0097] In the disclosed embodiment, an obstacle recognition module is used to collect information about obstacles in front of the vehicle, wherein the information about obstacles in front of the vehicle includes distance, position, and attributes. The position and attributes of the obstacles in front of the vehicle are marked with boxes and / or text to indicate the precise position and attributes of the obstacles in the first panoramic image, and a second panoramic image is generated. The obstacle recognition module may be an ADAS system (Advanced Driver Assistance System).

[0098] S104: When the position of the obstacle in front of the vehicle overlaps with the predicted driving trajectory, a third panoramic image is generated.

[0099] In the disclosed embodiment, an ADAS system (Advanced Driver Assistance System) is used as an obstacle recognition module. Sensors in the ADAS system, such as laser radars, cameras, etc., obtain obstacle information of the vehicle's surrounding environment, convert the obstacle information into a data format, represent the spatial coordinates of the obstacle in the form of a point cloud, and determine the obstacle point cloud; based on the spatial overlap area between the vehicle's predicted driving trajectory and the obstacle point cloud, it is automatically identified whether the vehicle's predicted driving trajectory conflicts with the obstacle. If the vehicle's predicted driving trajectory overlaps with the obstacle point cloud, a third panoramic image is generated based on the second panoramic image. The third panoramic image will highlight the obstacle and send out a warning signal to alert the driver of potential danger.

[0100] Furthermore, taking the example of a vehicle driving on the road, the front camera of the ADAS system is first used to capture the image of the vehicle in front, and the front radar of the ADAS system uses a visual algorithm to determine the obstacle or the distance information of the vehicle in front, and the obstacle in front of the vehicle is marked in real time. The predicted driving trajectory is determined based on the vehicle's driving speed, steering wheel angle information and wheelbase. When the predicted driving trajectory overlaps with the marked obstacle, the system highlights the overlapping area as an obstacle warning, that is, a third panoramic image is generated. Combined with the PHUD display module, the third panoramic image containing the warning information is directly presented in the front cabin, thereby effectively reminding the driver of potential obstacle risks.

[0101] In the disclosed embodiment, a driver monitoring system is used to collect the driver's eye position data. The eye position data includes eye coordinates and gaze point information. In order to ensure that the driver's eye position data can be obtained in real time, the driver monitoring system periodically obtains and updates the eye position data to ensure that the driver's eye position data is updated in real time when the driver's line of sight changes.

[0102] According to the eye position data obtained by the driver monitoring system, the PHUD display module adjusts the position and angle of the panoramic image displayed in front of the vehicle, so that the projected panoramic image is integrated with the driver's actual line of sight to avoid visual conflict.

[0103] On the second aspect, based on the same inventive concept, Figure 3 A block diagram of a front panoramic image display system provided by an embodiment of the present disclosure, such as Figure 3 As shown, the disclosed embodiment also provides a system for displaying a panoramic image in front of a vehicle, and the system includes: an image processing module, a trajectory prediction module, and an obstacle warning module. The image processing module is used to obtain the image in front of the vehicle, and generate a panoramic display image based on the image in front of the vehicle. The trajectory prediction module is used to obtain the real-time vehicle speed and steering wheel angle data of the vehicle, calculate the predicted driving trajectory of the vehicle based on the real-time vehicle speed and steering wheel angle data, and generate a first panoramic image in combination with the panoramic display image. The obstacle warning module is used to obtain the obstacle information in front of the vehicle, and the obstacle information in front of the vehicle is combined with the first panoramic image to determine the position of the obstacle in front of the vehicle, and the position of the obstacle in front of the vehicle is marked in the first panoramic image to generate a second panoramic image; the obstacle warning module is also used to generate a third panoramic image based on the second panoramic image when the position of the obstacle in front of the vehicle overlaps with the predicted driving trajectory.

[0104] Furthermore, the front panoramic image display system also includes a PHUD display module, which is used to display a second panoramic image to the driver, and when the position of the obstacle in front of the vehicle overlaps with the predicted driving trajectory, display a third panoramic image to send a warning signal to the driver.

[0105] In the disclosed embodiment, the front panoramic image display system further includes an image acquisition module and an obstacle recognition module. The image acquisition module is used to capture the front image of the vehicle, convert it into the front image data of the vehicle, and send the front image data of the vehicle to the image processing module. The obstacle recognition module is used to collect the information of the obstacle in front of the vehicle, wherein the information of the obstacle in front of the vehicle includes the distance, position and attributes, etc.

[0106] When the front panoramic image display system is actually deployed, the system modules are divided into data layer, business logic layer and application layer. The data layer includes image acquisition module and obstacle recognition module; the business logic layer includes image processing module, trajectory prediction module and obstacle warning module; the application layer includes PHUD display module, which is used to display the fused panoramic image, predicted driving trajectory, obstacle location and possible warning information.

[0107] On the third aspect, based on the same inventive concept, the embodiment of the present disclosure also provides a computer-readable storage medium, storing one or more programs, which, when executed, can implement the aforementioned method for displaying a panoramic image in front of the vehicle.

[0108] In a fourth aspect, based on the same inventive concept, an embodiment of the present disclosure further provides an electronic device, including a processor, a communication interface, the aforementioned computer-readable storage medium, and a communication bus. The processor, the communication interface, and the computer-readable storage medium communicate with each other via the communication bus. The processor is used to execute the program stored in the aforementioned computer-readable storage medium.

[0109] It should be noted that the electrical connection between the above-mentioned units does not necessarily mean the connection between lines. An indirect connection method can be applied to the embodiments of the present disclosure as long as the purpose of the present disclosure is achieved.

[0110] Although the present disclosure has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present disclosure.

Claims

1. A method for displaying a panoramic image in front of a vehicle, characterized in that: The method comprises: Acquire a front image of the vehicle, and generate a panoramic display image based on the front image of the vehicle; Acquiring real-time vehicle speed and steering wheel angle data of the vehicle, calculating a predicted vehicle driving trajectory based on the real-time vehicle speed and steering wheel angle data, and generating a first panoramic image in combination with the panoramic display image; Obstacle information in front of the vehicle is obtained, and the obstacle information in front of the vehicle is combined with the first panoramic image to determine the position of the obstacle in front of the vehicle, and the position of the obstacle in front of the vehicle is marked in the first panoramic image to generate a second panoramic image, and the PHUD display module displays the second panoramic image.

2. The method according to claim 1, characterized in that The method for displaying a panoramic image in front of a vehicle further includes: When the position of the obstacle in front of the vehicle overlaps with the predicted driving trajectory, a third panoramic image is generated based on the second panoramic image, and the PHUD display module displays the third panoramic image to send a warning signal to the driver.

3. The method according to claim 1, characterized in that The step of acquiring the image in front of the vehicle and generating a panoramic display image according to the image in front of the vehicle includes: Using the image acquisition module to capture the front image of the vehicle, and converting it into the front image data of the vehicle, and sending the front image data of the vehicle to the image processing module; The image processing module extracts the sub-images in front of the vehicle taken by each sub-camera of the image acquisition module according to the image data in front of the vehicle, wherein each sub-camera takes the image in front of the vehicle at a different angle; Extracting the plantago sub-images corresponding to the same frame of each plantago sub-image, performing image processing on each plantago sub-image, and determining a front panoramic image of the plantago; The vehicle front sub-images corresponding to the same frame of each vehicle front sub-image are extracted frame by frame, and the vehicle front panoramic image is determined frame by frame to synthesize the panoramic display image.

4. The method according to claim 3, characterized in that The extracting of the plantago sub-images corresponding to the same frame of each plantago sub-image, performing image processing on each plantago sub-image, and determining the front panoramic image of the vehicle includes: De-distorting the plantain image to correct image distortion caused by camera distortion; Adjusting the brightness and contrast of each of the plantago images in the same frame to make the light of each of the plantago images consistent; Using a feature detector, extracting feature points and descriptors of each of the Plantago sub-images to characterize features in the Plantago sub-images; Using a matcher to match the feature points between different Plantago sub-images, and comparing the similarities of the descriptors to determine the corresponding relationship between the different Plantago sub-images; Calculating a homography matrix according to the matching results of the feature points of the Plantago sub-images, aligning the Plantago sub-images, and performing perspective transformation using the homography matrix to map the Plantago sub-images to the same coordinate system, thereby correcting the Plantago sub-images; The corrected Plantago sub-images of the same frame are spliced ​​together, and the overlapping areas of the Plantago sub-images after splicing are processed to remove the obvious seams at the splicing of the Plantago sub-images.

5. The method according to claim 1, characterized in that The method of acquiring the real-time vehicle speed and steering wheel angle data of the vehicle, calculating the predicted vehicle driving trajectory according to the real-time vehicle speed and steering wheel angle data, and generating a first panoramic image in combination with the panoramic display image includes: According to the real-time vehicle speed and steering wheel angle data of the vehicle, a single track model is used to calculate the vehicle's driving trajectory to generate the predicted vehicle's driving trajectory; The predicted driving trajectory and the panoramic display image are fused to generate the second panoramic image.

6. The method according to claim 1, characterized in that The obtaining of the obstacle information in front of the vehicle, combining the obstacle information in front of the vehicle with the first panoramic image, determining the position of the obstacle in front of the vehicle, and marking the position of the obstacle in front of the vehicle in the first panoramic image to generate the second panoramic image includes: Collecting the obstacle information in front of the vehicle and sending the obstacle information in front of the vehicle to the obstacle warning module; The obstacle warning module combines the obstacle information in front of the vehicle with the first panoramic image to determine the precise location and attributes of the obstacle; wherein the obstacle information in front of the vehicle includes distance, location and attributes; The position and attributes of the obstacle in front of the vehicle are marked in the first panoramic image using boxes and / or text to generate the second panoramic image.

7. The method according to any one of claims 1 to 6, characterized in that: When the position of the obstacle in front of the vehicle overlaps with the predicted driving trajectory, generating a third panoramic image based on the second panoramic image to send a warning signal to the driver includes: The obstacle information in front of the vehicle is converted into a data format, the spatial coordinates of the obstacle in front of the vehicle are represented in the form of a point cloud, and the point cloud of the obstacle in front of the vehicle is determined; Determine whether there is a spatial overlap between the predicted driving trajectory of the vehicle and the point cloud of the obstacle in front of the vehicle; If there is a spatial overlapping area between the predicted driving trajectory and the point cloud of the obstacle in front of the vehicle, the third panoramic image is generated based on the second panoramic image to send a warning signal to the driver.

8. A front panoramic image display system, characterized in that: The system comprises: an image processing module, a trajectory prediction module and an obstacle warning module; The image processing module is used to obtain the image in front of the vehicle and generate a panoramic display image based on the image in front of the vehicle; The trajectory prediction module is used to obtain the real-time vehicle speed and steering wheel angle data of the vehicle, calculate the predicted vehicle driving trajectory according to the real-time vehicle speed and steering wheel angle data, and generate a first panoramic image in combination with the panoramic display image; The obstacle warning module is used to obtain information about obstacles in front of the vehicle, combine the information about obstacles in front of the vehicle with the first panoramic image, determine the position of the obstacle in front of the vehicle, and mark the position of the obstacle in front of the vehicle in the first panoramic image to generate a second panoramic image; and is also used to generate a third panoramic image based on the second panoramic image when the position of the obstacle in front of the vehicle overlaps with the predicted driving trajectory.

9. A computer-readable storage medium storing one or more programs, characterized in that: When the one or more programs are executed, the method for displaying the front panoramic image of the vehicle as described in any one of claims 1 to 7 can be implemented.

10. An electronic device comprising a processor, a communication interface, the computer-readable storage medium of claim 9, and a communication bus; wherein: The processor, the communication interface, and the computer-readable storage medium communicate with each other via a communication bus; It is characterized in that The processor is configured to execute a program stored in a computer-readable storage medium.

Images