A front-view panoramic image display method, system, storage medium and electronic device

Abstract

The present disclosure belongs to the technical field of intelligent driving of automobiles, and provides a front-vehicle panoramic image display method, a system, a storage medium and an electronic device, the method comprising: acquiring a front-vehicle image, generating a panoramic display image according to the front-vehicle image; acquiring real-time vehicle speed and steering wheel angle data of vehicle driving, calculating a predicted driving track of the vehicle according to the real-time vehicle speed and the steering wheel angle data, generating a first panoramic image in combination with the panoramic display image; acquiring front-vehicle obstacle information, determining a front-vehicle obstacle position in combination with the first panoramic image, marking the front-vehicle obstacle position in the first panoramic image, generating a second panoramic image, and displaying the second panoramic image by a PHUD display module. The driver can obtain a wider front-vehicle field of view, the image displayed on the front windshield is clearer, and is not easily affected by external light and interference, and the safety and convenience of driving are improved.

Description

Technical Field

[0001] The present disclosure belongs to the field of automobile intelligent driving technology, and in particular relates to a method and system for displaying a panoramic image in front of a vehicle. Background Art

[0002] The mainstream solution for displaying the vehicle's front view currently uses a 360-degree camera system to capture images of the vehicle's surroundings and project them onto the central control panel. This system captures images of the vehicle's front view through the front camera and then projects these images onto the central control panel, allowing the driver to clearly observe the environment ahead, including blind spots and obstacles. However, the drawback of this solution is that the front view is displayed on the central control panel, which can distract the driver from focusing on the image on the central control panel.

[0003] In the prior art, the invention patent application with application number CN202210438601.5 discloses a display system for a smart cockpit and a smart vehicle. The display system for the smart cockpit includes an immersive instrument, an A-pillar display, and a naked-eye 3D instrument. The immersive instrument is used to display image information from the front blind spot of the vehicle. The image information from the front blind spot is the image information in the driver's field of view blocked by the front engine cover of the smart vehicle. The A-pillar display is used to display image information from the A-pillar blind spot. The image information from the A-pillar blind spot is the image information in the driver's field of view blocked by the A-pillars on both sides of the smart vehicle. The naked-eye 3D instrument is set on the dashboard of the smart vehicle and displays map information of the current location of the smart vehicle in a floating manner. Although the image information of the blind spot of the front field of view can be displayed, the physical border exists, and the displayed field of view in front of the vehicle is relatively narrow. The image displayed in a floating manner cannot be better integrated with the real scene. Summary of the Invention

[0004] To solve the above problems, the present disclosure provides a method and system for displaying a panoramic image in front of a vehicle, which collects images in front of the vehicle, processes the collected images in front of the vehicle, and generates a panoramic display image; calculates the predicted driving trajectory of the vehicle based on real-time vehicle speed and steering wheel angle data, and combines the panoramic display image to generate a first panoramic image; obtains information about obstacles in front of the vehicle, combines the information about obstacles in front of the vehicle with the first panoramic image, determines the position of the obstacles in front of the vehicle, and marks the position of the obstacles in front of the vehicle in the first panoramic image to generate a second panoramic image, and the PHUD display module displays the second panoramic image; and realizes the display of a panoramic image in front of the vehicle including the predicted driving trajectory and the position of the obstacles to the driver.

[0005] The present invention is achieved through the following technical solutions:

[0006] In a first aspect, an embodiment of the present disclosure provides a method for displaying a panoramic image in front of a vehicle, the method comprising:

[0007] Acquire the front image of the vehicle, and generate a panoramic display image based on the front image of the vehicle;

[0008] Acquiring real-time vehicle speed and steering wheel angle data of the vehicle, calculating a predicted vehicle trajectory based on the real-time vehicle speed and steering wheel angle data, and combining the predicted trajectory with the panoramic display image to generate a first panoramic image;

[0009] Obtain information about obstacles in front of the vehicle, combine the information with the first panoramic image, determine the location of the obstacles in front of the vehicle, mark the location of the obstacles in front of the vehicle in the first panoramic image, generate a second panoramic image, and display the second panoramic image on the PHUD display module.

[0010] Further,

[0011] When the position of the obstacle in front of the vehicle overlaps with the predicted driving trajectory, a third panoramic image is generated based on the second panoramic image, and the PHUD display module displays the third panoramic image to issue a warning signal to the driver.

[0012] Further,

[0013] Use the image acquisition module to capture the front image of the vehicle, convert it into front image data, and send the front image data to the image processing module;

[0014] The image processing module extracts the front sub-images of the vehicle taken by each sub-camera of the image acquisition module based on the front image data of the vehicle;

[0015] Extracting the plantago image corresponding to the same frame of each plantago image, performing image processing on each plantago image, and determining a front panoramic image of the plantago;

[0016] The vehicle front sub-image corresponding to the same frame of each vehicle front sub-image is extracted frame by frame, and the vehicle front panoramic image is determined frame by frame to synthesize the panoramic display image.

[0017] Further,

[0018] Dedistortion processing is performed on the Plantago seed image to correct the image distortion caused by camera distortion;

[0019] Adjust the brightness and contrast of each plantago image in the same frame to make the lighting of each plantago image consistent;

[0020] Using a feature detector, extracting feature points and descriptors of each Plantago image, and characterizing features in the Plantago image;

[0021] The matching device is used to match the feature points between different Plantago sub-images and compare the similarity of the descriptors to determine the corresponding relationship between the different Plantago sub-images;

[0022] According to the matching results of the feature points of the plantain sub-images, the homography matrix is ​​calculated to align the plantain sub-images, and the perspective transformation is performed using the homography matrix to map the plantain sub-images to the same coordinate system to correct the plantain sub-images;

[0023] The corrected Plantago sub-images of the same frame are stitched together, and the overlapping areas after the stitching of the Plantago sub-images are processed to remove the obvious seams at the splicing of the Plantago sub-images.

[0024] Further,

[0025] Based on the real-time vehicle speed and steering wheel angle data, the vehicle's driving trajectory is calculated using a single-track model to generate a predicted vehicle driving trajectory.

[0026] The predicted driving trajectory and the panoramic display image are fused to generate a first panoramic image.

[0027] Further,

[0028] Collect obstacle information in front of the vehicle and send it to the obstacle warning module;

[0029] The obstacle warning module combines the obstacle information in front of the vehicle with the first panoramic image to determine the precise location and attributes of the obstacle; wherein the obstacle information in front of the vehicle includes distance, location and attributes;

[0030] The position and attributes of the obstacle in front of the vehicle are marked in the first panoramic image using a box and / or text to generate a second panoramic image.

[0031] Further,

[0032] Convert the data format of the obstacle information in front of the vehicle, express the spatial coordinates of the obstacle in front of the vehicle in the form of a point cloud, and determine the point cloud of the obstacle in front of the vehicle;

[0033] Determine whether there is a spatial overlap between the vehicle's predicted driving trajectory and the point cloud of the obstacle in front of the vehicle;

[0034] If there is a spatial overlap between the predicted driving trajectory and the point cloud of the obstacle in front of the vehicle, a third panoramic image is generated based on the second panoramic image to issue a warning signal to the driver.

[0035] In a second aspect, based on the same inventive concept, an embodiment of the present disclosure further provides a vehicle front panoramic image display system, the system comprising: an image processing module, a trajectory prediction module, and an obstacle warning module;

[0036] An image processing module is used to obtain the image in front of the vehicle and generate a panoramic display image based on the image in front of the vehicle;

[0037] a trajectory prediction module, configured to obtain real-time vehicle speed and steering wheel angle data of the vehicle, calculate a predicted vehicle trajectory based on the real-time vehicle speed and steering wheel angle data, and generate a first panoramic image by combining the predicted trajectory with the panoramic display image;

[0038] The obstacle warning module is used to obtain information about obstacles in front of the vehicle, combine the information with the first panoramic image, determine the location of the obstacle in front of the vehicle, and mark the location of the obstacle in front of the vehicle in the first panoramic image to generate a second panoramic image. It is also used to generate a third panoramic image based on the second panoramic image when the location of the obstacle in front of the vehicle overlaps with the predicted driving trajectory.

[0039] On the third aspect, based on the same inventive concept, the embodiment of the present disclosure also provides a computer-readable storage medium, which stores one or more programs. When the one or more programs are executed, the aforementioned method for displaying a panoramic image in front of the vehicle can be implemented.

[0040] In a fourth aspect, based on the same inventive concept, an embodiment of the present disclosure further provides an electronic device, comprising a processor, a communication interface, the aforementioned computer-readable storage medium, and a communication bus; wherein the processor, the communication interface, and the computer-readable storage medium communicate with each other via the communication bus;

[0041] The processor is configured to execute the program stored in the aforementioned computer-readable storage medium.

[0042] Compared with the prior art, the present disclosure has the following advantages:

[0043] 1. It reduces the driver's visual blind spots during driving, allowing the driver to have a wider field of view in front of the car, and the image displayed to the driver can be better integrated with the real scene.

[0044] 2. The visual area completely covers the front cabin of the vehicle. The image displayed on the front windshield is clearer and less susceptible to external light and interference. The driver can obtain key road information without shifting his or her gaze or distracting his or her attention, thereby improving driving safety and convenience.

[0045] Other features and advantages of the present disclosure will be described in the following description, and in part will become apparent from the description, or will be understood by practicing the present disclosure. The purposes and other advantages of the present disclosure can be realized and obtained by the structures indicated in the description, claims and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings described below are some embodiments of the present disclosure, and other drawings can be obtained by those skilled in the art without creative labor under the premise of the drawings.

[0047] Figure 1 A flow chart of a vehicle front panoramic image display method provided by the embodiment of the present disclosure is shown in the figure.

[0048] Figure 2 A PHUD display schematic diagram provided by the embodiment of the present disclosure is shown in the figure.

[0049] Figure 3 A vehicle front panoramic image display system block diagram provided by the embodiment of the present disclosure is shown in the figure. DETAILED DESCRIPTION

[0050] In order to make the purpose, technical solutions and advantages of the embodiments of the present disclosure clearer, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below with reference to the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are some of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present disclosure.

[0051] In a first aspect, Figure 1 A flow chart of a vehicle front panoramic image display method provided by the embodiment of the present disclosure is shown in the figure. Figure 1 The embodiment of the present disclosure provides a vehicle front panoramic image display method, which comprises:

[0052] S1: Obtain a vehicle front image, and generate a panoramic display image according to the vehicle front image.

[0053] S2: Obtain real-time vehicle speed and steering wheel angle data of vehicle driving, calculate the predicted driving track of the vehicle according to the real-time vehicle speed and steering wheel angle data, and generate a first panoramic image in combination with the panoramic display image.

[0054] S3: Obtain vehicle front obstacle information, determine the vehicle front obstacle position in combination with the first panoramic image, mark the vehicle front obstacle position in the first panoramic image, and generate a second panoramic image.

[0055] In an embodiment of the present disclosure, an image acquisition module is used to acquire images in front of the vehicle, and the acquired images in front of the vehicle are preprocessed to generate a panoramic display image; a predicted driving trajectory of the vehicle is calculated based on the real-time vehicle speed and steering wheel angle data, and combined with the panoramic display image to generate a first panoramic image, which includes the predicted driving trajectory on the basis of the panoramic display image; the position of an obstacle in front of the vehicle is determined, and the position of the obstacle in front of the vehicle is marked in the first panoramic image to generate a second panoramic image, which includes the position of the obstacle in front of the vehicle on the basis of the first panoramic image; and a PHUD display module is used to display the second panoramic image to the driver.

[0056] like Figure 2 As shown, existing WHUD (Windshield Head-Up Display) technology floats above the vehicle's front cabin and displays at eye level, directly in the driver's normal field of view. The WHUD image is located in front of the driver's line of sight and may overlap with the actual road scene, interfering with the driver's normal field of view, causing visual confusion and distraction. Furthermore, the information displayed on the WHUD may be affected by ambient light, reflections, and other factors, reducing image clarity and visibility.

[0057] Typically, a car's instrument panel is located in the center or front of the driver's cockpit. To view the image ahead, the driver must shift their gaze from the road ahead to the instrument panel. This shift in focus can divert the driver's attention from the road, reducing their ability to observe and be alert to the road ahead, increasing driving risks. This can lead to missed critical information or even dangerous situations, especially at high speeds or in complex road conditions.

[0058] Therefore, this solution uses PHUD (Panoramic Head-Up Display) technology to display a panoramic image of the vehicle's front view to the driver. The PHUD image is projected between the WHUD display and the instrument panel, and the PHUD display technology's visible area completely covers the vehicle's front cabin. The PHUD display module features a highly reflective black ink area printed on the underside of the imaging windshield. This design makes the image displayed on the front windshield clearer and less susceptible to external light and interference. By using PHUD display technology, drivers can obtain key road information without diverting their gaze or distracting their attention, improving driving safety and convenience.

[0059] In some examples, obtaining a front image of a vehicle and generating a panoramic display image based on the front image of the vehicle specifically includes:

[0060] S11: Use the image acquisition module to capture the front image of the vehicle, convert it into front image data, and send the front image data to the image processing module.

[0061] S12: The image processing module extracts the sub-images in front of the vehicle captured by each sub-camera of the image acquisition module based on the image data in front of the vehicle.

[0062] Specifically, the image acquisition module includes multiple sub-cameras for capturing images in front of the vehicle at different angles. Therefore, the sub-images in front of the vehicle captured by each sub-camera of the image acquisition module are extracted for subsequent determination of the panoramic display image.

[0063] S13: extracting the plantago sub-images corresponding to the same frame of each plantago sub-image, performing image processing on each plantago sub-image, and determining a panoramic image of the front of the vehicle.

[0064] S14: extracting the vehicle front sub-image corresponding to the same frame of each vehicle front sub-image frame by frame, determining the vehicle front panoramic image frame by frame, and synthesizing the panoramic display image.

[0065] Specifically, the vehicle's front sub-image is split frame by frame, and each of these split images is processed frame by frame. After processing each frame, a complete image is synthesized, which is the panoramic display image. The panoramic display image includes blind spots that the driver cannot directly observe, such as the car's A-pillar, thus effectively reducing the driver's blind spots during driving.

[0066] In some examples, extracting the plankton sub-images corresponding to the same frame of each plankton sub-image, performing image processing on each plankton sub-image, and determining a panoramic image of the front of the vehicle specifically includes:

[0067] S131: Dedistorting the Plantago seed image to correct image distortion caused by camera distortion.

[0068] S132: adjusting the brightness and contrast of each plantago seed image in the same frame to make the lighting of each plantago seed image consistent.

[0069] S133: Using a feature detector, extracting feature points and descriptors of each Plantago image to characterize features in the Plantago image.

[0070] S134: Matching feature points between different Plantago sub-images using a matcher, and comparing similarities of descriptors to determine the corresponding relationship between the different Plantago sub-images.

[0071] Specifically, feature detectors include ORB (Oriented FAST and Rotated BRIEF) or SIFT (Scale-Invariant Feature Transform), and matchers include FLANN (Fast Library for Approximate Nearest Neighbors) or BFMatcher (Brute-Force Matcher).

[0072] S135: Calculate the homography matrix based on the matching results of the feature points of the plantago sub-images, align the plantago sub-images, and use the homography matrix to perform perspective transformation to map the plantago sub-images to the same coordinate system to correct the plantago sub-images.

[0073] S136: stitching the corrected Plantago sub-images of the same frame, processing the overlapping areas after stitching the Plantago sub-images, and removing the obvious seams at the splicing of the Plantago sub-images.

[0074] In some examples, obtaining real-time vehicle speed and steering wheel angle data, calculating a predicted vehicle trajectory based on the real-time vehicle speed and steering wheel angle data, and combining the panoramic display image to generate a first panoramic image specifically includes:

[0075] S21: Based on the real-time vehicle speed and steering wheel angle data of the vehicle, a single-track model is used to calculate the vehicle's driving trajectory and generate a predicted vehicle driving trajectory.

[0076] S22: Fusing the predicted driving trajectory with the panoramic display image to generate a first panoramic image.

[0077] In some examples, obtaining information about an obstacle in front of the vehicle, combining the information with the first panoramic image, determining the location of the obstacle in front of the vehicle, and marking the location of the obstacle in front of the vehicle in the first panoramic image to generate a second panoramic image specifically includes:

[0078] S31: Collecting obstacle information in front of the vehicle and sending the obstacle information in front of the vehicle to the obstacle warning module.

[0079] S32: The obstacle warning module combines the information of the obstacle in front of the vehicle with the first panoramic image to determine the precise location and attributes of the obstacle.

[0080] Specifically, obstacle information in front of the vehicle includes distance, location, and attributes. Computer vision algorithms can be used to identify obstacles appearing in the first panoramic image. These algorithms can automatically analyze various objects and obstacles in the image in front of the vehicle, accurately identify their locations and features, and determine the attributes of the obstacles in front of the vehicle by identifying the obstacle features. Alternatively, an obstacle recognition module can be used to directly collect obstacle information in front of the vehicle. Obstacle attributes include obstacle type (e.g., person, vehicle, cone), and corresponding information such as height and width.

[0081] S33: Mark the position and attributes of the obstacle in front of the vehicle in the first panoramic image using a box and / or text to generate a second panoramic image.

[0082] In some examples, the method for displaying a panoramic image in front of the vehicle also includes generating a third panoramic image based on the second panoramic image to issue a warning signal to the driver when the location of an obstacle in front of the vehicle overlaps with the predicted driving trajectory. In addition to displaying a clearer and more comprehensive panoramic image in front of the vehicle, a specific warning may also be provided to the driver, prompting the driver to slow down, stop, or evade the obstacle.

[0083] Furthermore, when the position of the obstacle in front of the vehicle overlaps with the predicted driving trajectory, a third panoramic image is generated based on the second panoramic image to issue a warning signal to the driver, specifically including:

[0084] S41: Convert the data format of the obstacle information in front of the vehicle, express the spatial coordinates of the obstacle in front of the vehicle in the form of a point cloud, and determine the point cloud of the obstacle in front of the vehicle.

[0085] S42: Determine whether there is a spatial overlap between the predicted driving trajectory of the vehicle and the point cloud of the obstacle in front of the vehicle.

[0086] S43: If there is a spatial overlap between the predicted driving trajectory and the point cloud of the obstacle in front of the vehicle, a third panoramic image is generated based on the second panoramic image to issue a warning signal to the driver.

[0087] Example 1:

[0088] S101: Acquire a front image of the vehicle, and generate a panoramic display image based on the front image of the vehicle.

[0089] The image acquisition module in the present disclosure includes multiple sub-cameras for shooting in multiple directions and angles. The image acquisition module captures the front image of the vehicle and converts it into front image data, which is then transmitted to the image processing module for processing.

[0090] The image processing module receives the front vehicle image data, extracts the front vehicle sub-image corresponding to the same frame of each front vehicle sub-image according to the front vehicle image data, performs image processing on each front vehicle sub-image, determines a front vehicle panoramic image, and uses the same frame of the front vehicle sub-image to determine a front vehicle panoramic image as an example for description:

[0091] The front vehicle sub-image is subjected to distortion correction processing to correct the image distortion caused by the camera distortion. The brightness and contrast of each front vehicle sub-image of the same frame are adjusted to make the light of each front vehicle sub-image consistent, so as to ensure consistent visual effects of each front vehicle sub-image. A feature detector is used to extract feature points and descriptors of each front vehicle sub-image to represent the features in the front vehicle sub-image. A matcher is used to match the feature points between different front vehicle sub-images and compare the similarity of the descriptors to determine the correspondence between different front vehicle sub-images. The feature detector includes ORB (Oriented FAST and Rotated BRIEF) or SIFT (Scale-Invariant Feature Transform), and the matcher includes FLANN (Fast Library for Approximate Nearest Neighbors) or BFMatcher (Brute-Force Matcher).

[0092] According to the matching result of the feature points of the front vehicle sub-image, a homography matrix is calculated to align each front vehicle sub-image, and a perspective transformation is performed using the homography matrix to map each front vehicle sub-image to the same coordinate system to correct the front vehicle sub-image. The corrected front vehicle sub-images of the same frame are spliced, and the overlapping area after splicing the front vehicle sub-images is processed to remove the obvious joints at the splicing position of the front vehicle sub-images, and finally the front vehicle panoramic image is determined.

[0093] Using the above method, the front vehicle sub-image corresponding to the same frame of each front vehicle sub-image is extracted frame by frame, and the front vehicle panoramic image is determined frame by frame. The front vehicle panoramic images of consecutive frames are combined to synthesize a panoramic display image.

[0094] S102: Calculate the predicted driving trajectory of the vehicle, combine the panoramic display image, and determine a first panoramic image.

[0095] In the disclosed embodiment, the vehicle's driving trajectory is calculated based on the vehicle's real-time speed and steering wheel angle data. During the vehicle's driving process, the vehicle's trajectory is calculated using a single-track model using the real-time speed and steering wheel angle data. The single-track model is a kinematic model of the vehicle and can be used to accurately calculate the vehicle's forward trajectory. Using the vehicle's real-time speed, steering wheel angle, and wheelbase, the single-track model can calculate the vehicle's turning radius and update the vehicle's current position within continuous time steps, thereby generating a predicted driving trajectory for the vehicle. The predicted driving trajectory can be combined with a panoramic display image and clearly displayed in the driver's field of view through the PHUD display module, thereby improving the driver's visibility and understanding.

[0096] S103: Determine the position of the obstacle in front of the vehicle, and mark the position of the obstacle in front of the vehicle in the first panoramic image to generate a second panoramic image.

[0097] In the disclosed embodiments, an obstacle recognition module is used to collect information about obstacles in front of the vehicle. This information includes distance, location, and attributes. The location and attributes of the obstacle in front of the vehicle are annotated in the first panoramic image using boxes and / or text to indicate the precise location and attributes of the obstacle, generating a second panoramic image. The obstacle recognition module can be an ADAS (Advanced Driver Assistance System).

[0098] S104: When the position of the obstacle in front of the vehicle overlaps with the predicted driving trajectory, a third panoramic image is generated.

[0099] In the disclosed embodiment, an ADAS (Advanced Driver Assistance System) is used as the obstacle recognition module. Sensors in the ADAS, such as lidar and cameras, acquire obstacle information from the vehicle's surroundings, convert this information into a data format, and represent the obstacle's spatial coordinates in the form of a point cloud, thereby determining an obstacle point cloud. Based on the spatial overlap between the vehicle's predicted trajectory and the obstacle point cloud, the system automatically identifies whether the vehicle's predicted trajectory conflicts with an obstacle. If the vehicle's predicted trajectory overlaps with the obstacle point cloud, a third panoramic image is generated based on the second panoramic image. The third panoramic image highlights the obstacle and issues a warning signal, alerting the driver to the potential danger.

[0100] Furthermore, taking a vehicle driving on the road as an example, the ADAS system's front camera first captures the image of the vehicle in front, and the ADAS system's front radar uses a visual algorithm to determine obstacles or distance information of the vehicle in front, and marks obstacles in front of the vehicle in real time. The predicted driving trajectory is determined based on the vehicle's driving speed, steering wheel angle information, and wheelbase. When the predicted driving trajectory overlaps with the marked obstacle, the system highlights the overlapping area as an obstacle warning, that is, generates a third panoramic image. Combined with the PHUD display module, the third panoramic image containing the warning information is directly presented in the front cabin, thereby effectively reminding the driver of potential obstacle risks.

[0101] In the disclosed embodiment, a driver monitoring system is used to collect the driver's eye position data. The eye position data includes eye coordinates and gaze point information. To ensure that the driver's eye position data can be obtained in real time, the driver monitoring system periodically obtains and updates the eye position data to ensure that the driver's eye position data is updated in real time when the driver's line of sight changes.

[0102] Based on the eye position data obtained by the driver monitoring system, the PHUD display module adjusts the position and angle of the panoramic image displayed in front of the vehicle so that the projected panoramic image is integrated with the driver's actual line of sight to avoid visual conflict.

[0103] Secondly, based on the same inventive concept, Figure 3 A block diagram of a front-view panoramic image display system provided by an embodiment of the present disclosure is shown as follows: Figure 3 As shown, the embodiment of the present disclosure also provides a panoramic image display system in front of the vehicle, and the system includes: an image processing module, a trajectory prediction module and an obstacle warning module. The image processing module is used to obtain the image in front of the vehicle and generate a panoramic display image based on the image in front of the vehicle. The trajectory prediction module is used to obtain the real-time vehicle speed and steering wheel angle data of the vehicle, calculate the predicted driving trajectory of the vehicle based on the real-time vehicle speed and steering wheel angle data, and generate a first panoramic image in combination with the panoramic display image. The obstacle warning module is used to obtain the obstacle information in front of the vehicle, and the obstacle information in front of the vehicle is combined with the first panoramic image to determine the position of the obstacle in front of the vehicle, and the position of the obstacle in front of the vehicle is marked in the first panoramic image to generate a second panoramic image; the obstacle warning module is also used to generate a third panoramic image based on the second panoramic image when there is an overlap between the position of the obstacle in front of the vehicle and the predicted driving trajectory.

[0104] Furthermore, the front panoramic image display system also includes a PHUD display module for displaying a second panoramic image to the driver, and when the position of the obstacle in front of the vehicle overlaps with the predicted driving trajectory, displaying a third panoramic image to send a warning signal to the driver.

[0105] In the embodiments of the present disclosure, the vehicle front panoramic image display system further comprises an image acquisition module and an obstacle identification module. The image acquisition module is configured to capture a vehicle front image and convert the vehicle front image into vehicle front image data, and send the vehicle front image data to the image processing module. The obstacle identification module is configured to acquire vehicle front obstacle information, wherein the vehicle front obstacle information comprises distance, position, attribute, etc.

[0106] In actual deployment of the vehicle front panoramic image display system, the system modules are divided into a data layer, a business logic layer and an application layer. The data layer comprises the image acquisition module and the obstacle identification module. The business logic layer comprises the image processing module, the trajectory prediction module and the obstacle warning module. The application layer comprises the PHUD display module, which is configured to display the fused display panoramic image, the predicted driving trajectory, the obstacle position and the possible warning information.

[0107] In a third aspect, based on the same inventive concept, the embodiments of the present disclosure further provide a computer-readable storage medium, which stores one or more programs, and when the one or more programs are executed, the vehicle front panoramic image display method described above can be implemented.

[0108] In a fourth aspect, based on the same inventive concept, the embodiments of the present disclosure further provide an electronic device, which comprises a processor, a communication interface, the aforementioned computer-readable storage medium and a communication bus. The processor, the communication interface and the computer-readable storage medium communicate with each other through the communication bus. The processor is configured to execute the program stored in the computer-readable storage medium.

[0109] It should be noted that the electrical connection between the above-mentioned various units does not necessarily mean the connection between the lines, the indirect connection mode, as long as the purpose of the present disclosure can be achieved.

[0110] Although the present disclosure has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that the technical solutions recorded in the foregoing embodiments can be modified, or some technical features can be replaced by equivalent ones; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present disclosure.

Claims

1. A method for displaying a panoramic image in front of a vehicle, characterized in that: The method comprises: Acquire a front image of the vehicle and generate a panoramic display image based on the front image of the vehicle; specifically, use an image acquisition module to capture the front image of the vehicle and convert it into front image data, and send the front image data to an image processing module; the image processing module extracts the front sub-images of the vehicle taken by each sub-camera of the image acquisition module based on the front image data, each sub-camera taking the front image of the vehicle at a different angle; extract the front sub-images corresponding to the same frame of each front sub-image, perform image processing on each front sub-image, and determine the front panoramic image of the vehicle; extract the front sub-images corresponding to the same frame of each front sub-image frame by frame, determine the front panoramic image of the vehicle frame by frame, and synthesize the panoramic display image; Acquiring real-time vehicle speed and steering wheel angle data of the vehicle, calculating a predicted vehicle trajectory based on the real-time vehicle speed and steering wheel angle data, and combining the predicted trajectory with the panoramic display image to generate a first panoramic image; Obtain information about an obstacle in front of the vehicle, combine the information with the first panoramic image, determine the location of the obstacle in front of the vehicle, mark the location of the obstacle in front of the vehicle in the first panoramic image, generate a second panoramic image, and display the second panoramic image on the PHUD display module.

2. The method according to claim 1, characterized in that The method for displaying a panoramic image in front of a vehicle further includes: When the position of the obstacle in front of the vehicle overlaps with the predicted driving trajectory, a third panoramic image is generated based on the second panoramic image, and the PHUD display module displays the third panoramic image to issue a warning signal to the driver.

3. The method according to claim 1, characterized in that The extracting of the plantago sub-images corresponding to the same frame of each plantago sub-image, performing image processing on each plantago sub-image, and determining a panoramic image of the front of the vehicle includes: performing a dedistortion process on the plantago image to correct image distortion caused by camera distortion; Adjusting the brightness and contrast of each of the plantago images in the same frame to make the lighting of each of the plantago images consistent; Using a feature detector, extracting feature points and descriptors of each of the plantago seed images to characterize features in the plantago seed images; Matching the feature points between different Plantago sub-images using a matcher, and comparing the similarities of the descriptors to determine the corresponding relationship between the different Plantago sub-images; Calculating a homography matrix based on the matching results of the feature points of the Plantago sub-images, aligning the Plantago sub-images, and performing a perspective transformation using the homography matrix to map the Plantago sub-images to the same coordinate system, thereby correcting the Plantago sub-images; The corrected Plantago sub-images of the same frame are spliced ​​together, and overlapping areas of the spliced ​​Plantago sub-images are processed to remove obvious seams at the spliced ​​Plantago sub-images.

4. The method according to claim 1, wherein The method of obtaining real-time vehicle speed and steering wheel angle data, calculating a predicted vehicle trajectory based on the real-time vehicle speed and steering wheel angle data, and generating a first panoramic image by combining the panoramic display image, includes: Calculating the vehicle's driving trajectory using a single-track model based on the vehicle's real-time speed and steering wheel angle data to generate the vehicle's predicted driving trajectory; The predicted driving trajectory and the panoramic display image are fused to generate the first panoramic image.

5. The method according to claim 1, wherein The obtaining of information about an obstacle in front of the vehicle, combining the information about the obstacle in front of the vehicle with the first panoramic image, determining a position of the obstacle in front of the vehicle, and marking the position of the obstacle in front of the vehicle in the first panoramic image to generate a second panoramic image includes: Collecting the obstacle information in front of the vehicle and sending the obstacle information in front of the vehicle to the obstacle warning module; The obstacle warning module combines the obstacle information in front of the vehicle with the first panoramic image to determine the precise location and attributes of the obstacle; wherein the obstacle information in front of the vehicle includes distance, location, and attributes; The position and attributes of the obstacle in front of the vehicle are marked in the first panoramic image using a box and / or text to generate the second panoramic image.

6. The method according to claim 2, characterized in that When the position of the obstacle in front of the vehicle overlaps with the predicted driving trajectory, generating a third panoramic image based on the second panoramic image to issue a warning signal to the driver includes: Convert the data format of the obstacle information in front of the vehicle, express the spatial coordinates of the obstacle in front of the vehicle in the form of a point cloud, and determine the point cloud of the obstacle in front of the vehicle; Determining whether there is a spatial overlap between the predicted driving trajectory of the vehicle and the point cloud of the obstacle in front of the vehicle; If there is a spatial overlap area between the predicted driving trajectory and the point cloud of the obstacle in front of the vehicle, the third panoramic image is generated based on the second panoramic image to issue a warning signal to the driver.

7. A front-view panoramic image display system, characterized in that: The system includes: an image processing module, a trajectory prediction module and an obstacle warning module; The image processing module is used to obtain a front image of the vehicle and generate a panoramic display image based on the front image of the vehicle; specifically, the image acquisition module is used to capture the front image of the vehicle and convert it into front image data, and the front image data is sent to the image processing module; the image processing module extracts the front sub-images of the vehicle taken by each sub-camera of the image acquisition module based on the front image data, and each sub-camera takes the front image of the vehicle at a different angle; extracts the front sub-images corresponding to the same frame of each front sub-image, performs image processing on each front sub-image, and determines the front panoramic image of the vehicle; extracts the front sub-images corresponding to the same frame of each front sub-image frame by frame, and determines the front panoramic image of the vehicle frame by frame to synthesize the panoramic display image; The trajectory prediction module is configured to obtain real-time vehicle speed and steering wheel angle data of the vehicle, calculate a predicted vehicle trajectory based on the real-time vehicle speed and steering wheel angle data, and generate a first panoramic image in combination with the panoramic display image; The obstacle warning module is used to obtain information about obstacles in front of the vehicle, combine the information about obstacles in front of the vehicle with the first panoramic image, determine the position of the obstacle in front of the vehicle, mark the position of the obstacle in front of the vehicle in the first panoramic image, and generate a second panoramic image; and is also used to generate a third panoramic image based on the second panoramic image when the position of the obstacle in front of the vehicle overlaps with the predicted driving trajectory.

8. A computer-readable storage medium storing one or more programs, characterized in that: When the one or more programs are executed, the method for displaying a front panoramic image of a vehicle as described in any one of claims 1 to 6 can be implemented.

9. An electronic device comprising a processor, a communication interface, the computer-readable storage medium according to claim 8, and a communication bus; wherein: The processor, the communication interface, and the computer-readable storage medium communicate with each other via a communication bus; It is characterized in that The processor is configured to execute a program stored in a computer-readable storage medium.

Images