Panoramic image generation method, storage medium, vehicle-mounted equipment and vehicle

By constructing depth maps and sequentially independent translucent rendering algorithms, visual chaos and chassis perspective problems in panoramic surround view systems are solved, and clear panoramic images and real body rendering are achieved, improving visual effects and driving safety.

CN119067841APending Publication Date: 2024-12-03SHENZHEN DEEPROUTE AI CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411068298.X
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-06
Publication Date
2024-12-03

AI Technical Summary

Technical Problem

Traditional panoramic surround view systems have visual chaos and unnatural transparency when dealing with rain screens, mists or translucent obstacles, and are difficult to provide a clear perspective view of the vehicle chassis.

Method used

By obtaining the scene images around the vehicle, the initial panoramic image is constructed, and the depth map is constructed based on the depth information. The sequentially independent translucent rendering algorithm is used to mix the vehicle digital model into the panoramic image, combining the fragment shader and cumulative buffer for correct translucent objects Sort, simulate the perspective screen of the vehicle chassis.

Benefits of technology

It realizes providing clear panoramic images in complex visual environments, avoiding ghosting and unnatural effects caused by wrong drawing order, improving the visual effects of the vehicle in panoramic images, and enhancing the real-time perception of chassis perspective.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119067841A_ABST
    Figure CN119067841A_ABST
Patent Text Reader

Abstract

The invention discloses a panoramic image generation method, a storage medium, vehicle-mounted equipment and a vehicle, and the method comprises the steps: obtaining all scene images around the vehicle, and constructing an initial panoramic image surrounding the vehicle through all scene images; obtaining depth information of each scene image, and constructing a depth map corresponding to the initial panoramic image based on the depth information; and mixing a preset vehicle digital model into the initial panoramic image by adopting a sequence-independent semitransparent rendering algorithm based on the information of the depth map to form a panoramic image containing the vehicle. Through the above mode, the problem of ghosting or unnatural effect of the displayed vehicle image can be avoided, and the visual effect of the panoramic image is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of image processing, and in particular to a panoramic image generation method, a storage medium, a vehicle-mounted device and a vehicle. Background Art

[0002] As one of the key technologies to improve the safety of modern vehicles, the panoramic view system captures and synthesizes images of the surrounding environment through cameras integrated on all sides of the vehicle body, providing the driver with 360-degree field of view coverage.

[0003] However, traditional rendering mechanisms often cause visual confusion, such as ghosting or unnatural transparency effects, when dealing with rain curtains, mist, or semi-transparent obstacles, because they rely on a specific rendering order. In addition, when implementing the vehicle chassis perspective function, existing systems are often limited by the limitations of image processing technology and find it difficult to provide a clear and real-time perspective view.

[0004] Therefore, developing an efficient semi-transparent rendering solution that is independent of object drawing order and does not require complex dynamic tracking and a chassis perspective view to optimize the vehicle's panoramic view is the key to improving the visual effects of panoramic surround images. Summary of the invention

[0005] The present application mainly provides a panoramic image generation method, a storage medium, a vehicle-mounted device and a vehicle to solve the problem of poor visual effects of panoramic surround images.

[0006] In order to solve the above technical problems, a technical solution adopted in the present application is: to provide a panoramic image generation method, including: obtaining scene images around a vehicle, and constructing an initial panoramic image surrounding the vehicle from each of the scene images; obtaining depth information of each of the scene images, and constructing a depth map corresponding to the initial panoramic image based on the depth information; and mixing a preset vehicle digital model into the initial panoramic image using an order-independent semi-transparent rendering algorithm based on the information of the depth map to form a panoramic image containing the vehicle.

[0007] By constructing a depth map containing depth information, the digital model of the vehicle is mixed into the panoramic image in the form of a semi-transparent object, so that the mixing process does not depend on a certain drawing order, avoiding visual confusion, and making the panoramic image containing the vehicle provide clear visual effects even in complex visual environments.

[0008] In some embodiments, the preset vehicle digital model is mixed into the initial panoramic image using an order-independent semi-transparent rendering algorithm based on the information of the depth map, including: rendering each pixel of the vehicle digital model through a fragment shader to obtain color information and depth information of each pixel; based on the depth map and the depth information of each pixel, mixing the color information of each pixel on the vehicle digital model into the corresponding pixel point in the initial panoramic image.

[0009] By using a fragment shader to process translucent objects on each pixel, and using an accumulation buffer to store and blend the color and depth information of these translucent objects, the depth information provided by the depth map is used to correctly sort the translucent objects. This makes the blending of translucent objects independent of a certain drawing order, but is done according to the sorting of their depth information, thus avoiding image display problems caused by incorrect order and improving the visual effects of panoramic images.

[0010] In some embodiments, based on the depth map and the depth information of each pixel, the color information of the pixel is mixed with the corresponding pixel in the initial panoramic image; based on the depth map, the occlusion relationship between the pixels on the vehicle digital model and the mixing order obtained according to the depth information are determined; and the color information of each pixel on the vehicle digital model is mixed with the corresponding pixel in the initial panoramic image according to the occlusion relationship and the mixing order.

[0011] Semi-transparent objects are blended into the panoramic image through order-independent semi-transparent rendering, so that the blending of semi-transparent objects does not depend on their drawing order, avoiding the problem of image ghosting or unnatural effects caused by incorrect drawing order.

[0012] In some embodiments, after mixing the color information of each pixel on the vehicle digital model to the corresponding pixel point in the initial panoramic image according to the occlusion relationship and the mixing order, it also includes: comparing the panoramic image obtained by mixing with a reference image to evaluate the accuracy of the depth map and the correctness of the mixing order for the vehicle digital model; adjusting the construction method of the depth map and the sorting method of the vehicle digital model based on the accuracy of the depth map and the correctness of the mixing order.

[0013] By evaluating the accuracy of the depth map and the correctness of the mixing order of the vehicle digital model, the depth map construction and the sorting method of the vehicle digital model can be adjusted to improve the accuracy of the vehicle digital model during the sorting process, which is beneficial to improving the visual effect after mixing.

[0014] In some embodiments, after forming the panoramic image including the vehicle, the method further includes: using image-based lighting and physically based rendering materials to perform realistic vehicle body rendering on the vehicle body in the panoramic image.

[0015] Image-based lighting technology is used to simulate the impact of complex lighting environments on the brightness of the vehicle surface, so that the vehicle can show more realistic and delicate reflection and shadow effects under various lighting conditions, improving the overall visual realism of the vehicle surface. Physically based rendering materials are used to ensure that the rendering effect of the vehicle surface material follows the laws of physics, so that the vehicle surface has the reflective characteristics of materials such as metal or paint, and maintains consistent visual effects under different lighting and viewing angles, further improving the visual effect of the vehicle surface.

[0016] In some embodiments, the panoramic image generation method further includes: simulating a chassis perspective picture of the vehicle in the panoramic image based on acquiring image data and sensor data corresponding to a lane.

[0017] The ground conditions under the chassis are simulated through lane-related data in previously stored scene images and sensor data acquired in real time. The scene under the chassis is virtualized on the panoramic image to create a perspective visual effect, allowing users to intuitively understand the ground conditions through the interactive interface, further optimizing the visual effect of the panoramic image.

[0018] In some embodiments, constructing an initial panoramic image surrounding the vehicle from each of the scene images includes: preprocessing the scene images through an image processing algorithm, including: denoising, color balancing and exposure correction of the scene images; and stitching the preprocessed scene images to form the initial panoramic image.

[0019] By performing preliminary optimization such as noise reduction, color balance and exposure correction on the scene images collected by cameras at various angles, the optimized scene images are stitched together to obtain a clearer panoramic image covering the full range of vehicle viewing angles.

[0020] In some embodiments, obtaining the depth information of each of the scene images includes: calculating the depth information of each pixel in each of the scene images; constructing a depth map corresponding to the initial panoramic image based on the depth information includes: mapping the depth information of each pixel in each of the scene images to the corresponding pixel in the initial panoramic image to form a depth map of the initial panoramic image.

[0021] The depth information of each pixel is obtained through the camera parameters and the images taken by the camera, and a depth map containing the depth information is constructed, which provides a depth information basis for the correct sorting of semi-transparent objects in the panoramic image.

[0022] In order to solve the above technical problems, another technical solution adopted by the present application is: providing a storage medium on which program data is stored, and when the program data is executed by a processor, the steps of the panoramic image generation method as described above are implemented.

[0023] The beneficial effects of the storage medium refer to the beneficial effects of the panoramic image generation method mentioned above, which will not be repeated here.

[0024] The present application also provides a vehicle-mounted device, including a processor and a memory connected to each other, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of the panoramic image generation method as described above are implemented.

[0025] The vehicle-mounted device is a computer-type device. The beneficial effects of the vehicle-mounted device refer to the beneficial effects of the panoramic image generation method mentioned above, which will not be repeated here.

[0026] The present application also provides a vehicle, comprising the above-mentioned vehicle-mounted device. During the driving process of the vehicle, the steps of the above-mentioned panoramic image generation method can be completed through the control of the vehicle-mounted device.

[0027] The beneficial effects of the present application are as follows: Different from the prior art, the present application discloses a panoramic image generation method, storage medium, vehicle-mounted equipment and vehicle. An initial panoramic image surrounding the vehicle is constructed by acquiring images of scenes around the vehicle, and depth information is acquired to form a depth map, so that the digital model of the vehicle can be mixed into the initial panoramic image in the form of a semi-transparent object according to the depth information contained in the depth map to form a complete panoramic image. The depth-based order-independent semi-transparent rendering enables the vehicles on the panoramic image to have better visual effects, avoids the problem of ghosting or unnatural effects in the displayed vehicle image, and improves the visual effect of the panoramic image. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the prior art descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative work, among which:

[0029] Figure 1 It is a flowchart of an embodiment of a panoramic image generation method provided by the present application;

[0030] Figure 2 Yes Figure 1 A schematic flow chart of an embodiment of step 10 in the panoramic image generation method shown;

[0031] Figure 3 Yes Figure 1 A schematic diagram of a flow chart of an embodiment of step 20 in the panoramic image generation method shown;

[0032] Figure 4 Yes Figure 1 A schematic diagram of a flow chart of an embodiment of step 30 in the panoramic image generation method shown;

[0033] Figure 5 Yes Figure 4 A schematic diagram of a flow chart of an embodiment of step 32 in the panoramic image generation method shown;

[0034] Figure 6 Yes Figure 4 A schematic flow chart of another embodiment of step 32 in the panoramic image generation method shown;

[0035] Figure 7 is a flowchart of another embodiment of the panoramic image generation method provided by the present application;

[0036] Figure 8 It is a structural schematic diagram of an embodiment of a storage medium provided by the present application;

[0037] Fig. 9 It is a structural schematic diagram of an embodiment of the vehicle-mounted device provided by the present application;

[0038] Fig.10 It is a structural schematic diagram of a vehicle embodiment provided by the present application. DETAILED DESCRIPTION

[0039] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0040] The terms "first", "second", "third" in the embodiments of the present application are only used for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features. Thus, the features defined as "first", "second", "third" can expressly or implicitly include at least one of the features. In the description of the present application, the meaning of "multiple" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device comprising a series of steps or units is not limited to the listed steps or units, but optionally also includes steps or units that are not listed, or optionally also includes other steps or units inherent to these processes, methods, products or devices.

[0041] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0042] See also Figure 1 , Figure 1 : is a flowchart of a first embodiment of a panoramic image generation method provided by the present application, the panoramic image generation method comprising:

[0043] Step 10: Acquire images of scenes around the vehicle, and construct an initial panoramic image around the vehicle from the images of the scenes.

[0044] By configuring cameras at the front, back, left and right of the vehicle, all-round scene images around the vehicle are acquired and preliminarily processed, and based on the scene images around the vehicle, an initial panoramic image around the vehicle is constructed.

[0045] The scene image is the scene in a certain direction of the vehicle captured by the camera. Each scene image can cover all angles around the vehicle to form an Around View Monitor (AVM). When capturing scene images, it is necessary to ensure that all cameras capture images at the same time or almost at the same time to reduce image misalignment caused by time difference.

[0046] The acquired scene images will be stored in the cloud or local image storage for subsequent calls. The image storage only stores scene images within a certain time period. Scene images in the image storage that exceed a certain time will be replaced by newly stored scene images.

[0047] Specifically, the scene image can be acquired by a wide-angle camera, a pinhole camera or other high-definition cameras that can be configured to cover the scene around the vehicle.

[0048] The initial panoramic image is an image that surrounds the vehicle but does not contain the vehicle, which is constructed by optimizing, cutting, splicing and superimposing scene images from various angles. Specifically, in the process of constructing the initial panoramic image, when the scene image captured by the camera contains part of the vehicle, the features of the vehicle in the image are extracted through image processing technology to distinguish the vehicle part from the non-vehicle part in the image, calculate the border of the vehicle part and the non-vehicle part in the scene image, and cut the part of the scene image containing the vehicle along the border to form a panoramic image in which the edge of the area to be mixed is consistent with the surface position of the vehicle, so as to avoid repeated mixing of part of the vehicle area in the panoramic image and cause confusion in the mixing position, so as to ensure that the digital model of the vehicle can be correctly mixed according to the depth in the initial panoramic image.

[0049] Further, see Figure 2 , constructing an initial panoramic image around the vehicle from each scene image also includes:

[0050] Step 11: Preprocess the scene image through image processing algorithm.

[0051] Since the captured image may be blurred, distorted or insufficiently bright, after acquiring the scene image, it is necessary to pre-process the scene image through an image processing algorithm to obtain accurate and clear image information.

[0052] Specifically, the method of preprocessing the scene image by using the image processing algorithm includes: performing noise reduction, color balance, exposure correction or distortion correction on the scene image.

[0053] Noise reduction is a commonly used technique in image processing to reduce noise in images and improve image clarity and quality. Noise reduction can be performed using software such as image processing (Adobe Photoshop, PS), photo lab (DxO PhotoLab) or noise reduction AI (DeNoise AI), which usually provide noise reduction functions that can automatically detect and remove noise in images.

[0054] Color balance is the process of adjusting the proportion of various color components in an image to make the image color more natural and realistic. Improper color balance may cause color cast or color distortion in the image. You can use functions such as "Auto White Balance" or "Auto Tone" in LR (Adobe Lightroom) or image processing to automatically adjust the color balance of the image based on the color distribution and brightness information in the image.

[0055] Exposure correction is the process of adjusting the brightness of an image to ensure that the details in the image are fully displayed without being too bright or too dark.

[0056] Distortion correction is a technique used to correct distortion or distortion in images or videos. Specifically, the distortion can be modeled using a mathematical model, and the correction parameters can be calculated by an algorithm. For example, a polynomial transformation or correction function can be used to reconstruct the distorted image. By correcting the distortion, the quality of the image and the accuracy of the information can be improved.

[0057] By preprocessing the scene image and repairing the underexposed or overexposed image, the acquired scene image is made clearer, the quality of the scene image and the accuracy of the information are improved, and the image color is made more natural and harmonious.

[0058] Step 12: Stitch the preprocessed scene images to form an initial panoramic image.

[0059] Characteristic feature points are detected on each preprocessed scene image, and the feature points are matched between different scene images to determine the positional relationship between adjacent images. Based on the positional relationship between the scene images, an initial panoramic image is formed after transformation and fusion.

[0060] For example, if a bicycle is included in both the rearward image and the leftward image of the vehicle, the bicycle is used as a characteristic feature point on the rearward image and the leftward image to determine the positional relationship between the rearward image and the leftward image.

[0061] Optionally, after the initial panoramic image is formed, the initial panoramic image is optimized and adjusted, including adjusting brightness, contrast, color balance, etc., to ensure the overall quality of the panoramic image.

[0062] By stitching the pre-processed scene images to form an initial panoramic image, the field of view limitation can be solved and the image quality and information expression capability can be improved.

[0063] Step 20: Obtain depth information of each scene image, and construct a depth map corresponding to the initial panoramic image based on the depth information.

[0064] Based on the parameter data of the camera that captures the scene image, the depth information of the scene image is obtained. The depth information of the scene images captured at multiple different angles is combined with more visual clues such as feature matching and triangulation under different viewing angles to reconstruct the three-dimensional structure of the object and construct a depth map that reflects the depth information of the object in the image.

[0065] A depth map is an image in which each depth value represents the distance of the corresponding point in the scene from the observer (camera or vehicle surface). The position of each object on the depth map corresponds to the initial panoramic image to ensure the spatial consistency of the two. According to the depth value of each object in the depth image, the corresponding depth value is assigned to the object in the initial panoramic image, thereby constructing a depth map corresponding to the initial panoramic image, which is used to provide a basis for the correct sorting of the semi-transparent vehicle digital model.

[0066] For example, a scene image contains multiple cars, pedestrians, road signs, and buildings on both sides of the road. In the depth map, the nearest front vehicles will appear as lighter colors or smaller values, indicating that these vehicles are closer to the camera of the autonomous vehicle. As the distance increases, other vehicles on the road will appear as gradually darker colors or gradually larger values ​​in the depth map, reflecting their distance from the vehicle. Road signs and pedestrians near the vehicle will also be represented in the depth map with lighter colors or smaller values, while signs and pedestrians in the distance will be darker colors or higher values. The edge of the road and the buildings on both sides will appear as the darkest color or the largest value in the depth map because they are farther away.

[0067] Specifically, see Figure 3 , obtaining the depth information of each scene image includes:

[0068] Step 21: Calculate the depth information of each pixel in each scene image.

[0069] The depth information of each pixel in the scene image is calculated by a depth sensor or a stereo vision system.

[0070] Specifically, to obtain depth information through stereo vision, it is necessary to first obtain two or more images containing the same object or pixel in the scene image, find the corresponding corner points or edges in the images, calculate the horizontal pixel difference of the corner points or edges, and calculate the depth information by combining the camera parameters and the distance between the two cameras.

[0071] For example, there is a red fire hydrant in front of the right side of the vehicle, and the images taken by the right and front cameras both contain this fire hydrant. Then, based on the position of the fire hydrant in the right and front scene images, combined with the horizontal pixel difference, the front and right camera parameters, and the front and right camera distances, the depth information of each pixel on the fire hydrant is calculated.

[0072] The depth information of each pixel in the image is calculated through the spatial position relationship between each camera and the scene image, which is used to construct a depth map containing each scene image.

[0073] Specifically, constructing a depth map corresponding to the initial panoramic image based on the depth information includes:

[0074] Step 22: Map the depth information of each pixel in each scene image to the corresponding pixel in the initial panoramic image to form a depth map of the initial panoramic image.

[0075] The position correspondence between the scene image and the initial panoramic image is determined by feature point matching, and the depth information of each pixel in the scene image is mapped to the corresponding position of the initial panoramic image to form a depth map of the initial panoramic image.

[0076] For example, multiple scene images include depth information of a fire hydrant in front of the right side of a vehicle. According to the corresponding positions of each scene image in the panoramic image, the depth information of the fire hydrant is mapped to the panoramic image and the repeated areas are processed, so that the depth information of the fire hydrant is formed in the right front position of the vehicle in the depth map.

[0077] According to the depth value of each pixel in the depth image, a corresponding depth value is assigned to the object in the panoramic image coordinate system, thereby constructing a depth map corresponding to the initial panoramic image, which is used to provide a basis for the correct sorting of the semi-transparent vehicle digital model.

[0078] By using a fragment shader to process translucent objects on each pixel, and using an accumulation buffer to store and blend the color and depth information of these translucent objects, the depth information provided by the depth map is used to correctly sort the translucent objects. This makes the blending of translucent objects independent of a certain drawing order, but is done according to the sorting of their depth information, thus avoiding image display problems caused by incorrect order and improving the visual effects of panoramic images.

[0079] Step 30: Based on the information of the depth map, an order-independent semi-transparent rendering algorithm is used to mix the preset vehicle digital model into the initial panoramic image to form a panoramic image containing the vehicle.

[0080] Traditional methods require sorting of translucent objects to determine the rendering order. However, the sorting operation not only consumes performance, but is also difficult to handle for some self-intersecting, cyclically occluded transparent objects. The order-independent translucent rendering algorithm enables rendering to be unaffected by the order in which objects are drawn. The digital model of the vehicle is mixed into the initial panoramic image in the form of a translucent object based on the depth sorting information, filling in the missing vehicle part in the initial panoramic image to form a complete full-view panoramic image. At this point, the displayed content of the scene image includes the vehicle and the scene around the vehicle at a fixed viewing angle, and the displayed viewing angle can be switched in the interactive interface according to the actual needs of the user.

[0081] Order Independent Transparency (OIT) is a graphics rendering technique that allows correct blending of translucent objects without relying on the uncertain drawing order of the objects themselves, thus overcoming the problems caused by order errors in traditional rendering.

[0082] The digital model of the vehicle is created through digital modeling technology. It can simultaneously display the appearance and internal structure of the vehicle and present it with a translucent effect. The translucent digital model of the vehicle allows the observer to intuitively understand the internal structure and relative position of the vehicle without disassembling the vehicle. The translucent effect of the digital model of the vehicle is a mixture of the pixel colors of the foreground pattern and the background pattern. Specifically, according to the opacity of the foreground pattern pixel, its color is mixed with the color of the corresponding pixel of the background pattern in a certain proportion to obtain a translucent effect. By processing the color value and opacity of each pixel, complex translucent effects can be achieved and high performance can be maintained in real-time rendering.

[0083] By blending the digital model of the vehicle into the panoramic image in the form of a semi-transparent object, the blending process is not dependent on a certain drawing order, avoiding visual confusion such as ghosting or unnatural transparent effect superposition, so that the panoramic image containing the vehicle can provide clear visual effects even in complex visual environments.

[0084] Further, see Figure 4 , step 30 further includes:

[0085] Step 31: Render each pixel of the digital vehicle model through a fragment shader to obtain color information and depth information of each pixel.

[0086] Use the fragment shader in the graphics processor to process the fragment color containing the pixel, and compare the depth information of the colored fragment with the depth information in the depth buffer of the pixel. If the depth of the pixel in the current fragment is less than the depth buffer information, update the depth buffer and continue to render the pixel through the fragment shader, and store the results of previous renderings in the accumulation buffer until the depth of the pixel in the current fragment is equal to the depth buffer information. Finally, the color information and depth information of each pixel are obtained.

[0087] A fragment shader is a stage in the rendering pipeline that is responsible for calculating the color and characteristics of each pixel or fragment on the screen. It is a program executed on the graphics processing unit to determine the color of the pixels that are ultimately rendered to the screen.

[0088] The Accumulation Buffer allows the values ​​rendered to the color buffer to be copied to the accumulation buffer. When copying to the accumulation buffer multiple times, the color buffer contents can be repeatedly mixed with the current accumulation buffer contents in different ways to achieve effects such as blurring, anti-aliasing, etc.

[0089] The depth buffer area allows the graphics processor to perform depth testing by storing the depth value of each pixel. During the rendering process, the graphics processor compares the depth value of the newly generated pixel with the depth value of the corresponding position in the depth buffer to determine whether the pixel should be rendered to the screen. This mechanism ensures that only the objects closest to the viewer are displayed, and farther objects are blocked.

[0090] Step 32: Based on the depth map and the depth information of each pixel, the color information of each pixel on the digital model of the vehicle is mixed to the corresponding pixel point in the initial panoramic image.

[0091] After the vehicle digital model is positioned at the correct position in the panoramic image, the rendered pixels are mixed with the original colors of the corresponding pixels in the initial panoramic image based on the depth map and the depth information of each pixel, thereby completing the mixing of the semi-transparent vehicle digital model in the panoramic image.

[0092] Based on the depth information provided by the depth map, combined with the fragment shader and the cumulative buffer, each pixel is colored and correctly mixed into the initial panoramic image according to the depth, realizing the order-independent semi-transparent rendering algorithm, so that the mixing process does not depend on a certain drawing order, avoiding visual confusion and improving the visual effect of the image.

[0093] Further, see Figure 5 , step 32 further includes:

[0094] Step 321: Determine the occlusion relationship between pixels on the vehicle digital model and the mixing order obtained according to the depth information based on the depth map.

[0095] For each pixel on the digital model of the vehicle, its depth value is compared with the depth values ​​of other objects in the scene. If the depth value of a point is less than the depth value of another point, it means that the point is occluded by the latter from the observer's perspective. By comparing the depth values ​​of all relevant points, the occlusion relationship between the pixels can be determined. The pixels on the digital model of the vehicle are sorted from near to far according to the depth information. The sorted pixel list represents the mixing order obtained according to the depth information. The color information of the digital model of the vehicle is mixed into the initial panoramic image in this order. It ensures that only the objects closest to the observer are displayed, and the objects farther away are occluded to achieve the correct transparency effect.

[0096] Step 322: Mix the color information of each pixel on the digital model of the vehicle to the corresponding pixel point in the initial panoramic image according to the occlusion relationship and the mixing order.

[0097] The color information of each pixel on the vehicle digital model is mixed into the initial panoramic image according to the mixing order obtained from the depth information, ensuring that only the objects closest to the observer are displayed, while the objects farther away are blocked, forming a more spatial three-dimensional visual effect.

[0098] For example, in the panoramic image, there is a pedestrian standing in front of the autonomous vehicle, and part of the pedestrian's body overlaps with the vehicle's hood. During the fusion process, the occlusion relationship between the pedestrian and the vehicle's hood is first determined, and then according to the depth information, the pedestrian's color information is first drawn into the panoramic image because the pedestrian is closer to the camera. Next, for the part of the vehicle's hood that is not blocked by the pedestrian, its color information is mixed into the corresponding pixel points in the panoramic image. In this way, in the final composite image, the pedestrian will correctly block part of the vehicle's hood, presenting a realistic visual effect.

[0099] Optionally, see Figure 6 After step 322, the following steps may also be included:

[0100] Step 323: Compare the blended panoramic image with the reference image to evaluate the accuracy of the depth map and the correctness of the blending sequence for the digital vehicle model.

[0101] The selected reference image and the blended panoramic image are visually compared using an image similarity algorithm. The similarity between the two images is calculated by observing the differences between the blended image and the reference image in terms of overall visual effect, color consistency, texture details, etc., to verify whether the blending process successfully retains the information of the original image and accurately integrates the color information of the vehicle digital model into the panoramic image.

[0102] Specifically, check whether there are errors in the mixed image caused by improper handling of occlusion relations. For example, part of the vehicle may be mistakenly displayed on the occluded background, or the top of the vehicle may be occluded by side obstacles in the bird's-eye view.

[0103] Check the blended image for blending order errors. For example, different parts of the vehicle appear inconsistent in depth, or there are obvious color blending issues.

[0104] Check the impact of the accuracy of the depth map on the blending result. If the depth map has noise, holes, or inaccurate depth values, these will directly affect the quality of the blended image.

[0105] The reference image is a pre-formed reference perspective image that can accurately reflect the appearance and position of the vehicle in the actual scene without errors introduced by the mixing process.

[0106] The evaluation results are used to adjust parameters such as depth map acquisition, model processing and blending algorithm, and the blended image is regenerated for evaluation. This helps to improve the accuracy of the position, occlusion and coloring of the vehicle digital model blended onto the initial panoramic image, and optimize the visual effect of the blended panoramic view.

[0107] Step 324: Adjust the depth map construction method and the vehicle digital model sorting method based on the accuracy of the depth map and the correctness of the mixing order.

[0108] According to the accuracy evaluation results of the depth map, the data acquisition method, data processing algorithm and sorting method of the digital model of the depth map are improved respectively. According to the evaluation results, the blending process is iteratively improved to improve the quality and accuracy of the panoramic image. The sorting results and the blending process are optimized to improve the blending effect of the panoramic image.

[0109] Specifically, the mixed results of the vehicle digital model are comprehensively analyzed by combining visual inspection, quantitative evaluation, occlusion relationship inspection and light and shadow effects, and the accuracy of the depth map and the correctness of the mixed order of the vehicle digital model are comprehensively judged. If problems are found in the depth map or the mixed order, the results of the problems are fed back to the system in a timely manner, and the generation of the depth map and the mixed order of the vehicle digital model are adjusted and optimized until the vehicle digital model can be mixed into the panoramic image with the correct depth.

[0110] Specifically, the optimization of the depth map construction method also includes the optimization of the data collection method of the depth map. If there is noise or holes in the depth map, it may be because the sampling density during data collection is not enough. Increasing the sampling points can improve the accuracy of the depth map; you can also check and optimize the sensor configuration used to obtain depth data, such as adjusting the position, angle and resolution of the sensor; or combine the data of multiple sensors to construct the depth map to improve the reliability and accuracy of the data.

[0111] The optimization of the depth map construction method also includes the optimization of the data processing algorithm, such as applying a filtering algorithm to remove noise in the depth map, or using an interpolation algorithm to fill holes in the depth map, and using an optimization algorithm to further improve the accuracy and consistency of the depth map.

[0112] The optimization of the sorting method of the vehicle digital model includes the optimization of the depth information and the sorting algorithm. Priorities can be introduced to set priorities for different parts of the vehicle model so that more important parts can be processed first during the mixing process; or after each adjustment of the depth map construction method and the vehicle digital model sorting method, the mixed image is regenerated and compared with the reference image for evaluation, and the relevant parameters and methods are adjusted according to the evaluation results until a satisfactory depth and mixing effect is achieved; the optimized method can also be tested in different scenes and lighting conditions to ensure its generalization ability and robustness.

[0113] By optimizing the depth map construction method and the vehicle digital model sorting method, we ensure that the depth information extracted from the depth map is accurate enough to accurately reflect the relative position relationship between the various parts of the vehicle model. The sorting algorithm is adjusted according to the characteristics of the depth map to ensure that complex occlusion relationships and depth changes can be correctly handled.

[0114] Step 40: Photorealistic vehicle body rendering of the vehicle body in the panoramic image is performed using image-based lighting and physically based rendering materials.

[0115] The lighting effect and material of the vehicle body in the panoramic image are rendered through image-based lighting and physically based rendering materials respectively, so as to enhance the realism of the vehicle body in the panoramic image.

[0116] Image-Based Lighting (IBL) uses scene images pre-captured or rendered by the vehicle camera to provide lighting effects for objects in the scene. The environmental image is used as the light source, and these images are sampled to simulate the indirect lighting received by the surface of the object, such as reflected light, scattered light, etc. Specifically, image-based lighting samples the stereo image formed by stitching the scene images around the vehicle, and applies the sampling results as lighting information to the surface of the object, thereby achieving realistic lighting effects. It simulates the impact of complex lighting environments on the surface of the vehicle body, so that the vehicle can show delicate reflection and shadow effects under various lighting conditions, enhancing the overall visual realism.

[0117] Physically Based Rendering (PBR) materials ensure that the total amount of light reflected by a surface does not exceed the total amount of light it receives, thereby accurately depicting the material's reaction to light. This ensures that the rendering of the body surface material follows the laws of physics, such as the specular reflection of metal textures and the diffuse reflection characteristics of paint surfaces. Physically Based Rendering materials accurately calculate the absorption and scattering of light by defining the microscopic surface properties of the material, such as roughness or metallicity, so as to maintain the consistency of the visual effects of the body under different lighting and viewing angles, thereby improving the consistency and realism of the visual effects.

[0118] For example,

[0119] Image-based lighting technology is used to simulate the impact of complex lighting environments on the brightness of the vehicle surface, so that the vehicle can show more realistic and delicate reflection and shadow effects under various lighting conditions, improving the overall visual realism of the vehicle surface. Physically based rendering materials are used to ensure that the rendering effect of the vehicle surface material follows the laws of physics, so that the vehicle surface has the reflective characteristics of materials such as metal or paint, and maintains consistent visual effects under different lighting and viewing angles, further improving the visual effect of the vehicle surface.

[0120] See also Figure 1 and Figure 7 After step 30 or step 40, the panoramic image generation method further includes:

[0121] Step 50: Based on acquiring the image data corresponding to the lane and the sensor data, a chassis perspective picture of the vehicle is simulated in the panoramic image.

[0122] Since the bottom of the vehicle is usually not directly visible to the camera, it is necessary to use the sensor data installed at the bottom or around the vehicle to detect the terrain near the current vehicle, and determine the position of the current vehicle in the corresponding historical scene image based on the image data corresponding to the lane in the historical scene image retrieved from the cloud or local image storage; crop the lane image at that position and combine it with the sensor data; combine the vehicle's position, posture and sensor data, and project the information in the depth map onto a virtual chassis plane. The algorithm simulates the picture under the current vehicle chassis and displays it on the panoramic image, so that the user can observe the vehicle's chassis perspective picture from the human-computer interaction interface.

[0123] Specifically, the sensor types include: GPS position, speed, acceleration, gyroscope data, ultrasonic radar or lidar. The sensor is used to help determine the exact position and posture of the vehicle, as well as the depth information of the surrounding environment.

[0124] The simulated vehicle chassis perspective image can intuitively display the status of the road surface under the chassis, improve the visual effect of the panoramic image, and help the driver perceive the road conditions in real time, avoid potholes, bumps, obstacles on the road surface, etc., improve driving safety, reduce driving bumps, and assist driving.

[0125] See also Figure 8 , Figure 8 It is a structural diagram of an embodiment of the storage medium provided by the present application.

[0126] The storage medium 300 stores program data 301. When the program data 301 is executed by the processor, the following is achieved: Figure 1 The steps of the panoramic image generation method are described.

[0127] The program data 301 is stored in a storage medium 300, and includes a number of instructions for enabling a network device (such as a router, a personal computer, a server, or other network device) or a processor to execute all or part of the steps of the method described in each embodiment of the present application.

[0128] Optionally, the storage medium 300 may be a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, an optical disk, or other medium that can store program data.

[0129] See also Fig. 9 , Fig. 9 It is a structural schematic diagram of an embodiment of the vehicle-mounted device provided by the present application.

[0130] The vehicle-mounted device 400 is installed in the vehicle and is a computer-type device. It works in conjunction with various sensor modules and control modules in the vehicle to achieve visual perception, sensor data processing, and image display. The vehicle-mounted device 400 includes a processor 420 and a memory 410 that are interconnected. The memory 410 stores a computer program. When the processor 420 executes the computer program, the steps of the panoramic image generation method as described above are implemented.

[0131] See also Fig.10 , Fig.10 A vehicle 500 is provided, comprising the vehicle-mounted device 400 as described above. The vehicle 500 can complete the process of generating a panoramic image through the control of the vehicle-mounted device 400 during driving.

[0132] Different from the existing technology, the depth map around the corresponding vehicle is constructed through the scene image and the panoramic image to provide a basis for the correct sorting of the vehicle digital model, so that the vehicle digital model can be mixed into the panoramic image with sequence-independent semi-transparent rendering, thus overcoming the problem caused by sequence errors in traditional rendering. At the same time, the visual effects of the vehicle body surface are rendered to improve the visual authenticity of the vehicle body, and the transparent visual effects under the vehicle chassis are simulated to help the driver perceive the road conditions in real time.

[0133] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the storage medium embodiment and the computer device embodiment, since they are basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0134] The present application can be used in many general or special vehicle-mounted computing system environments or configurations, such as personal computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, network PCs, minicomputers, and distributed computing environments including any of the above systems or devices.

[0135] In the several embodiments provided in this application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device embodiments described above are only illustrative, for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not performed.

[0136] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0137] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0138] The above descriptions are merely embodiments of the present application and are not intended to limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A method for generating a panoramic image, characterized in that: include: Acquire scene images around the vehicle, and construct an initial panoramic image around the vehicle from the scene images; Acquire depth information of each of the scene images, and construct a depth map corresponding to the initial panoramic image based on the depth information; Based on the information of the depth map, an order-independent semi-transparent rendering algorithm is adopted to mix the preset vehicle digital model into the initial panoramic image to form a panoramic image containing the vehicle.

2. The panoramic image generation method according to claim 1, characterized in that: The step of mixing the preset vehicle digital model into the initial panoramic image using an order-independent semi-transparent rendering algorithm based on the information of the depth map includes: Rendering each pixel of the digital vehicle model by a fragment shader to obtain color information and depth information of each pixel; Based on the depth map and the depth information of each pixel, the color information of each pixel on the digital vehicle model is mixed to a corresponding pixel point in the initial panoramic image.

3. The panoramic image generation method according to claim 2, characterized in that: The mixing, based on the depth map and the depth information of each pixel, the color information of the pixel to the corresponding pixel point in the initial panoramic image comprises: Determining, based on the depth map, an occlusion relationship between pixels on the vehicle digital model and a mixing order obtained according to depth information; The color information of each pixel on the digital model of the vehicle is mixed onto the corresponding pixel point in the initial panoramic image according to the occlusion relationship and the mixing order.

4. The panoramic image generation method according to claim 3, characterized in that: After the color information of each pixel on the digital vehicle model is mixed with the corresponding pixel point in the initial panoramic image according to the occlusion relationship and the mixing order, the method further includes: Comparing the blended panoramic image with a reference image to evaluate the accuracy of the depth map and the correctness of the blending order for the digital vehicle model; The method for constructing the depth map and the method for sorting the digital vehicle model are adjusted based on the accuracy of the depth map and the correctness of the mixing order.

5. The panoramic image generation method according to claim 1, characterized in that: After forming the panoramic image including the vehicle, the method further includes: Image-based lighting and physically based rendering materials are used to perform realistic vehicle body rendering on the vehicle body in the panoramic image.

6. The panoramic image generation method according to claim 1 or 5, characterized in that: The panoramic image generation method further includes: Based on acquiring image data and sensor data corresponding to a lane, a chassis perspective view of the vehicle is simulated in the panoramic image.

7. The panoramic image generation method according to claim 1, characterized in that: The step of constructing an initial panoramic image surrounding the vehicle from the scene images comprises: Preprocessing the scene image by using an image processing algorithm; The preprocessed scene images are stitched together to form the initial panoramic image.

8. The panoramic image generation method according to claim 1, characterized in that: The acquiring the depth information of each of the scene images comprises: Calculating the depth information of each pixel in each of the scene images; The constructing a depth map corresponding to the initial panoramic image based on the depth information includes: The depth information of each pixel in each of the scene images is mapped to a corresponding pixel in the initial panoramic image to form a depth map of the initial panoramic image.

9. A storage medium having program data stored thereon, characterized in that: When the program data is executed by a processor, the steps of the panoramic image generation method according to any one of claims 1 to 8 are implemented.

10. A vehicle-mounted device, characterized in that: The method comprises a processor and a memory connected to each other, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of the panoramic image generation method according to any one of claims 1 to 8 are implemented.

11. A vehicle, characterized in that: The vehicle includes the vehicle-mounted device according to claim 10.