A blind area dynamic elimination and three-dimensional reconstruction method and system for vehicle-mounted surround view images

Abstract

The application provides a blind area dynamic elimination and three-dimensional reconstruction method and system for a vehicle-mounted surround view image, relates to the technical field of image processing, and comprises the following steps: identifying a current driving condition according to a vehicle state signal; performing occlusion type division on pixel points in an initial depth map based on a visibility confidence, a motion mutation index and a ground fitting residual error, and identifying a dynamic occlusion type, a non-occlusion type and an occlusion boundary type; and performing differential dynamic compensation based on pixel points in different driving conditions and belonging to different occlusion types, combining the visibility confidence and / or the motion mutation index and / or the ground fitting residual error and a depth value in the initial depth map, and generating a compensated depth map. The application effectively solves the problems of depth distortion and blind area residue caused by changes in the body posture in high-gradient and dynamic scenes.

Description

Technical Field

[0001] This application relates to the field of surround view image processing technology, and in particular to a method and system for dynamic elimination of blind spots and three-dimensional reconstruction of vehicle surround view images. Background Technology

[0002] With the rapid development of automotive intelligence and autonomous driving technologies, in-vehicle surround view imaging systems have become an indispensable active safety feature in modern vehicles. This system generates a bird's-eye view of the vehicle's surroundings by fusing images from multiple fisheye cameras around the vehicle, performing distortion correction, stitching, and inverse perspective transformation. Its aim is to eliminate blind spots for the driver and assist with parking and low-speed driving. However, existing surround view imaging systems suffer from significant technical bottlenecks in complex dynamic scenarios. Their core deficiency lies in the lack of adaptive processing capabilities for drastic changes in vehicle posture and complex operating conditions.

[0003] Specifically, when a vehicle enters conditions such as high inclines, low speeds, and sharp turns, its posture changes drastically, causing a significant tilt in the ground plane equation. Existing technologies typically perform depth backprojection and 3D reconstruction based on the static assumption of a level ground surface. Once the vehicle is in the aforementioned tilted state, the traditional level plane assumption immediately becomes invalid. This mismatch in the geometric model leads to two serious consequences: First, a systematic bias occurs in depth calculation, causing pixels that are actually close to the ground to be incorrectly mapped as suspended or recessed, resulting in distorted and broken road textures in the bird's-eye view. Second, it becomes impossible to accurately distinguish between dynamically occluded and unoccluded areas, because the incorrect geometric reference makes it difficult for multi-view matching algorithms to correctly determine which pixels are occluded by moving objects and have lost depth, and which represent actual terrain undulations.

[0004] The aforementioned deficiencies directly lead to severe challenges in the practical application of the blind spot dynamic elimination and 3D reconstruction tasks addressed in this application. First, due to deviations in depth calculation, the generated compensated depth map exhibits severe geometric distortion in slope scenarios, resulting in unnatural curvature or a floating appearance of the road surface in the image after blind spot elimination. This not only fails to reproduce the real road conditions but may also mislead drivers into making incorrect driving decisions, severely weakening the safety assistance of the surround-view system. Second, because it cannot accurately distinguish between dynamic occlusion and real terrain undulations, existing occlusion compensation strategies often misjudge pixel attributes: mistaking real slope undulations for dynamic occlusion and performing incorrect smoothing fills, or misjudging real dynamic obstacles as background and retaining their occlusion state. This directly results in fragmented object outlines, blurred edges, and even ghosting phenomena in the final output image. This failure of processing logic significantly reduces the reliability of the system under high-risk complex conditions. Summary of the Invention

[0005] This application aims to at least partially address one of the technical problems in the related art.

[0006] To achieve the above objectives, this application proposes a method for dynamic blind spot elimination and 3D reconstruction of vehicle surround view images, including the following steps:

[0007] Step 1: Acquire multi-view raw images using multiple fisheye cameras installed around the vehicle; obtain current vehicle status signals through vehicle sensors, including pitch and roll angles.

[0008] Step 2: Obtain an initial depth map based on the multi-view original images;

[0009] Step 3: Identify the current driving condition based on the vehicle status signal. The driving condition includes a first composite driving condition, a second composite driving condition, and a third composite driving condition. The first composite driving condition is a condition that simultaneously satisfies a slope ≥ 8%, a target speed steering, and the presence of dynamic obstacles. The second composite driving condition is a condition that satisfies only one of the three conditions: slope ≥ 8%, target speed steering, and dynamic obstacles. The third composite driving condition is a condition that satisfies any two of the three conditions: slope ≥ 8%, target speed steering, and dynamic obstacles. Based on the identified current driving condition, calculate the visibility confidence, motion mutation index, and ground adhesion residual for each pixel in the initial depth map.

[0010] Step 4: Based on the visibility confidence, motion mutation index and ground bonding residual, classify the occlusion types of the pixels in the initial depth map and identify dynamic occlusion type, non-occlusion type and occlusion boundary type.

[0011] Step 5: Based on pixels under different driving conditions and belonging to different occlusion types, combined with the visibility confidence and / or motion mutation index and / or ground adhesion residual and the depth value in the initial depth map, perform differentiated dynamic compensation to generate a compensated depth map; based on the compensated depth map and the original images from multiple perspectives, perform three-dimensional inverse perspective transformation and image fusion to output a surround view image with blind spots eliminated.

[0012] Obtain the initial depth map of the current frame and the initial depth map of the previous frame, calculate the absolute value of the depth difference of the same pixel and the frame time interval, and obtain the motion mutation index based on the absolute value of the depth difference and the frame time interval.

[0013] Based on the pitch and roll angles, construct the inclined local ground plane equation; back-project each pixel in the initial depth map into a three-dimensional first space point in the vehicle coordinate system according to the camera intrinsic parameters and depth value, calculate the vertical Euclidean distance between the three-dimensional first space point and the inclined local ground plane equation, and mark it as the ground fitting residual;

[0014] The first fisheye camera among the multiple fisheye cameras installed around the vehicle is selected as the reference viewpoint. Each pixel in the reference viewpoint image is defined as the target pixel. Based on the intrinsic parameter matrix of the first fisheye camera and the depth value corresponding to the target pixel in the initial depth map, the target pixel is back-projected from the image coordinate system to the vehicle coordinate system to obtain the corresponding three-dimensional second space point. The three-dimensional second space point is then reprojected onto the image plane corresponding to the remaining fisheye cameras using the intrinsic parameter matrix and extrinsic parameter matrix of the remaining fisheye cameras to obtain the reprojected coordinate set.

[0015] Based on each reprojection coordinate in the corresponding fisheye camera image, a neighborhood window of a preset size is extracted centered on the current reprojection coordinate; it is then determined whether the pixels within the neighborhood window satisfy the following two conditions:

[0016] Depth consistency: The difference between the effective pixel depth value within the neighborhood window and the depth value of the three-dimensional second space point is less than a preset depth threshold;

[0017] Feature similarity: The similarity between the features of the pixels within the neighborhood window and the features of the target pixel is greater than a preset similarity threshold;

[0018] The number of fisheye cameras that simultaneously meet both of the above conditions is counted, and the total number is summed. If the total is greater than 0, the total is used as the visibility confidence of the target pixel; otherwise, the visibility confidence of the target pixel is determined to be 0.

[0019] If the visibility confidence is 0, the current pixel is determined to be a dynamic occlusion type; if the visibility confidence belongs to the set [1,2] and the motion mutation index is greater than 0.8 m / s, the current pixel is determined to be an occlusion boundary type; if the visibility confidence is greater than or equal to 3, or if the visibility confidence belongs to the set [1,2] and the motion mutation index is less than or equal to 0.8 m / s, the current pixel is determined to be a non-occlusion type.

[0020] When the current working condition is the first composite working condition, a prediction compensation strategy is executed based on the pixels determined to be of the dynamic occlusion type to obtain a first compensation depth map, specifically including:

[0021] Step 5a1: Obtain the two-dimensional motion vector of the current pixel on the image plane; based on the depth value of the current pixel in the initial depth map and the camera intrinsic parameters, solve the two-dimensional motion vector into a three-dimensional motion vector in the vehicle coordinate system.

[0022] Step 5a2: Taking the current pixel as the center, determine the dynamic radius of the neighborhood window based on the magnitude of the three-dimensional motion vector and the preset neighborhood window radius. Determine the neighborhood expansion window based on the dynamic radius of the neighborhood window, and define all pixels within the neighborhood expansion window as candidate pixels. Traverse all candidate pixels, calculate the gradient difference between the current pixel and the candidate pixels in the multi-view original image, and mark all candidate pixels with gradient differences less than a preset texture threshold as candidate pixels with similar texture features.

[0023] Step 5a3: Perform triangulation consistency verification based on the candidate pixels with similar texture features to obtain reliable candidate pixel points;

[0024] Step 5a4: Obtain the depth value of the trusted candidate pixel in the initial depth map of the previous frame, and perform time extrapolation along the direction of the three-dimensional motion vector to obtain the predicted depth value;

[0025] Step 5a5: The predicted depth value is weighted and fused with the depth value in the initial depth map of the current frame to obtain the first compensated depth value. The first compensated depth value is assigned to the current pixel. Then, all pixels marked as dynamic occlusion type in the initial depth map are traversed. Steps 5a1-5a5 are repeated to generate the first compensated depth map.

[0026] When the current working condition is the first composite working condition, based on the pixels determined to be of the occlusion boundary type, an edge smoothing compensation strategy is executed to obtain a second compensation depth map, specifically including:

[0027] Step 5b1: Using the current pixel as the center, within a preset neighborhood, select pixels with a visibility confidence level greater than or equal to 3 as trusted reference pixels.

[0028] Step 5b2: Calculate the arithmetic mean of the depth values ​​of all the trusted reference pixels in the initial depth map;

[0029] Step 5b3: Preset correction weight coefficients based on the visibility confidence of the current pixel;

[0030] Step 5b4: The arithmetic mean is combined with the depth value in the initial depth map of the current frame and the correction weight coefficient to perform weighted fusion to obtain the second compensated depth value;

[0031] Step 5b5: Assign the second compensation depth value to the current pixel and repeat steps 5b1-5b5. Traverse all pixels marked as occlusion boundary type until the preset maximum number of iterations is reached or the depth change converges, and finally generate the second compensation depth map.

[0032] When the current working condition is the first composite working condition, based on the pixels determined to be non-occluded, a terrain following compensation strategy is executed to obtain a third compensated depth map, including the following steps:

[0033] Step 5c1: If the ground adhesion residual of the current pixel is less than 0.1m, mark it as ground-attached state; otherwise, mark it as suspended state.

[0034] Step 5c2: Obtain the three-dimensional third space point of the current pixel point back-projected onto the vehicle coordinate system;

[0035] Step 5c3: Based on the current pixel marked as being in a ground-hugging state, construct a first rotation matrix according to the pitch and roll angles, and obtain a first transformed 3D point based on the 3D third space point and the first rotation matrix; based on the current pixel marked as being in a suspended state, construct a second rotation matrix according to the pitch angle, and obtain a second transformed 3D point based on the 3D third space point and the second rotation matrix.

[0036] Step 5c4: Project the first transformed 3D point and the second transformed 3D point back onto the camera imaging plane, and extract the Z-axis component in the camera coordinate system as the third compensation depth value.

[0037] Step 5c5: Assign the third compensation depth value to the current pixel, traverse all pixels marked as non-occluded, and repeat steps 5c1-5c5 to obtain the third compensation depth map.

[0038] When the current working condition is the second or third composite working condition, based on the pixels determined to be of the dynamic occlusion type, a neighborhood smoothing compensation strategy is executed to obtain a fourth compensation depth map, including the following steps:

[0039] Step 5d1: Set a basic neighborhood window centered on the current pixel and filter out candidate pixels with texture similarity higher than a preset threshold.

[0040] Step 5d2: Perform brightness gradient consistency verification on the prepared pixels to obtain smooth reference points;

[0041] Step 5d3: Calculate the weighted average of the depth values ​​of all smoothed reference points in the initial depth map of the current frame;

[0042] Step 5d4: The weighted average value is weighted and fused with the depth value in the initial depth map of the current frame to obtain the fourth compensated depth value;

[0043] Step 5d5: Assign the fourth compensation depth value to the current pixel, traverse all pixels marked as dynamic occlusion type, and repeat steps 5d1-5d5 to obtain the fourth compensation depth map.

[0044] When the current working condition is the second or third composite working condition, based on the pixels that are determined to be occlusion boundary type, the depth values ​​of the four direct neighboring pixels above, below, left, and right of the current pixel are obtained and the arithmetic mean is calculated as the fifth compensation depth value, which is assigned to the current pixel. All pixels marked as occlusion boundary type are traversed, and the fifth compensation depth value is repeatedly calculated to obtain the fifth compensation depth map. When the current working condition is the second or third composite working condition, based on the pixels that are determined to be non-occlusion type, the basic compensation strategy is executed.

[0045] This embodiment also discloses a system for dynamic elimination and 3D reconstruction of blind spots in vehicle surround view images, including the following modules:

[0046] The data acquisition module is used to acquire multi-view raw images based on multiple fisheye cameras installed around the vehicle; and to obtain the current vehicle status signal through vehicle sensors.

[0047] The data processing module is used to obtain an initial depth map based on the multi-view original image;

[0048] The identification module is used to identify the current driving condition based on the vehicle status signal;

[0049] The segmentation module is used to classify the occlusion types of pixels in the initial depth map based on the visibility confidence, motion mutation index and ground adhesion residual.

[0050] The compensation analysis module is used to perform differential dynamic compensation based on pixels under different driving conditions and belonging to different occlusion types, combined with the visibility confidence and / or motion mutation index and / or ground adhesion residual and the depth value in the initial depth map, to generate a compensated depth map; based on the compensated depth map and the original images from multiple perspectives, it performs three-dimensional inverse perspective transformation and image fusion to output a surround view image with blind spots eliminated.

[0051] Compared with existing technologies, the method and system for dynamic elimination of blind spots and 3D reconstruction of vehicle surround view images provided in this application innovatively introduces real-time vehicle status signals (pitch angle and roll angle), constructs a local ground plane equation for the inclined plane, and calculates the vertical Euclidean distance (i.e., ground contact residual) between the pixels in the initial depth map and the inclined ground after back-projecting to the vehicle coordinate system. This corrects the systematic deviation of depth when driving on a slope at the source, effectively avoids the problem of ground-contact pixels being incorrectly mapped as suspended or recessed, and ensures the continuity and geometric authenticity of road texture in the bird's-eye view. Secondly, this invention constructs a working condition recognition and pixel classification mechanism. By defining first, second, and third composite working conditions, and combining three core indicators—visibility confidence, motion mutation index, and ground adhesion residual—it accurately classifies image pixels into dynamic occlusion, non-occlusion, and occlusion boundary types. Based on this, differentiated dynamic compensation strategies are implemented for different types of pixels under different working conditions: In the first composite working condition (high slope, low-speed steering, and dynamic obstacles), a predictive compensation strategy based on 3D motion vector inverse kinematics and triangulation consistency verification is adopted for dynamically occluded pixels. A reliable depth is generated by temporal extrapolation of candidate pixels with similar texture features, solving the depth loss problem under high-speed motion. Edge smoothing compensation is implemented for occlusion boundary pixels, improving edge continuity through iterative weighted fusion. For non-occluded pixels, terrain following compensation is performed, using a rotation matrix to dynamically correct the influence of vehicle posture, ensuring accurate reconstruction of suspended and ground-hugging areas. In the second and third composite working conditions, neighborhood smoothing compensation and basic compensation strategies are adopted to optimize computational efficiency while ensuring accuracy. It not only overcomes the problem of occlusion misjudgment caused by geometric benchmark errors in multi-view matching of traditional algorithms, but also realizes the leap from single static compensation to multi-dimensional dynamic adaptive compensation, which significantly improves the depth accuracy and visual coherence of surround view images under complex road conditions.

[0052] This invention effectively solves the problems of depth distortion and blind spot residue caused by changes in vehicle posture under high slopes and dynamic scenarios by constructing a tilted local ground model and a multi-condition adaptive compensation mechanism. This technology can accurately distinguish between dynamic occlusion, boundaries, and unoccluded areas, and accordingly execute differentiated compensation strategies such as prediction, smoothing, and terrain following, significantly eliminating the geometric distortion caused by the traditional horizontal plane assumption. The surround-view images generated by this method under complex and composite conditions have higher depth accuracy and edge clarity, significantly reducing ghosting and texture breakage phenomena, providing drivers with a realistic and reliable panoramic view, and greatly improving the active safety and assisted driving experience of vehicles in slope, sharp curve, and dynamic obstacle environments. It has significant practical value and application prospects. Attached Figure Description

[0053] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein:

[0054] Figure 1 A flowchart illustrating a method for dynamic elimination and 3D reconstruction of blind spots in vehicle surround view images provided in an embodiment of this application;

[0055] Figure 2 A structural block diagram of a vehicle surround view image blind spot dynamic elimination and three-dimensional reconstruction system provided in an embodiment of this application;

[0056] Figure 3 This is a block diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0057] The embodiments of this application are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this application, and should not be construed as limiting this application.

[0058] The following description, with reference to the accompanying drawings, describes a method and system for dynamic elimination of blind spots and three-dimensional reconstruction of vehicle surround view images according to an embodiment of this application.

[0059] like Figure 1 As shown, a method and system for dynamic blind spot elimination and 3D reconstruction of vehicle surround view images includes the following steps:

[0060] Step 1: Collect multi-view raw images using multiple fisheye cameras installed around the vehicle; obtain the current vehicle status signal through vehicle sensors, including pitch angle and roll angle.

[0061] First, multiple fisheye cameras are symmetrically installed at specific locations around the vehicle (typically below the front bumper, rear bumper, and side mirrors). These cameras form a multi-view raw image acquisition array to simultaneously capture a wide-angle field of view within a 360-degree range around the vehicle. During startup or operation in this embodiment, all fisheye cameras perform frame-synchronous acquisition to obtain unprocessed raw video stream data, i.e., multi-view raw images, ensuring alignment of each camera on the time axis to eliminate timing errors caused by motion blur. Simultaneously, the vehicle sensor module begins operation, reading the vehicle's dynamic status parameters via the onboard CAN bus or a dedicated interface. The focus is on extracting vehicle status signals including pitch and roll angles. The pitch angle is sensed by accelerometers or inertial measurement units (IMUs) mounted on the vehicle body, detecting changes in the vehicle's longitudinal tilt, while the roll angle reflects the degree of lateral tilt.

[0062] Step 2: Obtain an initial depth map based on the original multi-view images.

[0063] First, this embodiment preprocesses the acquired multi-view raw images. Using a pre-calibrated camera intrinsic parameter matrix (including focal length and principal point coordinates) and distortion coefficients, radial distortion correction is performed on the images captured by each fisheye camera to eliminate the barrel distortion effect produced by wide-angle lenses and ensure that the geometric correspondence between image pixels and real light projection is accurate.

[0064] Subsequently, using the current frame as a reference frame, other multi-view images from adjacent times or angles are selected as target frames. For each valid pixel in the reference frame, a feature descriptor is extracted within the corresponding search region of the target frame. Specifically, convolutional neural networks or traditional operators are used to analyze the texture gradient, color distribution, and structural edges of the image in the local neighborhood, encoding the visual features of the region surrounding each pixel into a set of high-dimensional vectors, i.e., feature descriptors. This descriptor uniquely represents the local appearance characteristics of the pixel, ensuring its distinguishability even under changes in illumination or slight shifts in viewing angle. Next, the feature descriptors of the reference frame pixels are compared one by one with the candidate pixel descriptors within the search region of the target frame. By calculating the similarity score between the two (e.g., calculating the Euclidean distance or Hamming distance of the vectors), the pixel pair with the highest matching degree is selected. This process aims to establish a pixel-level correspondence, i.e., to determine the precise location of the reference frame pixel in the target frame.

[0065] After obtaining successfully matched pixel pairs, the disparity value is calculated using the principles of stereo vision. The disparity value is defined as the lateral displacement between the image positions of the same point in space from two different viewpoints. According to the principle of triangulation, the distance of an object from the camera is inversely proportional to this disparity value: when the object is closer, the difference in its image position between the two viewpoints is larger, resulting in a larger disparity value; conversely, when the object is farther away, the difference in its image position is smaller, resulting in a smaller disparity value. Combining the known camera focal length and the baseline length between the optical centers of the two cameras, the calculated disparity value is converted into a specific depth value, i.e., the straight-line distance from the camera's optical center to the surface of an object in the scene, using the aforementioned inverse relationship. For multi-view scenes, the disparity calculation results from all available viewpoints are weighted and fused or optimally selected to eliminate errors caused by occlusion and duplicate textures, obtaining the accurate depth value corresponding to each pixel.

[0066] Finally, each pixel and its corresponding depth value are mapped into a two-dimensional matrix to form an initial depth map covering the area around the vehicle.

[0067] Step 3: Identify the current driving condition based on the vehicle status signal. The driving condition includes a first composite driving condition, a second composite driving condition, and a third composite driving condition. The first composite driving condition is a condition that simultaneously satisfies a gradient ≥ 8%, a target speed steering, and the presence of a dynamic obstacle. The second composite driving condition is a condition that satisfies only one of the three conditions: gradient ≥ 8%, target speed steering, and a dynamic obstacle. The third composite driving condition is a condition that satisfies any two of the three conditions: gradient ≥ 8%, target speed steering, and a dynamic obstacle.

[0068] First, regarding the acquisition of driving slope, the pitch angle value from the vehicle status signal is used to perform real-time calculation using the trigonometric function relationship: Slope = tan(pitch angle) × 100%. When the calculated slope percentage is greater than or equal to 8%, it is determined that the current driving state is on a high slope. This is because in depth reconstruction tasks, a slope exceeding 8% causes a drastic change in the angle between the camera's optical axis and the ground normal, resulting in severe geometric distortion in the initial depth map generated based on the conventional horizontal plane assumption. This distortion is the core error source that subsequent compensation algorithms must correct. If this parameter is ignored or a fuzzy threshold is used, it will be impossible to distinguish the projection distortion caused by normal road surfaces and extreme slopes, causing the compensation strategy to oscillate between overcorrection and undercorrection, failing to guarantee the continuity of the depth map. The advantage of this identification method is that it can accurately quantify the degree of damage to the imaging geometry caused by the terrain, providing a clear quantitative basis for subsequently selecting the degree of geometric correction, thereby ensuring that this embodiment has sufficient robustness when facing nonlinear abrupt change regions.

[0069] Secondly, regarding the acquisition of low-speed turning states: Real-time reading of the instantaneous speed value output by the vehicle speed sensor and the angle change rate data from the steering wheel angle sensor are used. A vehicle speed threshold of 15 km / h and a steering angle change threshold of 2° / s are set. When the instantaneous speed value is consistently below 15 km / h and the steering angle change per unit time (i.e., the angle difference between adjacent sampling times) exceeds 2° / s, both values ​​simultaneously meet specific conditions to be considered a low-speed turning (target speed turning) state. This numerical definition effectively eliminates misjudgments during high-speed straight driving or when stationary. The reason for introducing a joint judgment of instantaneous speed value and steering angle change is that simple low speed may lead to the vehicle being stationary or crawling in a straight line, while simple turning may occur in high-speed cornering scenarios. Only low speed combined with a large angular velocity change indicates that the vehicle is in a U-turn or a sharp turn in an extremely narrow space. In this situation, even small changes in vehicle posture can cause severe nonlinear drift in the parallax baseline between multi-view images, easily leading to depth matching errors. The advantage of this identification method is that it can effectively filter out regular steering interference during high-speed cruising and false alarms when stationary, concentrating valuable computing power on extreme maneuvering scenarios where occlusion and feature loss are most likely to occur. This is because if the upper limit of vehicle speed is not limited, motion blur during high-speed steering will obscure the true geometric relationship, leading to feature extraction failure. If the lower limit of steering rate is not limited, slight directional adjustments will be misjudged as complex operating conditions, causing the system to frequently switch to high-precision compensation mode, resulting in wasted resources and failing to solve the fundamental problem.

[0070] Secondly, regarding the acquisition of dynamic obstacles, frame-by-frame differencing is performed on the original image sequence from multiple perspectives. The grayscale change vector of the same pixel in two consecutive frames is calculated. If the displacement vector magnitude of a specific region between consecutive frames exceeds a preset pixel threshold (e.g., 5 pixels) and is not due to background texture fluctuations, then a dynamic obstacle is identified. This identification is crucial because depth map construction in a static background relies on the consistency of feature points. Dynamic objects (such as pedestrians and other vehicles) directly disrupt this consistency, leading to gaps or erroneous depth jumps in depth calculations. This is especially true in high-slope scenes, where dynamic objects further exacerbate the occlusion effect, rendering traditional algorithms ineffective. The advantage of this identification method is that it removes interference from moving targets before generating the depth map, preventing the incorrect mapping of dynamic object depth information onto the static road surface, thus ensuring the purity of the local ground plane equation. The presence of dynamic obstacles directly negates the fundamental premise of depth reconstruction—the static scene assumption. Without explicit identification and isolation, any compensation algorithm based on the static assumption will completely fail, unable to output reliable depth data.

[0071] Based on the real-time quantification results of the above three parameters, a strict working condition identification logic is executed: The first composite working condition is identified only when all three conditions are met simultaneously: gradient ≥ 8%, low-speed steering (vehicle speed < 15 km / h and steering angle change > 2° / s), and the presence of dynamic obstacles. This represents a fully coupled state of all adverse factors: high gradient leads to geometric baseline collapse, low-speed sharp turns cause drastic baseline changes, and dynamic objects cause feature matching failure; the superposition of these three factors produces catastrophic deep reconstruction errors. If only one of the three conditions is met, it is identified as the second composite working condition; this is a single-factor interference situation, such as only high gradient without steering and no dynamic obstacles, or only dynamic obstacles but flat road conditions. In this case, the main contradiction is singular, and geometric distortion or occlusion effects are relatively independent, without forming complex coupling effects. If exactly two of the three conditions are met, it is identified as the third composite working condition, which is a two-factor coupling situation, such as high gradient + low-speed steering without dynamic obstacles, or high gradient + dynamic obstacles without steering. At this point, the two main error sources interact, and their impact is greater than that of a single factor but less than that of the superposition of three factors, exhibiting nonlinear characteristics of moderate complexity.

[0072] Based on the identified current working condition, the visibility confidence, motion abrupt change index, and ground adhesion residual of each pixel in the initial depth map are calculated respectively:

[0073] Obtain the initial depth map of the current frame and the initial depth map of the previous frame, calculate the absolute value of the depth difference of the same pixel and the frame time interval, and obtain the motion mutation index based on the absolute value of the depth difference and the frame time interval.

[0074] After acquiring the initial depth map of the current frame and the initial depth map of the previous frame, a strict pixel-level alignment and mapping operation is first performed. Specifically, using the camera intrinsic parameter matrix and the vehicle rigid body motion model (which is calculated by fusing data from the inertial measurement unit (IMU) and wheel speed sensors), the coordinates of all pixels in the previous frame's depth map are precisely projected into the coordinate system of the current frame's image. This eliminates global parallax caused by vehicle translation and rotation, ensuring that the positions of the same physical space point in the two frames correspond strictly. Subsequently, for each pair of successfully matched pixels, its depth value in the current frame and its mapped depth value in the previous frame are extracted, and the absolute value of the difference between the two is calculated. This value directly reflects the radial displacement of the pixel per unit time. Simultaneously, the timestamps of the two frame acquisition times recorded by a high-precision clock are read, and the time interval between the two frames is calculated as a benchmark for the time dimension. The absolute value of the calculated depth difference is divided by the frame time interval to obtain the motion mutation index. Because simple depth difference cannot reflect the drasticness of change: in long exposures or low frame rates, even if an object moves a large distance, its actual speed may not be fast if the time interval is long; conversely, a tiny depth jump in a very short time may indicate a high-speed collision or violent shaking. Only by dividing the depth difference by the frame interval can the instantaneous radial velocity of the object relative to the camera be accurately characterized. This is the physical essence for judging whether a sudden change in motion has occurred in the scene; without introducing the time interval, it is impossible to distinguish whether it is an object rapidly approaching or the camera itself moving at high speed, leading to misjudgment. The advantage of this identification method is that it can accurately capture those instantaneous, violent geometric deformations or dynamic object intrusions that are ignored under the traditional static assumption.

[0075] Based on the pitch and roll angles, construct the inclined local ground plane equation; back-project each pixel in the initial depth map into a three-dimensional first spatial point in the vehicle coordinate system according to the camera intrinsic parameters and depth value, calculate the vertical Euclidean distance between the three-dimensional first spatial point and the inclined local ground plane equation, and mark it as the ground fitting residual.

[0076] First, using the vehicle pitch and roll angles calculated in real time by the inertial measurement unit (IMU), a rotation matrix is ​​constructed to obtain the true normal vector of the tilted local ground. Specifically, the initial normal vector of the ideal horizontal ground in the vehicle coordinate system is [0, 0, 1]. T T represents the transpose operation, indicating an upward rotation perpendicular to the Z-axis of the vehicle body. To reflect the actual vehicle attitude, a composite rotation matrix consisting of pitch and parallel rotations is constructed: first, the pitch angle is rotated around the vehicle's Y-axis, then the roll angle is rotated around the X-axis. This rotation matrix is ​​applied to the initial normal vector, and the initial normal vector of the ideal horizontal ground in the vehicle coordinate system is multiplied by the initial normal vector to obtain the transformed real road surface normal vector [A, B, C]. TIn this process, components A and B represent the degree of inclination of the road surface in the forward and backward directions (X-axis) and the left and right directions (Y-axis) of the vehicle, respectively, while C represents the vertical observation component of the road surface. This process transforms abstract angle data into specific geometric vectors, which accurately describe the spatial attitude of the current road conditions relative to the vehicle body.

[0077] Based on this, the generated equation for the inclined local ground plane is in the form Ax + By + Cz + D = 0. Here, D is a constant term. The geometric features described by this equation mainly include the road surface slope, the road surface azimuth angle, and the relative height relationship between the vehicle and the road surface. These features form the basis for subsequent high-precision obstacle detection. Finally, the vertical Euclidean distance from each generated three-dimensional first spatial point to the inclined local ground plane equation is calculated and marked as the ground fit residual. Because the vehicle pitches and rolls during travel, the road surface normal direction no longer coincides with the vehicle's Z-axis; therefore, the Z-axis coordinate difference cannot be simply used to measure the fit. The standard point-to-plane distance formula is used for calculation: First, substitute the coordinates of the point in the three-dimensional first space into the plane equation, that is, multiply the X, Y, and Z coordinates of the point by the corresponding coefficients A, B, and C in the plane equation, and add the constant term D to calculate their algebraic sum. This result represents the directed distance of the point relative to the plane. Next, take the absolute value of this algebraic sum. This step is to eliminate the influence of the positive and negative signs and only focus on the magnitude of the distance. Regardless of whether the point is above or below the plane, its deviation is always positive. At the same time, calculate the square root of the sum of the squares of each component of the plane normal vector and use it as the denominator to normalize the distance. Finally, divide the numerator by the denominator to obtain the true distance of the point in the direction perpendicular to the road surface.

[0078] When the distance approaches zero, it indicates that the spatial point corresponding to the pixel is closely attached to the currently calculated inclined road surface and belongs to the normal road surface area; when the distance is significantly greater than the preset threshold (0.1-0.3m), it indicates that the point is located above the road surface (such as an obstacle) or that the depth estimation is abnormal.

[0079] The first fisheye camera among multiple fisheye cameras installed around the vehicle is selected as the reference viewpoint. Each pixel in the reference viewpoint image is defined as a target pixel. Based on the intrinsic parameter matrix of the first fisheye camera and the depth value corresponding to the target pixel in the initial depth map, the target pixel is back-projected from the image coordinate system to the vehicle coordinate system to obtain the corresponding three-dimensional second space point. Using the intrinsic and extrinsic parameter matrices of the remaining fisheye cameras, the three-dimensional second space point is reprojected onto the image plane corresponding to the remaining fisheye cameras to obtain the reprojected coordinate set. The process of back-projecting pixels to the vehicle coordinate system and then projecting them onto the image plane is a conventional operation in this field, and the specific implementation process will not be described in detail in this embodiment.

[0080] In this embodiment, the first fisheye camera among multiple fisheye cameras installed around the vehicle is selected as the reference viewpoint (e.g., the forward-facing main camera), and each valid pixel on its image plane is defined as a target pixel. The row and column coordinates of the target pixel in the two-dimensional image coordinate system are read, and combined with the intrinsic parameter matrix of the first fisheye camera (which includes focal length parameters and principal point coordinates) and the depth value corresponding to the position in the pre-generated initial depth map, back-projection calculation is performed using the inverse process of the pinhole imaging model. Specifically, the pixel coordinates are first normalized to the camera normalization plane, that is, by subtracting the principal point coordinates from the pixel's horizontal and vertical coordinates respectively, and then dividing by the corresponding focal length component, to obtain the normalized planar coordinates; then, these two normalized coordinates are multiplied by the current depth value to obtain the three-dimensional spatial coordinates of the point in the reference viewpoint camera coordinate system; then, using the pre-calibrated extrinsic parameter matrix of the first fisheye camera (composed of rotation matrix and translation vector), the coordinates in the camera coordinate system are transformed to a unified vehicle coordinate system, thereby obtaining an accurate three-dimensional second spatial point, which represents the three-dimensional geometric information of the corresponding physical location in the real world.

[0081] After obtaining high-precision 3D points in the vehicle coordinate system, the multi-view consistency verification and data fusion stage begins. For the remaining fisheye cameras (such as left-view, right-view, and rear-view cameras), their respective intrinsic and extrinsic parameter matrices are called sequentially. The calculation process is as follows: First, the obtained 3D second-space points are transformed from the vehicle coordinate system to the camera coordinate system of the i-th fisheye camera. The calculation formula is to subtract the translation vector from the point coordinates and then multiply by the transpose of the rotation matrix. Subsequently, the 3D coordinates are projected back onto the 2D image plane of the i-th camera using the intrinsic parameter matrix of the i-th camera. The theoretical pixel coordinates are calculated by dividing the horizontal and vertical components of the 3D coordinates by the depth component, multiplying by the corresponding focal length, and adding the principal point coordinates. This set of calculation results is marked as the reprojected coordinate set.

[0082] Theoretical projected pixel coordinate sequence: This is the most crucial result. For each remaining camera, one or more specific 2D coordinate pairs (row number, column number) are calculated. This set of coordinates represents the specific pixel location in the camera image where the 3D point would fall if it existed in the real world.

[0083] Multi-view visibility determination marker: Based on the calculated theoretical projection coordinates, determine whether the coordinates fall within the effective imaging area of ​​the remaining cameras (i.e., whether they are within the image boundary and not occluded). This generates a Boolean marker result to indicate whether the 3D point is visible or invisible / occluded from a specific viewpoint.

[0084] Reprojection error quantization: The calculated theoretical projected pixel coordinates are compared with the actual coordinates of image feature points or depth map sampling points actually acquired by the remaining cameras, and the deviation distance between the two is calculated. This difference is used to evaluate the accuracy of the current depth estimation.

[0085] Based on each reprojection coordinate in the corresponding fisheye camera image, a neighborhood window of a preset size is extracted centered on the current reprojection coordinate; it is then determined whether the pixels within the neighborhood window satisfy the following two conditions:

[0086] Depth consistency: The difference between the effective pixel depth value within the neighborhood window and the depth value of the three-dimensional second space point is less than a preset depth threshold;

[0087] Feature similarity: The similarity between the features of the pixels within the neighborhood window and the features of the target pixel is greater than a preset similarity threshold;

[0088] The number of fisheye cameras that simultaneously meet both of the above conditions is counted, and the total number is summed. If the total is greater than 0, the total is used as the visibility confidence of the target pixel; otherwise, the visibility confidence of the target pixel is determined to be 0.

[0089] In this embodiment, for each target pixel that is back-projected onto the vehicle coordinate system and then projected back onto the remaining fisheye camera, its corresponding theoretical projection coordinates are extracted. Centered on these theoretical projection coordinates, a neighborhood window of a preset size (e.g., a 5×5 or 7×7 pixel rectangular area) is extracted from the corresponding fisheye camera image data, serving as the sampling range for subsequent verification. All valid pixels within this neighborhood window undergo dual logical checks: the first is a depth consistency check, where the depth value corresponding to each valid pixel in the initial depth map is read, and the absolute difference between its depth and the depth of the original three-dimensional second space point is calculated. If the depth difference of at least one pixel within the neighborhood is less than a preset depth threshold, the viewpoint is deemed to meet the depth consistency condition, indicating that the region geometrically matches the reference point. The second step is feature similarity verification. The visual feature vectors, such as the color histogram and gradient direction of the pixels in the neighborhood window, are compared with the features of the target pixel in the reference viewpoint, and the similarity score between the two is calculated. If the score is greater than the preset similarity threshold, it is determined that the feature similarity condition is met, indicating that the region belongs to the same object surface as the reference point in terms of visual texture.

[0090] After completing the dual verification of the neighborhood window, the statistical and confidence generation stage begins. Each remaining fisheye camera is traversed, checking whether it simultaneously passes the tests for both depth consistency and feature similarity. For each camera that passes the test, its count is incremented by one; if a camera's neighborhood window fails to meet both conditions simultaneously (e.g., excessive depth difference or texture mismatch), that camera is not counted as a valid vote. Finally, the total number of cameras that pass the test is summed. Based on this sum, the visibility confidence of the target pixel is defined: if the sum is greater than 0, it means that the 3D point has been confirmed both geometrically and visually from multiple independent viewpoints, and this value (i.e., the number of cameras that passed the verification) is directly used as the visibility confidence of the target pixel. The higher the value, the greater the probability that the point is observed by multiple viewpoints, and the higher the confidence. If the sum equals 0, it means that no auxiliary viewpoint can successfully verify the authenticity of the point, and the visibility confidence of the target pixel is determined to be 0, marking it as a low-confidence or potential noise point.

[0091] The depth threshold is typically set between 0.1 and 0.25 meters. In low-speed urban road conditions, a threshold of 0.1 meters can be used to strictly eliminate depth jumps; in high-speed or complex road conditions, it can be relaxed to 0.25 meters to tolerate minor displacements caused by vehicle vibrations. The similarity threshold mainly depends on lighting conditions and texture richness, and is generally set between 0.75 and 0.9. In well-lit scenes with clear textures, the threshold can be set above 0.85 to ensure extremely high matching accuracy; in low-texture areas such as low light, backlight, or large areas of solid-color walls, to avoid mistakenly identifying real objects, the threshold can be appropriately lowered to 0.75.

[0092] This embodiment achieves an upgrade in verification from point-to-point to point-to-multipoint by focusing on the theoretically projected pixel coordinates and extending to the neighborhood window. It introduces depth consistency, requiring that the distance to objects seen from the auxiliary viewpoint must match the distance calculated by the reference, effectively eliminating false 3D points caused by depth map noise. Introducing feature similarity ensures that different viewpoints observe the same physical entity, preventing irrelevant objects in the background from being mistaken for foreground obstacles. Furthermore, quantifying the results into visibility confidence levels distinguishes between completely unreliable (confidence level 0) and partially reliable (confidence levels 1, 2, ...) points, allowing subsequent algorithms to adopt tiered processing strategies based on confidence levels (e.g., high-confidence points are directly rendered, while low-confidence points are smoothly interpolated or ignored). This significantly improves the system's robustness and security in complex dynamic environments, ensuring that the final generated 3D map is free of holes and artifacts.

[0093] Step 4: Based on the visibility confidence, motion mutation index and ground bonding residual, classify the occlusion types of the pixels in the initial depth map and identify dynamic occlusion type, non-occlusion type and occlusion boundary type.

[0094] If the visibility confidence is 0, the current pixel is determined to be a dynamic occlusion type; if the visibility confidence belongs to the set [1,2] and the motion mutation index is greater than 0.8 m / s, the current pixel is determined to be an occlusion boundary type; if the visibility confidence is greater than or equal to 3, or if the visibility confidence belongs to the set [1,2] and the motion mutation index is less than or equal to 0.8 m / s, the current pixel is determined to be a non-occlusion type.

[0095] First, the visibility confidence value of the pixel is read. If the value is 0, it means that no auxiliary fisheye camera can verify the existence of the point, i.e., the point is completely invisible from multiple perspectives, and the system directly determines it to be a dynamic occlusion type. Second, if the visibility confidence value is in the set {1, 2} (i.e., only 1 or 2 auxiliary perspectives barely pass the verification), the motion mutation index is further introduced as a key criterion; if the index is greater than 0.8 m / s, it indicates that the area where the point is located has undergone drastic position or shape changes in a very short time, and it is determined to be an occlusion boundary type; conversely, if the visibility confidence value is greater than or equal to 3 (i.e., 3 or more perspectives consistently confirm), or although the confidence value is 1 or 2 but the motion mutation index is less than or equal to 0.8 m / s, the point is determined to be a non-occluded type and is regarded as a static or stable object in the environment.

[0096] Because if a 3D point cannot be observed from the reference viewpoint and all auxiliary viewpoints during vehicle movement (confidence level of 0), this is usually not due to depth estimation errors (because depth errors often result in mismatches from at least one viewpoint), but rather because the point is momentarily occluded by a fast-moving object (such as a pedestrian suddenly rushing out or a vehicle changing lanes), causing it to completely disappear or be severely distorted in the field of view of other cameras, thus failing feature matching and depth verification. This full-view failure is the most typical geometric feature of dynamic occlusion. When the confidence level is only 1 or 2, it indicates that the point is at the edge of visual information, possibly a blurred or weakly textured area. If the motion abrupt change index is very high at this time (>0.8 m / s), it indicates that the area is undergoing violent relative motion. This is usually the projection tearing or ghosting phenomenon produced by the interface between two objects of different depths (such as the edge of a stationary wall and a moving vehicle) under high-speed motion, i.e., the occlusion boundary. High confidence (≥3) represents multi-view consistency under strong geometric constraints, almost eliminating the possibility of occlusion; while low confidence with low motion (≤0.8m / s) indicates that although the feature matching of the point is weak (it may be far away or have a simple texture), its spatial position is relatively stable and belongs to a normal static environment point, rather than being occluded or disturbed by the boundary.

[0097] Step 5: Based on pixels under different driving conditions and belonging to different occlusion types, combined with the visibility confidence and / or motion mutation index and / or ground adhesion residual and the depth value in the initial depth map, perform differentiated dynamic compensation to generate a compensated depth map; based on the compensated depth map and the original images from multiple perspectives, perform three-dimensional inverse perspective transformation and image fusion to output a surround view image with blind spots eliminated.

[0098] When the current working condition is the first composite working condition, a prediction compensation strategy is executed based on the pixels determined to be of the dynamic occlusion type to obtain a first compensation depth map, specifically including:

[0099] Step 5a1: Obtain the two-dimensional motion vector of the current pixel on the image plane; based on the depth value of the current pixel in the initial depth map and the camera intrinsic parameters, solve the two-dimensional motion vector into a three-dimensional motion vector in the vehicle coordinate system.

[0100] This embodiment utilizes an optical flow algorithm to perform feature matching between the current frame and the previous frame, directly obtaining the two-dimensional motion vector of the target pixel on the image plane. This vector consists of the pixel displacement in the horizontal and vertical directions. The depth value corresponding to the target pixel in the initial depth map is read, and pre-calibrated camera intrinsic data (including horizontal and vertical focal lengths, and the principal point coordinates of the imaging center) is called. Based on the inverse projection principle of the pinhole camera model, the two-dimensional motion vector is decomposed into a three-dimensional motion vector in the camera coordinate system: First, the pixel displacement is divided by the horizontal and vertical focal lengths respectively to obtain the angular velocity change on the normalized plane, and then multiplied by the depth value to calculate the relative displacement components of the object along the left-right, up-down, and front-back axes in the camera coordinate system. Subsequently, using the extrinsic data of the fisheye camera (including rotation matrices and translation vectors), the three-dimensional motion vector in the camera coordinate system is rotated and translated, finally mapping it to a unified vehicle coordinate system to obtain an accurate three-dimensional motion vector in the vehicle coordinate system. This vector fully describes the three-dimensional velocity state of the object in the real world. Rotating and translating the three-dimensional motion vector in the camera coordinate system and finally mapping it to a unified vehicle coordinate system is a conventional technique, which will not be described in detail in this embodiment.

[0101] This embodiment performs a 2D-to-3D inverse kinematics transformation because pixel displacement on the image plane only reflects apparent motion within the viewpoint, and its magnitude is greatly affected by distance: a distant object, even if moving very fast, may only move a few pixels on the image; while a nearby object, even if stationary, will produce a huge parallax displacement if the vehicle itself is moving. Without introducing depth values ​​for inverse kinematics, the system cannot distinguish whether an object or a vehicle is moving, nor can it quantify the object's true speed. By combining depth values, the scale ambiguity caused by distance factors can be eliminated, restoring the absolute displacement of the object in the real world; further transformation to the vehicle coordinate system via an extrinsic parameter matrix eliminates the influence of differences in the viewpoints of different cameras, ensuring that the motion data from all cameras are uniformly measured under the same global reference system. This processing method ensures that regardless of the target object's location or distance from the vehicle, the calculation standard for its motion vector is physically consistent and comparable.

[0102] Step 5a2: Taking the current pixel as the center, determine the dynamic radius of the neighborhood window based on the magnitude of the three-dimensional motion vector and the preset neighborhood window radius. Determine the neighborhood expansion window based on the dynamic radius of the neighborhood window, and define all pixels within the neighborhood expansion window as candidate pixels. Traverse all candidate pixels, calculate the gradient difference between the current pixel and the candidate pixels in the multi-view original image, and mark all candidate pixels with gradient differences less than a preset texture threshold as candidate pixels with similar texture features.

[0103] Using the pixel currently being analyzed as the geometric center, the system reads the 3D motion vector magnitude calculated in the vehicle coordinate system in real time. The system internally presets a basic neighborhood window radius (e.g., set to five pixels) as the baseline coverage area in low-speed or stationary states. When the system detects that the 3D motion vector magnitude exceeds a preset baseline speed threshold (e.g., three meters per second), it determines that the target is in a high-speed motion state with a large magnitude. At this point, the ratio of the current speed value to the baseline speed value is multiplied by an empirical adjustment coefficient (a value between 0 and 1) based on lens focal length and field of view calibration to calculate the speed compensation increment. This increment is then added to the basic radius to generate the dynamic radius of the neighborhood window. If the calculated dynamic radius exceeds the set maximum safety limit (e.g., thirty pixels), it is automatically truncated to the upper limit to prevent the search range from being too large and introducing background noise from distant areas. This process ensures that regardless of whether the target object is moving slowly or speeding at high speed, its associated image region can be accurately and completely bounded, avoiding feature omissions caused by high-speed motion and preventing over-searching in low-speed scenes.

[0104] Based on the generated neighborhood window's dynamic radius, a fixed safety margin (typically one to two pixels) is further extended outwards, forming a neighborhood expansion window. This expansion window is designed to accommodate sub-pixel displacement deviations that may occur due to optical flow tracking errors, minor camera vibrations, or object edge blurring. All valid pixels within this expanded neighborhood window are uniformly defined as candidate pixels. Truly similar texture points will inevitably fall within the core region of the dynamic radius, but to be on the safe side, a certain margin of error must be reserved to capture feature points that are slightly offset due to motion blur. In this way, the system constructs a candidate set that includes core features and possesses error tolerance, laying a broad foundation for subsequent precise selection.

[0105] Iterate through all candidate pixels defined above, performing a rigorous gradient consistency check on each candidate pixel. During this process, convolutional kernels (such as the Sobel operator) are used to extract the horizontal and vertical gradient components of the current center pixel and each candidate pixel in the multi-view original image. These two components together form a two-dimensional gradient vector. The magnitude of this vector represents the intensity of grayscale change at that point (i.e., edge strength), while the direction of the vector points to the direction of the fastest grayscale change (i.e., perpendicular to the edge). Next, the Euclidean distance between the gradient vector of the center pixel and the gradient vector of the candidate pixels is calculated to quantify the degree of difference between them. If the calculated gradient difference is less than a preset texture threshold (preferably between 0.15 and 0.25), the point is determined to have a highly consistent texture structure with the center pixel and is marked as a candidate pixel with similar texture features; conversely, if the gradient difference is greater than the threshold, it is determined to be background clutter or the boundary between different objects and is discarded. This filtering process leverages the characteristic that the surfaces of the same physical objects in natural scenes typically have continuous and similar texture gradient distributions, effectively distinguishing pixels belonging to the same object from background pixels that are merely spatially adjacent but have distinct textures.

[0106] Two-dimensional gradient vectors can capture the comprehensive change information of a point in all directions. Calculating the difference between two gradient vectors essentially compares whether the texture patterns at the two points are consistent: if the two points belong to the same smooth object surface (such as a car paint surface), their gradient vectors should be very close (similar magnitudes and consistent directions); if the two points belong to different objects (such as a car body and a road surface), their gradient vectors often have huge differences (disparate magnitudes or orthogonal directions). This vector space-based metric method is more robust than simply comparing grayscale values ​​or one-dimensional gradients, and can effectively resist the effects of uneven lighting, because changes in lighting usually only change the magnitude of the gradient, without drastically changing the direction relationship of the gradient. By comprehensively considering both magnitude and direction, the system can more accurately determine the consistency of the texture.

[0107] Step 5a3: Perform triangulation consistency verification based on the candidate pixels with similar texture features to obtain reliable candidate pixel points.

[0108] This embodiment is based on a set of candidate pixels with similar texture features. For each candidate pixel, its corresponding two-dimensional image coordinates are extracted from multiple original images (such as left-view, right-view, or surround-view cameras). Since there is a physical baseline distance between the installation positions of different cameras and distortion in the imaging plane, the coordinates of candidate pixels from each viewpoint are first uniformly transformed into a three-dimensional spatial ray model in the vehicle coordinate system using pre-calibrated intrinsic parameter matrices (focal length, principal point coordinates) and extrinsic parameter matrices (rotation and translation matrices). Specifically, for the same candidate pixel, its two-dimensional coordinates from different viewpoints are back-projected into three spatial rays emanating from the optical centers of their respective cameras. These rays should theoretically converge at the same point on the surface of the real object. All candidate pixels are traversed, and the triangulation model composed of multiple viewpoint rays is solved. One ray (e.g., the left-view ray) is selected as the reference line, and then the perpendicular distance from any point on the other two rays (e.g., the right-view and third-view rays) to the reference ray is calculated. To eliminate distance fluctuations caused by depth estimation errors, an optimally fitted depth point is selected on the reference ray (usually determined by minimizing the sum of squared distances from all rays to this point), and this point is defined as the theoretical intersection focus. The Euclidean distance from points at corresponding depths on other rays to this theoretical intersection focus is calculated; this distance is the focus deviation distance. If a candidate pixel belongs to the same real physical point in multiple viewpoints, its corresponding spatial ray should be highly convergent, resulting in a minimal focus deviation distance. Conversely, if the point is falsely detected background noise, an occluded boundary, or a repetitive texture (e.g., misidentifying distant lane lines as the same object), its rays cannot intersect in space, leading to a significant increase in the focus deviation distance. The calculated focus deviation distance is compared with a preset geometric consistency threshold. This threshold is typically set as a sub-pixel tolerance range derived from scene depth and camera baseline distance (e.g., 1-3 cm corresponding to actual physical distance). When the focal deviation distance of a candidate pixel is less than the threshold, the system determines that it meets the triangulation consistency requirement, confirms that the point is a real 3D spatial point, and marks it as a reliable candidate pixel. If the distance exceeds the threshold, it is determined to be an anomalous point with geometric inconsistency and is removed. This process utilizes the physical law that real objects in stereo vision must satisfy strict geometric constraints under multiple viewpoints, effectively filtering out artifacts that are based solely on texture similarity but contradictory spatial positions from a single viewpoint.

[0109] Step 5a4: Obtain the depth value of the trusted candidate pixel in the initial depth map of the previous frame, and extrapolate it over time along the direction of the three-dimensional motion vector to obtain the predicted depth value.

[0110] The initial depth map data of the previous frame is obtained. This depth map records the 3D depth information of all valid pixels in the previous moment. For each reliable candidate pixel obtained through the previous steps in the current frame, its current 2D image coordinates and 3D motion vector are used to reverse-calculate the corresponding position of the point on the image plane of the previous frame. The initial depth value of the point in the previous frame is obtained by weighted calculation based on the depth values ​​of the four nearest neighbor pixels around the corresponding position of the pixel: assuming that the horizontal coordinate of the sub-pixel position in the coordinate system of the previous frame is x and the vertical coordinate is y, the coordinates of its four nearest neighbor integer pixels are: top left (x0, y0), top right (x1, y0), bottom left (x0, y1), and bottom right (x1, y1), where x0 is the floor value of x, x1 is x0+1, y0 is the floor value of y, and y1 is y0+1. In this embodiment, the horizontal relative distance is defined as x-x0, and the vertical relative distance is y-y0. Based on this, the weight coefficients of the four neighboring pixels are precisely set as follows:

[0111] Top-left pixel weight: set to (1 - horizontal relative distance) x (1 - vertical relative distance);

[0112] Top right corner pixel weight: set to horizontal relative distance x (1 - vertical relative distance);

[0113] The weight of the bottom left pixel is set to (1 - horizontal relative distance) x vertical relative distance.

[0114] Bottom right corner pixel weight: set as horizontal relative distance x vertical relative distance. Multiply the four determined weight coefficients by the depth values ​​of the corresponding neighboring pixels stored in the depth map of the previous frame, and add these four products together to obtain the initial depth value of the previous frame for a smooth transition.

[0115] The system reads the calculated 3D motion vector at the current moment. This vector is composed of the vehicle's translational velocity components (including instantaneous velocities in the forward, backward, left, right, and up / down directions) and rotational angular velocity components in its own coordinate system, comprehensively describing the instantaneous motion trend of the pixel in 3D space. To eliminate the influence of depth nonlinearity caused by changes in viewing angle, a unit vector of the current pixel's gaze direction in the vehicle coordinate system is first constructed. This vector points from the camera's optical center to the corresponding 3D space point, and its direction is uniquely determined by the camera's intrinsic parameter matrix (focal length, principal point coordinates) and the pixel's normalized coordinates. The dot product of the 3D motion vector and the unit vector of the gaze direction is calculated (i.e., the sum of the product of the corresponding components of the two vectors). If the translational component in the 3D motion vector points away from the camera, the dot product is positive, indicating that the object is moving away; if it points towards the camera, the dot product is negative, indicating that the object is approaching. Furthermore, considering that the vehicle's own rotation (such as turning or bumping) introduces a tangential motion component, the linear velocity component caused by rotation is calculated by cross-product of the rotational angular velocity and the distance from the pixel to the optical center. This tangential component is then removed from the total motion vector, retaining only the radial velocity component purely along the line of sight. This step ensures that the calculated rate of depth change is caused only by the relative approach or departure motion between the object and the camera, eliminating interference from lateral movement or rotation, thus accurately quantifying the rate of depth change over time.

[0116] Based on this, and adhering to the assumption of temporal continuity—that is, assuming that the motion of an object is uniform and smooth within an extremely short time interval (typically one frame, such as thirty milliseconds or less)—time extrapolation is performed using the initial depth value from the previous frame plus the product of the radial velocity component and that time interval to obtain the predicted depth value at the current moment. During this process, if the radial velocity component is positive, the predicted depth value increases; if it is negative, the predicted depth value decreases. This process follows the laws of three-dimensional spatial geometric transformation, converting the motion information of the two-dimensional image plane into the evolution trend of depth in three-dimensional space, thereby generating a physically meaningful and predictive depth reference value.

[0117] Step 5a5: The predicted depth value is weighted and fused with the depth value in the initial depth map of the current frame to obtain the first compensated depth value. The first compensated depth value is assigned to the current pixel. Then, all pixels marked as dynamic occlusion type in the initial depth map are traversed. Steps 5a1-5a5 are repeated to generate the first compensated depth map.

[0118] The weighting coefficient for the predicted depth value is 0.7, and the weighting coefficient for the current depth value is 0.3. The predicted depth value is combined with the depth value in the initial depth map of the current frame and their corresponding weighting coefficients, and then weighted and fused to obtain the first compensated depth value. After completing the fusion calculation for a single pixel, a global traversal and iterative update mechanism is executed. The calculated first compensated depth value is directly assigned to the corresponding position of the current pixel in the depth map, overwriting any invalid or low-confidence data. Subsequently, for all pixels marked as dynamically occluded, steps 5a1-5a5 are repeated to generate the first compensated depth map.

[0119] When dynamic occlusion occurs in a scene, the measured depth value directly acquired by the sensor in the current frame often reflects the surface distance of the occluder (such as a vehicle or pedestrian in front), rather than the true spatial position of the occluded target (such as a vehicle behind or a pedestrian in the distance). This results in severe false foreground jumps or holes in the depth data. In contrast, the predicted depth value, obtained by extrapolating the true state of the previous frame and the 3D motion vector, retains the true trajectory information of the occluded target in the continuous time dimension, based on the inertial laws of object motion. By fusing the two, the system is actually performing a physical model-based de-occlusion reconstruction process: using the historical true information contained in the predicted depth value that is not affected by occlusion, it strongly pulls back and covers the erroneous measured data generated by occlusion in the current frame, thereby mathematically achieving perspective restoration of the depth information of the occluded area. The fundamental value of this compensation mechanism lies in breaking the limitations of single-frame visual perception, elevating perception capabilities from relying on two-dimensional image features at the current moment to understanding the laws of motion in three-dimensional space. This ensures that even at the moment of occlusion, the system can still maintain accurate perception of the distance, speed, and trajectory of the occluded target, avoiding emergency braking or path planning errors caused by misjudging the occluded object as a real obstacle. It also prevents the accumulation of errors caused by relying entirely on historical predictions under extreme conditions, thereby constructing continuous, stable, and physically accurate deep perception results in dynamic and complex environments, providing an indispensable and highly reliable spatial benchmark for autonomous driving decisions.

[0120] When the current working condition is the first composite working condition, based on the pixels determined to be of the occlusion boundary type, an edge smoothing compensation strategy is executed to obtain a second compensation depth map, specifically including:

[0121] Step 5b1: Using the current pixel as the center, within a preset neighborhood, select pixels with a visibility confidence level greater than or equal to 3 as trusted reference pixels.

[0122] For the current pixel to be processed, a predefined geometric neighborhood is defined centered on the pixel's coordinates in the depth map. This neighborhood typically uses a square window (such as a sliding window of 3×3, 5×5, or 7×7 pixels) or a circular radius window (such as a radius of 3 pixels). Within this window, the system scans pixel by pixel to obtain the coordinates of all candidate pixels and their corresponding attribute information. The focus is on extracting the visibility confidence score of each candidate pixel, and only pixels with a visibility confidence score greater than or equal to 3 are marked as reliable reference pixels.

[0123] Step 5b2: Calculate the arithmetic mean of the depth values ​​of all the trusted reference pixels in the initial depth map.

[0124] Step 5b3: Preset correction weight coefficients based on the visibility confidence of the current pixel.

[0125] The visibility confidence of the current pixel is divided into three logical intervals: The first interval is the high confidence interval. When the visibility confidence is greater than or equal to 4, it is determined that the observation is extremely clear, and a correction weight of 1 is assigned, indicating that the original depth value of the current point is completely trusted and no neighborhood smoothing correction is performed to preserve edge details. The second interval is the medium confidence interval. When the visibility confidence is greater than or equal to 3 and less than 4 (including the boundary), it is determined that there is slight occlusion or texture blur. The correction weight is calculated as 1.0 − 0.2 × (4 − visibility confidence), so that the weight value smoothly decreases from 1 to 0.8 as the confidence decreases. This means that as the sharpness decreases, the neighborhood average value is gradually introduced for auxiliary correction. The third interval is the low confidence interval. When the visibility confidence is less than 3, a minimum weight of 0.6 is forcibly assigned, indicating that the reliability of the current point data is highly doubted and its own value is significantly reduced.

[0126] Step 5b4: The arithmetic mean is combined with the depth value in the initial depth map of the current frame and the correction weight coefficient to perform weighted fusion to obtain the second compensated depth value.

[0127] The original depth value of the current point is assigned a confidence level using a correction weight coefficient, while the remaining proportion (1 - correction weight coefficient) is allocated to the neighborhood arithmetic mean. When the visibility confidence of the current pixel is high, the correction weight coefficient approaches its maximum value, and the fused second compensated depth value closely resembles the original observation, thus preserving the edge details and texture features of the object to the greatest extent. Conversely, when the current pixel is in an occluded or weakly textured area resulting in low confidence, the correction weight coefficient automatically decreases, and the proportion of the neighborhood arithmetic mean increases accordingly. This makes the final output depth value more inclined to a smoother neighborhood statistical result, effectively suppressing noise interference and filling depth holes. The second compensated depth value is then generated.

[0128] Step 5b5: Assign the second compensation depth value to the current pixel and repeat steps 5b1-5b5. Traverse all pixels marked as occlusion boundary type until the preset maximum number of iterations is reached or the depth change converges, and finally generate the second compensation depth map.

[0129] After completing the depth correction at that point, the processing pointer moves to the next pixel marked as an occlusion boundary, and the entire process from step S5b1 to step S5b4 is repeated. The core of this iterative mechanism lies in using the results of previous corrections to drive subsequent corrections: as the depth value of the first occlusion boundary pixel is updated, its new depth information immediately becomes one of the input data for calculating the arithmetic mean of other surrounding occlusion boundary pixels in subsequent iterations. This chain reaction allows the depth correction effect to spread rapidly and diffuse within the occluded area like ripples, gradually eliminating depth discontinuities and voids caused by occlusion.

[0130] Set the maximum number of iterations to 8 to 10. If convergence is not achieved after this number, it indicates that there may be abnormal large-area occlusion or noise interference. Continuing to iterate will not only fail to improve accuracy, but may also lead to error accumulation or waste of computational resources.

[0131] After each complete traversal iteration, the absolute difference between the depth value of all processed pixels in the current iteration and the depth value in the previous iteration is calculated. A very small preset threshold (e.g., 0.01 meters or 1 centimeter) is set as the standard for judging whether the depth map is stable. If, after a certain iteration, the maximum absolute difference of all pixels is less than this threshold, the depth map is considered to have reached a convergent state. This means that the current depth distribution has become stable, and the correction amplitude produced by further iterations is negligible, contributing little to the final perception result. Once either the maximum number of iterations or the depth change convergence condition is met, the loop terminates immediately, and the final second compensated depth map is output. This depth map not only repairs depth breaks at occlusion boundaries but also maintains the smooth transition of object surfaces, providing high-confidence 3D spatial information for downstream path planning and obstacle avoidance decisions.

[0132] When the current working condition is the first composite working condition, based on the pixels determined to be non-occluded, a terrain following compensation strategy is executed to obtain a third compensated depth map, including the following steps:

[0133] Step 5c1: If the ground adhesion residual of the current pixel is less than 0.1m, mark it as ground-attached state; otherwise, mark it as suspended state.

[0134] First, the depth measurement value of the pixel to be processed in the initial depth map is retrieved. Combined with high-precision vehicle self-localization data (such as IMU attitude angle, wheel speed odometer) and pre-calibrated camera extrinsic parameter matrix, the three-dimensional spatial coordinates corresponding to the pixel are constructed. Using the vehicle's current pitch and roll angle information, the three-dimensional coordinates are projected onto an ideal horizontal ground plane in the vehicle coordinate system to calculate the theoretical ground contact height. If the absolute value of the calculated ground contact residual is less than 0.1 meters (i.e., 10 centimeters), the pixel is immediately marked as ground contact. This usually means that the point belongs to the road surface itself, a low curb, or a stationary obstacle close to the ground, and its height change conforms to the physical characteristics of the road. Conversely, if the ground contact residual is greater than or equal to 0.1 meters, it is marked as suspended, indicating that the point may belong to a moving vehicle, pedestrian, road sign, tree, or other dynamic / static obstacle detached from the ground.

[0135] Step 5c2: Obtain the three-dimensional third space point of the current pixel point back-projected onto the vehicle coordinate system;

[0136] The system reads the two-dimensional coordinates of the current pixel on the image sensor and its corresponding depth value in the initial depth map. Then, it calls the pre-calibrated and real-time calibrated camera intrinsic parameter matrix and uses the pinhole camera model to convert the depth value into three-dimensional coordinates in the camera coordinate system with the camera optical center as the origin. Next, the system combines the vehicle's current real-time pose data, including the vehicle roll angle, pitch angle, and yaw angle calculated by the inertial measurement unit (IMU) and wheel speed sensor, as well as the vehicle extrinsic parameter transformation matrix obtained by high-precision GNSS / RTK positioning (i.e., the rotation matrix and translation vector from the camera coordinate system to the vehicle coordinate system), to perform a rigid transformation operation on the three-dimensional point in the camera coordinate system: first, the point in the camera coordinate system is aligned in attitude by the rotation matrix, and then the translation vector is superimposed to complete the position offset. Finally, the complete three-dimensional coordinates of the third space point corresponding to the pixel in the vehicle coordinate system are accurately calculated.

[0137] Step 5c3: Based on the current pixel marked as grounded, construct a first rotation matrix according to the pitch and roll angles, and obtain a first transformed 3D point based on the 3D third space point and the first rotation matrix; based on the current pixel marked as suspended, construct a second rotation matrix according to the pitch angle, and obtain a second transformed 3D point based on the 3D third space point and the second rotation matrix.

[0138] The system reads the pitch angle calculated by the vehicle's inertial measurement unit (IMU) in real time. This angle accurately represents the degree of tilt of the vehicle's longitudinal axis relative to the horizontal plane during acceleration, braking, or going uphill / downhill. Then, a second rotation matrix is ​​constructed based on this pitch angle. Mathematically, this matrix only operates on the vehicle's lateral and vertical coordinate planes. Its core function is to project and correct spatial points in the vertical direction while strictly preserving their relative positional relationships in the vehicle's lateral and depth directions, thus ensuring that the transformation process does not introduce geometric distortions caused by tilt. Next, the system performs matrix multiplication with the calculated coordinate vector of the third-dimensional spatial point and the second rotation matrix. Through this linear transformation, the spatial point, originally in the tilted vehicle coordinate system, is pulled back to an ideal reference system assuming the vehicle is in an absolutely horizontal state, ultimately obtaining the second transformed three-dimensional point. This processing result allows the vertical height component of suspended objects (such as pedestrians, vehicles in front, road signs, etc.) to directly reflect their true physical height relative to the ideal horizontal ground, completely eliminating visual height errors caused by the vehicle's own pitch movements.

[0139] Step 5c4: Project the first transformed 3D point and the second transformed 3D point back onto the camera imaging plane, and extract the Z-axis component in the camera coordinate system as the third compensation depth value.

[0140] The first and second transformed 3D points are reprojected back to the 2D camera imaging plane from their 3D spatial coordinates in the world or vehicle coordinate system using the inverse operation of the camera intrinsic parameter matrix. This projection process not only restores the 2D position of the pixel on the image sensor but, more importantly, preserves its 3D spatial attributes after attitude normalization. Subsequently, the Z-axis component (i.e., the depth value along the optical axis) in the camera coordinate system is extracted from these two transformed 3D points and defined as the third compensated depth value. For pixels in a ground-hugging state, this third compensated depth value eliminates the dual interference of vehicle tilt and pitch on road distance perception, accurately reflecting the vertical distance from the vehicle to the road surface in an ideal horizontal state. For pixels in a suspended state, this value eliminates the height visual error caused by the pitch angle, retaining only the pure depth information of the object along the optical axis. The high-precision depth feature of the third compensated depth value, after complex kinematic calculations and geometric corrections, directly characterizes the true optical depth of the target object relative to the camera's optical center under the current dynamic attitude of the vehicle.

[0141] Step 5c5: Assign the third compensation depth value to the current pixel, traverse all pixels marked as non-occluded, and repeat steps 5c1-5c5 to obtain the third compensation depth map.

[0142] The third compensation depth value is directly assigned to the currently processed pixel, completing the update and solidification of the pixel in the depth data stream. All valid pixel regions marked as non-occluded are accurately selected, and pixels occluded by foreground objects or in invalid regions are automatically skipped to ensure the purity and accuracy of the depth data. For each selected valid pixel, it is used as a new input reference, and the entire processing flow from step S5c1 to step S5c5 is repeatedly executed. This iterative mechanism ensures that every valid piece of information in the image can independently undergo complete pose decoupling, geometric correction, and depth reconstruction, thereby eliminating systematic depth deviations caused by vehicle dynamics (such as acceleration, braking, and turning). After traversing and covering all non-occluded pixels in the image, all compensated and corrected depth values ​​are aggregated and reorganized according to the row and column structure of the original image to finally synthesize a third compensation depth map containing high-precision depth information of the entire scene.

[0143] When the current working condition is the second or third composite working condition, based on the pixels determined to be of the dynamic occlusion type, a neighborhood smoothing compensation strategy is executed to obtain a fourth compensation depth map, including the following steps:

[0144] Step 5d1: Set a basic neighborhood window centered on the current pixel and filter out candidate pixels with texture similarity higher than a preset threshold.

[0145] Using the current pixel to be processed as the geometric center, a basic neighborhood window of a preset size (e.g., a 5×5 or 7×7 square sliding window) is constructed, which covers the spatial range around the current pixel. Then, by extracting the local texture feature vectors (e.g., gray-level gradient histograms) of all candidate pixels within the window, the vectors are compared with the feature vector of the center pixel, and the similarity score between the two is calculated (usually using cosine similarity or the reciprocal of Euclidean distance). Based on this, the calculation result is compared in real time with a preset texture similarity threshold (0.7), and only those pixels with similarity scores higher than the threshold are retained as reserve pixels. These reserve pixels show a high degree of consistency in visual texture and are highly likely to belong to the same continuous surface or the same type of object, thereby effectively eliminating interference items such as edge noise, the boundary between different materials, or distant background.

[0146] Step 5d2: Perform a brightness gradient consistency check on the prepared pixel points to obtain a smooth reference point.

[0147] The smoothness of the surface of a region is quantified by calculating the rate of grayscale change (i.e., the first-order partial derivative) between each pre-selected pixel and its immediate neighbors (e.g., an eight-neighborhood). The local gradient magnitude of each pre-selected pixel is compared with the average gradient magnitude of all pre-selected pixels in its neighborhood. If the gradient magnitude of a pixel significantly deviates from the average, it is determined to be a gradient inconsistency point. Based on this, these outliers that do not conform to the smoothing assumption are automatically removed, retaining only pixels with gentle gradient changes and consistent directions, which are defined as smoothing reference points. This process essentially performs high-pass filtering on the image in the frequency domain, effectively filtering out false texture details caused by reflections from minor road surface potholes, raindrop streaks, or random sensor noise, while strictly preserving the natural transition features of real object edges and continuous curved surfaces. The final set of smoothing reference points not only inherits the spatial location information of the original pre-selected pixels but also possesses a high degree of photometric continuity.

[0148] Step 5d3: Calculate the weighted average of the depth values ​​of all smoothed reference points in the initial depth map of the current frame;

[0149] For smooth reference points located within 1-3 pixels of the center pixel (i.e., the immediate neighboring region), a high weight of 0.45 is assigned to emphasize the spatial relevance of the core neighborhood; for smooth reference points located within 4-6 pixels (i.e., the secondary neighborhood), a medium weight of 0.30 is assigned to supplement the spatial extension; for smooth reference points located within 7-9 pixels (i.e., the outer neighborhood), a low weight of 0.15 is assigned to provide macroscopic geometric constraints; and for reference points located more than 9 pixels away, they are ignored (weight 0.00). Based on this, the system multiplies the depth values ​​of all valid reference points by their corresponding specific weights (0.45, 0.30, or 0.15), sums them, and then divides by the sum of all non-zero weights (i.e., 0.90) to calculate the final weighted average depth value.

[0150] Step 5d4: The weighted average value is weighted and fused with the depth value in the initial depth map of the current frame to obtain the fourth compensated depth value;

[0151] Step 5d5: Assign the fourth compensation depth value to the current pixel, traverse all pixels marked as dynamic occlusion type, and repeat steps 5d1-5d5 to obtain the fourth compensation depth map.

[0152] A fixed weight of 70% is applied to the weighted average, emphasizing the dominance of geometric consistency information derived from surrounding high-confidence regions, to effectively correct holes caused by random noise or occlusion that may exist in the initial depth map. At the same time, a fixed weight of 30% is applied to the depth value in the initial depth map of the current frame to preserve the original geometric features observed by the current sensor and prevent the loss of details due to excessive smoothing in areas with extremely rich textures or abrupt edge changes. 70% of the weighted average and 30% of the initial depth value are superimposed to obtain a fourth compensated depth value that is both smooth and retains details. Then, in step 5d5, the generated fourth compensation depth value is directly mapped and assigned to the currently processed dynamic occlusion type pixel as the final depth output for that location. Next, the coordinates of all pixels in the image marked as dynamic occlusion type are scanned sequentially. For each target pixel, the entire processing flow from extracting the neighborhood, filtering preparatory points, gradient verification to weighted fusion is repeated until the entire marked region is traversed. Finally, all compensation results are integrated to construct a complete and high-precision fourth compensation depth map. This depth map can effectively eliminate depth breaks caused by dynamic object occlusion, ensuring the perception continuity and safety of the autonomous driving system in complex traffic scenarios.

[0153] When the current working condition is the second or third composite working condition, based on the pixels that are determined to be occlusion boundary type, the depth values ​​of the four direct neighboring pixels above, below, left, and right of the current pixel are obtained and the arithmetic mean is calculated as the fifth compensation depth value, which is assigned to the current pixel. All pixels marked as occlusion boundary type are traversed, and the fifth compensation depth value is repeatedly calculated to obtain the fifth compensation depth map. When the current working condition is the second or third composite working condition, based on the pixels that are determined to be non-occlusion type, the basic compensation strategy is executed.

[0154] First, using the current pixel as the center, accurately extract the depth data of its four direct neighboring pixels: directly above, below, to the left, and to the right. Then, sum the depth values ​​of these four neighboring pixels and divide by four to calculate the arithmetic mean. This value is considered the fifth compensation depth value, capable of bridging boundary gaps and maintaining local continuity. This value is immediately assigned to the pixel currently at the occlusion boundary, effectively repairing depth jumps and breaks caused by object movement or sudden changes in lighting. Next, sequentially scan and process all pixels in the image marked as occlusion boundaries, repeating the above operations of extracting neighbors, calculating the mean, and assigning values ​​until a complete fifth compensation depth map is generated. This map significantly improves the depth coherence at occlusion edges. Meanwhile, for pixels in the same frame that are determined to be non-occluded, a theoretical depth reference value is constructed. The theoretical depth reference value is a smooth transition value between 95% and 105% of the initial depth value of the current pixel to simulate the physical motion trajectory in the real world. The original measurement value of the current pixel is compared with the theoretical depth reference value to calculate the difference. If the difference exceeds a preset threshold range (e.g., 5% of the initial depth value), a specific correction operation is performed: the corrected depth = original depth value × 80% + theoretical depth reference value × 20%, and the corresponding compensated depth map is obtained.

[0155] After generating the compensated depth map, the system first uses this map to correct the geometric errors of traditional algorithms. The original images captured by multiple cameras are uniformly flattened onto the bird's-eye view plane below the vehicle through inverse perspective transformation (IPM), ensuring that the proportions of objects near and far are realistic and without stretching. Next, intelligent fusion and blind spot filling are performed: based on the compensated depth map, invalid pixels caused by vehicle body occlusion are automatically identified and removed. At the same time, based on the continuity of depth information, complementary images are extracted from other unobstructed perspectives to seamlessly fill the edges of blind spots, generating a complete and unbroken 360-degree panoramic image. Finally, the system combines this corrected panoramic image with accurate depth data and directly maps it into a three-dimensional spatial model. This not only restores the real texture of the environment around the vehicle but also reconstructs the height and distance information of obstacles, thus outputting a clear, intuitive, and accurately geometric three-dimensional surround view scene, enabling the autonomous driving system to accurately perceive the surrounding environment.

[0156] like Figure 2 As shown, this embodiment also discloses a system for dynamic elimination and 3D reconstruction of blind spots in vehicle surround view images, including the following modules:

[0157] The data acquisition module is used to acquire multi-view raw images based on multiple fisheye cameras installed around the vehicle; and to obtain the current vehicle status signal through vehicle sensors.

[0158] The data processing module is used to obtain an initial depth map based on the multi-view original image;

[0159] The identification module is used to identify the current driving condition based on the vehicle status signal;

[0160] The segmentation module is used to classify the occlusion types of pixels in the initial depth map based on the visibility confidence, motion mutation index and ground adhesion residual.

[0161] The compensation analysis module is used to perform differential dynamic compensation based on pixels under different driving conditions and belonging to different occlusion types, combined with the visibility confidence and / or motion mutation index and / or ground adhesion residual and the depth value in the initial depth map, to generate a compensated depth map; based on the compensated depth map and the original images from multiple perspectives, it performs three-dimensional inverse perspective transformation and image fusion to output a surround view image with blind spots eliminated.

[0162] To implement the above embodiments, this application also proposes an electronic device. Please see [link to relevant documentation]. Figure 3 , Figure 3 This is a schematic diagram of the structure of the electronic device provided in an embodiment of this application. For example... Figure 3 As shown, the electronic device 500 includes: a processor 501 and a memory 502 communicatively connected to the processor 501; the memory 502 stores computer-executable instructions; the processor 501 executes the computer-executable instructions stored in the memory to implement the method provided in the foregoing embodiments.

[0163] To implement the above embodiments, this application also proposes a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the methods provided in the foregoing embodiments.

[0164] To implement the above embodiments, this application also proposes a computer program product, including a computer program that, when executed by a processor, implements the methods provided in the foregoing embodiments.

[0165] The storage medium mentioned above can be a read-only memory, a disk, or an optical disk, etc. Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of this application.

Claims

1. A blind area dynamic elimination and 3D reconstruction method for vehicle surround view images, characterized in that, Includes the following steps: Step 1: Collect multi-view raw images using multiple fisheye cameras installed around the vehicle; obtain the current vehicle status signal through vehicle sensors, including pitch angle and roll angle. Step 2: Obtain an initial depth map based on the multi-view original images; Step 3: Identify the current driving condition based on the vehicle status signal. The driving condition includes a first composite driving condition, a second composite driving condition, and a third composite driving condition. The first composite driving condition is a condition that simultaneously satisfies a gradient of ≥8%, a target speed steering, and the presence of dynamic obstacles. The second composite working condition is a working condition in which only one of the following three conditions is met: slope ≥ 8%, target speed turning, and dynamic obstacle; the third composite working condition is a working condition in which any two of the following three conditions are met: slope ≥ 8%, target speed turning, and dynamic obstacle; based on the identified current working condition, the visibility confidence, motion mutation index, and ground adhesion residual of each pixel in the initial depth map are calculated respectively. Step 4: Based on the visibility confidence, motion mutation index and ground bonding residual, classify the occlusion types of the pixels in the initial depth map and identify dynamic occlusion type, non-occlusion type and occlusion boundary type. Step 5: Based on pixels under different driving conditions and belonging to different occlusion types, combined with the visibility confidence and / or motion mutation index and / or ground adhesion residual and the depth value in the initial depth map, perform differentiated dynamic compensation to generate a compensated depth map; based on the compensated depth map and the original images from multiple perspectives, perform three-dimensional inverse perspective transformation and image fusion to output a surround view image with blind spots eliminated. 2.The blind area dynamic elimination and 3D reconstruction method of vehicle surround view image according to claim 1, wherein, Obtain the initial depth map of the current frame and the initial depth map of the previous frame, calculate the absolute value of the depth difference of the same pixel and the frame time interval, and obtain the motion mutation index based on the absolute value of the depth difference and the frame time interval. Based on the pitch and roll angles, construct the inclined local ground plane equation; back-project each pixel in the initial depth map into a three-dimensional first space point in the vehicle coordinate system according to the camera intrinsic parameters and depth value, calculate the vertical Euclidean distance between the three-dimensional first space point and the inclined local ground plane equation, and mark it as the ground fitting residual; The first fisheye camera among the multiple fisheye cameras installed around the vehicle is selected as the reference viewpoint. Each pixel in the reference viewpoint image is defined as the target pixel. Based on the intrinsic parameter matrix of the first fisheye camera and the depth value corresponding to the target pixel in the initial depth map, the target pixel is back-projected from the image coordinate system to the vehicle coordinate system to obtain the corresponding three-dimensional second space point. Using the intrinsic and extrinsic parameter matrices of the remaining fisheye cameras respectively, the three-dimensional second space points are reprojected onto the image plane corresponding to the remaining fisheye cameras to obtain the reprojection coordinate set. Based on each reprojection coordinate in the corresponding fisheye camera image, a neighborhood window of a preset size is extracted centered on the current reprojection coordinate; it is then determined whether the pixels within the neighborhood window satisfy the following two conditions: Depth consistency: The difference between the effective pixel depth value within the neighborhood window and the depth value of the three-dimensional second space point is less than a preset depth threshold; Feature similarity: The similarity between the features of the pixels within the neighborhood window and the features of the target pixel is greater than a preset similarity threshold; The number of fisheye cameras that simultaneously meet both of the above conditions is counted, and the total number is summed. If the total is greater than 0, the total is used as the visibility confidence of the target pixel; otherwise, the visibility confidence of the target pixel is determined to be 0. 3.The blind area dynamic elimination and 3D reconstruction method of vehicle surround view image according to claim 2, characterized in that, If the visibility confidence is 0, the current pixel is determined to be a dynamic occlusion type; if the visibility confidence belongs to the set [1,2] and the motion mutation index is greater than 0.8 m / s, the current pixel is determined to be an occlusion boundary type; if the visibility confidence is greater than or equal to 3, or if the visibility confidence belongs to the set [1,2] and the motion mutation index is less than or equal to 0.8 m / s, the current pixel is determined to be a non-occlusion type.

4. The method for dynamic elimination of blind spots and three-dimensional reconstruction of vehicle surround view images according to claim 3, characterized in that, When the current working condition is the first composite working condition, a prediction compensation strategy is executed based on the pixels determined to be of the dynamic occlusion type to obtain a first compensation depth map, specifically including: Step 5a1: Obtain the two-dimensional motion vector of the current pixel on the image plane; based on the depth value of the current pixel in the initial depth map and the camera intrinsic parameters, solve the two-dimensional motion vector into a three-dimensional motion vector in the vehicle coordinate system. Step 5a2: Taking the current pixel as the center, determine the dynamic radius of the neighborhood window based on the magnitude of the three-dimensional motion vector and the preset neighborhood window radius. Determine the neighborhood expansion window based on the dynamic radius of the neighborhood window, and define all pixels within the neighborhood expansion window as candidate pixels. Traverse all candidate pixels, calculate the gradient difference between the current pixel and the candidate pixels in the multi-view original image, and mark all candidate pixels with gradient differences less than a preset texture threshold as candidate pixels with similar texture features. Step 5a3: Perform triangulation consistency verification based on the candidate pixels with similar texture features to obtain reliable candidate pixel points; Step 5a4: Obtain the depth value of the trusted candidate pixel in the initial depth map of the previous frame, and perform time extrapolation along the direction of the three-dimensional motion vector to obtain the predicted depth value; Step 5a5: The predicted depth value is weighted and fused with the depth value in the initial depth map of the current frame to obtain the first compensated depth value. The first compensated depth value is assigned to the current pixel. Then, all pixels marked as dynamic occlusion type in the initial depth map are traversed. Steps 5a1-5a5 are repeated to generate the first compensated depth map.

5. The method for dynamic elimination of blind spots and three-dimensional reconstruction of vehicle surround view images according to claim 4, characterized in that, When the current working condition is the first composite working condition, based on the pixels determined to be of the occlusion boundary type, an edge smoothing compensation strategy is executed to obtain a second compensation depth map, specifically including: Step 5b1: Using the current pixel as the center, within a preset neighborhood, select pixels with a visibility confidence level greater than or equal to 3 as trusted reference pixels. Step 5b2: Calculate the arithmetic mean of the depth values ​​of all the trusted reference pixels in the initial depth map; Step 5b3: Preset correction weight coefficients based on the visibility confidence of the current pixel; Step 5b4: The arithmetic mean is combined with the depth value in the initial depth map of the current frame and the correction weight coefficient to perform weighted fusion to obtain the second compensated depth value; Step 5b5: Assign the second compensation depth value to the current pixel and repeat steps 5b1-5b5. Traverse all pixels marked as occlusion boundary type until the preset maximum number of iterations is reached or the depth change converges, and finally generate the second compensation depth map.

6. The method for dynamic elimination of blind spots and three-dimensional reconstruction of vehicle surround view images according to claim 5, characterized in that, When the current working condition is the first composite working condition, based on the pixels determined to be non-occluded, a terrain following compensation strategy is executed to obtain a third compensated depth map, including the following steps: Step 5c1: If the ground adhesion residual of the current pixel is less than 0.1m, mark it as ground-attached state; otherwise, mark it as suspended state. Step 5c2: Obtain the three-dimensional third space point of the current pixel point back-projected onto the vehicle coordinate system; Step 5c3: Based on the current pixel marked as being in a ground-hugging state, construct a first rotation matrix according to the pitch and roll angles, and obtain a first transformed 3D point based on the 3D third space point and the first rotation matrix; based on the current pixel marked as being in a suspended state, construct a second rotation matrix according to the pitch angle, and obtain a second transformed 3D point based on the 3D third space point and the second rotation matrix. Step 5c4: Project the first transformed 3D point and the second transformed 3D point back onto the camera imaging plane, and extract the Z-axis component in the camera coordinate system as the third compensation depth value. Step 5c5: Assign the third compensation depth value to the current pixel, traverse all pixels marked as non-occluded, and repeat steps 5c1-5c5 to obtain the third compensation depth map.

7. The method for dynamic elimination of blind spots and three-dimensional reconstruction of vehicle surround view images according to claim 6, characterized in that, When the current working condition is the second or third composite working condition, based on the pixels determined to be of the dynamic occlusion type, a neighborhood smoothing compensation strategy is executed to obtain a fourth compensation depth map, including the following steps: Step 5d1: Set a basic neighborhood window centered on the current pixel and filter out candidate pixels with texture similarity higher than a preset threshold. Step 5d2: Perform brightness gradient consistency verification on the prepared pixels to obtain smooth reference points; Step 5d3: Calculate the weighted average of the depth values ​​of all smoothed reference points in the initial depth map of the current frame; Step 5d4: The weighted average value is weighted and fused with the depth value in the initial depth map of the current frame to obtain the fourth compensated depth value; Step 5d5: Assign the fourth compensation depth value to the current pixel, traverse all pixels marked as dynamic occlusion type, and repeat steps 5d1-5d5 to obtain the fourth compensation depth map.

8. The method for dynamic elimination of blind spots and three-dimensional reconstruction of vehicle surround view images according to claim 7, characterized in that, When the current working condition is the second or third composite working condition, based on the pixels that are determined to be occlusion boundary type, the depth values ​​of the four direct neighboring pixels above, below, left, and right of the current pixel are obtained and the arithmetic mean is calculated as the fifth compensation depth value, which is assigned to the current pixel. All pixels marked as occlusion boundary type are traversed, and the fifth compensation depth value is repeatedly calculated to obtain the fifth compensation depth map. When the current working condition is the second or third composite working condition, based on the pixels that are determined to be non-occlusion type, the basic compensation strategy is executed.

9. A system for dynamic blind spot elimination and 3D reconstruction of vehicle surround view images, used to execute the method for dynamic blind spot elimination and 3D reconstruction of vehicle surround view images as described in any one of claims 1-8, characterized in that, Includes the following modules: The data acquisition module is used to acquire multi-view raw images based on multiple fisheye cameras installed around the vehicle; and to obtain the current vehicle status signal through vehicle sensors. The data processing module is used to obtain an initial depth map based on the multi-view original image; The identification module is used to identify the current driving condition based on the vehicle status signal; The segmentation module is used to classify the occlusion types of pixels in the initial depth map based on the visibility confidence, motion mutation index and ground adhesion residual. The compensation analysis module is used to perform differential dynamic compensation based on pixels under different driving conditions and belonging to different occlusion types, combined with the visibility confidence and / or motion mutation index and / or ground adhesion residual and the depth value in the initial depth map, to generate a compensated depth map; based on the compensated depth map and the original images from multiple perspectives, it performs three-dimensional inverse perspective transformation and image fusion to output a surround view image with blind spots eliminated.

Images