Transparent tray filling method, transparent tray filling device, and computer storage medium

Abstract

The application provides a transparent bottom disc filling method, a transparent bottom disc filling device and a computer storage medium. The transparent bottom disc filling method comprises the following steps: acquiring a current frame panoramic overhead view and a historical frame vehicle panoramic image; a matching grid area is positioned in the current frame panoramic overhead view, wherein the matching grid area does not coincide with a transparent bottom disc filling area; a vehicle running parameter from a historical frame time to a current frame time is determined; the historical frame vehicle panoramic image is transformed into a coordinate system of the current frame panoramic overhead view based on the vehicle running parameter, so as to obtain a transformed vehicle panoramic image; a historical frame transparent bottom disc image of the transparent bottom disc filling area is cropped from the transformed vehicle panoramic image; and the historical frame transparent bottom disc image is overlaid on the transparent bottom disc filling area of the current frame panoramic overhead view, so as to generate a vehicle panoramic image. Through the above transparent bottom disc filling method, a high-quality surround view transparent bottom disc effect is realized.

Description

Technical Field

[0001] This application relates to the field of vehicle panoramic imaging technology, and in particular to a transparent chassis filling method, a transparent chassis filling device, and a computer storage medium. Background Technology

[0002] Transparent chassis technology is a technique that uses cameras mounted on the underside of a vehicle to capture and display the ground beneath it. The images captured by these cameras are transmitted to a display screen inside the vehicle, allowing the driver to see the actual conditions under the chassis as if it were transparent. This technology helps improve driving safety, especially in complex road conditions, such as when needing to avoid obstacles like manhole covers or potholes, or when engaging in off-road driving.

[0003] However, existing transparent chassis solutions face implementation bottlenecks in aftermarket vehicle scenarios, making it difficult to achieve real-time chassis visibility. Summary of the Invention

[0004] To address the aforementioned technical problems, this application proposes a transparent chassis filling method, a transparent chassis filling device, and a computer storage medium.

[0005] To address the aforementioned technical problems, this application proposes a transparent chassis filling method, which includes: Obtain the panoramic top-down view of the current frame, as well as panoramic images of the vehicle in historical frames; In the current frame panoramic top view, a matching grid region is located, wherein the matching grid region does not coincide with the transparent chassis filling region; Determine the vehicle operating parameters from the historical frame time to the current frame time; Based on the vehicle operating parameters, the historical frame panoramic vehicle image is transformed to the coordinate system of the current frame panoramic top view to obtain the transformed panoramic vehicle image. The historical frame transparent chassis image of the transparent chassis filling area is cropped from the transformed vehicle panoramic image; The transparent chassis image from the historical frame is overlaid on the transparent chassis-filled area of ​​the current frame panoramic top view to generate a panoramic vehicle image.

[0006] The determination of vehicle operating parameters from the historical frame time to the current frame time includes: Transform the image region corresponding to the matching grid region in the historical frame vehicle panoramic image to the current frame coordinate system using the initial running parameters; A nonlinear least squares problem is constructed, with the pixel error between the transformed region and the matching grid region in the current frame panoramic top view as the cost function; The operating parameters that minimize the cost function are obtained by using an iterative optimization algorithm and are then used as the vehicle operating parameters.

[0007] The transparent chassis filling method further includes: Construct a pyramid structure for the matching grid region, wherein the pyramid structure includes multiple image layers of different resolutions, with the top layer having the lowest resolution and the bottom layer having the highest resolution; Starting from the top layer of the pyramid structure, perform the following operations: At the current layer resolution, the corresponding region in the historical frame vehicle panoramic image is transformed to the current frame coordinate system using the initial running parameters; A nonlinear least squares problem is constructed, with the pixel error between the transformed region and the matching grid region of the current frame as the cost function; An iterative optimization algorithm is used to solve for the operating parameters that minimize the cost function; The optimized operating parameters of the current layer are amplified and used as the initial operating parameters for the next layer with a higher resolution. The final optimized operating parameters obtained from the bottom layer of the pyramid structure are output as the vehicle operating parameters.

[0008] The initial operating parameters are the vehicle operating parameters obtained by optimizing the historical frame vehicle panoramic image.

[0009] The vehicle operating parameters include: horizontal translation, vertical translation, and rotation angle.

[0010] The step of locating and matching the grid region in the current frame panoramic top view includes: The current frame panoramic top view is divided into several grid regions; Calculate the gradient magnitude of each grid region and select the high gradient grid regions as the matching grid regions.

[0011] The step of locating and matching the grid region in the current frame panoramic top view includes: Calculate the gradient magnitude of each pixel in the current frame panoramic top view, and select high gradient pixels as matching pixels; The matching grid region is determined with the matching pixel as the center.

[0012] The transparent chassis filling method further includes, after obtaining the current frame panoramic top view and the historical frame vehicle panoramic image, the method further includes: The average pixel value of the historical frame vehicle panoramic image is obtained as a reference value, wherein the average pixel value is the average value of a pixel point or the average value of a pixel region; The ratio of the reference value to the pixel value of the current frame panoramic top view is calculated as the optimization value; The optimized value is used in conjunction with the current frame panoramic top view for optimization.

[0013] The transparent chassis filling method further includes, after determining the vehicle operating parameters from the historical frame time to the current frame time: Robust smoothing processing is performed on the vehicle operating parameters, wherein the robust smoothing processing includes: Calculate the increment of vehicle operating parameters in the current frame panoramic top view relative to the operating parameters in the previous normal frame panoramic top view; Based on the incremental statistics window of the running parameters of the previous normal frame panoramic top view, calculate the median and absolute median difference of the increments in each dimension. The robust Z-score of the current frame increment is calculated based on the median and the absolute median difference, and the current frame panoramic top view is determined to be an abnormal frame when the robust Z-score exceeds a preset threshold. For abnormal frames, the vehicle operating parameters of the current frame are predicted using a constant speed model as a substitute output.

[0014] The step of predicting vehicle operating parameters for the current frame using a constant speed model includes: Calculate the velocity vector based on the vehicle operating parameters of the two most recent normal frame panoramic top view; The velocity vector is superimposed onto the vehicle operating parameters of the most recent normal frame panoramic top view to obtain the vehicle operating parameters predicted by the current frame panoramic top view.

[0015] To address the aforementioned technical problems, this application also proposes a transparent chassis filling device, which includes a memory and a processor coupled to the memory; wherein the memory is used to store program data, and the processor is used to execute the program data to implement the transparent chassis filling method as described above.

[0016] To address the aforementioned technical problems, this application also proposes a computer storage medium for storing program data, which, when executed by a computer, is used to implement the aforementioned transparent chassis filling method.

[0017] Compared with the prior art, the beneficial effects of this application are: the transparent chassis filling device uses the grid area matching of the historical frame vehicle panoramic image and the current frame panoramic top view to solve the vehicle operation parameters from the historical frame to the current frame, thereby completing the transfer of image data of the historical frame vehicle panoramic image to the transparent chassis filling area of ​​the current frame panoramic top view, so as to complete the generation of the vehicle panoramic image. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. Wherein: Figure 1 This is a flowchart illustrating the first embodiment of the transparent chassis filling method provided in this application; Figure 2 This is a complete flowchart of the transparent chassis filling solution provided in this application; Figure 3 This is a flowchart illustrating the second embodiment of the transparent chassis filling method provided in this application; Figure 4 This is a schematic diagram of an embodiment of the division of the transparent chassis filling area and the image optimization area provided in this application; Figure 5 This is a flowchart illustrating the third embodiment of the transparent chassis filling method provided in this application; Figure 6 This is a flowchart illustrating the fourth embodiment of the transparent chassis filling method provided in this application; Figure 7 This is a flowchart illustrating the fifth embodiment of the transparent chassis filling method provided in this application; Figure 8 yes Figure 7 The diagram shows the detailed process flow of steps S52 and S53 of the transparent chassis filling method. Figure 9 This is a schematic flowchart of an embodiment of the robust smoothing process in the transparent chassis filling method provided in this application; Figure 10 This is a schematic diagram of the overall process of the robust smoothing and continuous exception reset mechanism provided in this application; Figure 11 This is a schematic diagram of the vehicle coordinate system provided in this application; Figure 12 This is a schematic diagram of an embodiment of the panoramic vehicle image provided in this application; Figure 13 This is a schematic diagram of another embodiment of the panoramic vehicle image provided in this application; Figure 14 This is a schematic diagram of the structure of an embodiment of the transparent chassis filling device provided in this application; Figure 15 This is a schematic diagram of the structure of an embodiment of the computer storage medium provided in this application. Detailed Implementation

[0019] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0020] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a particular order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the application described herein can be implemented, for example, in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0021] This application proposes a method for implementing a surround-view transparent chassis based on photometric errors for aftermarket vehicle applications. This method aims to address the limitations in transparent chassis functionality and poor display quality caused by missing CAN message information or misaligned image data timestamps.

[0022] Please continue reading. Figure 1 and Figure 2 , Figure 1 This is a flowchart illustrating the first embodiment of the transparent chassis filling method provided in this application. Figure 2 This is a complete flowchart of the transparent chassis filling solution provided in this application.

[0023] The transparent chassis filling method of this application is applied to a transparent chassis filling device. This transparent chassis filling device can be a server, a terminal device, or a system in which the server and terminal device cooperate. Accordingly, all components of the transparent chassis filling device, such as units, subunits, modules, and submodules, can be entirely housed in the server, entirely housed in the terminal device, or separately housed in the server and the terminal device.

[0024] Furthermore, the aforementioned server can be either hardware or software. When the server is hardware, it can be implemented as a distributed server cluster consisting of multiple servers, or as a single server. When the server is software, it can be implemented as multiple software programs or software modules, such as software or software modules used to provide distributed server functionality, or as a single software program or software module; no specific limitations are made here.

[0025] It should be noted that the transparent chassis filling device of this application can be an intelligent terminal mounted on a vehicle, especially an autonomous vehicle.

[0026] like Figure 1 As shown, the specific steps are as follows: Step S11: Obtain the panoramic top view of the current frame and the panoramic images of the vehicles in the historical frames.

[0027] In this embodiment of the application, the transparent chassis filling device acquires a panoramic view of the vehicle in each acquisition cycle. For example, this application deploys four (six or eight) high-definition fisheye cameras in the front, rear, left and right directions of the vehicle to acquire fisheye distortion correction images around the vehicle.

[0028] Then, this application generates a panoramic top view by projecting fisheye distortion correction images captured by cameras at different locations. In this process, this application needs to determine the calibration of the projection mapping relationship from the fisheye distortion correction images to the panoramic top view.

[0029] Specifically, this application uses the camera's intrinsic parameter K and extrinsic parameter T to calculate the mapping between the ground coordinate system and pixel coordinates, i.e., P(X, Y, Z) → p(u, v):

[0030] in, This refers to camera i. Indicates camera The pixel coordinates (u, v), Indicates camera Attitude in the ground coordinate system Represents a point (X, Y, Z) in the ground coordinate system. express In the camera Depth in the coordinate system.

[0031] Assumption Represents the coordinates in the top view The corresponding point in the ground world coordinate system is The relationship between the two is as follows:

[0032] in, The transformation matrix from the ground world coordinate system to the top view coordinate system is defined as:

[0033] in, , These are the physical dimensions of each pixel in the top view. and These represent the width and height in pixels for the top view, respectively.

[0034] Ultimately, the fisheye camera Points on the image and projection points on the top view The relationship can be represented as:

[0035] This application can generate a panoramic top view by projecting a fisheye distortion correction image using the above mapping relationship.

[0036] Furthermore, in the process of projecting and integrating fisheye distortion-corrected images captured by different cameras to generate a panoramic top-down view, the four fisheye cameras (front, rear, left, and right) of the vehicle may have different lighting, white balance, and exposure, resulting in inconsistent colors near the stitching seams after stitching. Therefore, this application can correct the outputs of each camera to a unified reference frame based on the pixel distribution in the overlapping area (such as mean, histogram, or gain / bias model).

[0037] Specifically, this application first extracts pixel information from the overlapping area of ​​adjacent cameras in the panoramic top view, typically calculating the color statistics (such as mean or histogram distribution) of both cameras in that area. Then, using one camera as a reference, a mapping function from the original pixel value to the target pixel value is established for the other camera. Next, using the corresponding pixels in the overlapping area, the optimal transformation parameters are solved by least squares or histogram matching to minimize the color difference between the two sides of the overlapping area. Finally, the transformation parameters are applied to the entire camera image, thereby eliminating the color difference between adjacent cameras in space and ensuring a smooth and natural color transition in the panoramic stitched image.

[0038] In addition, the above optimizations improve the color balance of the panoramic top view in the spatial dimension. This application can also optimize the luminosity consistency of the panoramic top view in the temporal dimension.

[0039] Please refer to details. Figure 3 , Figure 3 This is a flowchart illustrating the second embodiment of the transparent chassis filling method provided in this application.

[0040] like Figure 3 As shown, the specific steps are as follows: Step S21: Obtain the average pixel value of the historical frame vehicle panoramic image as a reference value, wherein the average pixel value is the average pixel point value or the average pixel region value.

[0041] In this embodiment, the application obtains panoramic top-down views (all of which are panoramic vehicle images) of N historical frames preceding the current frame, and calculates the average pixel value of the N panoramic top-down views. The average pixel value can be the average value of individual pixels or the average value of a pixel region.

[0042] The pixel mean is the average pixel value of pixels at the same position in N frames of the panoramic top-down view, while the pixel region mean is the average pixel value of all pixels in the same region in N frames of the panoramic top-down view. For example, if a region in each frame of the panoramic top-down view contains 50 pixels, then the pixel region mean is the average pixel value of 50 * N pixels.

[0043] In other embodiments, the pixel mean can also be the mean of the entire frame image, which will not be elaborated here.

[0044] Step S22: Calculate the ratio of the reference value to the pixel value of the current frame panoramic top view as the optimization value.

[0045] In this embodiment of the application, if the average pixel value in step S21 is the average pixel value R, the application determines the pixel value C of the pixel at the pixel position in the panoramic top view of the current frame, and calculates the ratio G=R / C of the reference value to the pixel value as the gain coefficient of the pixel.

[0046] If the pixel mean value in step S21 is the pixel region mean value R, this application determines the pixel mean value C of the region at the pixel region position in the panoramic top view of the current frame, and calculates the ratio G=R / C of the reference value to the pixel mean value as the gain coefficient of all pixels in the region.

[0047] Step S23: Optimize the current frame panoramic top view using the optimized value.

[0048] In this embodiment of the application, the gain coefficient of step S22 is used to optimize the panoramic top view of the current frame:

[0049] Step S12: Locate the matching grid region in the current frame panoramic top view, wherein the matching grid region does not overlap with the transparent chassis filling region.

[0050] In this embodiment, a matching grid region is selected within the current frame's panoramic top-down view. This matching grid region does not contain any pixels from the transparent chassis filling region. Please refer to [link / reference] for details. Figure 4 , Figure 4This is a schematic diagram of an embodiment of the division of the transparent chassis filling area and the image optimization area provided in this application.

[0051] like Figure 4 As shown, the undercarriage filling area, i.e., the transparent chassis filling area, is an area that the camera cannot capture. This application requires selecting a matching grid area outside the transparent chassis filling area. The matching grid area can be... Figure 4 The rectangular area shown can also be any other shape; no specific limitation is made here.

[0052] In other embodiments, the matching grid region may also exclude dynamic objects, blind spots under vehicles, or unreliable regions, where the selected points do not include pixels within these regions.

[0053] The matching grid area can be selected by the staff through input commands, or a fixed grid area can be preset as the matching grid area in advance; no specific limitation is made here.

[0054] Furthermore, to improve the accuracy of vehicle operating parameters, this application also provides a selection scheme for high-quality matching grid areas; please refer to [link / reference needed]. Figure 5 , Figure 5 This is a flowchart illustrating the third embodiment of the transparent chassis filling method provided in this application.

[0055] like Figure 5 As shown, the specific steps are as follows: Step S31: Divide the current frame panoramic top view into several grid regions.

[0056] In this embodiment, the current frame panoramic top-down view is divided into M grid regions, and each grid region is numbered. The number M can be determined according to the actual operating efficiency of the device.

[0057] Step S32: Calculate the gradient magnitude of each grid region and select the high gradient grid regions as the matching grid regions.

[0058] In this embodiment of the application, the gradient magnitude of each grid region determined in step S31 is obtained. Figure 4 The set of gradient magnitudes shown. Then, this application from... Figure 4 The grid regions shown are selected from high-gradient grid regions as matching grid regions. There can be one or more matching grid regions.

[0059] When there is only one matching grid region, this application can select the grid region with the highest gradient magnitude as the matching grid region. When there are multiple matching grid regions, this application can filter out grid regions with gradient magnitudes higher than the threshold as matching grid regions by setting a threshold; this application can also set the number of matching grid regions, sort the M grid regions according to their gradient magnitudes from high to low, and select that number of grid regions as matching grid regions.

[0060] Furthermore, in addition to calculating the gradient magnitude for the grid region, this application can also calculate the gradient magnitude for each pixel, which can be used to filter high-quality pixels and high-quality matching grid regions. Please refer to [link / reference] for details. Figure 6 , Figure 6 This is a flowchart illustrating the fourth embodiment of the transparent chassis filling method provided in this application.

[0061] like Figure 6 As shown, the specific steps are as follows: Step S41: Calculate the gradient magnitude of each pixel in the current frame panoramic top view, and select high gradient pixels as matching pixels.

[0062] In this application embodiment, the application can also calculate the gradient magnitude of each pixel in the current frame panoramic top view and select the pixel with the highest gradient magnitude as the matching pixel, or select several pixels as matching pixels by setting a quantity or setting a threshold.

[0063] Step S42: Determine the matching grid region centered on the matching pixel.

[0064] In this embodiment, the matching grid region is generated by taking the matching pixel as the center point and according to a preset region size.

[0065] Step S13: Determine the vehicle operation parameters from the historical frame time to the current frame time.

[0066] In this embodiment, the vehicle operation parameters from the historical frame time to the current frame time are the translation and rotation amounts experienced by the vehicle from the historical frame time to the current frame time, which are used to describe the change in the vehicle's operating state between the two times.

[0067] Unlike pre-installed vehicles that directly obtain the aforementioned vehicle operating parameters via CAN bus signals, this application's transparent chassis filling method proposes an implementation scheme for determining vehicle operating parameters between two moments in a post-installed vehicle scenario. Please refer to [link / reference] for details. Figure 7 , Figure 7 This is a flowchart illustrating the fifth embodiment of the transparent chassis filling method provided in this application.

[0068] like Figure 7 As shown, the specific steps are as follows: Step S51: Transform the image region corresponding to the matching grid region in the historical frame vehicle panoramic image to the current frame coordinate system using the initial running parameters.

[0069] In this embodiment, the initial operating parameters can be vehicle operating parameters of (0,0,0), vehicle operating parameters optimized from the previous frame panoramic top view, or vehicle operating parameters optimized from the previous image layer.

[0070] Step S52: Construct a nonlinear least squares problem, using the pixel error between the transformed region and the matching grid region in the current frame panoramic top view as the cost function.

[0071] This application constructs a nonlinear least squares problem for matching grid regions, using the photometric error of image blocks in corresponding grid regions of adjacent frames (i.e., the difference between the transformed pixel brightness and the target frame pixel brightness) as the residual. The cost function is as follows:

[0072] in, For residuals, Match the pixels of the grid region in the current frame. The pixels in the historical frame are mapped to the current frame coordinate system based on the vehicle's operating parameters.

[0073] Step S53: Use an iterative optimization algorithm to find the operating parameters that minimize the cost function and use them as the vehicle operating parameters.

[0074] In the embodiments of this application, the CeresSolver or Levenberg-Marquardt algorithm is used to solve the motion parameters in the process of solving the nonlinear least squares problem.

[0075] Furthermore, this application also provides a pyramid structure to improve the accuracy of solving vehicle operating parameters. For details of the pyramid multi-size strategy and nonlinear optimization solution strategy of this application, please refer to [link / reference needed]. Figure 8 , Figure 8 yes Figure 7 The diagram shows the detailed process flow of steps S52 and S53 in the transparent chassis filling method.

[0076] like Figure 8 As shown, the specific steps are as follows: Step S61: Construct a pyramid structure for the matching grid region, wherein the pyramid structure includes multiple resolution image layers.

[0077] In this embodiment, a pyramid structure is constructed based on the matching grid region determined in the above embodiments. The pyramid structure includes multiple image layers with varying resolutions, with the top layer having the lowest resolution and the bottom layer the highest. The bottom image layer is the original image data of the current frame's panoramic top-down view, while the other image layers are obtained by downsampling the original image data.

[0078] In this embodiment, the vehicle operating parameters are solved starting from the top image layer of the pyramid structure, and the optimization results (vehicle operating parameters) of the top image layer are scaled up proportionally as the initial values ​​of the next image layer, thus refining the vehicle operating parameters layer by layer.

[0079] Step S62: At the current layer resolution, transform the corresponding region in the historical frame vehicle panoramic image to the current frame coordinate system using the initial running parameters.

[0080] In this embodiment, the initial operating parameters can be vehicle operating parameters of (0,0,0), vehicle operating parameters optimized from the previous frame's panoramic top-down view, or vehicle operating parameters optimized from the previous image layer. Specifically, the vehicle operating parameters include: horizontal translation amount. Vertical translation amount and rotation angle .

[0081] Step S63: Construct a nonlinear least squares problem, using the pixel error between the transformed region and the matching grid region of the current frame as the cost function.

[0082] In this embodiment, for each image layer, a nonlinear least squares problem is constructed, using the photometric error of the image blocks in the corresponding grid regions of adjacent frames (i.e., the difference between the transformed pixel brightness and the target frame pixel brightness) as the residual. The cost function is as follows:

[0083] in, For residuals, Match the pixels of the grid region in the current frame. The pixels in the historical frame are mapped to the current frame coordinate system based on the vehicle's operating parameters.

[0084] Step S64: Use an iterative optimization algorithm to solve for the operating parameters that minimize the cost function.

[0085] In the embodiments of this application, the CeresSolver or Levenberg-Marquardt algorithm is used to solve the motion parameters in the process of solving the nonlinear least squares problem.

[0086] Step S65: Magnify the running parameters obtained from the current layer optimization and use them as the initial running parameters for the next layer with a higher resolution.

[0087] In this embodiment, if the current layer is not the bottom image layer of the pyramid structure, the vehicle operation parameters optimized by the current layer are amplified and used as the initial operation parameters when constructing the nonlinear least squares problem in the next higher resolution image layer.

[0088] Furthermore, the initial value selection algorithm uses the optimization result of the previous frame or the previous layer as the initial value of the current frame or the current layer. In order to avoid data jitter and getting trapped in local minima, a low-pass filter is introduced to smooth the optimization results of the previous few frames or the previous few layers, and the smoothed result is used as the initial value of the current frame or the current layer.

[0089] Specifically, taking the previous frame as an example, the initial value for motion information optimization is the optimization result of the previous frame used as the initial value for the current frame. For instance, when the vehicle starts and the first frame begins, the initial value is 0, and the optimized motion information is... In the second frame As the initial value for optimization, the optimization result is obtained. Using low-pass filtering for and Smooth the surface and multiply by the weights. (0.8) In the third frame As the initial value for optimization, the optimization result is obtained. Using low-pass filtering , and Smoothing process is performed to obtain Similarly, the number of filters in the low-pass filter can be set to N (N is 5), that is, smoothing using the historical optimization data of the most recent N frames and multiplying by the weights. .

[0090] Step S66: Output the final optimized operating parameters of the bottom layer of the pyramid structure as the vehicle operating parameters.

[0091] In this embodiment of the application, if the current layer is the bottom image layer of a pyramid structure, the vehicle operation parameters optimized by the current layer are output as the final vehicle operation parameters.

[0092] Furthermore, after determining the vehicle operating parameters corresponding to the panoramic top-down view of the current frame, this application can also perform robust smoothing processing on the vehicle operating parameters. Due to factors such as image noise, interference from dynamic objects, and sudden changes in illumination, the output vehicle operating parameters may contain outliers (i.e., gross errors). If operating parameters containing gross errors are directly used for subsequent transparent chassis image cropping and overlay, it will lead to visual defects such as stitching misalignment and image tearing. Therefore, this application adds a robust smoothing processing step after the motion parameter output to detect and replace outlier frames, thereby improving the robustness and stability of transparent chassis stitching. Please refer to [link / reference] for details. Figure 8 and Figure 9 , Figure 9 This is a schematic flowchart of an embodiment of the robust smoothing process in the transparent chassis filling method provided in this application. Figure 10 This is a schematic diagram of the overall process of the robust smoothing and continuous exception reset mechanism provided in this application.

[0093] like Figure 9 As shown, the specific steps are as follows: Step S71: Calculate the increment of vehicle operating parameters in the current frame panoramic top view relative to the operating parameters in the previous normal frame panoramic top view.

[0094] In this embodiment, the application calculates the increment of the vehicle operating parameters obtained by optimization in the current frame relative to the operating parameters in the previous normal frame. The increment includes horizontal translation increment, vertical translation increment, and rotation angle increment. The previous normal frame is the most recent frame that was not determined to be abnormal.

[0095] Step S72: Based on the incremental statistics window of the running parameters of the previous normal frame panoramic top view, calculate the median and absolute median difference of the increments of each dimension.

[0096] In this embodiment, a sliding window is maintained, which stores the runtime parameter increments of the most recent N normal frames, where N is a preset window size. For each dimension of the increment sequence in the sliding window, the median and median absolute deviation (MAD) are calculated respectively.

[0097] Specifically, for an increment sequence {x1, x2, ..., x} in a certain dimension... n The median (med) is defined as the median value after sorting the sequence; the absolute median difference (MAD) is defined as the median of the absolute values ​​of the differences between each increment and the median, i.e.: MAD=median(|x i -med|) Compared to the mean and standard deviation, the median and absolute median deviation are naturally robust to outliers and are not skewed by a small number of extreme values, making them suitable for motion parameter anomaly detection scenarios.

[0098] Step S73: Calculate the robust Z-score of the current frame increment based on the median and the absolute median difference, and determine the current frame panoramic top view as an abnormal frame when the robust Z-score exceeds a preset threshold.

[0099] In this embodiment of the application, the robust Z-score of the increment of each dimension of the current frame is calculated based on the median and the absolute median difference: Z = 0.6745 × |x - med| / (MAD + ε) Where x is the increment value of the current frame, med is the median of the increment in this dimension within the sliding window, MAD is the absolute median difference, ε is a small constant to prevent division by zero, and 0.6745 is the conversion coefficient between MAD and standard deviation under a normal distribution.

[0100] When the robust Z-score in any one of the dimensions—horizontal translation increment, vertical translation increment, or rotation angle increment—exceeds a preset threshold, the current frame is determined to be an abnormal frame. The typical range of the preset threshold is 3.0 to 4.0.

[0101] Furthermore, this application sets a physical hard threshold as a priori constraint: when the increment of the running parameters of the current frame exceeds the physically possible maximum value, it is directly determined to be an abnormal frame regardless of whether the statistical window is ready. The physical hard threshold is consistent with the parameter boundary constraints in the iterative optimization algorithm.

[0102] In addition, when the number of normal frame samples in the sliding window does not reach the preset minimum number of samples, this application temporarily does not enable anomaly detection based on statistics, and only relies on physical hard thresholds for judgment.

[0103] Step S74: For abnormal frames, use the constant speed model to predict the vehicle operating parameters of the current frame as a substitute output.

[0104] In this embodiment of the application, when the current frame is determined to be an abnormal frame, this application does not use the optimized vehicle operating parameters, but instead uses the constant speed model to predict the vehicle operating parameters of the current frame as a substitute output.

[0105] Specifically, the constant speed model includes: calculating a speed vector based on the vehicle operating parameters of the two most recent normal frames, wherein the speed vector is the difference between the operating parameters of the most recent normal frame and the operating parameters of the previous normal frame; and superimposing the speed vector onto the operating parameters of the most recent normal frame to obtain the predicted vehicle operating parameters for the current frame. When there is only one normal frame in the history, the model degenerates into keeping the operating parameters of the previous normal frame unchanged.

[0106] It should be noted that the abnormal frames in this application do not update the pose history of normal frames, nor do they update the incremental statistics window, in order to prevent outliers from polluting subsequent statistical judgments.

[0107] like Figure 10 As shown in this embodiment, when the number of consecutive abnormal frames exceeds a preset reset threshold, the incremental statistics window for the operating parameters is cleared so that subsequent frames can re-accumulate normal frame samples for statistics. The typical range of the reset threshold is 10 to 20 frames.

[0108] The purpose of the continuous anomaly reset mechanism is to: when a vehicle experiences a long period of scene abrupt change (such as passing through a tunnel or being exposed to strong light), the incremental distribution in the historical statistical window is no longer applicable to the current scene. At this time, the smoother is re-converged by resetting the window, thus avoiding continuous misjudgment caused by model drift.

[0109] For normal frames, this application directly outputs the optimized vehicle operating parameters and adds the incremental operating parameters of the current frame to the sliding window, while updating the pose history of normal frames.

[0110] This application employs robust smoothing to detect and replace anomalies in the vehicle operating parameters output by iterative optimization. It uses robust statistics based on the median and absolute median difference to calculate robust Z-scores, which, compared to traditional Z-score methods based on the mean and standard deviation, are naturally immune to outliers and are not swayed by a few extreme values. This allows for accurate identification of abnormal frames even in complex scenarios such as dynamic object interference and sudden changes in lighting. For abnormal frames, a constant velocity model is used for prediction and replacement, avoiding misalignment and tearing caused by abnormal motion parameters directly participating in transparent chassis image stitching. A continuous anomaly reset mechanism enables the smoother to automatically reconverge after long-term scene changes, ensuring the long-term operational stability of the system.

[0111] Step S14: Based on the vehicle operating parameters, transform the historical frame panoramic vehicle image to the coordinate system of the current frame panoramic top view to obtain the transformed panoramic vehicle image.

[0112] In this embodiment, the vehicle operating parameters determined in step S13 are used to perform motion compensation stitching repair on the transparent chassis filling area of ​​the current frame transparent chassis image. First, this application needs to obtain the historical frame transparent chassis image for stitching repair, transform the historical frame panoramic vehicle image to the coordinate system of the current frame according to the vehicle operating parameters, and then crop the historical frame transparent chassis image at the location of the transparent chassis filling area of ​​the current frame.

[0113] Please see Figure 11 , Figure 11This is a schematic diagram of the vehicle coordinate system provided in this application. The center of the rear axle of the vehicle is taken as the origin, the direction of the vehicle's front is the Y-axis, and the horizontal rightward direction is the X-axis.

[0114] like Figure 11 As shown, this application is based on the vehicle's overall parameters and the pixel size of the top view. , Determine the rotation center for top view transformation The vehicle motion model is defined as follows: based on the Ackermann steering principle, the motion parameter vector between two adjacent frames is defined. ,in , The translation amount, It represents the rotation angle.

[0115] The transformation relationship is as follows: for the previous frame image any pixel in It is mapped to the current frame image. The corresponding point in is:

[0116]

[0117] In the embodiments of this application, based on the estimated motion parameters and the center of rotation Construct a 2×3 affine transformation matrix M such that:

[0118] In this embodiment, the historical frame vehicle panoramic image is the color image of the previous frame that has been repaired. The image is obtained by performing an affine transformation on it using the affine transformation matrix M. Its content indicates that the repair result of the previous frame will be "flowed" into the coordinate system of the current frame according to the estimated motion.

[0119] Step S15: Crop the historical frame transparent chassis image of the transparent chassis filling area from the transformed vehicle panoramic image.

[0120] In the embodiments of this application, from The area that corresponds to the transparent chassis filling area in the current frame is cropped out and directly copied to the corresponding position in the color image of the current frame. This covers the defective areas in the blind spot of the vehicle's undercarriage in the original top view, and is blended with the background while filling to eliminate the seams around the edges. The blending technique can employ currently common progressive fade-in and non-linear blending schemes.

[0121] Step S16: Overlay the transparent chassis filling area of ​​the current frame panoramic top view with the historical frame transparent chassis image to generate a vehicle panoramic image.

[0122] In this embodiment of the application, the currently repaired color image can also be used as the next frame. The grayscale reference image is updated for the next round of motion estimation. The entire image stream is processed sequentially. The bottom area of ​​the vehicle in the top view is filled using historical data, resulting in a transparent chassis effect.

[0123] Please refer to the panoramic image of the vehicle after the transparent chassis splicing and filling is completed in this application. Figure 12 and Figure 13 , Figure 12 This is a schematic diagram of an embodiment of the panoramic vehicle image provided in this application. Figure 13 This is a schematic diagram of another embodiment of the panoramic vehicle image provided in this application. Figure 12 and Figure 13 As shown, the transparent chassis filling scheme of this application can obtain high-precision panoramic images of vehicles, and can achieve stable results for different scenarios (such as different traffic lines).

[0124] The transparent chassis filling method of this application utilizes the nonlinear optimization technology of photometric error of image blocks and pyramids to further improve the stitching effect of transparent chassis in panoramic view of vehicles. It can optimize the error at the pixel level, improve the flat stitching effect of transparent chassis, and avoid the stitching misalignment problem caused by vehicle tire sideslip and track integration.

[0125] The transparent chassis filling method of this application does not rely on sensors other than surround view cameras (such as wheel speedometers, inertial navigation, and high-precision positioning), and balances computational cost and stitching effect, making it widely applicable.

[0126] Those skilled in the art will understand that, in the above-described method of the specific implementation, the order in which each step is written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of each step should be determined by its function and possible internal logic.

[0127] To achieve the above-mentioned transparent chassis filling method, this application also proposes a transparent chassis filling device, please refer to the details below. Figure 14 , Figure 14 This is a schematic diagram of an embodiment of the transparent chassis filling device provided in this application.

[0128] The transparent chassis filling device 400 of this embodiment includes a processor 41, a memory 42, an input / output device 43, and a bus 44.

[0129] The processor 41, memory 42, and input / output device 43 are respectively connected to the bus 44. The memory 42 stores program data, and the processor 41 is used to execute the program data to implement the transparent chassis filling method described in the above embodiment.

[0130] In this embodiment, processor 41 can also be referred to as CPU (Central Processing Unit). Processor 41 may be an integrated circuit chip with signal processing capabilities. Processor 41 can also be a general-purpose processor, digital signal processor (DSP), application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), or other programmable logic device, discrete gate or transistor logic device, or discrete hardware component. The general-purpose processor can be a microprocessor, or processor 41 can be any conventional processor.

[0131] This application also provides a computer storage medium; please refer to the following: Figure 15 , Figure 15 This is a schematic diagram of a computer storage medium according to an embodiment of the present application. The computer storage medium 600 stores a computer program 61, which, when executed by a processor, is used to implement the transparent chassis filling method of the above embodiment.

[0132] When the embodiments of this application are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0133] The above description is merely an embodiment of this application and does not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A method for filling a transparent chassis, characterized in that, The transparent chassis filling method includes: Obtain the panoramic top-down view of the current frame, as well as panoramic images of the vehicle in historical frames; In the current frame panoramic top view, a matching grid region is located, wherein the matching grid region does not coincide with the transparent chassis filling region; Determine the vehicle operating parameters from the historical frame time to the current frame time; Based on the vehicle operating parameters, the historical frame panoramic vehicle image is transformed to the coordinate system of the current frame panoramic top view to obtain the transformed panoramic vehicle image. The historical frame transparent chassis image of the transparent chassis filling area is cropped from the transformed vehicle panoramic image; The transparent chassis image from the historical frame is overlaid on the transparent chassis-filled area of ​​the current frame panoramic top view to generate a panoramic vehicle image.

2. The transparent chassis filling method according to claim 1, characterized in that, The determination of vehicle operating parameters from historical frame time to current frame time includes: Transform the image region corresponding to the matching grid region in the historical frame vehicle panoramic image to the current frame coordinate system using the initial running parameters; A nonlinear least squares problem is constructed, with the pixel error between the transformed region and the matching grid region in the current frame panoramic top view as the cost function; The operating parameters that minimize the cost function are obtained by using an iterative optimization algorithm and are then used as the vehicle operating parameters.

3. The transparent chassis filling method according to claim 1, characterized in that, The transparent chassis filling method further includes: Construct a pyramid structure for the matching grid region, wherein the pyramid structure includes multiple image layers of different resolutions, with the top layer having the lowest resolution and the bottom layer having the highest resolution; Starting from the top layer of the pyramid structure, perform the following operations: At the current layer resolution, the corresponding region in the historical frame vehicle panoramic image is transformed to the current frame coordinate system using the initial running parameters; A nonlinear least squares problem is constructed, with the pixel error between the transformed region and the matching grid region of the current frame as the cost function; An iterative optimization algorithm is used to solve for the operating parameters that minimize the cost function; The running parameters obtained from the current layer optimization are amplified and used as the initial running parameters for the next layer with a higher resolution. The final optimized operating parameters obtained from the bottom layer of the pyramid structure are output as the vehicle operating parameters.

4. The transparent chassis filling method according to claim 1, characterized in that, The initial operating parameters are the vehicle operating parameters obtained by optimizing the historical frame vehicle panoramic image.

5. The transparent chassis filling method according to any one of claims 1-4, characterized in that, The vehicle operating parameters include: horizontal translation, vertical translation, and rotation angle.

6. The transparent chassis filling method according to claim 1, characterized in that, The step of locating and matching the grid region in the current frame's panoramic top-down view includes: The current frame panoramic top view is divided into several grid regions; Calculate the gradient magnitude of each grid region and select the high gradient grid regions as the matching grid regions.

7. The transparent chassis filling method according to claim 1, characterized in that, The step of locating and matching the grid region in the current frame's panoramic top-down view includes: Calculate the gradient magnitude of each pixel in the current frame panoramic top view, and select high gradient pixels as matching pixels; The matching grid region is determined with the matching pixel as the center.

8. The transparent chassis filling method according to claim 1, characterized in that, After obtaining the current frame panoramic top view and the historical frame vehicle panoramic images, the transparent chassis filling method further includes: The average pixel value of the historical frame vehicle panoramic image is obtained as a reference value, wherein the average pixel value is the average value of a pixel point or the average value of a pixel region; The ratio of the reference value to the pixel value of the current frame panoramic top view is calculated as the optimization value; The optimized value is used in conjunction with the current frame panoramic top view for optimization.

9. The transparent chassis filling method according to claim 1, characterized in that, After determining the vehicle operating parameters from the historical frame time to the current frame time, the transparent chassis filling method further includes: Robust smoothing processing is performed on the vehicle operating parameters, wherein the robust smoothing processing includes: Calculate the increment of vehicle operating parameters in the current frame panoramic top view relative to the operating parameters in the previous normal frame panoramic top view; Based on the incremental statistics window of the running parameters of the previous normal frame panoramic top view, calculate the median and absolute median difference of the increments in each dimension. The robust Z-score of the current frame increment is calculated based on the median and the absolute median difference, and the current frame panoramic top view is determined to be an abnormal frame when the robust Z-score exceeds a preset threshold. For abnormal frames, the vehicle operating parameters of the current frame are predicted using a constant speed model as a substitute output.

10. The transparent chassis filling method according to claim 9, characterized in that, The method of predicting vehicle operating parameters for the current frame using a constant speed model includes: Calculate the velocity vector based on the vehicle operating parameters of the two most recent normal frame panoramic top view; The velocity vector is superimposed onto the vehicle operating parameters of the most recent normal frame panoramic top view to obtain the vehicle operating parameters predicted by the current frame panoramic top view.

11. A transparent chassis filling device, characterized in that, The transparent chassis filling device includes a memory and a processor coupled to the memory; The memory is used to store program data, and the processor is used to execute the program data to implement the transparent chassis filling method as described in any one of claims 1 to 10.

12. A computer storage medium, characterized in that, The computer storage medium is used to store program data, which, when executed by the computer, is used to implement the transparent chassis filling method as described in any one of claims 1 to 10.

Images