Transparent base plate image generation method and device, vehicle, medium and product

Abstract

The present application relates to the technical field of intelligent driving, and discloses a transparent chassis image generation method and device, a vehicle, a medium and a product. The present application acquires a bird's eye view sequence, a historical chassis image and historical state parameters corresponding to each frame of the bird's eye view, and ensures synchronization of timestamps. Frame-to-frame motion parameters of adjacent two frames of the bird's eye view sequence are determined, the frame-to-frame motion parameters and historical state parameters collected by a sensor are fused to generate a vehicle pose trajectory, precise prediction of the vehicle pose is realized, and delay and inaccurate parameters caused by direct use of current sensor data are avoided. Delay time of the historical chassis image and the current frame of the bird's eye view is determined, the historical chassis image is motion compensated in combination with the vehicle pose trajectory, vehicle motion deviation is offset, and a compensated chassis image is generated. Finally, the compensated chassis image and the current frame of the bird's eye view are fused, natural transition of the transparent chassis image is realized, image misplacement is avoided, and display effect of the transparent chassis and vehicle driving safety are significantly improved.

Description

Technical Field

[0001] This invention relates to the field of intelligent driving technology, specifically to a method, apparatus, vehicle, medium, and product for generating transparent chassis images. Background Technology

[0002] With the development of intelligent driving technology, panoramic transparent chassis has become a standard feature of mid-to-high-end models. It uses multiple surround-view cameras to collect images around the vehicle, generates a bird's-eye view (BEV) through inverse projection mapping (IPM), and fills the current vehicle chassis position with historical road surface images under the chassis, making the chassis transparent and helping the driver to observe ground obstacles, assist parking, and pass through complex road surfaces.

[0003] Existing technologies acquire real-time motion parameters such as vehicle speed, steering angle, and gear position via CAN bus or CAN FD bus, calculate the vehicle's displacement and rotation angle within the time difference between the current frame and historical frames, then geometrically transform the image of the corresponding chassis area in the bird's-eye view of the historical frame according to the calculated displacement and rotation angle, mount it onto the chassis position in the current frame's bird's-eye view, and output a transparent chassis image to the vehicle's central control screen.

[0004] However, when the vehicle is traveling at high speed, due to the inherent delay in the real-time motion parameters collected by the CAN bus or CAN FD bus, the calculated displacement and rotation angle will have errors compared with the actual driving data. The transparent chassis image will lag behind the actual ground where the vehicle is currently located, resulting in image ghosting or discontinuity. It will also cause deviations in the display of obstacle positions, affecting the accuracy of the driver's judgment and driving operation. Summary of the Invention

[0005] This invention provides a method, apparatus, vehicle, medium, and product for generating transparent chassis images, in order to solve the problem of misalignment between the transparent chassis images displayed in the prior art and the real ground.

[0006] In a first aspect, the present invention provides a method for generating a transparent chassis image, the method comprising: The system acquires bird's-eye view sequences, sensor data sequences, and historical chassis images generated during vehicle operation; the sensor data sequences include historical state parameters corresponding to each frame of the bird's-eye view. Based on the bird's-eye view sequence, the inter-frame motion parameters of two adjacent bird's-eye view frames are obtained. Based on the inter-frame motion parameters and historical state parameters, the vehicle pose at each time moment is predicted, and the pose trajectory is obtained. Determine the time delay between the historical acquisition time and the current rendering time of the historical chassis image, and determine the target motion parameters of the vehicle within the time delay based on the pose trajectory; Motion compensation is performed on historical chassis images based on target motion parameters to obtain compensated chassis images. The compensated chassis images are then fused with the current frame bird's-eye view corresponding to the current rendering time to obtain transparent chassis images.

[0007] This invention ensures the synchronization of timestamps across multiple data sources by acquiring a sequence of bird's-eye view images, historical chassis images, and historical state parameters corresponding to each frame of the bird's-eye view. Secondly, it determines the inter-frame motion parameters between adjacent frames in the bird's-eye view sequence, and fuses these parameters with historical state parameters collected by sensors to generate a vehicle pose trajectory. This achieves accurate and smooth prediction of the vehicle pose, avoiding the delay and inaccurate parameters caused by directly using current sensor data. Next, it determines the time delay between the historical chassis image and the current frame of the bird's-eye view, and combines this with the vehicle pose trajectory to perform motion compensation on the historical chassis image, effectively offsetting vehicle motion offset and generating a compensated chassis image. Finally, it fuses the compensated chassis image with the current frame of the bird's-eye view to achieve a natural transition of the transparent chassis image, avoiding image misalignment and fragmentation, and significantly improving the display effect of the transparent chassis and vehicle driving safety.

[0008] In some optional implementations, the vehicle pose at each time step is predicted based on inter-frame motion parameters and historical state parameters to obtain the pose trajectory, including: For each time step, based on the historical state parameters of the previous time step, the inter-frame motion parameters between the previous frame bird's-eye view and the current frame bird's-eye view, the current state vector corresponding to the current time step is predicted. Extract the vehicle pose from the current state vector at each time step to generate the vehicle's pose trajectory.

[0009] This invention uses historical state parameters and vision-based inter-frame motion parameters to collaboratively predict vehicle state vectors, reducing the interference of sensor delay on motion estimation. It extracts vehicle pose based on the state vectors at each time point and generates a coherent pose trajectory, thereby accurately representing the vehicle's motion state and providing a data foundation for subsequent chassis image compensation.

[0010] In some optional implementations, based on the historical state parameters of the previous time step and the inter-frame motion parameters between the previous frame's bird's-eye view and the current frame's bird's-eye view, the current state vector corresponding to the current time step is predicted, including: Based on the historical state parameters of the previous moment, the historical state vector is obtained, and the prior state vector of the vehicle at the current moment is determined. Visual observations are constructed based on the inter-frame motion parameters between the previous and current bird's-eye view images. Based on visual observations, historical state vectors, and prior state vectors, the observation noise and observation equations are obtained, and the prior covariance matrix is ​​determined based on the observation noise and observation equations. The Kalman gain is calculated based on the prior covariance matrix, the observation equation, and the observation noise. The prior state vector is then updated based on the Kalman gain and the observation noise to obtain the current state vector at the current time.

[0011] This invention accurately predicts the vehicle's current state vector by recursively calculating the prior state vector based on historical state parameters and combining it with visual observations obtained from a bird's-eye view to correct the vehicle's state. By utilizing state transitions to predict the current prior state in advance, compensating for delay biases, and then correcting prediction errors with visual observations, it achieves accurate prediction of vehicle pose. This effectively avoids motion lag caused by data transmission and output delays from onboard sensors, prevents chassis image misalignment due to sensor delays, and improves the alignment accuracy and display stability of transparent chassis images.

[0012] In some optional implementations, the historical state parameters include a first state parameter and a second state parameter characterizing the changing trend of the first state parameter; based on the historical state parameters from the previous moment, a historical state vector is obtained, and the prior state vector of the vehicle at the current moment is determined, including: Based on the vehicle's first state parameters at the previous moment, a historical state vector is constructed; where the first state parameters include the vehicle's position, speed, heading angle, and heading angular velocity. The state transition is performed based on the second state parameters and historical state vectors from the previous moment to obtain the vehicle's prior state vector at the current moment; the second state parameters include acceleration and heading angular acceleration.

[0013] This invention constructs a historical state vector containing vehicle position, speed, heading angle, and heading angular velocity, and combines acceleration and heading angular acceleration for state transition recursion to accurately predict the prior state vector at the current moment. This avoids image misalignment caused by using sensor signals with long delays to calculate the vehicle's current motion parameters, thereby improving the overall accuracy and stability of subsequent vehicle pose trajectory calculations.

[0014] In some optional implementations, the compensated chassis image and the current frame bird's-eye view corresponding to the current rendering time are fused to obtain a transparent chassis image, including: Determine the first region and the second region in the current frame bird's-eye view; wherein, the first region is the overlapping area between the compensation chassis image and the current frame bird's-eye view, and the second region is the area other than the first region; The fusion weight is determined based on the region type of the overlapping region. The corresponding regions of the first region and the compensation chassis image are fused according to the fusion weight to obtain the fused compensation region. The fusion compensation area and the second area are stitched together to obtain a transparent chassis image.

[0015] This invention achieves segmented image processing by dividing the image into an overlapping first region and a non-overlapping second region. It adaptively configures the fusion weights and adjusts the fusion ratio based on the type of overlapping region, weakening the stitching marks at image junctions and making the transition between overlapping areas smoother and more natural. The fused compensation region is then completely stitched together with the non-overlapping region to obtain a transparent chassis image, avoiding misalignment and improving overall image quality and display effect.

[0016] In some alternative implementations, based on the pose trajectory, the target motion parameters of the vehicle during the delay time are determined, including: Based on the pose trajectory, the first pose corresponding to the historical acquisition time and the second pose corresponding to the current rendering time are obtained; Based on the first and second poses, the target displacement and target rotation angle of the vehicle during the delay time are obtained.

[0017] This invention accurately extracts the vehicle pose corresponding to the historical acquisition time and the current rendering time based on the vehicle pose trajectory, and then calculates the target displacement and target rotation angle of the vehicle throughout the entire delay time to restore the true motion state of the vehicle during the delay time. This facilitates motion compensation of historical chassis images and reduces the risk of image offset caused by time delay.

[0018] In some optional implementations, motion compensation is performed on historical chassis images based on target motion parameters to obtain compensated chassis images, including: Based on the target displacement and target rotation angle, the coordinates of the first pixel in the historical chassis image are inversely transformed to obtain the coordinates of the second pixel. The compensated chassis image is obtained based on the second pixel coordinates.

[0019] This invention utilizes the target displacement and target rotation angle within a delay time to perform a geometric inverse transformation on the pixel coordinates of historical chassis images, offsetting the positional shift and angular deflection of the vehicle during the delay time pixel by pixel, so that the pixel position of the historical chassis image matches the current vehicle chassis area, eliminating the image misalignment caused by the time difference between image acquisition and rendering, and optimizing the visual presentation effect of the transparent chassis.

[0020] In some optional implementations, based on the bird's-eye view sequence, the inter-frame motion parameters of two adjacent bird's-eye view frames are obtained, including: Based on the bird's-eye view sequence, determine the inter-frame homography matrix of two adjacent bird's-eye view frames; By decomposing the inter-frame homography matrix, the inter-frame displacement and inter-frame rotation angles of two adjacent bird's-eye view frames are obtained.

[0021] This invention determines the inter-frame homography matrix of adjacent bird's-eye view images and decomposes it to obtain the inter-frame displacement and rotation angle, accurately characterizing the motion relationship between adjacent bird's-eye view frames. This provides a data foundation for subsequent vehicle pose trajectory generation and motion compensation, ensuring the alignment accuracy and display stability of the transparent chassis image.

[0022] In a second aspect, the present invention provides a transparent chassis image generation apparatus, the apparatus comprising: The first processing module is used to acquire the bird's-eye view sequence, sensor data sequence, and historical chassis images generated during vehicle operation; wherein, the sensor data sequence includes historical state parameters corresponding to each frame of the bird's-eye view. The second processing module is used to obtain the inter-frame motion parameters of two adjacent bird's-eye view frames based on the bird's-eye view sequence, and predict the vehicle pose at each time based on the inter-frame motion parameters and historical state parameters to obtain the pose trajectory. The third processing module is used to determine the time delay between the historical acquisition time and the current rendering time of the historical chassis image, and to determine the target motion parameters of the vehicle within the time delay based on the pose trajectory. The fourth processing module is used to perform motion compensation on the historical chassis image based on the target motion parameters to obtain the compensated chassis image, and to fuse the compensated chassis image with the current frame bird's-eye view corresponding to the current rendering time to obtain the transparent chassis image.

[0023] Thirdly, the present invention provides a vehicle comprising: a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the computer instructions to perform the transparent chassis image generation method of the first aspect or any corresponding embodiment described above.

[0024] Fourthly, the present invention provides a computer-readable storage medium storing computer instructions for causing a computer to perform the transparent chassis image generation method of the first aspect or any corresponding embodiment described above.

[0025] Fifthly, the present invention provides a computer program product, including computer instructions for causing a computer to execute the transparent chassis image generation method of the first aspect or any corresponding embodiment described above.

[0026] The beneficial effects of this invention are as follows: This invention ensures the synchronization of timestamps across multiple data sources by acquiring a sequence of bird's-eye view images, historical chassis images, and historical state parameters corresponding to each frame of the bird's-eye view. Secondly, it determines the inter-frame motion parameters between adjacent frames in the bird's-eye view sequence, and fuses these parameters with historical state parameters collected by sensors to generate a vehicle pose trajectory. This achieves accurate and smooth prediction of the vehicle pose, avoiding the delay and inaccurate parameters caused by directly using current sensor data. Next, it determines the time delay between the historical chassis image and the current frame of the bird's-eye view, and combines this with the vehicle pose trajectory to perform motion compensation on the historical chassis image, effectively offsetting vehicle motion offset and generating a compensated chassis image. Finally, it fuses the compensated chassis image with the current frame of the bird's-eye view to achieve a natural transition of the transparent chassis image, avoiding image misalignment and fragmentation, and significantly improving the display effect of the transparent chassis and vehicle driving safety. Attached Figure Description

[0027] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0028] Figure 1 This is a schematic diagram of the first process of a transparent chassis image generation method according to an embodiment of the present invention; Figure 2 This is a schematic diagram of a second process for generating a transparent chassis image according to an embodiment of the present invention; Figure 3 This is a structural block diagram of a transparent chassis image generation device according to an embodiment of the present invention; Figure 4 This is a schematic diagram of the hardware structure of a vehicle according to an embodiment of the present invention. Detailed Implementation

[0029] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0030] It is understood that before using the technical solutions disclosed in the various embodiments of the present invention, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in the present invention and their authorization should be obtained in accordance with relevant laws and regulations through appropriate means.

[0031] The terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0032] A panoramic transparent chassis (also known as a 540° panoramic imaging system) helps drivers observe ground obstacles (such as rocks, potholes, and steps), assists in parking, and helps navigate complex road surfaces. The related transparent chassis image filling technology mainly consists of the following three steps: 1) Image acquisition and preprocessing: Real-time images of the surrounding environment are acquired using four fisheye cameras installed at the front, rear, left, and right of the vehicle. After preprocessing such as distortion correction, white balance, and exposure synchronization, a standardized image is output to generate a bird's-eye view; 2) Motion parameter acquisition and displacement calculation: Motion parameters such as vehicle speed, steering angle, and gear position are obtained from the vehicle's electronic control unit (ECU) via the CAN bus or CAN FD bus. The displacement and rotation angle of the vehicle within the time difference are calculated by combining the time difference between the current frame and historical frames; 3) Image stitching and filling: The image of the corresponding chassis area in the historical frame's bird's-eye view is geometrically transformed according to the calculated displacement and rotation angle, and then pasted onto the chassis position in the current frame's bird's-eye view. Finally, the transparent chassis image is output to the vehicle's central control screen.

[0033] Although related technologies have achieved the basic functions of a transparent chassis, in actual use, especially when the vehicle is moving or turning, there is a common problem of misalignment between the chassis image and the real ground. Specifically, this manifests as: longitudinal misalignment, where at high speeds, the chassis image lags behind the real ground, resulting in ghosting or gaps, and errors in the display of obstacle positions; and lateral misalignment, where, during turns, the chassis image does not blend naturally with the images of the vehicles on both sides, resulting in a step-like misalignment. Furthermore, the misalignment is more pronounced when the vehicle accelerates or decelerates, crawls at low speeds, or traverses bumpy roads, severely affecting the driver's judgment.

[0034] Analysis revealed that the core reason for the misalignment between the chassis image and the actual ground lies in the inherent delay in the motion parameter signals such as vehicle speed and steering angle transmitted via the CAN bus or CAN FD bus. This inherent delay consists of three main parts: signal acquisition delay (the delay in the ECU acquiring signals from the wheel speed and steering angle sensors); bus transmission delay (the delay in signal transmission and queuing within the CAN bus); and parsing and processing delay (the delay in protocol parsing and unit conversion after the panoramic image controller receives the signal). The total delay, when these three factors are combined, is typically between 50 and 120 ms, and can exceed 150 ms in some extreme scenarios.

[0035] Furthermore, the relevant technology calculates displacement solely based on vehicle speed and time difference, neglecting acceleration and lateral displacement during steering. Since the vehicle speed signal is a discrete value, it cannot reflect the continuous motion of the vehicle, leading to discrepancies between the calculated displacement and the actual motion, resulting in low accuracy. In addition, the asynchronous timing of camera image acquisition, CAN signal generation, and the processing timestamp of the panoramic controller further exacerbates the inaccurate time difference calculation, amplifying the misalignment between the chassis image and the actual ground.

[0036] To address the aforementioned issues, related technologies have optimized CAN bus transmission by increasing the priority of CAN signals and shortening the transmission cycle (e.g., from 50 ms to 20 ms). However, this only reduces the latency by 10-20 ms and cannot completely eliminate the problem of long latency. Furthermore, pure visual technology using multi-frame image stitching to generate chassis images eliminates the need for CAN signals. However, this technology suffers from stitching failures on weakly textured surfaces (such as concrete or asphalt roads) and problems caused by dynamic objects (such as pedestrians or vehicles) resulting in chaotic chassis images, indicating insufficient robustness.

[0037] To address the aforementioned issues, this invention provides a method for generating transparent chassis images. By acquiring a bird's-eye view sequence, historical chassis images, and historical state parameters corresponding to each frame of the bird's-eye view, the synchronization of timestamps from multiple data sources is ensured. Secondly, the inter-frame motion parameters of two adjacent frames in the bird's-eye view sequence are determined. These inter-frame motion parameters are then fused with historical state parameters collected by sensors to generate a vehicle pose trajectory, achieving accurate and smooth prediction of the vehicle pose and avoiding delays and inaccurate parameter values ​​caused by directly using current sensor data. Next, the delay time between the historical chassis image and the current frame of the bird's-eye view is determined. Motion compensation is then applied to the historical chassis image based on the vehicle pose trajectory, effectively offsetting vehicle motion offsets and generating a compensated chassis image. Finally, the compensated chassis image is fused with the current frame of the bird's-eye view to achieve a natural transition in the transparent chassis image, avoiding image misalignment and fragmentation, and significantly improving the display effect of the transparent chassis and vehicle driving safety.

[0038] According to an embodiment of the present invention, a method for generating a transparent chassis image is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0039] This embodiment provides a method for generating transparent chassis images, which can be used in vehicle displays, such as vehicle central control screens and instrument display systems. Figure 1 This is a flowchart of a transparent chassis image generation method according to an embodiment of the present invention, such as... Figure 1 As shown, the process includes the following steps: Step S101: Obtain the bird's-eye view sequence, sensor data sequence, and historical chassis images generated during vehicle operation; wherein, the sensor data sequence includes the historical state parameters corresponding to each frame of the bird's-eye view.

[0040] In this embodiment, hardware triggering, such as using a General Purpose Input / Output (GPIO) signal, is employed to synchronously control the simultaneous exposure of four fisheye cameras. The exposure time synchronization error is ≤1 ms, avoiding motion blur and inter-frame misalignment caused by asynchronous exposure. Furthermore, the camera frame rate is ≥30 fps to ensure high-frequency motion measurement. Each frame of fisheye image output by the camera is timestamped using hardware for subsequent time synchronization. Specifically, the four fisheye cameras have a horizontal field of view ≥180°, a resolution ≥1080P, a frame rate ≥30fps, and support hardware-triggered synchronization. The camera interface uses a MIPI CSI-2 interface (transmission rate ≥2Gbps) and supports exposure synchronization signal input.

[0041] Furthermore, based on camera intrinsic parameters (such as focal length and center point coordinates) and distortion parameters (such as radial distortion and tangential distortion), a Brown-Conrady model is used to correct the distortion of the fisheye image, resulting in a corrected fisheye image. Then, based on the vehicle coordinate system (with the vehicle's centroid as the origin, X-axis forward, Y-axis left, and Z-axis upward), a projection matrix between the ground plane and the image plane is established. The corrected fisheye image is projected as a bird's-eye view. The projection matrix... The process involves calibration, acquiring images of the checkerboard pattern from different angles by laying it out on a ground screen, determining the correspondence between the checkerboard corner points in the vehicle coordinate system and the image coordinate system, and then fitting the projection matrix using the least squares method. For detailed information, please refer to the descriptions of the relevant technologies; they will not be repeated here.

[0042] Specifically, during vehicle operation, a bird's-eye view is generated at each moment according to the steps described above, resulting in a sequence of bird's-eye view images (each frame corresponding to a moment). Simultaneously, while generating the bird's-eye view for each moment, sensor data output from onboard sensors is read. This sensor data includes CAN bus data, data transmitted from the Inertial Measurement Unit (IMU), etc., generating a sensor data sequence. An MCU chip can be used to receive CAN and CAN FD signals (vehicle speed, steering angle) and IMU data. Furthermore, each frame of sensor data corresponds to historical state parameters at the same timestamp in the bird's-eye view sequence. These historical state parameters include vehicle position, vehicle speed, heading angle, heading angular velocity, acceleration, and heading angular acceleration. From the cached historical bird's-eye view images, images of the vehicle chassis area at each moment are extracted to obtain historical chassis images. This completes the synchronous acquisition and storage of the bird's-eye view sequence, sensor data sequence, and historical chassis images, ensuring timestamp synchronization.

[0043] Step S102: Based on the bird's-eye view sequence, obtain the inter-frame motion parameters of two adjacent bird's-eye view frames, and based on the inter-frame motion parameters and historical state parameters, predict the vehicle pose at each time point to obtain the pose trajectory.

[0044] Specifically, ground features are extracted and tracked for any two adjacent frames in the bird's-eye view sequence. After removing dynamic interference features and outliers, inter-frame motion parameters such as inter-frame displacement and inter-frame rotation angle are obtained. For each moment, based on the historical state parameters of the previous moment in the sensor data sequence, state recursion and optimization are performed in combination with the inter-frame motion parameters to obtain the vehicle pose at that moment. The vehicle pose information at each moment is then predicted frame by frame according to the above process to generate a continuous and smooth vehicle pose trajectory. By combining the inter-frame motion parameters obtained from visual analysis and the collected historical state parameters, the vehicle pose at each moment can be accurately predicted, avoiding the direct use of sensor data with a long delay collected at the current moment to calculate motion parameters such as vehicle displacement and rotation angle, thus avoiding image display misalignment caused by long delays.

[0045] It should be noted that an onboard SOC chip with a computing power of ≥10 TOPS can be used for feature extraction, optical flow tracking, and inter-frame motion parameter calculation, effectively reducing the processing latency of a single frame.

[0046] Step S103: Determine the time delay between the historical acquisition time and the current rendering time of the historical chassis image, and determine the target motion parameters of the vehicle within the time delay based on the pose trajectory.

[0047] Specifically, the timestamps corresponding to the historical chassis images are extracted to obtain their historical acquisition times, and the current rendering time corresponding to the current frame's bird's-eye view is determined. The time difference between the historical acquisition time and the current rendering time is calculated to determine the delay time. From the vehicle's pose trajectory, trajectory interpolation and parameter extraction are performed based on the delay time to determine the target motion parameters of the vehicle from the historical acquisition time to the current rendering time, such as the target displacement (i.e., cumulative displacement) and the target rotation angle (i.e., cumulative rotation angle), accurately representing the vehicle's motion changes within the delay time.

[0048] Step S104: Motion compensation is performed on the historical chassis image based on the target motion parameters to obtain a compensated chassis image. The compensated chassis image and the current frame bird's-eye view corresponding to the current rendering time are fused to obtain a transparent chassis image.

[0049] Specifically, based on the target motion parameters within the delay time, an inverse geometric transformation compensation is performed on the historical chassis image to offset the motion offset of the vehicle during the delay period, generating a compensated chassis image aligned with the current chassis position. Then, pixel fusion and stitching are performed on the corresponding areas of the compensated chassis image and the current frame bird's-eye view to finally generate a transparent chassis image with no misalignment and a natural transition, which is then displayed on the vehicle's central control screen.

[0050] For example, the fused transparent chassis image is received by the display driver chip and output to the central control screen for display. The resolution of the vehicle's central control screen is ≥1920×1080 and the refresh rate can be 60Hz.

[0051] The transparent chassis image generation method provided in this embodiment ensures the synchronization of timestamps from multiple data sources by acquiring a bird's-eye view sequence, historical chassis images, and historical state parameters corresponding to each frame of the bird's-eye view. Secondly, it determines the inter-frame motion parameters between two adjacent frames of the bird's-eye view sequence, and fuses these parameters with historical state parameters collected by sensors to generate a vehicle pose trajectory. This achieves accurate and smooth prediction of the vehicle pose, avoiding the delay and inaccurate parameter information caused by directly using current sensor data. Next, it determines the delay time between the historical chassis image and the current frame of the bird's-eye view, and performs motion compensation on the historical chassis image based on the vehicle pose trajectory, effectively offsetting vehicle motion offset and generating a compensated chassis image. Finally, it fuses the compensated chassis image with the current frame of the bird's-eye view to achieve a natural transition of the transparent chassis image, avoiding image misalignment and fragmentation, and significantly improving the display effect of the transparent chassis and vehicle driving safety.

[0052] This embodiment provides a method for generating transparent chassis images, which can be used in vehicle displays, such as vehicle central control screens and instrument display systems. Figure 2 This is a flowchart of a transparent chassis image generation method according to an embodiment of the present invention, such as... Figure 2 As shown, the process includes the following steps: Step S201: Acquire the bird's-eye view sequence, sensor data sequence, and historical chassis images generated during vehicle operation; wherein, the sensor data sequence includes historical state parameters corresponding to each frame of the bird's-eye view. See details for further information. Figure 1 Step S101 of the illustrated embodiment will not be described again here.

[0053] Step S202: Based on the bird's-eye view sequence, obtain the inter-frame motion parameters of two adjacent bird's-eye view frames, and based on the inter-frame motion parameters and historical state parameters, predict the vehicle pose at each time point to obtain the pose trajectory.

[0054] Specifically, step S202 includes: Step S2021: Based on the bird's-eye view sequence, determine the inter-frame homography matrix of two adjacent bird's-eye view frames.

[0055] Specifically, a corner detection algorithm (such as the Shi-Tomasi algorithm, balancing real-time performance and stability) is used to extract ground corner features from each frame of the bird's-eye view. N (e.g., 200-500) feature points can be extracted per frame, and the extraction area is limited to the ground area surrounding the vehicle (excluding non-ground areas such as the vehicle body and sky). Furthermore, corner detection thresholds are set, including a minimum feature value threshold q and a minimum distance threshold. For example, the minimum feature value threshold q can be 0.04, and the minimum distance threshold is 10 pixels (to avoid overly dense feature points). Among these, the feature points... The corresponding minimum eigenvalue The calculation method is as follows:

[0056]

[0057] in, These are the two eigenvalues ​​of the autocorrelation matrix of the corner detection algorithm. Indicates the current feature point The corresponding eigenvalues, This represents the largest eigenvalue of the entire image.

[0058] In this embodiment, for any two adjacent bird's-eye view frames, an optical flow method, such as the Lucas-Kanade (LK) algorithm, is used to track feature points between them. The optical flow window size can be set to 15×15 pixels, the number of iterations is 10, and the error threshold is 0.5 pixels. When tracking feature points, the brightness of the previous bird's-eye view frame is set to... (t-1 is the time point of the previous frame), the current frame is the brightness. (t is the time point of the current frame), the feature point set of the previous frame is N is the number of feature points. The feature point set corresponding to the current frame is solved using the optical flow method. In the bird's-eye view, the feature points satisfy the constant brightness constraint:

[0059] in, Feature points of the previous bird's-eye view The corresponding brightness, (u1, v1) is the optical flow vector between two adjacent bird's-eye view frames, Feature points in the current frame and the bird's-eye view from the previous frame Corresponding feature points The brightness.

[0060] In this embodiment, the consistency between the optical flow vector of the feature point and the vehicle's direction of motion is calculated. If the angle between the optical flow vector and the vehicle's direction of motion is greater than a preset angle (e.g., 90°) and the amplitude of the optical flow vector is greater than a preset amplitude (e.g., 10 pixels), it is determined to be a dynamic target (such as a pedestrian or vehicle), and the corresponding feature point is removed.

[0061] In this embodiment, feature point pairs are generated between two adjacent bird's-eye view frames. The Random Sample Consensus (RANSAC) algorithm is used to randomly select four sets of non-collinear samples from all feature point pairs, and the homography matrix is ​​solved. Then, based on the interior point determination criteria and the homography matrix... Select interior points from all feature point pairs and calculate the current homography matrix. Number of interior points in the model and interior point ratio Repeat the above steps 100 times to retain the maximum number of interior points and the interior point ratio that satisfies the condition. The optimal homography matrix model and corresponding set of inliers that reach the inlier ratio threshold (e.g., 60%) are obtained; all outliers are removed, and the filtered inlier pairs are used for subsequent inter-frame motion parameter calculation.

[0062] Specifically, the interior point determination criterion is: for any pair of feature points... Through the currently estimated homography matrix calculate Projection point in the current frame's bird's-eye view homogeneous coordinates Convert to non-homogeneous coordinates ,in, For horizontal homogeneous components, These are vertical homogeneous components. This is the homogeneous scaling factor. Then, based on the feature points... With projection point The Euclidean distance between them yields the projection error. Projection error The calculation formula is as follows:

[0063] in,( ) is a feature point The pixel coordinates in the current frame. (Due to projection error) Not greater than the preset interior point threshold (For example When the pixel value is less than or equal to a certain value, the corresponding feature point pair is determined to be an interior point; otherwise, it is an exterior point. This process filters the feature point pairs, resulting in filtered feature point pairs.

[0064] Furthermore, based on the filtered feature point pairs The Direct Linear Transform (DLT) algorithm is used to solve the inter-frame homography matrix. Among them, the inter-frame homography matrix It can be a 3×3 matrix, or an inter-frame homography matrix. satisfy:

[0065] in,( () represents the feature points of the previous frame. pixel coordinates, ( ) represents the feature point corresponding to the current frame. The pixel coordinates are determined. Furthermore, to improve accuracy, a nonlinear least squares method is used to optimize the inter-frame homography matrix. Determine the inter-frame homography matrix of two adjacent bird's-eye view frames. The objective function is:

[0066] Where M is the number of filtered feature point pairs, M≥30.

[0067] Step S2022: Decompose the inter-frame homography matrix to obtain the inter-frame displacement and inter-frame rotation angle of two adjacent bird's-eye view frames.

[0068] Specifically, since the ground corresponding to the bird's-eye view is a plane, the inter-frame homography matrix... It can be decomposed into the inter-frame displacement of the vehicle ( ) and inter-frame rotation angle The equations are decomposed as follows:

[0069] Where: s is the scale factor (since the bird's-eye view has been scaled, s=1); Let be a rotation matrix. ; It is a translation matrix. .

[0070] Furthermore, by solving the above decomposition equations using Singular Value Decomposition (SVD), the inter-frame shift is obtained. ) and inter-frame rotation angle ,in, For lateral displacement, For longitudinal displacement, , The decomposition accuracy is ≤0.01m, and the decomposition accuracy of Δθ is ≤0.1°.

[0071] This invention determines the inter-frame homography matrix of adjacent bird's-eye view images and decomposes it to obtain the inter-frame displacement and rotation angle, accurately characterizing the motion relationship between adjacent bird's-eye view frames. This provides a data foundation for subsequent vehicle pose trajectory generation and motion compensation, ensuring the alignment accuracy and display stability of the transparent chassis image.

[0072] Step S2023: For each time moment, based on the historical state parameters of the previous time moment, the inter-frame motion parameters between the previous frame bird's-eye view and the current frame bird's-eye view, predict the current state vector corresponding to the current time moment.

[0073] In some optional implementations, step S2023 above includes: Step a1: Based on the historical state parameters of the previous moment, obtain the historical state vector and determine the prior state vector of the vehicle at the current moment.

[0074] Specifically, the historical state parameters include a first state parameter and a second state parameter used to characterize the changing trend of the first state parameter. Step a1 above includes: Step a11: Construct a historical state vector based on the vehicle's first state parameters at the previous moment; wherein the first state parameters include the vehicle position, vehicle speed, heading angle, and heading angular velocity.

[0075] Specifically, a historical state vector is constructed based on the vehicle's position, speed, heading angle, and heading angular velocity from the previous moment. for:

[0076] in, This represents the x-coordinate of the vehicle in the world coordinate system at time t-1. This represents the ordinate of the vehicle in the world coordinate system at time t-1. Let be the heading angle of the vehicle at time t-1. This represents the lateral velocity of the vehicle at time t-1. This represents the longitudinal velocity of the vehicle at time t-1. This represents the angular velocity of the vehicle at time t-1.

[0077] Step a12: Based on the second state parameters and historical state vector of the previous moment, perform state transition to obtain the prior state vector of the vehicle at the current moment; wherein, the second state parameters include acceleration and heading angular acceleration.

[0078] Specifically, based on the acceleration, heading angular acceleration, and historical state vector from the previous moment, the vehicle's prior state vector at the current moment is calculated using the following formula. :

[0079] in, This is the time difference between the previous moment and the current moment (i.e., the frame interval). For the vehicle's lateral acceleration, For the longitudinal acceleration of the vehicle, This is the angular acceleration of the heading.

[0080] This embodiment constructs a historical state vector containing vehicle position, speed, heading angle, and heading angular velocity, and combines acceleration and heading angular acceleration for state transition recursion to accurately predict the prior state vector at the current moment. This avoids image misalignment caused by using sensor signals with long delays to calculate the vehicle's current motion parameters, thereby improving the overall accuracy and stability of subsequent vehicle pose trajectory calculations.

[0081] Step a2: Based on the inter-frame motion parameters between the previous and current bird's-eye view images, construct visual observations.

[0082] Specifically, visual observations are constructed based on the inter-frame displacement and inter-frame rotation angle between two adjacent bird's-eye view frames:

[0083] in, This represents the visual observation value at the current moment.

[0084] Step a3: Based on visual observations, historical state vectors, and prior state vectors, obtain observation noise and observation equations, and determine the prior covariance matrix based on observation noise and observation equations.

[0085] Specifically, theoretical observations are constructed based on the following formula. :

[0086] in, for The x-coordinate of the vehicle in the data. for The vehicle's ordinate in the diagram. for The vehicle's heading angle in the middle, for The lateral speed of the vehicle in the middle, for The longitudinal speed of the vehicle in the middle, for The angular velocity of the vehicle's heading in the middle.

[0087] Furthermore, the observation noise is obtained based on the difference between the visual observations and the observation equation. Then the observation equation is ,in, This represents the state transition function.

[0088] Furthermore, the prior covariance matrix is ​​determined based on the following formula. :

[0089] in, Let represent the covariance matrix at time t-1. This represents the observation Jacobian matrix (obtained by taking the partial derivative of the observation equation). Initial time... It is a diagonal matrix generated based on the initial state or a preset covariance matrix based on common empirical values.

[0090] Step a4: Calculate the Kalman gain based on the prior covariance matrix, observation equation, and observation noise. Then, update the prior state vector based on the Kalman gain and observation noise to obtain the current state vector at the current time.

[0091] Specifically, the Kalman gain is calculated using the following formula:

[0092] in, Indicates Kalman gain, This is the observation Jacobian matrix obtained by taking the partial derivative of the observation equation.

[0093] Furthermore, the prior state vector is updated according to the following formula. This yields the current state vector at the current moment. :

[0094] Furthermore, calculate the covariance matrix at the current time. This is used to calculate the prior covariance matrix for the next time step, where E is the identity matrix.

[0095] This invention accurately predicts the vehicle's current state vector by recursively calculating the prior state vector based on historical state parameters and combining it with visual observations obtained from a bird's-eye view to correct the vehicle's state. By utilizing state transitions to predict the current prior state in advance, compensating for delay biases, and then correcting prediction errors with visual observations, it achieves accurate prediction of vehicle pose. This effectively avoids motion lag caused by data transmission and output delays from onboard sensors, prevents chassis image misalignment due to sensor delays, and improves the alignment accuracy and display stability of transparent chassis images.

[0096] Step S2024: Extract the vehicle pose from the current state vector at each time step to generate the vehicle's pose trajectory.

[0097] Specifically, the current state vector at the current moment Extract the vehicle pose at each time step. Generate the vehicle's pose trajectory.

[0098] This invention uses historical state parameters and vision-based inter-frame motion parameters to collaboratively predict vehicle state vectors, reducing the interference of sensor delay on motion estimation. It extracts vehicle pose based on the state vectors at each time point and generates a coherent pose trajectory, thereby accurately representing the vehicle's motion state and providing a data foundation for subsequent chassis image compensation.

[0099] Step S203: Determine the time delay between the historical acquisition time and the current rendering time of the historical chassis image, and determine the target motion parameters of the vehicle within the time delay based on the pose trajectory.

[0100] Specifically, step S203 includes: Step S2031: Determine the time delay between the historical acquisition time and the current rendering time of the historical chassis image.

[0101] Specifically, let the historical acquisition time corresponding to the historical chassis image be . The current frame rendering time is Then the delay time The delay time is calculated from the image hardware timestamp and the rendering timestamp.

[0102] Step S2032: Based on the pose trajectory, obtain the first pose corresponding to the historical acquisition time and the second pose corresponding to the current rendering time.

[0103] Specifically, based on historical data collection time The current frame rendering time is Match the pose trajectory to obtain the first pose. Second pose .

[0104] Step S2033: Based on the first pose and the second pose, obtain the target displacement and target rotation angle of the vehicle within the delay time.

[0105] Specifically, based on the vehicle's pose trajectory, the fitting interpolation is obtained. to The target displacement of the vehicle within the time period ( ) and target displacement ,Right now:

[0106] in, To accumulate lateral displacement, This represents the cumulative longitudinal displacement.

[0107] This invention accurately extracts the vehicle pose corresponding to the historical acquisition time and the current rendering time based on the vehicle pose trajectory, and then calculates the target displacement and target rotation angle of the vehicle throughout the entire delay time to restore the real motion state of the vehicle during the delay time. This facilitates motion compensation of historical chassis images and reduces the risk of image offset caused by time delay.

[0108] Step S204: Motion compensation is performed on the historical chassis image based on the target motion parameters to obtain a compensated chassis image. The compensated chassis image and the current frame bird's-eye view corresponding to the current rendering time are fused to obtain a transparent chassis image.

[0109] Specifically, step S204 includes: Step S2041: Based on the target displacement and target rotation angle, perform an inverse transformation on the first pixel coordinates in the historical chassis image to obtain the second pixel coordinates.

[0110] Specifically, based on the target displacement and target rotation angle, historical chassis images are analyzed. Perform an inverse transform to cancel out vehicle motion during the delay time, aligning the chassis image with the current image. The chassis position corresponding to a given moment is determined by the following formula:

[0111] in, The coordinates of the first pixel in the historical chassis image. Let the coordinates be the second pixel coordinates, and the two satisfy:

[0112] in, This is a pixel scale for the bird's-eye view. Inverse rotation matrix (cancellation) .

[0113] Step S2042: Obtain the compensated chassis image based on the second pixel coordinates.

[0114] Specifically, through the second pixel coordinates Generate supplementary chassis images .

[0115] This invention utilizes the target displacement and target rotation angle within a delay time to perform a geometric inverse transformation on the pixel coordinates of historical chassis images, offsetting the positional shift and angular deflection of the vehicle during the delay time pixel by pixel, so that the pixel position of the historical chassis image matches the current vehicle chassis area, eliminating the image misalignment caused by the time difference between image acquisition and rendering, and optimizing the visual presentation effect of the transparent chassis.

[0116] Step S2043: Determine the first region and the second region in the current frame bird's-eye view; wherein, the first region is the overlapping region between the compensation chassis image and the current frame bird's-eye view, and the second region is the region other than the first region.

[0117] Specifically, the overlapping area between the compensated chassis image and the current frame bird's-eye view is determined and designated as the first region. The area outside the first region in the current frame bird's-eye view is designated as the second region. It should be noted that the overlapping area of ​​the transparent chassis is the chassis position in the current frame bird's-eye view, that is, the pixel area corresponding to the vehicle outline, and its size can be determined according to the actual vehicle model.

[0118] Step S2044: Determine the fusion weight based on the region type of the overlapping region, and fuse the corresponding regions of the first region and the compensation chassis image according to the fusion weight to obtain the fused compensation region.

[0119] Specifically, a fading weighted fusion algorithm is used to process the compensated historical chassis image. Bird's-eye view of the current frame The overlapping areas are merged using the following formula:

[0120] in, In order to integrate the compensation area, For the first area, This is the fusion weight, which is matched to the region type of the overlapping region. If the overlapping region is a central region, That is, using the compensated chassis image entirely; if the overlapping area is an edge area, The distance between the edge region and the center decreases linearly from 1 to 0, that is, the image smoothly transitions from the compensated chassis image to the bird's-eye view of the current frame. The weighted transition bandwidth can be 20 pixels to avoid visible stitching seams.

[0121] Step S2045: The fusion compensation area and the second area are stitched together to obtain a transparent chassis image.

[0122] Specifically, the fusion compensation area and the second area in the current frame's bird's-eye view are seamlessly stitched together to obtain a transparent chassis image, thus achieving the transition from the compensated chassis image to the current frame's bird's-eye view.

[0123] This invention achieves segmented image processing by dividing the image into an overlapping first region and a non-overlapping second region. It adaptively configures the fusion weights and adjusts the fusion ratio based on the type of overlapping region, weakening the stitching marks at image junctions and making the transition between overlapping areas smoother and more natural. The fused compensation region is then completely stitched together with the non-overlapping region to obtain a transparent chassis image, avoiding misalignment and improving overall image quality and display effect.

[0124] In some embodiments, the display of the transparent chassis image can be adaptively optimized according to the scene type. For example, in a weak texture scene, the number of feature points is automatically reduced to decrease the transparency intensity of the transparent chassis (transparency is reduced from 80% to 50%), and grid auxiliary lines are superimposed to reduce misalignment perception; in a high-speed scene (vehicle speed > 80km / h), the display area of ​​the transparent chassis is reduced.

[0125] The transparent chassis image generation method provided in this embodiment achieves inter-frame motion parameter measurement through visual feature tracking, solving the root cause of misalignment caused by latency; based on centimeter-level precision visual inter-frame motion parameter measurement and reverse geometric compensation, it solves the misalignment problem between the transparent chassis image and the real ground, avoiding image ghosting and discontinuity; it is adaptable to various road surfaces such as strong textures (zebra crossings, paving stones) and weak textures (cement, asphalt roads), and resists dynamic interference from pedestrians, vehicles, etc. through dynamic target filtering and outlier removal, exhibiting strong robustness; it avoids visual drift by fusing multi-source data of inter-frame motion parameters and historical state parameters, ensuring stable operation over a long period of time; by reusing existing surround-view camera hardware, it eliminates the need for new sensors (such as LiDAR), achieving this only through algorithm upgrades, thus reducing the overall vehicle cost; and it reduces the total processing latency per frame, meeting the real-time requirements of in-vehicle scenarios.

[0126] This embodiment also provides a transparent chassis image generation device, which is used to implement the above embodiments and preferred embodiments; details already described will not be repeated. As used below, the term "module" can refer to a combination of software and / or hardware that performs a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0127] This embodiment provides a transparent chassis image generation device, such as... Figure 3 As shown, it includes: The first processing module 301 is used to acquire the bird's-eye view sequence, sensor data sequence and historical chassis image generated by the vehicle during driving; wherein, the sensor data sequence includes the historical state parameters corresponding to each frame of the bird's-eye view; The second processing module 302 is used to obtain the inter-frame motion parameters of two adjacent bird's-eye view frames according to the bird's-eye view sequence, and predict the vehicle pose at each time based on the inter-frame motion parameters and historical state parameters to obtain the pose trajectory. The third processing module 303 is used to determine the delay time between the historical acquisition time and the current rendering time of the historical chassis image, and to determine the target motion parameters of the vehicle within the delay time based on the pose trajectory. The fourth processing module 304 is used to perform motion compensation on the historical chassis image based on the target motion parameters to obtain a compensated chassis image, and to fuse the compensated chassis image with the current frame bird's-eye view corresponding to the current rendering time to obtain a transparent chassis image.

[0128] In some optional implementations, the second processing module 302 is further configured to: Based on the bird's-eye view sequence, determine the inter-frame homography matrix of two adjacent bird's-eye view frames; By decomposing the inter-frame homography matrix, the inter-frame displacement and inter-frame rotation angles of two adjacent bird's-eye view frames are obtained.

[0129] In some optional implementations, the second processing module 302 is further configured to: For each time step, based on the historical state parameters of the previous time step, the inter-frame motion parameters between the previous frame bird's-eye view and the current frame bird's-eye view, predict the current state vector corresponding to the current time step; Extract the vehicle pose from the current state vector at each time step to generate the vehicle's pose trajectory.

[0130] In some optional implementations, the second processing module 302 is further configured to: Based on the historical state parameters of the previous moment, the historical state vector is obtained, and the prior state vector of the vehicle at the current moment is determined. Visual observations are constructed based on the inter-frame motion parameters between the previous and current bird's-eye view images. Based on visual observations, historical state vectors, and prior state vectors, the observation noise and observation equations are obtained, and the prior covariance matrix is ​​determined based on the observation noise and observation equations. The Kalman gain is calculated based on the prior covariance matrix, the observation equation, and the observation noise. The prior state vector is then updated based on the Kalman gain and the observation noise to obtain the current state vector at the current time.

[0131] In some optional implementations, the historical state parameters include a first state parameter and a second state parameter for characterizing the changing trend of the first state parameter; the second processing module 302 is further configured to: Based on the vehicle's first state parameters at the previous moment, a historical state vector is constructed; where the first state parameters include the vehicle's position, speed, heading angle, and heading angular velocity. The state transition is performed based on the second state parameters and historical state vector of the previous moment to obtain the prior state vector of the vehicle at the current moment; where the second state parameters include acceleration and heading angular acceleration.

[0132] In some optional implementations, the third processing module 303 is further configured to: Based on the pose trajectory, the first pose corresponding to the historical acquisition time and the second pose corresponding to the current rendering time are obtained; Based on the first and second poses, the target displacement and target rotation angle of the vehicle during the delay time are obtained.

[0133] In some alternative implementations, the fourth processing module 304 is further configured to: Determine the first region and the second region in the current frame bird's-eye view; wherein, the first region is the overlapping area between the compensation chassis image and the current frame bird's-eye view, and the second region is the area other than the first region; The fusion weight is determined based on the region type of the overlapping region. The corresponding regions of the first region and the compensation chassis image are fused according to the fusion weight to obtain the fused compensation region. The fusion compensation area and the second area are stitched together to obtain a transparent chassis image.

[0134] In some alternative implementations, the fourth processing module 304 is further configured to: Based on the target displacement and target rotation angle, the coordinates of the first pixel in the historical chassis image are inversely transformed to obtain the coordinates of the second pixel. The compensated chassis image is obtained based on the second pixel coordinates.

[0135] The transparent chassis image generation apparatus provided in this embodiment of the invention can execute the transparent chassis image generation method provided in any embodiment of the invention, and has the corresponding functional modules and beneficial effects for executing the method. Further functional descriptions of the above modules and units are the same as in the corresponding embodiments described above, and will not be repeated here.

[0136] Figure 4 This is a structural schematic diagram of a vehicle provided in an embodiment of the present invention.

[0137] The following is a detailed reference. Figure 4The diagram illustrates a structural schematic suitable for implementing a vehicle according to an embodiment of the present invention. The vehicle may include a processor (e.g., a central processing unit, graphics processor, etc.) 401, which can perform various appropriate actions and processes according to a program stored in read-only memory (ROM) 402 or a program loaded from memory 408 into random access memory (RAM) 403. The RAM 403 also stores various programs and data required for vehicle operation. The processor 401, ROM 402, and RAM 403 are interconnected via a bus 404. An input / output (I / O) interface 405 is also connected to the bus 404.

[0138] Typically, the following devices can be connected to I / O interface 405: input devices 406 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 407 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; memory devices 408 including, for example, magnetic tapes, hard disks, etc.; and communication devices 409. Communication device 409 allows the vehicle to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 4 Vehicles with various devices are shown, but it should be understood that it is not required to implement or have all of the devices shown, and more or fewer devices may be implemented or have instead.

[0139] In particular, according to embodiments of the present invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of the present invention include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 409, or installed from a memory 408, or installed from a ROM 402. When the computer program is executed by the processor 401, it performs the functions defined in the transparent chassis image generation method of the embodiments of the present invention.

[0140] Figure 4 The vehicle shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments of the present invention.

[0141] This invention also provides a computer-readable storage medium. The methods described above according to embodiments of the invention can be implemented in hardware or firmware, or implemented as computer code that can be recorded on a storage medium, or implemented as computer code downloaded via a network and originally stored on a remote storage medium or a non-transitory machine-readable storage medium and then stored on a local storage medium. Thus, the methods described herein can be processed by software stored on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. The storage medium can be a magnetic disk, optical disk, read-only memory, random access memory, flash memory, hard disk, or solid-state drive, etc.; further, the storage medium can also include combinations of the above types of memory. It is understood that computers, processors, microprocessor controllers, or programmable hardware include storage components capable of storing or receiving software or computer code. When the software or computer code is accessed and executed by the computer, processor, or hardware, the transparent chassis image generation method shown in the above embodiments is implemented.

[0142] A portion of this invention can be applied as a computer program product, such as computer program instructions, which, when executed by a computer, can invoke or provide the methods and / or technical solutions according to the invention through the operation of the computer. Those skilled in the art will understand that the forms in which computer program instructions exist in a computer-readable medium include, but are not limited to, source files, executable files, installation package files, etc. Correspondingly, the ways in which computer program instructions are executed by a computer include, but are not limited to: the computer directly executing the instructions, or the computer compiling the instructions and then executing the corresponding compiled program, or the computer reading and executing the instructions, or the computer reading and installing the instructions and then executing the corresponding installed program. Here, the computer-readable medium can be any available computer-readable storage medium or communication medium accessible to a computer.

[0143] Although embodiments of the invention have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of the invention, and such modifications and variations all fall within the scope defined by the appended claims.

Claims

1. A method for generating a transparent chassis image, characterized in that, The method includes: The system acquires a sequence of bird's-eye view images, a sequence of sensor data, and historical chassis images generated during vehicle operation; wherein the sensor data sequence includes historical state parameters corresponding to each frame of the bird's-eye view image. Based on the bird's-eye view sequence, the inter-frame motion parameters of two adjacent bird's-eye view frames are obtained, and based on the inter-frame motion parameters and the historical state parameters, the vehicle pose at each moment is predicted to obtain the pose trajectory. Determine the time delay between the historical acquisition time and the current rendering time of the historical chassis image, and based on the pose trajectory, determine the target motion parameters of the vehicle within the time delay. Motion compensation is performed on the historical chassis image based on the target motion parameters to obtain a compensated chassis image. The compensated chassis image and the current frame bird's-eye view corresponding to the current rendering time are then fused to obtain a transparent chassis image.

2. The method for generating a transparent chassis image according to claim 1, characterized in that, The step of predicting the vehicle pose at each moment based on the inter-frame motion parameters and the historical state parameters to obtain the pose trajectory includes: For each time step, based on the historical state parameters of the previous time step, the inter-frame motion parameters between the previous frame bird's-eye view and the current frame bird's-eye view, the current state vector corresponding to the current time step is predicted. Extract the vehicle pose from the current state vector at each time step to generate the vehicle's pose trajectory.

3. The method for generating a transparent chassis image according to claim 2, characterized in that, The prediction of the current state vector corresponding to the current moment based on the historical state parameters of the previous moment and the inter-frame motion parameters between the previous frame bird's-eye view and the current frame bird's-eye view includes: Based on the historical state parameters of the previous moment, the historical state vector is obtained, and the prior state vector of the vehicle at the current moment is determined. Visual observations are constructed based on the inter-frame motion parameters between the previous and current bird's-eye view images. Based on the visual observations, the historical state vector, and the prior state vector, the observation noise and the observation equation are obtained, and the prior covariance matrix is ​​determined based on the observation noise and the observation equation. The Kalman gain is calculated based on the prior covariance matrix, the observation equation, and the observation noise. The prior state vector is then updated based on the Kalman gain and the observation noise to obtain the current state vector at the current moment.

4. The method for generating a transparent chassis image according to claim 3, characterized in that, The historical state parameters include a first state parameter and a second state parameter characterizing the changing trend of the first state parameter; the step of obtaining the historical state vector based on the historical state parameters of the previous moment and determining the prior state vector of the vehicle at the current moment includes: Based on the vehicle's first state parameters at the previous moment, a historical state vector is constructed; wherein, the first state parameters include the vehicle position, vehicle speed, heading angle, and heading angular velocity; Based on the second state parameter of the previous moment and the historical state vector, a state transition is performed to obtain the vehicle's prior state vector at the current moment; wherein, the second state parameter includes acceleration and heading angular acceleration.

5. The method for generating a transparent chassis image according to claim 3, characterized in that, The process of fusing the compensated chassis image and the bird's-eye view of the current frame corresponding to the current rendering time to obtain a transparent chassis image includes: Determine a first region and a second region in the current frame bird's-eye view; wherein, the first region is the overlapping region between the compensated chassis image and the current frame bird's-eye view, and the second region is the region other than the first region; Based on the region type of the overlapping region, a fusion weight is determined, and the corresponding regions of the first region and the compensated chassis image are fused according to the fusion weight to obtain a fused compensation region. The fusion compensation area and the second area are stitched together to obtain a transparent chassis image.

6. The method for generating a transparent chassis image according to any one of claims 1-5, characterized in that, Determining the target motion parameters of the vehicle within the delay time based on the pose trajectory includes: Based on the pose trajectory, the first pose corresponding to the historical acquisition time and the second pose corresponding to the current rendering time are obtained; Based on the first pose and the second pose, the target displacement and target rotation angle of the vehicle during the delay time are obtained.

7. The method for generating a transparent chassis image according to claim 6, characterized in that, The step of performing motion compensation on the historical chassis image based on the target motion parameters to obtain a compensated chassis image includes: Based on the target displacement and the target rotation angle, the first pixel coordinates in the historical chassis image are inversely transformed to obtain the second pixel coordinates; The compensated chassis image is obtained based on the second pixel coordinates.

8. The method for generating a transparent chassis image according to any one of claims 1-5, characterized in that, The step of obtaining the inter-frame motion parameters of two adjacent bird's-eye view frames based on the bird's-eye view sequence includes: Based on the bird's-eye view sequence, determine the inter-frame homography matrix of two adjacent bird's-eye view frames; The inter-frame homography matrix is ​​decomposed to obtain the inter-frame displacement and inter-frame rotation angle of two adjacent bird's-eye view frames.

9. A transparent chassis image generation device, characterized in that, The device includes: The first processing module is used to acquire the bird's-eye view sequence, sensor data sequence, and historical chassis images generated during vehicle operation; wherein, the sensor data sequence includes historical state parameters corresponding to each frame of the bird's-eye view. The second processing module is used to obtain the inter-frame motion parameters of two adjacent bird's-eye view frames according to the bird's-eye view sequence, and predict the vehicle pose at each time based on the inter-frame motion parameters and the historical state parameters to obtain the pose trajectory. The third processing module is used to determine the delay time between the historical acquisition time and the current rendering time of the historical chassis image, and to determine the target motion parameters of the vehicle within the delay time based on the pose trajectory. The fourth processing module is used to perform motion compensation on the historical chassis image based on the target motion parameters to obtain a compensated chassis image, and to fuse the compensated chassis image with the current frame bird's-eye view corresponding to the current rendering time to obtain a transparent chassis image.

10. A vehicle, characterized in that, include: A memory and a processor are communicatively connected, the memory stores computer instructions, and the processor executes the computer instructions to perform the transparent chassis image generation method according to any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions for causing a computer to perform the transparent chassis image generation method according to any one of claims 1 to 8.

12. A computer program product, characterized in that, Includes computer instructions for causing a computer to perform the transparent chassis image generation method according to any one of claims 1 to 8.

Images