Data processing method and device and vehicle

Abstract

The embodiment of the invention provides a data processing method and device and a vehicle, and the method comprises the steps: obtaining data collected by a plurality of sensors in the vehicle in a current time period, and obtaining multi-modal data; based on the multi-modal data and a historical external parameter matrix determined in a historical time period, determining an external parameter residual amount; converting the historical external parameter matrix by using the external parameter residual to obtain a current external parameter matrix, the current external parameter matrix being used for representing relative poses of the plurality of sensors during working in the current time period; the current external parameter matrix is utilized to fuse the multiple pieces of multi-modal data to obtain fused data, and the fused data is used for representing position information of at least one obstacle existing in the area where the vehicle is located. The technical problem of low accuracy of multi-sensor external parameter calibration is solved.

Description

Technical Field

[0001] This application relates to the field of vehicles, and more specifically, to a data processing method, apparatus, and vehicle. Background Technology

[0002] Currently, calibration methods for the external parameters of multiple vehicle sensors mainly include offline calibration methods and online calibration methods based on natural scenes. Offline calibration methods cannot promptly correct calibration errors caused by vehicle vibration, temperature changes, or component aging; online calibration methods based on natural scenes are easily affected by certain environmental conditions. Both of these methods suffer from low accuracy in calibrating the external parameters of multiple sensors.

[0003] There is currently no good solution to the above problems. Summary of the Invention

[0004] This application provides a data processing method, apparatus, and vehicle to at least address the technical problem of low accuracy in calibrating extrinsic parameters of multiple sensors.

[0005] According to one aspect of the embodiments of this application, a data processing method is provided. The method may include: acquiring data collected by multiple sensors in a vehicle during a current time period to obtain multimodal data; determining extrinsic parameter residuals based on the multimodal data and a historical extrinsic parameter matrix determined during a historical time period, wherein the historical extrinsic parameter matrix is ​​used to characterize the relative poses of the multiple sensors when they were working during the historical time period, and the extrinsic parameter residuals are used to characterize the degree of deviation between the historical extrinsic parameter matrix and the current extrinsic parameter matrix to be determined; transforming the historical extrinsic parameter matrix using the extrinsic parameter residuals to obtain a current extrinsic parameter matrix, wherein the current extrinsic parameter matrix is ​​used to characterize the relative poses of the multiple sensors when they were working during the current time period; and fusing the multiple multimodal data using the current extrinsic parameter matrix to obtain fused data, wherein the fused data is used to characterize the location information of at least one obstacle in the area where the vehicle is located.

[0006] Furthermore, the multimodal data includes point cloud data acquired by radar sensors and image data acquired by image sensors. Based on the multimodal data and the historical extrinsic parameter matrix determined within the historical time period, the extrinsic parameter residuals are determined, including: projecting the point cloud data into planar image data using the historical extrinsic parameter matrix; and determining the extrinsic parameter residuals based on the planar image data, the image data, and the historical extrinsic parameter matrix.

[0007] Furthermore, based on the planar image data, the image data, and the historical extrinsic parameter matrix, the extrinsic parameter residuals are determined, including: performing convolution processing on the planar image data to obtain key vectors and value vectors, and performing convolution processing on the image data to obtain a query vector, wherein the key vector is used to characterize the spatial feature distribution of the planar image data, the value vector is used to characterize the pixel value information of the planar image data, and the query vector is used to characterize the contextual relevance of the image data; transforming the query vector using the key vector to obtain an attention correlation matrix, wherein the attention correlation matrix is ​​used to characterize the correlation strength between spatial location points in the image data and spatial location points in the planar image data; fusing the attention correlation matrix and the value vector to obtain a fused feature; and determining the extrinsic parameter residuals based on the fused feature and the historical extrinsic parameter matrix.

[0008] Furthermore, based on the fusion features and the historical extrinsic parameter matrix, the extrinsic parameter residuals are determined, including: transforming the historical extrinsic parameter matrix to obtain Lie algebraic coordinates, where the Lie algebraic coordinates are used to characterize the rigid body transformation state of the historical extrinsic parameter matrix; and determining the extrinsic parameter residuals based on the fusion features and the Lie algebraic coordinates.

[0009] Furthermore, based on the fusion features and Lie algebra coordinates, the extrinsic parameter residuals are determined, including: encoding the Lie algebra coordinates to obtain the extrinsic parameter embedding vector; expanding the extrinsic parameter embedding vector to obtain the prior information features; concatenating the multimodal fusion features and the prior information features to obtain the concatenated features; decoding the concatenated features to obtain the feature image; and transforming the feature image to obtain the extrinsic parameter residuals.

[0010] Furthermore, the historical external parameter matrix is ​​transformed using the external parameter residuals to obtain the current external parameter matrix. This includes summing the external parameter residuals and the historical external parameter matrix to obtain the current external parameter matrix.

[0011] Furthermore, using the current extrinsic parameter matrix, multiple multimodal data are fused to obtain fused data, including: using the current extrinsic parameter matrix to project point cloud data onto image data to obtain fused data.

[0012] Furthermore, the method also includes: identifying the location information of obstacles by analyzing the fused data; and determining the vehicle's trajectory in a future time period based on the location information, wherein the future time period is later than the current time period.

[0013] According to another aspect of the embodiments of this application, an information processing apparatus is also provided, which may include: an acquisition unit, configured to acquire data collected by multiple sensors in a vehicle during the current time period to obtain multimodal data; a determination unit, configured to determine extrinsic parameter residuals based on the multimodal data and a historical extrinsic parameter matrix determined during a historical time period, wherein the historical extrinsic parameter matrix is ​​used to characterize the relative poses of the multiple sensors when they were working during the historical time period, and the extrinsic parameter residuals are used to characterize the degree of deviation between the historical extrinsic parameter matrix and the current extrinsic parameter matrix to be determined; a conversion unit, configured to convert the historical extrinsic parameter matrix using the extrinsic parameter residuals to obtain a current extrinsic parameter matrix, wherein the current extrinsic parameter matrix is ​​used to characterize the relative poses of the multiple sensors when they were working during the current time period; and a fusion unit, configured to fuse the multiple multimodal data using the current extrinsic parameter matrix to obtain fused data, wherein the fused data is used to characterize the location information of at least one obstacle in the area where the vehicle is located.

[0014] According to another aspect of the embodiments of this application, a vehicle is also provided, including: a memory storing an executable program; and a processor for running the program, wherein the program executes the methods of various embodiments of this application when it runs.

[0015] According to another aspect of the embodiments of this application, a computer-readable storage medium is also provided, the computer-readable storage medium including a stored executable program, wherein, when the executable program is running, it controls the device where the computer-readable storage medium is located to run the methods of various embodiments of this application.

[0016] According to another aspect of the embodiments of this application, a computer program product is also provided, including a computer program that implements the methods of various embodiments of this application when run by a processor.

[0017] According to another aspect of the embodiments of this application, a computer program product is also provided, including a non-volatile computer-readable storage medium storing a computer program, which is executed by a processor to implement the methods in various embodiments of this application.

[0018] According to another aspect of the embodiments of this application, a computer program is also provided, which is executed by a processor to implement the methods of the various embodiments of this application.

[0019] In this embodiment, data collected by multiple sensors in the vehicle during the current time period is obtained to obtain multimodal data; based on the multimodal data and the historical extrinsic parameter matrix determined during the historical time period, the extrinsic parameter residual is determined; the historical extrinsic parameter matrix is ​​transformed using the extrinsic parameter residual to obtain the current extrinsic parameter matrix, which is used to characterize the relative poses of multiple sensors when they are working during the current time period; the multiple multimodal data are fused using the current extrinsic parameter matrix to obtain fused data. In other words, in this embodiment, by collecting multimodal data in real time and determining the extrinsic parameter residuals based on the historical extrinsic parameter matrix determined within a historical time period, the historical extrinsic parameter matrix is ​​transformed based on the extrinsic parameter residuals to obtain the current extrinsic parameter matrix. The updated current extrinsic parameter matrix is ​​then used for multi-sensor data fusion, achieving the goal of real-time and accurate calibration of the relative poses between sensors and outputting high-quality fusion perception results. Therefore, it overcomes the problem that traditional calibration methods are easily affected by the aging of vehicle components and the environment in which the vehicle sensors are located, which leads to a reduction in the accuracy of multi-sensor extrinsic parameter calibration. This achieves the technical effect of improving the accuracy of multi-sensor extrinsic parameter calibration and solves the technical problem of low accuracy in multi-sensor extrinsic parameter calibration. Attached Figure Description

[0020] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0021] Figure 1 This is a flowchart of an information processing method according to an embodiment of this application;

[0022] Figure 2 This is a flowchart of an end-to-end online calibration method according to an embodiment of this application;

[0023] Figure 3 This is a flowchart illustrating the construction of an end-to-end deep neural network model according to an embodiment of this application;

[0024] Figure 4 This is a flowchart illustrating the application of an attention mechanism in multimodal feature fusion according to an embodiment of this application;

[0025] Figure 5 This is a flowchart of a calibration quality assessment and degradation process according to an embodiment of this application;

[0026] Figure 6 This is a schematic diagram of an information processing apparatus according to an embodiment of this application. Detailed Implementation

[0027] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0028] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0029] According to an embodiment of this application, a method embodiment for determining the load rate of a processor is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0030] Currently, high-performance autonomous driving systems heavily rely on multi-sensor fusion technology, and accurate extrinsic parameter calibration (i.e., determining the spatial transformation relationship between different sensor coordinate systems, including rotation and translation) is a prerequisite for multi-sensor fusion.

[0031] In related technologies, extrinsic parameter calibration methods for vehicle-mounted multi-sensor systems are mainly divided into two categories: offline calibration methods and online calibration methods based on natural scenes. Offline calibration methods rely on specific calibration objects (such as checkerboard patterns or 3D calibration boards), and these methods are performed by professionals while the vehicle is stationary. Online calibration methods based on natural scenes aim to overcome the limitations of offline calibration by utilizing natural features on the road (such as lane lines, edges, and building outlines) for online calibration. These methods aim to achieve extrinsic parameter calibration without relying on calibration objects.

[0032] The aforementioned existing technologies have the following main drawbacks and shortcomings. First, they require professional personnel and specific calibration sites, making them inconvenient to use in daily vehicle operation. Second, during long-term vehicle use, vibration, impact, temperature changes, and component aging can cause minor changes in sensor installation positions (i.e., "extrinsic parameter degradation"), which gradually renders the offline calibration parameters ineffective, severely impacting the accuracy of fused perception, and cannot be automatically compensated for. Results obtained in an ideal, static calibration environment may not be optimal in complex, dynamic real-world road environments. Furthermore, these methods heavily rely on manually designed features (e.g., edges, corners), and feature extraction and matching are prone to failure in feature-sparse (e.g., highways), repetitive textures, or adverse weather conditions (e.g., rain, snow, fog), leading to unstable or even erroneous calibration.

[0033] Optionally, the above methods also involve complex iterative optimization algorithms (such as nonlinear least squares), resulting in high computational costs and making it difficult to meet the stringent real-time requirements of autonomous driving systems (typically needing to be completed within 100 milliseconds). Furthermore, these methods are susceptible to interference from dynamic objects, as numerous vehicles, pedestrians, and other dynamic objects on the road can contaminate the features used for calibration, leading to estimation bias.

[0034] In summary, the relevant technologies cannot meet the stringent requirements of autonomous driving systems for the accuracy, stability, and real-time performance of multi-sensor extrinsic parameter calibration under long lifecycles and all operating conditions.

[0035] In this embodiment, data collected by multiple sensors in a vehicle during the current time period is obtained to form multimodal data. Based on the multimodal data and the historical extrinsic parameter matrix determined during historical time periods, the extrinsic parameter residual is determined. The historical extrinsic parameter matrix is ​​transformed using the extrinsic parameter residual to obtain the current extrinsic parameter matrix, which is used to characterize the relative poses of multiple sensors when they are working during the current time period. The multiple multimodal data are fused using the current extrinsic parameter matrix to obtain fused data. This achieves the technical effect of improving the accuracy of multi-sensor extrinsic parameter calibration and solves the technical problem of low accuracy in multi-sensor extrinsic parameter calibration.

[0036] Optionally, the embodiments of this application have the following significant advantages: No manual intervention or specific calibration materials are required; the entire process from data input to extrinsic parameter output is completed automatically, achieving "seamless" calibration. Through residual learning and online learning, the slow drift of extrinsic parameters can be continuously tracked and compensated, ensuring the reliability of multiple sensors throughout the vehicle's lifecycle. Deep learning models can learn complex nonlinear mappings from massive amounts of data; combined with attention mechanisms, they are insensitive to noise and interference, maintaining high accuracy under various complex road conditions. The forward inference process has low computational cost, meeting the real-time requirements of automotive-grade systems, and can be embedded into relevant sensor data processing pipelines.

[0037] This application provides an information processing method in its embodiments. Figure 1 This is a flowchart of an information processing method according to an embodiment of this application. Figure 1 As shown, the method may include the following steps.

[0038] Step S102: Obtain data collected by multiple sensors in the vehicle during the current time period to obtain multimodal data.

[0039] In the technical solution provided in step S102 of this application, the aforementioned sensors may include, but are not limited to, sensors in the vehicle such as cameras, lidar, millimeter-wave radar, inertial measurement units (IMUs), and global positioning systems (GPS). The aforementioned multimodal data may include sensor data collected by multiple sensors, such as images collected by cameras and point cloud data collected by radar sensors. This data can be used to characterize the operating status or location information of different obstacles in the area where the vehicle is located, and can also be referred to as multimodal data pairs.

[0040] Optionally, data from at least two different types of sensors can be collected simultaneously during the current time period to form a multimodal data pair.

[0041] Optionally, the vehicle's multi-sensor data acquisition module can synchronously acquire raw data from each sensor, i.e., sensor data. This multi-sensor data acquisition module can be an integrated hardware and software system comprising a series of vehicle sensors and corresponding control units. The control units can control the corresponding sensors to acquire sensor data.

[0042] For example, during the current time period, image data of the vehicle's surrounding environment can be captured by the vehicle's camera. This image data can be used to provide visual information, including roads, vehicles, pedestrians, signs, etc. Furthermore, a lidar sensor in the vehicle can emit laser pulses and receive reflected signals to generate three-dimensional point cloud data of the vehicle's surrounding environment. Millimeter-wave radar sensors in the vehicle can provide detection information for distant targets, including but not limited to the distance, speed, and azimuth angle of the aforementioned distant targets.

[0043] Step S104: Based on multimodal data and the historical extrinsic parameter matrix determined within the historical time period, determine the extrinsic parameter residual. The historical extrinsic parameter matrix is ​​used to characterize the relative poses of multiple sensors when they are working within the historical time period, and the extrinsic parameter residual is used to characterize the degree of deviation between the historical extrinsic parameter matrix and the current extrinsic parameter matrix to be determined.

[0044] In the technical solution provided in step S104 of this application, the aforementioned historical extrinsic parameter matrix can be used to characterize the relative poses of multiple sensors during historical periods. It can be a rigid body transformation matrix, for example, a 4x4 rigid body transformation matrix, or a matrix containing a rotation matrix (R) and a translation vector (T). The aforementioned historical extrinsic parameter matrix can contain multiple extrinsic parameters. The aforementioned extrinsic parameter residuals are used to characterize the degree of deviation between the historical extrinsic parameter matrix and the current extrinsic parameter matrix to be determined, and can include a rotation residual matrix (ΔT) and a translation residual matrix (ΔR). The aforementioned relative poses can be used to characterize the relative positions and attitudes between different sensors.

[0045] Optionally, after obtaining multimodal data, the residual amount of the extrinsic parameters can be determined based on the multimodal data and the historical extrinsic parameter matrix determined within the historical period.

[0046] Optionally, records of historical time periods from one or more vehicles are first compiled and used. These records may contain relative position information (i.e., relative pose) between sensors during the historical time periods. The historical extrinsic parameter matrix (which may also be referred to as the extrinsic parameter estimate in this embodiment) can be determined using the above method.

[0047] Optionally, the multimodal data (e.g., image data and radar data) at the current moment is standardized and then input into a pre-trained deep learning model. The deep learning model is based on the multimodal data and the historical extrinsic parameter matrix, and outputs the extrinsic parameter residuals through a series of nonlinear transformations.

[0048] Step S106: Using the extrinsic residual, the historical extrinsic parameter matrix is ​​transformed to obtain the current extrinsic parameter matrix. The current extrinsic parameter matrix is ​​used to characterize the relative poses of multiple sensors when they are working in the current time period.

[0049] In the technical solution provided by step S106 of this application, the aforementioned current extrinsic parameter matrix can be used to characterize the relative poses of multiple sensors when they are working in the current time period, and can also be referred to as the estimated value of extrinsic parameters at the current time.

[0050] Optionally, after determining the extrinsic residuals, the historical extrinsic parameter matrix can be transformed using the extrinsic residuals to obtain the current extrinsic parameter matrix.

[0051] Optionally, the aforementioned extrinsic residuals can be applied to the historical extrinsic parameter matrix to update the relative poses between sensors and generate a current extrinsic parameter matrix that reflects the current moment. For example, the rotation and translation residual matrices ΔR and ΔT can be applied to the historical extrinsic parameter matrix, and matrix multiplication and / or vector addition can be performed in the space of the rotation matrix (R) and translation vector (T) to obtain a new extrinsic parameter matrix, i.e., the current extrinsic parameter matrix.

[0052] Step S108: Using the current external parameter matrix, multiple multimodal data are fused to obtain fused data, wherein the fused data is used to characterize the location information of at least one obstacle in the area where the vehicle is located.

[0053] In the technical solution provided by step S108 of this application, the fused data can be used to characterize the location information of at least one obstacle in the area where the vehicle is located. The fused data can also be a perception result with significantly improved quality generated by downstream perception modules (such as 3D target detection, semantic segmentation, drivable area segmentation, etc.) based on multimodal data that has been optimized and aligned in real time. It can also be called a fused perception result.

[0054] Optionally, after determining the current external parameter matrix, multiple multimodal data can be fused to obtain fused data.

[0055] Optionally, the multimodal data is spatiotemporally aligned based on the current external parameter matrix; environmental features, such as edges, textures, and point cloud density distribution, are extracted from the aligned multimodal data; the environmental features are fused together using a pre-designed multi-sensor fusion algorithm; and the fused environmental features are input into a pre-designed obstacle detection algorithm, such as 3D target detection, to identify and locate obstacles around the vehicle and their corresponding location information (i.e., fused data).

[0056] Through steps S102 to S108 above, data collected by multiple sensors in the vehicle during the current time period are obtained to obtain multimodal data; based on the multimodal data and the historical extrinsic parameter matrix determined in the historical time period, the extrinsic parameter residual is determined; using the extrinsic parameter residual, the historical extrinsic parameter matrix is ​​transformed to obtain the current extrinsic parameter matrix, which is used to characterize the relative poses of multiple sensors when they are working in the current time period; using the current extrinsic parameter matrix, multiple multimodal data are fused to obtain fused data. In other words, in this embodiment, by collecting multimodal data in real time and determining the extrinsic parameter residuals based on the historical extrinsic parameter matrix determined within a historical time period, the historical extrinsic parameter matrix is ​​transformed based on the extrinsic parameter residuals to obtain the current extrinsic parameter matrix. The updated current extrinsic parameter matrix is ​​then used for multi-sensor data fusion, achieving the goal of real-time and accurate calibration of the relative poses between sensors and outputting high-quality fusion perception results. Therefore, it overcomes the problem that traditional calibration methods are easily affected by the aging of vehicle components and the environment in which the vehicle sensors are located, which leads to a reduction in the accuracy of multi-sensor extrinsic parameter calibration. This achieves the technical effect of improving the accuracy of multi-sensor extrinsic parameter calibration and solves the technical problem of low accuracy in multi-sensor extrinsic parameter calibration.

[0057] The above-mentioned method of this application will be further described below.

[0058] As an optional implementation, step S104, the multimodal data includes point cloud data collected by radar sensors and image data collected by image sensors. Based on the multimodal data and the historical extrinsic parameter matrix determined within the historical time period, the extrinsic parameter residual is determined, including: projecting the point cloud data into planar image data using the historical extrinsic parameter matrix; and determining the extrinsic parameter residual based on the planar image data, the image data, and the historical extrinsic parameter matrix.

[0059] In this embodiment, the point cloud data can be used to represent the three-dimensional spatial distribution of the surrounding environment detected by radar sensors in the vehicle. The image data can be used to represent environmental visual information captured by the vehicle's camera. The planar image data can be the visualization data of the radar point cloud data on the image plane after coordinate transformation, and can also be called a dense depth map or depth map.

[0060] Optionally, the coordinates of the point cloud data are transformed using the rotation matrix and translation vector recorded in the historical external parameter matrix; the Z coordinate corresponding to each transformed point cloud data is the depth information; the X and Y coordinates of each point cloud data are converted into pixel coordinates; based on the depth information and pixel coordinates calculated above, a planar image data (i.e., a depth map or dense depth map) with the same size as the image data is generated.

[0061] For example, a historical extrinsic parameter matrix (T_offline) pre-calibrated offline (e.g., using a checkerboard pattern) can be loaded from the vehicle's non-volatile memory. T_offline can be a 4x4 rigid body transformation matrix containing a rotation matrix (R) and a translation vector (T). The rotation matrix can be a 3x3 orthogonal rotation matrix, defining the rotation of the coordinate system. The translation vector can be a 3x1 translation vector, defining the translation of the coordinate system. It should be noted that this is merely an example; the values ​​of the numbers are not specifically limited, and the content of the historical extrinsic parameter matrix can be selected according to the actual situation without specific restrictions.

[0062] For another example, multimodal data can be acquired synchronously and preprocessed. This preprocessing process can include hardware-triggered or software-based synchronization based on Precision Time Protocol Timestamp (PTP) to ensure that each frame of image data (I) and each frame of point cloud data (P_lidar) are strictly corresponding in time. Furthermore, the sparse 3D point cloud data can be projected onto a 2D image plane using the historical extrinsic parameter matrix (T_prev) to generate a depth map (D) that is pixel-aligned with the image data.

[0063] Alternatively, in addition to transformations in the Lie algebra space, a gated cyclic unit or long short-term memory network module can be designed to explicitly model the time-series dynamics of extrinsic parameters, thereby enabling more intelligent fusion and filtering.

[0064] Optionally, a two-dimensional matrix corresponding to the image data needs to be initialized as a depth map based on the image data size. Each element in this two-dimensional matrix is ​​initialized to zero, indicating that there is no valid depth value at the pixel location. Therefore, the three-dimensional coordinates of each point cloud data acquired by the LiDAR deployed in the vehicle can be converted into homogeneous coordinates. Based on the historical extrinsic parameter matrix, these homogeneous coordinates are transformed into the image coordinate system to obtain the three-dimensional position of each point cloud in the image coordinate system.

[0065] Optionally, in the above process, points behind the vehicle's camera (i.e., points with a Z value less than or equal to zero in the image coordinate system) can be removed. For points in front of the vehicle's camera, using the camera's intrinsic parameter matrix, the 3D position of these points in the camera coordinate system is projected onto the 2D image pixel plane to obtain the corresponding pixel coordinates. The pixel coordinates are then rounded, and it is determined whether the rounded pixel coordinates fall within the valid image pixel range. If the rounded pixel coordinates are within the valid range, the currently recorded depth value of the pixel coordinates is checked: if no depth value has been recorded for the pixel coordinates, or if the distance of the current point relative to the camera (i.e., the Z value of the point in the image coordinate system) is less than the recorded distance, the depth value of the corresponding pixel coordinate in the depth map is updated using the distance value of the current point. After traversing the point cloud data, the resulting 2D matrix is ​​the generated depth map, where non-zero values ​​represent the closest distance of the corresponding pixel position in the image to the camera in 3D space. The image coordinate system can be used to represent the coordinate system of the image captured by the vehicle's camera on a 2D plane. The pixel coordinates can be used to represent the position of a specific pixel in the image.

[0066] Optionally, depth map features and image features are extracted from the planar image data and image data respectively through the encoder network in the pre-trained deep learning model; the extracted depth map features and image features are fused through an attention mechanism to obtain a fused feature containing information from both modalities; the fused feature and the historical extrinsic parameter matrix are input into the regression layer of the deep learning model to predict the extrinsic parameter residual between the current relative pose between the sensors and the historical parameters, which can also be called the small change.

[0067] For example, the multimodal data pairs and the previous time step extrinsic parameter estimates (i.e., the historical extrinsic parameter matrix in this embodiment) are input into a pre-trained end-to-end deep neural network model; the end-to-end deep neural network model outputs the residual of the current time step extrinsic parameters (i.e., the current extrinsic parameter matrix) relative to the previous time step extrinsic parameter estimates (i.e., the extrinsic parameter residual).

[0068] For another example, the preprocessed multimodal data and the Lie algebra coordinates (ξ_prev) corresponding to the historical extrinsic parameter matrix, i.e., {I, D, ξ_prev}, can be fed into an end-to-end deep learning calibration engine. This end-to-end deep learning calibration engine is deployed in the vehicle's computing unit to run the aforementioned deep neural network model to process the input data and output the extrinsic parameter residuals. The core of this end-to-end deep learning calibration engine is a pre-trained encoder-decoder-attention network structure. Through feature extraction and feature fusion, the historical extrinsic parameter matrix is ​​combined with the fused features to obtain the extrinsic parameter residuals.

[0069] Alternatively, in addition to generating depth maps, the laser point cloud can be converted into a voxel grid or the original point cloud can be directly processed using a point cloud neural network and then fused with image features.

[0070] In this embodiment, the above method utilizes a historical extrinsic parameter matrix to transform the coordinate system of point cloud data collected by the vehicle's radar sensors, generating a depth map pixel-aligned with the image data. This method ensures strict temporal and spatial matching between point cloud data and image data, providing high-quality multimodal input for subsequent deep learning models. The encoder of the deep learning network extracts depth map features and image features from the image data and planar image data. Through attention mechanisms and other methods, effective fusion of cross-modal features is achieved, resulting in a comprehensive fused feature that reflects environmental changes. Based on the fused feature and the historical extrinsic parameter matrix, the extrinsic parameter residual between the relative poses of the sensors at the current moment and historical parameters can be predicted. This ensures that the calibration stability and accuracy are maintained even under long-term vehicle operation and various environmental conditions.

[0071] As an optional implementation, the extrinsic parameter residuals are determined based on planar image data, image data, and a historical extrinsic parameter matrix. This includes: convolving the planar image data to obtain key vectors and value vectors, and convolving the image data to obtain a query vector. The key vector represents the spatial feature distribution of the planar image data, the value vector represents the pixel value information of the planar image data, and the query vector represents the contextual relevance of the image data. The query vector is then transformed using the key vector to obtain an attention correlation matrix, which represents the correlation strength between spatial location points in the image data and spatial location points in the planar image data. The attention correlation matrix and value vector are then fused to obtain a fused feature. Finally, the extrinsic parameter residuals are determined based on the fused feature and the historical extrinsic parameter matrix.

[0072] In this embodiment, the key vector can be used to capture the spatial feature distribution in planar image data, such as edges, textures, and structural layouts. The value vector can be used to characterize the pixel value information of the planar image data, reflecting details such as color and brightness. During the feature fusion stage, the information carried by the value vector can be utilized by the query vector to obtain richer cross-modal features. The query vector can be used to characterize the contextual relevance of the image data and to determine the information in the key vector associated with the image data. The attention correlation matrix can be used to characterize the correlation strength between spatial location points in the image data and spatial location points in the planar image data. The fused feature can be a fused visual feature or a fused feature identifier, and can be represented by F_fused.

[0073] Optionally, the image data (F_img) is input into a 1x1 convolutional layer via a convolutional neural network to generate a query vector (Q). Simultaneously, the depth map features (F_depth) are input into two independent 1x1 convolutional layers to generate a key vector (K) and a value vector (V). These steps map the planar image data and its corresponding original features to a common subspace for calculating association. Further, the matrix product between the query vector Q and the transpose of the key vector K is calculated, and the result is scaled using a scaling factor (which can be the reciprocal of the square root of the key vector dimension d_k) to stabilize gradients during training. Subsequently, a softmax normalization function is applied along the specified dimension to the scaled result to obtain the attention association matrix A. Each element A_ij in this attention association matrix can be used to quantify the association strength between the i-th spatial location in the image feature map (i.e., the image data) and the j-th spatial location in the depth map feature map (i.e., the planar image data). Here, the depth map features can be planar image data.

[0074] Alternatively, in addition to convolutional neural networks, visual transformers (ViTs) can also be used as the backbone network for image feature extraction.

[0075] Optionally, the calculated attention correlation matrix A is used to perform a weighted summation on the value vector V to achieve the purpose of fusing the attention correlation matrix and the value vector. That is, matrix A can be multiplied by matrix V to obtain the fused feature representation F_fused. This operation ensures that the output fused feature contains information from all locations of the depth map features, weighted according to their correlation with the image features, at each location. The output F_fused is the multimodal fused feature modulated by the attention mechanism.

[0076] Optionally, the fused visual features F_fused obtained in the above steps are combined with the prior extrinsic information from the previous time step (i.e., the historical extrinsic parameter matrix) and input together into the subsequent decoder network module to obtain the residual amount of the extrinsic parameters at the current time step relative to the prior extrinsic information.

[0077] For example, image data and depth maps can be processed separately by two structurally identical but weighted convolutional encoders to extract high-level features; deep learning networks can autonomously focus on static, cross-modal consistent features sensitive to pose changes and suppress interference from dynamic objects and noise; historical extrinsic parameter matrices can be combined with fused visual features for prediction of extrinsic residuals. Alternatively, a transformer architecture can replace the aforementioned attention correlation matrix, utilizing the Transformer's self-attention layers to better model long-range cross-modal dependencies.

[0078] In this embodiment of the application, the above method can achieve high-level fusion of cross-modal data and accurate prediction of extrinsic residuals. By using deep learning and attention mechanisms, the limitations of traditional calibration methods are overcome, and long-term stable online calibration of extrinsic parameters of vehicle-mounted multi-sensor systems is achieved.

[0079] As an optional implementation, the extrinsic parameter residuals are determined based on the fusion features and the historical extrinsic parameter matrix, including: transforming the historical extrinsic parameter matrix to obtain Lie algebraic coordinates, wherein the Lie algebraic coordinates are used to characterize the rigid body transformation state of the historical extrinsic parameter matrix; and determining the extrinsic parameter residuals based on the fusion features and the Lie algebraic coordinates.

[0080] In the embodiments of this application, the above-mentioned Lie algebraic coordinates (ξ) can be used to characterize the rigid body transformation state of the historical external parameter matrix, and the above-mentioned Lie algebraic coordinates can be a six-dimensional vector.

[0081] Optionally, a historical extrinsic parameter matrix (T_offline) obtained in advance through high-precision offline calibration (e.g., using a checkerboard pattern) can be loaded from the vehicle's non-volatile memory; the T_offline can be a 4x4 rigid body transformation matrix; the T_offline can be converted into the corresponding Lie algebraic coordinates ξ_prev.

[0082] Optionally, a 3x3 rotation matrix and a 3x1 translation vector are separated from the aforementioned 4x4 historical extrinsic parameter matrix; the rotation matrix can be converted into a 3D rotation vector using the Rodriguez formula; the 3x1 translation vector is directly read from the historical extrinsic parameter matrix; the obtained rotation vector and translation vector are combined to form a six-dimensional vector [w x， w y， w z， v x， v y， v z], that is, the Lie algebra coordinates (ξ). Among them, W can be used to represent the rotation vector, V can represent the translation vector, and x, y, z are used to represent the projections of each component of the Lie algebra vector onto the corresponding three-dimensional coordinate axes; based on the fusion features determined above and the Lie algebra coordinates above, the extrinsic parameter residuals can be determined.

[0083] In the embodiments of this application, the historical extrinsic parameter matrix can be converted into Lie algebra coordinates by the above method, which simplifies mathematical operations, provides a direct input form for deep learning networks, and facilitates the prediction of extrinsic parameter residuals.

[0084] As an optional implementation, the extrinsic residual is determined based on the fused features and Lie algebra coordinates, including: encoding the Lie algebra coordinates to obtain the extrinsic embedding vector; expanding the extrinsic embedding vector to obtain the prior information features; concatenating the multimodal fused features and the prior information features to obtain the concatenated features; decoding the concatenated features to obtain the feature image; and transforming the feature image to obtain the extrinsic residual.

[0085] In this embodiment, the aforementioned extrinsic parameter embedding vector can be used to represent a vector that can be embedded into a deep learning network, obtained by transforming Lie algebra coordinates. The aforementioned prior information features can be used to represent a feature map with a certain spatial resolution, obtained by expanding the spatial dimension based on the extrinsic parameter embedding vector, such as a prior information feature map. The aforementioned concatenated features can be used to represent a comprehensive feature that integrates multimodal fusion features and prior information features. The aforementioned feature image can be used to represent a feature map containing internal and external information obtained after a series of decoding operations on the concatenated features.

[0086] Optionally, the Lie algebra coordinates ξ_prev can be input into a fully connected layer or a series of fully connected layers for encoding transformation to obtain a higher-dimensional extrinsic embedding vector (ξ_prev_encoded).

[0087] Optionally, the extrinsic embedding vector (ξ_prev_encoded) obtained above is expanded (or broadcast) in the corresponding spatial dimensions, so that the extrinsic embedding vector is consistent with the size of the fused visual feature map (F_fused) in the height and width dimensions, thereby obtaining a spatial dimension (H). f W f The prior information feature map (ξ_prev_broadcast) of H. Where, H f It can be used to represent the height dimension, W f It can be used to represent the width dimension.

[0088] Optionally, in the channel dimension, the multimodal fusion feature (F_fused) is concatenated with the prior information feature map (ξ_prev_broadcast) to form a combined feature (F_combined) containing more information. This combined feature can be a concatenated feature.

[0089] Optionally, the combined features (F_combined) are input into a lightweight decoder network consisting of several convolutional layers. The decoder network further combines F_fused with ξ_prev_broadcast and outputs the decoded feature image (F_decoded).

[0090] Optionally, a global average pooling operation is performed on the decoded feature image (F_decoded) to aggregate the spatial information of the feature image and obtain a global feature vector (F_global). The global feature vector (F_global) is then input into a fully connected regression head network to directly regress and output a 6-DOF extrinsic residual Δξ (i.e., a small increment [Δω_x, Δω_y, Δω_z, Δv_x, Δv_y, Δv_z] in Lie algebra coordinates). Simultaneously, the F_global is input into another parallel fully connected network branch, and the output is processed by an exponential function to obtain the uncertainty measure of the extrinsic estimation of the current frame, i.e., the variance σ².

[0091] In this embodiment, the Lie algebra coordinates can be encoded into extrinsic embedding vectors through a fully connected layer using the above method. This achieves information compression and reshaping, enhances feature representation capabilities, and makes the encoded Lie algebra coordinates more suitable for processing and optimization within a deep learning framework. The decoder network gradually restores the spliced ​​comprehensive features into a feature image, which can recover the spatial distribution information of the features. This allows the deep learning network to understand and predict the changes in relative pose between sensors from a local to a global perspective.

[0092] As an optional implementation, the historical external parameter matrix is ​​transformed using the external parameter residuals to obtain the current external parameter matrix, including: summing the external parameter residuals and the historical external parameter matrix to obtain the current external parameter matrix.

[0093] In the embodiments of this application, the aforementioned current extrinsic parameter matrix can be used to characterize an extrinsic parameter estimate that includes the relative pose relationship between sensors after real-time adjustment and optimization.

[0094] Optionally, based on the externally calibrated residual quantity predicted by the deep learning network, the prior external calibration is updated. The externally calibrated residual quantity (Δξ) determined above can be added to the Lie algebra coordinates (ξ_prev) to obtain the optimized externally calibrated estimated value (ξ_curr) at the current moment. The externally calibrated estimated value (ξ_curr) obtained above is converted back into a 4x4 rigid body transformation matrix, which is the current external parameter matrix.

[0095] Optionally, during the inference of the above deep neural network, in addition to outputting the externally calibrated residual quantity, the last layer of the regression head can also output an uncertainty estimate value (U) in parallel. The above method can be achieved by adding a branch after the regression head to output the predicted variance or through techniques such as Monte Carlo Dropout.

[0096] Optionally, an uncertainty threshold (U_th) is set. The calculated uncertainty estimate value (U) is compared with the above uncertainty threshold (U_th); if U < U_th, it is determined that the current externally calibrated estimated value is reliable and is normally used for data fusion. If U >= U_th, it is determined that the current estimated value is unreliable, and a degradation processing strategy is triggered. For example, in the sensor fusion algorithm, the weight of the fusion result based on this external calibration is reduced. Discard the result of the external calibration of the sensor for the current frame and continue to use the externally calibrated estimated value of the previous frame. A warning about the degradation of the performance of the external calibration of the sensor can be sent to the driver or the vehicle's back-end system. Through the above method, the external calibration of the vehicle camera and lidar can be continuously optimized in a closed loop, ensuring that even if there are slight changes in the installation position of the sensor, the performance of the external calibration of the sensor can always maintain the best performance.

[0097] Optionally, in addition to regression variance, an ensemble learning method can also be used to train multiple networks and use the output variance as the uncertainty measure.

[0098] For example, the externally calibrated residual quantity output by the deep neural network is fused with the Lie algebra coordinates to obtain the optimized external calibration at the current moment, which is used for the sensor data fusion of the next frame, forming a continuous closed-loop optimization system. In addition, by calculating the uncertainty measure of the features in the middle layer of the deep neural network, the quality of the external calibration of the current vehicle sensor is evaluated. When the uncertainty measure exceeds the preset threshold, it is determined that the current externally calibrated estimated value is unreliable, and a degradation processing mechanism is triggered when the evaluation result is unreliable, including but not limited to reducing the weight of the current externally calibrated estimated value in data fusion; prompting the above closed-loop optimization system to perform degradation processing.

[0099] For another example, the optimized extrinsic parameter estimates described above can be immediately applied, forming a closed loop. The current extrinsic parameter matrix is ​​used to project point cloud data onto an image, providing high-quality fusion data for downstream tasks such as object detection and semantic segmentation. The optimized extrinsic parameter estimates (ξ_curr) are used as the prior values ​​(ξ_prev) for the next frame, allowing continuous tracking of extrinsic parameter changes. Data within a time window {I, D, ξ_prev, ξ_curr} can be stored in a circular buffer. Periodically sampling data from the buffer allows for online fine-tuning of the network, enabling the deep learning network to adapt to the slow aging of the vehicle itself and long-term changes in the external environment.

[0100] In the embodiments of this application, the above method can reflect and correct the dynamic changes in pose between sensors in real time through the current external parameter matrix, providing the vehicle with a continuously updated and high-precision multi-sensor fusion benchmark, which significantly enhances the vehicle's environmental perception capability and decision reliability.

[0101] As an optional implementation, multiple multimodal data are fused using the current extrinsic parameter matrix to obtain fused data, including: projecting point cloud data onto image data using the current extrinsic parameter matrix to obtain fused data.

[0102] In the embodiments of this application, the fused data can be used to characterize a multimodal data set with high spatiotemporal synchronization obtained by precisely aligning point cloud data to the pixel coordinate system of image data in space.

[0103] Optionally, the coordinate system of each point in the point cloud data is transformed according to the current extrinsic parameter matrix; the projection position of each transformed point on the image data, i.e., the corresponding image coordinates, is calculated. The above method can convert three-dimensional coordinates into two-dimensional pixel coordinates using the camera's intrinsic parameters. When the projection of a point falls within the effective pixel range of the image, the corresponding depth value can be calculated, and the depth values ​​of the point and the corresponding pixels in the image can be compared and matched. After depth matching and filtering of the point cloud data, the successfully matched point cloud data is fused with the image data to obtain fused data. The camera intrinsic parameters may include, but are not limited to: focal length, principal point, and distortion coefficients.

[0104] In this embodiment, the above method enables highly spatiotemporally consistent fusion of multimodal sensor data to obtain fused data. By aligning point cloud data with image data, spatiotemporal synchronization between the two is ensured. This allows each point in the point cloud data to find its corresponding position in three-dimensional space within the image, providing a unified spatiotemporal reference for subsequent tasks.

[0105] As an optional implementation, the method further includes: identifying the location information of the obstacle by analyzing the fused data; and determining the vehicle's trajectory in a future time period based on the location information, wherein the future time period is later than the current time period.

[0106] In the embodiments of this application, the aforementioned future time can be used to characterize a certain point in time or time period in the future relative to the current time. The aforementioned movement trajectory can be used to characterize the expected travel path of the vehicle within a predetermined future time period.

[0107] Optionally, based on the obstacle location information contained in the above-mentioned fused data, the traffic environment around the vehicle is perceived and understood; the possible behavior of obstacles in the future time period is predicted; and by combining the above-mentioned obstacle location information and predicted behavior, as well as the vehicle's current position, speed and destination, a driving path is determined based on a path planning algorithm, that is, the vehicle's movement trajectory in the future time period.

[0108] In this embodiment, the above method can acquire data collected by multiple sensors in the vehicle during the current time period to obtain multimodal data. Based on the multimodal data and the historical extrinsic parameter matrix determined during the historical time period, the extrinsic parameter residual is determined. Using the extrinsic parameter residual, the historical extrinsic parameter matrix is ​​transformed to obtain the current extrinsic parameter matrix, which is used to characterize the relative poses of multiple sensors when they are working during the current time period. Using the current extrinsic parameter matrix, the multiple multimodal data are fused to obtain fused data. That is, in this application, by acquiring multimodal data in real time and determining the extrinsic parameter residual based on the historical extrinsic parameter matrix determined during the historical time period, the historical extrinsic parameter matrix is ​​updated. The updated matrix is ​​then used for multisensor data fusion, achieving the goal of real-time and accurate calibration of the relative poses between sensors and outputting high-quality fused perception results. Therefore, it overcomes the problem that traditional calibration methods are easily affected by the aging of vehicle components and the environment in which the vehicle sensors are located, leading to a reduction in the accuracy of multisensor extrinsic parameter calibration. This achieves the technical effect of improving the accuracy of multisensor extrinsic parameter calibration and solves the technical problem of low accuracy in multisensor extrinsic parameter calibration.

[0109] The method described above in this application will be further described below with reference to preferred embodiments.

[0110] In this embodiment, multimodal data is applicable not only to camera-LiDAR combinations but also alternatively to camera-millimeter-wave radar. The point cloud projection map can be replaced with a two-dimensional density map or feature map generated by the millimeter-wave radar.

[0111] Optionally, two types of 3D point cloud data can be processed directly, and the structure of the deep learning network can be adapted to process 3D voxels or point cloud neural networks such as PointNet Plus Plus (PointNet++). No specific restrictions are placed on the structure of the deep learning network being processed here.

[0112] Alternatively, extrinsic parameters between surround-view cameras can be calibrated using a multi-camera system.

[0113] Optionally, for scenarios where there are no high-precision offline calibration results, an initial extrinsic parameter with a large error range can be provided. After a period of operation, the residual learning mechanism and online learning capability of this application can still gradually converge to the accurate value.

[0114] Figure 2 This is a flowchart of an end-to-end online calibration method according to an embodiment of this application. Figure 2 As shown, the end-to-end online calibration method includes the following steps.

[0115] Step S202: Simultaneously acquire image data and point cloud data from multiple sensors.

[0116] Optionally, a timestamp mechanism can be used to ensure that image data and point cloud data collected from different sensors are strictly correlated in time.

[0117] Step S204, data preprocessing (point cloud projection / feature extraction).

[0118] Optionally, point cloud projection technology can be used to map the point cloud data acquired by the vehicle's LiDAR onto the two-dimensional coordinate system of the image data, generating a depth map (i.e., planar image data) to facilitate subsequent fusion of the image and point cloud data. Feature extraction can be performed separately on the image data and the point cloud data. For example, a convolutional neural network can be used to extract features from the image data. Optionally, in addition to a convolutional neural network, a Vision Transformer (ViT) can also be used as the backbone network for image feature extraction.

[0119] Step S206, Model Inference.

[0120] Optionally, the preprocessed image features, depth map features (or point cloud features), and the previous time step's extrinsic parameter estimates are input into the deep neural network model for forward propagation to obtain the prediction residuals of the extrinsic parameters at the current time step.

[0121] Step S208: Determine the residual amount of the external parameters.

[0122] Optionally, the output of deep neural network inference can be directly interpreted as the change in extrinsic parameters at the current time relative to the previous time, i.e., the extrinsic parameter residual, including small adjustments for rotation and translation.

[0123] Optionally, after the deep neural network acquires image data, planar image data, and historical extrinsic parameter matrices, it can perform convolution processing on the planar image data to obtain key vectors and value vectors, and convolution processing on the image data to obtain query vectors. The key vectors are then used to transform the query vectors to obtain an attention correlation matrix. The attention correlation matrix and the value vectors are fused to obtain fused features. The historical extrinsic parameter matrix is ​​transformed to obtain Lie algebra coordinates. The Lie algebra coordinates are encoded to obtain extrinsic parameter embedding vectors. These extrinsic parameter embedding vectors are expanded to obtain prior information features. The multimodal fused features and prior information features are concatenated to obtain concatenated features. The concatenated features are decoded to obtain a feature image. The feature image is then transformed to obtain the extrinsic parameter residuals.

[0124] Step S210: External parameter fusion and update.

[0125] Alternatively, complementary filtering or a Lie group-based update algorithm can be used to combine the residual predicted by the deep neural network with the extrinsic parameter estimates from the previous time step to obtain the updated extrinsic parameter values ​​for the current time step.

[0126] Step S212: Update the estimated external parameters.

[0127] Optionally, after the fusion calculation, the optimized extrinsic parameter estimates (i.e., the current extrinsic parameter matrix) can be saved to the system's state memory as prior information for the next time step.

[0128] Step S214: Data fusion based on the new extrinsic parameters.

[0129] Optionally, the acquired image data and point cloud data are spatiotemporally aligned using the updated extrinsic parameter matrix (i.e., the current extrinsic parameter matrix) and then fused together to generate multimodal fused sensing data.

[0130] Step S216: Output the fusion perception result.

[0131] Optionally, the fused data can be further processed through perception algorithms such as object detection and semantic segmentation to produce specific perception information such as obstacle recognition and road boundary extraction, which can then be provided to the autonomous driving decision-making system.

[0132] Step S218, parameter prior update.

[0133] Optionally, the determined extrinsic residuals are added to the Lie algebra coordinates to obtain the optimized extrinsic estimate (prior parameters) at the current time.

[0134] Step S220: Determine the estimated historical external parameters.

[0135] Optionally, a fixed-length historical extrinsic parameter queue can be maintained. Whenever a new extrinsic parameter estimate is generated, the oldest estimate in the queue is removed, and the new value is added to the end of the queue. Historical extrinsic parameter estimates can be determined using this historical extrinsic parameter queue.

[0136] Figure 3 This is a flowchart illustrating the construction of an end-to-end deep neural network model according to an embodiment of this application. Figure 3 As shown, constructing an end-to-end deep neural network model includes the following steps.

[0137] Step S302, Image input.

[0138] Optionally, the image input can be preprocessed, such as normalized or resized, to suit the needs of the model and improve the efficiency and accuracy of feature extraction.

[0139] Step S304, point cloud input.

[0140] Optionally, the point cloud data can be preprocessed, such as downsampling, voxel meshing, or direct point cloud encoding, to reduce computational complexity and prepare it for input into the neural network.

[0141] Step S306: Determine the historical extrinsic parameters prior.

[0142] Optionally, one or more historical extrinsic parameter estimates can be selected as prior information to provide a preliminary estimate of the model's extrinsic parameters at the current time.

[0143] Step S308, Image Feature Extraction Branch.

[0144] Optionally, a convolutional neural network or other visual feature extraction network can be used to extract features from the input image to obtain an image feature vector.

[0145] Step S310, point cloud feature extraction branch.

[0146] Optionally, a three-dimensional convolutional neural network or other specialized point cloud feature extraction network can be used to process the input point cloud data and extract point cloud feature vectors.

[0147] Step S312, prior encoder.

[0148] Optionally, historical extrinsic information can be encoded into a compact feature vector through a small subnetwork consisting of one or more fully connected layers, which facilitates fusion with image and point cloud features.

[0149] Step S314, Multimodal feature fusion module.

[0150] Optionally, attention mechanisms or other adaptive fusion strategies can be used to combine image features, point cloud features, and prior extrinsic feature vectors to generate a fusion feature vector that integrates multimodal information.

[0151] Step S316: Determine the prior feature vector.

[0152] Optionally, before feature fusion, the feature vector output by the prior encoder can be optimized through normalization, dimensionality reduction, etc., to improve the quality of fusion.

[0153] Step S318: Determine the fusion feature vector.

[0154] Optionally, after the multimodal feature fusion is completed, the fused feature vector can be further processed and adjusted through a fully connected layer or other layers.

[0155] Step S320, Residual Regression Head Fully Connected Layer.

[0156] Alternatively, a residual regression network consisting of one or more fully connected layers can be used to directly predict the 6-DoF extrinsic residuals of the current time relative to the historical prior from the fused feature vectors.

[0157] Step S322: Determine the 6-DoF extrinsic residuals ΔRx, ΔRy, ΔRz, ΔTx, ΔTY, and ΔTz.

[0158] Optionally, the residuals of the model output can be decomposed into three rotational components (ΔRx, ΔRy, ΔRz) and three translational components (ΔTx, ΔTy, ΔTz). These residuals will be used for the next step of extrinsic parameter update to improve the relative positioning between sensors.

[0159] Figure 4 This is a flowchart illustrating the application of an attention mechanism in multimodal feature fusion according to an embodiment of this application. Figure 4 As shown, applying an attention mechanism in multimodal feature fusion can include the following steps.

[0160] Step S402: Determine the image feature map.

[0161] Alternatively, deep features can be extracted from the input RGB image using a pre-trained image feature extraction network, such as a residual network, an efficient network, or a visual transformer.

[0162] Step S404: Determine the point cloud feature map.

[0163] Alternatively, the point cloud data can be obtained from the LiDAR point cloud using PointNet++ or a similar network. The LiDAR point cloud can first be converted into a voxel grid or bird's-eye view representation before being input into the aforementioned network for feature extraction.

[0164] Optionally, after determining the image feature map and the point cloud feature map, the image feature map and the point cloud feature map can be input into the core of the attention mechanism.

[0165] Step S406, Query: Image features.

[0166] Optionally, the query vector can be a local or global representation of image features, which can be generated from the image feature map through additional fully connected or convolutional layers.

[0167] Step S408, Key: Point cloud features.

[0168] Optionally, the key vector reflects the importance distribution of the point cloud features, and key values ​​with a higher matching degree to the query vector will receive higher attention weights. The key of the point cloud can be generated from the point cloud feature map using the same or similar network structure to ensure consistency between modalities.

[0169] Step S410, attention weight features.

[0170] Optionally, the attention weight matrix can be obtained by calculating the dot product or inner product of the Query and Key and applying a soft-maximum function. Each element of the attention weight matrix represents the association strength between a location in the image feature map and all locations in the point cloud feature map.

[0171] Step S412, Value: Point cloud features.

[0172] Optionally, Value is the original feature of the point cloud feature or the transformed feature, which can be weighted and summed according to attention weights to generate weighted point cloud features.

[0173] Step S414: Weighted summation.

[0174] Optionally, the Value vector is multiplied by the attention weight matrix obtained in step S410, and then the results are summed to obtain a point cloud weighted feature that integrates the attention of image features.

[0175] Step S416, weighted point cloud features.

[0176] Optionally, the result of the weighted summation is the weighted point cloud feature. The above result can preserve the original information of the point cloud, while incorporating the attention of image features to important regions in the point cloud.

[0177] Step S418, skip connection.

[0178] Alternatively, skip connections can be performed directly after the image feature map is determined.

[0179] Optionally, during feature fusion, skip connections can be used to directly pass image features to deeper layers of the network or the final output, avoiding feature loss or over-abstraction during the transfer process and preserving more detailed image information. Skip connections can include residual blocks or simple pointwise addition.

[0180] Step S420, feature splicing.

[0181] Optionally, the weighted point cloud features can be concatenated with the image features obtained through skip connections to form a multimodal feature fusion. Feature concatenation can merge the representations of the two features into a wider feature map along the channel dimension.

[0182] Step S422: Determine the fused feature map.

[0183] Optionally, the fused feature maps will be further processed by subsequent decoders or fully connected layers to integrate multimodal information and generate the final output, such as residual estimation of extrinsic parameters.

[0184] Figure 5 This is a flowchart illustrating a calibration quality assessment and degradation process according to an embodiment of this application. Figure 5 As shown, the calibration quality assessment and downgrade process includes the following steps.

[0185] Step S502, end-to-end deep learning calibration engine.

[0186] Optionally, the deep learning calibration engine outputs the estimated values ​​of the extrinsic parameters at the current moment, as well as the accompanying uncertainty estimates.

[0187] Step S504: Extract the intermediate layer feature statistics.

[0188] Optionally, feature statistics such as variance, entropy, or confidence intervals can be extracted from the intermediate layers of the calibration engine network. These statistics can reflect the reliability of the network's predictions and the complexity of the environment.

[0189] Step S506: Calculate feature uncertainty based on variance / entropy / confidence.

[0190] Optionally, a quantitative measure of uncertainty can be calculated using the extracted feature statistics, which can be achieved through statistical analysis or machine learning methods, to quantify the confidence level of the current extrinsic parameter estimates.

[0191] Step S508, uncertainty assessment, determine whether the uncertainty is less than the threshold.

[0192] Optionally, the calculated uncertainty measure is compared with a pre-set threshold U_threshold to determine the subsequent processing strategy. If Uncertainty < U_threshold, the current external parameter estimation is considered reliable; otherwise, it is regarded as unreliable.

[0193] Optionally, if Uncertainty is less than the threshold, step S510 is executed; if Uncertainty is equal to the threshold, step S514 is executed; if Uncertainty is greater than the threshold, step S518 is executed.

[0194] Step S510: Normally use the calibration parameters with high weights to participate in the fusion.

[0195] Optionally, if Uncertainty is less than the threshold, the current external parameter estimation has a high degree of credibility. Therefore, when performing multi-sensor data fusion, a higher weight can be assigned to make the most of the latest and most accurate alignment information.

[0196] Step S512: Continue with high-precision fusion perception.

[0197] Optionally, when the reliability of the external parameter estimation is high, data fusion and perception calculations can be performed normally to obtain high-precision environmental perception results.

[0198] Step S514: Reduce the weight of the current parameters and increase the weight of the historical parameters.

[0199] Optionally, if Uncertainty is equal to the threshold, although the uncertainty is at the critical level, the current parameters can still be used, but with a slightly lower weight. At the same time, the weight of the historical parameters is increased to maintain the stability of the system and the accuracy of data fusion.

[0200] Step S516: Continue with stable fusion perception.

[0201] Optionally, after weighing the external parameter estimations at the current moment and the historical moment, a stable fusion strategy is selected to ensure that a certain perception ability can be maintained even under conditions of high uncertainty.

[0202] Step S518: Execute a hierarchical degradation processing strategy.

[0203] Optionally, if Uncertainty is greater than the threshold, that is, when the uncertainty predicted by the network exceeds the safety threshold, a degradation processing strategy can be initiated to prevent inaccurate external parameter estimations from posing risks to the entire autonomous driving system.

[0204] Optionally, the above-mentioned graded degradation processing strategy includes: parameter-level degradation using historical moving average parameters; fusion-level degradation reducing the fusion weight of the sensor; and system-level degradation, prompting system alarms / driver takeover. Specifically, parameter-level degradation using historical moving average parameters, at the mildest degradation level, can ignore the current high-uncertainty extrinsic parameters and replace them with historical moving average extrinsic parameters, achieving the goal of not interrupting the perception process while ensuring a certain degree of robustness of the fused data; fusion-level degradation reducing the fusion weight of the sensor, in cases of severe uncertainty, reduces the weight of the current sensor data in the fusion process to reduce the contribution of inaccurate data to the entire perception system; and system-level degradation, prompting system alarms / driver takeover, when uncertainty reaches the highest level, should trigger a system-level alarm mechanism to notify the driver or remote monitoring center, and if necessary, switch to a backup sensor configuration or request manual intervention to ensure driving safety.

[0205] Step S520 ensures basic safe operation.

[0206] Optionally, while performing the downgrade process, measures can be taken to ensure the basic safe operation of the vehicle, such as switching to a low-power mode or restricting the use of certain advanced driver assistance functions.

[0207] Step S522, the system runs normally.

[0208] Optionally, the system can recover the reliability of the extrinsic parameter estimation from subsequent data, return to normal operation, and continue to perform high-precision sensing and control tasks.

[0209] In this embodiment, data collected by multiple sensors in the vehicle during the current time period is acquired to obtain multimodal data; based on the multimodal data and the historical extrinsic parameter matrix determined during historical time periods, the extrinsic parameter residual is determined; the historical extrinsic parameter matrix is ​​transformed using the extrinsic parameter residual to obtain the current extrinsic parameter matrix, which is used to characterize the relative poses of multiple sensors when they are working during the current time period; the multiple multimodal data are fused using the current extrinsic parameter matrix to obtain fused data. In other words, in this embodiment, by collecting multimodal data in real time and determining the extrinsic parameter residuals based on the historical extrinsic parameter matrix determined within a historical time period, the historical extrinsic parameter matrix is ​​transformed based on the extrinsic parameter residuals to obtain the current extrinsic parameter matrix. The updated current extrinsic parameter matrix is ​​then used for multi-sensor data fusion, achieving the goal of real-time and accurate calibration of the relative poses between sensors and outputting high-quality fusion perception results. Therefore, it overcomes the problem that traditional calibration methods are easily affected by the aging of vehicle components and the environment in which the vehicle sensors are located, which leads to a reduction in the accuracy of multi-sensor extrinsic parameter calibration. This achieves the technical effect of improving the accuracy of multi-sensor extrinsic parameter calibration and solves the technical problem of low accuracy in multi-sensor extrinsic parameter calibration.

[0210] According to an embodiment of this application, an embodiment of a processor load rate determination device is also provided. It should be noted that the device can be used to run the above-described processor load rate determination method.

[0211] Figure 6 This is a schematic diagram of an information processing apparatus according to an embodiment of this application. Figure 6 As shown, the information processing device may include: an acquisition unit 602, a determination unit 604, a conversion unit 606, and a fusion unit 608.

[0212] The acquisition unit 602 is used to acquire data collected by multiple sensors in the vehicle during the current time period to obtain multimodal data.

[0213] The determining unit 604 is used to determine the extrinsic parameter residual based on multimodal data and the historical extrinsic parameter matrix determined within the historical time period. The historical extrinsic parameter matrix is ​​used to characterize the relative poses of multiple sensors when they are working within the historical time period, and the extrinsic parameter residual is used to characterize the degree of deviation between the historical extrinsic parameter matrix and the current extrinsic parameter matrix to be determined.

[0214] The conversion unit 606 is used to convert the historical external parameter matrix using the external parameter residual to obtain the current external parameter matrix. The current external parameter matrix is ​​used to characterize the relative poses of multiple sensors when they are working in the current time period.

[0215] The fusion unit 608 is used to fuse multiple multimodal data using the current external parameter matrix to obtain fused data, wherein the fused data is used to characterize the location information of at least one obstacle in the area where the vehicle is located.

[0216] In the processor load rate determination device of this embodiment, the acquisition unit acquires data collected by multiple sensors in the vehicle during the current time period to obtain multimodal data; the determination unit determines the extrinsic parameter residual based on the multimodal data and the historical extrinsic parameter matrix determined during the historical time period, wherein the historical extrinsic parameter matrix is ​​used to characterize the relative poses of the multiple sensors when they were working during the historical time period, and the extrinsic parameter residual is used to characterize the degree of deviation between the historical extrinsic parameter matrix and the current extrinsic parameter matrix to be determined; the conversion unit uses the extrinsic parameter residual to convert the historical extrinsic parameter matrix to obtain the current extrinsic parameter matrix, which is used to characterize the relative poses of the multiple sensors when they were working during the current time period; the fusion unit uses the current extrinsic parameter matrix to fuse the multiple multimodal data to obtain fused data, wherein the fused data is used to characterize the position information of at least one obstacle in the area where the vehicle is located, thereby achieving the technical effect of improving the accuracy of multi-sensor extrinsic parameter calibration and solving the technical problem of low accuracy of multi-sensor extrinsic parameter calibration.

[0217] Embodiments of this application also provide a vehicle, including: a memory storing an executable program; and a processor for running the program, wherein the program executes the methods described in various embodiments of this application.

[0218] Embodiments of this application also provide a computer-readable storage medium including a stored executable program, wherein, when the executable program is running, it controls the device where the computer-readable storage medium is located to perform the methods of various embodiments of this application.

[0219] Embodiments of this application also provide a computer program product, including a computer program that implements the methods of various embodiments of this application when run by a processor.

[0220] Embodiments of this application also provide a computer program product, including a non-volatile computer-readable storage medium for storing a computer program, which is executed by a processor to implement the methods in various embodiments of this application.

[0221] Embodiments of this application also provide a computer program that, when run by a processor, implements the methods described in the various embodiments of this application.

[0222] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0223] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units can be a logical functional division, and in actual implementation, there may be other division methods. For instance, multiple units or components may be combined or integrated into another system, or some features may be ignored or not operational. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection of units or modules may be electrical or other forms.

[0224] The units described as separate components may or may not be physically separate. Similarly, the components shown as units may or may not be physical units; they may be located in one place or distributed across multiple units. Some or all of the units can be selected to achieve the purpose of this embodiment, depending on actual needs.

[0225] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0226] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to run all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard drive, magnetic disk, or optical disk.

[0227] The above are merely preferred embodiments of this application. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A data processing method, characterized in that, include: Data collected by multiple sensors in the vehicle during the current time period is obtained to obtain multimodal data; Based on the multimodal data and the historical extrinsic parameter matrix determined within the historical time period, the extrinsic parameter residual is determined. The historical extrinsic parameter matrix is ​​used to characterize the relative poses of the multiple sensors when they are working within the historical time period, and the extrinsic parameter residual is used to characterize the degree of deviation between the historical extrinsic parameter matrix and the current extrinsic parameter matrix to be determined. The historical extrinsic parameter matrix is ​​transformed using the extrinsic parameter residual to obtain the current extrinsic parameter matrix, which is used to characterize the relative poses of the multiple sensors when they are working in the current time period. Using the current external parameter matrix, multiple multimodal data are fused to obtain fused data, wherein the fused data is used to characterize the location information of at least one obstacle in the area where the vehicle is located.

2. The method according to claim 1, characterized in that, The multimodal data includes point cloud data acquired by radar sensors and image data acquired by image sensors. The determination of the extrinsic parameter residuals based on the multimodal data and the historical extrinsic parameter matrix determined within historical time periods includes: The point cloud data is projected into planar image data using the historical extrinsic parameter matrix. The residual value of the extrinsic parameter is determined based on the planar image data, the image data, and the historical extrinsic parameter matrix.

3. The method according to claim 2, characterized in that, The determination of the extrinsic residual based on the planar image data, the image data, and the historical extrinsic parameter matrix includes: The planar image data is convolved to obtain a key vector and a value vector, and the image data is convolved to obtain a query vector. The key vector is used to represent the spatial feature distribution of the planar image data, the value vector is used to represent the pixel value information of the planar image data, and the query vector is used to represent the contextual relevance of the image data. The query vector is transformed using the key vector to obtain an attention association matrix, wherein the attention association matrix is ​​used to characterize the association strength between spatial location points in the image data and spatial location points in the planar image data; The attention correlation matrix and the value vector are fused to obtain fused features; Based on the fusion features and the historical extrinsic parameter matrix, the extrinsic parameter residual is determined.

4. The method according to claim 3, characterized in that, The determination of the extrinsic residual based on the fusion features and the historical extrinsic parameter matrix includes: The historical extrinsic parameter matrix is ​​transformed to obtain Lie algebraic coordinates, wherein the Lie algebraic coordinates are used to characterize the rigid body transformation state of the historical extrinsic parameter matrix; Based on the fusion features and the Lie algebra coordinates, the extrinsic parameter residual is determined.

5. The method according to claim 4, characterized in that, The determination of the extrinsic parameter residual based on the fusion features and the Lie algebra coordinates includes: The Lie algebra coordinates are encoded to obtain the extrinsic parameter embedding vector; The extrinsic parameter embedding vector is expanded to obtain prior information features; The multimodal fusion features and the prior information features are concatenated to obtain concatenated features; The spliced ​​features are decoded to obtain a feature image; The feature image is transformed to obtain the extrinsic parameter residual.

6. The method according to any one of claims 2 to 5, characterized in that, The step of transforming the historical extrinsic parameter matrix using the extrinsic parameter residuals to obtain the current extrinsic parameter matrix includes: The current external parameter matrix is ​​obtained by summing the external parameter residuals and the historical external parameter matrix.

7. The method according to any one of claims 2 to 5, characterized in that, The process of fusing multiple multimodal data using the current external parameter matrix to obtain fused data includes: The point cloud data is projected onto the image data using the current external parameter matrix to obtain the fused data.

8. The method according to any one of claims 1 to 5, characterized in that, The method further includes: The location information of the obstacle is obtained by identifying the fused data; Based on the location information, the vehicle's movement trajectory in a future time period is determined, wherein the future time period is later than the current time period.

9. An information processing apparatus, characterized in that, include: The acquisition unit is used to acquire data collected by multiple sensors in the vehicle during the current time period to obtain multimodal data; The determining unit is used to determine the extrinsic parameter residual based on the multimodal data and the historical extrinsic parameter matrix determined within the historical time period, wherein the historical extrinsic parameter matrix is ​​used to characterize the relative poses of the multiple sensors when they are working within the historical time period, and the extrinsic parameter residual is used to characterize the degree of deviation between the historical extrinsic parameter matrix and the current extrinsic parameter matrix to be determined. The conversion unit is used to convert the historical extrinsic parameter matrix using the extrinsic parameter residual to obtain the current extrinsic parameter matrix, which is used to characterize the relative poses of the multiple sensors when they are working in the current time period. The fusion unit is used to fuse multiple multimodal data using the current external parameter matrix to obtain fused data, wherein the fused data is used to characterize the location information of at least one obstacle in the area where the vehicle is located.

10. A vehicle, characterized in that, include: Memory, which stores executable programs; A processor for running the program, wherein the program, when running, performs the method according to any one of claims 1 to 8.

Images