Intelligent driving data acquisition method and device and automobile
The joint calibration of lidar and camera sensors using motion compensation and graph neural networks addresses alignment challenges, providing precise and reliable data alignment for intelligent driving systems.
Patent Information
- Application Number
- CN202411234789.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-04
- Publication Date
- 2024-12-06
AI Technical Summary
Existing intelligent driving data collection methods face challenges in accurately aligning lidar and camera sensors due to calibration errors and hardware delays, leading to misaligned depth features and inconsistent data fusion, which affects the accuracy and robustness of perception systems.
A method involving joint calibration of lidar and camera sensors using motion compensation and a pre-trained graph neural network to achieve time and spatial alignment of point cloud and image data, ensuring precise alignment in a unified coordinate system.
Ensures accurate and reliable raw data alignment for intelligent driving systems, laying a solid foundation for subsequent model training and evaluation tasks by correcting time and space misalignments.
Smart Images

Figure CN119091280A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of intelligent driving technology, and in particular to an intelligent driving data collection method, device and automobile. Background Art
[0002] Intelligent driving data collection refers to the process of collecting various driving-related data through sensors and other technical means during the development, testing and optimization of intelligent driving (such as self-driving cars) systems. These data are used to train, test, verify and improve autonomous driving algorithms and models to enhance the driving capabilities and safety of vehicles.
[0003] At present, the key to intelligent driving data collection is how to collect data accurately, efficiently and safely. Existing intelligent driving data collection methods usually rely on manually defined scenarios and rules to label data and transmit data through the CAN bus. This method is inefficient.
[0004] For example, in the field of autonomous driving, integrating lidar and camera information into a bird's-eye view (i.e., Bird's-Eye-View perception map, generally abbreviated as BEV) representation has become an important direction for the development of perception technology. However, a major challenge currently exists is the accuracy of the calibration parameters of lidar and camera sensors. The inherent complexity in real-world scenarios, where most projection matrices between lidar and cameras are manually calibrated, and factors such as road vibration exacerbate this error. These inherent errors cannot be corrected by online calibration, which is a major challenge. This calibration inevitably introduces projection errors, resulting in depth misalignment (the depth of surrounding neighbors is projected as the depth of pixels). This depth misalignment leads to inaccurate depth features, which in turn leads to misalignment during the conversion of multiple views to BEV representation. In order to be able to align the lidar and camera information, that is, to ensure that the data collected by different sensors can accurately correspond to a unified time and space coordinate system, avoid the problem of spatiotemporal misalignment of data. In terms of time alignment, the current practice is generally to trigger the camera to expose once when the radar passes through the center of the camera, so as to achieve time synchronization. However, this will cause some problems. Although the radar triggering the camera can achieve time consistency, it is actually difficult to achieve very accurate time alignment due to the delay and accuracy limitations of the hardware equipment. Slight delays or inconsistencies may cause time misalignment of the data, which in turn affects subsequent data fusion and processing.
[0005] Therefore, how to adjust the features generated by both lidar and camera sensors so that they are aligned in the BEV space to improve the accuracy and robustness of the perception system is a technical problem currently faced by intelligent driving data collection. Summary of the invention
[0006] In view of this, the embodiments of the present application provide a method, device and automobile for collecting intelligent driving data to solve the technical problem of how to realize intelligent driving data collection in the prior art.
[0007] According to a first aspect of an embodiment of the present application, a method for collecting intelligent driving data is provided, which includes: respectively acquiring data collected by at least two perception sensors, the data including point cloud data collected by a lidar and image data collected by a camera; jointly calibrating the at least two perception sensors, the joint calibration including: temporally aligning the point cloud data and image data using motion compensation, and inputting the point cloud data and image data into a pre-trained graph neural network, and obtaining spatially aligned point cloud data and image data at the output of the graph neural network; uploading the jointly calibrated point cloud data and image data to a vehicle cloud server.
[0008] According to a second aspect of an embodiment of the present application, there is provided an intelligent driving data acquisition device, comprising: a data acquisition module, configured to respectively acquire data collected by at least two perception sensors, the data comprising point cloud data collected by a lidar and image data collected by a camera; a data calibration module, configured to jointly calibrate the at least two perception sensors, the joint calibration comprising: temporally aligning the point cloud data and image data using motion compensation, and inputting the point cloud data and image data into a pre-trained graph neural network, and obtaining spatially aligned point cloud data and image data at the output of the graph neural network; and a data feedback module, configured to upload the jointly calibrated point cloud data and image data to a vehicle cloud server.
[0009] According to a third aspect of an embodiment of the present application, a car is provided, comprising an intelligent driving data acquisition system, wherein the intelligent driving system comprises a laser radar, a camera and a control device, wherein the control device comprises at least a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above method when executing the computer program.
[0010] Compared with the prior art, the embodiments of the present application have the following beneficial effects: the intelligent driving data acquisition method obtains data collected by at least two perception sensors respectively, wherein the data include point cloud data collected by a lidar and image data collected by a camera; the at least two perception sensors are jointly calibrated, and the joint calibration includes: time-aligning the point cloud data and image data using motion compensation, and inputting the point cloud data and image data into a pre-trained graph neural network, and obtaining spatially aligned point cloud data and image data at the output of the graph neural network; uploading the jointly calibrated point cloud data and image data to a vehicle cloud server to collect intelligent driving data fused by a camera and a lidar, and since the motion-compensated time synchronization and the graph neural network-based spatial synchronization of the data collected by the two perception sensors are performed, the precise alignment of multi-sensor data is ensured, thereby providing accurate and reliable raw data, laying a solid foundation for subsequent model training and evaluation tasks. BRIEF DESCRIPTION OF THE DRAWINGS
[0011] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.
[0012] Figure 1 It is a flowchart of a smart driving data collection method provided in an embodiment of the present application;
[0013] Figure 2 It is a structural schematic diagram of an intelligent driving data collection device provided in an embodiment of the present application;
[0014] Figure 3 This is an internal architecture diagram of a car provided by an embodiment of the present application;
[0015] Figure 4 It is a schematic diagram of the architecture of the intelligent driving collection system provided in the embodiment of the present application;
[0016] Figure 5 This is a structural diagram of a spatial alignment algorithm based on a graph neural network provided in an embodiment of the present application;
[0017] Figure 6 It is a structural schematic diagram of a control device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0018] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.
[0019] See also Figure 1 In a first embodiment of the present application, a method for collecting intelligent driving data is provided, comprising the steps of:
[0020] S1: acquiring data collected by at least two perception sensors respectively, the data including point cloud data collected by a laser radar and image data collected by a camera;
[0021] S2: Jointly calibrate at least two perception sensors, the joint calibration comprising: temporally aligning the point cloud data and the image data using a motion compensation method, inputting the point cloud data and the image data into a pre-trained graph neural network, and obtaining spatially aligned point cloud data and image data at the output of the graph neural network;
[0022] S3: Upload the jointly calibrated point cloud data and image data to the vehicle cloud server.
[0023] The working principle of the above-mentioned intelligent driving data collection method is: after collecting multiple required sensor data, motion compensation is used to consider the collection time of each sensor data and the motion state of the vehicle, and the lidar and camera data are time-calibrated to ensure that they correspond to the same time point, thereby avoiding time inconsistency; and a pre-trained graph neural network is used to introduce deep features of image neighborhood perception in a graph matching manner, thereby correcting local errors, aligning the BEV features generated by the lidar and camera in the three-dimensional BEV space, and ensuring that the data collected by the lidar and camera sensors can accurately correspond to a unified time and space coordinate system, avoiding time and space dislocation of the data.
[0024] Cameras and laser radar (LiDAR) are very critical perception sensors in intelligent driving (also referred to as smart driving) systems. Cameras are mainly used to capture visual information around the vehicle, and LiDAR generates high-precision three-dimensional point cloud data by emitting lasers and receiving reflected signals. In intelligent driving, cameras and LiDAR are often used in combination. LiDAR is relatively weak in object classification and recognition, while cameras are insufficient in accurate ranging and three-dimensional modeling. Combining the camera's two-dimensional image data with the LiDAR's three-dimensional point cloud data can compensate for each other and generate more comprehensive environmental perception information, thereby improving the accuracy and reliability of vehicle perception.
[0025] This embodiment ensures the precise alignment of multi-sensor data through motion-compensated time synchronization and graph neural network-based spatial synchronization, provides accurate and reliable raw data, and lays a solid foundation for subsequent model training and evaluation tasks.
[0026] In an optional embodiment, the point cloud data and the image data are time-aligned using a motion compensation method, including: recording the timestamps of the point cloud data and the image data; obtaining the vehicle motion state at the time of collecting the point cloud data and the image data, the vehicle motion state including the average speed of the vehicle; determining the time offset according to the displacement of the vehicle between the two timestamps and the average speed of the vehicle; when the time offset is not zero, using the time offset to update the timestamp of the point cloud data or the image data, so as to time-align the point cloud data with the image data.
[0027] Specifically, the timestamps of the lidar and camera data are recorded as t LiDAR and t Camera , the vehicle motion state at the two sensor data collection moments is obtained through IMU (Inertial Measurement Unit) and wheel speed sensors, including but not limited to vehicle speed v and vehicle acceleration a. According to the vehicle's motion state, the time offset is calculated to calibrate the time of the lidar and camera data. For example, the displacement d of the vehicle between two timestamps is known, and the time offset Δt=dv is estimated, where d is the displacement of the vehicle (this is known, including but not limited to calculation through the vehicle's position data), and v is the average speed of the vehicle. Therefore, the calculated time offset Δt is applied to adjust the timestamps of the camera or lidar data to ensure that they correspond to the same time point. Assume that for the camera data, if calibration is required, the new timestamp is: t Camera,corrected =t Camera +Δt, ensure that the timestamp after synchronization satisfies: t LiDAR =t Camera,corrected .
[0028] The embodiment of the present application performs time calibration on the lidar and camera data by considering the collection time of the sensor data and the motion state of the vehicle to ensure that they correspond to the same time point, thereby avoiding time inconsistency.
[0029] In an optional embodiment, the point cloud data and image data are input into a pre-trained graph neural network, and spatially aligned point cloud data and image data are obtained at the output of the graph neural network, including: using a pre-trained SECOND network model and a Swin-Transformer model as encoders to convert the point cloud data into point cloud BEV features and the image data into camera BEV features, respectively; flattening the point cloud BEV features and performing feature fusion after deep graph convolution on the camera BEV features, wherein the feature fusion includes three-dimensional spatial alignment of the point cloud BEV features and the camera BEV features; inputting the fused features into a decoder, and obtaining spatially aligned point cloud data and image data at the output of the decoder.
[0030] Specifically, SECOND (sparsely embedded convolutional detection) is a deep learning network architecture based on sparse convolution for processing and parsing LiDAR point cloud data. Swin-Transformer is a visual Transformer model based on the self-attention mechanism, which captures long-range dependencies in images through the self-attention mechanism. In this embodiment, input is obtained from different perception sensors (including lidar and camera), Swin-Transformer is used as the camera encoder, and Second is used as the lidar encoder to extract the characteristics of the data collected by the lidar and camera respectively.
[0031] Among them, the Swin-Transformer model acts as an encoder to convert the image data collected by the camera into camera BEV features, and then performs depth map convolution on these camera BEV features. The depth map convolution can effectively associate information in the depth image, that is, through the convolution operation, the depth information is viewed in the local area, and the depth features of adjacent pixels are integrated and processed. At the same time, this helps to capture the spatial correlation between depth information at the pixel level. Through the depth map convolution, the feature expression ability of the depth image can be enhanced. Traditional RGB image convolution mainly focuses on color and texture features, while the depth map convolution is more focused on the propagation and integration of depth information, so as to better reflect the spatial layout and distance relationship of objects in the scene, that is, to obtain the height information of the image in three-dimensional space, so as to overcome the local misalignment caused by the projection error between LiDAR and camera in the camera-to-BEV process in previous BEV-based methods.
[0032] At the same time, the SECOND network model acts as an encoder to convert point cloud data into point cloud BEV features. The process of feature flattening the point cloud features extracted by the encoder includes: compressing the point cloud BEV features along the axis, representing the 3D features as 2D LiDAR BEV features, and finally using a learnable offset to achieve global multimodal BEV feature alignment, that is, feature fusion of the camera BEV features after depth map convolution and the point cloud BEV features after feature flattening.
[0033] In practical applications, combined with the above-mentioned Swin-Transformer model and SECOND network architecture, a global alignment model can be introduced through pre-training to perform spatial alignment. The global alignment model uses the neighborhood-aware deep features through graph matching to achieve the correction of local errors, aiming to correct the misalignment problem between the BEV features generated by the lidar and the camera. It should be noted that the supervisory information used in the training of the global alignment model is not limited to the labels of the training samples used, but also includes some supervisory information derived from the features obtained by the fusion convolution of the LiDAR and camera BEV features. It can be seen that this feature fusion after deep graph convolution and feature flattening of the image features extracted by the Swin-Transformer model and the point cloud features extracted by the SECOND network architecture can derive new supervisory information and participate in the training of the global alignment model, thereby improving the effect of the model in aligning the camera and radar space. Preferably, in the training process of the global alignment model, global offset noise is introduced, and a learnable offset is used to improve the model accuracy, thereby effectively adjusting the features generated by the two sensors to ensure that they are aligned in the BEV space, and finally achieving high-precision camera and radar space alignment.
[0034] In addition, data collection for intelligent driving systems covers multiple key elements, including time, weather conditions, visibility, urban environment, road grade, road conditions, and traffic flow. For example, describing "a car driving on a highway in the evening on a rainy day" is a specific scenario example. These labels can be further subdivided as needed, such as classifying rainy days as heavy rain, moderate rain, or light rain levels. In terms of data screening, rule-based methods require frequent and flexible configuration of rules. Over time, the value of data filtered out using specific rules may decrease, so it becomes increasingly challenging to continuously adjust the rules. In the long run, an effective way to collect data is to train a model in the cloud that can accurately evaluate data that is beneficial to the current autonomous driving system. Then deploy a simplified version of the model on the vehicle to filter the data that needs to be uploaded to the cloud. This approach can optimize the collection and utilization of data, ensuring that the system can effectively learn and improve the accuracy and safety of driving decisions.
[0035] In an optional embodiment, after the joint calibration of at least two perception sensors, the steps are also included: using model failure analysis and model decision boundary analysis to pre-set the scenarios and acquisition logic to be collected, and setting a corresponding trigger mechanism on the vehicle-mounted end; based on the trigger mechanism, the calibrated point cloud data and image data are monitored, and when a pre-set scenario is monitored, the data set under the corresponding acquisition scenario is automatically acquired based on the pre-set acquisition logic.
[0036] This embodiment introduces an efficient and automated scenario extraction algorithm based on model failure analysis and model decision boundary analysis, so that data collection can be automatically screened according to set scenarios, thereby significantly improving the efficiency and accuracy of data collection.
[0037] First, the purpose of model failure analysis is to identify the circumstances under which the model cannot correctly predict or classify, that is, the failure point of the model. By monitoring the difference between the model's prediction results and the actual labels, detecting instances of model failure, statistically analyzing the failure cases, finding out the mode and cause of model failure, and determining the boundary conditions of model failure. For example, specific input feature values or environmental conditions. Based on failure analysis, optimize the data collection logic to collect more data on model failure for further training and model improvement. Model failure can be evaluated using accuracy and recall.
[0038] Secondly, in classification problems, the model decision boundary is the dividing line that distinguishes different categories. Analyze the probability distribution of the model output and identify data points near the decision boundary. Calculate the distance from the data point to the decision boundary as a measure of the uncertainty of the data point. Identify data points close to the decision boundary, which may have an important impact on the generalization ability of the model. Through decision boundary analysis, prioritize the collection of data that can provide a deeper understanding of the model's decision boundary. For example, the Euclidean distance can be used to calculate the distance from the data point to the decision boundary.
[0039] Finally, according to the results of model failure analysis and decision boundary analysis, set trigger conditions to automatically collect the required data. Evaluate the value of the collected data to avoid returning worthless data. Design triggers to improve data accuracy and recall to ensure that valuable data is not missed. Set priorities for different triggers based on the importance and scarcity of the event. Considering the bandwidth and traffic limitations of the vehicle end, prioritize the return of data with high importance. Weights can be assigned to different triggers, and the priority of data return can be determined based on the weights.
[0040] In summary, this optional embodiment uses model failure analysis and model decision boundary analysis to pre-set the scenarios and collection logic to be collected, set a data return trigger (i.e., trigger layer) on the vehicle side, and automatically obtain the required scenario data set according to the scenario algorithm. It is worth mentioning that when designing the trigger layer, it is necessary to maximize the accuracy and recall rate of the data, so as to avoid returning worthless data and prevent the omission of valuable data. Taking into account the 4 / 5G bandwidth and traffic issues of a single vehicle, different triggers can be grouped according to the priority of the event, and high-importance data, such as data of scarce scenarios, can be returned first.
[0041] For example, Priority = w1*Importance+w2*Scarcity, where w1 and w2 are weight coefficients, and Importance and Scarcity represent the importance and scarcity of the event, respectively. In this way, the most valuable data for model improvement can be automatically screened and collected, while taking into account the limitations and priorities of data transmission.
[0042] Furthermore, during the on-board data collection process, it is crucial to ensure that there is an efficient data transmission link in the vehicle, as this can maximize the utilization of data transmission speed and capacity. Although the traditional CAN bus transmission method is reliable, its data transmission volume is greatly limited. Therefore, the introduction of 4G or 5G modules has become a solution, which can effectively upload data to the cloud quickly on the vehicle side. Traditional data collection methods are usually difficult to capture data accurately, and usually require extensive collection of video data and uploading all data to the cloud. This method not only wastes network traffic and increases data storage pressure, but also increases the workload of subsequent annotation and training. In order to optimize the data collection process, this patent accurately captures data from five seconds before and after the vehicle by triggering events, including camera, radar and bus data. After data deduplication and other processing, the system packages the data into a compressed file and uploads it to the cloud.
[0043] In an optional embodiment, before uploading the jointly calibrated point cloud data and image data to the vehicle cloud server, the method further includes:
[0044] The value of the collected data is evaluated using the two dimensions of time and scene. Based on the value, the latest collected data is retained and the old data is deleted. At the same time, the scene data that is not processed enough by the current autonomous driving system is retained, and the scene data that has been processed sufficiently is deleted to obtain the uploaded target collected data. This embodiment minimizes the amount of stored data by deleting the old data and the scene data that has been processed sufficiently, thereby improving the efficiency of data return.
[0045] In an optional embodiment, the jointly calibrated point cloud data and image data are uploaded to the vehicle cloud server, including: compressing the data to be uploaded to obtain a corresponding compressed data packet, and generating a metadata file of the compressed data packet, the metadata file is associated with the compressed data packet, wherein the metadata file includes a label automatically added for each compressed data packet; uploading the metadata file together with the compressed data packet to the vehicle cloud server. When uploading data to the cloud, this embodiment uses a data compression method to compress the data before uploading, which can further reduce storage space and alleviate storage pressure on the vehicle cloud server; and, while compressing the data, detailed labels are added to each data packet, so that users can obtain key information without opening the data packet, making users more efficient in storing, searching and using data.
[0046] Specifically, metadata is data about data, usually including the data's description, format, size, creation time, modification time, author, copyright information, etc. Tags are a form of metadata that are used to mark key features of data packets. Tags can be keywords, categories, version numbers, summaries, etc., which help users quickly identify and search for data. During the compression process, metadata itself may also be compressed to reduce storage space. The algorithms for achieving data compression are not unique, including but not limited to Huffman coding, LZ77, LZ78 and other text compression algorithms. Among them, the compression of tags can be achieved by encoding short codes for commonly used tags, similar to the principle of Huffman coding. Commonly used tags are assigned shorter codes. The Huffman coding tree construction formula is:
[0047]
[0048] Among them, H(X) is the entropy of the data set X, P(x i ) is the data item x i The basic principle is to convert the probability distribution into an interval and then use this interval to represent the data.
[0049] Preferably, following the above optional embodiment, the method further includes the steps of storing the metadata file and the compressed data packet in the vehicle cloud server, and establishing index information of the compressed data packet based on the metadata file and the tag.
[0050] Specifically, metadata and tags are used to build indexes so that users can quickly locate the data packets of interest through the index without traversing the entire data set. Metadata and tags are stored in the header or tail of the data packet to ensure that they can be quickly accessed. Metadata and tags are designed to be self-descriptive so that users can understand their meaning even without external documentation. When data is updated or modified, the corresponding metadata and tags should also be updated to maintain the accuracy of the information.
[0051] Furthermore, in order to ensure the security of metadata and tags and prevent them from being tampered with, digital signatures are added during the message storage process to ensure information security.
[0052] For example, in an optional embodiment, the metadata file and the compressed data packet are stored in the vehicle cloud server, including: using a digital signature algorithm to encrypt and store the metadata file and the compressed data packet; when a read request for the metadata file or the compressed data packet is received, a digital signature identity authentication is performed on the object of the read request, and reading of the metadata file or the compressed data packet is allowed only after the authentication is passed.
[0053] Specifically, the digital signature data encryption process is as follows:
[0054] 1. Sender A uses a hash function (i.e., a hash function) to perform a hash operation on the plaintext message m to obtain a digital summary A;
[0055] 2. Sender A uses his own private key to encrypt digital summary A and obtain a digital signature;
[0056] 3. Sender A packages the digital signature, TA certificate (proving the authenticity of the public key) and plaintext into an electronic signature;
[0057] 4. Sender A encrypts the electronic signature using the public key of recipient B;
[0058] 5. Sender A sends the encrypted electronic signature package to receiver B;
[0059] 6. Recipient B uses his own private key to decrypt the encrypted electronic signature package and obtain the original text of the electronic signature package;
[0060] 7. After receiving the ciphertext sent by sender A, receiver B uses his own private key to decrypt the ciphertext and obtain the original data m;
[0061] 8. Receiver B uses the same hash function as sender A to perform a hash operation on the original data m to obtain a new digital summary B;
[0062] 9. Receiver B compares the new digital summary B with the digital summary A obtained from sender A. If they are consistent, the identity authentication is successful; if they are inconsistent, the identity authentication fails.
[0063] It can be seen that only objects that have successfully passed the digital signature identity authentication can read and use the original data, ensuring the security of data transmission.
[0064] All the above optional technical solutions can be arbitrarily combined to form optional embodiments of the present application, which will not be described one by one here.
[0065] The following is an embodiment of the device of the present application, which can be used to execute the embodiment of the method of the present application. For details not disclosed in the embodiment of the device of the present application, please refer to the embodiment of the method of the present application.
[0066] See also Figure 2 In a second embodiment of the present application, a smart driving data collection device is provided, which includes:
[0067] The data acquisition module 201 is configured to respectively acquire data collected by at least two perception sensors, the data including point cloud data collected by the laser radar and image data collected by the camera;
[0068] The data calibration module 202 is configured to perform joint calibration on at least two perception sensors, wherein the joint calibration includes: temporally aligning the point cloud data and the image data by using a motion compensation method, and inputting the point cloud data and the image data into a pre-trained graph neural network, and obtaining spatially aligned point cloud data and image data at the output of the graph neural network;
[0069] The data return module 203 is configured to upload the jointly calibrated point cloud data and image data to the vehicle cloud server.
[0070] According to the technical solution provided in the embodiment of the present application, precise alignment of multi-sensor data is ensured through motion-compensated time synchronization and graph neural network-based spatial synchronization, providing accurate and reliable raw data, laying a solid foundation for subsequent model training and evaluation tasks.
[0071] In some optional embodiments, the data calibration module 202 is configured to record the timestamps of the point cloud data and the image data; obtain the vehicle motion state at the time of collecting the point cloud data and the image data, the vehicle motion state including the average speed of the vehicle; determine the time offset based on the displacement of the vehicle between the two timestamps and the average speed of the vehicle; when the time offset is not zero, use the time offset to update the timestamp of the point cloud data or the image data to align the time of the point cloud data with the image data.
[0072] In some optional embodiments, the data calibration module 202 is configured to use a pre-trained SECOND network model and a Swin-Transformer model as encoders to convert point cloud data into point cloud BEV features and image data into camera BEV features, respectively; to flatten the point cloud BEV features and perform feature fusion after deep graph convolution on the camera BEV features, wherein the feature fusion includes three-dimensional spatial alignment of the point cloud BEV features and the camera BEV features; to input the fused features into a decoder, and to obtain spatially aligned point cloud data and image data at the output of the decoder.
[0073] In some optional embodiments, the above-mentioned intelligent driving data collection device further includes:
[0074] The data screening module 204 is configured to, after jointly calibrating at least two perception sensors, use model failure analysis and model decision boundary analysis to pre-set the scenarios and acquisition logic that need to be collected, and set a corresponding trigger mechanism on the vehicle-mounted end; monitor the calibrated point cloud data and image data based on the trigger mechanism, and when a pre-set scenario is detected, automatically acquire the data set under the corresponding acquisition scenario based on the pre-set acquisition logic.
[0075] In some optional embodiments, the data feedback module 203 is configured to evaluate the value of the collected data using the two dimensions of time and scene before uploading the jointly calibrated point cloud data and image data to the vehicle cloud server, retain the latest collected data and delete the old data based on the value, and retain the scene data that is insufficiently processed by the current autonomous driving system and delete the scene data that has been fully processed, so as to obtain the uploaded target collected data.
[0076] In some optional embodiments, the above-mentioned data feedback module 203 is configured to compress the data to be uploaded to obtain a corresponding compressed data packet, and generate a metadata file of the compressed data packet, wherein the metadata file is associated with the compressed data packet, wherein the metadata file includes a label automatically added to each compressed data packet; the metadata file is uploaded together with the compressed data packet to the vehicle cloud server.
[0077] In some optional embodiments, the data transmission module 203 is configured to store the metadata file and the compressed data packet in the vehicle cloud server, and to establish index information of the compressed data packet based on the metadata file and the tag.
[0078] In some optional embodiments, the above-mentioned data feedback module 203 is configured to use a digital signature algorithm to encrypt and store metadata files and compressed data packets; when a read request for a metadata file or a compressed data packet is received, a digital signature identity authentication is performed on the object of the read request, and reading of the metadata file or compressed data packet is allowed only after the authentication is passed.
[0079] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.
[0080] See also Figure 3 In a third embodiment of the present application, a car 3 is provided, including an intelligent driving data acquisition system, and the intelligent driving system includes a laser radar 31, a camera 32 and a control device 33.
[0081] LiDAR31 and camera32 are critical perception sensors in intelligent driving systems. They help autonomous vehicles perceive their surroundings in real time and support vehicles to make safe and reliable driving decisions. The following is the relationship between cameras and LiDAR and intelligent driving and their main functions.
[0082] The control device 33 can be used to implement the steps in the above method embodiments, or the functions of the modules in the above device embodiments.
[0083] For example, see Figure 4 After acquiring the point cloud data collected by the laser radar and the image data collected by the camera, the control device 33 can be based on Figure 4 The architecture shown is used to calibrate, filter and return point cloud data and image data.
[0084] Among them, for data calibration, combined with Figure 4 For example, first, motion compensation is used to perform time alignment. First, the timestamps of the lidar and camera data are recorded, denoted as t LiDAR and t Camera , the motion state at the time of data collection of the two sensors is obtained through IMU and wheel speed sensors, including velocity v and acceleration a. According to the motion state of the vehicle, the time offset Δt is calculated to calibrate the time of the lidar and camera data. The displacement d of the vehicle between the two timestamps is known, and the time offset Δt=dv is estimated, where d is the displacement of the vehicle (this is known, including but not limited to calculation through the vehicle's position data), and v is the average speed of the vehicle. Therefore, the calculated time offset Δt is applied to adjust the timestamps of the camera or lidar data to ensure that they correspond to the same time point. Assume that for the camera data, if calibration is required, the new timestamp is: t Camera,corrected =t Camera +Δt, ensure that the timestamp after synchronization satisfies: t LiDAR =t Camera,corrected Secondly, we use graph neural network to achieve spatial alignment of the two perceptual data, see Figure 5, taking inputs from different sensors (including LiDAR and camera), firstly applying modality-specific encoders, Swin-Transformer as camera encoder, Swin-Transformer is a model based on Transformer architecture, which captures long-range dependencies in images through self-attention mechanism. Second, as LiDAR encoder, to extract their respective characteristics. Then, the camera features are converted into camera BEV features through global alignment module, and deep graph convolution is performed on these features. Deep graph convolution can effectively associate information in depth images, that is, through convolution operation, depth information is viewed in local areas, and the depth features of adjacent pixels are integrated and processed. This helps to capture the spatial correlation between depth information at the pixel level. Through deep graph convolution, the feature expression ability of depth images can be enhanced. Traditional RGB image convolution mainly focuses on color and texture features, while deep graph convolution is more focused on the propagation and integration of depth information, so as to better reflect the spatial layout and distance relationship of objects in the scene. It aims to alleviate the local misalignment caused by the projection error between LiDAR and camera in the camera-to-BEV process of previous BEV-based methods. The LiDAR features are then compressed along the axis to represent the 3D features as 2D LiDAR BEV features, and finally a learnable offset is used to achieve global multimodal BEV feature alignment. The supervised information is derived from the features obtained by convolution of LiDAR and camera BEV features. During the training process, global offset noise is introduced and a learnable offset is used, which ultimately achieves high-precision spatial alignment of camera and radar.
[0085] For data screening, model failure analysis and model decision boundary analysis are used to set the scenarios and collection logic to be collected in advance. A data return trigger (trigger layer) is set on the vehicle side to automatically obtain the required scenario data set according to the scenario algorithm. When designing the trigger layer, the accuracy and recall rate of the data must be maximized to avoid returning worthless data and preventing the omission of valuable data.
[0086] For data transmission, when data is uploaded to the cloud, it is highly compressed and detailed tags are added to each data packet, so that users can obtain key information without opening the data packet. This method makes users more efficient in storing, searching and using data.
[0087] Specifically, the structure of the control device is not unique, see Figure 6In an optional embodiment, the control device 33 includes: a processor 331, a memory 332, and a computer program 333 stored in the memory 332 and executable on the processor 331. When the processor 331 executes the computer program 333, the steps in the above-mentioned method embodiments are implemented. Alternatively, when the processor 331 executes the computer program 333, the functions of the modules in the above-mentioned device embodiments are implemented.
[0088] The electronic device 6 may be a desktop computer, a notebook, a PDA, a cloud server, or other electronic device. The electronic device 6 may include, but is not limited to, a processor 331 and a memory 332. Those skilled in the art will appreciate that Figure 6 The electronic device 6 is merely an example and does not limit the electronic device 6 . The electronic device 6 may include more or less components than those shown in the figure, or different components.
[0089] The processor 331 can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.
[0090] The memory 332 may be an internal storage unit of the electronic device 6, for example, a hard disk or memory of the electronic device 6. The memory 332 may also be an external storage device of the electronic device 6, for example, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the electronic device 6. The memory 332 may also include both an internal storage unit and an external storage device of the electronic device 6. The memory 332 is used to store computer programs and other programs and data required by the electronic device.
[0091] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional modules in the embodiment can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units.
[0092] If the integrated module is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium (e.g., a computer-readable storage medium). Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. The computer program may include computer program code, which may be in source code form, object code form, executable file or some intermediate form. Computer-readable storage media may include: any entity or device capable of carrying computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc.
[0093] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.
Claims
1. A method for collecting intelligent driving data, characterized in that: include: Acquire data collected by at least two perception sensors respectively, wherein the data includes point cloud data collected by a laser radar and image data collected by a camera; Performing joint calibration on the at least two perception sensors, the joint calibration comprising: temporally aligning the point cloud data and the image data by using a motion compensation method, and inputting the point cloud data and the image data into a pre-trained graph neural network, and obtaining spatially aligned point cloud data and image data at the output of the graph neural network; The jointly calibrated point cloud data and image data are uploaded to the vehicle cloud server.
2. The method according to claim 1, characterized in that The step of temporally aligning the point cloud data and the image data by using a motion compensation method includes: Record the timestamp of point cloud data and image data; Obtain the vehicle motion state at the time of collecting point cloud data and image data, where the vehicle motion state includes the average speed of the vehicle; Determine a time offset based on the displacement of the vehicle between two time stamps and the average speed of the vehicle; When the time offset is not zero, the time offset is used to update the timestamp of the point cloud data or the image data, so as to align the time of the point cloud data with that of the image data.
3. The method according to claim 1, characterized in that: The step of inputting the point cloud data and the image data into a pre-trained graph neural network and obtaining spatially aligned point cloud data and image data at the output of the graph neural network includes: Use the pre-trained SECOND network model and Swin-Transformer model as encoders to convert point cloud data into point cloud BEV features and image data into camera BEV features respectively; Flatten the point cloud BEV features respectively, and perform feature fusion after performing depth graph convolution on the camera BEV features, wherein the feature fusion includes three-dimensional spatial alignment of the point cloud BEV features and the camera BEV features; The fused features are input into the decoder, and spatially aligned point cloud data and image data are obtained at the output of the decoder.
4. The method according to claim 1, characterized in that: After jointly calibrating the at least two perception sensors, the method further includes: Using model failure analysis and model decision boundary analysis, set the scenarios and collection logic that need to be collected in advance, and set the corresponding trigger mechanism on the vehicle side; The calibrated point cloud data and image data are monitored based on the trigger mechanism. When a pre-set scene is monitored, the data set under the corresponding acquisition scene is automatically acquired based on the pre-set acquisition logic.
5. The method according to claim 4, characterized in that Before uploading the jointly calibrated point cloud data and image data to the vehicle cloud server, the method further includes: The value of the collected data is evaluated using the two dimensions of time and scene. Based on the value, the latest collected data is retained and the old data is deleted. At the same time, the scene data that is insufficiently processed by the current autonomous driving system is retained and the scene data that has been fully processed is deleted to obtain the uploaded target collected data.
6. The method according to any one of claims 1 to 5, characterized in that The step of uploading the jointly calibrated point cloud data and image data to the vehicle cloud server includes: Compressing the data to be uploaded to obtain a corresponding compressed data packet, and generating a metadata file of the compressed data packet, wherein the metadata file is associated with the compressed data packet, wherein the metadata file includes a label automatically added to each compressed data packet; The metadata file is uploaded to the vehicle cloud server together with the compressed data package.
7. The method according to claim 6, characterized in that Also includes: The metadata file and the compressed data packet are stored in the vehicle cloud server, and index information of the compressed data packet is established based on the metadata file and the tag.
8. The method according to claim 7, characterized in that Storing the metadata file and the compressed data packet in the vehicle cloud server includes: Use digital signature algorithm to encrypt and store metadata files and compressed data packets; When a read request for a metadata file or a compressed data packet is received, a digital signature identity authentication is performed on the object of the read request, and reading of the metadata file or compressed data packet is allowed only after the authentication is passed.
9. An intelligent driving data collection device, characterized in that: include: A data acquisition module is configured to respectively acquire data collected by at least two perception sensors, wherein the data includes point cloud data collected by a laser radar and image data collected by a camera; A data calibration module is configured to jointly calibrate the at least two perception sensors, wherein the joint calibration includes: temporally aligning the point cloud data and the image data by using a motion compensation method, and inputting the point cloud data and the image data into a pre-trained graph neural network, and obtaining spatially aligned point cloud data and image data at the output of the graph neural network; The data feedback module is configured to upload the jointly calibrated point cloud data and image data to the vehicle cloud server.
10. An automobile, comprising an intelligent driving data acquisition system, wherein the intelligent driving system comprises a laser radar, a camera and a control device, wherein the control device comprises at least a memory, a processor and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 8 are implemented.
Citation Information
Cited By
Indoor map construction method and system based on BIM and laser radar
CN119849016A
