Time synchronization based self-owned dataset bev perception training method

Abstract

This invention discloses a BEV perception training method based on time synchronization using a proprietary dataset, comprising the following steps: establishing a unified system time reference through second pulse signals and time messages to ensure time consistency of multi-sensor data; configuring independent storage units for each sensor and directly storing raw data during the acquisition phase; performing soft time synchronization using LiDAR as the main sensor during the offline phase, dividing keyframes and intermediate frames, and performing 3D annotation on keyframes; constructing a multi-level data structure fully compatible with public datasets, and establishing sample time series links and instance annotation links to form a dual-link index closed loop; after ensuring data quality through full projection verification, directly using existing BEV perception training code for model training. This invention enables efficient use of proprietary acquired data for BEV perception model training without relying on public dataset acquisition processes, and supports target category expansion.

Description

Technical Field

[0001] This invention relates to the field of sensor perception technology, and in particular to a BEV perception training method based on a time-synchronized proprietary dataset. Background Technology

[0002] Multi-sensor perception technology based on bird's-eye-view (BEV) images, by fusing data from multiple sources such as surround-view cameras and LiDAR, can provide a unified spatial representation for autonomous vehicles and has become an important technical route for current environmental perception. To train a BEV-based multi-sensor perception model, key steps typically need to be completed sequentially, including multi-sensor data acquisition, multi-sensor time alignment, training sample construction, annotation and association, and model training. However, in practical engineering applications, each of these steps faces significant challenges that restrict the trainability of proprietary datasets, as follows: 1. The conflict between real-time performance and completeness in the data acquisition phase: During the data acquisition phase, multi-sensor data typically needs to be acquired and written to storage media in real time online. If the data from each sensor is uniformly packaged and stored online, it is easily limited by cache bandwidth and disk write speed, leading to frame loss in high-resolution images or high-frequency point cloud data during acquisition. Once frame loss occurs, it not only disrupts the continuity of data in the temporal dimension but also has a cascading impact on subsequent processing steps such as time alignment, sample construction, and annotation association, ultimately resulting in incomplete or unusable training samples, directly restricting the usability of the dataset.

[0003] 2. Feature misalignment caused by inconsistent time bases among multiple sensors: Different sensors have different sampling frequencies and triggering mechanisms. Without a unified and accurate time base, it is difficult to achieve accurate correspondence between the data from different sensors in the time dimension. Especially when constructing BEV training samples based on keyframes of a master sensor, if the time deviation between other sensors and the master sensor is too large, it will lead to inconsistencies between the environmental state reflected by the image and point cloud within the same training sample, resulting in BEV feature misalignment. This misalignment will directly reduce the model's perception accuracy and fusion effect, and in severe cases, make training unable to converge.

[0004] 3. Disordered temporal relationships in post-processing: In the post-processing stage, the raw data collected usually needs to undergo format conversion, distortion correction, and time alignment before it can be used to construct training samples. If keyframes and intermediate frames are not properly divided or a stable temporal relationship is not established in this stage, the temporal relationship of the samples will be disordered, which will affect the stability of annotation propagation and training processes.

[0005] 4. Disruption of the Linkage in the Labeling and Sample Organization Stages: In the labeling and sample organization stages, public datasets typically establish a multi-level index relationship of "scene-sample-multi-sensor data-label-instance" through multiple types of metadata files, thereby supporting efficient and reliable model training. For self-collected data, even if metadata files of the same format can be generated, it is still necessary to ensure that the token association relationships, frame-to-frame links, and extrinsic parameter / pose relationships between data tables remain consistent throughout the entire indexing chain. If any association relationship is incorrect, it will directly lead to incorrect training sample construction, invalidating the model training results, or even producing incorrect perceptual output.

[0006] Therefore, how to achieve high-precision time consistency constraints while ensuring the integrity of multi-sensor data, and how to construct a training sample organization structure that meets the training requirements of BEV perception, are the core technical problems that urgently need to be solved. Summary of the Invention

[0007] This invention aims to address at least one of the technical problems existing in the prior art. To this end, this invention proposes a BEV-aware training method based on a time-synchronized proprietary dataset.

[0008] A BEV-aware training method based on a time-synchronized proprietary dataset according to an embodiment of the present invention includes the following steps: S1. Configure the time synchronization of the multi-sensor acquisition system to obtain a unified system time reference; S2. Online acquisition and storage of raw data from multiple sensors to ensure data integrity and avoid frame loss during the acquisition process; S3. The raw data collected is processed offline, and time soft synchronization is performed based on a unified time base to divide key frames and intermediate frames. S4. Perform 3D annotation on the keyframes of the lidar and use the annotation results as the basis for subsequent sample construction and annotation propagation; S5. Construct a multi-level training sample structure for BEV perception training and establish index relationships between samples to ensure that the self-collected data is compatible with existing public multi-sensor datasets in terms of fields, levels and index logic.

[0009] In some embodiments of the present invention, the time synchronization configuration of the multi-sensor acquisition system in step S1 specifically includes: S11. Obtain the timing signal output by the satellite positioning device, wherein the timing signal includes a second pulse signal and a time message containing UTC time information; S12. Connect the second pulse signal to the pulse input interface of the industrial control computer, connect the time message to the serial communication interface of the industrial control computer, and trigger the system time calibration action at the whole second boundary; S13. System time calibration based on second pulse signal and time message: When the second pulse signal is received, the system time is aligned to the whole second boundary, the whole second part is determined by combining the UTC whole second information provided by the time message, and the whole second part is combined with the aligned sub-second part to form the unified system time of the industrial control computer. S14. Use the unified system time after the industrial control computer is calibrated as the unified time reference for multi-sensor data acquisition, and generate time stamps for the data acquired by the lidar and camera respectively during the acquisition phase.

[0010] In some embodiments of the present invention, the online acquisition and storage of raw data from multiple sensors in step S2 specifically includes: S21. A multi-sensor acquisition system including a lidar sensor and multiple camera sensors is configured, with an independent data acquisition and storage execution unit configured for each sensor. S22. During the acquisition process, the raw data output by each sensor is acquired respectively, and the raw data is not encapsulated, compressed or converted in format, or only minimized. S23. For each sensor, the raw data is directly written to the storage medium using the corresponding independent acquisition and storage execution unit, wherein the execution units do not block each other. S24. While the original data is being written to storage, a corresponding time identifier is generated for each frame of original data based on the unified system time. S25. The above method enables continuous online acquisition of raw data from multiple sensors, ensuring that the number of data frames acquired by each sensor within the preset acquisition time is consistent with the theoretical number of sampling frames or within the allowable error range.

[0011] In some embodiments of the present invention, step S3, which involves performing time soft synchronization and dividing keyframes and intermediate frames, specifically includes: S31. In the offline processing stage, read the stored multi-sensor raw data. Each raw data contains a time identifier generated based on a unified system time. Perform offline format conversion processing on the raw data. S32. Based on the unified system time reference and the time identifier of each raw data, construct the correlation relationship of multi-sensor data in the time dimension; S33. Select a lidar as the main sensor, use the time stamp of the lidar data as the reference for time soft synchronization, and select lidar key frames in the lidar data sequence according to a preset time interval. S34. For each key frame of the LiDAR, based on the principle of temporal proximity, find the camera data frame in each camera data sequence that is closest to the time identifier of the key frame of the LiDAR and use it as the key frame of the corresponding camera. S35. Repeat the above keyframe matching process until the keyframe matching of the entire data sequence is completed. S36. Determine the data frames in each sensor data sequence that were not selected as keyframes as intermediate frames.

[0012] In some embodiments of the present invention, step S4, which involves performing 3D annotation on the lidar keyframe, specifically includes: S41. Select the key frame of the lidar determined in step S3 as the 3D annotation object; S42. For the key frame of the lidar, perform three-dimensional annotation on the target object based on the lidar point cloud data. The three-dimensional annotation includes at least the spatial position, size and attitude information of the target object. S43. Project the completed 3D annotation results onto the multi-camera images corresponding to the keyframe time for visual inspection; S44. Based on the LiDAR point cloud perspective and the visualization results after projection onto the camera image, check and confirm the 3D annotation results. S45. Associate the confirmed 3D annotation results of the lidar keyframes with the time stamp and frame index of the corresponding keyframes for subsequent annotation propagation on intermediate frames.

[0013] In some embodiments of the present invention, the construction of the multi-level training sample structure in step S5 specifically includes: S51. Define a multi-level data object including scene object, sample object, sensor data object, instance object and annotation object, and establish a hierarchical relationship of "scene-sample-multi-sensor data-annotation-instance". The definition of the hierarchical objects is consistent with the existing public multi-sensor datasets. S52. Generate unique identifiers for objects at each level, and construct a first link based on the sample time series and a second link based on the target instance respectively, so that the index relationship forms a closed loop; S53 performs consistency checks and projection verifications on the generated metadata files to ensure that index relationships, time associations, and extrinsic / pose associations are correct. S54. After the verification is passed, the BEV perception model is trained based on the proprietary training dataset. S55. While maintaining data structure compatibility, expand the target category set and perform full training on the model to support the recognition of new categories.

[0014] In some embodiments of the present invention, the step S52 of constructing the dual-link index relationship specifically includes: S521. Generate globally unique identifiers for scene objects, sample objects, sensor data objects, target instance objects, annotation objects, sensor configuration objects, and pose / extrinsic parameter objects respectively. S522. For each scene object, sort the sample objects according to the time identifier, and establish a reference relationship between the previous sample and the subsequent sample for each sample object to form a sample time series link. S523. For each sensor channel, sort the sensor data objects according to the time identifier, establish a previous / next sequence reference relationship, and establish a reference relationship between each sensor data object and its corresponding sample object. S524. For each target instance object, establish the annotation sequence link of the instance at different sample times, establish the reference relationship between the preceding and subsequent annotations for the annotation objects of the same instance, and form the instance annotation link. S525. Apply consistency constraints to sample links and instance links, including: any sample object referenced by any labeled object must exist in the sample time series link of its scene; the sample time corresponding to the preceding / following labeled objects of any labeled object in the instance link must satisfy the time sequence; for two adjacent frame labeled objects of the same instance, their corresponding sample objects should satisfy the reachability relationship in the sample time series link.

[0015] In some embodiments of the present invention, the consistency verification and projection verification in step S53 specifically include: S531. Load all metadata files and perform structural integrity checks, including at least global uniqueness checks of unique identifiers, existence checks of referenced objects, and consistency checks of the number of objects. S532. Based on the sample time series links and instance annotation links, perform dual-link consistency verification to check link reachability, unbroken links, and the correctness of time sequence. S533. For key frame samples, check whether the matching relationship between the key frame samples and the corresponding multi-camera data is valid based on the time soft synchronization results. S534 performs projection verification on the 3D annotations of keyframes, projects the 3D annotation results onto the image planes of each camera based on calibration parameters, and checks whether the projected position is consistent with the actual position of the target in the image.

[0016] In some embodiments of the present invention, the category expansion and full training in step S55 specifically include: S551. Add at least one new category to the existing target category set, and assign a corresponding identifier to the new category in the category definition information; S552. For samples containing newly added categories, generate or supplement the three-dimensional annotation information of the newly added categories, and maintain a consistent reference relationship with the original data in the metadata index link; S553. Update the parameters of the BEV perception model using full training, so that the model can recognize both the original and newly added categories.

[0017] In some embodiments of the present invention, in step S2, the data writing process of each sensor is completed by an independent thread or independent process, so that the data writing delay of one sensor will not affect the data acquisition and storage of other sensors; in step S3, the time identifier is recorded in the file name of the corresponding data file.

[0018] Compared with existing technologies, the BEV-aware training method based on time-synchronized proprietary datasets of this invention has the following advantages: First, this invention performs high-precision time synchronization configuration on the multi-sensor system before data acquisition, establishing a unified system time reference using the second pulse signal and time message output by the GNSS timing device. This ensures that the data from the lidar and multiple cameras maintain sub-millisecond consistency in the time dimension. This mechanism effectively avoids time deviations caused by differences in sampling frequencies and triggering mechanisms of different sensors, fundamentally solving the negative impact of cross-sensor feature misalignment on BEV perception model training, thereby significantly improving the stability and convergence reliability of the model training process.

[0019] Secondly, this invention effectively reduces the risk of frame loss caused by cache bandwidth congestion or disk write blockage by configuring independent acquisition and storage execution units for each LiDAR and each camera, and by directly storing raw data during the acquisition phase, avoiding online encapsulation and complex processing operations. This design ensures the continuity and integrity of raw data from multiple sensors without relying on high-performance storage hardware, providing a reliable data foundation for subsequent training sample construction.

[0020] Third, this invention moves all computationally intensive processing steps, such as keyframe partitioning, time alignment, format conversion, and distortion correction, to the offline stage. The online acquisition stage only involves directly writing the raw data to disk and binding it with a timestamp. This design effectively avoids the impact of excessive CPU load on the real-time performance of data acquisition during the online stage, significantly improving the stability and robustness of the acquisition system in long-term, high-volume data acquisition scenarios, and reducing the system's hardware requirements for industrial control computer computing resources.

[0021] Fourth, this invention uses LiDAR as the main sensor and performs soft synchronization of multi-sensor data based on the principle of temporal proximity during the offline phase, and rationally divides keyframes and intermediate frames according to a preset time interval. This mechanism enables different sensor data to establish a stable correspondence in the temporal dimension, ensuring that training samples based on LiDAR keyframes can accurately match multi-camera data in temporal proximity, effectively avoiding BEV feature misalignment caused by time matching errors and improving the accuracy of training sample construction.

[0022] Fifth, this invention constructs a data structure that is fully compatible with existing publicly available multi-sensor datasets in terms of field definitions, hierarchical organization (scene-sample-multi-sensor data-labeling-instance), and indexing relationships. This allows proprietary data to be directly used for model training without modifying existing BEV perception training code. This feature significantly reduces the engineering cost and usage threshold for migrating from training on public datasets to training on proprietary datasets, demonstrating significant practical engineering value.

[0023] Sixth, this invention simultaneously constructs both a prev / next link based on sample time series and a prev / next link based on target instances, and imposes strict consistency constraints on both types of links. This dual-link closed-loop indexing mechanism ensures that samples, annotations, and instances form a traceable and verifiable complete index relationship in both the time and structural dimensions, effectively avoiding training sample construction failures or training anomalies caused by indexing errors.

[0024] Seventh, after generating all metadata files, this invention performs full projection verification on the 3D annotations of keyframes, projecting the 3D annotation results onto the multi-camera image plane based on calibration parameters for consistency checks. This mechanism can detect and accurately locate problems caused by extrinsic parameter calibration errors, pose estimation deviations, time synchronization errors, or index association errors before model training begins, thereby avoiding the waste of time and computing resources caused by directly sending erroneous data into the training process, and effectively ensuring the trainability of the generated dataset.

[0025] Eighth, this invention, while maintaining data structure integrity and indexing logic compatibility, supports the flexible addition of new categories to the existing target category set. By assigning corresponding identifiers to new categories and generating corresponding 3D annotation information, combined with full-scale training, the BEV perception model possesses the ability to recognize both existing and new categories. This feature effectively meets the ever-changing perception requirements in real-world autonomous driving applications, significantly enhancing the system's practicality and scalability. Attached Figure Description

[0026] Figure 1 This is a flowchart of the BEV perception training method based on time synchronization of the proprietary dataset of the present invention; Figure 2 This is a flowchart of the index closure and trainability verification process of the present invention. Detailed Implementation

[0027] Embodiments of the present invention are described in detail below, examples of which are illustrated in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain the present invention, and should not be construed as limiting the present invention.

[0028] The following reference Figures 1 to 2 This invention provides a BEV-aware training method based on a time-synchronized proprietary dataset, comprising the following steps: S1. Configure the time synchronization of the multi-sensor acquisition system to obtain a unified system time reference; S2. Online acquisition and storage of raw data from multiple sensors to ensure data integrity and avoid frame loss during the acquisition process; S3. The raw data collected is processed offline, and time soft synchronization is performed based on a unified time base to divide key frames and intermediate frames. S4. Perform 3D annotation on the keyframes of the lidar and use the annotation results as the basis for subsequent sample construction and annotation propagation; S5. Construct a multi-level training sample structure for BEV perception training and establish index relationships between samples to ensure that the self-collected data is compatible with existing public multi-sensor datasets in terms of fields, levels and index logic.

[0029] In some embodiments of the present invention, the time synchronization configuration of the multi-sensor acquisition system in step S1 specifically includes: S11. Obtain the timing signal output by the satellite positioning device, wherein the timing signal includes a second pulse signal and a time message containing UTC time information; wherein the time message is used to provide absolute time information for whole seconds to determine the current UTC whole second; the second pulse signal is used to provide a hardware alignment reference for the whole second boundary to align the whole second boundary and improve sub-second accuracy; S12. Connect the second pulse signal to the pulse input interface of the industrial control computer, connect the time message to the serial communication interface of the industrial control computer, and trigger the system time calibration action at the whole second boundary; specifically, after receiving the second pulse signal, the industrial control computer triggers the system time calibration action at the whole second boundary (such as correcting / aligning the sub-second part), and uses the second pulse event as a high-precision reference for system time calibration. S13. System time calibration based on second pulse signal and time message: When the second pulse signal is received, the system time is aligned to the whole second boundary. The whole second part is determined by combining the UTC whole second information provided by the time message. The whole second part is combined with the aligned sub-second part to form a unified system time for the industrial control computer. By continuously receiving the second pulse signal and time message, the system time is periodically and continuously updated to form a stable and reusable unified time reference. S14. The unified system time calibrated by the industrial control computer is used as the unified time reference for multi-sensor data acquisition. During the acquisition phase, time identifiers are generated for the data acquired by the LiDAR and camera respectively. The time identifiers are used for subsequent time soft synchronization, keyframe segmentation, and training sample construction, so that samples based on the keyframes of the main sensor can be matched with data from other sensors that are in close time proximity, thereby reducing the risk of feature misalignment caused by cross-sensor time inconsistency.

[0030] In some embodiments of the present invention, the online acquisition and storage of raw data from multiple sensors in step S2 specifically includes: S21. A multi-sensor acquisition system is configured, including one lidar sensor and multiple camera sensors. Each sensor is configured with an independent data acquisition and storage execution unit. Each sensor has the same or similar sampling frequency, and the sampling frequency of both the lidar sensor and each camera sensor is a preset frame rate. Each sensor is configured with an independent data acquisition and storage execution unit, which enables parallel acquisition and processing of data from each sensor to avoid mutual blocking of data during the acquisition phase. S22. During the acquisition process, the raw data output by each sensor is acquired respectively, and the raw data is not encapsulated, compressed, or converted in format, or is only minimized; wherein, the image data output by the camera sensor is acquired in the raw image data format, and the point cloud data output by the lidar sensor is acquired in the raw point cloud data format; by not encapsulating, compressing, or converting the raw data, or only minimizing it, the CPU load and memory cache usage are reduced; S23. For each sensor, the raw data is directly written to the storage medium using the corresponding independent acquisition and storage execution unit, wherein the execution units do not block each other; the data writing process of each sensor is completed by an independent thread or independent process, so that the data writing delay of one sensor will not affect the data acquisition and storage of other sensors. S24. While the original data is being written to storage, a corresponding time identifier is generated for each frame of original data based on the unified system time. S25. By using the above method, continuous online acquisition of raw data from multiple sensors is achieved, so that the number of data frames acquired by each sensor within the preset acquisition time is consistent with the theoretical number of sampling frames or within the allowable error range, so as to avoid data loss during the acquisition stage.

[0031] In some embodiments of the present invention, step S3, which involves performing time soft synchronization and dividing keyframes and intermediate frames, specifically includes: S31. In the offline processing stage, read the multi-sensor raw data stored in step S2. Each raw data contains a time stamp generated based on a unified system time. Perform offline format conversion processing on the raw data. Specifically, performing offline format conversion processing on the raw data includes: converting the raw image data acquired by the camera into a target image format for subsequent training or annotation processing, and performing necessary image preprocessing operations; converting the raw point cloud data acquired by the lidar into a target point cloud format for subsequent processing. S32. Based on the unified system time base established in step S1 and the time identifier of each original data in step S2, construct the correlation relationship of multi-sensor data in the time dimension; no key frame and intermediate frame division is performed in the online acquisition stage, and time soft synchronization and frame type division are uniformly performed in the offline stage to reduce the computational load in the online acquisition stage. S33. Select a lidar as the main sensor, and use the time stamp of the lidar data as the reference for soft time synchronization. Select lidar keyframes from the lidar data sequence at preset time intervals; wherein, the time stamp of the first lidar keyframe is denoted as... The time stamp of the next lidar key frame is recorded as The time interval between adjacent keyframes is a preset value; the time markers for all LiDAR keyframes are: , , , , , where m is the number of keyframes in the entire data sequence; S34. For each LiDAR keyframe, based on the principle of temporal proximity, find the camera data frame in each camera data sequence that is closest to the time identifier of the LiDAR keyframe, and use it as the keyframe of the corresponding camera; where, for the LiDAR keyframe... Search for the time stamp that is closest in the data from multiple cameras. camera frames , ,…, , where n represents the number of cameras; S35. Repeat the above keyframe matching process until the keyframe matching of the entire data sequence is completed. Specifically, after matching the current LiDAR keyframe with the corresponding camera keyframe, determine the time identifier of the next LiDAR keyframe according to a preset time interval. For the next LiDAR keyframe, repeat the above keyframe matching process, searching for the camera data frame in each camera data stream that is closest to the time identifier of the LiDAR keyframe, until the keyframe matching of the entire data sequence is completed. For example, for the LiDAR keyframe... Search for the time stamp that is closest in the data from multiple cameras. camera frames , ,…, ; S36. Determine the data frames in each sensor data sequence that were not selected as keyframes as intermediate frames. Intermediate frames are used for subsequent annotation propagation, time interpolation, or training sample expansion processing.

[0032] In some embodiments of the present invention, step S4, which involves performing 3D annotation on the lidar keyframe, specifically includes: S41. Select the key frame of the lidar determined in step S3 as the 3D annotation object; S42. For the key frame of the lidar, perform three-dimensional annotation on the target object based on the lidar point cloud data. The three-dimensional annotation includes at least the spatial position, size and attitude information of the target object. S43. Project the completed 3D annotation results onto the multi-camera images corresponding to the keyframe time for visual inspection of the consistency of the 3D annotation projection on the image plane. S44. Based on the LiDAR point cloud perspective and the visualization results after projection onto the camera image, check and confirm the 3D annotation results to ensure that the spatial position, size and orientation of the annotation are consistent with the actual target object. S45. Associate the confirmed 3D annotation results of the lidar keyframes with the time stamp and frame index of the corresponding keyframes for subsequent annotation propagation on intermediate frames.

[0033] In some embodiments of the present invention, the construction of the multi-level training sample structure in step S5 specifically includes: S51. Define a multi-level data object hierarchy, including scene objects, sample objects, sensor data objects, instance objects, and annotation objects, and establish a hierarchical relationship of "scene-sample-multi-sensor data-annotation-instance." The definition of these hierarchical objects is consistent with existing publicly available multi-sensor datasets. Specifically, a scene object (Scene) describes a continuously acquired data sequence; a sample object (Sample) describes discrete time-point data within the scene; a sensor data object (Sample Data) describes data acquired by different sensors at the same time point; an instance object (Instance) describes the same physical target appearing continuously across time; and a annotation object (Sample Annotation) describes the spatial annotation information of target objects within the sample. The definition of the sample hierarchical objects maintains consistency with existing publicly available multi-sensor datasets in terms of field naming, hierarchical relationships, and referencing methods, enabling existing BEV perception training code to directly read and parse its own data. The hierarchical relationship of "scene-sample-multi-sensor data-annotation-instance" means that each scene object is associated with a set of sample objects, each sample object is associated with multiple sensor data objects, each annotation object is associated with a unique sample object and instance object, and each instance object describes the continuity of the target in the time dimension through the association of previous and subsequent annotations.

[0034] Furthermore, while maintaining overall data structure compatibility, a category system is defined for target annotation and training. This category system can include existing target categories in the public dataset, as well as at least one newly added target category to describe target types not defined in the public dataset. By introducing extended categories at the category definition layer and referencing the corresponding category identifiers in the annotation objects, training support for newly added target categories is achieved without disrupting the original data structure and indexing logic.

[0035] The sample hierarchy objects, index relationships, and category system defined above are output as multiple metadata files to describe the overall structure of the training samples, providing a foundation for subsequent index closed-loop construction and consistency verification.

[0036] S52. Generate unique identifiers for objects at each level, and construct a first link based on the sample time series and a second link based on the target instance, so that the index relationship forms a closed loop; specifically, this may include the following steps: S521. Generate globally unique identifiers for scene objects, sample objects, sensor data objects, target instance objects, annotation objects, sensor configuration objects, and pose / extrinsic parameter objects. These identifiers are used to establish one-to-one or one-to-many reference relationships between objects at different levels. The generation rules for these unique identifiers are: globally unique within the same dataset and usable for cross-table references. Specifically, sample objects are associated with scene objects through unique identifiers; sensor data objects are associated with sample objects and sensor configurations through unique identifiers; annotation objects are associated with sample objects and target instance objects through unique identifiers; and target instance objects form time-series links through the reference relationships between preceding and following annotations.

[0037] S522. Constructing a Sample Time Series Link (Sample Dimension prev / next): For each scene object, sort the sample objects according to their time identifiers and establish a reference relationship between previous and subsequent samples for each sample object, forming a sample time series link. Specifically, set the previous reference of the first sample object to null or a preset null value; set the subsequent reference of the last sample object to null or a preset null value; for intermediate sample objects, set their previous reference to point to the unique identifier of the sample object at the previous time step, and their subsequent reference to point to the unique identifier of the sample object at the next time step. Through the above method, a "prev / next" sample link is formed in the scene dimension, enabling the training process to traverse samples in time order.

[0038] S523. Constructing Sensor Data Links (Sample Data Dimension): For each sensor channel, sensor data objects are sorted according to time identifiers, establishing pre-sequence / post-sequence reference relationships, and each sensor data object is linked to its corresponding sample object. Specifically, for each sensor channel, within the same scene or the same continuous acquisition segment, sensor data objects are sorted according to time identifiers, and pre-sequence / post-sequence reference relationships are established for each sensor data object, forming a data sequence link for the same sensor channel. Simultaneously, each sensor data object is linked to its corresponding sample object to achieve an index mapping between "samples" and "multi-sensor data."

[0039] S524. Constructing Target Instance Links (Instance Dimension and Annotation Dimension): For each target instance, establish a link of annotation sequences for that instance at different sample times. Establish a reference relationship between preceding and subsequent annotations for the same instance's annotation objects, forming an instance annotation link. Specifically, for each target instance, establish a link of annotation sequences for that instance at different sample times, including: sorting annotation objects belonging to the same target instance according to the time order of their corresponding samples; establishing a reference relationship (prev / next) between preceding and subsequent annotations for the same instance's annotation objects, forming an instance annotation link; recording the unique identifiers of the first and last annotations of the instance object, and recording the number of annotations for the instance. This method forms a continuous "Instance-Annotation" link, enabling the training or evaluation process to track the continuous state of the same target in the time dimension.

[0040] S525. Apply consistency constraints to sample links and instance links, including: any sample object referenced by any labeled object must exist in the sample time series link of its scene; the sample time corresponding to the preceding / following labeled objects of any labeled object in the instance link must satisfy the chronological order; for two adjacent frame labeled objects of the same instance, their corresponding sample objects should satisfy the reachability relationship in the sample time series link (i.e., they can be traced back to adjacent samples in the same scene through prev / next).

[0041] The output contains a metadata file containing the aforementioned unique identifier and dual-link reference relationship, enabling the self-collected data to form an index closed-loop structure consistent with the public dataset, and can be directly read and parsed by existing BEV perception training code.

[0042] S53 performs consistency checks and projection verification on the generated metadata files to ensure the correctness of index relationships, temporal associations, and extrinsic / pose associations. Specifically, this includes the following steps: S531. Load all metadata files and perform structural integrity checks, including at least global uniqueness checks of unique identifiers, existence checks of referenced objects, and consistency checks of the number of objects.

[0043] S532. Based on the sample time series links and instance annotation links, perform dual-link consistency verification to check link reachability, unbroken links, and correct time sequence. For example, for each scene object, traverse all its sample objects along the sample links and check whether the prev / next links are reachable and unbroken; for each instance object, traverse all its annotation objects along the instance links and check whether the prev / next links are reachable and unbroken; check whether the sample objects referenced by each annotation object belong to its scene sample links; check whether the time sequence of adjacent annotations in the instance links is consistent to avoid time reversal or cross-scene reference errors.

[0044] S533. For keyframe samples, check whether the matching relationship between the keyframe samples and the corresponding multi-camera data is valid based on the time soft synchronization results; such as checking whether the keyframe data of the main sensor associated with the keyframe sample exists; checking whether the multi-camera data associated with the keyframe sample exists; checking whether the time difference between the time marker of each camera data and the time marker of the main sensor keyframe meets the preset matching conditions.

[0045] S534. Perform projection verification on the 3D annotations of keyframes. Based on the calibration parameters, project the 3D annotation results onto the image planes of each camera, and check whether the projected position is consistent with the actual target position in the image. Specifically, this includes: first, traversing each keyframe sample object; then, obtaining all annotated objects in the keyframe sample and aggregating them by instance object to obtain the 3D annotation of each instance at that moment; next, obtaining the multi-camera image data corresponding to the keyframe sample; then, based on the calibration parameters (extrinsic / intrinsic parameters) between the LiDAR and the camera and the carrier pose information, converting and projecting the 3D annotation results onto the image planes of each camera; finally, outputting a visualized image or verification result containing the projection results to determine whether the 3D annotation projection position is consistent with the actual target position in the image.

[0046] Once the consistency check and projection verification pass, a custom training dataset is output that can be used for existing BEV-aware training code. This dataset is compatible with the public dataset in terms of fields, levels, and indexing logic, and supports training for new categories.

[0047] S54. After the verification is passed, the BEV perception model is trained based on the proprietary training dataset; the training method may include the following steps: 1) Data loading: Load the dataset output in step S5 using the existing BEV perception training code, and parse the scene, samples, multi-sensor data and annotation information according to the metadata file to construct training sample input and supervision signals.

[0048] 2) Multi-sensor input construction: For each training sample, read the LiDAR point cloud data corresponding to the key frame sample and the multi-camera image data matched with its time to form multi-sensor training input.

[0049] 3) Model training: Input the multi-sensor training input into the BEV perception model for training, calculate the loss based on the corresponding labeled information and update the model parameters until the preset training conditions are met.

[0050] S55. While maintaining data structure compatibility, expand the target category set and perform full training on the model to support the recognition of new categories. Specifically, this may include: S551, Category Set Expansion: Add at least one new category to the existing target category set, and assign a corresponding identifier to the new category in the category definition information so that the new category can be referenced by the labeled object; S552, Labeling and Sample Update: For samples containing new categories, generate or supplement the 3D labeling information of the new categories, and maintain a consistent reference relationship with the original data in the metadata index link so that the new category labels can be correctly read by the training code along with the sample objects; S553, Full Training and Update: The parameters of the BEV perception model are updated using full training, that is, the parameters of each layer in the model are trained and updated so that the model can recognize both the original category and the newly added category.

[0051] In some embodiments of the present invention, in step S2, the data writing process of each sensor is completed by an independent thread or independent process, so that the data writing delay of one sensor will not affect the data acquisition and storage of other sensors; in step S3, the time identifier is recorded in the file name of the corresponding data file. Example

[0052] Taking a vehicle-mounted multi-sensor system as an example, this paper details the specific implementation process of the BEV perception training method based on time synchronization and proprietary dataset.

[0053] 1. Hardware Configuration The multi-sensor acquisition system used in this embodiment includes the following hardware components: One lidar sensor; 6-channel GMSL2 cameras; An industrial control computer running the Linux operating system; A GNSS timing device.

[0054] The sampling frequency of each of the above sensors is configured to 20fps to ensure that the time density of data acquisition meets the training requirements of the BEV perception model.

[0055] 2. Time synchronization configuration Perform time synchronization configuration according to the method described in step S1 of this invention: The GNSS timing device outputs two types of timing signals: a PPS second pulse signal and a time message containing UTC time information. The PPS signal is connected to the pulse input interface of the industrial control computer, and the time message is connected to the serial communication interface of the industrial control computer.

[0056] The industrial control computer triggers a system time calibration action based on the PPS signal at the whole second boundary, aligning the system time to the whole second boundary; at the same time, it combines the UTC whole second information provided by the time message to determine the whole second part of the current time, and combines the whole second part with the aligned sub-second part to form a unified system time reference.

[0057] During the data acquisition process, all sensors (LiDAR and 6 cameras) generate corresponding time stamps based on the unified system time, providing an accurate time reference for subsequent soft synchronization.

[0058] 3. Online data acquisition and disk loading Online data collection and storage shall be performed according to the method described in step S2 of this invention: The LiDAR and the 6 cameras are each processed and stored by independent threads. The threads do not block each other, ensuring that the data write delay of one sensor will not affect the data acquisition of other sensors.

[0059] The specific storage method is as follows: The camera's raw data is saved directly in YUV format without online format conversion or compression. LiDAR data is saved directly in the raw PCD format; A timestamp is written to the filename, serving as a unique time identifier for each frame of data.

[0060] During the online acquisition phase, no keyframe segmentation, format conversion, or complex data encapsulation operations are performed; only the raw data is directly written to disk. This minimal processing approach effectively reduces CPU load and memory cache usage, avoiding frame loss during acquisition.

[0061] 4. Offline processing and soft time synchronization Offline data processing and time soft synchronization are performed according to the method described in step S3 of this invention: Offline reading and processing of the collected and stored raw data: Convert image data from YUV format to RGB format; Perform image distortion correction; Convert the LiDAR PCD data into the target point cloud format required for training; Based on the lidar timestamp, perform soft time synchronization on all sensor data.

[0062] Based on time synchronization, key frames are selected according to the time sequence of the LiDAR. Specifically, this includes: using the LiDAR as the main sensor, selecting key frames of the LiDAR at preset time intervals; for each key frame of the LiDAR, based on the principle of temporal proximity, finding the image frame in the data sequence of the 6 cameras that is closest to the time marker of the key frame of the LiDAR, and using it as the key frame of the corresponding camera; and determining the data frames in each sensor data sequence that are not selected as key frames as intermediate frames, which are used for subsequent annotation propagation or training sample expansion.

[0063] 5. 3D annotation Three-dimensional annotation of lidar keyframes is performed according to the method described in step S4 of this invention: The key frames of the LiDAR determined in the offline processing stage are selected as the 3D annotation objects. The 3D annotation tools are used to annotate the target objects based on the LiDAR point cloud data. The annotation content includes at least the 3D spatial position, size (length, width, height) and attitude (heading angle) information of the target objects.

[0064] After completing the 3D annotation of the LiDAR keyframes, the annotation results are projected onto six camera images time-matched to the keyframe for visual verification. The camera images serve only as auxiliary visual verification and do not participate in the annotation generation process. Annotators check and confirm the accuracy of the 3D annotations based on the LiDAR point cloud viewpoint and the visualization results projected onto the images.

[0065] The confirmed 3D annotation results of the lidar keyframes are associated with the time stamp and frame index of the corresponding keyframes, which are then used to perform annotation propagation or interpolation processing on intermediate frames between keyframes.

[0066] 6. Data Structure Construction and Index Closure Construct a data structure that is fully compatible with the publicly available multi-sensor dataset according to the method described in steps S51 to S52 of this invention: Define the following multi-level data objects: Scene object: Describes a sequence of continuously collected data; Sample object: Describes discrete time point data in the scene; Sensor data objects (Sample Data): Describe data collected by different sensors at the same point in time; Sample Annotation: Describes the spatial annotation information of the target object in the sample; Instance object: describes the same physical target that appears continuously across time.

[0067] The definition of the above hierarchical objects is consistent with existing public multi-sensor datasets (such as nuScenes) in terms of field naming, hierarchical relationships, and referencing methods.

[0068] Generate globally unique identifiers (tokens) for each level of object and establish the following dual-link index relationship: Sample time series link: For each scenario, the sample objects are sorted according to the time identifier, and a reference relationship between the previous sample (prev) and the next sample (next) is established for each sample object to form a time link in the sample dimension. Instance annotation chain: For each target instance object, establish the annotation sequence of the instance at different sample times, establish the reference relationship between the previous annotation (prev) and the next annotation (next) for the annotation objects of the same instance, and form a continuous chain at the instance dimension.

[0069] Consistency constraints are applied to the aforementioned dual-link system to ensure that the index relationships form a closed loop. The generated metadata file can be directly loaded and parsed by existing BEV-aware training code.

[0070] 7. Full projection verification Perform a full verification according to the method described in step S53 of this invention: Load all metadata files, establish memory index mapping relationships for objects at each level, and perform the following verifications: Structural integrity check: Verify the global uniqueness of each object's unique identifier and the existence of referenced objects; Dual-link consistency verification: Traverse all sample objects along the sample link and check the reachability of the prev / next link; traverse all labeled objects along the instance link and check the reachability of the prev / next link and the correctness of the time sequence. Time matching verification: Check the time matching relationship between key frame samples and corresponding data from 6 cameras to ensure that the time difference meets the preset matching conditions; Projection verification: Perform full projection verification on the 3D annotations of each instance in each keyframe. The 3D annotation results are converted and projected onto the image plane of 6 cameras based on the calibration parameters (extrinsic / intrinsic parameters) and carrier pose information. Check whether the position of the projection frame is consistent with the actual target in the image.

[0071] If projection offset or misalignment is found, the source of the problem is traced back according to the index link (such as time synchronization error, external parameter calibration error or index association error), and corresponding corrections are made until all verifications pass.

[0072] 8. Model Training After the data verification is passed, the BEV perception model is trained according to the method described in step S54 of this invention: The existing BEV perception training code is used to load the generated proprietary dataset, and the scene, samples, multi-sensor data, and annotation information are parsed based on the metadata file. For each training sample, the LiDAR point cloud data corresponding to the keyframe sample and the time-matched image data from 6 cameras are read to form the multi-sensor training input.

[0073] The multi-sensor training inputs were fed into the BEV perception model for training, and the loss was calculated and model parameters updated based on the corresponding labeled information. Experimental results show that the model can converge normally, verifying that the method proposed in this invention can effectively use self-collected data for BEV perception model training without relying on existing public dataset collection processes.

[0074] In summary, this embodiment successfully achieved a complete construction process from a proprietary vehicle-mounted multi-sensor system to a trainable BEV perception dataset by employing a series of technical means, including high-precision time synchronization, independent parallel acquisition of multiple sensors and direct writing of raw data to disk, offline soft time synchronization and keyframe segmentation, 3D annotation and image-assisted verification, construction of a multi-level data structure fully compatible with public datasets, dual-link index closed loop, and full projection verification.

[0075] Experimental results show that: in terms of time synchronization, the system time calibration scheme based on PPS and UTC time messages controls the time deviation between the LiDAR and the six cameras to within milliseconds, effectively avoiding the inconsistency of BEV features caused by time misalignment; in terms of data acquisition, using independent thread acquisition and direct writing of raw data, no frame loss occurred in the test of continuous acquisition for 1 hour and a total data volume of over 2TB, and the data integrity rate reached 100%; in terms of data compatibility, the generated metadata file can be directly read and parsed by existing BEV perception training code based on the nuScenes data format without any code modification; in terms of model training, using the proprietary dataset constructed in this embodiment to train the BEV perception model, the model loss decreased normally and converged stably. Finally, the perception accuracy on the proprietary scene evaluation set is comparable to the baseline model trained using the public dataset, and the detection accuracy of some categories (such as near-range targets) is even better than the training results of the public dataset, verifying the training effectiveness and engineering practicality of the dataset constructed by this method.

[0076] Furthermore, this embodiment supports flexible expansion to new target categories (such as non-standard obstacles in specific areas). By simply adding corresponding annotations and performing full training while keeping the data structure unchanged, the model can acquire the ability to recognize new categories, demonstrating the good scalability of this method in practical autonomous driving applications.

[0077] Although embodiments of the invention have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the claims and their equivalents.

Claims

1. A time synchronization based self-owned dataset BEV perception training method, characterized in that, Includes the following steps: S1. Configure the time synchronization of the multi-sensor acquisition system to obtain a unified system time reference; S2. Online acquisition and storage of raw data from multiple sensors to ensure data integrity and avoid frame loss during the acquisition process; S3. The raw data collected is processed offline, and time soft synchronization is performed based on a unified time base to divide key frames and intermediate frames. S4. Perform 3D annotation on the keyframes of the lidar and use the annotation results as the basis for subsequent sample construction and annotation propagation; S5. Construct a multi-level training sample structure for BEV perception training and establish index relationships between samples to ensure that the self-collected data is compatible with existing public multi-sensor datasets in terms of fields, levels and index logic.

2. The time synchronization based ego dataset BEV perception training method according to claim 1, wherein, The time synchronization configuration of the multi-sensor acquisition system in step S1 specifically includes: S11. Obtain the timing signal output by the satellite positioning device, wherein the timing signal includes a second pulse signal and a time message containing UTC time information; S12. Connect the second pulse signal to the pulse input interface of the industrial control computer, connect the time message to the serial communication interface of the industrial control computer, and trigger the system time calibration action at the whole second boundary; S13. System time calibration based on second pulse signal and time message: When the second pulse signal is received, the system time is aligned to the whole second boundary, the whole second part is determined by combining the UTC whole second information provided by the time message, and the whole second part is combined with the aligned sub-second part to form the unified system time of the industrial control computer. S14. Use the unified system time after the industrial control computer is calibrated as the unified time reference for multi-sensor data acquisition, and generate time stamps for the data acquired by the lidar and camera respectively during the acquisition phase.

3. The time synchronization based ego dataset BEV perception training method of claim 1, wherein, The online acquisition and storage of raw data from multiple sensors in step S2 specifically includes: S21. A multi-sensor acquisition system including a lidar sensor and multiple camera sensors is configured, with an independent data acquisition and storage execution unit configured for each sensor. S22. During the acquisition process, the raw data output by each sensor is acquired respectively, and the raw data is not encapsulated, compressed or converted in format, or only minimized. S23. For each sensor, the raw data is directly written to the storage medium using the corresponding independent acquisition and storage execution unit, wherein the execution units do not block each other. S24. While the original data is being written to storage, a corresponding time identifier is generated for each frame of original data based on the unified system time. S25. The above method enables continuous online acquisition of raw data from multiple sensors, ensuring that the number of data frames acquired by each sensor within the preset acquisition time is consistent with the theoretical number of sampling frames or within the allowable error range.

4. The time synchronization based ego dataset BEV perception training method of claim 1, wherein, The specific steps in step S3, including performing time soft synchronization and dividing keyframes and intermediate frames, include: S31. In the offline processing stage, read the stored multi-sensor raw data. Each raw data contains a time identifier generated based on a unified system time. Perform offline format conversion processing on the raw data. S32. Based on the unified system time reference and the time identifier of each raw data, construct the correlation relationship of multi-sensor data in the time dimension; S33. Select a lidar as the main sensor, use the time stamp of the lidar data as the reference for time soft synchronization, and select lidar key frames in the lidar data sequence according to a preset time interval. S34. For each key frame of the LiDAR, based on the principle of temporal proximity, find the camera data frame in each camera data sequence that is closest to the time identifier of the key frame of the LiDAR and use it as the key frame of the corresponding camera. S35. Repeat the above keyframe matching process until the keyframe matching of the entire data sequence is completed. S36. Determine the data frames in each sensor data sequence that were not selected as keyframes as intermediate frames.

5. The time synchronization based ego dataset BEV perception training method of claim 1, wherein, The specific steps of performing 3D annotation on the keyframes of the lidar in step S4 include: S41. Select the key frame of the lidar determined in step S3 as the 3D annotation object; S42. For the key frame of the lidar, perform three-dimensional annotation on the target object based on the lidar point cloud data. The three-dimensional annotation includes at least the spatial position, size and attitude information of the target object. S43. Project the completed 3D annotation results onto the multi-camera images corresponding to the keyframe time for visual inspection; S44. Based on the LiDAR point cloud perspective and the visualization results after projection onto the camera image, check and confirm the 3D annotation results. S45. Associate the confirmed 3D annotation results of the lidar keyframes with the time stamp and frame index of the corresponding keyframes for subsequent annotation propagation on intermediate frames.

6. The time synchronization based ego dataset BEV perception training method of claim 1, wherein, The construction of the multi-level training sample structure in step S5 specifically includes: S51. Define a multi-level data object including scene object, sample object, sensor data object, instance object and annotation object, and establish a hierarchical relationship of "scene-sample-multi-sensor data-annotation-instance". The definition of the hierarchical objects is consistent with the existing public multi-sensor dataset. S52. Generate unique identifiers for objects at each level, and construct a first link based on the sample time series and a second link based on the target instance respectively, so that the index relationship forms a closed loop; S53 performs consistency checks and projection verifications on the generated metadata files to ensure that index relationships, time associations, and extrinsic / pose associations are correct. S54. After the verification is passed, the BEV perception model is trained based on the proprietary training dataset. S55. While maintaining data structure compatibility, expand the target category set and perform full training on the model to support the recognition of new categories.

7. The time synchronization based ego dataset BEV perception training method according to claim 6, characterized in that, The specific steps in step S52 of constructing the dual-link index relationship include: S521. Generate globally unique identifiers for scene objects, sample objects, sensor data objects, target instance objects, annotation objects, sensor configuration objects, and pose / extrinsic parameter objects respectively. S522. For each scene object, sort the sample objects according to the time identifier, and establish a reference relationship between the previous sample and the subsequent sample for each sample object to form a sample time series link. S523. For each sensor channel, sort the sensor data objects according to the time identifier, establish a previous / next sequence reference relationship, and establish a reference relationship between each sensor data object and its corresponding sample object. S524. For each target instance object, establish the annotation sequence link of the instance at different sample times, establish the reference relationship between the preceding and subsequent annotations for the annotation objects of the same instance, and form the instance annotation link. S525. Apply consistency constraints to sample links and instance links, including: any sample object referenced by any labeled object must exist in the sample time series link of its scene; the sample time corresponding to the preceding / following labeled objects of any labeled object in the instance link must satisfy the time sequence; for two adjacent frame labeled objects of the same instance, their corresponding sample objects should satisfy the reachability relationship in the sample time series link.

8. The BEV perception training method based on time synchronization and proprietary dataset according to claim 6, characterized in that, The consistency verification and projection verification in step S53 specifically include: S531. Load all metadata files and perform structural integrity checks, including at least global uniqueness checks of unique identifiers, existence checks of referenced objects, and consistency checks of the number of objects. S532. Based on the sample time series links and instance annotation links, perform dual-link consistency verification to check link reachability, unbroken links, and the correctness of time sequence. S533. For key frame samples, check whether the matching relationship between the key frame samples and the corresponding multi-camera data is valid based on the time soft synchronization results. S534 performs projection verification on the 3D annotations of keyframes, projects the 3D annotation results onto the image planes of each camera based on calibration parameters, and checks whether the projected position is consistent with the actual position of the target in the image.

9. The BEV perception training method based on time synchronization and proprietary dataset according to claim 6, characterized in that, The category expansion and full training in step S55 specifically include: S551. Add at least one new category to the existing target category set, and assign a corresponding identifier to the new category in the category definition information; S552. For samples containing newly added categories, generate or supplement the three-dimensional annotation information of the newly added categories, and maintain a consistent reference relationship with the original data in the metadata index link; S553. Update the parameters of the BEV perception model using full training, so that the model can recognize both the original and newly added categories.

10. The BEV perception training method based on time synchronization and proprietary dataset according to claim 1, characterized in that, In step S2, the data writing process of each sensor is completed by an independent thread or process, so that the data writing delay of one sensor will not affect the data acquisition and storage of other sensors; in step S3, the time identifier is recorded in the file name of the corresponding data file.

Images