Intelligent driving multi-sensor fusion data processing system

Abstract

The invention discloses an intelligent driving multi-sensor fusion data processing system, relates to the technical field of intelligent driving, and solves the problems that time synchronization of multi-sensor heterogeneous data, scientific completion of missing data and dynamic optimization of computing power distribution cannot be accurately solved. According to the invention, through a standardized period calibration and translation adaptation mechanism of a time feature verification end, a main timeline is constructed by taking a maximum acquisition period as a reference, an optimal solution of coincident acquisition points is locked through multi-round translation verification, the translation features of the timelines of all sensors are accurately calibrated, and the accuracy of time line calibration is improved. The problem of time asynchronization caused by acquisition period difference of different sensors is effectively eliminated; and by matching with a linear interpolation processing end to accurately complement the acquired neutral period data, the data integrity and consistency of the sensors on the unified time dimension are ensured, a high-precision data foundation is laid for subsequent fusion processing, and fusion errors caused by time deviation are prevented from being introduced.

Description

Technical Field

[0001] This invention relates to the field of intelligent driving technology, specifically to an intelligent driving multi-sensor fusion data processing system. Background Technology

[0002] The core of intelligent driving technology lies in achieving accurate cognition of complex traffic environments through multi-sensor collaborative perception. However, due to differences in their operating principles, sensors such as lidar, cameras, and millimeter-wave radar generally exhibit heterogeneous characteristics in terms of acquisition frequency, data format, and time reference, which poses a fundamental challenge to the fusion of multi-source data.

[0003] Currently, most intelligent driving multi-sensor data processing solutions have significant shortcomings: Firstly, the time synchronization mechanism is rudimentary, relying mainly on the sensor's own crystal oscillator clock or simple timestamp alignment, without fully considering the differences in the acquisition cycles of different sensors, which can easily lead to asynchronous data time and thus introduce fusion errors. Secondly, regarding the issue of data loss during the sensor acquisition gap, there is a lack of accurate completion strategies, which often results in data breaks or interpolation distortion, affecting the continuity and integrity of the fused data. Third, the allocation of computing power lacks scientific rigor, often using fixed weights or empirical values ​​instead of dynamically adapting to the actual processing needs of different data types. This results in some redundant computing power for data processing and may lead to insufficient fusion rates for critical data, making it difficult to meet the stringent real-time requirements of intelligent driving. In addition, the core processing logic of existing solutions (such as time calibration and weight confirmation) lacks standardized and implementable execution processes, has poor flexibility in adapting to different sensor combinations, and requires a lot of manual debugging during deployment, which increases application costs and implementation difficulty.

[0004] Therefore, developing a processing system that can accurately solve the time synchronization of heterogeneous data from multiple sensors, scientifically complete missing data, and dynamically optimize computing power allocation has become a key requirement for improving the accuracy and real-time performance of intelligent driving environment perception and promoting the large-scale application of intelligent driving technology. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention provides an intelligent driving multi-sensor fusion data processing system that solves the problems of inaccurate time synchronization of heterogeneous data from multiple sensors, scientific completion of missing data, and dynamic optimization of computing power allocation.

[0006] To achieve the above objectives, the present invention provides the following technical solution: an intelligent driving multi-sensor fusion data processing system, comprising: At the data feature extraction end, the acquisition cycle associated with the corresponding sensor is determined based on the acquisition frequency associated with different sensors. Specifically, the method is as follows: The sampling frequency involved in each group of sensors is labeled as P. i , where i represents different sensors; Use: T i =1÷P i Confirm the acquisition period T associated with the corresponding sensor i ; The time feature verification end, based on the different acquisition cycles of different sensors, determines the overlap process of acquired data from multiple sensors within a certain period by shifting the timeline forward and backward. This confirms the shift characteristics for each sensor and executes the verification. The specific method is as follows: Using the current time as the calibration time, based on the acquisition period involved in each group of sensors, an acquisition timeline associated with the corresponding sensor is generated. From the acquisition periods involved in several sensors, the maximum value is selected, and the maximum acquisition period is recorded as the standard period. The acquisition timeline associated with the standard period is recorded as the standard timeline. Starting from the initial acquisition time within the standard timeline, four groups of acquisition times are confirmed step by step. The time length associated with the five groups of acquisition times is recorded as the standard time length. The timeline associated with the standard time length is recorded as the main timeline. From the standard timelines involved in other sensors, we identify the timelines with the same standard time length starting from the first initial acquisition time and record them as the corresponding sensor's calibration timeline. The main timeline and the verification timeline are shifted forward and backward for confirmation. A standard process is selected from several forward and backward shift processes. The selection methods include: While keeping the time characteristics of the main timeline unchanged, the verification timeline is shifted back and forth for confirmation. Several sets of shift confirmation processes are executed, and the overlapping acquisition points associated with each set of shift confirmation processes are marked. The overlapping acquisition points are the acquisition points that have the same time on different timelines. The total number of overlapping acquisition points marked in the corresponding shift confirmation process is recorded as the process feature of the corresponding shift confirmation process. From the process features associated with different shift confirmation processes, the maximum value is selected, and the shift confirmation process associated with the maximum value is recorded as the standard process. The translation characteristics of each set of calibration timelines are identified from the standard process. Based on these translation characteristics, the acquisition status of each set of sensors is recalibrated so that the parameters of each set of sensors are relatively uniform within the corresponding time period. The linear interpolation processing unit confirms the data collected by each group of sensors and, in conjunction with the acquisition times of other sensors, identifies the associated data of the corresponding sensors during the acquisition gap period, and performs linear interpolation processing. The specific method is as follows: The data collected by each group of sensors is confirmed and the collection time is identified. Then, it is confirmed whether other sensors have collected data at the same time. If they do, no processing is required. If they do not, linear interpolation is performed on the specified sensor. Mark the acquisition time as C0, the acquisition time of the specified sensor before C0 as C1, the acquisition time of the specified sensor after C0 as C2, and the acquisition data associated with C1 as SJ1, and the acquisition data associated with C2 as SJ2. use: Confirm the associated data SJ0 corresponding to the acquisition time C0, and perform linear interpolation based on the confirmed associated data SJ0 to facilitate subsequent data synchronization confirmation.

[0007] Preferred options also include: The feature fusion processing end, for multiple sets of acquired data associated with the same acquisition time, identifies the fusion features involved in each set of acquired data from historical data, determines the fusion weights associated with different acquired data based on the fusion features, and transmits the determined fusion weights to the dynamic weight allocation end, including: The data types of multiple sets of collected data at the same collection time are recorded as undetermined types. The processing data involved in the undetermined types are identified from historical data. Different batches of fused data are extracted from the processing data. The execution computing power and the resulting average fusion rate are confirmed from the fused data. The fusion characteristics of this fused data are confirmed by the processing method of: average fusion rate ÷ execution computing power = fusion characteristics. The maximum value is extracted from the fusion characteristics associated with different fused data. The execution computing power associated with the maximum value is recorded as the fusion characteristics involved in the corresponding undetermined type. Based on the different fusion features associated with different undetermined types, multiple sets of fusion features are processed by ratio analysis to confirm the feature ratio columns associated with multiple different undetermined types. Then, the inherent computing power set in the current intelligent driving system is identified, and the inherent computing power is evenly distributed according to the ratio arrangement of the feature ratio columns. The allocated computing power associated with each undetermined type is confirmed. If the allocated computing power associated with each undetermined type exceeds the fusion feature, the associated computing power with the same value as the fusion feature is directly extracted from the inherent computing power as the allocated computing power associated with the corresponding undetermined type. If the allocated computing power associated with each undetermined type does not exceed the fusion feature, the confirmed allocated computing power remains unchanged. The allocated computing power for each undetermined type is recorded as the fusion weight of the corresponding collected data, and the confirmed fusion weight is transmitted to the dynamic weight allocation terminal.

[0008] Preferably, the dynamic weight allocation terminal allocates the inherent computing power within the intelligent driving system according to the different fusion weights associated with different collected data, thereby completing the unified fusion processing of different collected data.

[0009] This invention provides an intelligent driving multi-sensor fusion data processing system. Compared with the prior art, it has the following advantages: This invention utilizes a standardized periodic calibration and translation adaptation mechanism at the time feature verification end to construct the main timeline based on the maximum acquisition period. Through multiple rounds of translation verification, it locks the optimal solution for overlapping acquisition points, accurately calibrates the translation characteristics of each sensor's timeline, and effectively eliminates the time asynchrony problem caused by differences in the acquisition periods of different sensors. Combined with the linear interpolation processing end, it accurately completes the data during the acquisition gap period, ensuring the data integrity and consistency of each sensor in a unified time dimension. This lays a high-precision data foundation for subsequent fusion processing and avoids fusion errors introduced by time deviations. Based on historical data extraction and fusion features, the processing requirements for different data types are determined through quantification, and then the inherent computing power is precisely allocated by combining feature ratios. The dynamic weight allocation terminal adapts the computing power according to the fusion weight, ensuring that the fusion rate of each type of data meets the requirements while avoiding redundant and wasted computing power. This "on-demand allocation" computing power scheduling mode effectively improves computing power utilization, significantly shortens the unified fusion processing time of multi-sensor data, and adapts to the stringent real-time requirements of intelligent driving. Attached Figure Description

[0010] Figure 1 This is a schematic diagram of the principle framework of the present invention. Detailed Implementation

[0011] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0012] First Embodiment Please see Figure 1 This application provides an intelligent driving multi-sensor fusion data processing system, including a data feature extraction end, a time feature verification end, a linear interpolation processing end, a feature fusion processing end, and a dynamic weight allocation end, wherein the data feature extraction end, the time feature verification end, the linear interpolation processing end, the feature fusion processing end, and the dynamic weight allocation end are electrically connected from the output node to the input node in sequence. The data feature extraction end identifies the sensors involved in the intelligent driving process, recognizes the acquisition frequencies associated with different sensors from the identified multiple sensor groups, and then confirms the acquisition period associated with the corresponding sensor. The specific confirmation method is as follows: The sampling frequency involved in each group of sensors is labeled as P. i , where i represents different sensors; Use: T i =1÷P i Confirm the acquisition period T associated with the corresponding sensor i .

[0013] Among them, the time feature verification end locks the acquisition period involved by the corresponding sensor, and based on the different acquisition periods involved by different sensors, confirms the overlap process of the acquired data of multiple groups of sensors within a certain period by shifting the timeline forward and backward, thereby confirming the shift characteristics of each sensor and executing the procedure.

[0014] In the linear interpolation processing end, after the translation features of each group of sensors have been calibrated, the data collected by each group of sensors is confirmed, and combined with the collection time of other sensors, the associated data of the corresponding sensor during the collection gap period is identified and linear interpolation processing is performed. Among them, the feature fusion processing end, for multiple sets of collected data associated with the same collection time, confirms the fusion features involved in each set of collected data from historical data, and confirms the fusion weights associated with different collected data based on the fusion features, and transmits the confirmed fusion weights to the dynamic weight allocation end. Among them, the dynamic weight allocation end allocates the inherent computing power in the intelligent driving system according to the different fusion weights associated with different collected data, and completes the unified fusion processing of different collected data.

[0015] Second Embodiment In its specific implementation, compared to the above embodiments, this embodiment mainly focuses on the confirmation process of the time verification feature at the time stop verification end, which specifically includes: The specific method for verifying the time calibration characteristics of each sensor at the time feature verification end is as follows: Using the current time as the calibration time, based on the acquisition period involved in each group of sensors, an acquisition timeline associated with the corresponding sensor is generated. From the acquisition periods involved in several sensors, the maximum value is selected, and the maximum acquisition period is recorded as the standard period. The acquisition timeline associated with the standard period is recorded as the standard timeline. Starting from the initial acquisition time within the standard timeline, four groups of acquisition times are confirmed step by step. The time length associated with the five groups of acquisition times is recorded as the standard time length. The timeline associated with the standard time length is recorded as the main timeline. From the standard timelines involved in other sensors, we identify the timelines with the same standard time length starting from the first initial acquisition time and record them as the corresponding sensor's calibration timeline. Forward and backward translation confirmation of the main timeline and the verification timeline: Keep the time characteristics of the main timeline unchanged, perform forward and backward translation confirmation of the verification timeline, execute several sets of translation confirmation processes, and mark the overlapping acquisition points associated with each set of translation confirmation processes. The overlapping acquisition points are the acquisition points that have the same time on different timelines (that is, the timelines associated with different sensors are all in the acquisition state at the same time). Record the total number of overlapping acquisition points marked in the corresponding translation confirmation process as the process feature of the corresponding translation confirmation process. Select the maximum value from the process features associated with different translation confirmation processes, and record the translation confirmation process associated with the maximum value as the standard process. The translation characteristics (how much time to move forward or backward) of each set of calibration timelines are identified from the standard process. Based on these translation characteristics, the acquisition status of each set of sensors is recalibrated so that the parameters of each set of sensors are relatively uniform within the corresponding time period. Specifically, each different timeline exhibits forward or backward translation characteristics during the corresponding time-before-and-after translation confirmation process. Based on these translation characteristics, the acquisition status of each group of sensors is adjusted and verified, so that each group of sensors is in a relatively uniform state within the corresponding acquisition period, thereby achieving the optimal parameter acquisition effect.

[0016] Third Embodiment In the specific implementation process, compared with the above embodiments, this embodiment mainly focuses on the interpolation process of the linear interpolation processing end for the associated data; Specifically, the linear interpolation processing unit confirms the correlation data of different sensors during the data acquisition gap period as follows: The data collected by each group of sensors is confirmed and the collection time is identified. Then, it is confirmed whether other sensors have collected data at the same time. If they do, no processing is required. If they do not, linear interpolation is performed on the specified sensor. Mark the acquisition time as C0, the acquisition time of the specified sensor before C0 as C1, the acquisition time of the specified sensor after C0 as C2, and the acquisition data associated with C1 as SJ1, and the acquisition data associated with C2 as SJ2. use: Confirm the associated data SJ0 corresponding to the acquisition time C0, and perform linear interpolation based on the confirmed associated data SJ0 to facilitate subsequent data synchronization confirmation; Specifically, in the data fusion process, a certain sensor may collect data at a certain time. After time calibration, a certain sensor may also collect data, but sensor C may not collect data. Therefore, it is necessary to confirm the features based on the data collected by sensor C at the previous and next collection times at the current time, so as to confirm the related data at the corresponding time. This will achieve the best data synchronization confirmation effect, which will facilitate the subsequent linear interpolation processing of the related data to achieve a better data processing effect.

[0017] Fourth embodiment In the specific implementation process, compared with the above embodiments, this embodiment mainly focuses on the confirmation process of fusion weight, and its main execution end is the feature fusion processing end; The feature fusion processing stage includes the following steps for confirming the fusion features of different collected data: The data types of multiple sets of collected data at the same collection time are recorded as undetermined types. The processing data involved in the undetermined types are identified from historical data. Different batches of fused data are extracted from the processing data. The execution computing power and the resulting average fusion rate are confirmed from the fused data. The fusion characteristics of this fused data are confirmed by the processing method of: average fusion rate ÷ execution computing power = fusion characteristics. The maximum value is extracted from the fusion characteristics associated with different fused data. The execution computing power associated with the maximum value is recorded as the fusion characteristics involved in the corresponding undetermined type. Based on the different fusion features associated with different undetermined types, multiple sets of fusion features are processed by ratio analysis to confirm the feature ratio columns associated with multiple different undetermined types. Then, the inherent computing power set in the current intelligent driving system is identified, and the inherent computing power is evenly distributed according to the ratio arrangement of the feature ratio columns. The allocated computing power associated with each undetermined type is confirmed (the ratio generated by multiple allocated computing power in the subsequent ratio analysis process is consistent with the feature ratio columns). If the allocated computing power associated with each undetermined type exceeds the fusion feature, the associated computing power with the same value as the fusion feature is directly extracted from the inherent computing power as the allocated computing power associated with the corresponding undetermined type. If the allocated computing power associated with each undetermined type does not exceed the fusion feature, the confirmed allocated computing power remains unchanged. The allocated computing power for each undetermined type is recorded as the fusion weight of the corresponding collected data, and the confirmed fusion weight is transmitted to the dynamic weight allocation terminal. Specifically, each different pending type has a different fusion weight during the feature confirmation process. Based on the confirmed fusion weight, the fusion rate of each pending type in the fusion processing process is fully guaranteed, so that the corresponding system can achieve the best fusion effect and significantly reduce the fusion time when performing data fusion processing.

[0018] Some of the data in the above formulas are numerical calculations with dimensions removed, and the contents not described in detail in this specification are all prior art known to those skilled in the art.

[0019] The above embodiments are only used to illustrate the technical methods of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical methods of the present invention without departing from the spirit and scope of the technical methods of the present invention.

Claims

1. A multi-sensor fusion data processing system for intelligent driving, characterized in that, include: At the data feature extraction end, the acquisition cycle associated with the corresponding sensor is determined based on the acquisition frequency associated with different sensors. The time feature verification end, based on the different acquisition cycles of different sensors, confirms the overlap process of the acquired data of multiple sets of sensors within a certain period by shifting the timeline forward and backward, thereby confirming the shift characteristics of each sensor and executing the verification. The linear interpolation processing unit confirms the data collected by each group of sensors, and, in conjunction with the collection times of other sensors, identifies the associated data of the corresponding sensors during the collection gap period, and performs linear interpolation processing.

2. The intelligent driving multi-sensor fusion data processing system according to claim 1, characterized in that, The specific method for confirming the collection period at the data feature extraction end is as follows: The sampling frequency involved in each group of sensors is labeled as P. i , where i represents different sensors; Use: T i =1÷P i Confirm the acquisition period T associated with the corresponding sensor i .

3. The intelligent driving multi-sensor fusion data processing system according to claim 1, characterized in that, The specific method by which the time feature verification terminal confirms the time calibration features of each sensor is as follows: Using the current time as the calibration time, based on the acquisition period involved in each group of sensors, an acquisition timeline associated with the corresponding sensor is generated. From the acquisition periods involved in several sensors, the maximum value is selected, and the maximum acquisition period is recorded as the standard period. The acquisition timeline associated with the standard period is recorded as the standard timeline. Starting from the initial acquisition time within the standard timeline, four groups of acquisition times are confirmed step by step. The time length associated with the five groups of acquisition times is recorded as the standard time length. The timeline associated with the standard time length is recorded as the main timeline. From the standard timelines involved in other sensors, we identify the timelines with the same standard time length starting from the first initial acquisition time and record them as the corresponding sensor's calibration timeline. The main timeline and the verification timeline are shifted forward and backward for confirmation, and a standard process is selected from several forward and backward shifting processes; The translation characteristics of each set of calibration timelines are identified from the standard process, and the acquisition status of each set of sensors is recalibrated based on these translation characteristics.

4. The intelligent driving multi-sensor fusion data processing system according to claim 3, characterized in that, The selection methods for the standard process include: While maintaining the time characteristics of the main timeline, the verification timeline is shifted back and forth for confirmation. Several sets of shift confirmation processes are executed, and the overlapping acquisition points associated with each set of shift confirmation processes are marked. The overlapping acquisition points are the acquisition points that have the same time on different timelines. The total number of overlapping acquisition points marked in the corresponding shift confirmation process is recorded as the process feature of the corresponding shift confirmation process. From the process features associated with different shift confirmation processes, the maximum value is selected, and the shift confirmation process associated with the maximum value is recorded as the standard process.

5. The intelligent driving multi-sensor fusion data processing system according to claim 3, characterized in that, The linear interpolation processing terminal confirms the correlation data of different sensors during the data acquisition gap period in the following specific way: The data collected by each group of sensors is confirmed and the collection time is identified. Then, it is confirmed whether other sensors have collected data at the same time. If they do, no processing is required. If they do not, linear interpolation is performed on the specified sensor. Mark the acquisition time as C0, the acquisition time of the specified sensor before C0 as C1, the acquisition time of the specified sensor after C0 as C2, and the acquisition data associated with C1 as SJ1, and the acquisition data associated with C2 as SJ2. use: Confirm the associated data SJ0 corresponding to the acquisition time C0, and perform linear interpolation based on the confirmed associated data SJ0 to facilitate subsequent data synchronization confirmation.

6. The intelligent driving multi-sensor fusion data processing system according to claim 1, characterized in that, Also includes: The feature fusion processing end identifies the fusion features involved in each set of collected data from historical data for multiple sets of collected data associated with the same collection time, and determines the fusion weight associated with different collected data based on the fusion features, and transmits the confirmed fusion weight to the dynamic weight allocation end.

7. The intelligent driving multi-sensor fusion data processing system according to claim 6, characterized in that, The feature fusion processing terminal includes the following confirmation process for the fusion features of different collected data: The data types of multiple sets of collected data at the same collection time are recorded as undetermined types. The processing data involved in the undetermined types are identified from historical data. Different batches of fused data are extracted from the processing data. The execution computing power and the resulting average fusion rate are confirmed from the fused data. The fusion characteristics of this fused data are confirmed by the processing method of: average fusion rate ÷ execution computing power = fusion characteristics. The maximum value is extracted from the fusion characteristics associated with different fused data. The execution computing power associated with the maximum value is recorded as the fusion characteristics involved in the corresponding undetermined type.

8. The intelligent driving multi-sensor fusion data processing system according to claim 7, characterized in that, The specific method by which the feature fusion processing terminal confirms the fusion weights of different collected data is as follows: Based on the different fusion features associated with different undetermined types, multiple sets of fusion features are processed by ratio analysis to confirm the feature ratio columns associated with multiple different undetermined types. Then, the inherent computing power set in the current intelligent driving system is identified, and the inherent computing power is evenly distributed according to the ratio arrangement of the feature ratio columns. The allocated computing power associated with each undetermined type is confirmed. If the allocated computing power associated with each undetermined type exceeds the fusion feature, the associated computing power with the same value as the fusion feature is directly extracted from the inherent computing power as the allocated computing power associated with the corresponding undetermined type. If the allocated computing power associated with each undetermined type does not exceed the fusion feature, the confirmed allocated computing power remains unchanged. The allocated computing power for each undetermined type is recorded as the fusion weight of the corresponding collected data, and the confirmed fusion weight is transmitted to the dynamic weight allocation terminal.

9. The intelligent driving multi-sensor fusion data processing system according to claim 6, characterized in that, The dynamic weight allocation terminal allocates the inherent computing power within the intelligent driving system based on the different fusion weights associated with different collected data, thereby completing the unified fusion processing of different collected data.

Images