A multi-sensor-based target detection method, system and vehicle-mounted terminal

Abstract

This application provides a multi-sensor-based target detection method, system, and vehicle terminal. The multi-sensor-based target detection method includes: acquiring initial sensor data from multiple sensors based on a primary time alignment rule; then processing the initial sensor data sequentially according to spatial alignment and secondary time alignment rules to obtain target sensor data; further acquiring input data based on the target sensor data and inputting the input data into a Transformer model to output fused data as the target detection result. Compared with existing rule-based post-fusion algorithms, this application exhibits greater stability and robustness under conditions of sensor characteristic changes and data anomalies. Combined with deep fusion of multi-sensor data, it enhances the perception and understanding of the environment.

Description

Technical Field

[0001] This application relates to the field of autonomous driving technology, and in particular to a target detection method, system and vehicle terminal based on multiple sensors. Background Technology

[0002] Current multi-sensor fusion technology in autonomous driving systems aims to integrate data from multiple sensors to reduce measurement errors and uncertainties from individual sensors, thereby improving data accuracy and reliability. Post-fusion algorithms then make decisions and combine the target outputs from each sensor to obtain a more accurate single-frame target, and perform multi-frame tracking of the target.

[0003] However, when sensor characteristics change, the results of rule-based and prior knowledge-based post-fusion algorithms may also change. Furthermore, these algorithms rely too heavily on single-frame target results provided by sensors; when an attribute provided by a particular sensor is abnormal, the post-fusion results may exhibit multi-frame unsmoothness, failing to utilize data accumulated from multiple sensors. Summary of the Invention

[0004] To address the aforementioned technical problems, this application provides a multi-sensor-based target detection method, system, and vehicle terminal that can fuse data from different sensors to achieve accurate perception and understanding of the environment.

[0005] Specifically, this application provides a target detection method based on multiple sensors, comprising the following steps:

[0006] Initial sensor data from multiple sensors were acquired based on a single time alignment rule.

[0007] The initial sensor data is processed sequentially according to spatial alignment rules and secondary time alignment rules to obtain target sensor data.

[0008] Furthermore, input data is acquired based on the target sensor data, and the input data is input into the Transformer model to output fused data as the target detection result.

[0009] In the above technical solution, compared with the traditional rule-based post-fusion algorithm, this application is more stable and robust under the conditions of sensor characteristic changes and data anomalies. Combined with the deep fusion of multi-sensor data, it improves the ability to perceive and understand the environment.

[0010] Furthermore, each sensor is pre-configured with a preset server; the initial sensor data acquisition from multiple sensors based on a single time alignment rule includes:

[0011] The sensor synchronizes its clock with the preset server based on a preset clock synchronization protocol.

[0012] Furthermore, during the acquisition of initial sensor data, a first timestamp is applied to each frame of the initial sensor data based on the synchronization clock, so as to align the initial sensor data through the first timestamp; wherein, the initial sensor data includes at least initial radar data, initial laser data and initial camera data.

[0013] In the above technical solution, based on the preset clock synchronization protocol, the timestamps of the sensors and the server are highly synchronized, improving the accuracy of time alignment; based on the synchronized clock, each frame of data is stamped with the first timestamp, ensuring the precise alignment of the data of each sensor in the time dimension and improving the accuracy of data fusion; the application of multi-source data enhances the system's ability to perceive the environment, which helps to improve the safety and reliability of the autonomous driving system.

[0014] Furthermore, the initial camera data includes at least first camera data and second camera data; after acquiring the initial sensor data, it also includes:

[0015] The first camera data and the second camera data are perceptually fused to update the initial camera data.

[0016] In the above technical solution, by fusing the data output by the camera, the differences in the camera data output results can be effectively eliminated, thereby ensuring the accuracy and reliability of the final target detection results.

[0017] Furthermore, the processing of the initial sensor data according to spatial alignment rules includes:

[0018] Obtain the intrinsic and extrinsic parameter matrices of each sensor.

[0019] The initial sensor data is transformed from its own coordinate system to the sensor coordinate system based on the intrinsic parameter matrix.

[0020] Furthermore, the initial sensor data is transformed from the sensor coordinate system to the vehicle coordinate system based on the extrinsic parameter matrix.

[0021] In the above technical solution, the intrinsic and extrinsic parameter matrices of each sensor can be obtained through calibration experiments, ensuring the accuracy and reliability of the parameters and providing high-quality input for subsequent spatial alignment. Through the transformation of the intrinsic parameter matrix, the data from different sensors are unified into the sensor coordinate system, ensuring data alignment in the same coordinate system and improving data accuracy and reliability. Through the transformation of the extrinsic parameter matrix, the data from all sensors are unified into the vehicle coordinate system, ensuring global alignment of data throughout the system and improving system integration and consistency.

[0022] Furthermore, the processing of the initial sensor data according to a secondary time alignment rule includes:

[0023] The timestamp corresponding to the initial sensor data that meets the preset conditions is obtained as the reference time.

[0024] A second timestamp is added to each frame of initial sensor data after processing according to spatial alignment rules, and the difference between the second timestamp and the reference time is calculated. The initial sensor data with the second timestamp is then time-aligned based on the difference to obtain the target sensor data.

[0025] In the above technical solution, a reference time is selected by setting preset conditions to ensure the stability and reliability of the reference time, providing a stable time base for subsequent time alignment; by adding a second timestamp, the data from each sensor are ensured to be under a precise time base, improving the accuracy of time alignment; the difference between the second timestamp and the reference time is calculated to obtain the time offset of each sensor data in real time, facilitating subsequent time alignment processing; and time alignment processing based on the difference ensures high-precision alignment of each sensor data in the time dimension, improving the accuracy of data fusion.

[0026] Furthermore, the acquisition of input data includes:

[0027] A time window is obtained, and target features of each sensor are extracted from the target sensor data based on the time window to form a target feature sequence, which is used as the input data.

[0028] In the above technical solution, by setting a time window, the time range of data processing is reduced, the efficiency and real-time performance of data processing are improved, and the system can quickly respond to and process sensor data. By constructing a target feature sequence, the data structure is optimized, which facilitates efficient processing and analysis by subsequent algorithms and improves the overall performance of the system.

[0029] Furthermore, before inputting the input data into the Transformer model, the following steps are also included:

[0030] The input data is preprocessed; wherein the preprocessing includes at least frame interpolation and normalization.

[0031] In the above technical solution, frame interpolation ensures the integrity and consistency of input data, avoids processing errors caused by missing data, and improves the stability and reliability of data processing; normalization scales each feature to the same scale, ensuring the fairness and accuracy of model processing and avoiding processing deviations caused by differences in feature scale.

[0032] Furthermore, the step of inputting the input data into the Transformer model to output fused data as the target detection result includes:

[0033] The preprocessed input data is input into the Transformer model; wherein the Transformer model includes at least a first inference layer and a second inference layer.

[0034] The temporal features between the input data of the same sensor are extracted through the first inference layer and used as the first feature.

[0035] The second inference layer extracts the correlation features between the input data of each sensor as the second feature.

[0036] The first feature and the second feature are fused to obtain fused data, which is then output as the target detection result.

[0037] In the above technical solution, normalization is used to scale all features to the same scale, ensuring the fairness and accuracy of model processing and avoiding processing deviations caused by differences in feature scale. Temporal features are extracted through the first inference layer to ensure that the model can capture the changing trends and dynamic information of sensor data, thereby improving the real-time performance and accuracy of target detection. Correlation features are extracted through the second inference layer to ensure that the model can capture the interrelationships and dependencies between multi-sensor data, thereby improving the globality and consistency of target detection.

[0038] Furthermore, by fusing temporal and correlation features, the model can comprehensively consider the dynamic changes in data and the relationships between multiple sensors, thereby improving the accuracy and robustness of target detection. The target detection results obtained through feature fusion ensure the accuracy and reliability of the detection results, thereby improving the overall performance and safety of the autonomous driving system.

[0039] Furthermore, based on the same concept, this application also provides a system for a multi-sensor-based target detection method, comprising:

[0040] The acquisition module is used to acquire initial sensor data from multiple sensors based on a single time alignment rule.

[0041] The processing module is used to process the initial sensor data sequentially according to spatial alignment rules and secondary time alignment rules to obtain target sensor data.

[0042] It also includes a fusion and detection module, which is used to acquire input data based on the target sensor data and input the input data into the Transformer model to output fused data as the target detection result.

[0043] The above technical solution achieves the fusion of data from different sensors, thereby realizing the ability to accurately perceive and understand the environment.

[0044] Furthermore, based on the same concept, this application also provides an in-vehicle terminal, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the steps of the multi-sensor-based target detection method.

[0045] Compared with the prior art, the beneficial effects of this application are as follows:

[0046] This application acquires initial sensor data from multiple sensors based on a primary time alignment rule; then, the initial sensor data is processed sequentially according to spatial alignment and secondary time alignment rules to obtain target sensor data; further, input data is obtained based on the target sensor data and input into a Transformer model to output fused data as the target detection result. Compared with existing rule-based post-fusion algorithms, this application exhibits greater stability and robustness under conditions of sensor characteristic changes and data anomalies. Combined with deep fusion of multi-sensor data, it enhances the ability to perceive and understand the environment. Attached Figure Description

[0047] Figure 1 This is a flowchart of the multi-sensor-based target detection method described in this application.

[0048] Figure 2 To adopt Figure 1 The system framework diagram of the multi-sensor-based target detection method is shown. Detailed Implementation

[0049] The following describes in further detail a multi-sensor-based target detection method, system, and vehicle terminal of this application, with reference to specific embodiments and accompanying drawings.

[0050] Please see Figure 1 This application provides a target detection method based on multiple sensors, including the following steps S100-S300.

[0051] Step S100: Collect initial sensor data from multiple sensors based on a single time alignment rule.

[0052] Furthermore, each sensor is pre-configured with a preset server; the initial sensor data acquisition from multiple sensors based on a single time alignment rule includes:

[0053] The sensor synchronizes its clock with the preset server based on a preset clock synchronization protocol.

[0054] Furthermore, during the acquisition of initial sensor data, a first timestamp is applied to each frame of the initial sensor data based on the synchronization clock, so as to align the initial sensor data through the first timestamp; wherein, the initial sensor data includes at least initial radar data, initial laser data and initial camera data.

[0055] In some embodiments, the vehicle terminal needs to be equipped with a server, such as an NTP (Network Time Protocol) server. First, the server needs to be configured to synchronize with a reliable time source (such as the National Time Service Center, GPS, etc.), while ensuring the server's time accuracy. Then, the server needs to be deployed in the vehicle network environment, ensuring its time synchronization service functions correctly.

[0056] Furthermore, configure a client corresponding to the server, such as an NTP client, on each sensor, and specify the server's IP address or domain name.

[0057] Specifically, sensor devices such as millimeter-wave radar, lidar, and cameras periodically synchronize their clocks with the server using a protocol (such as the NTP protocol) to ensure that their internal clocks are consistent with the server's clock. During the sensor data acquisition process, the synchronized clock is used to timestamp each frame of data. Since the timestamps of all sensor devices have been synchronized through the server, the data from different sensors can be aligned using the timestamps.

[0058] It should be noted that the first timestamp and the subsequent second timestamp are essentially both timestamps. The terms "first" and "second" are only used to distinguish between the two steps and have no priority or other meaning.

[0059] In addition, in other embodiments, those skilled in the art may also choose other servers to complete the time synchronization operation, preferably NTP servers, but are not limited to this.

[0060] In the above technical solution, based on the preset clock synchronization protocol, the timestamps of the sensors and the server are highly synchronized, improving the accuracy of time alignment; based on the synchronized clock, each frame of data is stamped with the first timestamp, ensuring the precise alignment of the data of each sensor in the time dimension and improving the accuracy of data fusion; the application of multi-source data enhances the system's ability to perceive the environment, which helps to improve the safety and reliability of the autonomous driving system.

[0061] Furthermore, the initial camera data includes at least first camera data and second camera data; after acquiring the initial sensor data, it also includes:

[0062] The first camera data and the second camera data are perceptually fused to update the initial camera data.

[0063] In some embodiments, the first camera data is such as forward-view camera data, and the second camera data is such as surround-view camera data; wherein, the forward-view camera is mainly used to capture road information, obstacles, traffic signals, etc. in front of the vehicle, and can provide a detailed view of the front of the vehicle, suitable for target detection and tracking in front; the surround-view camera is usually installed around the vehicle, providing a 360-degree panoramic view, and can capture environmental information around the vehicle, including targets on the sides and rear, suitable for all-round target detection and environmental perception of the vehicle.

[0064] By using a perception post-processing algorithm, target information from forward-looking camera data and surround-view camera data is fused. By leveraging the complementary nature of information from multiple sources, target discrepancies are eliminated, improving the accuracy and robustness of target detection.

[0065] Furthermore, during the fusion process, data association and correction technologies can be used to ensure the accurate correspondence and consistency of target information from each camera, thereby further improving the reliability of target detection.

[0066] In the above technical solution, by fusing the data output by the camera, the differences in the camera data output results can be effectively eliminated, thereby ensuring the accuracy and reliability of the final target detection results.

[0067] Step S200: Process the initial sensor data sequentially according to spatial alignment rules and secondary time alignment rules to obtain target sensor data.

[0068] Furthermore, the processing of the initial sensor data according to spatial alignment rules includes:

[0069] Obtain the intrinsic and extrinsic parameter matrices of each sensor.

[0070] The initial sensor data is transformed from its own coordinate system to the sensor coordinate system based on the intrinsic parameter matrix.

[0071] Furthermore, the initial sensor data is transformed from the sensor coordinate system to the vehicle coordinate system based on the extrinsic parameter matrix.

[0072] In some embodiments, the intrinsic parameter matrix defines the parameters of the sensor itself. For cameras, it is usually necessary to calibrate the focal length, principal point coordinates, and distortion coefficient. For lidar, it is necessary to calibrate the angle and range of the laser beam. For millimeter-wave radar, it is necessary to calibrate the angle and resolution of the radar beam. The extrinsic parameter matrix defines the position and orientation of the sensor relative to the vehicle coordinate system.

[0073] Specifically, the pixel coordinates corresponding to the initial camera data are transformed into three-dimensional coordinates in the sensor coordinate system through an intrinsic parameter matrix; the initial laser data can be directly obtained by using laser point cloud data without going through the intrinsic parameter matrix; the initial radar data is transformed into three-dimensional coordinates in the sensor coordinate system using the corresponding beam angle and distance information.

[0074] Furthermore, the data in the sensor coordinate system is transformed to the vehicle coordinate system using the extrinsic parameter matrix (such as a rotation and translation matrix).

[0075] In the above technical solution, the intrinsic and extrinsic parameter matrices of each sensor can be obtained through calibration experiments, ensuring the accuracy and reliability of the parameters and providing high-quality input for subsequent spatial alignment. Through the transformation of the intrinsic parameter matrix, the data from different sensors are unified into the sensor coordinate system, ensuring data alignment in the same coordinate system and improving data accuracy and reliability. Through the transformation of the extrinsic parameter matrix, the data from all sensors are unified into the vehicle coordinate system, ensuring global alignment of data throughout the system and improving system integration and consistency.

[0076] Furthermore, the processing of the initial sensor data according to a secondary time alignment rule includes:

[0077] The timestamp corresponding to the initial sensor data that meets the preset conditions is obtained as the reference time.

[0078] A second timestamp is added to each frame of initial sensor data after processing according to spatial alignment rules, and the difference between the second timestamp and the reference time is calculated. The initial sensor data with the second timestamp is then time-aligned based on the difference to obtain the target sensor data.

[0079] In some embodiments, the time of the lidar is selected as the reference time because lidar typically has high time accuracy and frequency.

[0080] Specifically, the system receives data from the camera, LiDAR, and millimeter-wave radar that have completed spatial alignment in real time, records a timestamp for each frame of data to ensure that the data from each sensor has a corresponding timestamp; then it calculates the difference between the timestamps of the camera and the millimeter-wave radar and the timestamp of the LiDAR; further, it interpolates the initial sensor data corresponding to the camera and the millimeter-wave radar according to the difference to align them with the timestamp of the LiDAR.

[0081] Those skilled in the art can choose linear interpolation or spline interpolation methods to achieve time alignment according to actual application requirements, and there are no restrictions on this.

[0082] In the above technical solution, a reference time is selected by setting preset conditions to ensure the stability and reliability of the reference time, providing a stable time base for subsequent time alignment; by adding a second timestamp, the data from each sensor are ensured to be under a precise time base, improving the accuracy of time alignment; the difference between the second timestamp and the reference time is calculated to obtain the time offset of each sensor data in real time, facilitating subsequent time alignment processing; and time alignment processing based on the difference ensures high-precision alignment of each sensor data in the time dimension, improving the accuracy of data fusion.

[0083] Step S300: Obtain input data based on the target sensor data, and input the input data into the Transformer model to output fused data as the target detection result.

[0084] Furthermore, the acquisition of input data includes:

[0085] A time window is obtained, and target features of each sensor are extracted from the target sensor data based on the time window to form a target feature sequence, which is used as the input data.

[0086] In some embodiments, the size of the time window W needs to be determined first to ensure that data from all sensors within that time period is covered; the size of the selected time window is typically determined based on the frequency of the sensor data and the application scenario.

[0087] For example, assuming the LiDAR frequency is 10Hz, the camera frequency is 30Hz, and the millimeter-wave radar frequency is 20Hz, then W can be selected as 0.1 seconds (100 milliseconds) to ensure that at least one frame of data from the LiDAR is included.

[0088] Based on the current timestamp, determine the start and end times of the time window, and extract the target features of each sensor within the time window to ensure that all target features are on the same time reference.

[0089] Specifically, target features are extracted from the initial camera data within the time window, such as target location, size, color, and texture; target features are extracted from the initial laser data within the time window, such as target location, point cloud density, and shape; and target features are extracted from the initial radar data within the time window, such as target location, speed, and reflection intensity.

[0090] Furthermore, the extracted target features are concatenated into vectors, and the concatenated feature vectors from multiple sensors are then concatenated into a time series tensor, thus forming the target feature sequence x. os , to be used as input data for the Transformer model.

[0091] Where s is the type of sensor, o is the number of the target feature, and xos It includes the historical target motion attributes of W frames.

[0092] In the above technical solution, by setting a time window, the time range of data processing is reduced, the efficiency and real-time performance of data processing are improved, and the system can quickly respond to and process sensor data. By constructing a target feature sequence, the data structure is optimized, which facilitates efficient processing and analysis by subsequent algorithms and improves the overall performance of the system.

[0093] Furthermore, before inputting the input data into the Transformer model, the following steps are also included:

[0094] The input data is preprocessed; wherein the preprocessing includes at least frame interpolation and normalization.

[0095] In some embodiments, before inputting the input data into the Transformer model, frame loss may occur during the acquisition process due to the different frequencies of the sensors. It is necessary to pad the target feature sequence of the sensors with zeros in the time dimension, and at the same time, the target feature sequence needs to be normalized to eliminate the scale difference between data from different sensors.

[0096] In the above technical solution, frame interpolation ensures the integrity and consistency of input data, avoids processing errors caused by missing data, and improves the stability and reliability of data processing; normalization scales each feature to the same scale, ensuring the fairness and accuracy of model processing and avoiding processing deviations caused by differences in feature scale.

[0097] Furthermore, the step of inputting the input data into the Transformer model to output fused data as the target detection result includes:

[0098] The preprocessed input data is input into the Transformer model; wherein the Transformer model includes at least a first inference layer and a second inference layer.

[0099] The temporal features between the input data of the same sensor are extracted through the first inference layer and used as the first feature.

[0100] The second inference layer extracts the correlation features between the input data of each sensor as the second feature.

[0101] The first feature and the second feature are fused to obtain fused data, which is then output as the target detection result.

[0102] In some embodiments, a pre-trained Transformer model is loaded, and pre-processed input data is fed into the Transformer model for forward propagation.

[0103] The Transformer model consists of multiple layers of self-attention and cross-attention, used to extract features between time-series data from the same sensor and between data from different sensors.

[0104] Each self-attention layer is used to extract features between time-series data from the same sensor, and each cross-attention layer is used to extract features between data from different sensors. Through multiple layers of self-attention and cross-attention, the model outputs the fused feature representation (i.e., the fused data). Finally, the model performs a decoding operation based on the fused data to output the final target features, i.e., the target detection result.

[0105] In the above technical solution, normalization is used to scale all features to the same scale, ensuring the fairness and accuracy of model processing and avoiding processing deviations caused by differences in feature scale. Temporal features are extracted through the first inference layer to ensure that the model can capture the changing trends and dynamic information of sensor data, thereby improving the real-time performance and accuracy of target detection. Correlation features are extracted through the second inference layer to ensure that the model can capture the interrelationships and dependencies between multi-sensor data, thereby improving the globality and consistency of target detection.

[0106] Furthermore, by fusing temporal and correlation features, the model can comprehensively consider the dynamic changes in data and the relationships between multiple sensors, thereby improving the accuracy and robustness of target detection. The target detection results obtained through feature fusion ensure the accuracy and reliability of the detection results, thereby improving the overall performance and safety of the autonomous driving system.

[0107] Furthermore, based on the same concept, please refer to Figure 2 This application also provides a target detection system based on multiple sensors, including:

[0108] The acquisition module is used to acquire initial sensor data from multiple sensors based on a single time alignment rule.

[0109] The processing module is used to process the initial sensor data sequentially according to spatial alignment rules and secondary time alignment rules to obtain target sensor data.

[0110] It also includes a fusion and detection module, which is used to acquire input data based on the target sensor data and input the input data into the Transformer model to output fused data as the target detection result.

[0111] It should be noted that the multi-sensor-based target detection system is implemented using the multi-sensor-based target detection method described above. The specific implementation of this system is the same as the target detection method, so it will not be described again.

[0112] The above technical solution achieves the fusion of data from different sensors, thereby realizing the ability to accurately perceive and understand the environment.

[0113] Furthermore, based on the same concept, this application also provides an in-vehicle terminal, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the steps of the multi-sensor-based target detection method.

[0114] In some embodiments, the memory and the processor are interconnected via a bus; the processor may be one or more CPUs. If the processor is a single CPU, it may be a single-core CPU or a multi-core CPU. The processor is used to control the various functional modules of the vehicle terminal and process signals.

[0115] The memory includes, but is not limited to, RAM (Random Access Memory), ROM (Read-Only Memory), EPROM (Erasable Programmable Read-Only Memory), and CD-ROM (Compact Disc Read-Only Memory). This memory is used to store computer programs, operating systems, various applications, and data, such as storing computer programs used to implement the multi-sensor-based target detection method.

[0116] In summary, this application provides a multi-sensor-based target detection method, system, and vehicle terminal. First, initial sensor data from multiple sensors is acquired based on a primary time alignment rule. Then, the initial sensor data is processed sequentially according to spatial alignment and secondary time alignment rules to obtain target sensor data. Further, input data is acquired based on the target sensor data and input into a Transformer model to output fused data as the target detection result. Compared to existing rule-based post-fusion algorithms, this application demonstrates greater stability and robustness under conditions of sensor characteristic changes and data anomalies. The deep fusion of multi-sensor data enhances the perception and understanding of the environment.

[0117] Although exemplary embodiments have been described herein with reference to the accompanying drawings, it should be understood that the above exemplary embodiments are merely illustrative and are not intended to limit the scope of this application. Various changes and modifications can be made therein by those skilled in the art without departing from the scope and spirit of this application. All such changes and modifications are intended to be included within the scope of this application as claimed in the appended claims.

[0118] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0119] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed.

[0120] The various component embodiments of this application can be implemented in hardware, or as software modules running on one or more processors, or a combination thereof. Those skilled in the art will understand that microprocessors or digital signal processors (DSPs) can be used in practice to implement some or all of the functions of some modules according to the embodiments of this application. This application can also be implemented as an apparatus program (e.g., a computer program and computer program product) for performing part or all of the methods described herein. Such an implementation of this application can be stored on a computer-readable medium, or can be in the form of one or more signals. Such signals can be downloaded from an Internet website, provided on a carrier signal, or provided in any other form.

[0121] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0122] Although the description of this application has been made in conjunction with the specific embodiments described above, it will be apparent to those skilled in the art that many substitutions, modifications, and variations can be made based on the foregoing. Therefore, all such substitutions, modifications, and variations are included within the spirit and scope of the appended claims.

Claims

1. A target detection method based on multiple sensors, characterized in that, Includes the following steps: Initial sensor data from multiple sensors are acquired based on a single time alignment rule; The initial sensor data is processed sequentially according to spatial alignment rules and secondary time alignment rules to obtain target sensor data; Furthermore, input data is acquired based on the target sensor data, and the input data is input into the Transformer model to output fused data as the target detection result.

2. The target detection method based on multiple sensors according to claim 1, characterized in that, Each sensor is pre-configured with a preset server; The initial sensor data collected from multiple sensors based on a single time alignment rule includes: The sensor synchronizes its clock with the preset server based on a preset clock synchronization protocol. In addition, during the process of collecting initial sensor data, a first timestamp is added to each frame of initial sensor data based on the synchronization clock, so as to align each initial sensor data through the first timestamp. The initial sensor data includes at least initial radar data, initial laser data, and initial camera data.

3. The target detection method based on multiple sensors according to claim 2, characterized in that, The initial camera data includes at least the first camera data and the second camera data; After acquiring the initial sensor data, the process also includes: The first camera data and the second camera data are perceptually fused to update the initial camera data.

4. The target detection method based on multiple sensors according to claim 3, characterized in that, The process of processing the initial sensor data according to spatial alignment rules includes: Obtain the intrinsic and extrinsic parameter matrices of each sensor; Based on the intrinsic parameter matrix, the initial sensor data is transformed from its own coordinate system to the sensor coordinate system; Furthermore, the initial sensor data is transformed from the sensor coordinate system to the vehicle coordinate system based on the extrinsic parameter matrix.

5. The target detection method based on multiple sensors according to claim 4, characterized in that, The process of processing the initial sensor data according to a secondary time alignment rule includes: Obtain the timestamp corresponding to the initial sensor data that meets the preset conditions as the reference time; A second timestamp is added to each frame of initial sensor data after processing according to spatial alignment rules, and the difference between the second timestamp and the reference time is calculated. The initial sensor data with the second timestamp is then time-aligned based on the difference to obtain the target sensor data.

6. The target detection method based on multiple sensors according to claim 5, characterized in that, The acquisition of input data includes: A time window is obtained, and target features of each sensor are extracted from the target sensor data based on the time window to form a target feature sequence, which is used as the input data.

7. The target detection method based on multiple sensors according to claim 6, characterized in that, Before inputting the input data into the Transformer model, the following steps are also included: The input data is preprocessed; wherein the preprocessing includes at least frame interpolation and normalization.

8. The target detection method based on multiple sensors according to claim 7, characterized in that, The step of inputting the input data into the Transformer model and outputting fused data as the target detection result includes: The preprocessed input data is input into the Transformer model; wherein the Transformer model includes at least a first inference layer and a second inference layer; The temporal features between the input data of the same sensor are extracted through the first inference layer and used as the first feature; The second inference layer extracts the correlation features between the input data of each sensor as the second feature; The first feature and the second feature are fused to obtain fused data, which is then output as the target detection result.

9. A system employing the multi-sensor-based target detection method as described in any one of claims 1-8, characterized in that, include: The acquisition module is used to acquire initial sensor data from multiple sensors based on a single time alignment rule. The processing module is used to process the initial sensor data sequentially according to spatial alignment rules and secondary time alignment rules to obtain target sensor data; It also includes a fusion and detection module, which is used to acquire input data based on the target sensor data and input the input data into the Transformer model to output fused data as the target detection result.

10. A vehicle-mounted terminal, characterized in that, It includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the steps of the multi-sensor-based target detection method as described in any one of claims 1-8.

Images