A multi-sensor target fusion method and device

By obtaining the time stamp and receiving order of sensor data, the problem of sensor time is solved, the accuracy and flexibility of multi-sensor target fusion of unmanned driving systems is improved, and the decentralized time synchronization solution is realized, and the The robustness and real-time nature of the system.

CN114528940BActive Publication Date: 2024-11-15SHENZHEN HAIXING ZHIJIA TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202210155624.5
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-02-18
Publication Date
2024-11-15
Estimated Expiration
2042-02-18

AI Technical Summary

Technical Problem

In unmanned driving and unmanned operations, the time out-of-synchronization between the sensor and the computing platform leads to a reduction in the accuracy of the multi-sensor target fusion, affecting decision-making and control.

Method used

By obtaining the data to be fused and its timestamps sent by each target sensor, the data is selected and updated in the order of reception, and the relationship between the timestamp and the previous fusion time is used for prediction and update, and a decentralized time synchronization scheme is realized.

Benefits of technology

It improves the robustness and flexibility of the fusion target list, allows fusion operations at any time, without being limited by sensor sampling frequency, and enhances the universality and real-time nature of the system.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114528940B_ABST
    Figure CN114528940B_ABST
Patent Text Reader

Abstract

The present invention provides a multi-sensor target fusion method and device, the method comprising: obtaining the data to be fused sent by each target sensor and the fusion target list corresponding to the last fusion time, the data to be fused carrying the timestamp information of the data generation time; selecting the first data to be fused from the data to be fused according to the receiving order of the data to be fused; based on the relationship between the first timestamp corresponding to the first data to be fused and the last fusion time, using the first data to be fused to update the fusion target list. Thus, a decentralized multi-sensor target fusion time synchronization scheme is realized, the robustness of the fusion target list update result is improved, and it is not limited to the sampling frequency of the sensor, and the sampling frequency of all sensors does not need to be synchronized. The fusion operation can be performed at any time. The fusion scheme is more universal and improves the flexibility of the scheme application.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of unmanned driving and unmanned operation, and in particular to a multi-sensor target fusion method and device. Background Art

[0002] In unmanned driving and unmanned operation, many sensors installed on vehicles and equipment are responsible for sensing and collecting data about the surrounding environment, providing a basis for decision-making and control of unmanned driving and unmanned operation. Different sensors perceive different environmental information, so the perception results of different sensors must be fused. However, due to the differences in sampling cycles, transmission delays, and processing times of each sensor, there is a time asynchrony between sensors and between sensors and computing platforms. If data at different times are directly used for target fusion, the accuracy of the fused target position and motion information will be affected, which in turn affects subsequent decision-making and control. Summary of the invention

[0003] In view of this, an embodiment of the present invention provides a multi-sensor target fusion method and device to overcome the problem in the prior art that the multi-sensor target fusion accuracy is affected by the time asynchrony between the sensor and the computing platform.

[0004] According to a first aspect, an embodiment of the present invention provides a multi-sensor target fusion method, including:

[0005] Obtain the data to be fused sent by each target sensor and the fusion target list corresponding to the last fusion time, wherein the data to be fused carries the timestamp information of the time when the data was generated;

[0006] Selecting first data to be fused from the data to be fused according to the order in which the data to be fused are received;

[0007] Based on the relationship between the first timestamp corresponding to the first data to be fused and the last fusion time, the fusion target list is updated using the first data to be fused.

[0008] Optionally, the updating the fusion target list by using the first data to be fused based on the relationship between the first timestamp corresponding to the first data to be fused and the last fusion time includes:

[0009] Determine whether a first moment corresponding to the first timestamp is earlier than the last fusion moment;

[0010] When a first moment corresponding to the first timestamp is earlier than the last fusion moment, predicting and updating the first to-be-fused data based on a first time difference between the first moment and the last fusion moment;

[0011] The fusion target list is updated based on the updated first data to be fused.

[0012] Optionally, when a first moment corresponding to the first timestamp is not earlier than the last fusion moment, the fusion target list is predicted and updated based on a first time difference between the first moment and the last fusion moment;

[0013] The updated fusion target list is updated based on the first data to be fused.

[0014] Optionally, the method further comprises:

[0015] The previous fusion moment is updated based on the first moment to obtain a current fusion moment.

[0016] Optionally, the updating the fusion target list based on the updated first data to be fused includes:

[0017] Based on the updated first data to be fused, target attribute information of each target in the fusion target list is fused and updated;

[0018] Based on the second time difference between the last fusion moment and the current moment, the target attribute of each target in the fusion target list after fusion and update is predicted to obtain the prediction result corresponding to each target at the current moment.

[0019] Optionally, before predicting the target attribute of each target in the fused target list after fusion update, the method further includes:

[0020] Get the target type of each target in the fusion target list after fusion update;

[0021] Determine the corresponding update strategy based on the target type of each target;

[0022] Based on the update strategy corresponding to each target, the target attribute of each target in the fusion target list after fusion and update is predicted.

[0023] Optionally, updating the updated fusion target list based on the first to-be-fused data includes:

[0024] Based on the first data to be fused, target attribute information of each target in the updated fusion target list is fused and updated;

[0025] Based on the third time difference between the current fusion moment and the current moment, the target attribute of each target in the fusion target list after fusion and update is predicted to obtain the prediction result corresponding to each target at the current moment.

[0026] According to a second aspect, an embodiment of the present invention provides a device, including:

[0027] An acquisition module is used to acquire the data to be fused sent by each target sensor and a fusion target list corresponding to the last fusion time, wherein the data to be fused carries the timestamp information of the time when the data is generated;

[0028] A first processing module, configured to select first data to be fused from the data to be fused according to the order in which the data to be fused are received;

[0029] The second processing module is used to update the fusion target list using the first data to be fused based on the relationship between the first timestamp corresponding to the first data to be fused and the previous fusion time.

[0030] According to a third aspect, an embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, and when the computer instructions are executed by a processor, the method described in the first aspect of the present invention and any optional manner thereof is implemented.

[0031] According to a fourth aspect, an embodiment of the present invention provides an electronic device, including:

[0032] A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the method described in the first aspect of the present invention and any optional method thereof by executing the computer instructions.

[0033] The technical solution of the present invention has the following advantages:

[0034] The embodiment of the present invention provides a multi-sensor target fusion method and device, by obtaining the data to be fused sent by each target sensor and the fusion target list corresponding to the last fusion time, the data to be fused carries the timestamp information of the data generation time; the first data to be fused is selected from the data to be fused according to the receiving order of the data to be fused; based on the relationship between the first timestamp corresponding to the first data to be fused and the last fusion time, the fusion target list is updated using the first data to be fused. Therefore, when the data fusion operation is triggered, the data fusion processing is performed in sequence according to the receiving time of the data to be fused, and the relationship between the timestamp of the data to be fused and the fusion time corresponding to the last fusion operation is judged, and the fusion target list is updated using the data to be fused, so as to realize the decentralized multi-sensor target fusion time synchronization scheme, improve the robustness of the fusion target list update result, and is not limited to the sampling frequency of the sensor, and does not need to synchronize the sampling frequency of all sensors. The fusion operation can be performed at any time. The fusion scheme is more universal and improves the flexibility of the scheme application. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0036] Figure 1 is a flow chart of a multi-sensor target fusion method in an embodiment of the present invention;

[0037] Figure 2 A schematic diagram of the structure of a time synchronization system in an embodiment of the present invention;

[0038] Figure 3 Another structural principle diagram of time soft synchronization in the time synchronization system in an embodiment of the present invention;

[0039] Figure 4 A schematic diagram of a time soft synchronization working process of a time synchronization system in an embodiment of the present invention;

[0040] Figure 5 A schematic diagram of an example of a time soft synchronization application of a time synchronization system in an embodiment of the present invention;

[0041] Figure 6 Schematic diagram of the structure of a multi-sensor target fusion device in an embodiment of the present invention;

[0042] Figure 7 Schematic diagram of the structure of an electronic device in an embodiment of the present invention. DETAILED DESCRIPTION

[0043] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0044] In the description of the present invention, it should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", etc., indicating the orientation or positional relationship, are based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first", "second", and "third" are used for descriptive purposes only, and cannot be understood as indicating or implying relative importance.

[0045] In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or an indirect connection through an intermediate medium, or it can be the internal connection of two components, it can be a wireless connection, or it can be a wired connection. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0046] The technical features involved in the different embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0047] In the fusion perception system of unmanned driving and unmanned operation, the accuracy of time synchronization between sensors and between sensors and controllers directly affects the final output result of fusion perception. However, due to factors such as the deviation of the hardware's own clock crystal and the different sampling periods of each sensor, when the data arrives at the centralized processing module, the timestamps of each data packet obtained by the processing module are not consistent. If the multi-sensor data with inconsistent sampling times are directly fused, the fusion accuracy will be seriously affected. Some fusion perception systems use the time when the data arrives at the centralized processing module, subtract the signal transmission delay and the signal processing time, obtain the sampling time of the signal, and then perform fusion processing. However, due to the inability to accurately estimate the signal transmission delay and the uncertainty of the signal processing time, it is impossible to obtain the accurate sampling time, which affects the fusion effect.

[0048] Based on the above problems, an embodiment of the present invention provides a multi-sensor target fusion method, which can be applied to scenarios such as unmanned driving or unmanned operation systems that require multi-sensor target fusion. In the embodiment of the present invention, an unmanned driving system is taken as an example for explanation. The unmanned driving system includes an unmanned driving central domain controller, a laser radar, a millimeter wave radar, an ultrasonic radar, an infrared sensor, a camera, and a positioning navigator such as a GPS. Figure 1 As shown, the multi-sensor target fusion method specifically includes the following steps:

[0049] Step S101: Acquire the data to be fused sent by each target sensor and the fusion target list corresponding to the last fusion moment.

[0050] The data to be fused carries the timestamp information of the time when the data was generated.

[0051] Specifically, in the embodiment of the present invention, a thread for receiving data is created for each sensor respectively, and then a first-in-first-out (FIFO) queue is used to uniformly store the data received by each thread, and the data are stored in the order of arrival at the queue. The queue will be exposed to the data processing module, namely the fusion module, to execute the multi-sensor target fusion method provided by the embodiment of the present invention.

[0052] Exemplarily, taking the target sensor as a camera, the data to be fused is the image data collected by the camera and the timestamp information corresponding to the moment when the image data was collected.

[0053] Step S102: selecting first data to be fused from the data to be fused according to the order in which the data to be fused are received.

[0054] Specifically, in an embodiment of the present invention, a separate thread is opened for the fusion module, and the thread takes out a packet of data from the above-mentioned FIFO queue at a fixed frequency and processes it. Generally, the update frequency of the thread is higher than the sampling frequency of any sensor to ensure data fusion efficiency. It should be noted that during the reading and writing operations of the queue, the queue needs to be locked to avoid read and write conflicts on the same memory block. Therefore, by decoupling from the sensor sampling cycle, time synchronization can be performed for any sampling cycle and transmission delay, thereby improving the flexibility and universality of the multi-sensor target fusion method.

[0055] Step S103: Based on the relationship between the first timestamp corresponding to the first data to be fused and the last fusion time, the fusion target list is updated using the first data to be fused.

[0056] Specifically, in multi-sensor target fusion, an internal fusion target list is usually maintained, and this list is used as the output of the fusion module. As mentioned above, the data sent by all sensors will be uniformly stored in a FIFO queue. The fusion module will process the data packets in the FIFO queue in sequence during the fusion cycle. The specific processing method is: a packet of data is fused with the internal fusion target list in terms of position, speed, category, size and other dimensions. Therefore, when the first packet of data is received, the target list will be initialized with this packet of data, and the timestamp of this packet of data will be assigned to the timestamp of the fusion target as the initial fusion moment. Subsequently, the time relationship between the previous fusion moment and the timestamp corresponding to the data to be fused will be used to update the corresponding data to be fused and the fusion target list, and the updated fusion target list will be output.

[0057] By executing the above steps, the multi-sensor target fusion method provided by the embodiment of the present invention, when the data fusion operation is triggered, performs data fusion processing in sequence according to the receiving time of the data to be fused, and updates the fusion target list using the data to be fused by judging the relationship between the timestamp of the data to be fused and the fusion moment corresponding to the previous fusion operation, thereby realizing a decentralized multi-sensor target fusion time synchronization scheme, improving the robustness of the fusion target list update result, and is not limited to the sampling frequency of the sensor, and there is no need to synchronize the sampling frequencies of all sensors. The fusion operation can be performed at any time, and the fusion scheme is more universal, and the flexibility of the scheme application is improved.

[0058] Specifically, in one embodiment, the above step S103 specifically includes the following steps:

[0059] Step S301: Determine whether the first moment corresponding to the first timestamp is earlier than the last fusion moment.

[0060] Specifically, when the sensor generates a signal, it will add a timestamp to the signal. When the fusion module processes this data packet, it will obtain its timestamp t_sensor, and assume that the last fusion time is t_fusion, then determine whether t_fusion>t_sensor is true, if true, then execute step S302, otherwise execute step S304.

[0061] Step S302: when the first moment corresponding to the first timestamp is earlier than the last fusion moment, predict and update the first data to be fused based on a first time difference between the first moment and the last fusion moment.

[0062] Specifically, if it is judged that t_fusion>t_sensor is true, that is, the sensor timestamp is earlier than the fusion time, this situation is called the signal generated first and then reaching the fusion module, and the data is also called out-of-order data. For out-of-order data, prediction processing is required. Before prediction, the time interval difference dt=t_fusion-t_sensor is obtained first, and then the prediction model x=f(x, dt) is used to predict the data to be fused, where x refers to the state vector of the target. This ensures the time synchronization between the fusion target list and the data to be fused.

[0063] This can effectively solve the disorder problem of sensor data. Delayed sensor data is not discarded, but is input into the fusion module after prediction. The sensor frame rate and data are not lost, which can effectively improve the fusion accuracy and real-time performance.

[0064] Step S303: updating the fusion target list based on the updated first data to be fused.

[0065] Specifically, the target attribute information of each target in the fusion target list is fused and updated based on the updated first data to be fused; based on the second time difference between the last fusion moment and the current moment, the target attribute of each target in the fusion target list after the fusion update is predicted to obtain the prediction result corresponding to each target at the current moment.

[0066] In practical applications, the above-mentioned time is derived from the time when the sensor signal is generated. In the process from signal generation to the output of the fusion target list by the fusion module, the signal processing time, data transmission time, and fusion processing time are involved, which leads to the occurrence of delays. Therefore, the above-mentioned time is not the real-time time of the current moment, but belongs to the historical time. If the fusion module directly outputs the fusion target list corresponding to the t_fusion moment instead of the fusion target list of the current moment to the downstream module, the performance of the entire system will be affected due to the large error. Therefore, this part of the delay needs to be compensated. In an embodiment of the present invention, the target attributes of each target in the fusion target list after fusion update are predicted by calculating the time difference between the current moment and the previous fusion moment, thereby ensuring that the prediction result is the real-time prediction result of each target at the current moment.

[0067] Step S304: when the first moment corresponding to the first timestamp is not earlier than the last fusion moment, the fusion target list is predicted and updated based on a first time difference between the first moment and the last fusion moment.

[0068] Specifically, if it is judged that t_fusion>t_sensor is false, that is, the fusion time is earlier than the sensor timestamp, the internal fusion target list is predicted. Before the prediction, the time interval difference dt=t_sensor-t_fusion is obtained, and then the prediction model x=f(x, dt) is used to predict the fusion target list, where x refers to the state vector of the target.

[0069] Step S305: updating the updated fusion target list based on the first data to be fused.

[0070] Step S306: Update the previous fusion time based on the first time to obtain the current fusion time.

[0071] Specifically, if it is judged that t_fusion>t_sensor is false, that is, the fusion time is earlier than the sensor timestamp, then the fusion time t_fusion needs to be updated, that is, the value of t_sensor is assigned to t_fusion, so as to ensure the time synchronization between the fusion target list and the data to be fused.

[0072] Furthermore, the above step S305 performs fusion update on the target attribute information of each target in the updated fusion target list based on the first data to be fused; and predicts the target attribute of each target in the fusion target list after fusion update based on the third time difference between the current fusion moment and the current moment, and obtains the prediction result corresponding to each target at the current moment. Thus, the time synchronization between the fusion target list and the data to be fused is ensured.

[0073] In practical applications, the above-mentioned time is derived from the time when the sensor signal is generated. In the process from signal generation to the output of the fusion target list by the fusion module, the signal processing time, data transmission time, and fusion processing time are involved, which leads to the occurrence of delays. Therefore, the above-mentioned time is not the real-time time of the current moment, but belongs to the historical time. If the fusion module directly outputs the fusion target list corresponding to the t_fusion moment instead of the fusion target list of the current moment to the downstream module, the performance of the entire system will be affected due to the large error, so this part of the delay needs to be compensated. In an embodiment of the present invention, the target attributes of each target in the fusion target list after fusion update are predicted by calculating the time difference between the current moment and the current fusion moment, thereby ensuring that the prediction result is the real-time prediction result of each target at the current moment.

[0074] Specifically, in one embodiment, before predicting the target attribute of each target in the fusion target list after fusion update, the multi-sensor target fusion method provided by the embodiment of the present invention further includes the following steps:

[0075] Step S401: Obtain the target type of each target in the fusion target list after fusion update.

[0076] Specifically, the fusion target list after the above fusion update is a collection of all targets. Due to the different motion characteristics of different target types, in order to further improve the prediction accuracy of each target, each target is classified and predicted. The fusion target list is all targets in the past time, so it is necessary to predict the internal fusion target list, and predict it to the current time, and then output it to the downstream module. It should be noted that in the embodiment of the present invention, the target types are divided into two types: pedestrians and vehicles. In actual applications, the target types can also be divided according to the actual application scenarios and prediction accuracy requirements, and the present invention is not limited to this.

[0077] Step S402: Determine a corresponding update strategy based on the target type of each target.

[0078] Specifically, in the embodiment of the present invention, a constant acceleration model (CA) or a constant angular velocity model (CTRV) is used for the motion model of the vehicle, and the present invention adopts the constant acceleration model (CA). Exemplarily, the motion model of the vehicle is shown in formula (1).

[0079]

[0080] Among them, x, y, v x ,v y ,a x ,a y They represent the position, velocity, and acceleration of the horizontal and vertical coordinates respectively, t represents time, subscript t represents the state value at the current moment, and subscript t-1 represents the state value at the previous moment.

[0081] Specifically, in the embodiment of the present invention, a constant velocity model (CV) is used for pedestrians. Exemplarily, the motion model of pedestrians is shown in formula (2).

[0082]

[0083] Among them, x, y, v x ,v y They represent the position, velocity, and acceleration of the horizontal and vertical coordinates respectively, t represents time, subscript t represents the state value at the current moment, and subscript t-1 represents the state value at the previous moment.

[0084] Step S403: Based on the update strategy corresponding to each target, predict the target attribute of each target in the fused target list after fusion and update.

[0085] Therefore, by using different motion models for prediction for different categories of targets, higher precision compensation can be achieved.

[0086] In addition, in the embodiment of the present invention, the above multi-sensor target fusion method is to realize the software time synchronization of the multi-sensor target fusion. In order to further ensure the time synchronization of the multi-sensor target fusion, the embodiment of the present invention also provides a hardware time synchronization solution, which is as follows:

[0087] like Figure 2 As shown, in the process of implementing time hard synchronization, an absolute time synchronization algorithm is used, and a sensor is selected as the reference clock source. For example, in the embodiment of the present invention, the time of the GPS system is selected as the reference clock source, and the PPS pulse output pin of the GPS is respectively connected to the laser radar and the unmanned driving controller. After the laser radar receives the PPS pulse signal, it calibrates the internal hardware clock so that the timestamp obtained at the time of sampling is an absolute timestamp consistent with the GPS system. Similarly, the unmanned driving controller obtains the same absolute timestamp. The unmanned driving controller can output a pulse signal and connect to the trigger pin of the camera to give the camera time. At this point, the laser radar, camera and unmanned driving controller have completed time hard synchronization, and all three use the absolute time output by the GPS system as the reference clock source. After time hard synchronization, the timestamps of all sensors will use the absolute time output by the GPS system as the reference clock source. When the sensor generates a signal, it will timestamp the signal, and then execute the multi-sensor target fusion steps of the above-mentioned software time synchronization.

[0088] The time synchronization system constructed by the multi-sensor target fusion method provided by the embodiment of the present invention will be described in detail below with reference to specific application examples.

[0089] like Figure 2 As shown, the time synchronization system constructed in the embodiment of the present invention includes two parts: time hard synchronization and time soft synchronization. In addition, in practical applications, Figure 3 As shown, in time soft synchronization, a thread for receiving data can be created for each sensor, and then a FIFO queue is used to store the data received by each thread. When the fusion module is triggered, a packet of data is taken out from each FIFO queue and inserted into the main buffer area, which is exposed to the fusion module, and the fusion module processes all the data within the fusion cycle. The present invention is not limited to this.

[0090] Figure 4 The schematic diagram of the time soft synchronization working process of the time synchronization system is shown in FIG. Figure 5 Detailed description is given. Figure 5 Take two sensors as an example, coded as a and b respectively.

[0091] From the first time axis, i.e., the signal generation time, it can be seen that both sensors a and b emit signals at a fixed period, and the signal generation times of the two sensors are t a 0, t a 1, t a 2… and t b 0, t b 1, t b 2….

[0092] The second time axis is the time when the fusion module runs. It can be seen that the fusion module runs in a fixed period, which is t0, t1, t2, t3, t4, ... t 13 ,…t n .

[0093] The time evolution is based on the second time axis, and the program starts running from time t0.

[0094] In t' a At time 0, a packet of data arrives at the fusion module. This data is from sensor a at t a At time t0, as above, the signal will be stored in the FIFO queue and in a waiting state. At time t3, the fusion module is triggered again. At this time, the data will be taken out from the FIFO queue and used to initialize the internal fusion target list. The fusion time t_fusion is t a 0. Before time t3, since the internal fusion target list is not initialized, that is, the number of targets in the target list is 0, no fusion target will be output to the downstream module. At time t3, the fusion module is triggered, the internal fusion target list is initialized, and the fusion target result will also be output to the downstream module at this time. As mentioned above, since the internal fusion target list corresponds to the historical time, that is, t a 0, so before outputting the target at time t3, the target list needs to be predicted to time t3, as shown in the attached Figure 5 It is shown by the black solid arrow in . It should be noted that the target category needs to be judged before prediction, whether the target is a car or a pedestrian, and the CA model or CV model is used for prediction respectively.

[0095] At t4 and t5, since the fusion module does not receive new sensor data, it will a The internal fusion target list at time 0 is used as the benchmark for prediction, as shown in the attached Figure 5 As shown by the solid black arrow in the middle, the fused target is then output to the downstream module.

[0096] In t' b At time 0, a packet of data arrives at the fusion module. This data is from sensor b at t bThe signal is generated at time 0. As above, the signal will be stored in the FIFO queue and will be in a waiting state. At time t6, the fusion module is triggered again. At this time, the data will be taken out of the FIFO queue and the timestamp t_sensor of this data packet will be obtained as t b 0. Then get the fusion time t a 0. Due to t b 0 is greater than t a 0, so to predict the fusion target, as shown in the attached Figure 5 As shown by the gray arrow in the middle, the fusion time t_fusion is updated to t b 0. At this time, the fusion target result will also be output to the downstream module. As mentioned above, since the internal fusion target list corresponds to the historical time, i.e., t b 0, so before outputting the target at time t6, the target list needs to be predicted to time t6, as shown in the attached Figure 5 It is shown by the black solid arrow in . It should be noted that the target category needs to be judged before prediction, whether the target is a car or a pedestrian, and the CA model or CV model is used for prediction respectively.

[0097] At time t7, a packet of data arrives at the fusion module. This data is from sensor a at t a The signal is generated at time 1. As above, the signal will be stored in the FIFO queue and is in a waiting state. At time t7, the fusion module is triggered again. At this time, the data will be taken out from the FIFO queue and the timestamp t_sensor of this data packet will be obtained as t a 1. Then get the fusion time t b 0. Due to t a 1 is greater than t b 0, so to predict the fusion target, as shown in the attached Figure 5 As shown by the gray arrow in the middle, the fusion time t_fusion is updated to t a 1. At this time, the fusion target result will also be output to the downstream module. As mentioned above, since the internal fusion target list corresponds to the historical time, i.e., t a 1, so before outputting the target at time t7, the target list needs to be predicted to time t7, as shown in the attached Figure 5 It is shown by the black solid arrow in . It should be noted that the target category needs to be judged before prediction, whether the target is a car or a pedestrian, and the CA model or CV model is used for prediction respectively.

[0098] At time t8, since the fusion module does not receive new sensor data, it will a The internal fusion target list at time 1 is used as the benchmark for prediction, as shown in the attached Figure 5 As shown by the solid black arrow in the middle, the fused target is then output to the downstream module.

[0099] In t'b At time 1, a packet of data arrives at the fusion module. This data is from sensor b at t b The signal is generated at time 1. As above, the signal will be stored in the FIFO queue and is in a waiting state. At time t9, the fusion module is triggered again. At this time, the data will be taken out from the FIFO queue and the timestamp t_sensor of this data packet will be obtained as t b 1. Then get the fusion time t a 1. Due to t a 1 is greater than t b 0, so to predict the sensor target, such as the attached Figure 5 As shown by the black hollow arrow in the figure, in this case, the fusion time t_fusion is not updated. At this time, the fusion target result is also output to the downstream module. As mentioned above, since the internal fusion target list corresponds to the historical time, i.e., t a 1, so before outputting the target at time t9, the target list needs to be predicted to time t9, as shown in the attached Figure 5 It is shown by the black solid arrow in . It should be noted that the target category needs to be judged before prediction, whether the target is a car or a pedestrian, and the CA model or CV model is used for prediction respectively.

[0100] In summary, the software time synchronization method for multi-sensor target fusion provided by the embodiment of the present invention has the characteristics of better robustness, stronger generalization ability and higher precision. A decentralized time synchronization scheme is adopted, which does not distinguish between master and slave sensors, can effectively improve the robustness of the system, and at the same time can ensure the real-time output of the fusion target; by decoupling from the sensor sampling period, time synchronization can be performed for any sampling period and transmission delay, so it is not limited to the sampling frequency of the sensor, and there is no need to synchronize the sampling frequencies of all sensors, and there is no need to fuse at integer times, so the system is more universal; it can effectively solve the disorder problem of sensor data; the delayed sensor data is not discarded, but predicted and then input into the fusion module, without losing the frame rate and data of the sensor, which can effectively improve the fusion accuracy and real-time performance; the compensation for processing delay and transmission delay is not a simple interpolation, but the selection of appropriate motion models according to different target types, and the compensation accuracy is higher.

[0101] By executing the above steps, the multi-sensor target fusion method provided by the embodiment of the present invention, when the data fusion operation is triggered, performs data fusion processing in sequence according to the receiving time of the data to be fused, and updates the fusion target list using the data to be fused by judging the relationship between the timestamp of the data to be fused and the fusion moment corresponding to the previous fusion operation, thereby realizing a decentralized multi-sensor target fusion time synchronization scheme, improving the robustness of the fusion target list update result, and is not limited to the sampling frequency of the sensor, and there is no need to synchronize the sampling frequencies of all sensors. The fusion operation can be performed at any time, and the fusion scheme is more universal, and the flexibility of the scheme application is improved.

[0102] The embodiment of the present invention also provides a multi-sensor target fusion device, such as Figure 6 As shown, the multi-sensor target fusion device specifically includes:

[0103] The acquisition module 101 is used to acquire the data to be fused sent by each target sensor and the fusion target list corresponding to the last fusion time. The data to be fused carries the timestamp information of the data generation time. For details, please refer to the detailed description of the above step S101, which will not be repeated here.

[0104] The first processing module 102 is used to select first data to be fused from the data to be fused according to the order in which the data to be fused are received. For details, please refer to the detailed description of the above step S102, which will not be repeated here.

[0105] The second processing module 103 is used to update the fusion target list using the first data to be fused based on the relationship between the first timestamp corresponding to the first data to be fused and the last fusion time. For details, please refer to the detailed description of the above step S103, which will not be repeated here.

[0106] Through the collaborative cooperation of the above-mentioned components, the multi-sensor target fusion device provided by the embodiment of the present invention, when triggering the data fusion operation, performs data fusion processing in sequence according to the reception time of the data to be fused, and updates the fusion target list using the data to be fused by judging the relationship between the timestamp of the data to be fused and the fusion moment corresponding to the previous fusion operation, thereby realizing a decentralized time synchronization scheme for multi-sensor target fusion, improving the robustness of the fusion target list update result, and is not limited to the sampling frequency of the sensor, and there is no need to synchronize the sampling frequencies of all sensors. The fusion operation can be performed at any time. The fusion scheme is more universal and improves the flexibility of the scheme application.

[0107] like Figure 7As shown, an embodiment of the present invention further provides an electronic device, which may include a processor 901 and a memory 902, wherein the processor 901 and the memory 902 may be connected via a bus or other means. Figure 7 The example of connecting through bus is taken in the following.

[0108] The processor 901 may be a central processing unit (CPU). The processor 901 may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or a combination of the above chips.

[0109] The memory 902 is a non-transitory computer-readable storage medium that can be used to store non-transitory software programs, non-transitory computer executable programs and modules, such as program instructions / modules corresponding to the method in the embodiment of the present invention. The processor 901 executes various functional applications and data processing of the processor by running the non-transitory software programs, instructions and modules stored in the memory 902, that is, implementing the above method.

[0110] The memory 902 may include a program storage area and a data storage area, wherein the program storage area may store an application required for operating the device and at least one function; the data storage area may store data created by the processor 901, etc. In addition, the memory 902 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 902 may optionally include a memory remotely arranged relative to the processor 901, and these remote memories may be connected to the processor 901 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0111] One or more modules are stored in the memory 902, and when executed by the processor 901, the above method is performed.

[0112] The specific details of the above-mentioned server can be understood by referring to the corresponding related descriptions and effects in the above-mentioned method embodiment, and will not be repeated here.

[0113] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program, and the implemented program can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the storage medium can be a disk, an optical disk, a read-only memory (ROM), a random access memory (RAM), a flash memory, a hard disk drive (HDD) or a solid-state drive (SSD), etc.; the storage medium can also include a combination of the above-mentioned types of memory.

[0114] Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A multi-sensor target fusion method, characterized in that: include: Obtain the data to be fused sent by each target sensor and the fusion target list corresponding to the last fusion time, wherein the data to be fused carries the timestamp information of the time when the data was generated; Selecting first data to be fused from the data to be fused according to the order in which the data to be fused are received; Based on the relationship between the first timestamp corresponding to the first data to be fused and the last fusion time, the fusion target list is updated using the first data to be fused; The updating of the fusion target list by using the first data to be fused based on the relationship between the first timestamp corresponding to the first data to be fused and the last fusion time includes: Determine whether a first moment corresponding to the first timestamp is earlier than the last fusion moment; When a first moment corresponding to the first timestamp is earlier than the last fusion moment, predicting and updating the first to-be-fused data based on a first time difference between the first moment and the last fusion moment; Updating the fusion target list based on the updated first data to be fused; When a first moment corresponding to the first timestamp is not earlier than the last fusion moment, predictively updating the fusion target list based on a first time difference between the first moment and the last fusion moment; The updated fusion target list is updated based on the first data to be fused.

2. The method according to claim 1, characterized in that The method further comprises: The previous fusion moment is updated based on the first moment to obtain a current fusion moment.

3. The method according to claim 2, characterized in that: The updating of the updated fusion target list based on the first to-be-fused data includes: Based on the first data to be fused, target attribute information of each target in the updated fusion target list is fused and updated; Based on the third time difference between the current fusion moment and the current moment, the target attribute of each target in the fusion target list after fusion and update is predicted to obtain the prediction result corresponding to each target at the current moment.

4. The method according to claim 1, characterized in that: The updating of the fusion target list based on the updated first data to be fused includes: Based on the updated first data to be fused, target attribute information of each target in the fusion target list is fused and updated; Based on the second time difference between the last fusion moment and the current moment, the target attribute of each target in the fusion target list after fusion and update is predicted to obtain the prediction result corresponding to each target at the current moment.

5. The method according to claim 3 or 4, characterized in that: Before predicting the target attribute of each target in the fusion target list after fusion update, the method further includes: Get the target type of each target in the fusion target list after fusion update; Determine the corresponding update strategy based on the target type of each target; Based on the update strategy corresponding to each target, the target attribute of each target in the fusion target list after fusion and update is predicted.

6. A multi-sensor target fusion device, characterized in that: include: An acquisition module is used to acquire the data to be fused sent by each target sensor and a fusion target list corresponding to the last fusion time, wherein the data to be fused carries the timestamp information of the time when the data is generated; A first processing module, configured to select first data to be fused from the data to be fused according to the order in which the data to be fused are received; A second processing module, configured to update the fusion target list using the first data to be fused based on a relationship between a first timestamp corresponding to the first data to be fused and a previous fusion time; The updating of the fusion target list using the first data to be fused based on the relationship between the first timestamp corresponding to the first data to be fused and the previous fusion moment includes: judging whether the first moment corresponding to the first timestamp is earlier than the previous fusion moment; when the first moment corresponding to the first timestamp is earlier than the previous fusion moment, predicting and updating the first data to be fused based on the first time difference between the first moment and the previous fusion moment; updating the fusion target list based on the updated first data to be fused; when the first moment corresponding to the first timestamp is not earlier than the previous fusion moment, predicting and updating the fusion target list based on the first time difference between the first moment and the previous fusion moment; and updating the updated fusion target list based on the first data to be fused.

7. An electronic device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the method as claimed in any one of claims 1 to 5 by executing the computer instructions.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and when the computer instructions are executed by a processor, the method according to any one of claims 1 to 5 is implemented.