Sensor data fusion method, apparatus, device, and storage medium

Abstract

The present disclosure provides a sensor data fusion method, device and equipment and storage medium, relates to the technical field of data processing, and particularly relates to the technical field of perception and automatic driving. The specific implementation scheme is as follows: target data of a first sensor and fusion data of the first sensor are associated and fused to obtain fusion target data of the first sensor; and the fusion target data of the first sensor and global fusion data are associated and fused to obtain updated global fusion data. According to the present disclosure, the data of multiple sensors is associated and fused, the detection performance of the multiple sensors can be fully utilized, and the method is suitable for more application scenarios.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and more particularly to the field of perception and autonomous driving technology. Background Technology

[0002] Accurate and reliable environmental perception is a crucial prerequisite for the safe and comfortable operation of automated or unmanned vehicles. Environmental perception systems are sensor-based systems. These systems require high precision and reliability to correctly detect various traffic participants and obstacles, such as stationary backgrounds, in front of, behind, to the left and right, and adjacent to the vehicle. Therefore, numerous environmental perception sensors are needed to cover all possible angles. Due to varying application requirements, the type, number, and installation location of sensors selected may differ, and many different sensors can collect vast amounts of data. Summary of the Invention

[0003] This disclosure provides a sensor data fusion method, apparatus, device, and storage medium.

[0004] According to one aspect of this disclosure, a sensor data fusion method is provided, comprising:

[0005] The target data from the first sensor and the fused data from the first sensor are correlated and fused to obtain the fused target data from the first sensor.

[0006] The target data and global fusion data from the first sensor are correlated and fused to obtain updated global fusion data.

[0007] According to another aspect of this disclosure, a sensor data fusion apparatus is provided, comprising:

[0008] The first processing module is used to perform correlation and fusion processing on the target data of the first sensor and the fused data of the first sensor to obtain the fused target data of the first sensor;

[0009] The second processing module is used to perform correlation and fusion processing on the fused target data and global fused data of the first sensor to obtain updated global fused data.

[0010] According to another aspect of this disclosure, an electronic device is provided, comprising:

[0011] At least one processor; and

[0012] The memory is communicatively connected to the at least one processor; wherein,

[0013] The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the methods of any embodiment of the present disclosure.

[0014] According to another aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions, wherein the computer instructions are used to cause the computer to perform a method according to any embodiment of this disclosure.

[0015] According to another aspect of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements a method according to any embodiment of this disclosure.

[0016] The embodiments disclosed herein, by correlating and fusing data from multiple sensors, can fully utilize the detection performance of multiple sensors, making them applicable to a wider range of application scenarios.

[0017] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0018] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0019] Figure 1 This is a schematic flowchart of a sensor data fusion method according to an embodiment of the present disclosure;

[0020] Figure 2 This is a schematic flowchart of a sensor data fusion method according to another embodiment of the present disclosure;

[0021] Figure 3 This is a schematic flowchart of a sensor data fusion method according to another embodiment of the present disclosure;

[0022] Figure 4 This is a schematic flowchart of a sensor data fusion method according to another embodiment of the present disclosure;

[0023] Figure 5 This is a diagram illustrating the architecture of multi-sensor fusion according to an embodiment of this disclosure.

[0024] Figure 6 This is a single-sensor sensing and processing diagram according to an embodiment of this disclosure;

[0025] Figure 7 This is a flowchart of sensor fusion processing based on raw data between sensors according to an embodiment of this disclosure;

[0026] Figure 8 This is a flowchart of the correlation and fusion processing based on raw data according to an embodiment of this disclosure;

[0027] Figure 9This is a schematic diagram illustrating the determination of track correlation in an embodiment of this disclosure;

[0028] Figure 10 This is a schematic diagram of the structure of a sensor data fusion apparatus according to an embodiment of the present disclosure;

[0029] Figure 11 This is a schematic diagram of the structure of a sensor data fusion apparatus according to another embodiment of the present disclosure;

[0030] Figure 12 This is a block diagram of an electronic device used to implement embodiments of the present disclosure. Detailed Implementation

[0031] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0032] Figure 1 This is a flowchart illustrating a sensor data fusion method according to an embodiment of the present disclosure, which may include:

[0033] S101. The target data of the first sensor and the fused data of the first sensor are correlated and fused to obtain the fused target data of the first sensor;

[0034] S102. Perform correlation and fusion processing on the fusion target data and global fusion data of the first sensor to obtain updated global fusion data.

[0035] In this embodiment of the disclosure, the environmental perception system of a vehicle or other device may include a variety of sensors. For example, various sensors at different installation locations or in different locations can collect a large amount of data. Sensor types may include, for example, visual sensors, LiDAR, and millimeter-wave radar. Visual sensors, such as cameras, can acquire images; LiDAR can acquire point clouds; and millimeter-wave radar can acquire point clouds. Each sensor can perform perception processing independently. For example, a perception processing flow may include: target detection, target classification, scene semantic segmentation, target dynamic and static recognition, etc.; target association and tracking; target existence estimation; target filtering based on target information such as existence or confidence level; target data output according to a target protocol; and module output output according to a frame protocol.

[0036] In this embodiment of the disclosure, target data of the sensor can be obtained by performing perception processing on the raw data of the sensor. The target data of the sensor may include attribute information of one or more targets sensed by the sensor, such as the target's position, shape, orientation, velocity, acceleration, category, etc. The fused data of the first sensor may also include attribute information of one or more targets sensed by the first sensor and other sensors.

[0037] First, raw data from a single sensor and fused data are correlated and fused to obtain the target data for that single sensor. Then, the target data from multiple sensors is used to obtain global fused data. This global fused data can include not only the target data from multiple sensors but also the fused track data from multiple sensors. Then, the target data from each sensor and the global fused data are correlated and fused again to update the global fused data. By correlating and fusing data from multiple sensors, the detection performance of each sensor can be fully utilized, making it suitable for a wider range of application scenarios.

[0038] Figure 2 This is a flowchart illustrating a sensor data fusion method according to another embodiment of the present disclosure. The method may include one or more features of the sensor data fusion method described in the above embodiments. In one implementation, the method further includes:

[0039] S201. The target data from the first sensor and the target data from at least one second sensor are fused to obtain the fused data from the first sensor.

[0040] In this embodiment, fused data from the first sensor can be obtained by fusing raw data from the first sensor with raw data from other sensors and then performing sensory processing. Alternatively, fused data from the first sensor can also be obtained by fusing target data from the first sensor with target data from other sensors. Taking sensors such as LiDAR, vision sensors, and millimeter-wave radar as examples, the LiDAR sensor is the first sensor, and the vision sensor and millimeter-wave radar are the second sensors. The target data from the LiDAR sensor is {L}, the target data from the vision sensor is {C}, and the target data from the millimeter-wave radar sensor is {R}. The fused data obtained by fusing the LiDAR sensor and the vision sensor is {LC}, and the fused data obtained by fusing the LiDAR sensor and the millimeter-wave radar is {LR}. Fusing target data from different sensors can improve overall detection performance and expand applicable scenarios.

[0041] In one embodiment, S201 fuses the target data from the first sensor and the target data from at least one second sensor to obtain fused data from the first sensor, including:

[0042] The target data from the first sensor and the target data from multiple second sensors are combined based on the projection relationship to obtain the expression mode, and then the features are extracted and fused in pairs to obtain multiple fused data from the first sensor.

[0043] In this embodiment, sensors can be fused pairwise based on raw data. The pairwise fusion methods can include: extracting features based on projection relationships or extracting features separately from the raw data and then fusing them together. Since different sensors have different detection performances and are applicable to different scenarios, pairwise fusion based on raw data can improve performance and expand the range of applicable scenarios.

[0044] Taking LiDAR, visual sensors, and millimeter-wave radar as examples, data can be fused between each other based on raw data. Fusion of LiDAR point clouds and visual images, compared to single-LiDAR perception processing, can improve target category accuracy; compared to single-visual sensor perception processing, it can improve target position accuracy. Fusion of LiDAR point clouds and millimeter-wave radar point clouds, compared to single-LiDAR perception processing, can improve target detection performance in special weather conditions such as rain, fog, and sandstorms; compared to single-millimeter-wave radar perception processing, it can improve target position accuracy and detection performance in environments with low or high light; compared to single-millimeter-wave radar perception processing, it can improve target category accuracy. After the raw data between sensors is fused, target detection, target classification, and target dynamic / static recognition can be performed. The subsequent processing steps are similar to single-sensor perception processing, and the final module output can be output according to a frame protocol.

[0045] In one embodiment, S101 performs correlation and fusion processing on the target data of the first sensor and the fused data of the first sensor to obtain the fused target data of the first sensor, including:

[0046] The target data from the first sensor is compared with different types of fused data according to the correlation method to determine whether the expected target exists.

[0047] When there are multiple desired targets in a certain association method, the multiple desired targets of the association method are merged to obtain a merged target;

[0048] Based on the expected target or fusion target corresponding to different association methods, the fusion target data of the first sensor is obtained.

[0049] In this embodiment of the disclosure, the fusion of the first sensor with different second sensors may yield different types of fused data. For example, the target data from the lidar sensor is {L}, the fused data obtained by fusing the lidar sensor and the vision sensor is {LC}, and the fused data obtained by fusing the lidar sensor and the millimeter-wave radar is {LR}. {L}, {LC}, and {LR} can have multiple association methods. Based on one association method, one or more desired targets may be obtained, or no desired targets may be obtained. If multiple desired targets are obtained based on one association method, these desired targets can be fused to obtain a fused target. Then, by summarizing the desired targets or fused targets corresponding to each association method, the fused target data of the first sensor can be obtained. The fused target data may include a list of fused targets of the first sensor. This list may include information such as the identifier and attributes of the fused targets.

[0050] In this embodiment of the disclosure, the desired target can be matched from the target data and fused data of the first sensor by association, and multiple desired targets with the same association method can be fused to obtain fused target data, which is beneficial to improving the overall detection performance and expanding the target detection scenario.

[0051] In one implementation, the association method includes at least one of the following:

[0052] The target data from the first sensor is not associated with any target among all the fused data from the first sensor;

[0053] The first fused data does not relate to the target data of the first sensor or the target data of the second fused data;

[0054] The second fused data is not associated with the target data of the first sensor or with the target data of the first fused data.

[0055] The target data from the first sensor is related to the first fused data but not to the second fused data.

[0056] The target data from the first sensor is related to the second fused data but is not related to the first fused data.

[0057] The first fused data is related to the second fused data, but the target data of the first sensor is not related to the target data of the first sensor;

[0058] The target data of the first sensor is associated with all the fused data of the first sensor;

[0059] The first fused data and the second fused data are fused data obtained by fusing the first sensor with different second sensors.

[0060] For example, the target data from the LiDAR sensor is {L}, the fused data obtained by fusing the LiDAR sensor and the vision sensor is {LC}, and the fused data obtained by fusing the LiDAR sensor and the millimeter-wave radar is {LR}. An example of the association method is as follows:

[0061] Method 1: Targets in {L} that are not associated with targets in {LC} and {LR};

[0062] Method 2: Targets in {LC} that are not associated with targets in {L} and {LR};

[0063] Method 3: Targets in {LR} that are not associated with targets in {L} and {LC};

[0064] Method 4: Targets that are associated with targets in {L} and {LC} but not with targets in {LR};

[0065] Method 5: Targets that are associated with targets in {L} and {LR} but not with targets in {LC};

[0066] Method 6: Targets that are associated with targets in {LC} and {LR} but not with targets in {L};

[0067] Method 7: Targets associated with {L}, {LC}, and {LR}.

[0068] In this embodiment, the association method between the target data of the first sensor and the first and second fused data is merely an example and not a limitation. In practical applications, the target data of the first sensor can be associated with more fused data. The association methods can be derived through permutations and combinations, which will not be exhaustively listed here. By using various association methods, the desired target can be matched from the target data of the first sensor and the fused data, thereby obtaining fused target data, which can improve overall detection performance and expand the applicable scenarios.

[0069] Figure 3 This is a flowchart illustrating a sensor data fusion method according to another embodiment of the present disclosure. The method may include one or more features of the sensor data fusion method described in the above embodiments. In one implementation, the method further includes:

[0070] S301. Based on the fused target data of the first sensor from multiple frames, obtain the fused trajectory data of the first sensor;

[0071] S302. Based on the global fusion target data of multiple frames, obtain the global fusion track data.

[0072] In this embodiment, the fused target data of a single frame may include tracking information for each fused target, such as the acquisition time and location of the fused target in the current frame. Fusion track data can be obtained from the fused target data of multiple frames. Global fused target data can be obtained from the fused target data of multiple sensors. Global fused track data can be obtained from the global fused target data of multiple frames. Based on the relationship between targets and tracks, correlated fusion tracking processing can be performed. For example, fused target lists based on different sensors can be fused together through the correlation results of targets and tracks or tracks and tracks to generate a global fused target list, and track management can be performed on the global fused targets in the global fused target list.

[0073] The fusion results based on raw data from different sensors may be asynchronous and unordered. The target-track or track-track association fusion tracking algorithm used can maintain a stable global track and consistency over time, even if a target moves within the fields of view of multiple sensors. Target-track or track-based association fusion tracking processing may include: associating the fused target or track with the global fused target or track to determine which targets from different sensor raw data might represent the same target in reality; performing various attribute fusion processing on the associated targets based on the association processing results; and finally, tracking the fused target.

[0074] In one implementation, S102 performs correlation fusion processing on the fused target data and global fusion data of the first sensor to obtain updated global fusion data, including:

[0075] Based on the fused target data and global fused data from the first sensor, an association matrix is ​​obtained;

[0076] Use at least one of the following from the fused target data: tracking information, inter-sensor correlation information, category information, dynamic and static attributes, and velocity information, to examine and update the correlation matrix.

[0077] In this embodiment, the elements in the correlation matrix can represent whether a target or track in the fused target data of the first sensor is correlated with a target or track in the global fused data. For example, if the fused target data of the first sensor includes M targets and the global fused data includes N targets, then the correlation matrix can include M×N elements. Different sensors may yield different correlation matrices. For example, millimeter-wave radar corresponds to an M1×N correlation matrix, a camera corresponds to an M2×N correlation matrix, and a lidar corresponds to an M3×N correlation matrix. After establishing the correlation matrix, the accuracy of the elements in the correlation matrix can be verified using various attribute information from the original sensor data or target data, thereby obtaining a more accurate correlation between targets or tracks. This, in turn, facilitates more accurate updates to the global fused data.

[0078] In one implementation, the elements in the association matrix are used to represent at least one of the following:

[0079] The correlation distance between the target in the fused target data of the first sensor and the target in the global fused data;

[0080] The correlation distance between the target in the fused target data of the first sensor and the track in the global fused data;

[0081] The correlation distance between the track in the fused track data of the first sensor and the track in the global fused data;

[0082] The correlation distance between the trajectory in the fused trajectory data of the first sensor and the target in the global fused data.

[0083] In this embodiment of the disclosure, the elements in the initially constructed correlation matrix can represent the correlation distance between two objects. This correlation distance can be the correlation distance between targets or the correlation matrix between a target and a track. Generally, the closer the correlation distance, the greater the probability that they belong to the same target. Therefore, constructing a correlation matrix based on correlation distance can accurately reflect the correlation relationship between targets or tracks.

[0084] In one implementation, if the association distance between two objects represented by an element in the association matrix is ​​greater than or equal to a threshold, the corresponding element in the association matrix is ​​a first value; if the association distance is less than the threshold, the corresponding element in the association matrix is ​​equal to the association distance divided by the threshold.

[0085] For example, if the correlation distance L1 between a target in the fused target data of the first sensor and a target in the global fused data is greater than a first threshold T1, the corresponding element is 0. Otherwise, the value of this element is equal to L1 / T1.

[0086] For example, if the correlation distance L2 between the target in the fused target data of the first sensor and the track in the global fused data is greater than the second threshold T2, the corresponding element is 0. Otherwise, the value of this element is equal to L2 / T2.

[0087] For example, if the correlation distance L3 between the track in the fused track data of the first sensor and the track in the global fused data is greater than the third threshold T3, the corresponding element is 0. Otherwise, the value of this element is equal to L3 / T3.

[0088] For example, if the correlation distance L4 between the trajectory in the fused trajectory data of the first sensor and the target in the global fused data is greater than the fourth threshold T4, the corresponding element is 0. Otherwise, the value of this element is equal to L4 / T4.

[0089] In this embodiment, the first value can be 0, or other values ​​such as 1, 2, etc. The values ​​of the elements of the correlation matrix are determined based on the correlation distance, and the constructed correlation matrix can accurately reflect the correlation between targets or tracks.

[0090] In one implementation, S102 performs correlation fusion processing on the fused target data and global fusion data of the first sensor to obtain updated global fusion data, and further includes:

[0091] Based on the association pairs obtained from the updated association matrix, attribute fusion processing is performed on the fused target data of a single sensor and the global fused target data;

[0092] Update the global fused track data based on the global fused target data after attribute fusion processing.

[0093] In this embodiment of the disclosure, the elements of the updated correlation matrix can represent which targets or tracks are correlated. Targets or tracks with correlation can be called correlation pairs. For example, if the value of an element is not 0, then target A1 in the fused target data of the first sensor represented by that element is correlated with target A2 in the globally fused target data. Attribute fusion processing can be performed on target A1 and target A2. As another example, if the value of an element is not 0, then target A1 in the fused target data of the first sensor represented by that element is correlated with track B2 in the globally fused target data. Attribute fusion processing can be performed on the attributes of target A2 included in target A1 and track B2 to update the track of target A2. For example, if track B2 includes the track of target A2 from time t1 to t2, the attributes of target A1 at time t3 can be added to track B2, and the updated track includes data from time t1 to t3.

[0094] In this embodiment of the disclosure, if a sensor's fused target does not match an associated track or target in the global fused data, a new track or target can be created for subsequent association and tracking. Through attribute fusion and track updates, target tracking can be performed more quickly and accurately.

[0095] In one implementation, the attribute fusion process includes at least one of the following: position-shape fusion; orientation fusion; category fusion; motion fusion; existence fusion; and semantic fusion. Each attribute can be fused separately, and different association pairs may require different attributes to be fused. Association pairs corresponding to different sensors may also require different attributes to be fused. By fusing multiple attributes, more types of sensors can be supported, improving target detection performance.

[0096] In one implementation, the position-shape fusion includes: filtering or smoothing position observations from different sensors, and performing time-series estimation on low-confidence position observations to obtain the position of the fused target; or, selecting high-precision shape observations from shape observations from different sensors, and correcting these high-precision shape observations using other shape observations to obtain the shape of the fused target. For example, if multiple sensors provide time-series position points of a target including P1, P2, and P3, where P1 and P3 have high confidence and P2 has low confidence, P2 can be estimated using P1 and P3 to update its value. Furthermore, if the shape observations detected by the camera have the highest accuracy, shape observations such as length, width, and height can be selected from the camera-detected values, and then the shape observations detected by millimeter-wave radar can be used to correct the camera-detected shape observations, such as the width.

[0097] In one implementation, the orientation fusion includes obtaining the orientation of the fusion target based on statistical information from the orientation observations. For example, the statistical information may include accuracy statistics, forward / reverse jump statistics, etc. The orientation detected by a sensor with higher accuracy or more accurate forward / reverse jump is selected as the orientation of the fusion target, depending on the requirements.

[0098] In one implementation, the category fusion includes: obtaining the classification probability corresponding to the classification vector based on the perceptual processing results of the original target data, and obtaining the category of the fused target based on the classification probability. For example, the fused target can be classified using methods such as machine learning or data mining to determine the category of the fused target, such as pedestrian, vehicle, animal, ground, guardrail, green plant, etc.

[0099] In one implementation, the motion fusion includes estimating the motion state variables of the fused target based on a motion model and a Kalman filter algorithm. For example, the motion model of the fused target may include motion state variables such as position, velocity, acceleration, orientation angle, and angular velocity. The Kalman filter algorithm can be used to estimate the motion state variables of the fused target at a certain moment based on the motion model.

[0100] In one implementation, the existence fusion includes: calculating the existence probability of the fused target. This can be achieved by first calculating the existence probability of the same target separately based on data from sensors from different sources, and then combining the results to calculate the existence probability of the fused target. Alternatively, the existence probability of the fused target can be calculated directly based on the fused target data from sensors from different sources.

[0101] In one implementation, the semantic fusion includes: obtaining dynamic and static occlusion regions based on the original target data, and providing disappearance semantics for fused targets entering the occlusion regions. For example, the point cloud of a millimeter-wave radar in front of a vehicle may not include the area behind the vehicle, which is an occlusion region. If a fused target enters this occlusion region, then the fused target can be provided with disappearance semantics.

[0102] By fusing various attributes of the target in the correlation pair, data from more types of sensors can be comprehensively utilized, improving the accuracy of target detection performance and expanding the applicable scenarios of target detection.

[0103] Figure 4 This is a flowchart illustrating a sensor data fusion method according to another embodiment of the present disclosure. The method may include one or more features of the sensor data fusion method described in the above embodiments. In one implementation, the method further includes:

[0104] S401. Filter targets based on their association information, attribute information, and tracking information; the targets include those in the updated global fusion data.

[0105] In this embodiment, the association information of the fusion target may include which sensors the fusion information is associated with. If a fusion target is associated with few sensors or has low accuracy, the fusion target can be filtered out. The attribute information of the fusion target may include attribute information fused according to the above attribute fusion method. If the fused attribute information is obviously unreasonable, for example, the speed obviously exceeds a reasonable range, the fusion target can be filtered out. The tracking information of the fusion target may include tracking information for each frame. If the tracking information is obviously unreasonable, for example, the position jump between two or more frames obviously exceeds a reasonable range, the fusion target can be filtered out.

[0106] In this embodiment of the disclosure, the specific operation of filtering out fusion targets may include deleting the fusion target from the fusion target data or global fusion data of the first sensor, or it may include not deleting the fusion target but not pushing the fusion target to a specific application. By filtering targets, unqualified targets can be removed, interfering targets can be reduced, and the accuracy of the overall detection results can be improved.

[0107] In one implementation, such as Figure 4 As shown, it also includes:

[0108] S402. Based on the application's data source and updated global fusion data, output the fusion targets of interest in the application.

[0109] In this embodiment, the application's data source may include data from the main vehicle, high-precision maps, road boundaries, and occupancy grid maps (dynamic or static). Depending on the specific application scenario, the application can output fusion targets of interest based on updated global fusion data. For example, the high-precision map can display the locations and tracking tracks of obstacles detected by the main vehicle's multi-sensor fusion data. By combining this with the application's data source, richer application scenarios can be expanded based on multi-sensor fusion data.

[0110] Because multiple sensors are often out of sync, significant post-processing and communication delays may occur. The sensor fusion method provided in this disclosure is a general multi-sensor asynchronous fusion method. This method is modular, practical, and scalable, allowing different application requirements or different sensor configurations to be applied to the same fusion architecture, and is also easy to implement in automotive applications. This method can be based on, for example... Figure 5 The process shown will be executed accordingly.

[0111] First, the frame protocol and target protocol used in this method will be introduced.

[0112] The frame protocol mainly includes the following: timestamp information, coordinate system transformation information, raw data information, raw quality information, raw data semantic information, target list information, association information between the target and the sensor fused target, and information associated with the global fused target.

[0113] The targets in the target list can be obtained according to the target protocol. The target protocol includes the following: target information (target location, size, shape, etc.), target tracking information (tracking number, state variables and covariance matrix, etc.), target category information (category, classification probability corresponding to the classification vector), target confidence information, target existence information (existence probability), target dynamic and static information, and the original data information corresponding to the target.

[0114] like Figure 5 As shown, the multi-sensor asynchronous fusion method may include the following steps:

[0115] S1. Perceptual processing based on raw data. Divided into two main categories:

[0116] (1) Individual sensor sensing and processing, that is, each sensor performs sensing and processing independently, such as Figure 6 As shown. Commonly used sensor output data mainly includes raw data or target-level data. If the output is raw data, it needs to undergo sensing processing to obtain target-level data, such as... Figure 6 As shown. For example, LiDAR sensors typically output raw point cloud data, which needs to be processed to obtain target data; millimeter-wave radars, depending on the manufacturer, provide different outputs, usually including raw point cloud data and target data; vision sensors typically output image data, which needs to be processed to obtain target data. A typical perception processing flow mainly includes: target detection, target classification, scene semantic segmentation, target dynamic and static recognition, etc.; target correlation and tracking; target existence estimation; target filtering based on existence or confidence level; target data output according to target protocol; and the perception processing module output can be according to frame protocol.

[0117] (2) Pairwise fusion processing of raw data between each sensor, such as... Figure 7 As shown. The fusion of raw data from different sensors commonly includes two methods: extracting features based on projection relationships or extracting features separately from the raw data and then fusing them together. After the raw data from the sensors is fused, sensory processing such as target detection, target classification, and target dynamic / static recognition is performed. These sensory processing steps are similar to single-sensor sensory processing, and the final output of the sensory processing module can be output according to a frame protocol.

[0118] S2. Target association fusion processing based on raw data.

[0119] like Figure 8As shown, the target list of the frame information obtained by sensor ni perception processing includes the original data corresponding to target oi, and the target list of the frame information obtained by sensor ni and sensor nj based on the perception fusion processing of the original data includes the original data corresponding to target oj. The association fusion processing based on the original data of sensor ni includes: if the original data corresponding to target oi and target oj have a certain proportion of the sameness, and the relevant attributes of target oi and target oj may have consistency, then target oi and target oj are associated. The fused target oij is obtained by fusing target oi and target oj, and the semantic information (e.g., point category: target, ground, guardrail, greenery, noise, etc.) based on the original data of sensor ni and sensor nj can be updated separately. Finally, the target association fusion processing module outputs the fused target based on the original data of sensor ni according to the frame protocol. Furthermore, the frame information obtained by sensor nj perception processing, and the frame information obtained by sensor ni and sensor nj based on the perception fusion processing of the original data, can be associated and fused based on the original data of sensor nj. Then, the fused target based on the original data of sensor nj can be output according to the frame protocol. For three or more types of sensors, please refer to [link to relevant documentation]. Figure 8 And so on.

[0120] Taking LiDAR, visual sensors, and millimeter-wave radar as examples, this section illustrates the association and fusion results based on raw LiDAR point cloud data. Individual sensor processing yields target data as LiDAR {L}, camera {C}, and radar {R}. Fusion processing of LiDAR and visual sensors yields target data {LC}, and fusion processing of LiDAR and millimeter-wave radar yields target data {LR}. Association result explanation: The number 1 after each letter indicates association, and 0 indicates no association. {L}, {LC}, and {LR}, based on the raw LiDAR point cloud data, yield {L1*LC0*LR0, L0*LC1*LR0, L0*LC0*LR1, L1*LC1*LR0, L1*LC0*LR1, L0*LC1*LR1, L1*LC1*LR1}.

[0121] The objectives represented by each association method are as follows:

[0122] L1*LC0*LR0 represents a target in {L} that is not associated with any target in {LC} or {LR};

[0123] L0*LC1*LR0 represents a target in {LC} that is not associated with any target in {L} or {LR};

[0124] L0*LC0*LR1 represents a target in {LR} that is not associated with any target in {L} or {LC};

[0125] L1*LC1*LR0 represents a target that is associated with targets in {L} and {LC} but not with targets in {LR};

[0126] L1*LC0*LR1 represents a target that is associated with targets in {L} and {LR} but not with targets in {LC};

[0127] L0*LC1*LR1 represents a target that is associated with targets in {LC} and {LR} but not with targets in {L};

[0128] L1*LC1*LR1 represents the target that is associated in {L}, {LC} and {LR}.

[0129] A specific example is as follows:

[0130] There are 4 objectives in {L}: l1, l2, l3, l4;

[0131] There are 4 objectives in {LC}: lc1, lc2, lc3, and lc4;

[0132] There are 4 targets in {LR}: lr1, lr2, lr3, lr4;

[0133] The associations {L} and {LC} result in the association pair: (l2, lc1)(l3, lc4);

[0134] The associations {L} and {LR} result in the association pairs: (l2, lr3) and (l4, lr2).

[0135] The associations {LC} and {LR} result in association pairs: (lc1, lr3) and (lc2, lr4).

[0136] Association results: (l1), (lc3), (lr1), (l3,lc4), (l4,lr2), (lc2,lr4), (l2,lc1,lr3);

[0137] The correspondence between each association method and the association result is as follows:

[0138] L1*LC0*LR0:(l1);

[0139] L0*LC1*LR0: (lc3);

[0140] L0*LC0*LR1:(lr1);

[0141] L1*LC1*LR0: (l3,lc4);

[0142] L1*LC0*LR1: (l4,lr2);

[0143] L0*LC1*LR1: (lc2,lr4);

[0144] L1*LC1*LR1: (l2,lc1,lr3);

[0145] The latter four categories of the above association results (i.e., target association matching) each include multiple targets, which can be fused separately to obtain the fused target.

[0146] S3. Target or track-based correlation fusion tracking processing.

[0147] In this step, the fusion target lists obtained from raw data from different sensors are fused together through the association results of targets and tracks, or through the association results of tracks, to generate a global fusion target list. Track management is performed on the global fusion targets. Scene management is also performed, including global fusion targets, raw data, and semantic information of the raw data.

[0148] (1) Target or track-based association matching processing.

[0149] Typically, the association matrix is ​​obtained by calculating the association distance between the fused target or track obtained from raw data from different sensors and the globally fused target or track. An initial association distance threshold is set; if the distance is within the threshold, it is considered associated, and the value is the association distance divided by the threshold; otherwise, the value is 0. During the association process, attributes are used to verify the accuracy of the association. For example, whether the original tracking information of the fused target is consistent; whether the association information between the original sensors of the fused target is consistent; for category information, the similarity between two classification vectors is calculated to obtain a similarity metric, which is used as a weighted value and applied to the association matrix; for example, whether the dynamic and static attributes or velocity information of the target are consistent, etc., to verify the accuracy of the association. For the association matrix, the optimal one-to-one matching is then achieved according to common matching algorithms.

[0150] like Figure 9 As shown, it's possible to compare the established target track (global) with the track of the new frame of target data (single frame) to see if they are related to the target. If so, the new frame of target data can be filtered before track updates to obtain the target track. Otherwise, unrelated track preservation can be performed, establishing a new track based on the unrelated new target. Then, it's determined whether to delete the track based on the number of tracking losses. If the number of losses is too high, the new track can be deleted to continue comparing other target tracks; otherwise, the new track can be retained as a target track for subsequent comparisons.

[0151] (2) Fusion processing based on target or trajectory.

[0152] Based on the above association processing results, the associated targets are then subjected to fusion processing. Fusion processing can be divided into: position and shape fusion; orientation fusion; category fusion; motion fusion, including velocity, acceleration, and static / dynamic fusion; existence fusion, including existence probability fusion; semantic fusion, including scene semantics, occlusion semantics, and vanishing semantics fusion; and fusion of other attributes of interest.

[0153] Location-shape fusion: For location, based on results from different sources and processed as observations, filtering or smoothing is used to perform time-series estimation of low-confidence locations. For shape, since the accuracy varies due to results from different sensors, high-accuracy results are selected from the observations and corrected using other observations.

[0154] Orientation fusion: The orientation of the target is obtained by comprehensively considering the statistical information of the observations. For example, the orientation information of a target based on the raw data of LiDAR is highly accurate, but it is prone to jumping between forward and reverse directions; the orientation information of a target based on a vision camera has high overall accuracy in both forward and reverse directions, but lower precision.

[0155] Category fusion: Based on the perceptual processing results of the raw data, the classification probability corresponding to the classification vector can be obtained. This typically involves using some form of machine learning or data mining method to classify the target data.

[0156] Motion fusion is performed to construct a motion module, and commonly used Kalman filtering is employed to estimate acceleration and velocity.

[0157] Existence fusion: Target existence is measured by the probability of its existence. Existence probability describes the likelihood that a target will actually be a real, detectable target, rather than a false detection. Target existence probability estimation can significantly improve target management and selection, thereby increasing the probability of true detection and reducing the probability of false detection. For example, based on Dempster-Shafer evidence theory, target existence probabilities can be fused when sensor uncertainties exist.

[0158] In semantic fusion, occlusion semantics are used to obtain dynamic and static occlusion regions based on the original data, and to provide disappearance semantics for targets entering the occlusion regions.

[0159] (3) Fusion target tracking processing.

[0160] The tracks are updated based on the fusion results.

[0161] S4. Merge target filtering.

[0162] Target fusion filtering primarily relies on a comprehensive consideration of its association information, fusion attribute information, and tracking information to filter out targets that do not meet expectations. For example, the association information of the fused target, i.e., which sensors the target was detected and processed from, is crucial. If the location of the fused target is an overlapping area that is stably detected by multiple sensors, but it only comes from the processing of one sensor, the probability that the target is not a real target will be lower. The existence of the fused target is also important; the probability of the fused target being present is very low. The tracking information also includes short tracking time.

[0163] For example: Within 30m directly in front of the main vehicle, in a non-blind spot, theoretically all three types of sensors should be able to detect the target stably. The fused target association information shows that the target is obtained from the LiDAR perception processing; the target obtained from the perception processing of other modules is not associated with it, and this target may be a false detection target.

[0164] S5. Combine with other data sources to output the target focus and information according to the needs.

[0165] Other commonly used data sources include vehicle data, high-precision maps, lane lines, road boundaries, and occupancy grid maps (dynamic or static). Depending on the application requirements, these data output the targets and information of interest. For example, in public road environments, high-precision maps are often used to match targets onto the map, determining which targets are located in which lanes and identifying which targets are in the vehicle's lane and adjacent lanes.

[0166] Figure 10 This is a schematic diagram of a sensor data fusion apparatus according to an embodiment of the present disclosure, the apparatus may include:

[0167] The first processing module 1001 is used to perform correlation and fusion processing on the target data of the first sensor and the fused data of the first sensor to obtain the fused target data of the first sensor.

[0168] The second processing module 1002 is used to perform correlation and fusion processing on the fused target data and global fused data of the first sensor to obtain updated global fused data.

[0169] Figure 11 This is a schematic diagram of a sensor data fusion apparatus according to another embodiment of the present disclosure. The apparatus may include one or more features of the obstacle detection apparatus of the above embodiments. In one possible implementation, the apparatus further includes:

[0170] The third processing module 1003 is used to fuse the target data of the first sensor and the target data of at least one second sensor to obtain the fused data of the first sensor.

[0171] In one possible implementation, such as Figure 11As shown, the third processing module 1003 includes:

[0172] The feature fusion submodule 1102 is used to extract features from the target data of the first sensor and the target data of multiple second sensors based on the projection relationship, and then fuse them in pairs to obtain multiple fused data of the first sensor.

[0173] In one possible implementation, such as Figure 11 As shown, the first processing module 1001 includes:

[0174] The target association submodule 1103 is used to compare the target data of the first sensor with different types of fused data according to the association method to see if the expected target exists.

[0175] The target fusion submodule 1104 is used to fuse multiple expected targets in a certain association method to obtain a fused target when multiple expected targets exist in a certain association method.

[0176] The data processing submodule 1105 is used to obtain the fusion target data of the first sensor according to the expected target or fusion target corresponding to different association methods.

[0177] In one possible implementation, the association method includes at least one of the following:

[0178] The target data from the first sensor is not associated with any target among all the fused data from the first sensor;

[0179] The first fused data does not relate to the target data of the first sensor or the target data of the second fused data;

[0180] The second fused data is not associated with the target data of the first sensor or with the target data of the first fused data.

[0181] The target data from the first sensor is related to the first fused data but not to the second fused data.

[0182] The target data from the first sensor is related to the second fused data but is not related to the first fused data.

[0183] The first fused data is related to the second fused data, but the target data of the first sensor is not related to the target data of the first sensor;

[0184] The target data of the first sensor is associated with all the fused data of the first sensor;

[0185] The first fused data and the second fused data are fused data obtained by fusing the first sensor with different second sensors.

[0186] In one possible implementation, such as Figure 11 As shown, the device also includes:

[0187] The sensor trajectory module 1004 is used to obtain the fused trajectory data of the first sensor based on the fused target data of the first sensor in multiple frames.

[0188] The global trajectory module 1005 is used to obtain global fused trajectory data based on multi-frame global fused target data.

[0189] In one possible implementation, such as Figure 11 As shown, the second processing module 1002 includes:

[0190] The matrix establishment submodule 1108 is used to obtain the correlation matrix based on the fused target data and global fused data of the first sensor;

[0191] The matrix update submodule 1109 is used to examine and update the correlation matrix using at least one of the tracking information, sensor correlation information, category information, dynamic and static attributes, and velocity information in the fused target data.

[0192] In one possible implementation, the elements in the association matrix are used to represent at least one of the following:

[0193] The correlation distance between the target in the fused target data of the first sensor and the target in the global fused data;

[0194] The correlation distance between the target in the fused target data of the first sensor and the track in the global fused data;

[0195] The correlation distance between the track in the fused track data of the first sensor and the track in the global fused data;

[0196] The correlation distance between the trajectory in the fused trajectory data of the first sensor and the target in the global fused data.

[0197] In one possible implementation, if the association distance between two objects represented by an element in the association matrix is ​​greater than or equal to a threshold, the corresponding element in the association matrix is ​​a first value; if the association distance is less than the threshold, the corresponding element in the association matrix is ​​equal to the association distance divided by the threshold.

[0198] In one possible implementation, such as Figure 11 As shown, the second processing module 1002 further includes:

[0199] The attribute fusion submodule 1110 is used to perform attribute fusion processing on the fusion target data of a single sensor and the global fusion target data based on the association pairs obtained from the updated association matrix.

[0200] The global update submodule 1111 is used to update the global fused track data based on the global fused target data after attribute fusion processing.

[0201] In one possible implementation, the attribute fusion process includes at least one of the following: position-shape fusion; orientation fusion; category fusion; motion fusion; existence fusion; semantic fusion.

[0202] In one possible implementation, the position shape fusion includes: filtering or smoothing position observations from different sensors and performing time-series estimation on low-confidence position observations to obtain the position of the fused target; or, selecting high-precision shape observations from shape observations from different sensors and correcting the high-precision shape observations using other shape observations to obtain the shape of the fused target.

[0203] This orientation fusion includes: obtaining the orientation of the fusion target based on statistical information from the orientation observations;

[0204] This category fusion includes: obtaining the classification probability corresponding to the classification vector based on the perceptual processing results of the original target data, and obtaining the category of the fused target based on the classification probability;

[0205] The motion fusion includes estimating the motion state variables of the fusion target based on the motion model and the Kalman filter algorithm;

[0206] This existence fusion includes: calculating the existence probability of the fusion target;

[0207] This semantic fusion includes: obtaining dynamic and static occlusion regions based on the original target data, and providing disappearance semantics for the fusion target entering the occlusion region.

[0208] In one possible implementation, such as Figure 11 As shown, the device also includes:

[0209] The filtering module 1006 is used to filter targets based on the association information, attribute information and tracking information of the fusion targets; the fusion targets include the fusion targets in the updated global fusion data.

[0210] In one possible implementation, such as Figure 11 As shown, the device also includes:

[0211] Output module 1007 is used to output the fusion target of interest in the application based on the application's data source and updated global fusion data.

[0212] The specific functions and examples of each module and submodule of the apparatus in this disclosure can be found in the relevant descriptions of the corresponding steps in the above method embodiments, and will not be repeated here.

[0213] The acquisition, storage, and application of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0214] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0215] Figure 12 A schematic block diagram of an example electronic device 1200 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0216] like Figure 12 As shown, device 1200 includes a computing unit 1201, which can perform various appropriate actions and processes according to a computer program stored in read-only memory (ROM) 1202 or a computer program loaded from storage unit 1208 into random access memory (RAM) 1203. The RAM 1203 may also store various programs and data required for the operation of device 1200. The computing unit 1201, ROM 1202, and RAM 1203 are interconnected via bus 1204. Input / output (I / O) interface 1205 is also connected to bus 1204.

[0217] Multiple components in device 1200 are connected to I / O interface 1205, including: input unit 1206, such as keyboard, mouse, etc.; output unit 1207, such as various types of monitors, speakers, etc.; storage unit 1208, such as disk, optical disk, etc.; and communication unit 1209, such as network card, modem, wireless transceiver, etc. Communication unit 1209 allows device 1200 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0218] The computing unit 1201 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 1201 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 1201 performs the various methods and processes described above, such as sensor data fusion methods. For example, in some embodiments, the sensor data fusion method may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 1208. In some embodiments, part or all of the computer program may be loaded and / or installed on device 1200 via ROM 1202 and / or communication unit 1209. When the computer program is loaded into RAM 1203 and executed by the computing unit 1201, one or more steps of the sensor data fusion method described above may be performed. Alternatively, in other embodiments, the computing unit 1201 may be configured to perform a sensor data fusion method by any other suitable means (e.g., by means of firmware).

[0219] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0220] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0221] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0222] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0223] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with embodiments of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0224] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0225] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0226] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A sensor data fusion method, comprising: The process involves associating and fusing target data from a first sensor with fused data from the first sensor to obtain fused target data from the first sensor. This includes: comparing the target data from the first sensor with fused data of different types according to an association method to determine if a desired target exists; fusing multiple desired targets in one association method to obtain a fused target; and obtaining the fused target data from the first sensor based on the desired target or fused target corresponding to different association methods. The fused data from the first sensor is obtained by fusing the target data from the first sensor with target data from at least one second sensor. The first sensor and the second sensor are sensors of different types. The fusion target data and global fusion data of the first sensor are correlated and fused to obtain updated global fusion data; wherein, the global fusion data includes fusion target data of multiple sensors.

2. The method according to claim 1, further comprising: The target data from the first sensor and the target data from at least one second sensor are fused together to obtain the fused data from the first sensor.

3. The method according to claim 2, wherein the target data from the first sensor and the target data from at least one second sensor are fused to obtain the fused data from the first sensor, comprising: The target data from the first sensor and the target data from multiple second sensors are combined based on the projection relationship to obtain the expression mode, and then the features are extracted and fused in pairs to obtain multiple fused data from the first sensor.

4. The method according to claim 1, wherein, The association method includes at least one of the following: The target data from the first sensor has no associated target with any of the fused data from the first sensor; The first fused data does not relate to the target data from the first sensor or the second fused data; The second fused data is not associated with the target data of the first sensor or the target data of the first fused data; The target data from the first sensor is related to the first fused data but not to the second fused data; The target data from the first sensor is related to the second fused data, but is not related to the first fused data. The first fused data and the second fused data are related to a target, but the target data of the first sensor is not related to the target data of the first sensor. The target data from the first sensor has a related target with all fused data from the first sensor; The first fused data and the second fused data are fused data obtained by fusing the first sensor with different second sensors.

5. The method according to any one of claims 1 to 4, further comprising: Based on the fused target data of the first sensor from multiple frames, the fused trajectory data of the first sensor is obtained; Global fused trajectory data is obtained based on multi-frame global fused target data.

6. The method according to any one of claims 1 to 4, wherein, The fused target data and global fused data from the first sensor are correlated and fused to obtain updated global fused data, including: Based on the fused target data and global fused data from the first sensor, an association matrix is ​​obtained; The correlation matrix is ​​examined and updated using at least one of the tracking information, inter-sensor correlation information, category information, dynamic and static attributes, and velocity information from the fused target data.

7. The method according to claim 6, wherein, The elements in the correlation matrix are used to represent at least one of the following: The correlation distance between the target in the fused target data of the first sensor and the target in the global fused data; The correlation distance between the target in the fused target data of the first sensor and the track in the global fused data; The correlation distance between the track in the fused track data of the first sensor and the track in the global fused data; The correlation distance between the trajectory in the fused trajectory data of the first sensor and the target in the global fused data.

8. The method according to claim 6, wherein, If the association distance between two objects represented by an element in the association matrix is ​​greater than or equal to a threshold, the corresponding element in the association matrix is ​​a first value; if the association distance is less than the threshold, the corresponding element in the association matrix is ​​equal to the association distance divided by the threshold.

9. The method according to claim 6, wherein, The target data and global fusion data from the first sensor are correlated and fused to obtain updated global fusion data, which further includes at least one of the following: Based on the association pairs obtained from the updated association matrix, attribute fusion processing is performed on the fused target data of a single sensor and the global fused target data; Update the global fused track data based on the global fused target data after attribute fusion processing.

10. The method according to claim 9, wherein, The attribute fusion process includes at least one of the following: position and shape fusion; orientation fusion; category fusion; motion fusion; existence fusion; semantic fusion.

11. The method according to claim 10, wherein, The position shape fusion includes: filtering or smoothing position observations from different sensors and performing time-series estimation on low-confidence position observations to obtain the position of the fused target; or, selecting high-precision shape observations from shape observations from different sensors and correcting the high-precision shape observations using other shape observations to obtain the shape of the fused target. The orientation fusion includes: obtaining the orientation of the fusion target based on statistical information of the orientation observations; The category fusion includes: obtaining the classification probability corresponding to the classification vector based on the perception processing result of the original target data, and obtaining the category of the fused target based on the classification probability; The motion fusion includes: estimating the motion state variables of the fusion target based on the motion model and the Kalman filter algorithm; The existence fusion includes: calculating the existence probability of the fusion target; The semantic fusion includes: obtaining dynamic and static occlusion regions based on the original target data, and providing disappearance semantics for the fusion target entering the occlusion region.

12. The method according to any one of claims 1 to 4, further comprising: Target filtering is performed based on the association information, attribute information, and tracking information of the fusion targets; the fusion targets include the fusion targets in the updated global fusion data.

13. The method according to any one of claims 1 to 4, further comprising: Based on the application's data source and updated global fusion data, the application outputs the fusion targets of interest.

14. A sensor data fusion device, comprising: A first processing module is used to perform correlation and fusion processing on the target data of a first sensor and the fused data of the first sensor to obtain fused target data of the first sensor. The first processing module includes: a target correlation submodule, used to compare the target data of the first sensor with fused data of different types according to the correlation method to see if there is a desired target; a target fusion submodule, used to fuse multiple desired targets of the correlation method when multiple desired targets exist in one correlation method to obtain a fused target; and a data processing submodule, used to obtain the fused target data of the first sensor according to the desired target or fusion target corresponding to different correlation methods; wherein, the fused data of the first sensor is obtained by fusing the target data of the first sensor and the target data of at least one second sensor; the first sensor and the second sensor are sensors of different types; The second processing module is used to perform correlation and fusion processing on the fusion target data and global fusion data of the first sensor to obtain updated global fusion data; wherein, the global fusion data includes fusion target data of multiple sensors.

15. The apparatus of claim 14, further comprising: The third processing module is used to fuse the target data from the first sensor and the target data from at least one second sensor to obtain the fused data from the first sensor.

16. The apparatus according to claim 15, wherein the third processing module comprises: The feature fusion submodule is used to extract features from the target data of the first sensor and the target data of multiple second sensors based on the projection relationship, and then fuse them in pairs to obtain multiple fused data of the first sensor.

17. The apparatus according to claim 14, wherein, The association method includes at least one of the following: The target data from the first sensor has no associated target with any of the fused data from the first sensor; The first fused data does not relate to the target data from the first sensor or the second fused data; The second fused data is not associated with the target data of the first sensor or the target data of the first fused data; The target data from the first sensor is related to the first fused data but not to the second fused data; The target data from the first sensor is related to the second fused data, but is not related to the first fused data. The first fused data and the second fused data are related to a target, but the target data of the first sensor is not related to the target data of the first sensor. The target data from the first sensor has a related target with all fused data from the first sensor; The first fused data and the second fused data are fused data obtained by fusing the first sensor with different second sensors.

18. The apparatus according to any one of claims 14 to 17, further comprising: The sensor trajectory module is used to obtain the fused trajectory data of the first sensor based on the fused target data of the first sensor in multiple frames; The global trajectory module is used to obtain global fused trajectory data based on multi-frame global fused target data.

19. The apparatus according to any one of claims 14 to 17, wherein, The second processing module includes: The matrix building submodule is used to obtain the correlation matrix based on the fused target data and global fused data from the first sensor; The matrix update submodule is used to examine and update the correlation matrix using at least one of the tracking information, inter-sensor correlation information, category information, dynamic and static attributes, and velocity information from the fused target data.

20. The apparatus according to claim 19, wherein, The elements in the correlation matrix are used to represent at least one of the following: The correlation distance between the target in the fused target data of the first sensor and the target in the global fused data; The correlation distance between the target in the fused target data of the first sensor and the track in the global fused data; The correlation distance between the track in the fused track data of the first sensor and the track in the global fused data; The correlation distance between the trajectory in the fused trajectory data of the first sensor and the target in the global fused data.

21. The apparatus according to claim 19, wherein, If the association distance between two objects represented by an element in the association matrix is ​​greater than or equal to a threshold, the corresponding element in the association matrix is ​​a first value; if the association distance is less than the threshold, the corresponding element in the association matrix is ​​equal to the association distance divided by the threshold.

22. The apparatus according to any one of claims 19, wherein, The second processing module further includes: The attribute fusion submodule is used to perform attribute fusion processing on the fusion target data of a single sensor and the global fusion target data based on the association pairs obtained from the updated association matrix. The global update submodule is used to update the global fused track data based on the global fused target data after attribute fusion processing.

23. The apparatus according to claim 22, wherein, The attribute fusion process includes at least one of the following: position and shape fusion; orientation fusion; category fusion; motion fusion; existence fusion; semantic fusion.

24. The apparatus according to claim 23, wherein, The position shape fusion includes: filtering or smoothing position observations from different sensors and performing time-series estimation on low-confidence position observations to obtain the position of the fused target; or, selecting high-precision shape observations from shape observations from different sensors and correcting the high-precision shape observations using other shape observations to obtain the shape of the fused target. The orientation fusion includes: obtaining the orientation of the fusion target based on statistical information of the orientation observations; The category fusion includes: obtaining the classification probability corresponding to the classification vector based on the perception processing result of the original target data, and obtaining the category of the fused target based on the classification probability; The motion fusion includes: estimating the motion state variables of the fusion target based on the motion model and the Kalman filter algorithm; The existence fusion includes: calculating the existence probability of the fusion target; The semantic fusion includes: obtaining dynamic and static occlusion regions based on the original target data, and providing disappearance semantics for the fusion target entering the occlusion region.

25. The apparatus according to any one of claims 14 to 17, further comprising: The filtering module is used to filter targets based on their association information, attribute information, and tracking information; the fusion targets include the fusion targets in the updated global fusion data.

26. The apparatus according to any one of claims 14 to 17, further comprising: The output module is used to output the fusion target of interest in the application based on the application's data source and updated global fusion data.

27. An electronic device comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-13.

28. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-13.

29. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1-13.

Images