Sensor Data Fusion Method, Device, and Computer-Readable Storage Medium

Abstract

The present invention discloses a sensor data fusion method, device, and computer-readable storage medium, relating to the field of sensor data processing technology. The method includes the following steps: determining the environmental target of a vehicle system through a sensor module and collecting environmental target information and vehicle motion information; determining a primary target at the current moment and obtaining primary target information at the current moment based on the vehicle motion information and environmental target information; determining a primary target fused at the previous moment of an existing track and obtaining primary target information fused at the previous moment; determining whether the primary target at the current moment is related to the primary target fused at the previous moment based on the primary target information at the current moment and the primary target information fused at the previous moment; and if the primary target at the current moment is related to the primary target fused at the previous moment, filtering and calculating the primary target information at the current moment and the primary target information fused at the previous moment using a fusion algorithm to determine the primary target information fused at the current moment.

Description

Technical Field

[0001] The present invention relates to the field of sensor data processing technology, and in particular to a sensor data fusion method, device, and computer-readable storage medium. Background Art

[0002] With the advancement of automotive electronics and intelligent technology, various sensor technologies have been adopted. Cameras and millimeter-wave radar, as key sensors for sensing the surrounding environment, have become recognized as mainstream choices in the industry. In real-world environments, a single sensor typically detects multiple targets. Current mainstream fusion frameworks often fuse all filtered targets and then select the primary target of interest from the fused set. However, in practice, the excessive number of targets sensed by sensors can lead to problems in data fusion algorithms, such as miscorrelation, multiple correlations, high complexity, and poor robustness. Summary of the Invention

[0003] The main purpose of the present invention is to provide a sensor data fusion method, which aims to solve the technical problems that the data fusion algorithm may have misassociation, multiple associations, high complexity and poor robustness due to the excessive number of sensor-perceived targets.

[0004] To achieve the above object, the present invention provides a sensor data fusion method, which is applied to a vehicle system, wherein the vehicle system includes a vehicle and a sensor module, and the sensor data fusion method includes the following steps:

[0005] Determining the environmental target of the vehicle system and collecting environmental target information and vehicle motion information through a sensor module;

[0006] Determining a primary target at a current moment and obtaining primary target information at a current moment based on the vehicle motion information and the environmental target information;

[0007] Determine the main target fused at the previous moment of the existing track and obtain the main target information fused at the previous moment, and determine whether the main target at the current moment is related to the main target fused at the previous moment based on the main target information at the current moment and the main target information fused at the previous moment;

[0008] If the main target at the current moment is related to the main target fused at the previous moment, filtering calculation is performed on the main target information at the current moment and the main target information fused at the previous moment through a fusion algorithm to determine the main target information fused at the current moment.

[0009] Optionally, the step of determining the primary target at the current moment and obtaining the primary target information at the current moment based on the vehicle motion information and the environmental target information includes:

[0010] Determining each environmental target in a preset vehicle coordinate system based on the environmental target information;

[0011] Determining a predicted path of the vehicle based on the vehicle motion information;

[0012] Determining the relative distance and relative movement speed between each of the environmental targets and the ego vehicle based on the predicted path of the ego vehicle, the environmental target information, and the ego vehicle movement information;

[0013] According to the relative distance and the relative movement speed, the first collision target among the environmental targets within the preset range of the predicted path of the vehicle is determined, and the first collision target is used as the main target at the current moment, and the main target information at the current moment is obtained.

[0014] Optionally, the step of determining, based on the relative distance and the relative movement speed, the first collision target among the environmental targets within a preset range of the predicted path of the ego vehicle includes:

[0015] Based on the relative distance, environmental targets located within a first preset distance on both sides of the predicted path of the ego vehicle are regarded as first priority targets, and environmental targets located within a second preset distance outside the first preset distance on both sides of the predicted path of the ego vehicle are regarded as second priority targets, wherein the second preset distance is greater than the first preset distance;

[0016] Determining the longitudinal collision duration and the lateral separation duration of the secondary target according to the relative distance and the relative movement speed, and selecting the secondary target whose longitudinal collision duration is shorter than the lateral separation duration as the second key target;

[0017] The target with the shortest longitudinal collision time between the first key target and the second key target is taken as the first collision target.

[0018] Optionally, the step of obtaining the main target information fused at the previous moment and determining whether the main target at the current moment is related to the main target fused at the previous moment based on the main target information at the current moment and the main target information fused at the previous moment includes:

[0019] Acquire the fused main target information at the last moment, wherein the fused main target information at the last moment includes the historical horizontal distance, historical vertical distance, and historical movement speed of the fused main target at the last moment;

[0020] When the lateral distance difference between the main target at the current moment and the historical lateral distance is less than the first preset lateral threshold, and the longitudinal distance difference between the main target at the current moment and the historical longitudinal distance is less than the first preset longitudinal threshold, and the movement speed difference between the main target at the current moment and the historical movement speed is less than the first preset speed threshold, it is determined that the main target at the current moment is related to the fused main target at the previous moment.

[0021] Optionally, the step of obtaining the fused main target information at the previous moment and determining whether the main target at the current moment is related to the main target at the previous moment based on the main target information at the current moment and the main target information at the previous moment, the method further includes:

[0022] Acquire the main target information fused at the last moment, wherein the main target information fused at the last moment includes historical identification information of the main target fused at the last moment;

[0023] When the current identification information of the main object at the current moment is consistent with the historical identification information, it is determined that the main object at the current moment is related to the main fusion main object at the previous moment.

[0024] Optionally, the step of performing filtering calculation on the current moment main target information and the fused main target information at the previous moment by a fusion algorithm to determine the fused main target information at the current moment includes:

[0025] Obtaining the process noise covariance matrix of the Kalman filter, and calculating the current moment prior error based on the posterior error of the main target fused at the previous moment, the state transfer matrix, and the process noise covariance matrix;

[0026] Obtaining a measurement matrix and a measurement noise covariance matrix of a Kalman filter, and calculating a Kalman gain based on the current moment prior error, the measurement matrix, and the measurement noise covariance matrix;

[0027] The a posteriori state estimate at the current moment is calculated based on the Kalman gain, the measurement matrix, the a priori state estimate at the current moment and the main target at the current moment, and the a posteriori state estimate at the current moment is used as the fusion main target at the current moment.

[0028] Optionally, the sensor module includes a first sensor and a second sensor, and the current main target includes a first current main target of the first sensor and a second current main target of the second sensor;

[0029] If the current main target is related to the previous fused main target, the step of filtering and calculating the current main target information and the previous fused main target information by a fusion algorithm to determine the current fused main target information includes:

[0030] If the first current main target is related to the fused main target at the previous moment, performing filtering calculation on the first current main target and the fused main target at the previous moment based on Kalman filtering to obtain the fused main target at the current moment;

[0031] If the second current main target is related to the fused main target at the previous moment, filtering calculation is performed on the second current main target and the fused main target at the previous moment based on Kalman filtering to obtain a pending fused main target, and a new track is established based on the first current main target; a first longitudinal collision duration between the pending fused main target and the ego vehicle, and a second longitudinal collision duration between the first current main target and the ego vehicle are obtained; if the first longitudinal collision duration is less than the second longitudinal collision duration, the pending fused main target is used as the fused main target at the current moment; if the first longitudinal collision duration is not less than the second longitudinal collision duration, the first current main target is used as the fused main target at the current moment;

[0032] If the first current main target and the second current main target are both related to the main target fused at the previous moment, then according to the acquisition time sequence of the first current main target and the second current main target, the first current main target, the second current main target and the main target fused at the previous moment are filtered and calculated in sequence based on Kalman filtering to obtain the main target fused at the current moment.

[0033] Optionally, the current main target includes a first current main target;

[0034] After the step of determining whether the current main target is related to the previous fused main target based on the current main target information and the previous fused main target information, the method further includes:

[0035] If the first current main target is related to the fused main target at the previous moment, performing filtering calculation on the first current main target and the fused main target at the previous moment based on Kalman filtering to obtain the fused main target at the current moment;

[0036] If the first current main target and the fused main target at the previous moment are unrelated, obtaining a state transfer matrix of a Kalman filter, calculating the fused main target at the previous moment based on the state transfer matrix, obtaining a priori state estimate at the current moment, and establishing a new track based on the first current main target;

[0037] Obtain the third longitudinal collision duration between the current moment prior state estimate and the ego vehicle, and the fourth longitudinal collision duration between the first current main target and the ego vehicle; if the third longitudinal collision duration is less than the fourth longitudinal collision duration, use the current moment prior state as the current moment fusion main target; if the third longitudinal collision duration is not less than the fourth longitudinal collision duration, use the first current main target as the current moment fusion main target.

[0038] Optionally, the current main target includes a second current main target;

[0039] After the step of determining whether the current main target is related to the previous fused main target based on the current main target information and the previous fused main target information, the method further includes:

[0040] If the second current main target is related to the fused main target at the previous moment, performing filtering calculation on the second current main target and the fused main target at the previous moment based on Kalman filtering to obtain the fused main target at the current moment;

[0041] If the second current main target is not related to the main target fused at the previous moment, the state transfer matrix of the Kalman filter is obtained, and the main target fused at the previous moment is calculated based on the state transfer matrix to obtain the prior state estimate at the current moment, and the prior state estimate at the current moment is used as the main target fused at the current moment.

[0042] Optionally, the step of determining the primary target at the current moment and obtaining the primary target information at the current moment based on the vehicle motion information and the environmental target information, the method further includes:

[0043] If the main target at the current moment does not exist, the state transfer matrix of the Kalman filter is obtained, and the main target fused at the previous moment is calculated based on the state transfer matrix to obtain the prior state estimate at the current moment, and the prior state estimate at the current moment is used as the main target fused at the current moment.

[0044] Optionally, the steps of determining the main target fused at the last moment of the existing track and obtaining the information of the main target fused at the last moment, the method further includes:

[0045] If the fused main target at the previous moment does not exist, determining whether the first current main target exists in the main targets at the current moment;

[0046] If there is a first current main target among the main targets at the current moment, the first current main target is used as the fused main target at the current moment, and a new track is established according to the first current main target;

[0047] If the first current main target does not exist in the main targets at the current moment, it is determined that there is no track main target at the current moment.

[0048] Optionally, the sensor data fusion method further includes:

[0049] Obtaining an unassociated time duration during which the existing track is not associated with the primary target at the current moment, and determining whether the unassociated time duration reaches a preset time duration threshold;

[0050] If the unassociated time reaches a preset time threshold, the existing track is cancelled.

[0051] In addition, to achieve the above-mentioned purpose, the present invention also provides a sensor data fusion device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program implements the steps of any of the methods described above when executed by the processor.

[0052] In addition, to achieve the above objectives, the present invention also provides a computer-readable storage medium, on which a sensor data fusion program is stored. When the sensor data fusion program is executed by a processor, the steps of the sensor data fusion method described in any one of the above items are implemented.

[0053] The present invention proposes a sensor data fusion method, which is applied to a vehicle system. The vehicle system includes a vehicle and a sensor module. The sensor data fusion method includes the following steps: determining the environmental target of the vehicle system and collecting environmental target information and vehicle motion information through the sensor module; determining the main target at the current moment and obtaining the main target information at the current moment based on the vehicle motion information and the environmental target information; determining the fused main target at the previous moment of the existing track and obtaining the fused main target information at the previous moment, and determining whether the main target at the current moment is related to the fused main target at the previous moment based on the main target information at the current moment and the fused main target information at the previous moment; if the main target at the current moment is related to the fused main target at the previous moment, filtering and calculating the main target information at the current moment and the fused main target information at the previous moment through a fusion algorithm to determine the fused main target information at the current moment. Before data fusion, the present invention determines the current primary target from the environmental targets of the vehicle system perceived by the sensor module. This significantly reduces the number of targets that need to be fused, thereby improving the accuracy of the final fusion result. It also reduces the complexity of the fusion algorithm and improves its robustness, thereby improving algorithm performance while reducing hardware performance requirements. Furthermore, by fusing the primary target fused at the previous moment with the current primary target, the tracking of historical fused primary targets is achieved, further improving the accuracy of the fusion result. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 A schematic diagram of the device structure of the hardware operating environment involved in the embodiment of the present invention;

[0055] Figure 2 This is a flow chart of a first embodiment of a sensor data fusion method according to the present invention;

[0056] Figure 3 This is a flow chart of a second embodiment of the sensor data fusion method of the present invention;

[0057] Figure 4 A schematic diagram of a scenario of the sensor data fusion method of the present invention

[0058] Figure 5 This is a flow chart of an optional embodiment of the sensor data fusion method of the present invention;

[0059] Figure 6 This is a flow chart of a third embodiment of the sensor data fusion method of the present invention;

[0060] Figure 7 This is a flow chart of a fourth embodiment of the sensor data fusion method of the present invention;

[0061] Figure 8 The figure is a flow chart of another optional embodiment of the sensor data fusion method of the present invention.

[0062] The purpose, features and advantages of the present invention will be further described with reference to the accompanying drawings and in conjunction with the embodiments. DETAILED DESCRIPTION

[0063] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0064] The term "and / or" in this article is merely a description of the association relationship between associated objects, indicating that three relationships may exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone.

[0065] In the description and claims of the embodiments of this application, the terms "first" and "second" are used to distinguish different objects, rather than to describe a specific order of objects. For example, the terms "first target object" and "second target object" are used to distinguish different objects, rather than to describe a specific order of objects.

[0066] In the embodiments of this application, words such as "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of this application should not be interpreted as being preferred or advantageous over other embodiments or designs. Rather, the use of words such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0067] It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0068] With the advancement of automotive electronics and intelligence, various sensor technologies are being applied. In the field of active automotive safety, millimeter-wave radar and cameras exemplify the perfect integration and complementary nature of these two sensor technologies. As key sensors for environmental perception, cameras and millimeter-wave radar have become recognized as mainstream choices in the industry, enjoying enormous market demand and, therefore, a major research and development direction for automotive electronics manufacturers. The combination of cameras and millimeter-wave radar is primarily used in active safety applications such as lane departure warning, traffic sign recognition, blind spot monitoring, automatic emergency braking, adaptive cruise control, and forward collision warning. In the multi-sensor data fusion process, the number of targets sensed by the sensors directly determines the complexity and performance of the required fusion algorithm. A larger number of targets requires more chip processing power and memory, increasing the likelihood of multiple associations and false associations during the fusion process.

[0069] Currently, in the multi-sensor data fusion process, methods such as removing signals with Doppler velocity, clutter threshold segmentation, and Euclidean clustering are often used at the underlying layer to reduce the number of targets sensed by millimeter-wave radars, thereby reducing fusion algorithm complexity, improving performance, and minimizing the possibility of multiple associations and false associations. At the application layer, the industry generally employs filtering of millimeter-wave sensed targets based on radar cross-section or target tracking time. In real-world environments, a single sensor typically detects multiple targets, but we are often only interested in one of them. Current mainstream fusion frameworks often fuse all filtered targets and then select the primary target of interest from the fused set. However, the excessive number of sensor-sensed targets can lead to problems in data fusion algorithms, such as false associations, multiple associations, increased complexity, and poor robustness, resulting in low accuracy in the final fusion results.

[0070] like Figure 1 As shown, Figure 1 This is a schematic diagram of the device structure of the hardware operating environment involved in the embodiment of the present invention.

[0071] Specifically, the sensor data fusion device may be a VCU (Vehicle Control Unit), a PC (Personal Computer), a tablet computer, a portable computer, or a server.

[0072] like Figure 1As shown, the sensor data fusion device may include: a processor 1001, such as a central processing unit (CPU), a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. The communication bus 1002 is used to implement communication between these components. The user interface 1003 may include a display screen and an input unit such as a keyboard. Optionally, the user interface 1003 may also include a standard wired interface or a wireless interface. The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a wireless fidelity (WI-FI) interface). The memory 1005 may be a high-speed random access memory (RAM) or a stable non-volatile memory (NVM), such as a disk storage device. The memory 1005 may also be a storage device independent of the processor 1001.

[0073] Those skilled in the art will understand that Figure 1 The device structure shown in the figure does not constitute a limitation on the sensor data fusion device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0074] like Figure 1 As shown, the memory 1005 as a computer storage medium may include an operating system, a network communication module, a user interface module, and a sensor data fusion application.

[0075] exist Figure 1 In the device shown, the processor 1001 can be used to call the sensor data fusion application stored in the memory 1005 and perform the operations of the sensor data fusion method in the following embodiments.

[0076] Reference Figure 2 , Figure 2 FIG. 4 is a flow chart of the first embodiment of the sensor data fusion method of the present invention.

[0077] A first embodiment of the present invention provides a sensor data fusion method, which is applied to a vehicle system. The vehicle system includes a vehicle and a sensor module. The sensor data fusion method includes the following steps:

[0078] Step S100, determining the environmental target of the vehicle system through a sensor module and collecting environmental target information and vehicle motion information;

[0079] It can be understood that the sensor data fusion method of this embodiment is applied to a vehicle system, which includes a vehicle and a sensor module. The sensor module may include two or more sensors, such as millimeter-wave radar, lidar, camera, etc.

[0080] In this embodiment, by collecting information about the surrounding environment of the vehicle system through a sensor module, the environmental target of the vehicle system can be determined and environmental target information, as well as vehicle motion information, can be collected. The environmental target information can include motion information such as the position and speed of the environmental target, as well as identification information such as the target ID. It can also include other information such as the target type (e.g., large vehicle, medium-sized vehicle, small vehicle, etc.) and the time of collection. The vehicle motion information can include information such as the vehicle's position, speed, historical driving trajectory, steering wheel angle, tire angle, and lane markings of the lane it is in.

[0081] Step S200, determining a primary target at a current moment and obtaining primary target information at a current moment based on the vehicle motion information and the environmental target information;

[0082] After obtaining the self-vehicle motion information and the environmental target information, the environmental target of the vehicle system determined by the sensor module and the corresponding environmental target information can be uniformly converted to the preset self-vehicle coordinate system, thereby obtaining the environmental target in the preset self-vehicle coordinate system. Then, according to the preset selection rules, the main target at the current moment is determined from the environmental targets, and the main target information at the current moment corresponding to the main target at the current moment is determined. Among them, the main target information at the current moment may include the current motion information of the main target at the current moment, such as the lateral distance, longitudinal distance, and movement speed, as well as the current identification information such as the current target ID. The preset selection rules are set according to the target of interest in the actual scenario. For example, in the actual application scenario of assisted driving, the forward ADAS (Advanced Driving Assistance System) function is only interested in the nearest path target.

[0083] In this embodiment, by determining the current primary target and obtaining its information based on the vehicle's motion information and the environmental target information, the number of targets requiring fusion processing is significantly reduced. This, in turn, reduces the likelihood of multiple associations and false associations during the subsequent data fusion process, thereby improving the accuracy of the final fusion result. Furthermore, this reduces the complexity of the fusion algorithm and improves its robustness, thereby enhancing algorithm performance while reducing hardware requirements.

[0084] Step S300: determining the main target fused at the previous moment of the existing track and obtaining the main target information fused at the previous moment; and determining whether the main target at the current moment is related to the main target fused at the previous moment based on the main target information at the current moment and the main target information fused at the previous moment;

[0085] A track is a trajectory formed by a set of measurements of the same target over a period of time. The existing track is the currently saved track, which may include tracks corresponding to one or more targets, and the main target fused at the previous moment is the main target obtained after the fusion of the existing track at the previous moment. In this embodiment, the main target fused at the previous moment of the existing track is determined, and the main target information fused at the previous moment corresponding to the main target fused at the previous moment is obtained, wherein the main target information fused at the previous moment includes historical motion information such as the historical lateral distance, historical longitudinal distance, and historical motion speed of the main target fused at the previous moment, as well as historical identification information such as the historical target ID. Then, based on the main target information at the current moment and the main target information fused at the previous moment, it is determined whether the main target at the current moment is related to the main target fused at the previous moment. For example, the historical motion information such as the historical lateral distance, historical longitudinal distance, and historical motion speed of the main target fused at the previous moment can be matched with the current motion information such as the current lateral distance, current longitudinal distance, and current motion speed of the main target at the current moment. When the historical motion information matches the current motion information, it means that the main target fused at the previous moment and the main target at the current moment are the same target, and it can be determined that the main target at the current moment is related to the main target fused at the previous moment. When the historical motion information does not match the current motion information, it means that the main target fused at the previous moment and the main target at the current moment are not the same target, and it can be determined that the main target at the current moment and the main target fused at the previous moment are not related. In addition, the historical identification information such as the historical target ID of the main target fused at the previous moment can be compared with the current identification information such as the current target ID of the main target at the current moment. When the historical identification information is consistent with the current identification information, it can be determined that the main target at the current moment and the main target fused at the previous moment are related.

[0086] Step S400: If the current main target is related to the fused main target at the previous moment, filtering calculation is performed on the current main target information and the fused main target information at the previous moment through a fusion algorithm to determine the fused main target information at the current moment.

[0087] In traditional sensor data fusion methods, only the measurement results of the current sensor are generally used for fusion, while historical test results are discarded, which is not conducive to improving the accuracy of the measurement results. Therefore, in order to further improve the accuracy of the measurement results. In this embodiment, if the current main target and the fused main target at the previous moment are related, indicating that the current main target and the fused main target at the previous moment are the same target, a fusion algorithm can be used to filter the current main target information and the fused main target information at the previous moment to determine the fused main target information at the current moment. The fusion algorithm can be a Kalman filter algorithm.

[0088] For example, the current main target includes a first current main target and a second current main target, and the fusion algorithm is a Kalman filter algorithm. When the first current main target and / or the second current main target are related targets related to the fused main target at the previous moment, a filtering calculation is performed on the current main target and the fused main target at the previous moment based on the Kalman filter, and the fused main target at the previous moment and the current main target are fused to obtain the fused main target at the current moment. When the first current main target or the second current main target is a related target related to the fused main target at the previous moment, a filtering calculation is performed on the first current main target or the second current main target and the fused main target at the previous moment based on the Kalman filter to obtain the fused main target at the current moment. It is understandable that the data acquisition cycles of different sensors are often inconsistent. For example, the data acquisition cycle of a camera is 30ms, and the data acquisition cycle of a millimeter-wave radar is 50ms, which means that the camera collects data every 30 milliseconds and the millimeter-wave radar collects data every 50 milliseconds. Therefore, if the first current main target and the second current main target are both related targets related to the fused main target at the previous moment, the order of the acquisition times of the main targets at the current moment can be determined based on the acquisition times of the first current main target and the second current main target, and then, based on the Kalman filter, the fused main target at the previous moment and the main target at the current moment are filtered and calculated in sequence according to the acquisition time order to obtain the fused main target at the current moment. For example, if the acquisition time of the first current main target is earlier than the acquisition time of the second current main target, the first current main target and the fused main target at the previous moment are filtered and calculated based on the Kalman filter to obtain the first fused main target, and then the second current main target and the first fused main target are filtered and calculated based on the Kalman filter to obtain the fused main target at the current moment. In the data fusion process of this embodiment, the pipelined (i.e., the fused main target at the previous moment is fused with the related main target at the current moment) Kalman filter data fusion method utilizes the fusion of the fused main target at the previous moment with the related targets at the current moment to achieve tracking of the historical fused main target, thereby improving the accuracy of the fusion result.

[0089] Furthermore, in step S400, if the current main target is related to the fused main target at the previous moment, filtering and calculating the current main target information and the fused main target information at the previous moment by a fusion algorithm to determine the fused main target information at the current moment includes:

[0090] Step A10: If the first current main target is related to the fused main target at the previous moment, filtering calculation is performed on the first current main target and the fused main target at the previous moment based on Kalman filtering to obtain the fused main target at the current moment;

[0091] Step A20: If the second current main target is related to the fused main target at the previous moment, filtering calculation is performed on the second current main target and the fused main target at the previous moment based on Kalman filtering to obtain a pending fused main target, and a new track is established based on the first current main target; a first longitudinal collision duration between the pending fused main target and the ego vehicle, and a second longitudinal collision duration between the first current main target and the ego vehicle are obtained; if the first longitudinal collision duration is less than the second longitudinal collision duration, the pending fused main target is used as the fused main target at the current moment; if the first longitudinal collision duration is not less than the second longitudinal collision duration, the first current main target is used as the fused main target at the current moment;

[0092] Step A30: If the first current main target and the second current main target are both related to the main target fused at the previous moment, then according to the acquisition time sequence of the first current main target and the second current main target, the first current main target, the second current main target and the main target fused at the previous moment are filtered and calculated in turn based on Kalman filtering to obtain the main target fused at the current moment.

[0093] Specifically, the sensor module includes a first sensor and a second sensor, and the current primary target includes a first current primary target of the first sensor and a second current primary target of the second sensor. It is understandable that the false detection rates (i.e., misidentifying a target when there is no target) of the first sensor and the second sensor may differ.

[0094] In the case where the main target at the current moment includes a first current main target and a second current main target, if the first current main target is related to the main target fused at the previous moment, filtering calculation is performed on the first current main target and the main target fused at the previous moment based on Kalman filtering to obtain the main target fused at the current moment.

[0095] For example, let's take the first sensor as a camera and the second sensor as a millimeter-wave radar. In actual use, the target information collected by the camera is usually more accurate, while the millimeter-wave radar not only receives the reflected signal from the target, but also receives the reflected signal from the environment and unnecessary targets, resulting in the millimeter-wave radar signal often being accompanied by a large amount of noise. Therefore, the millimeter-wave radar has a higher false detection rate, that is, the false detection rate of the first sensor is lower than the false detection rate of the second sensor. Therefore, if only the second current main target is related to the fused main target at the previous moment, the second current main target and the fused main target at the previous moment are filtered and calculated based on the Kalman filter to obtain the pending fused main target. When the first current main target and the fused main target at the previous moment are not related, it means that the first current main target is another target detected by the camera that is different from the fused main target at the previous moment, so the first current main target can be used as the starting point to establish a new track. Then, the longitudinal collision duration between the pending fusion main target and the ego vehicle, as well as the longitudinal collision duration between the first current main target and the ego vehicle, is obtained. The target with the shortest longitudinal collision duration with the ego vehicle between the pending fusion main target and the first current main target is then used as the fusion main target at the current moment. That is, when the longitudinal collision duration of the pending fusion main target is less than that of the first current main target, the pending fusion main target is used as the fusion main target at the current moment; when the longitudinal collision duration of the pending fusion main target is not less than that of the first current main target, the first current main target is used as the fusion main target at the current moment. This can further improve the accuracy of the fusion result on the one hand, and on the other hand, establish a new track based on the first current main target, thereby enabling tracking of different target tracks.

[0096] If the first current main target and the second current main target are both related to the main target fused at the previous moment, the acquisition moments of the first current main target and the second current main target can be obtained, thereby determining the acquisition moment order of the first current main target and the second current main target, and then performing filtering calculations on the first current main target, the second current main target and the main target fused at the previous moment based on the Kalman filter according to the acquisition moment order to obtain the fused main target at the current moment.

[0097] During the data fusion process, in this embodiment, if the second current main target is correlated with the previously fused main target, the longitudinal collision duration between the pending fused main target and the ego vehicle, obtained by filtering and fusion of the second current main target and the previously fused main target, and the longitudinal collision duration between the first current main target and the ego vehicle are compared. The target with the shortest longitudinal collision duration with the ego vehicle between the pending fused main target and the first current main target is then selected as the current fused main target. This avoids errors caused by the high false detection rate of millimeter-wave radar, further improving the accuracy of the fusion results. Furthermore, by establishing a new track based on the first current main target, it is also possible to track the tracks of different targets.

[0098] In a first embodiment of the present invention, the sensor data fusion method is applied to a vehicle system, which includes a vehicle and a sensor module. The sensor data fusion method includes the following steps: determining the environmental target of the vehicle system and collecting environmental target information and vehicle motion information through the sensor module; determining the main target at the current moment and obtaining the main target information at the current moment based on the vehicle motion information and the environmental target information; determining the fused main target at the previous moment of the existing track and obtaining the fused main target information at the previous moment, and determining whether the main target at the current moment is related to the fused main target at the previous moment based on the main target information at the current moment and the fused main target information at the previous moment; if the main target at the current moment is related to the fused main target at the previous moment, filtering and calculating the main target information at the current moment and the fused main target information at the previous moment through a fusion algorithm to determine the fused main target information at the current moment. Before data fusion, the present invention determines the current primary target from the environmental targets of the vehicle system perceived by the sensor module. This significantly reduces the number of targets that need to be fused, thereby improving the accuracy of the final fusion result. It also reduces the complexity of the fusion algorithm and improves its robustness, thereby improving algorithm performance while reducing hardware performance requirements. Furthermore, by fusing the primary target fused at the previous moment with the current primary target, the tracking of historical fused primary targets is achieved, further improving the accuracy of the fusion result.

[0099] Further, refer to Figure 3 The second embodiment of the present invention provides a sensor data fusion method based on the above Figure 2 In the illustrated embodiment, step S200 determines the primary target at the current moment and obtains the primary target information at the current moment based on the vehicle motion information and the environmental target information, including:

[0100] Step S210, determining each environmental target in a preset vehicle coordinate system according to the environmental target information;

[0101] Step S220, determining a predicted path of the vehicle based on the vehicle motion information;

[0102] Step S230, determining the relative distance and relative movement speed between each environmental target and the ego vehicle based on the predicted path of the ego vehicle, the environmental target information, and the ego vehicle movement information;

[0103] In step S240 , based on the relative distance and the relative speed, the first collision target within the preset range of the predicted path of the vehicle is determined among the environmental targets, the first collision target is used as the primary target at the current moment, and the primary target information at the current moment is obtained.

[0104] In real-world environments, a single sensor typically detects multiple targets. For example, millimeter-wave radar signals are often accompanied by significant noise. Radar receives reflections not only from the target but also from the surrounding environment and unwanted objects, resulting in a high false detection rate. Current millimeter-wave radar filtering methods are mostly based on theoretical principles or algorithms, without integrating them into real-world application scenarios. For example, in the actual application of assisted driving, forward-facing ADAS (Advanced Driving Assistance System) functions are only interested in the nearest target.

[0105] In this embodiment, the environmental target information collected by the sensor module can be uniformly converted to the preset vehicle coordinate system based on the conversion relationship between the preset vehicle coordinate system and the coordinate system corresponding to the sensor module, thereby obtaining various environmental targets in the preset vehicle coordinate system.

[0106] The vehicle motion information may include information such as the vehicle's position, speed, historical driving trajectory, steering wheel angle, tire angle, lane markings of the lane it is in, and the like. The predicted path of the vehicle can be determined based on the historical driving trajectory, speed, and steering wheel angle in the vehicle motion information. In addition, the predicted path of the vehicle can also be determined based on the historical driving trajectory, speed, and lane markings of the lane it is in. Of course, considering the need to change lanes and overtake during actual use of the vehicle, the priority of the predicted path of the vehicle obtained based on the historical driving trajectory, speed, and steering wheel angle in the vehicle motion information can be set to the highest to ensure the accuracy of the predicted path of the vehicle.

[0107] After obtaining the predicted path of the ego vehicle, the environmental target information, and the ego vehicle motion information, the first collision environmental target among the environmental targets within a preset range on both sides of the predicted path of the ego vehicle (such as a distance of half the width of the vehicle on both sides) can be determined based on the predicted path of the ego vehicle, the relative distance and relative motion speed between each environmental target and the ego vehicle in the environmental target information. Specifically, the relative distance and relative motion speed between each environmental target and the ego vehicle can be determined based on the predicted path of the ego vehicle, the environmental target information, and the ego vehicle motion information; then, based on the relative distance and the relative motion speed, the first collision target among the environmental targets within the preset range of the predicted path of the ego vehicle can be determined, and the first collision target can be used as the main target at the current moment, and the main target information at the current moment can be obtained. It can be understood that the main target at the current moment can include the main targets at the current moment corresponding to different sensors in the sensor module.

[0108] In this embodiment, environmental targets are identified in a preset ego-vehicle coordinate system based on the environmental target information; a predicted ego-vehicle path is determined based on the ego-vehicle motion information; the relative distance and relative velocity between each environmental target and the ego-vehicle are determined based on the predicted ego-vehicle path, the environmental target information, and the ego-vehicle motion information; and the first collision target within the preset range of the ego-vehicle predicted path is determined based on the relative distance and relative velocity. This first collision target is then used as the primary target at the current moment, and the primary target information at the current moment is obtained. Thus, the target that first collides with the ego-vehicle (i.e., the first collision target) among the environmental targets corresponding to different sensors in the sensor module is determined as the primary target at the current moment. This solves the problems of multiple associations and a high probability of misassociation in the subsequent fusion process caused by an excessive number of perceived targets, as well as high algorithm complexity and poor robustness. It also reduces hardware performance requirements and significantly improves the robustness and real-time performance of the fusion algorithm in the subsequent fusion process.

[0109] Furthermore, in step S240, determining the first collision target among the environmental targets within a preset range of the predicted path of the vehicle according to the relative distance and the relative movement speed includes:

[0110] Step S241, based on the relative distances, identifying environmental targets within a first preset distance on both sides of the predicted path of the ego vehicle as first priority targets, and identifying environmental targets within a second preset distance outside the first preset distance on both sides of the predicted path of the ego vehicle as secondary priority targets, wherein the second preset distance is greater than the first preset distance;

[0111] Step S242, determining the longitudinal collision duration and lateral separation duration of the secondary target based on the relative distance and the relative movement speed, and selecting the secondary target whose longitudinal collision duration is shorter than the lateral separation duration as the second primary target;

[0112] Step S243: The target with the shortest longitudinal collision duration between the first key target and the second key target is selected as the first collision target.

[0113] Specifically, the relative distance between each of the environmental targets and the vehicle may include a lateral distance and a longitudinal distance of the environmental target, and the relative movement speed may include a lateral movement speed and a longitudinal movement speed of the environmental target.

[0114] Reference Figure 4 , Figure 4 A schematic diagram of a scenario of the sensor data fusion method of the present invention. Figure 4 In the equation, C1 is the ego vehicle, L0 is the predicted path of the ego vehicle, L1 is the boundary of the first preset distance, L2 is the boundary of the second preset distance, L3 is the lane line of the ego vehicle, T1 is the first important target, and T2 is the second important target. It can be determined whether each environmental target is within the first preset distance on both sides of the predicted path of the ego vehicle based on the lateral distance and longitudinal distance between each environmental target and the ego vehicle, that is, Figure 4 The dark gray area within the dividing line L1 of the first preset distance on both sides of the predicted path L0 of the self-vehicle. The first preset distance can be a distance value less than half of the width of the self-vehicle, such as 0.2 times, 0.25 times, 0.3 times, etc. of the width of the self-vehicle. If the environmental target is within the first preset distance on both sides of the predicted path of the self-vehicle, the environmental target can be used as the first key target. It can be understood that the first key target is an environmental target with a very high probability of colliding with the self-vehicle. It is also possible to determine the environmental targets within the second preset distance outside the first preset distance on both sides of the predicted path of the self-vehicle as secondary targets based on the relative distance between each environmental target and the self-vehicle, wherein the second preset distance is greater than the first preset distance, such as 0.5 times, 0.55 times, 0.6 times, etc. of the width of the self-vehicle. The range within the second preset distance outside the first preset distance on both sides of the predicted path of the self-vehicle, that is, Figure 4The light gray area between the first preset distance dividing line L1 and the second preset distance dividing line L2 on either side of the predicted path L0 of the ego vehicle is shown in FIG. If an environmental object is located outside the first preset distance and within the second preset distance on either side of the predicted path of the ego vehicle, the environmental object may be considered a secondary target. It is understood that a secondary target is an environmental object with a certain probability of colliding with the ego vehicle. Furthermore, the longitudinal collision time and lateral separation time of the secondary target can be obtained based on the relative distance and relative speed between the secondary target and the ego vehicle. Specifically, the longitudinal collision time can be obtained by taking the quotient of the longitudinal distance and the longitudinal speed in the relative distance between the secondary target and the ego vehicle, where the longitudinal collision time is the time required for the secondary target to collide with the ego vehicle in the longitudinal direction. The lateral separation time can be obtained by taking the quotient of the lateral distance and the lateral speed in the relative distance between the secondary target and the ego vehicle, where the lateral separation time is the time required for the secondary target to laterally separate from the second preset distance on either side of the predicted path of the ego vehicle. It can be understood that if the longitudinal collision duration of a secondary target is shorter than the lateral separation duration, it means that the secondary target will collide with the ego vehicle before departing from the second preset distance on both sides of the predicted path of the ego vehicle. In other words, there is a very high probability that the secondary target will collide with the ego vehicle. Therefore, the secondary target with a longitudinal collision duration shorter than the lateral separation duration can be used as the second key target. Then, the longitudinal collision durations of all the above-mentioned first key targets are obtained, and the target with the smallest longitudinal collision duration among the first and second key targets is selected as the first collision target.

[0115] In this embodiment, based on the relative distance and relative movement speed between each of the environmental targets and the ego vehicle, environmental targets with a very high probability of colliding with the ego vehicle are quickly screened out from the environmental targets as the first key target and the second key target, and then the target with the shortest longitudinal collision time (i.e., the earliest collision with the ego vehicle) among the first key target and the second key target is selected as the first collision target.

[0116] In addition, refer to Figure 5 , Figure 5 This is a flow chart of an optional embodiment of the sensor data fusion method of the present invention.

[0117] Specifically, Figure 5The targets in the equation are environmental targets. The number of targets is the number of targets that have been judged. If the number of targets is less than the total number of targets, the judgment process has not yet been completed for all targets. The width of the ego vehicle is the width of the ego vehicle. Xtemp = target lateral distance - ego vehicle width. The resulting Xtemp is the lateral distance between the target and the ego vehicle. When Xtemp is less than a first threshold (e.g., 0 cm, 10 cm, 20 cm, etc.), it indicates that the target is likely to collide with the ego vehicle.

[0118] Further determine whether there is a preferred target. If there is no preferred target, it can be further determined whether Xtemp is less than a second threshold value (such as -10cm, -20cm, -30cm, etc.), wherein the second threshold value is less than the first threshold value. When Xtemp is less than the second threshold value, it means that there is a very high probability that the target will collide with the ego vehicle, and the target can be used as the preferred target. When Xtemp is not less than the second threshold value, it can be further determined by the lateral movement direction of the target whether the target has a trend of approaching the predicted path of the ego vehicle. If there is a trend of approaching, the longitudinal collision duration and lateral separation duration of the target are calculated to determine whether there is a very high probability of collision between the target and the ego vehicle. If the longitudinal collision duration is less than the lateral separation duration, the target is used as the preferred target.

[0119] If a preferred target exists, determine whether the longitudinal distance of the target is less than the longitudinal distance of the preferred target. If it is less than the longitudinal distance of the preferred target, further determine whether the Xtemp of the target is less than a second threshold. If the Xtemp of the target is less than the second threshold, it means that the target will collide with the ego vehicle earlier than the preferred target, and then update the preferred target and use the target as the new preferred target. If the Xtemp of the target is not less than the second threshold, it can be further determined whether the target has a trend of approaching the predicted path of the ego vehicle. If there is a trend of approaching, calculate the longitudinal collision time and lateral separation time of the target, and determine whether the longitudinal collision time of the target is less than the lateral separation time. If the longitudinal collision time of the target is less than the lateral separation time, further determine whether the longitudinal collision time of the target is less than the longitudinal collision time of the preferred target. If the longitudinal collision time of the target is less than the longitudinal collision time of the preferred target, it means that the target will collide with the ego vehicle earlier than the preferred target, and then update the preferred target.

[0120] After the judgment of each target is completed, the target count is increased by 1 until the number of targets is equal to the total number of targets, indicating that the judgment of all targets has been completed. Then the preferred target at this time can be output and the preferred target can be used as the first collision target.

[0121] Further, refer to Figure 6 The third embodiment of the present invention provides a sensor data fusion method based on the above Figure 2 In the illustrated embodiment, in step S300, obtaining the fused main target information at the previous moment, and determining whether the main target at the current moment is related to the main target at the previous moment based on the main target information at the current moment and the main target information at the previous moment include:

[0122] Step S310: Acquire the fused main target information at the last moment, wherein the fused main target information at the last moment includes the historical horizontal distance, historical vertical distance, and historical movement speed of the fused main target at the last moment;

[0123] Step S311, when the lateral distance difference between the main target at the current moment and the historical lateral distance is less than the first preset lateral threshold, and the longitudinal distance difference between the main target at the current moment and the historical longitudinal distance is less than the first preset longitudinal threshold, and the movement speed difference between the main target at the current moment and the historical movement speed is less than the first preset speed threshold, it is determined that the main target at the current moment is related to the fused main target at the previous moment.

[0124] After determining the main target at the current moment and obtaining the main target information at the current moment, the main target fused at the previous moment of the existing track can be determined and the main target fused at the previous moment information can be obtained. Then, the main target information at the current moment and the main target fused at the previous moment information are matched to determine whether the main target at the current moment is related to the main target fused at the previous moment (that is, whether the main target at the current moment and the main target fused at the previous moment are the same target).

[0125] The information of the main target fused at the previous moment in the existing track includes the historical lateral distance, historical longitudinal distance and historical movement speed of the main target fused at the previous moment. When the lateral distance difference between the lateral distance of the main target at the current moment and the historical lateral distance is less than a first preset lateral threshold (such as 2cm, 3cm, 5cm, etc.), and the longitudinal distance difference between the longitudinal distance of the main target at the current moment and the historical longitudinal distance is less than a first preset longitudinal threshold (such as 5cm, 8cm, 10cm, etc.), it means that the position of the main target at the current moment is close to that of the main target fused at the previous moment. When the movement speed difference between the main target at the current moment and the historical movement speed is less than a first preset speed threshold (such as 0.2m / s, 0.3m / s, 0.5m / s, etc.), it means that the movement speed of the main target at the current moment is similar to that of the main target fused at the previous moment, and it can be determined that the main target at the current moment is related to the main target fused at the previous moment, that is, the main target at the current moment and the main target fused at the previous moment are the same target. When the lateral distance difference between the main target at the current moment and the historical lateral distance is not less than the first preset lateral threshold, and / or the longitudinal distance difference between the main target at the current moment and the historical longitudinal distance is not less than the first preset longitudinal threshold, and / or the movement speed difference between the main target at the current moment and the historical movement speed is not less than the first preset speed threshold, it is determined that the main target at the current moment and the main target fused at the previous moment are not related, that is, the main target at the current moment and the main target fused at the previous moment are not the same target.

[0126] It is understandable that, since the measurement accuracy of different sensors in the sensor module may vary, the first preset lateral threshold, first preset longitudinal threshold and first preset speed threshold used by the main target at the current moment corresponding to different sensors may be different.

[0127] In this embodiment, by comparing the position and movement speed of the main target fused at the previous moment with the position and movement speed of the main target at the current moment, it is determined whether the main target at the current moment is related to the main target fused at the previous moment, that is, whether the main target at the current moment is the same target as the main target fused at the previous moment.

[0128] Furthermore, in step S300, the fused main target information at the previous moment is obtained, and based on the current main target information and the fused main target information at the previous moment, it is determined whether the current main target is related to the fused main target at the previous moment. The method further includes:

[0129] Step S320: Acquire the main target information fused at the last moment, wherein the main target information fused at the last moment includes the historical identification information of the main target fused at the last moment;

[0130] Step S321: When the current identification information of the main target at the current moment is consistent with the historical identification information, it is determined that the main target at the current moment is related to the main fusion main target at the previous moment.

[0131] It is understood that since cameras typically identify environmental targets when recognizing targets, corresponding identification information, such as target IDs, is generated for different environmental targets. This identification information can be in the form of a string or text, such as large vehicle 001, large vehicle 002, etc. Therefore, the identification information of the previous fused main target of the existing track can be obtained. Then, a determination is made as to whether this identification information from the previous moment is consistent with the current identification information of the current camera. If the current identification information of the current main target is consistent with the identification information from the previous moment, it indicates that the current main target and the fused main target from the previous moment are the same target, and the current main target and the fused main target from the previous moment are determined to be related. If the current identification information of the current main target is inconsistent with the identification information from the previous moment, it indicates that the current main target and the fused main target from the previous moment are not the same target, and the current main target and the fused main target from the previous moment are determined to be unrelated.

[0132] This embodiment compares the current identification information of the current primary target with the previous identification information of the fused primary target at the previous moment to determine whether the current primary target is related to the previous fused primary target. This embodiment improves the accuracy of determining whether the current primary target is related to the previous fused primary target.

[0133] Further, refer to Figure 7 The fourth embodiment of the present invention provides a sensor data fusion method based on the above Figure 2 In the embodiment shown, the step S400 of filtering and calculating the primary target information at the current moment and the fused primary target information at the previous moment by a fusion algorithm to determine the fused primary target information at the current moment includes:

[0134] Step S410, obtaining the process noise covariance matrix of the Kalman filter, and calculating the current moment prior error based on the posterior error of the main target fusion at the previous moment, the state transfer matrix and the process noise covariance matrix;

[0135] Step S420: Obtain a measurement matrix and a measurement noise covariance matrix of a Kalman filter, and calculate a Kalman gain based on the current moment prior error, the measurement matrix, and the measurement noise covariance matrix;

[0136] Step S430: Calculate the current moment posterior state estimate based on the Kalman gain, the measurement matrix, the current moment prior state estimate and the current moment main target, and use the current moment posterior state estimate as the current moment fusion main target.

[0137] Specifically, the fusion algorithm is a Kalman filter algorithm. When the main target at the current moment is related to the main target fused at the previous moment, the main target fused at the previous moment can be fused with the main target at the current moment based on Kalman filtering to obtain the fused main target at the current moment.

[0138] Specifically, the a priori error at the current moment is obtained by obtaining the process noise covariance matrix of the Kalman filter and fusing the posterior error of the main target, the state transition matrix, and the process noise covariance matrix at the previous moment. The calculation equation for the a priori error at the current moment is as follows:

[0139]

[0140] in, represents the a priori error at the current moment (i.e., the covariance matrix of the mean square error of the a priori state estimate at the current moment), represents the posterior error of the main target fused at the previous moment, and Q represents the covariance matrix of the process noise.

[0141] Then, the measurement matrix and the measurement noise covariance matrix of the Kalman filter are obtained, and the Kalman gain is obtained according to the current moment prior error, the measurement matrix and the measurement noise covariance matrix. The calculation equation of the Kalman gain is as follows:

[0142]

[0143] Among them, K k represents the Kalman gain, H represents the measurement matrix, and R represents the measurement noise covariance matrix.

[0144] Finally, based on the Kalman gain, the measurement matrix, the current moment prior state estimate, and the current moment primary target, the current moment posterior state estimate is obtained, and the current moment posterior state estimate is used as the current moment fusion primary target. The calculation equation of the current moment posterior state estimate is as follows:

[0145]

[0146] in, Represents the posterior state estimate of the current state, z k Indicates a related target.

[0147] In the data fusion process, this embodiment utilizes the fusion main target at the previous moment to fuse with the related targets at the current moment, thereby achieving the tracking of the historical fusion main target, thereby further improving the accuracy of the fusion result.

[0148] In addition, this embodiment can also calculate the a posteriori error at the current moment for calculation of the a priori error at the next moment. The calculation equation for the a posteriori error at the current moment is as follows:

[0149]

[0150] in, Represents the posterior error of the fused main target at the current moment.

[0151] Further, refer to Figure 8 , Figure 8 The figure is a flow chart of another optional embodiment of the sensor data fusion method of the present invention.

[0152] In this embodiment, the sensor module includes a camera and a millimeter-wave radar. After the vehicle is powered on, the forward-facing millimeter-wave radar and the forward-facing camera respectively sense surrounding targets and environmental information, thereby obtaining environmental target information (i.e., first target information) and millimeter-wave radar information (i.e., second target information). Based on the vehicle coordinate system, the millimeter-wave radar coordinate system, and the camera coordinate system, the environmental target information and millimeter-wave radar target information sensed by the millimeter-wave radar and the camera are uniformly converted to the vehicle coordinate system to determine each environmental target and each millimeter-wave radar target. The corresponding predicted path of the vehicle is then derived in conjunction with the vehicle information, and a primary target is selected for each millimeter-wave radar target and each environmental target. The target that first collides with the vehicle within the preset range of the predicted path is selected as the primary target, thereby determining the primary target for the millimeter-wave radar and the camera. After determining the millimeter-wave radar and camera primary targets, the track management module obtains the previous fused primary target of the existing track from the track management module and performs correlation calculations with the millimeter-wave radar and camera primary targets to determine whether the millimeter-wave radar and / or camera primary targets are correlated with the previous fused primary targets. If the millimeter-wave radar and / or camera primary targets are not correlated with the previous fused primary targets, a one-step prediction of the current moment's prior state estimate is performed based on the Kalman filter and the previous fused primary targets to obtain the current moment's prior state estimate, and the current moment's prior state estimate is used as the current moment's fused primary target.

[0153] If the millimeter-wave radar main target and / or the camera main target are related to the main target fused at the previous moment, the prior error and Kalman gain at the current moment are calculated in sequence based on the Kalman filter, and finally the posterior state estimate at the current moment is calculated, and the posterior state estimate at the current moment is used as the main fusion target at the current moment.

[0154] Furthermore, in another embodiment, the current main target includes a first current main target;

[0155] After determining whether the current main target is related to the previous fused main target based on the current main target information and the previous fused main target information in step S300, the method further includes:

[0156] Step B10: If the first current main target is related to the fused main target at the previous moment, filtering calculation is performed on the first current main target and the fused main target at the previous moment based on Kalman filtering to obtain the fused main target at the current moment;

[0157] Step B20: If the first current primary target and the fused primary target at the previous moment are unrelated, obtaining a state transfer matrix of a Kalman filter, calculating the fused primary target at the previous moment based on the state transfer matrix, obtaining a priori state estimate at the current moment, and establishing a new track based on the first current primary target;

[0158] Step B30, obtaining the third longitudinal collision duration between the current moment prior state estimate and the ego vehicle, and the fourth longitudinal collision duration between the first current main target and the ego vehicle; if the third longitudinal collision duration is less than the fourth longitudinal collision duration, using the current moment prior state as the current moment fusion main target; if the third longitudinal collision duration is not less than the fourth longitudinal collision duration, using the first current main target as the current moment fusion main target.

[0159] In actual application, it is also possible that the first sensor detects the first current main target but the second sensor does not detect the second current main target (that is, only the fused main target at the previous moment and the first current main target exist, while the second current main target does not exist). Taking the false detection rate of the first sensor as an example, when the first current main target is related to the fused main target at the previous moment and the second current main target does not exist, the first current main target and the fused main target at the previous moment can be filtered and calculated based on the Kalman filter to obtain the fused main target at the current moment. However, if the first current main target is not related to the fused main target at the previous moment and the second current main target does not exist, the state transition matrix of the Kalman filter is obtained, and the fused main target at the previous moment is calculated based on the state transition matrix to obtain the prior state estimate at the current moment. At the same time, the first current main target is not related to the fused main target at the previous moment, which means that the first current main target is another target detected by the first sensor that is different from the fused main target at the previous moment. Therefore, a new track can be established using the first current main target as the starting point and a new track can be established based on the first current main target. Obtain the third longitudinal collision duration between the current moment prior state estimate and the ego vehicle, and the fourth longitudinal collision duration between the first current main target and the ego vehicle; if the third longitudinal collision duration is less than the fourth longitudinal collision duration, use the current moment prior state as the current moment fusion main target; if the third longitudinal collision duration is not less than the fourth longitudinal collision duration, use the first current main target as the current moment fusion main target.

[0160] Among them, the one-step prediction equation of the prior state estimate at the current moment is as follows:

[0161]

[0162] in, represents the posterior state estimation at the previous moment (i.e., the position and motion speed information of the main target fused at the previous moment), Represents the a priori state estimate at the current moment, and A represents the state transfer matrix. It can be understood that in the calculation process of the Kalman filter, the a priori state estimate and the a priori state estimate can be represented by the vertical and horizontal coordinates of the corresponding target in the preset ego vehicle coordinate system and the longitudinal and lateral movement speeds. For example, the coordinates of the fused main target in the preset ego vehicle coordinate system at the previous moment are (Xn, Yn), the lateral movement speed is Vxn, and the longitudinal movement speed is Vyn, then is (Xn, Yn, Vxn, Vyn).

[0163] Furthermore, in another embodiment, the current main target includes a second current main target;

[0164] After determining whether the current main target is related to the previous fused main target based on the current main target information and the previous fused main target information in step S300, the method further includes:

[0165] Step C10: If the second current main target is related to the fused main target at the previous moment, performing filtering calculation on the second current main target and the fused main target at the previous moment based on Kalman filtering to obtain the fused main target at the current moment;

[0166] Step C10: If the second current main target and the main target fused at the previous moment are not related, obtain the state transfer matrix of the Kalman filter, and calculate the main target fused at the previous moment based on the state transfer matrix to obtain the prior state estimate at the current moment, and use the prior state estimate at the current moment as the main target fused at the current moment.

[0167] Similarly, in actual application, the second sensor may detect the second current main target but the first sensor may not detect the first current main target (that is, only the fused main target at the previous moment and the second current main target exist, but the first current main target does not exist). When the second current main target is related to the fused main target at the previous moment and the first current main target does not exist, the second current main target and the fused main target at the previous moment are filtered and calculated based on the Kalman filter to obtain the fused main target at the current moment. When the second current main target is not related to the fused main target at the previous moment and the first current main target does not exist, due to the different false detection rates of different sensors, the false detection rate of the first sensor is lower than the false detection rate of the second sensor. Then, the second current main target can be ignored, and the state transfer matrix of the Kalman filter is obtained, and the fused main target at the previous moment is calculated according to the state transfer matrix to obtain the prior state estimate at the current moment, and the prior state estimate at the current moment is used as the fused main target at the current moment.

[0168] Furthermore, in another embodiment, in step S200, determining the primary target at the current moment and obtaining the primary target information at the current moment based on the vehicle motion information and the environmental target information, the method further includes:

[0169] Step D10: If the main target at the current moment does not exist, obtain the state transfer matrix of the Kalman filter, and calculate the fusion main target at the previous moment based on the state transfer matrix to obtain the prior state estimate at the current moment, and use the prior state estimate at the current moment as the fusion main target at the current moment.

[0170] In actual applications, situations may arise where neither the first sensor nor the second sensor detects the corresponding second current primary target nor the first current primary target. If the current primary target does not exist, the state transition matrix of the Kalman filter is obtained, and the fused primary target at the previous moment is calculated based on the state transition matrix to obtain a priori state estimate for the current moment. This priori state estimate is then used as the fused primary target at the current moment.

[0171] Furthermore, in another embodiment, in step S200, the method further includes the steps of determining the previous fused main target of the existing track and obtaining the previous fused main target information:

[0172] Step E10: If the fused main target at the previous moment does not exist, determining whether there is a first current main target among the main targets at the current moment;

[0173] Step E20: If there is a first current main target among the current main targets, the first current main target is used as the current fusion main target, and a new track is established according to the first current main target;

[0174] Step E30: If the first current main target does not exist in the main targets at the current moment, it is determined that there is no track main target at the current moment.

[0175] When the vehicle is just starting or has been driving in an open area for a long time, there may be no existing track for the vehicle, and therefore there is no main target fused at the previous moment. Since the false detection rates of different sensors are different, taking the false detection rate of the first sensor as an example, which is lower than the false detection rate of the second sensor, it is possible to determine whether the first current main target exists. Since the target information collected by the first sensor is usually more accurate, the credibility of the first current main target determined thereby is higher. Therefore, if the first current main target exists, the first current main target can be directly used as the main target fused at the current moment, and a corresponding new track can be established with the first current main target as the starting point. After the new track is established, if there is a second current main target, the first current main target can be used as the main target fused at the previous moment for fusion. If the first current main target does not exist, considering the higher false detection rate of the second sensor, even if the second current main target exists, its credibility is lower, so it can be determined that there is no track main target at the current moment.

[0176] Furthermore, in another embodiment, the sensor data fusion method further includes:

[0177] Step S500, obtaining an unassociated time duration during which the existing track is not associated with a camera primary target or a radar primary target, and determining whether the unassociated time duration reaches a preset time duration threshold;

[0178] Step S510: If the unassociated time reaches a preset time threshold, the existing track is cancelled.

[0179] A track refers to the trajectory formed by a set of measurements of the same target within a period of time. When making a correlation judgment, the duration during which the main fusion target at the previous moment in the existing track is not associated with the camera main target or the radar main target can be timed to obtain the unassociated duration of the existing track. Determine whether the unassociated duration has reached a preset duration threshold (such as 0.2s, 0.3s, 0.5s, etc.). If the unassociated duration has reached the preset duration threshold, it means that the target corresponding to the existing track may have moved away from the vehicle, and the existing track can be revoked. If the unassociated duration does not reach the preset duration threshold, the existing track is retained. Of course, it is understandable that since the sensor data fusion process is usually performed periodically, for example, each data fusion cycle is 50ms. The unassociated period during which the existing track is not associated with the camera main target or the radar main target can be obtained, and it can be determined whether the unassociated period reaches a preset period threshold (for example, 5 periods, 8 periods, 10 periods, etc.). If the unassociated period reaches the preset period threshold, the existing track is revoked; if the unassociated period does not reach the preset period threshold, the existing track can be retained. This embodiment manages the existing track by judging the unassociated duration of the existing track to determine whether the target corresponding to the existing track has moved away from the vehicle, and revoking the existing track whose unassociated duration reaches the preset duration threshold, thereby avoiding invalid existing tracks that reduce the efficiency of correlation determination.

[0180] In addition, an embodiment of the present invention further provides a computer storage medium, on which a computer program is stored. When the computer program is executed by a processor, the operations in the sensor data fusion method provided in the above embodiment are implemented. The specific steps are not described in detail here.

[0181] It should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity / operation / object from another entity / operation / object, and do not necessarily require or imply any actual relationship or order between these entities / operations / objects; the terms "include", "comprise", or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article, or system that includes a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or system. In the absence of further limitations, an element defined by the phrase "includes a ..." does not exclude the presence of other identical elements in the process, method, article, or system that includes the element.

[0182] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple. For relevant parts, please refer to the partial description of the method embodiment. The device embodiment described above is merely illustrative, and the units described as separate components may or may not be physically separated. Some or all of the modules can be selected according to actual needs to achieve the purpose of the present invention. Those skilled in the art can understand and implement it without paying any creative work.

[0183] The serial numbers of the above embodiments of the present invention are for description only and do not represent the advantages or disadvantages of the embodiments.

[0184] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus the necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes a number of instructions for enabling a terminal device (which can be a mobile phone, computer, server, vehicle, or network device, etc.) to execute the methods described in each embodiment of the present invention.

[0185] The above are only preferred embodiments of the present invention and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention description and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A sensor data fusion method, characterized in that: The sensor data fusion method is applied to a vehicle system, wherein the vehicle system includes a vehicle and a sensor module. The sensor data fusion method includes the following steps: Determining the environmental target of the vehicle system and collecting environmental target information and vehicle motion information through a sensor module; Determining a primary target at a current moment and obtaining primary target information at a current moment based on the vehicle motion information and the environmental target information; Determine the main target fused at the previous moment of the existing track and obtain the main target information fused at the previous moment, and determine whether the main target at the current moment is related to the main target fused at the previous moment based on the main target information at the current moment and the main target information fused at the previous moment; If the main target at the current moment is related to the main target fused at the previous moment, filtering calculation is performed on the main target information at the current moment and the main target information fused at the previous moment through a fusion algorithm to determine the main target information fused at the current moment.

2. The sensor data fusion method according to claim 1, wherein: The step of determining the primary target at the current moment and obtaining the primary target information at the current moment based on the vehicle motion information and the environmental target information includes: Determining each environmental target in a preset vehicle coordinate system based on the environmental target information; Determining a predicted path of the vehicle based on the vehicle motion information; Determining the relative distance and relative movement speed between each of the environmental targets and the ego vehicle based on the predicted path of the ego vehicle, the environmental target information, and the ego vehicle movement information; According to the relative distance and the relative movement speed, the first collision target among the environmental targets within the preset range of the predicted path of the vehicle is determined, and the first collision target is used as the main target at the current moment, and the main target information at the current moment is obtained.

3. The sensor data fusion method according to claim 2, wherein: The step of determining the first collision target among the environmental targets within a preset range of the predicted path of the vehicle according to the relative distance and the relative movement speed includes: Based on the relative distance, environmental targets located within a first preset distance on both sides of the predicted path of the ego vehicle are regarded as first priority targets, and environmental targets located within a second preset distance outside the first preset distance on both sides of the predicted path of the ego vehicle are regarded as second priority targets, wherein the second preset distance is greater than the first preset distance; Determining the longitudinal collision duration and the lateral separation duration of the secondary target according to the relative distance and the relative movement speed, and selecting the secondary target whose longitudinal collision duration is shorter than the lateral separation duration as the second key target; The target with the shortest longitudinal collision time between the first key target and the second key target is taken as the first collision target.

4. The sensor data fusion method according to claim 1, wherein: The step of obtaining the main target information fused at the previous moment and determining whether the main target at the current moment is related to the main target fused at the previous moment based on the main target information at the current moment and the main target information fused at the previous moment includes: Acquire the fused main target information at the last moment, wherein the fused main target information at the last moment includes the historical horizontal distance, historical vertical distance, and historical movement speed of the fused main target at the last moment; When the lateral distance difference between the main target at the current moment and the historical lateral distance is less than the first preset lateral threshold, and the longitudinal distance difference between the main target at the current moment and the historical longitudinal distance is less than the first preset longitudinal threshold, and the movement speed difference between the main target at the current moment and the historical movement speed is less than the first preset speed threshold, it is determined that the main target at the current moment is related to the fused main target at the previous moment.

5. The sensor data fusion method according to claim 1, wherein: The method further includes the step of obtaining the fused main target information at the previous moment, and determining whether the main target at the current moment is related to the main target at the previous moment based on the main target information at the current moment and the main target information at the previous moment: Acquire the main target information fused at the last moment, wherein the main target information fused at the last moment includes historical identification information of the main target fused at the last moment; When the current identification information of the main object at the current moment is consistent with the historical identification information, it is determined that the main object at the current moment is related to the main fusion main object at the previous moment.

6. The sensor data fusion method according to claim 1, wherein: The step of filtering and calculating the main target information at the current moment and the fused main target information at the previous moment by a fusion algorithm to determine the fused main target information at the current moment includes: Obtaining the process noise covariance matrix of the Kalman filter, and calculating the current moment prior error based on the posterior error of the main target fused at the previous moment, the state transfer matrix, and the process noise covariance matrix; Obtaining a measurement matrix and a measurement noise covariance matrix of a Kalman filter, and calculating a Kalman gain based on the current moment prior error, the measurement matrix, and the measurement noise covariance matrix; The a posteriori state estimate at the current moment is calculated based on the Kalman gain, the measurement matrix, the a priori state estimate at the current moment and the main target at the current moment, and the a posteriori state estimate at the current moment is used as the fusion main target at the current moment.

7. The sensor data fusion method according to claim 1, wherein: The sensor module includes a first sensor and a second sensor, and the current main target includes a first current main target of the first sensor and a second current main target of the second sensor; If the current main target is related to the previous fused main target, the step of filtering and calculating the current main target information and the previous fused main target information by a fusion algorithm to determine the current fused main target information includes: If the first current main target is related to the fused main target at the previous moment, performing filtering calculation on the first current main target and the fused main target at the previous moment based on Kalman filtering to obtain the fused main target at the current moment; If the second current main target is related to the fused main target at the previous moment, filtering calculation is performed on the second current main target and the fused main target at the previous moment based on Kalman filtering to obtain a pending fused main target, and a new track is established based on the first current main target; a first longitudinal collision duration between the pending fused main target and the ego vehicle, and a second longitudinal collision duration between the first current main target and the ego vehicle are obtained; if the first longitudinal collision duration is less than the second longitudinal collision duration, the pending fused main target is used as the fused main target at the current moment; if the first longitudinal collision duration is not less than the second longitudinal collision duration, the first current main target is used as the fused main target at the current moment; If the first current main target and the second current main target are both related to the main target fused at the previous moment, then according to the acquisition time sequence of the first current main target and the second current main target, the first current main target, the second current main target and the main target fused at the previous moment are filtered and calculated in sequence based on Kalman filtering to obtain the main target fused at the current moment.

8. The sensor data fusion method according to claim 1, wherein: The current main target includes a first current main target; After the step of determining whether the current main target is related to the previous fused main target based on the current main target information and the previous fused main target information, the method further includes: If the first current main target is related to the fused main target at the previous moment, performing filtering calculation on the first current main target and the fused main target at the previous moment based on Kalman filtering to obtain the fused main target at the current moment; If the first current main target and the fused main target at the previous moment are unrelated, obtaining a state transfer matrix of a Kalman filter, calculating the fused main target at the previous moment based on the state transfer matrix, obtaining a priori state estimate at the current moment, and establishing a new track based on the first current main target; Obtain the third longitudinal collision duration between the current moment prior state estimate and the ego vehicle, and the fourth longitudinal collision duration between the first current main target and the ego vehicle; if the third longitudinal collision duration is less than the fourth longitudinal collision duration, use the current moment prior state as the current moment fusion main target; if the third longitudinal collision duration is not less than the fourth longitudinal collision duration, use the first current main target as the current moment fusion main target.

9. The sensor data fusion method according to claim 8, wherein: The current main target includes a second current main target; After the step of determining whether the current main target is related to the previous fused main target based on the current main target information and the previous fused main target information, the method further includes: If the second current main target is related to the fused main target at the previous moment, performing filtering calculation on the second current main target and the fused main target at the previous moment based on Kalman filtering to obtain the fused main target at the current moment; If the second current main target is not related to the main target fused at the previous moment, the state transfer matrix of the Kalman filter is obtained, and the main target fused at the previous moment is calculated based on the state transfer matrix to obtain the prior state estimate at the current moment, and the prior state estimate at the current moment is used as the main target fused at the current moment.

10. The sensor data fusion method according to claim 1, wherein: The step of determining the primary target at the current moment and obtaining the primary target information at the current moment based on the vehicle motion information and the environmental target information, the method further includes: If the main target at the current moment does not exist, the state transfer matrix of the Kalman filter is obtained, and the main target fused at the previous moment is calculated based on the state transfer matrix to obtain the prior state estimate at the current moment, and the prior state estimate at the current moment is used as the main target fused at the current moment.

11. The sensor data fusion method according to claim 1, wherein: The steps of determining the main target fused at the last moment of the existing track and obtaining the information of the main target fused at the last moment, the method further includes: If the fused main target at the previous moment does not exist, determining whether the first current main target exists in the main targets at the current moment; If there is a first current main target among the main targets at the current moment, the first current main target is used as the fused main target at the current moment, and a new track is established according to the first current main target; If the first current main target does not exist in the main targets at the current moment, it is determined that there is no track main target at the current moment.

12. The sensor data fusion method according to any one of claims 1 to 11, characterized in that: The sensor data fusion method further includes: Obtaining an unassociated time duration during which the existing track is not associated with the primary target at the current moment, and determining whether the unassociated time duration reaches a preset time duration threshold; If the unassociated time reaches a preset time threshold, the existing track is cancelled.

13. A sensor data fusion device, characterized in that: The sensor data fusion device includes: a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and when the computer program is executed by the processor, the steps of the method according to any one of claims 1 to 12 are implemented.

14. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a sensor data fusion program, which, when executed by a processor, implements the steps of the sensor data fusion method according to any one of claims 1 to 12.

Images