Method for detecting targets by integrating multiple sensors, automatic driving system, and medium

Abstract

The present invention relates to a method for fusing detection targets of multiple sensors, a vehicle automatic driving system, a computer storage medium, and a vehicle. According to one aspect of the present invention, a method for fusing detection targets of multiple sensors is provided, the method comprising the following steps: at a first moment, acquiring a detection target of a first sensor and marking a state indicator of the acquired detection target of the first sensor as valid; at a second moment, acquiring a detection target of a second sensor or a detection target of the first sensor, and predicting the detection target of the first sensor based on the detection target of the first sensor acquired at the first moment; and marking the state indicator of the detection target of the second sensor or the detection target of the first sensor acquired at the second moment and the predicted detection target of the first sensor using a preset state transition rule.

Description

Technical Field

[0001] The present invention relates to the field of autonomous driving, and more particularly to a method for fusing detection targets of multiple sensors, a vehicle autonomous driving system, a computer storage medium, and a vehicle. Background Art

[0002] Autonomous driving systems use data from a variety of sensors to detect the vehicle's surroundings, objects, and their movement, and can control the vehicle based on this information. Therefore, multi-sensor fusion is a key technology for autonomous driving.

[0003] Currently, Kalman filtering-based target-level fusion of cameras and radars is a common approach. However, with the introduction of more sensors, such as lidar, a more unified framework algorithm is desired for fusion across multiple sensor targets. Fusion across multiple sensor targets requires accurate target association. Without this association, the individual sensors cannot function effectively. Incorrect association, on the other hand, can introduce erroneous information, leading to erroneous target perception.

[0004] However, when a new target appears in the sensor's target detection results, how to associate and track the new target, and how to detect when the target disappears. In addition, when associating targets, how to use the target's historical data and the sensor's own characteristics to more accurately associate the target, and how to detect the appearance or disappearance of the new target, are all urgent problems that need to be solved in existing technologies. Summary of the Invention

[0005] In order to solve or at least alleviate one or more of the above problems, the following technical solutions are provided.

[0006] According to a first aspect of the present invention, a method for fusing detection targets of multiple sensors is provided, the method comprising the following steps: at a first moment, acquiring a detection target of a first sensor and marking a status indicator of the acquired detection target of the first sensor as valid; at a second moment, acquiring a detection target of a second sensor or a detection target of the first sensor, and predicting the detection target of the first sensor based on the detection target of the first sensor acquired at the first moment; and marking the status indicator of the detection target of the second sensor or the detection target of the first sensor acquired at the second moment and the predicted detection target of the first sensor using a preset state transition rule.

[0007] In the method for fusing detection targets of multiple sensors according to an embodiment of the present invention, the state indicator of the detection target includes one or more of the following: valid, fused, tracked, and untracked.

[0008] According to the method for detecting a target by fusing multiple sensors according to one embodiment of the present invention or any of the above embodiments, the multiple sensors include one or more of the following: an image sensor, a lidar sensor, a millimeter-wave radar sensor, and an ultrasonic sensor.

[0009] According to the method for fusing detection targets of multiple sensors according to one embodiment of the present invention or any one of the above embodiments, the state indicator of the detection target of the second sensor acquired at the second moment or the detection target of the first sensor and the predicted detection target of the first sensor is marked using a preset state transition rule, including: at the second moment, acquiring the detection target of the second sensor and predicting the detection target of the first sensor based on the detection target of the first sensor acquired at the first moment; in response to determining that the detection target of the acquired second sensor is associated with the detection target of the predicted first sensor, marking the detection target of the acquired second sensor and the detection target of the predicted first sensor as fused; in response to determining that the detection target of the acquired second sensor is not associated with the detection target of the predicted first sensor, marking the detection target of the acquired second sensor as valid; and marking the detection target of the predicted first sensor outside the detection range of the second sensor as valid.

[0010] According to the method for fusing detection targets of multiple sensors according to one embodiment of the present invention or any one of the above embodiments, the state indicator of the detection target of the second sensor acquired at the second moment or the detection target of the first sensor and the predicted detection target of the first sensor is marked using a preset state transition rule, including: at the second moment, acquiring the detection target of the first sensor and predicting the detection target of the first sensor based on the detection target of the first sensor acquired at the first moment; in response to determining that the detection target of the acquired first sensor is associated with the predicted detection target of the first sensor, marking the acquired detection target of the first sensor and the predicted detection target of the first sensor as tracking; and in response to determining that the detection target of the acquired first sensor is not associated with the predicted detection target of the first sensor, marking the acquired detection target of the first sensor and the predicted detection target of the first sensor as valid.

[0011] According to the method for fusing detection targets of multiple sensors according to one embodiment of the present invention or any one of the above embodiments, the method further includes, at a third moment: acquiring a detection target of a third sensor; predicting a detection target of the second sensor based on the detection target of the second sensor acquired at the second moment; predicting a detection target of the first sensor based on the detection target of the first sensor predicted at the second moment; and marking the state indicators of the detection target of the third sensor acquired at the third moment, the predicted detection target of the second sensor, and the predicted detection target of the first sensor using a preset state transition rule.

[0012] According to the method for fusing detection targets of multiple sensors described in one embodiment of the present invention or any one of the above embodiments, the state indicators of the detection target of the third sensor, the predicted detection target of the second sensor and the predicted detection target of the first sensor acquired at the third moment are marked using a preset state transition rule, including: in response to determining that the detection target of the acquired third sensor is associated with the detection target of the predicted second sensor or the detection target of the predicted first sensor, marking the detection target of the acquired third sensor and the detection target of the predicted second sensor or the detection target of the predicted first sensor as fused; in response to determining that the detection target of the acquired third sensor is not associated with the detection target of the predicted second sensor or the detection target of the predicted first sensor, marking the detection target of the acquired third sensor as valid; and marking the state indicators of the detection target of the predicted second sensor and the detection target of the predicted first sensor outside the detection range of the third sensor as the same as the state indicators at the second moment.

[0013] According to the method for fusing detection targets of multiple sensors according to one embodiment of the present invention or any of the above embodiments, the correlation between the detection targets is determined based on a group consisting of one or more of the following: Mahalanobis distance, Euclidean distance, angular distance and velocity distance.

[0014] According to the method for fusing detection targets of multiple sensors according to one embodiment of the present invention or any of the above embodiments, the method further includes: repeatedly performing the operations at the second moment and the third moment; marking the detection targets that are not marked as tracked or fused after a predetermined number of moments as untracked; and discarding the detection targets marked as untracked.

[0015] According to a second aspect of the present invention, a vehicle automatic driving system is provided, which includes: multiple sensors; a memory; and a processor communicatively coupled to the memory and the multiple sensors, the processor being configured to: at a first moment, acquire a detection target of a first sensor and mark a status indicator of the acquired detection target of the first sensor as valid; at a second moment, acquire a detection target of a second sensor or a detection target of the first sensor, and predict the detection target of the first sensor based on the detection target of the first sensor acquired at the first moment; and mark the status indicator of the detection target of the second sensor or the detection target of the first sensor acquired at the second moment and the predicted detection target of the first sensor using a preset state transition rule.

[0016] According to the vehicle automatic driving system according to an embodiment of the present invention, the status indicator of the detected target includes one or more of the following: valid, fused, tracked, and untracked.

[0017] According to the vehicle automatic driving system according to one embodiment of the present invention or any one of the above embodiments, the multiple sensors include one or more of the following: image sensors, lidar sensors, millimeter wave radar sensors and ultrasonic sensors.

[0018] According to the vehicle automatic driving system of one embodiment or any one of the above embodiments of the present invention, the processor is further configured to mark the state indicator of the detection target of the second sensor or the detection target of the first sensor and the predicted detection target of the first sensor acquired at the second moment using a preset state transition rule in the following manner: at the second moment, the detection target of the second sensor is acquired and the detection target of the first sensor is predicted based on the detection target of the first sensor acquired at the first moment; in response to determining that the acquired detection target of the second sensor is associated with the predicted detection target of the first sensor, the acquired detection target of the second sensor and the predicted detection target of the first sensor are marked as fused; in response to determining that the acquired detection target of the second sensor is not associated with the predicted detection target of the first sensor, the acquired detection target of the second sensor is marked as valid; and the predicted detection target of the first sensor outside the detection range of the second sensor is marked as valid.

[0019] According to the vehicle automatic driving system of one embodiment or any one of the above embodiments of the present invention, the processor is further configured to mark the state indicator of the detection target of the second sensor acquired at the second moment or the detection target of the first sensor and the predicted detection target of the first sensor using a preset state transition rule in the following manner: at the second moment, the detection target of the first sensor is acquired and the detection target of the first sensor is predicted based on the detection target of the first sensor acquired at the first moment; in response to determining that the acquired detection target of the first sensor is associated with the predicted detection target of the first sensor, the acquired detection target of the first sensor and the predicted detection target of the first sensor are marked as tracking; and in response to determining that the acquired detection target of the first sensor is not associated with the predicted detection target of the first sensor, the acquired detection target of the first sensor and the predicted detection target of the first sensor are marked as valid.

[0020] According to the vehicle automatic driving system according to one embodiment of the present invention or any one of the above embodiments, the processor is further configured to: obtain the detection target of the third sensor at a third moment; predict the detection target of the second sensor based on the detection target of the second sensor obtained at the second moment; predict the detection target of the first sensor based on the detection target of the first sensor predicted at the second moment; and mark the state indicators of the detection target of the third sensor, the predicted detection target of the second sensor and the predicted detection target of the first sensor obtained at the third moment using a preset state transition rule.

[0021] According to the vehicle automatic driving system of one embodiment or any one of the above embodiments of the present invention, the processor is further configured to mark the state indicators of the detection target of the third sensor, the predicted detection target of the second sensor and the predicted detection target of the first sensor acquired at the third moment using a preset state transition rule in the following manner: in response to determining that the detection target of the acquired third sensor is associated with the detection target of the predicted second sensor or the detection target of the predicted first sensor, the detection target of the acquired third sensor and the detection target of the predicted second sensor or the detection target of the predicted first sensor are marked as fused; in response to determining that the detection target of the acquired third sensor is not associated with the detection target of the predicted second sensor or the detection target of the predicted first sensor, the detection target of the acquired third sensor is marked as valid; and the state indicators of the predicted detection target of the second sensor and the predicted detection target of the first sensor outside the detection range of the third sensor are marked as the same as the state indicators at the second moment.

[0022] According to the vehicle automatic driving system according to one embodiment of the present invention or any one of the above embodiments, the correlation between the detected targets is determined based on a group consisting of one or more of the following: Mahalanobis distance, Euclidean distance, angular distance and speed distance.

[0023] According to the vehicle automatic driving system of one embodiment or any one of the above embodiments of the present invention, the processor is further configured to: repeatedly perform the operations at the second moment and the third moment; mark the detection target that is not marked as tracked or fused after a predetermined number of moments as untracked; and discard the detection target marked as untracked.

[0024] According to a third aspect of the present invention, a computer storage medium is provided, wherein the computer storage medium comprises instructions, which, when run, execute the steps of the method for fusing detection targets of multiple sensors according to the first aspect of the present invention.

[0025] According to a fourth aspect of the present invention, a vehicle is provided, comprising the vehicle automatic driving system according to the second aspect of the present invention.

[0026] According to one or more embodiments of the present invention, the scheme for fusing detection targets of multiple sensors can achieve fusion between multiple sensor targets based on historical information of sensor detection targets, prior probability distribution and sensor characteristics, and use preset state transition rules to mark the state indicators of detection targets generated by multiple sensors at different times, so as to determine whether the disappearance of the detection target is due to the loss of one or several frames of the sensor or due to the detection target exceeding the detection range of the sensor, thereby improving the target association accuracy and fusion success rate. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] The above and / or other aspects and advantages of the present invention will become clearer and easier to understand through the following description of various aspects in conjunction with the accompanying drawings, in which the same or similar elements are represented by the same reference numerals. In the accompanying drawings:

[0028] Figure 1 A flowchart of a method for fusing target detection by multiple sensors according to one or more embodiments of the present invention is shown.

[0029] Figure 2 A flowchart of a method for fusing target detection by multiple sensors according to one or more embodiments of the present invention is shown.

[0030] Figures 3A-3G A fusion diagram is shown of a method for fusing detection targets of multiple sensors according to one or more embodiments of the present invention.

[0031] Figure 4 A schematic block diagram of an automatic driving system for a vehicle according to one or more embodiments of the present invention is shown. DETAILED DESCRIPTION

[0032] The description of the following specific embodiments is merely exemplary in nature and is not intended to limit the disclosed technology or the application and use of the disclosed technology. In addition, there is no intention to be bound by any express or implied theory presented in the foregoing technical field, background technology or the following specific embodiments.

[0033] In the following detailed description of the embodiments, numerous specific details are set forth to provide a more thorough understanding of the disclosed technology. However, it will be apparent to one of ordinary skill in the art that the disclosed technology can be practiced without these specific details. In other instances, well-known features are not described in detail to avoid unnecessarily complicating the description.

[0034] Terms such as "comprising" and "including" indicate that, in addition to the units and steps directly and explicitly stated in the specification, the technical solution of the present invention does not exclude the presence of other units and steps not directly or explicitly stated. Terms such as "first" and "second" do not indicate the order of units in terms of time, space, size, etc., but are merely used to distinguish between units.

[0035] It should be noted that the "sensor" mentioned in the context of this invention may refer to various types of sensors deployed on a vehicle for detecting targets, including but not limited to image sensors (such as cameras), lidar sensors, millimeter-wave radar sensors, and ultrasonic sensors. The "target" mentioned in the context of this invention may refer to various moving or stationary objects in front of, behind, or to the sides of a vehicle, such as vehicles, pedestrians, and buildings.

[0036] Hereinafter, exemplary embodiments according to the present invention will be described in detail with reference to the accompanying drawings.

[0037] Figure 1 A flowchart of a method for fusing target detection by multiple sensors according to one or more embodiments of the present invention is shown.

[0038] like Figure 1 As shown, in step S110, at a first moment, a detection target of a first sensor is acquired and a state indicator of the acquired detection target of the first sensor is marked as valid. In the context of the present invention, the state indicator of the detection target may include one or more of the following: valid, fused, tracked, and untracked.

[0039] In step S120, at the second moment, the target detected by the second sensor or the target detected by the first sensor is acquired, and the target detected by the first sensor is predicted based on the target detected by the first sensor acquired at the first moment. Optionally, a Kalman filter algorithm can be used to predict the target detected by the first sensor based on the target detected by the first sensor acquired at the first moment. By predicting the target detected by the first sensor at the second moment based on the target detected by the first sensor acquired at the first moment, errors in target fusion caused by asynchrony between the time frames of multiple sensors can be avoided.

[0040] In step S130 , a state indicator of the detection target of the second sensor or the detection target of the first sensor and the predicted detection target of the first sensor acquired at the second moment is marked using a preset state transition rule.

[0041] In one embodiment, in step S120, at a second moment, a detection target of the second sensor is acquired and a detection target of the first sensor is predicted based on the detection target of the first sensor acquired at the first moment. In step 130, marking the state indicator of the detection target at the second moment using a preset state transition rule may include: in response to determining that the detection target acquired by the second sensor is associated with the predicted detection target of the first sensor, marking the detection target acquired by the second sensor and the predicted detection target of the first sensor as fused; in response to determining that the detection target acquired by the second sensor is not associated with the predicted detection target of the first sensor, marking the detection target acquired by the second sensor as valid; and marking the predicted detection target of the first sensor as valid if it is outside the detection range of the second sensor.

[0042] In another embodiment, in step S120, at a second moment, a detection target of the first sensor is acquired and a detection target of the first sensor is predicted based on the detection target of the first sensor acquired at the first moment. In step 130, marking a state indicator of the detection target at the second moment using a preset state transition rule may include: in response to determining that the detection target acquired by the first sensor is associated with the predicted detection target of the first sensor, marking the acquired detection target of the first sensor and the predicted detection target of the first sensor as tracking; and in response to determining that the detection target acquired by the first sensor is not associated with the predicted detection target of the first sensor, marking the acquired detection target of the first sensor and the predicted detection target of the first sensor as valid.

[0043] Optionally, the correlation between the detection targets is determined based on one or more of the following: Mahalanobis distance, Euclidean distance, angular distance, and velocity distance.

[0044] In one embodiment, the Mahalanobis distance may be used to determine the correlation between the detection target acquired by the second sensor at the second moment and the predicted detection target of the first sensor.

[0045] Mahalanobis distance without threshold It can be calculated by the following formula:

[0046] Formula (1)

[0047] in represents the feature space of the detection target of the first sensor, represents the feature space of the detection target of the second sensor, T represents the matrix transpose, represents the covariance matrix of the detected target of the first sensor, and Represents the covariance matrix of the detected target by the second sensor. For example, the feature space of the detected target can be represented as (x, y, Vx, Vy) consisting of the target's planar position in the vehicle coordinate system and its velocity relative to the vehicle.

[0048] Mahalanobis distance with threshold It can be calculated by the following formula:

[0049] Formula (2)

[0050] in represents the feature space of the detection target of the first sensor, represents the feature space of the detection target of the second sensor, T represents the matrix transpose, represents the covariance matrix of the detected target of the first sensor, and represents the covariance matrix of the detected targets of the second sensor.

[0051] It should be noted that, in the context of the present invention, the Mahalanobis distance can be determined by either formula (1) or formula (2).

[0052] It should be noted that the Mahalanobis distance is derived from the posterior probability (i.e., it makes the association distance between identically distributed objects equal to 0). Therefore, when the Euclidean distance is constant, the greater the variance, the greater the likelihood of association; when the variance is constant, the closer the Euclidean distance, the greater the likelihood of association; if the two are identically distributed, the association distance is 0. For example, when the Mahalanobis distance between the detection target of the second sensor acquired at the second moment and the predicted detection target of the first sensor is less than a threshold Mahalanobis distance, it can be determined that the detection target of the second sensor acquired at the second moment is associated with the predicted detection target of the first sensor, and the detection target of the second sensor acquired at the second moment and the predicted detection target of the first sensor are marked as fused; when the Mahalanobis distance between the detection target of the second sensor acquired at the second moment and the predicted detection target of the first sensor is greater than or equal to the threshold Mahalanobis distance, it can be determined that the detection target of the second sensor acquired at the second moment is not associated with the predicted detection target of the first sensor, and the detection target of the second sensor acquired at the second moment is marked as valid.

[0053] In another embodiment, the association between the detection target of the second sensor acquired at the second moment and the predicted detection target of the first sensor can also be determined based on both the Mahalanobis distance and the Euclidean distance. For example, when the Mahalanobis distance and the Euclidean distance between the detection target of the second sensor acquired at the second moment and the predicted detection target of the first sensor are respectively less than a threshold Mahalanobis distance and a threshold Euclidean distance, it can be determined that the detection target of the second sensor acquired at the second moment is associated with the predicted detection target of the first sensor, and the detection target of the second sensor acquired at the second moment and the predicted detection target of the first sensor are marked as fused; when the Mahalanobis distance and the Euclidean distance between the detection target of the second sensor acquired at the second moment and the predicted detection target of the first sensor are respectively greater than or equal to the threshold Mahalanobis distance and the threshold Euclidean distance, it can be determined that the detection target of the second sensor acquired at the second moment is not associated with the predicted detection target of the first sensor, and the detection target of the second sensor acquired at the second moment is marked as valid.

[0054] Alternatively, to improve the accuracy of visual and radar angle measurements, the correlation between the target detected by the second sensor at the second moment and the predicted target detected by the first sensor can be determined based on the Mahalanobis distance, Euclidean distance, and angular distance. Alternatively, the correlation between the target detected by the second sensor at the second moment and the predicted target detected by the first sensor can be determined based on the Mahalanobis distance, Euclidean distance, angular distance, and speed distance. For example, the angular distance and speed distance can be represented by the angular difference and speed difference between the targets detected by different sensors in the vehicle coordinate system.

[0055] Alternatively, the angular distance or the speed distance may be used alone to determine the association between the detection target of the second sensor acquired at the second moment and the predicted detection target of the first sensor. For example, when the angular distance between the detection target of the second sensor acquired at the second moment and the predicted detection target of the first sensor is less than a threshold angular distance, it can be determined that the detection target of the second sensor acquired at the second moment is associated with the predicted detection target of the first sensor, and the detection target of the second sensor acquired at the second moment and the predicted detection target of the first sensor are marked as fused; when the angular distance between the detection target of the second sensor acquired at the second moment and the predicted detection target of the first sensor is greater than or equal to the threshold angular distance, it can be determined that the detection target of the second sensor acquired at the second moment is not associated with the predicted detection target of the first sensor, and the detection target of the second sensor acquired at the second moment is marked as valid. Exemplarily, when the speed distance between the detection target of the second sensor obtained at the second moment and the predicted detection target of the first sensor is less than a threshold speed distance, it can be determined that the detection target of the second sensor obtained at the second moment is associated with the predicted detection target of the first sensor, and the detection target of the second sensor obtained at the second moment and the predicted detection target of the first sensor are marked as fused; when the speed distance between the detection target of the second sensor obtained at the second moment and the predicted detection target of the first sensor is greater than or equal to the threshold speed distance, it can be determined that the detection target of the second sensor obtained at the second moment is not associated with the predicted detection target of the first sensor, and the detection target of the second sensor obtained at the second moment is marked as valid.

[0056] By determining the association between detected targets based on one or more of the following: Mahalanobis distance, Euclidean distance, angular distance, and velocity distance, the target association accuracy and fusion success rate can be improved.

[0057] Figure 2 A flowchart of a method for fusing target detection by multiple sensors according to one or more embodiments of the present invention is shown. Figures 3A-3G The following is a fusion diagram of a method for fusing detection targets of multiple sensors according to one or more embodiments of the present invention. Figure 2 and Figures 3A-3G A method for fusing target detection by multiple sensors according to one or more embodiments of the present invention is further described.

[0058] Figure 3ADetection ranges of sensor A, sensor B, and sensor C are shown. As an example, sensor A may be implemented as a camera, sensor B may be implemented as a millimeter-wave radar, and sensor C may be implemented as a lidar. It is understood that the detection ranges of sensor A, sensor B, and sensor C may partially overlap.

[0059] In the context of the present invention, marking the state indicator of the detection target using a preset state transition rule may include: marking the state indicator of the detection target of the sensor obtained by the first detection as valid; predicting the detection target of the valid sensor from the current moment to the next moment, and keeping the state indicator of the detection target outside the detection range of the new sensor at the next moment valid; predicting the detection target of the valid sensor from the current moment to the next moment, and in response to determining that the predicted detection target of the valid sensor is associated with the detection target detected by the new sensor at the next moment, marking the predicted detection target of the valid sensor and the detection target detected by the new sensor at the next moment as fused; predicting the detection target of the fused sensor from the current moment to the next moment, and in response to determining that the predicted detection target of the fused sensor is not associated with the detection target detected by the new sensor at the next moment, marking the predicted detection target of the fused sensor and the detection target detected by the new sensor at the next moment as valid; predicting the detection target of the fused sensor from the current moment to the next moment, and keeping the state indicator of the detection target outside the detection range of the new sensor at the next moment fused; predicting the fused detection target from the current moment to the next moment The detection target of the sensor is predicted, and in response to determining that the predicted detection target of the fused sensor is associated with the detection target detected by the new sensor at the next moment, the predicted detection target of the fused sensor and the detection target detected by the new sensor at the next moment are marked as tracked; from the current moment to the next moment, the detection target of the tracked sensor is predicted, and the state indicator of the detection target outside the detection range of the new sensor at the next moment is kept tracked; from the current moment to the next moment, the detection target of the tracked sensor is predicted, and in response to determining that the predicted detection target of the tracked sensor is not associated with the detection target detected by the new sensor at the next moment, the predicted detection target of the tracked sensor and the detection target detected by the new sensor at the next moment are marked as valid; from the current moment to the next moment, the detection target of the valid sensor is predicted, and in response to determining that the predicted detection target of the valid sensor is associated with the detection target detected by the new sensor at the next moment, the predicted detection target of the valid sensor and the detection target detected by the new sensor at the next moment are marked as tracked; after maintaining the valid state for several rounds, the detection targets that are not marked as tracked and fused are marked as untracked, and the detection targets marked as untracked are discarded.

[0060] Back to Figure 2In step S210, at the first moment T1, the detection target of sensor A is acquired and the acquired detection target of sensor A is marked as valid, such as Figure 3B As shown by the hollow pentagram in the figure.

[0061] In step S220, at the second time T2, the detection target of sensor B is obtained (eg Figure 3C The detection target of sensor A is predicted based on the detection target of sensor A obtained at the first moment T1, for example, by Kalman filtering (as shown in the hollow triangle in FIG). Figure 3C ), and the state indicators of the detection targets of sensor B and the predicted detection targets of sensor A acquired at the second moment T2 are marked using the preset state transition rule described above. For example, Figure 3D As shown in , when it is determined that the target detected by sensor B at time T2 is associated with the predicted target detected by sensor A, the target detected by sensor B at time T2 and the predicted target detected by sensor A are weighted averaged (e.g., based on confidence) and marked as fused. When it is determined that the target detected by sensor B at time T2 is not associated with the predicted target detected by sensor A, the target detected by sensor B at time T2 is marked as valid. Furthermore, the predicted target detected by sensor A outside the detection range of sensor B is marked as valid. Alternatively, the association between the detected targets can be determined based on one or more of the following: Mahalanobis distance, Euclidean distance, angular distance, and velocity distance.

[0062] In step S230, at the third moment T3, the detection target of sensor C is obtained (eg Figure 3E ), and based on the detection target of sensor B acquired at the second moment T2 and the predicted detection target of sensor A, for example, the detection target of sensor B is predicted by Kalman filtering (as shown in FIG. Figure 3E The hollow triangle in the figure) and the detection target of sensor A (as shown in Figure 3E ), and the state indicators of the detection target of sensor C, the predicted detection target of sensor B, and the predicted detection target of sensor A acquired at the third moment T3 are marked using the preset state transition rule described above. For example, Figure 3FAs shown in , when it is determined that the detection target of sensor C acquired at the third time T3 is associated with the predicted detection target of sensor B or the predicted detection target of sensor A, the detection target of sensor C acquired at the third time T3 is weighted averaged (for example, weighted averaged according to confidence) with the predicted detection target of sensor B or the predicted detection target of sensor A and marked as fused; when it is determined that the detection target of sensor C acquired at the third time T3 is not associated with the predicted detection target of sensor B or the predicted detection target of sensor A, the detection target of sensor C acquired at the third time T3 is marked as valid; and the state indicators of the predicted detection targets of sensor B and sensor A outside the detection range of sensor C are marked as the same as the state indicators at the second time T2 (refer to Figure 3D ). Optionally, the correlation between the detection targets may be determined based on one or more of the following: Mahalanobis distance, Euclidean distance, angular distance, and velocity distance.

[0063] In step S240, at the fourth moment T4, the detection target of sensor A is obtained (eg Figure 3G ), based on the detection target of sensor C acquired at the third time T3, the predicted detection target of sensor A and the predicted detection target of sensor B, for example, the detection target of sensor C is predicted by Kalman filtering (as shown in FIG. Figure 3G As shown by the hollow four-pointed star in the figure), the detection target of sensor A (as shown in Figure 3G The hollow five-pointed star in the figure) and the detection target of sensor B (as shown in Figure 3G ), and the state indicators of the detection target of sensor A, the predicted detection target of sensor C, the predicted detection target of sensor A, and the predicted detection target of sensor B acquired at the fourth moment T4 are marked using the preset state transition rule described above. For example, Figure 3G As shown, the state indicators of the predicted detection targets of sensor B and sensor C outside the detection range of sensor A are marked as the same as the state indicators at the third time point T3. When it is determined that the detection target of sensor A acquired at the fourth time point T4 is associated with the predicted detection target of sensor A marked as valid, the detection target of sensor A acquired at the fourth time point T4 and the predicted detection target of sensor A marked as valid are marked as tracked. When it is determined that the detection target of sensor A acquired at the fourth time point T4 is associated with the predicted detection target marked as fused, the detection target of sensor A acquired at the fourth time point T4 and the predicted detection target marked as fused are marked as fused. Optionally, the association between the detection targets can be determined based on one or more of the following: Mahalanobis distance, Euclidean distance, angular distance, and velocity distance.

[0064] In step S250, at each subsequent moment, for the detection targets acquired by sensor A, sensor B or sensor C, operations similar to those in the above steps S220-S240 are performed, and the detection targets that are not marked as tracked or fused after a predetermined number of moments are marked as untracked, and the detection targets marked as untracked are discarded.

[0065] The method for fusing detection targets of multiple sensors proposed in accordance with one or more embodiments of the present invention can achieve fusion between multiple sensor targets based on historical information of sensor detection targets, prior probability distribution, and sensor characteristics, and utilize preset state transition rules to mark the state indicators of detection targets generated by multiple sensors at different times, so as to determine whether the disappearance of the detection target is due to the loss of one or several frames of the sensor or due to the detection target exceeding the detection range of the sensor, thereby improving the target association accuracy and fusion success rate.

[0066] Figure 4 A schematic block diagram of an automatic driving system for a vehicle according to one or more embodiments of the present invention is shown.

[0067] like Figure 4 As shown in the figure, the vehicle automatic driving system 40 includes multiple sensors 410, a memory 420 (for example, a non-volatile memory such as a flash memory, a ROM, a hard disk drive, a magnetic disk, an optical disk), and a processor 430 communicatively coupled to the memory 420 and the multiple sensors 410, and the processor 430 is configured to implement a method for fusing detection targets of multiple sensors according to one or more embodiments of the present invention.

[0068] In addition, the present invention may also be implemented as a computer storage medium in which a program for causing a computer to execute the method for fusing detection targets of multiple sensors according to one aspect of the present invention is stored.

[0069] Here, as computer storage media, various types of computer storage media can be used, such as disks (for example, magnetic disks, optical disks, etc.), cards (for example, memory cards, optical cards, etc.), semiconductor memories (for example, ROMs, non-volatile memories, etc.), and tapes (for example, magnetic tapes, cassette tapes, etc.).

[0070] In the case of applicable, hardware, software or a combination of hardware and software can be used to realize the various embodiments provided by the present invention. Moreover, in the case of applicable, without departing from the scope of the present invention, the various hardware components and / or software components set forth herein can be combined into composite components comprising software, hardware and / or both. In the case of applicable, without departing from the scope of the present invention, the various hardware components and / or software components set forth herein can be divided into subcomponents comprising software, hardware or both. In addition, in the case of applicable, it is contemplated that software components can be implemented as hardware components, and vice versa.

[0071] Software according to the present invention (such as program code and / or data) can be stored on one or more computer storage media. It is also contemplated that the software identified herein can be implemented using one or more general-purpose or special-purpose computers and / or computer systems, networked and / or otherwise. Where applicable, the order of the various steps described herein can be changed, combined into composite steps, and / or divided into sub-steps to provide the features described herein.

[0072] The embodiments and examples set forth herein are provided to best illustrate embodiments according to the present invention and its specific applications, and thereby enable those skilled in the art to make and use the invention. However, those skilled in the art will appreciate that the above description and examples are provided for ease of illustration and example only. The descriptions set forth are not intended to encompass all aspects of the invention or to limit the invention to the precise forms disclosed.

Claims

1. A method for fusing detection targets of multiple sensors, characterized in that: The method comprises the following steps: At a first moment, acquiring a detection target of a first sensor and marking a state indicator of the acquired detection target of the first sensor as valid; At a second moment, acquiring a detection target of the second sensor or a detection target of the first sensor, and predicting a detection target of the first sensor based on the detection target of the first sensor acquired at the first moment; as well as Using a preset state transition rule to mark the state indicator of the detection target of the second sensor or the detection target of the first sensor and the predicted detection target of the first sensor acquired at the second moment, The state indicator of the detected target includes one or more of the following: valid, fused, tracked, and untracked. 2 . The method according to claim 1 , wherein the plurality of sensors comprises one or more of the following: an image sensor, a lidar sensor, a millimeter-wave radar sensor, and an ultrasonic sensor.

3. The method according to claim 1 , wherein marking the state indicator of the detection target of the second sensor or the detection target of the first sensor and the predicted detection target of the first sensor obtained at the second moment using a preset state transition rule comprises: At a second moment, acquiring a detection target of the second sensor and predicting a detection target of the first sensor based on the detection target of the first sensor acquired at the first moment; In response to determining that the acquired detection target of the second sensor is associated with the predicted detection target of the first sensor, recording the acquired detection target of the second sensor and the predicted detection target of the first sensor as fused; marking the acquired detection target of the second sensor as valid in response to determining that the acquired detection target of the second sensor is not associated with the predicted detection target of the first sensor; as well as The predicted detection target of the first sensor that is outside the detection range of the second sensor is marked as valid.

4. The method according to claim 1, wherein marking the state indicator of the detection target of the second sensor or the detection target of the first sensor and the predicted detection target of the first sensor obtained at the second moment using a preset state transition rule comprises: At a second moment, acquiring a detection target of the first sensor and predicting a detection target of the first sensor based on the detection target of the first sensor acquired at the first moment; marking the acquired first sensor detection target and the predicted first sensor detection target as tracked in response to determining that the acquired first sensor detection target is associated with the predicted first sensor detection target; as well as In response to determining that the acquired first sensor detection target is not associated with the predicted first sensor detection target, the acquired first sensor detection target and the predicted first sensor detection target are marked as valid.

5. The method according to claim 3, wherein the method further comprises, at a third moment: obtaining a detection target of a third sensor; predicting a detection target of the second sensor based on the detection target of the second sensor acquired at the second moment; predicting a detection target of the first sensor based on the detection target of the first sensor predicted at the second moment; as well as The state indicators of the detection target of the third sensor, the predicted detection target of the second sensor, and the predicted detection target of the first sensor acquired at the third moment are marked using a preset state transition rule.

6. The method according to claim 5, wherein marking the state indicators of the detection target acquired by the third sensor, the detection target predicted by the second sensor, and the detection target predicted by the first sensor at the third moment using a preset state transition rule comprises: marking the acquired detection target of the third sensor and the predicted detection target of the second sensor or the predicted detection target of the first sensor as fused in response to determining that the acquired detection target of the third sensor is associated with the predicted detection target of the second sensor or the predicted detection target of the first sensor; marking the acquired detection target of the third sensor as valid in response to determining that the acquired detection target of the third sensor is not associated with the predicted detection target of the second sensor or the predicted detection target of the first sensor; as well as The state indicators of the predicted detection targets of the second sensor and the predicted detection targets of the first sensor outside the detection range of the third sensor are marked as being the same as the state indicators at the second moment. 7 . The method according to claim 1 , wherein the association between the detected targets is determined based on one or more of the following: Mahalanobis distance, Euclidean distance, angular distance, and velocity distance.

8. The method according to any one of claims 5 to 6, further comprising: Repeating the operation at the second moment and the operation at the third moment; Marking detected targets that have not been marked as tracked or fused after a predetermined number of time instants as untracked; as well as The detected targets marked as untracked are discarded.

9. A vehicle automatic driving system, characterized in that: The system comprises: Multiple sensors; Memory; and a processor communicatively coupled to the memory and the plurality of sensors, the processor configured to: At a first moment, acquiring a detection target of a first sensor and marking a state indicator of the acquired detection target of the first sensor as valid; At a second moment, acquiring a detection target of the second sensor or a detection target of the first sensor, and predicting a detection target of the first sensor based on the detection target of the first sensor acquired at the first moment; and Using a preset state transition rule to mark the state indicator of the detection target of the second sensor or the detection target of the first sensor and the predicted detection target of the first sensor acquired at the second moment, The state indicator of the detected target includes one or more of the following: valid, fused, tracked, and untracked.

10. The system of claim 9, wherein the plurality of sensors comprises one or more of: an image sensor, a lidar sensor, a millimeter wave radar sensor, and an ultrasonic sensor.

11. The system according to claim 9, wherein the processor is further configured to mark the state indicator of the detection target of the second sensor or the detection target of the first sensor and the predicted detection target of the first sensor acquired at the second moment using a preset state transition rule in the following manner: At a second moment, acquiring a detection target of the second sensor and predicting a detection target of the first sensor based on the detection target of the first sensor acquired at the first moment; In response to determining that the acquired detection target of the second sensor is associated with the predicted detection target of the first sensor, recording the acquired detection target of the second sensor and the predicted detection target of the first sensor as fused; marking the acquired detection target of the second sensor as valid in response to determining that the acquired detection target of the second sensor is not associated with the predicted detection target of the first sensor; as well as The predicted detection target of the first sensor that is outside the detection range of the second sensor is marked as valid.

12. The system according to claim 9, wherein the processor is further configured to mark the state indicator of the detection target of the second sensor or the detection target of the first sensor and the predicted detection target of the first sensor acquired at the second moment using a preset state transition rule in the following manner: At a second moment, acquiring a detection target of the first sensor and predicting a detection target of the first sensor based on the detection target of the first sensor acquired at the first moment; marking the acquired first sensor detection target and the predicted first sensor detection target as tracked in response to determining that the acquired first sensor detection target is associated with the predicted first sensor detection target; as well as In response to determining that the acquired first sensor detection target is not associated with the predicted first sensor detection target, the acquired first sensor detection target and the predicted first sensor detection target are marked as valid.

13. The system of claim 11, wherein the processor is further configured to: obtaining a detection target of a third sensor; predicting a detection target of the second sensor based on the detection target of the second sensor acquired at the second moment; predicting a detection target of the first sensor based on the detection target of the first sensor predicted at the second moment; as well as The state indicators of the detection target of the third sensor, the predicted detection target of the second sensor, and the predicted detection target of the first sensor acquired at the third moment are marked using a preset state transition rule.

14. The system according to claim 13, wherein the processor is further configured to mark the state indicators of the detection target of the third sensor, the predicted detection target of the second sensor, and the predicted detection target of the first sensor acquired at the third moment using a preset state transition rule in the following manner: marking the acquired detection target of the third sensor and the predicted detection target of the second sensor or the predicted detection target of the first sensor as fused in response to determining that the acquired detection target of the third sensor is associated with the predicted detection target of the second sensor or the predicted detection target of the first sensor; marking the acquired detection target of the third sensor as valid in response to determining that the acquired detection target of the third sensor is not associated with the predicted detection target of the second sensor or the predicted detection target of the first sensor; and The state indicators of the predicted detection targets of the second sensor and the predicted detection targets of the first sensor outside the detection range of the third sensor are marked as being the same as the state indicators at the second moment.

15. The system according to any one of claims 9 to 14, wherein the association between the detected targets is determined based on one or more of the following: Mahalanobis distance, Euclidean distance, angular distance, and velocity distance.

16. The system of any one of claims 13-14, wherein the processor is further configured to: Repeating the operation at the second moment and the operation at the third moment; Marking detections that have not been marked as tracked or fused after a predetermined number of time instants as untracked; and The detected targets marked as untracked are discarded.

17. A computer storage medium, characterized in that The computer storage medium comprises instructions that, when executed, perform the method according to any one of claims 1 to 8.

18. A vehicle, characterized in that: The vehicle comprises a vehicle automatic driving system according to any one of claims 9 to 16.

Images