Method and apparatus for fusion of obstacle information

Abstract

This application discloses a method and apparatus for fusion of obstacle information. The method involves acquiring obstacle information from frame k and preset fused obstacle information from frame (k-1). The preset fused obstacle information from frame (k-1) is then predicted to obtain fused obstacle prediction information for frame k. The obstacle information from frame k and the fused obstacle prediction information from frame k are then correlated and matched to obtain first fused obstacle information for frame k. Finally, a confidence correction operation is performed on the first fused obstacle information for frame k to obtain second fused obstacle information for frame k. This method improves the accuracy of fusing obstacle information perceived by different sensors by removing obstacles with confidence levels below a preset threshold through the confidence correction operation on the first fused obstacle information for frame k.

Description

Technical Field

[0001] This application belongs to the field of driver assistance technology, and in particular relates to a method and apparatus for fusing obstacle information. Background Technology

[0002] Vehicles typically carry various types of sensors, such as millimeter-wave radar, lidar, and cameras, used for tasks like sensing the surrounding environment and identifying and tracking obstacles. Because different types of sensors have different characteristics and limitations, such as field of view, resolution, noise, and blind spots, a single sensor cannot meet all task requirements. Therefore, fusing information from different sensors can improve the accuracy and robustness of environmental perception. However, existing multi-sensor target-level fusion schemes suffer from problems such as low accuracy in correlation matching and insufficient handling of false obstacle detections. Summary of the Invention

[0003] This application provides a method and apparatus for fusing obstacle information, which can improve the accuracy of fusing obstacle information perceived by different sensors.

[0004] In a first aspect, embodiments of this application provide a method for fusing obstacle information, the method including:

[0005] Obtain obstacle information from frame k and preset fused obstacle information from frame (k-1), where k is an integer greater than 1. The obstacle information includes information about multiple obstacles perceived by visual and non-visual sensors.

[0006] Predict the fused obstacle information of the preset (k-1)th frame to obtain the fused obstacle prediction information of the kth frame.

[0007] The obstacle information in frame k is correlated and matched with the fused obstacle prediction information in frame k to obtain the first fused obstacle information in frame k.

[0008] Perform a confidence correction operation on the first fused obstacle information of the k-th frame to obtain the second fused obstacle information of the k-th frame.

[0009] In one embodiment, the process of associating and matching the obstacle information of the k-th frame with the fused obstacle prediction information of the k-th frame to obtain the first fused obstacle information of the k-th frame includes:

[0010] The obstacle identifiers in the obstacle information of the k-th frame are associated and matched with the obstacle identifiers in the fused obstacle prediction information of the k-th frame to obtain the first fused obstacle information of the k-th frame.

[0011] In one embodiment, the first fused obstacle information of the k-th frame mentioned above includes multiple obstacle identifiers. A confidence correction operation is performed on the first fused obstacle information of the k-th frame to obtain the second fused obstacle information of the k-th frame, which includes:

[0012] For each obstacle identifier in the first fused obstacle information of the k-th frame, a first correction operation and a second correction operation are performed to obtain the corrected confidence scores for each obstacle identifier in the first fused obstacle information of the k-th frame. The first correction operation is based on the perception range area, and the second correction operation is based on the confidence score correction based on the traversability judgment.

[0013] Based on the corrected confidence level corresponding to each obstacle identifier, the target obstacle identifier in the first fused obstacle information of the k-th frame is deleted to obtain the second fused obstacle information of the k-th frame. The target obstacle identifier is the obstacle identifier with a corrected confidence level less than a preset threshold among multiple obstacle identifiers.

[0014] In one embodiment, the aforementioned first correction operation is an operation that corrects the original confidence scores corresponding to each obstacle identifier in the first fused obstacle information of the k-th frame based on a first confidence correction function.

[0015] The first confidence level correction function includes:

[0016] conf_new=f(fn_frames)*conf_org

[0017] Where conf_new is the corrected confidence level, conf_org is the original confidence level, and f() is the correction function.

[0018] In one embodiment, the second correction operation mentioned above is an operation that corrects the original confidence level corresponding to each obstacle identifier in the first fused obstacle information of the k-th frame based on a second confidence correction function or a third confidence correction function.

[0019] The second confidence level correction function includes:

[0020] conf_new=ω ttc_1 *conf_org

[0021] Where conf_new is the revised confidence level, conf_org is the original confidence level, and ω ttc_1 This is the first correction factor.

[0022] The third confidence level correction function includes:

[0023] conf_new=ω ttc_2 *conf_org

[0024] Where conf_new is the revised confidence level, conf_org is the original confidence level, and ω ttc_2 ω is the second correction factor. ttc_2 Greater than ω ttc_1 .

[0025] In one embodiment, the process of associating and matching the obstacle identifiers in the obstacle information of the k-th frame with the obstacle identifiers in the fused obstacle prediction information of the k-th frame to obtain the first fused obstacle information of the k-th frame includes:

[0026] The obstacle markers in the first perception information and the obstacle markers in the third perception information are matched from the same source to obtain the first matching result. Similarly, the obstacle markers in the second perception information and the obstacle markers in the fourth perception information are matched from the same source to obtain the second matching result.

[0027] Based on the first and second matching results, the first fused obstacle information of the k-th frame is obtained.

[0028] The first perception information consists of obstacle information perceived by a visual sensor in the obstacle information of the k-th frame; the second perception information consists of obstacle information perceived by a non-visual sensor in the obstacle information of the k-th frame; the third perception information consists of obstacle information perceived by a visual sensor in the fused obstacle prediction information of the k-th frame; the fourth perception information consists of obstacle information perceived by a non-visual sensor in the fused obstacle prediction information of the k-th frame; the first matching result includes the fifth and sixth perception information, where the fifth perception information consists of obstacle information that did not successfully match the obstacle identifier in the first perception information; the sixth perception information consists of obstacle information that did not successfully match the obstacle identifier in the second perception information; the second matching result includes the seventh and eighth perception information, where the seventh perception information consists of obstacle information that did not successfully match the obstacle identifier in the third perception information; and the eighth perception information consists of obstacle information that did not successfully match the obstacle identifier in the fourth perception information.

[0029] In one embodiment, the process of obtaining the first fused obstacle information for the k-th frame based on the first matching result and the second matching result includes:

[0030] By performing heterogeneous matching on the obstacle identifiers in the fifth, sixth, seventh, and eighth perception information, the first fused obstacle information of the k-th frame is obtained.

[0031] In one embodiment, before performing heterogeneous matching of obstacle identifiers from the fifth, sixth, seventh, and eighth perception information to obtain the first fused obstacle information of the k-th frame, the method further includes:

[0032] The obstacle information in the fifth, sixth, seventh, and eighth sensory information is corrected using a coordinate system to obtain the corrected fifth, sixth, seventh, and eighth sensory information.

[0033] By performing heterogeneous matching on the obstacle identifiers in the fifth, sixth, seventh, and eighth sensory information, the first fused obstacle information of the k-th frame is obtained, including:

[0034] By performing heterogeneous matching on the obstacle identifiers in the corrected fifth, sixth, seventh, and eighth perception information, the first fused obstacle information of the k-th frame is obtained.

[0035] In one embodiment, the visual sensor mentioned above includes a camera, and the non-visual sensor includes any one of millimeter-wave radar, lidar, and ultrasonic sensors.

[0036] Secondly, embodiments of this application provide an obstacle information fusion device, which may include:

[0037] The acquisition module is used to acquire obstacle information of the k-th frame and preset fused obstacle information of the (k-1)-th frame, where k is an integer greater than 1. The obstacle information includes information on multiple obstacles perceived by visual and non-visual sensors.

[0038] The prediction module is used to predict the fused obstacle information of the preset (k-1)th frame to obtain the fused obstacle prediction information of the kth frame.

[0039] The matching module is used to correlate and match the obstacle information of the k-th frame with the fused obstacle prediction information of the k-th frame to obtain the first fused obstacle information of the k-th frame.

[0040] The correction module is used to perform a confidence correction operation on the first fused obstacle information of the k-th frame to obtain the second fused obstacle information of the k-th frame.

[0041] Thirdly, embodiments of this application provide an electronic device, the device comprising:

[0042] processor;

[0043] Memory used to store processor-executable instructions;

[0044] The processor is configured to execute instructions to implement the obstacle information fusion method as shown in any embodiment of the first aspect.

[0045] Fourthly, embodiments of this application provide a computer storage medium storing a computer program that, when executed by a processor, implements the obstacle information fusion method as shown in any embodiment of the first aspect.

[0046] Fifthly, embodiments of this application also provide a computer program product comprising a computer program stored in a readable storage medium, wherein at least one processor of the device reads from the storage medium and executes the computer program, causing the device to perform the obstacle information fusion method shown in any embodiment of the first aspect.

[0047] This application provides a method and apparatus for fusing obstacle information. Compared with the prior art, this application has the following advantages:

[0048] An obstacle information fusion method and apparatus according to an embodiment of this application acquires obstacle information of the k-th frame and preset fused obstacle information of the (k-1)-th frame. It then predicts the preset fused obstacle information of the (k-1)-th frame to obtain fused obstacle prediction information of the k-th frame. Next, it correlates and matches the obstacle information of the k-th frame with the fused obstacle prediction information of the k-th frame to obtain first fused obstacle information of the k-th frame. Finally, it performs a confidence correction operation on the first fused obstacle information of the k-th frame to obtain second fused obstacle information of the k-th frame. In this way, by performing a confidence correction operation on the first fused obstacle information of the k-th frame, obstacles with confidence levels below a preset threshold can be removed, thereby improving the accuracy of fusing obstacle information perceived by different sensors. Attached Figure Description

[0049] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0050] Figure 1 This is a flowchart illustrating a method for fusing obstacle information provided in an embodiment of this application;

[0051] Figure 2 This is a flowchart illustrating another obstacle information fusion method provided in an embodiment of this application;

[0052] Figure 3This is a schematic diagram of a camera millimeter-wave radar observation correction provided in an embodiment of this application;

[0053] Figure 4 This is a schematic diagram of the core field of view (FOV) region of an obstacle provided in an embodiment of this application;

[0054] Figure 5 This is a schematic diagram illustrating obstacle passability determination provided in an embodiment of this application;

[0055] Figure 6 This is a schematic diagram of the structure of an obstacle information fusion device provided in an embodiment of this application;

[0056] Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0057] The features and exemplary embodiments of various aspects of this application will be described in detail below. To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only intended to explain this application and not to limit it. For those skilled in the art, this application can be implemented without some of these specific details. The following description of the embodiments is merely to provide a better understanding of this application by illustrating examples.

[0058] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus that includes the element.

[0059] As discussed in the background section, existing related technologies suffer from problems such as low accuracy in association matching and insufficient handling of falsely reported obstacles. To address these issues, this application provides a method and apparatus for fusing obstacle information. The method involves acquiring obstacle information from frame k and preset fused obstacle information from frame (k-1), predicting the preset fused obstacle information from frame (k-1) to obtain predicted fused obstacle information for frame k, then associating and matching the obstacle information from frame k with the predicted fused obstacle information to obtain first fused obstacle information for frame k, and finally performing a confidence correction operation on the first fused obstacle information to obtain second fused obstacle information for frame k. This approach allows for the removal of obstacles with confidence levels below a preset threshold from the first fused obstacle information, thereby improving the accuracy of fusing obstacle information perceived by different sensors.

[0060] This application provides a method and apparatus for fusing obstacle information. The method for fusing obstacle information provided in this application will be described first. Figure 1 As shown in the embodiments of this application, the obstacle information fusion method includes the following steps:

[0061] S101: Obtain obstacle information from frame k and preset fused obstacle information from frame (k-1), where k is an integer greater than 1. The obstacle information includes information about multiple obstacles perceived by visual and non-visual sensors.

[0062] S102: Predict the fused obstacle information of the preset (k-1)th frame to obtain the fused obstacle prediction information of the kth frame.

[0063] S103: Associate and match the obstacle information of frame k with the fused obstacle prediction information of frame k to obtain the first fused obstacle information of frame k.

[0064] S104: Perform a confidence correction operation on the first fused obstacle information of the k-th frame to obtain the second fused obstacle information of the k-th frame.

[0065] This application provides a method and apparatus for fusing obstacle information. By acquiring obstacle information from frame k and preset fused obstacle information from frame (k-1), the preset fused obstacle information from frame (k-1) is predicted to obtain fused obstacle prediction information for frame k. Then, the obstacle information from frame k and the fused obstacle prediction information from frame k are correlated and matched to obtain first fused obstacle information for frame k. Finally, a confidence correction operation is performed on the first fused obstacle information for frame k to obtain second fused obstacle information for frame k. In this way, by performing a confidence correction operation on the first fused obstacle information for frame k, obstacles with confidence levels below a preset threshold can be removed, thereby improving the accuracy of fusing obstacle information perceived by different sensors.

[0066] In S101, in one example, the obstacle information in the k-th frame can be obstacle information detected by the camera and obstacle information detected by millimeter-wave radar. In one example, the visual sensor includes a camera, and the non-visual sensor includes any one of millimeter-wave radar, lidar, and ultrasonic sensors.

[0067] In S102, in one example, if obstacle information of the kth frame is currently obtained, the existing fused obstacle information of the (k-1)th frame will be predicted in one step. The prediction method includes, but is not limited to, Kalman filtering, to obtain the fused obstacle prediction information of the kth frame.

[0068] In S103, in one example, the obstacle information of the k-th frame and the fused obstacle prediction information of the k-th frame are associated and matched to obtain the first fused obstacle information of the k-th frame. For example, this includes associating and matching the obstacle information of the k-th frame and the fused obstacle prediction information of the k-th frame with the same source (camera-camera, radar-radar) and different source (camera-radar).

[0069] In one example, the obstacle information of frame k and the fused obstacle prediction information of frame k are correlated and matched to obtain the first fused obstacle information of frame k, including:

[0070] The obstacle identifiers in the obstacle information of the k-th frame are associated and matched with the obstacle identifiers in the fused obstacle prediction information of the k-th frame to obtain the first fused obstacle information of the k-th frame.

[0071] Since the fused obstacle prediction information in the k-th frame stores the matched obstacle identifiers, when tracking detected obstacles using the internal perception algorithm module based on these identifiers, the original internal features can be fully utilized to obtain more accurate association matching results. Therefore, association matching based on obstacle identifiers is equivalent to indirectly using the original perception features, which can effectively improve the accuracy of association matching.

[0072] In one example, the obstacle identifiers in the obstacle information of frame k are correlated and matched with the obstacle identifiers in the fused obstacle prediction information of frame k to obtain the first fused obstacle information of frame k, including:

[0073] The obstacle markers in the first perception information and the obstacle markers in the third perception information are matched from the same source to obtain the first matching result. Similarly, the obstacle markers in the second perception information and the obstacle markers in the fourth perception information are matched from the same source to obtain the second matching result.

[0074] Based on the first and second matching results, the first fused obstacle information of the k-th frame is obtained.

[0075] The first perception information consists of obstacle information perceived by a visual sensor in the obstacle information of the k-th frame; the second perception information consists of obstacle information perceived by a non-visual sensor in the obstacle information of the k-th frame; the third perception information consists of obstacle information perceived by a visual sensor in the fused obstacle prediction information of the k-th frame; the fourth perception information consists of obstacle information perceived by a non-visual sensor in the fused obstacle prediction information of the k-th frame; the first matching result includes the fifth and sixth perception information, where the fifth perception information consists of obstacle information that did not successfully match the obstacle identifier in the first perception information; the sixth perception information consists of obstacle information that did not successfully match the obstacle identifier in the second perception information; the second matching result includes the seventh and eighth perception information, where the seventh perception information consists of obstacle information that did not successfully match the obstacle identifier in the third perception information; and the eighth perception information consists of obstacle information that did not successfully match the obstacle identifier in the fourth perception information.

[0076] In one example, based on the first matching result and the second matching result, the first fused obstacle information of the k-th frame is obtained, including:

[0077] By performing heterogeneous matching on the obstacle identifiers in the fifth, sixth, seventh, and eighth perception information, the first fused obstacle information of the k-th frame is obtained.

[0078] To better illustrate the above embodiments, a specific implementation method is described below. For example, obstacle identification-based matching is first performed. This results in a list of matched and fused obstacles (fus_matched_1) and a list of unmatched and fused obstacles (fus_unmatched_1); a list of matched camera obstacles (cam_matched_1) and a list of unmatched camera obstacles (cam_unmatched_1); and a list of matched radar obstacles (rad_matched_1) and a list of unmatched radar obstacles (rad_unmatched_1).

[0079] The same-source matching includes: performing correlation matching on the fused obstacles updated by camera obstacles in the (k-1)th frame of the unmatched fused obstacle list fus_unmatched_1, and the unmatched camera obstacle list cam_unmatched_1, updating the matched and unmatched fused obstacle lists, and the matched and unmatched camera obstacle lists, resulting in fus_matched_2, fus_unmatched_2, cam_matched_2, and cam_unmatched_2. Similarly, performing correlation matching on the fused obstacles updated by radar obstacles in the (k-1)th frame of the unmatched fused obstacle list fus_unmatched_2, and the unmatched radar obstacle list rad_unmatched_1, updating the matched and unmatched fused obstacle lists, and the matched and unmatched radar obstacle lists, resulting in fus_matched_3, fus_unmatched_3, rad_matched_2, and rad_unmatched_2.

[0080] Heterogeneous matching includes: performing correlation matching on the unmatched fused obstacle list fus_unmatched_3, the unmatched radar obstacle list rad_unmatched_2, and the unmatched camera obstacle list cam_unmatched_2; updating the matched and unmatched fused obstacle lists, the matched and unmatched radar obstacle lists, and the matched and unmatched camera obstacle lists, resulting in fus_matched_4, fus_unmatched_4, rad_matched_3, rad_unmatched_3, cam_matched_3, and cam_unmatched_3.

[0081] In one example, before performing heterogeneous matching on obstacle identifiers from the fifth, sixth, seventh, and eighth sensory information to obtain the first fused obstacle information for the k-th frame, the process also includes:

[0082] The obstacle information in the fifth, sixth, seventh, and eighth sensory information is corrected using a coordinate system to obtain the corrected fifth, sixth, seventh, and eighth sensory information.

[0083] By performing heterogeneous matching on the obstacle identifiers in the fifth, sixth, seventh, and eighth sensory information, the first fused obstacle information of the k-th frame is obtained, including:

[0084] By performing heterogeneous matching on the obstacle identifiers in the corrected fifth, sixth, seventh, and eighth perception information, the first fused obstacle information of the k-th frame is obtained.

[0085] For example, when performing camera-radar correlation matching for heterogeneous sensor information, the observation positions output by the two sensors are different, so it is necessary to correct them. The obstacle detected by the camera needs to be transformed into the radar coordinate system. This can be achieved by translating the centroid coordinates of the obstacle into the radar coordinate system by a vector (the intersection of the line connecting the origin of the radar coordinate system and the coordinates of the obstacle detected by the radar and the camera obstacle).

[0086] In S104, a confidence correction operation is performed on the first fused obstacle information of the k-th frame to obtain the second fused obstacle information of the k-th frame. The confidence correction can include two parts: one is the correction based on the core FOV (Field of View, i.e., the field of view angle, which refers to the sensing range that the sensor can cover) area, and the other is the confidence correction based on the traversability judgment.

[0087] In one example, the first fused obstacle information in frame k includes multiple obstacle identifiers. Performing a confidence correction operation on the first fused obstacle information in frame k yields the second fused obstacle information in frame k, which includes:

[0088] For each obstacle identifier in the first fused obstacle information of the k-th frame, a first correction operation and a second correction operation are performed to obtain the corrected confidence scores for each obstacle identifier in the first fused obstacle information of the k-th frame. The first correction operation is based on the perception range area, and the second correction operation is based on the confidence score correction based on the traversability judgment.

[0089] Based on the corrected confidence level corresponding to each obstacle identifier, the target obstacle identifier in the first fused obstacle information of the k-th frame is deleted to obtain the second fused obstacle information of the k-th frame. The target obstacle identifier is the obstacle identifier with a corrected confidence level less than a preset threshold among multiple obstacle identifiers.

[0090] In one example, the first correction operation is an operation that corrects the original confidence scores corresponding to each obstacle identifier in the first fused obstacle information of the k-th frame based on a first confidence correction function.

[0091] The first confidence level correction function includes:

[0092] conf_new=f(fn_frames)*conf_org

[0093] Where conf_new is the corrected confidence level, conf_org is the original confidence level, and f() is the correction function.

[0094] In one example, the second correction operation is an operation that corrects the original confidence scores corresponding to each obstacle identifier in the first fused obstacle information of the k-th frame based on either a second confidence correction function or a third confidence correction function.

[0095] The second confidence level correction function includes:

[0096] conf_new=ω ttc_1 *conf_org

[0097] Where conf_new is the revised confidence level, conf_org is the original confidence level, and ω ttc_1 This is the first correction factor.

[0098] The third confidence level correction function includes:

[0099] conf_new=ω ttc_2 *conf_org

[0100] Where conf_new is the revised confidence level, conf_org is the original confidence level, and ω ttc_2 ω is the second correction factor. ttc_2 Greater than ω ttc_1 .

[0101] To better illustrate the embodiments of this application, a specific embodiment is given below as an example, such as... Figure 2 As shown, Figure 2 This is a flowchart illustrating a method for fusing obstacle information, which mainly includes the following steps:

[0102] Step S1 involves acquiring obstacle information from sensor input, primarily obstacle detection information from cameras and millimeter-wave radar. Since the acquired sensor input is generally in the sensor coordinate system (origin at the sensor installation location, x and y axes defined by the sensor), it's necessary to spatially align the camera and radar obstacle information to the vehicle's coordinate system. The origin of the vehicle's coordinate system is at the center of the rear axle of the vehicle, with the x-axis pointing in the vehicle's driving direction and the y-axis pointing to the right side of the vehicle.

[0103] Step S2: Assuming the current acquired sensor input information is the k-th frame, perform a one-step prediction on the existing obstacle information fused from the (k-1)-th frame (common methods include Kalman filtering) to obtain the fused obstacle prediction information for the k-th frame. Then, perform correlation matching between the fused prediction information and the sensor input information of the k-th frame. Fully utilizing the respective advantages of the camera and radar, the correlation matching is divided into two parts: same-source (camera-camera, radar-radar) and different-source (camera-radar). Correlation matching mainly includes the following four steps:

[0104] Step S2.1: The fused obstacle list stores the IDs of the matched perceived obstacles. First, matching is performed based on these perceived IDs. This results in a list of matched fused obstacles (fus_matched_1) and unmatched fused obstacles (fus_unmatched_1); a list of matched camera obstacles (cam_matched_1) and unmatched camera obstacles (cam_unmatched_1); and a list of matched radar obstacles (rad_matched_1) and unmatched radar obstacles (rad_unmatched_1). When the internal perception algorithm module tracks detected obstacles, it can fully utilize the original internal features to obtain more accurate association matching results. Therefore, association matching based on perceived IDs is equivalent to indirectly using the original perception features, effectively improving the accuracy of association matching.

[0105] Step S2.2: Perform association matching on the merged obstacles updated by camera obstacles in the (k-1)th frame of the unmatched fused obstacle list fus_unmatched_1 and the unmatched camera obstacle list cam_unmatched_1, and update the matched and unmatched fused obstacle lists and the matched and unmatched camera obstacle lists to obtain fus_matched_2, fus_unmatched_2, cam_matched_2, and cam_unmatched_2. Since the camera's perception input advantage information includes precise bounding box size, category, heading, etc., this information must be considered additionally during camera-to-camera association matching.

[0106] Step S2.3: Perform association matching on the fused obstacles updated by radar obstacles in the (k-1)th frame of the unmatched fused obstacle list fus_unmatched_2 and the unmatched radar obstacle list rad_unmatched_1, updating the matched and unmatched fused obstacle lists and the matched and unmatched radar obstacle lists, resulting in fus_matched_3, fus_unmatched_3, rad_matched_2, and rad_unmatched_2. Since the radar's perception input advantage information includes precise speed, this information must be additionally considered during radar-to-radar association matching.

[0107] Step S2.4: Perform correlation matching on the unmatched fused obstacle list fus_unmatched_3, the unmatched radar obstacle list rad_unmatched_2, and the unmatched camera obstacle list cam_unmatched_2. Update the matched and unmatched fused obstacle lists, the matched and unmatched radar obstacle lists, and the matched and unmatched camera obstacle lists to obtain fus_matched_4, fus_unmatched_4, rad_matched_3, rad_unmatched_3, cam_matched_3, and cam_unmatched_3. For heterogeneous sensor information, when performing camera-radar correlation matching, corrections are needed because the observation positions output by the two sensors are different, as illustrated in the diagram below. Figure 3 As shown. Transforming the obstacle detected by the camera into the radar coordinate system, its centroid coordinates are represented as O. c The coordinates of obstacles detected by radar are represented as O. r The origin of the radar coordinate system is parallel to O. r The intersection of the line and the camera obstacle bbox is point P. inter , is the theoretical observation point of the radar when there are no external sensor errors. O r By vector The translation yields the corrected observation point O. r,c Using O r,c Perform tasks such as calculating the distance for correlation matching and updating fused information.

[0108] Step S3 involves performing confidence correction on the unmatched camera obstacle set cam_unmatched_3 and the unmatched radar obstacle set rad_unmatched_3, deleting obstacles with a confidence level lower than the set threshold conf_thres. Confidence correction mainly includes two parts: one is correction based on the core FOV (Field of View, referring to the sensing range that the sensor can cover), and the other is confidence correction based on traversability judgment.

[0109] The core FOV region is a part of the original FOV region, as Figure 4 shown, representing the region with better sensor detection performance, where both the false negative rate and false positive rate are very low. It can be obtained by shrinking the original FOV by a certain ratio inward or manual setting, etc. If an obstacle does not appear starting from the edge of the core FOV region but directly appears within the core FOV region, it means that there is a missed detection from the FOV boundary to the current appearance position. Define the missed detection distance fn_dis (m) as the distance from the FOV boundary to the current appearance position, the current speed of the obstacle is obj_spd (m / s), and the missed detection time fn_time (s) is fn_dis / obj_spd. The sensor frame rate is frame_ratio (hz), then the number of missed detection frames fn_frames is fn_time / frame_ratio. Define the confidence correction function as:

[0110] conf_new = f(fn_frames) * conf_org (Equation 1)

[0111] where conf_new is the corrected confidence, conf_org is the original confidence, and f() is the correction function.

[0112] The schematic diagram of the confidence correction scenario for traversability judgment is as Figure 5 shown. Define the lanes of interest as the host vehicle's current lane, the left adjacent lane, and the right adjacent lane. Only the obstacles in the lanes of interest are considered in this part. For the perceived obstacle A that has not been matched and fused, calculate its time to collision ttc with the obstacle C behind in the same lane. If there is still no deceleration action when ttc is less than the set threshold ttc_thres_1, it is determined to be traversable, and define the confidence correction function as:

[0113] conf_new = ω ttc_1 * conf_org (Equation 2)

[0114] where conf_new is the corrected confidence, conf_org is the original confidence, and ω ttc_1 is the correction coefficient.

[0115] For the case where obstacle C does not exist, calculate the average time to collision ttc with the nearest obstacles B and C behind in the left and right adjacent lanes. If there is still no deceleration action when ttc is less than the set threshold ttc_thres_2 (ttc_thres_2 < ttc_thres_1), it is determined to be traversable, and define the confidence correction function as:

[0116] conf_new = ω ttc_2 * conf_org (Equation 2)

[0117] Where conf_new is the corrected confidence level, conf_org is the original confidence level, and ω ttc_2 ω is the correction factor. ttc_2 >ω ttc_1 .

[0118] Step S4 involves lifecycle management of fusion obstacles, including updating matched fusion obstacles, predicting unmatched fusion obstacles, creating new fusion obstacles for unmatched perceived obstacles, and deleting fusion obstacles predicted over a long period of time.

[0119] Based on the obstacle information fusion method provided in the above embodiments, correspondingly, as... Figure 6 As shown, this application embodiment provides an obstacle information fusion device 600, which may include:

[0120] The acquisition module 601 is used to acquire obstacle information of the k-th frame and preset fused obstacle information of the (k-1)-th frame, where k is an integer greater than 1. The obstacle information includes information on multiple obstacles perceived by visual sensors and non-visual sensors.

[0121] Prediction module 602 is used to predict the fused obstacle information of the preset (k-1)th frame to obtain the fused obstacle prediction information of the kth frame.

[0122] Matching module 603 is used to associate and match the obstacle information of frame k with the fused obstacle prediction information of frame k to obtain the first fused obstacle information of frame k.

[0123] The correction module 604 is used to perform a confidence correction operation on the first fused obstacle information of the k-th frame to obtain the second fused obstacle information of the k-th frame.

[0124] In one embodiment, the matching module may specifically be used for:

[0125] The obstacle identifiers in the obstacle information of the k-th frame are associated and matched with the obstacle identifiers in the fused obstacle prediction information of the k-th frame to obtain the first fused obstacle information of the k-th frame.

[0126] In one embodiment, the correction module may specifically be used for:

[0127] For each obstacle identifier in the first fused obstacle information of the k-th frame, a first correction operation and a second correction operation are performed to obtain the corrected confidence scores for each obstacle identifier in the first fused obstacle information of the k-th frame. The first correction operation is based on the perception range area, and the second correction operation is based on the confidence score correction based on the traversability judgment.

[0128] Based on the corrected confidence level corresponding to each obstacle identifier, the target obstacle identifier in the first fused obstacle information of the k-th frame is deleted to obtain the second fused obstacle information of the k-th frame. The target obstacle identifier is the obstacle identifier with a corrected confidence level less than a preset threshold among multiple obstacle identifiers.

[0129] In one embodiment, the matching module may specifically be used for:

[0130] The obstacle markers in the first perception information and the obstacle markers in the third perception information are matched from the same source to obtain the first matching result. Similarly, the obstacle markers in the second perception information and the obstacle markers in the fourth perception information are matched from the same source to obtain the second matching result.

[0131] Based on the first and second matching results, the first fused obstacle information of the k-th frame is obtained.

[0132] The first perception information consists of obstacle information perceived by a visual sensor in the obstacle information of the k-th frame; the second perception information consists of obstacle information perceived by a non-visual sensor in the obstacle information of the k-th frame; the third perception information consists of obstacle information perceived by a visual sensor in the fused obstacle prediction information of the k-th frame; the fourth perception information consists of obstacle information perceived by a non-visual sensor in the fused obstacle prediction information of the k-th frame; the first matching result includes the fifth and sixth perception information, where the fifth perception information consists of obstacle information that did not successfully match the obstacle identifier in the first perception information; the sixth perception information consists of obstacle information that did not successfully match the obstacle identifier in the second perception information; the second matching result includes the seventh and eighth perception information, where the seventh perception information consists of obstacle information that did not successfully match the obstacle identifier in the third perception information; and the eighth perception information consists of obstacle information that did not successfully match the obstacle identifier in the fourth perception information.

[0133] In one embodiment, the matching module may further include:

[0134] By performing heterogeneous matching on the obstacle identifiers in the fifth, sixth, seventh, and eighth perception information, the first fused obstacle information of the k-th frame is obtained.

[0135] In one embodiment, the matching module may further include:

[0136] The obstacle information in the fifth, sixth, seventh, and eighth sensory information is corrected using a coordinate system to obtain the corrected fifth, sixth, seventh, and eighth sensory information.

[0137] The matching module can also be specifically used for:

[0138] By performing heterogeneous matching on the obstacle identifiers in the corrected fifth, sixth, seventh, and eighth perception information, the first fused obstacle information of the k-th frame is obtained.

[0139] Based on the obstacle information fusion method and apparatus provided in the above embodiments, this application also provides an electronic device 700, such as... Figure 7 As shown:

[0140] It includes a processor 701, a memory 702, and a computer program stored in the memory 702 and executable on the processor 701. When the computer program is executed by the processor 701, it implements the various processes of the above-described obstacle information fusion method embodiment and achieves the same technical effect.

[0141] Specifically, the processor 701 may include a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits that can be configured to implement the embodiments of this application.

[0142] Memory 702 may include mass storage for data or instructions. For example, and not limitingly, memory 702 may include a hard disk drive (HDD), floppy disk drive, flash memory, optical disk, magneto-optical disk, magnetic tape, or Universal Serial Bus (USB) drive, or a combination of two or more of these. Where appropriate, memory 702 may include removable or non-removable (or fixed) media. Where appropriate, memory 702 may be internal or external to the integrated gateway disaster recovery device. In a particular embodiment, memory 702 is non-volatile solid-state memory.

[0143] In certain embodiments, the memory may include read-only memory (ROM), random access memory (RAM), disk storage media devices, optical storage media devices, flash memory devices, and electrical, optical, or other physical / tangible memory storage devices. Thus, typically, memory includes one or more tangible (non-transitory) computer-readable storage media (e.g., memory devices) encoded with software including computer-executable instructions, and when the software is executed (e.g., by one or more processors), it is operable to perform the operations described with reference to the method according to one aspect of this application.

[0144] The processor 701 reads and executes computer program instructions stored in the memory 702 to implement any of the obstacle information fusion methods in the above embodiments.

[0145] In one example, the electronic device may also include a communication interface 703 and a bus 710. As an example, such as... Figure 7 As shown, the processor 701, memory 702, and communication interface 703 are connected through bus 710 and complete communication with each other.

[0146] The communication interface 703 is mainly used to realize communication between various modules, devices, units and / or equipment in the embodiments of this application.

[0147] Bus 710 includes hardware, software, or both, that couples components of an online data traffic metering device together. For example, and not limitingly, the bus may include an Accelerated Graphics Port (AGP) or other graphics bus, an Enhanced Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), HyperTransport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an Infinite Bandwidth Interconnect, a Low Pin Count (LPC) bus, a memory bus, a Microchannel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local (VLB) bus, or other suitable buses, or combinations of two or more of these. Where appropriate, bus 710 may include one or more buses. Although specific buses are described and illustrated in embodiments of this application, any suitable bus or interconnect is contemplated herein.

[0148] This application also provides a computer-readable storage medium storing a computer program. When executed by a processor, the computer program implements the various processes of the obstacle information fusion method embodiments described above, and achieves the same technical effects. To avoid repetition, it will not be described again here. The computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, etc.

[0149] It should be clarified that this application is not limited to the specific configurations and processes described above and shown in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of this application is not limited to the specific steps described and shown. Those skilled in the art can make various changes, modifications, and additions, or change the order of steps, after understanding the spirit of this application.

[0150] The functional blocks shown in the above block diagram can be implemented as hardware, software, firmware, or a combination thereof. When implemented in hardware, they can be, for example, electronic circuits, application-specific integrated circuits (ASICs), appropriate firmware, plug-ins, function cards, etc. When implemented in software, the elements of this application are programs or code segments used to perform the required tasks. Programs or code segments can be stored on a machine-readable medium or transmitted over a transmission medium or communication link via data signals carried on a carrier wave. "Machine-readable medium" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, fiber optic media, radio frequency (RF) links, etc. Code segments can be downloaded via computer networks such as the Internet, intranets, etc.

[0151] It should also be noted that the exemplary embodiments mentioned in this application describe methods or systems based on a series of steps or apparatus. However, this application is not limited to the order of the above steps; that is, the steps can be performed in the order mentioned in the embodiments, or in a different order, or several steps can be performed simultaneously.

[0152] The aspects of this application have been described above with reference to flowchart illustrations and / or block diagrams of methods, apparatus, and computer program products according to embodiments of this application. It should be understood that each block in the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that these instructions, executable via the processor of the computer or other programmable data processing apparatus, enable the implementation of the functions / actions specified in one or more blocks of the flowchart illustrations and / or block diagrams. Such a processor can be, but is not limited to, a general-purpose processor, a special-purpose processor, a special application processor, or a field-programmable logic circuit. It is also understood that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can also be implemented by dedicated hardware performing the specified functions or actions, or can be implemented by a combination of dedicated hardware and computer instructions.

[0153] The above are merely specific embodiments of this application. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, modules, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here. It should be understood that the protection scope of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the protection scope of this application.

Claims

1. A method for fusing obstacle information, characterized in that, include: Obstacle information from frame k and preset fused obstacle information from frame (k-1), where k is an integer greater than 1, is obtained. The obstacle information includes information about multiple obstacles perceived by visual and non-visual sensors. The fused obstacle information of the preset (k-1)th frame is predicted to obtain the fused obstacle prediction information of the kth frame. The obstacle information in the k-th frame and the fused obstacle prediction information in the k-th frame are correlated and matched to obtain the first fused obstacle information in the k-th frame. A confidence correction operation is performed on the first fused obstacle information of the k-th frame to obtain the second fused obstacle information of the k-th frame; The first fused obstacle information of the k-th frame includes multiple obstacle identifiers. The step of performing a confidence correction operation on the first fused obstacle information of the k-th frame to obtain the second fused obstacle information of the k-th frame includes: performing a first correction operation and a second correction operation on the original confidence corresponding to each obstacle identifier in the first fused obstacle information of the k-th frame to obtain the corrected confidence corresponding to each obstacle identifier in the first fused obstacle information of the k-th frame. The first correction operation is a correction operation based on the perception range area, and the second correction operation is a confidence correction operation based on the traversability judgment. Based on the corrected confidence level corresponding to each obstacle identifier, the target obstacle identifier in the first fused obstacle information of the k-th frame is deleted to obtain the second fused obstacle information of the k-th frame. The target obstacle identifier is the obstacle identifier among the plurality of obstacle identifiers whose corrected confidence level is less than a preset threshold.

2. The method according to claim 1, characterized in that, The step of associating and matching the obstacle information of the k-th frame with the fused obstacle prediction information of the k-th frame to obtain the first fused obstacle information of the k-th frame includes: The obstacle identifiers in the obstacle information of the k-th frame are associated and matched with the obstacle identifiers in the fused obstacle prediction information of the k-th frame to obtain the first fused obstacle information of the k-th frame.

3. The method according to claim 2, characterized in that, The first correction operation is an operation that corrects the original confidence scores of each obstacle identifier in the first fused obstacle information of the k-th frame based on a first confidence score correction function. The first confidence correction function includes: conf_new=f(fn_frames)*conf_org Wherein, conf_new is the corrected confidence level, conf_org is the original confidence level, and f() is the correction function.

4. The method according to claim 2, characterized in that, The second correction operation is an operation that corrects the original confidence level of each obstacle identifier in the first fused obstacle information of the k-th frame based on a second confidence correction function or a third confidence correction function. The second confidence correction function includes: conf_new=ω ttc_1 *conf_org Wherein, conf_new is the modified confidence level, conf_org is the original confidence level, and ω ttc_1 The first correction factor is... The third confidence level correction function includes: conf_new=ω ttc_2 *conf_org Wherein, conf_new is the modified confidence level, conf_org is the original confidence level, and ω ttc_2 As the second correction coefficient, ω ttc_2 Greater than the ω ttc_1 .

5. The method according to claim 2, characterized in that, The step of associating and matching the obstacle identifiers in the obstacle information of the k-th frame with the obstacle identifiers in the fused obstacle prediction information of the k-th frame to obtain the first fused obstacle information of the k-th frame includes: The obstacle markers in the first perception information and the obstacle markers in the third perception information are matched from the same source to obtain the first matching result. Similarly, the obstacle markers in the second perception information and the obstacle markers in the fourth perception information are matched from the same source to obtain the second matching result. Based on the first matching result and the second matching result, the first fused obstacle information of the k-th frame is obtained. Wherein, the first perception information is the obstacle information perceived by the visual sensor in the obstacle information of the k-th frame; the second perception information is the obstacle information perceived by the non-visual sensor in the obstacle information of the k-th frame; the third perception information is the obstacle information perceived by the visual sensor in the fused obstacle prediction information of the k-th frame; the fourth perception information is the obstacle information perceived by the non-visual sensor in the fused obstacle prediction information of the k-th frame; the first matching result includes a fifth perception information and a sixth perception information, wherein the fifth perception information is the obstacle information in the first perception information that did not successfully match the obstacle identifier of the second perception information; the sixth perception information is the obstacle information in the second perception information that did not successfully match the obstacle identifier of the first perception information; the second matching result includes a seventh perception information and an eighth perception information, wherein the seventh perception information is the obstacle information in the third perception information that did not successfully match the obstacle identifier of the fourth perception information; and the eighth perception information is the obstacle information in the fourth perception information that did not successfully match the obstacle identifier of the third perception information.

6. The method according to claim 5, characterized in that, The step of obtaining the first fused obstacle information for the k-th frame based on the first matching result and the second matching result includes: The obstacle identifiers in the fifth, sixth, seventh, and eighth perception information are matched from different sources to obtain the first fused obstacle information in the k-th frame.

7. The method according to claim 6, characterized in that, Before performing heterogeneous matching on the obstacle identifiers in the fifth, sixth, seventh, and eighth perception information to obtain the first fused obstacle information of the k-th frame, the method further includes: The obstacle information in the fifth, sixth, seventh, and eighth perception information is corrected using a coordinate system to obtain the corrected fifth, sixth, seventh, and eighth perception information. The step of performing heterogeneous matching on obstacle identifiers in the fifth, sixth, seventh, and eighth perception information to obtain the first fused obstacle information of the k-th frame includes: The obstacle identifiers in the corrected fifth, sixth, seventh, and eighth perception information are subjected to heterogeneous matching to obtain the first fused obstacle information of the k-th frame.

8. The method according to claim 1, characterized in that, The visual sensor includes a camera, and the non-visual sensor includes any one of millimeter-wave radar, lidar, and ultrasonic sensors.

9. A device for fusing obstacle information, characterized in that, The device includes: The acquisition module is used to acquire obstacle information of the k-th frame and preset fused obstacle information of the (k-1)-th frame, where k is an integer greater than 1. The obstacle information includes information on multiple obstacles perceived by visual sensors and non-visual sensors. The prediction module is used to predict the fused obstacle information of the preset (k-1)th frame to obtain the fused obstacle prediction information of the kth frame. The matching module is used to associate and match the obstacle information of the k-th frame with the fused obstacle prediction information of the k-th frame to obtain the first fused obstacle information of the k-th frame. The correction module is used to perform a confidence correction operation on the first fused obstacle information of the k-th frame to obtain the second fused obstacle information of the k-th frame; The first fused obstacle information of the k-th frame includes multiple obstacle identifiers. The correction module is used to perform a first correction operation and a second correction operation on the original confidence scores corresponding to each obstacle identifier in the first fused obstacle information of the k-th frame to obtain the corrected confidence scores corresponding to each obstacle identifier in the first fused obstacle information of the k-th frame. The first correction operation is a correction operation based on the perception range area, and the second correction operation is a confidence score correction operation based on traversability judgment. According to the corrected confidence scores corresponding to each obstacle identifier, the target obstacle identifier in the first fused obstacle information of the k-th frame is deleted to obtain the second fused obstacle information of the k-th frame. The target obstacle identifier is the obstacle identifier among the multiple obstacle identifiers whose corrected confidence scores are less than a preset threshold.

Images