Obstacle detection method, device and storage medium for autonomous driving vehicle

By building a historical image library and combining positioning sensor information, obstacle detection of autonomous vehicles is realized, which solves the problem of difficult to identify low-frequency obstacles in the prior art, improves obstacle recognition accuracy, and provides effective control for autonomous driving refer to.

CN118135539BActive Publication Date: 2024-11-05COWA TECHNOLOGY CO LTD +1
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410263365.7
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-03-08
Publication Date
2024-11-05
Estimated Expiration
2044-03-08

AI Technical Summary

Technical Problem

Existing obstacle detection models for autonomous driving vehicles are difficult to identify obstacle types that occur at low frequencies, resulting in an increased risk of safety accidents.

Method used

By building and updating the historical image library of autonomous driving vehicles, using real-time data to detect differences between historical images, combining positioning sensor information, the frame-interval IOU matching and timing filtering of obstacles is achieved, and the detection results are output.

Benefits of technology

It improves the accuracy of obstacle recognition, solves the long tail problem, and provides effective reference information for planning and control of autonomous driving.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118135539B_ABST
    Figure CN118135539B_ABST
Patent Text Reader

Abstract

The present invention discloses an obstacle detection method, device and storage medium for an autonomous driving vehicle, including: building a historical image library of an autonomous driving vehicle and updating it; obtaining real-time data when the autonomous driving vehicle is running, and finding similar historical images from the historical image library according to the real-time data, and combining them with the real-time image into an image pair; based on a difference detection model, detecting the change area of ​​the image at the current moment in the image pair, and obtaining a change area mask; splitting the change area mask through a connected domain, and passing it into a result queue as a detection result of each frame; matching each mask connected domain in the result queue to achieve inter-frame IOU matching; and filtering in time series to output the detection result. Based on scene change detection, the present invention integrates positioning sensors to achieve open set obstacle detection on a fixed driving route of an autonomous driving vehicle, and provides effective reference information for planning and control of autonomous driving.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the technical field of autonomous driving, and in particular relates to an obstacle detection method, device and storage medium for an autonomous driving vehicle. Background Art

[0002] In the current era of rapid development of various high-tech technologies, autonomous driving technology has received more and more attention. Various car manufacturers have invested a lot of resources in the research and development of autonomous driving systems (Auto Driving System, ADS), which includes a complete set of workflows from perception to decision-making. Among them, the perception system uses various perception sensors installed on the vehicle to perceive the surrounding situation in real time, and uses the data returned by the sensors and advanced algorithms to provide a basis for downstream decision-making tasks.

[0003] As the first step to realize autonomous driving, the perception system plays a vital role. Obstacle detection is one of the core technologies of the perception system, which is related to the safe driving of the vehicle and the effective identification of the surrounding environment. Obstacle detection technology is designed to enable the vehicle to continuously identify and respond to various obstacles, including pedestrians, vehicles, roadblocks, etc.

[0004] At present, most obstacle detection schemes for autonomous vehicles have fixed obstacle category restrictions. In the real world, some obstacle types appear very rarely and are difficult to appear in regular data sets, making it difficult for obstacle detection models to learn how to accurately identify the obstacle, which may lead to serious safety accidents. Summary of the invention

[0005] In order to solve the above technical problems, the present invention proposes an obstacle detection method, device and storage medium for an autonomous driving vehicle.

[0006] In order to achieve the above object, the technical solution of the present invention is as follows:

[0007] In a first aspect, the present invention discloses an obstacle detection method for an autonomous driving vehicle, comprising:

[0008] Step S1: Build and update the historical image library of the autonomous driving vehicle;

[0009] The data in the historical image library includes: historical images and the corresponding time and vehicle posture information;

[0010] Step S2: when the autonomous driving vehicle is running, real-time data is obtained, and the real-time data includes: real-time images and corresponding time and vehicle posture information;

[0011] Find similar historical images from the historical image library based on real-time data and combine them with the real-time images to form an image pair;

[0012] Step S3: Based on the difference detection model, detect the changed area of ​​the image at the current moment in the image pair and obtain a changed area mask;

[0013] Step S4: split the changed area mask by the connected domain and pass it into the result queue as the detection result of each frame;

[0014] Step S5: Match each mask connected domain in the result queue to achieve inter-frame IOU matching;

[0015] Step S6: Based on the inter-frame IOU matching result of step S5, perform time filtering and output the detection result.

[0016] Based on the above technical solution, the following improvements can be made:

[0017] As a preferred solution, step S1 includes:

[0018] Step S1.1: The autonomous driving vehicle is equipped with a camera and a positioning sensor, and drives along its fixed driving route, collecting camera images and corresponding time and positioning sensor information along the way, and saving them;

[0019] Step S1.2: Calculate the vehicle posture information corresponding to each frame of image through the positioning sensor information to obtain the vehicle position coordinates and vehicle yaw angle;

[0020] Step S1.3: Screen each frame of image according to requirements;

[0021] Step S1.4: When updating the data, the data in the existing historical image library is mixed with the new data, and steps S1.2 to S1.3 are repeated.

[0022] As a preferred solution, step S1.3 includes:

[0023] Step S1.3.1: With the camera as the center, set the restricted neighborhood 1 with a range of R1, and set the restricted neighborhood 2 with a range of R2, R2<R1;

[0024] Step S1.3.2: Screen out images whose visual field is blocked within the restricted neighborhood one;

[0025] Step S1.3.3: Traverse all images after step S1.3.2 in reverse chronological order. For each frame of image whose field of view is not blocked in restricted neighborhood two, filter all data points whose difference with the vehicle's yaw angle in the surrounding restricted neighborhood one is within the threshold range to obtain a sparse historical image library.

[0026] As a preferred solution, step S1.2 is further as follows:

[0027] The vehicle position and posture corresponding to each frame of the image are calculated through the positioning sensor information to obtain the vehicle position coordinates and vehicle yaw angle;

[0028] Build a database to store images, corresponding time and vehicle posture information.

[0029] As a preferred solution, step S2 includes:

[0030] Step S2.1: When the autonomous driving vehicle is running, the camera image and the corresponding time and positioning sensor information are collected in real time along the way;

[0031] Step S2.2: Calculate the current vehicle posture through the positioning sensor information to obtain the vehicle position coordinates and vehicle yaw angle;

[0032] Step S2.3: Use the current vehicle position coordinates to retrieve neighborhood points in the historical image library, and select historical images whose yaw angle difference with the vehicle is within a threshold range, and combine them with the real-time image to form an image pair.

[0033] As a preferred solution, step S5 is further as follows:

[0034] Match each masked connected domain in the result queue and calculate the IOU between it and all the results of the upper and lower frames.

[0035] The intersection I is the number of pixels of the intersection of the two masks between the two frames, and the union U is the number of pixels of the union of the two masks between the two frames. IOU = I / U;

[0036] If the intersection-over-union (IOU) between two frames is greater than the threshold, they are matched.

[0037] As a preferred solution, step S6 is further as follows:

[0038] Based on the matching results of each target in the result queue with the targets of the upper and lower frames obtained in step S5, the IOU matching results of alternate frames are connected into a linked list. If the matching linked list of the target in the current frame can cover the entire result queue length, it is output as the final detection mask result.

[0039] As a preferred solution, step S3 includes:

[0040] Step S3.1: Input image pair, historical image t0 and real-time image t1;

[0041] Step S3.2: extract features from the historical image t0 and the real-time image t1 to obtain the corresponding FPN historical feature map f0 and FPN real-time feature map f1;

[0042] Step S3.3: Use a fusion feature module to fuse the features of the FPN historical feature map f0 and the FPN real-time feature map f1 to obtain a fusion feature map;

[0043] Step S3.4: Use the segmentation network to obtain the changed segmentation result, that is, the changed area mask, using the fused feature map.

[0044] In a second aspect, the present invention further discloses an obstacle detection device for an autonomous driving vehicle, comprising:

[0045] The historical image library building module is used to build and update the historical image library of autonomous driving vehicles;

[0046] The data in the historical image library includes: historical images and the corresponding time and vehicle posture information;

[0047] The image pair combination module is used to obtain real-time data when the autonomous driving vehicle is running. The real-time data includes: real-time images and corresponding time and vehicle posture information;

[0048] Find similar historical images from the historical image library based on real-time data and combine them with the real-time images to form an image pair;

[0049] A change region detection module is used to detect the change region of the image at the current moment in the image pair based on a difference detection model, and obtain a change region mask;

[0050] The connected domain recognition module is used to split the change region mask by the connected domain and pass it into the result queue as the detection result of each frame;

[0051] The inter-frame IOU module is used to match each mask connected domain in the result queue to achieve inter-frame IOU matching;

[0052] The timing filtering module is used to perform timing filtering based on the IOU matching results of every other frame and output the detection results.

[0053] As a preferred solution, the historical image library construction module includes:

[0054] The first acquisition unit is used for the autonomous driving vehicle equipped with a camera and a positioning sensor to drive along its fixed driving route, collect camera images and corresponding time and positioning sensor information along the way, and save them;

[0055] The first posture calculation unit is used to calculate the vehicle posture information corresponding to each frame of the image through the positioning sensor information to obtain the vehicle position coordinates and the vehicle yaw angle;

[0056] A screening unit, used for screening each frame of image according to requirements;

[0057] The data updating unit is used to mix the data in the existing historical image library with the new data when updating the data, and repeatedly execute the methods in the first pose calculation unit and the screening unit.

[0058] As a preferred solution, the screening unit includes:

[0059] A restricted neighborhood setting unit is used to set a restricted neighborhood 1 with a range of R1 and a restricted neighborhood 2 with a range of R2 with the camera as the center, where R2<R1;

[0060] A screening unit, used for screening out images whose visual field is blocked within a restricted neighborhood;

[0061] The filtering unit is used to traverse all images after being filtered out by the filtering unit in reverse chronological order, and for each frame of image whose field of view is not blocked in the restricted neighborhood two, filter all data points whose difference with the vehicle yaw angle within the threshold range within the surrounding restricted neighborhood one to obtain a sparse historical image library.

[0062] As a preferred solution, the first posture calculation unit is further used to calculate the vehicle posture corresponding to each frame image through the positioning sensor information to obtain the vehicle position coordinates and the vehicle yaw angle;

[0063] Build a database to store images, corresponding time and vehicle posture information.

[0064] As a preferred solution, the image pair combination module includes:

[0065] The second acquisition unit is used to collect camera images and corresponding time and positioning sensor information in real time along the way when the autonomous driving vehicle is running;

[0066] The second posture calculation unit is used to calculate the current vehicle posture through the positioning sensor information to obtain the vehicle position coordinates and the vehicle yaw angle;

[0067] The image pair combination unit is used to retrieve neighborhood points in the historical image library using the current vehicle position coordinates, and to screen the historical images whose differences with the vehicle yaw angle are within a threshold range, and to combine them with the real-time image into an image pair.

[0068] As a preferred solution, the inter-frame IOU module is further used to match each mask connected domain in the result queue and calculate the IOU between it and all the results of the upper and lower frames.

[0069] The intersection I is the number of pixels of the intersection of the two masks between the two frames, and the union U is the number of pixels of the union of the two masks between the two frames. IOU = I / U;

[0070] If the intersection-over-union (IOU) between two frames is greater than the threshold, they are matched.

[0071] As a preferred solution, the timing filtering module is further used to obtain the matching results of each target in the result queue with the targets of the upper and lower frames in the inter-frame IOU module, and connect the inter-frame IOU matching results into a linked list. If the matching linked list of the target in the current frame can cover the entire result queue length, it is output as the final detection mask result.

[0072] As a preferred solution, the changed area detection module includes:

[0073] An input unit, used for inputting an image pair, a historical image t0 and a real-time image t1;

[0074] A feature extraction unit is used to extract features from the historical image t0 and the real-time image t1 to obtain a corresponding FPN historical feature map f0 and a FPN real-time feature map f1;

[0075] A feature fusion unit, used for fusing features of the FPN historical feature map f0 and the FPN real-time feature map f1 using a fusion feature module to obtain a fusion feature map;

[0076] The segmentation unit is used to obtain the changed segmentation result, i.e., the changed region mask, by using the segmentation network to fuse the feature map.

[0077] In a third aspect, the present invention further discloses a storage medium, which includes a computer program. When the computer program is executed by one or more processors, any of the above-mentioned obstacle detection methods is implemented.

[0078] The present invention provides an obstacle detection method, device and storage medium for an autonomous driving vehicle, which have the following beneficial effects:

[0079] The present invention focuses on the long-tail problem of obstacle detection technology. It uses computer vision technology, based on scene change detection, and integrates positioning sensors to realize open-set obstacle detection on a fixed driving route of an autonomous driving vehicle, providing effective reference information for the planning and control of autonomous driving. BRIEF DESCRIPTION OF THE DRAWINGS

[0080] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0081] Figure 1 A flowchart of an obstacle detection method provided by an embodiment of the present invention.

[0082] Figure 2 A detection block diagram of an obstacle detection method provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0083] The preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0084] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0085] Using ordinal numbers “first,” “second,” “third,” etc. to describe common objects merely indicates that different instances of similar objects are involved and is not intended to imply that the objects so described must have a given order in time, space, order, or in any other manner.

[0086] In addition, the expression of “comprising” an element is an “open” expression, which merely means that corresponding components or steps exist, and should not be interpreted as excluding additional components or steps.

[0087] In order to achieve the purpose of the present invention, some embodiments of the obstacle detection method for an autonomous driving vehicle include: Figure 1-2 As shown, the obstacle detection method includes:

[0088] Step S1: Build and update the historical image library of the autonomous driving vehicle;

[0089] The data in the historical image library includes: historical images and corresponding timestamps, GPS positioning information, IMU information, vehicle utm (Universal Transverse Mercator Projection) projection coordinates and vehicle yaw angle;

[0090] Step S2: When the autonomous driving vehicle is running, real-time data is obtained, which includes: real-time images and corresponding timestamps, GPS positioning information, IMU information, vehicle UTM projection coordinates and vehicle yaw angle;

[0091] Find similar historical images from the historical image library based on real-time data and combine them with the real-time images to form an image pair;

[0092] Step S3: Based on the difference detection model, detect the changed area of ​​the image at the current moment in the image pair and obtain a changed area mask;

[0093] Step S4: split the changed area mask by the connected domain and pass it into the result queue as the detection result of each frame;

[0094] Step S5: Match each mask connected domain in the result queue to achieve inter-frame IOU matching;

[0095] Step S6: Based on the inter-frame IOU matching result of step S5, perform time filtering and output the detection result.

[0096] The idea of ​​change detection is used to compare real-time data with data in the historical image library to realize open set obstacle prediction, effectively solve the problems mentioned in the background technology, and improve the accuracy of obstacle recognition.

[0097] In order to better understand the present invention, each step is described in detail in sequence below.

[0098] Step S1 includes:

[0099] Step S1.1: The autonomous driving vehicle is equipped with a camera, a GPS positioning system, and an IMU sensor, and drives along its fixed driving route, collecting camera images and corresponding timestamps, GPS positioning information, and IMU sensor information along the way, and saving them;

[0100] Step S1.2: Calculate the vehicle posture corresponding to each frame of the image through GPS positioning information and IMU sensor information to obtain the vehicle UTM projection coordinates and vehicle yaw angle;

[0101] Using the vehicle's UTM projection coordinates as an index, build an R-tree (Retangle Tree) data structure to store images, timestamps, and vehicle yaw angles;

[0102] Step S1.3: Screen each frame of image according to requirements;

[0103] Step S1.4: When updating the data, the data in the existing historical image library is mixed with the new data, and steps S1.2 to S1.3 are repeated.

[0104] The above step S1.3 further comprises:

[0105] Step S1.3.1: Draw a restricted neighborhood 1 with a range of 5m and a restricted neighborhood 2 with a range of 3m centered on the camera through the camera intrinsic parameter K, distortion sparse D and camera extrinsic parameters Rc2b and Tc2b;

[0106] Step S1.3.2: manually filter out images whose visual field is blocked within the restricted neighborhood one;

[0107] Step S1.3.3: Traverse all images after step S1.3.2 in reverse chronological order. For each frame of the image whose field of view is not blocked in restricted neighborhood 2, filter all data points in the surrounding restricted neighborhood 1 whose yaw angle difference with the vehicle is within 20° to obtain a sparse historical image library.

[0108] It is worth noting that step S1.3.2 can also use other deep neural networks with depth estimation functions to screen out images whose field of view is blocked within the restricted neighborhood, such as BEV network.

[0109] When the historical image library is first built, a dedicated image acquisition process is no longer required and subsequent vehicle operation data can be used directly.

[0110] Further, step S2 comprises:

[0111] Step S2.1: When the autonomous driving vehicle is running, the camera images and corresponding timestamps, GPS positioning information, and IMU sensor information are collected in real time along the way;

[0112] Step S2.2: Calculate the current vehicle posture information through GPS positioning information and IMU sensor information to obtain the vehicle UTM projection coordinates and vehicle yaw angle;

[0113] Step S2.3: Use the current vehicle utm projection coordinates to retrieve the nearest neighbor point in the R-tree data structure in the historical image library, and select the historical images whose yaw angle difference with the vehicle is within 20°, and combine them with the real-time image into an image pair.

[0114] Furthermore, step S3 uses a difference detection model to detect the changed region of the image at the current moment in the image pair, and obtains a changed region mask.

[0115] Step S3 specifically includes:

[0116] Step S3.1: Input image pair, historical image t0 and real-time image t1;

[0117] Step S3.2: extract features from the historical image t0 and the real-time image t1 to obtain the corresponding FPN historical feature map f0 and FPN real-time feature map f1;

[0118] The above FPN is Feature Pyramid Network (FPN);

[0119] Step S3.3: Use the Merge Features (MF) module to fuse the features of the FPN historical feature map f0 and the FPN real-time feature map f1 to obtain a fused feature map;

[0120] Specifically, starting from the top layer of the network, each layer of the network is upsampled, added to the features of the lower layer and activated by convolution, and then each layer of the network is sampled to the bottom feature size and merged to obtain a fused feature map;

[0121] Step S3.4: Use the segmentation network to obtain the changed segmentation result, that is, the changed area mask, using the fused feature map.

[0122] Furthermore, in step S4, based on the changed region mask obtained in step S3, the changed region mask is split by the connected domain and is passed into the result queue as the detection result of each frame, and the queue length is fixed to 5 frames.

[0123] Further, step S5 is:

[0124] Match each masked connected domain in the result queue and calculate the IOU (Intersection over Union) between it and all the results of the upper and lower frames.

[0125] The intersection I is the number of pixels of the intersection of the two masks between the two frames, and the union U is the number of pixels of the union of the two masks between the two frames. IOU = I / U;

[0126] If the intersection-over-union (IOU) between two frames is greater than the threshold of 0.1, they are matched.

[0127] Further, step S6 is:

[0128] Based on the matching results of each target in the result queue with the targets of the upper and lower frames obtained in step S5, the IOU matching results of alternate frames are connected into a linked list. If the matching linked list of the target in the current frame can cover the entire result queue length, it is output as the final detection mask result.

[0129] In some embodiments of an obstacle detection device for an autonomous driving vehicle, the obstacle detection device includes:

[0130] The historical image library building module is used to build and update the historical image library of autonomous driving vehicles;

[0131] The data in the historical image library includes: historical images and corresponding timestamps, GPS positioning information, IMU information, vehicle UTM projection coordinates and vehicle yaw angle;

[0132] The image pair combination module is used to obtain real-time data when the autonomous driving vehicle is running. The real-time data includes: real-time image and corresponding timestamp, GPS positioning information, IMU information, vehicle UTM projection coordinates and vehicle yaw angle;

[0133] Find similar historical images from the historical image library based on real-time data and combine them with the real-time images to form an image pair;

[0134] A change region detection module is used to detect the change region of the image at the current moment in the image pair based on a difference detection model, and obtain a change region mask;

[0135] The connected domain recognition module is used to split the change region mask by the connected domain and pass it into the result queue as the detection result of each frame;

[0136] The inter-frame IOU module is used to match each mask connected domain in the result queue to achieve inter-frame IOU matching;

[0137] The timing filtering module is used to perform timing filtering based on the IOU matching results of every other frame and output the detection results.

[0138] Furthermore, the historical image library building module includes:

[0139] The first acquisition unit is used for the autonomous driving vehicle equipped with a camera, a GPS positioning system, and an IMU sensor to travel along its fixed driving route, collect camera images and corresponding timestamps, GPS positioning information, and IMU sensor information along the way, and save them;

[0140] The first posture calculation unit is used to calculate the vehicle posture information corresponding to each frame of the image through GPS positioning information and IMU sensor information, and obtain the vehicle utm projection coordinates and vehicle yaw angle; using the vehicle utm projection coordinates as the index, an R-tree data structure is constructed to store the image, timestamp and vehicle yaw angle;

[0141] A screening unit, used for screening each frame of image according to requirements;

[0142] The data updating unit is used to mix the data in the existing historical image library with the new data when updating the data, and repeatedly execute the methods in the first pose calculation unit and the screening unit.

[0143] Further, the screening unit comprises:

[0144] A restricted neighborhood setting unit is used to set a restricted neighborhood 1 with a range of R1 and a restricted neighborhood 2 with a range of R2 with the camera as the center, where R2<R1;

[0145] A screening unit, used for screening out images whose visual field is blocked within a restricted neighborhood;

[0146] The filtering unit is used to traverse all images after being filtered out by the filtering unit in reverse chronological order, and for each frame of image whose field of view is not blocked in the restricted neighborhood two, filter all data points whose difference with the vehicle yaw angle within the threshold range within the surrounding restricted neighborhood one to obtain a sparse historical image library.

[0147] Furthermore, the image pair combination module includes:

[0148] The second acquisition unit is used to collect camera images and corresponding timestamps, GPS positioning information, and IMU sensor information in real time along the way when the autonomous driving vehicle is running;

[0149] The second posture calculation unit is used to calculate the current vehicle posture information through GPS positioning information and IMU sensor information to obtain the vehicle UTM projection coordinates and vehicle yaw angle;

[0150] The image pair combination unit is used to use the current vehicle utm projection coordinates to retrieve the nearest neighbor point in the R-tree data structure in the historical image library, and to screen the historical images whose yaw angle difference with the vehicle is within a threshold range, and combine them with the real-time image into a picture pair.

[0151] Furthermore, the inter-frame IOU module is further used to match each mask connected domain in the result queue and calculate the IOU between it and all the results of the upper and lower frames.

[0152] The intersection I is the number of pixels of the intersection of the two masks between the two frames, and the union U is the number of pixels of the union of the two masks between the two frames. IOU = I / U;

[0153] If the intersection-over-union (IOU) between two frames is greater than the threshold, they are matched.

[0154] Furthermore, the timing filtering module is further used to obtain the matching results of each target in the result queue with the targets of the upper and lower frames in the inter-frame IOU module, and connect the inter-frame IOU matching results into a linked list. If the matching linked list of the target in the current frame can cover the entire result queue length, it is output as the final detection mask result.

[0155] Furthermore, the changed area detection module includes:

[0156] An input unit, used for inputting an image pair, a historical image t0 and a real-time image t1;

[0157] A feature extraction unit is used to extract features from the historical image t0 and the real-time image t1 to obtain a corresponding FPN historical feature map f0 and a FPN real-time feature map f1;

[0158] A feature fusion unit, used for fusing features of the FPN historical feature map f0 and the FPN real-time feature map f1 using a fusion feature module to obtain a fusion feature map;

[0159] The segmentation unit is used to obtain the changed segmentation result, i.e., the changed region mask, by using the segmentation network to fuse the feature map.

[0160] In this embodiment, the specific content of the obstacle detection device of the autonomous driving vehicle is similar to the content of the obstacle detection method of the autonomous driving vehicle disclosed in the above embodiment, and will not be repeated here.

[0161] In some embodiments of the storage medium, the storage medium includes a computer program, and when the computer program is executed by one or more processors, the obstacle detection method disclosed in any of the above embodiments is implemented.

[0162] The present invention provides an obstacle detection method, device and storage medium for an autonomous driving vehicle, which have the following beneficial effects:

[0163] The present invention focuses on the long-tail problem of obstacle detection technology. It uses computer vision technology, based on scene change detection, and integrates positioning sensors to realize open-set obstacle detection on a fixed driving route of an autonomous driving vehicle, providing effective reference information for the planning and control of autonomous driving.

[0164] It should be understood that the various techniques described herein can be implemented in combination with hardware or software, or a combination thereof. Thus, the method and apparatus of the present invention, or some aspects or portions of the method and apparatus of the present invention, can take the form of program codes (i.e., instructions) embedded in a tangible medium, such as a floppy disk, a CD-ROM, a hard disk drive, or any other machine-readable storage medium, wherein when the program is loaded into a machine such as a computer and executed by the machine, the machine becomes a device for practicing the present invention.

[0165] The above shows and describes the basic principles and main features of the present invention and the advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The above embodiments and descriptions are only for illustrating the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, which shall fall within the scope of the present invention to be protected. The scope of protection of the present invention is defined by the attached claims and their equivalents.

Claims

1. An obstacle detection method for an autonomous driving vehicle, characterized in that: include: Step S1: Build and update the historical image library of the autonomous driving vehicle; The data in the historical image library includes: historical images and corresponding time and vehicle posture information; Step S2: when the autonomous driving vehicle is running, real-time data is obtained, and the real-time data includes: real-time images and corresponding time and vehicle posture information; Find similar historical images from the historical image library based on real-time data and combine them with the real-time images to form an image pair; The step S2 comprises: Step S2.1: When the autonomous driving vehicle is running, the camera image and the corresponding time and positioning sensor information are collected in real time along the way; Step S2.2: Calculate the current vehicle posture through the positioning sensor information to obtain the vehicle position coordinates and vehicle yaw angle; Step S2.3: Using the current vehicle position coordinates to retrieve neighborhood points in the historical image library, and screening out historical images whose yaw angle difference with the vehicle is within a threshold range, and combining them with the real-time image into an image pair; Step S3: Based on the difference detection model, detect the changed area of ​​the image at the current moment in the image pair and obtain a changed area mask; Step S4: split the changed area mask by the connected domain algorithm and transfer it into the result queue as the detection result of each frame; Step S5: Match each mask connected domain in the result queue to achieve inter-frame IOU matching; Step S6: Based on the inter-frame IOU matching result of step S5, perform time filtering and output the detection result.

2. The obstacle detection method according to claim 1, characterized in that: The step S1 comprises: Step S1.1: The autonomous driving vehicle is equipped with a camera and a positioning sensor, and drives along its fixed driving route, collecting camera images and corresponding time and positioning sensor information along the way, and saving them; Step S1.2: Calculate the vehicle posture information corresponding to each frame of image through the positioning sensor information to obtain the vehicle position coordinates and vehicle yaw angle; Step S1.3: Screen each frame of image according to requirements; Step S1.4: When updating the data, the data in the existing historical image library is mixed with the new data, and steps S1.2 to S1.3 are repeated.

3. The obstacle detection method according to claim 2, characterized in that: The step S1.3 comprises: Step S1.3.1: With the camera as the center, set the restricted neighborhood 1 with a range of R1, and set the restricted neighborhood 2 with a range of R2, R2<R1; Step S1.3.2: Screen out images whose visual field is blocked within the restricted neighborhood one; Step S1.3.3: Traverse all images after step S1.3.2 in reverse chronological order. For each frame of image whose field of view is not blocked in restricted neighborhood two, filter all data points whose difference with the vehicle's yaw angle in the surrounding restricted neighborhood one is within the threshold range to obtain a sparse historical image library.

4. The obstacle detection method according to claim 2, characterized in that: The step S1.2 is further as follows: The vehicle position and posture corresponding to each frame of the image are calculated through the positioning sensor information to obtain the vehicle position coordinates and vehicle yaw angle; Build a database to store images, corresponding time and vehicle posture information.

5. The obstacle detection method according to claim 1, characterized in that: The step S5 is further as follows: Match each masked connected domain in the result queue and calculate the IOU between it and all the results of the upper and lower frames. The intersection I is the number of pixels of the intersection of the two masks between the two frames, and the union U is the number of pixels of the union of the two masks between the two frames. IOU = I / U; If the intersection-over-union (IOU) between two frames is greater than the threshold, they are matched.

6. The obstacle detection method according to claim 5, characterized in that: The step S6 is further as follows: Based on the matching results of each target in the result queue with the targets of the upper and lower frames obtained in step S5, the IOU matching results of alternate frames are connected into a linked list. If the matching linked list of the target in the current frame can cover the entire result queue length, it is output as the final detection mask result.

7. The obstacle detection method according to any one of claims 1 to 6, characterized in that: The step S3 comprises: Step S3.1: Input image pair, historical image t0 and real-time image t1; Step S3.2: extract features from the historical image t0 and the real-time image t1 to obtain the corresponding FPN historical feature map f0 and FPN real-time feature map f1; Step S3.3: Use a fusion feature module to fuse the features of the FPN historical feature map f0 and the FPN real-time feature map f1 to obtain a fusion feature map; Step S3.4: Use the segmentation network to obtain the changed segmentation result, that is, the changed area mask, using the fused feature map.

8. An obstacle detection device for an autonomous driving vehicle, characterized in that: include: The historical image library building module is used to build and update the historical image library of autonomous driving vehicles; The data in the historical image library includes: historical images and corresponding time and vehicle posture information; An image pair combination module is used to obtain real-time data when the autonomous driving vehicle is running, and the real-time data includes: real-time images and corresponding time and vehicle posture information; Find similar historical images from the historical image library based on real-time data and combine them with the real-time images to form an image pair; The image pair combination module comprises: The second acquisition unit is used to collect camera images and corresponding time and positioning sensor information in real time along the way when the autonomous driving vehicle is running; The second posture calculation unit is used to calculate the current vehicle posture through the positioning sensor information to obtain the vehicle position coordinates and the vehicle yaw angle; An image pair combination unit is used to retrieve neighborhood points in the historical image library using the current vehicle position coordinates, and to select historical images whose difference with the vehicle yaw angle is within a threshold range, and to combine them with the real-time image into an image pair; A change region detection module is used to detect the change region of the image at the current moment in the image pair based on a difference detection model, and obtain a change region mask; The connected domain recognition module is used to split the changed region mask through the connected domain algorithm and pass it into the result queue as the detection result of each frame; The inter-frame IOU module is used to match each mask connected domain in the result queue to achieve inter-frame IOU matching; The timing filtering module is used to perform timing filtering based on the IOU matching results of every other frame and output the detection results.

9. The obstacle detection device according to claim 8, characterized in that: The historical image library building module includes: The first acquisition unit is used for the autonomous driving vehicle equipped with a camera and a positioning sensor to drive along its fixed driving route, collect camera images and corresponding time and positioning sensor information along the way, and save them; The first posture calculation unit is used to calculate the vehicle posture information corresponding to each frame of the image through the positioning sensor information to obtain the vehicle position coordinates and the vehicle yaw angle; A screening unit, used for screening each frame of image according to requirements; The data updating unit is used to mix the data in the existing historical image library with the new data when updating the data, and repeatedly execute the methods in the first pose calculation unit and the screening unit.

10. The obstacle detection device according to claim 9, characterized in that: The screening unit comprises: A restricted neighborhood setting unit is used to set a restricted neighborhood 1 with a range of R1 and a restricted neighborhood 2 with a range of R2 with the camera as the center, where R2<R1; A screening unit, used for screening out images whose visual field is blocked within a restricted neighborhood; The filtering unit is used to traverse all images after being filtered out by the filtering unit in reverse chronological order, and for each frame of image whose field of view is not blocked in the restricted neighborhood two, filter all data points whose difference with the vehicle yaw angle within the threshold range within the surrounding restricted neighborhood one to obtain a sparse historical image library.

11. The obstacle detection device according to claim 9, characterized in that: The first posture calculation unit is further used to calculate the vehicle posture corresponding to each frame of image through the positioning sensor information to obtain the vehicle position coordinates and the vehicle yaw angle; Build a database to store images, corresponding time and vehicle posture information.

12. The obstacle detection device according to claim 8, characterized in that: The inter-frame IOU module is further used to match each mask connected domain in the result queue and calculate the IOU between it and all the results of the upper and lower frames. The intersection I is the number of pixels of the intersection of the two masks between the two frames, and the union U is the number of pixels of the union of the two masks between the two frames. IOU = I / U; If the intersection-over-union (IOU) between two frames is greater than the threshold, they are matched.

13. The obstacle detection device according to claim 12, characterized in that: The timing filtering module is further used to obtain the matching results of each target in the result queue with the targets of the upper and lower frames in the inter-frame IOU module, connect the inter-frame IOU matching results into a linked list, and if the matching linked list of the target in the current frame can cover the entire result queue length, it is output as the final detection mask result.

14. The obstacle detection device according to any one of claims 8 to 13, characterized in that: The change area detection module comprises: An input unit, used for inputting an image pair, a historical image t0 and a real-time image t1; A feature extraction unit is used to extract features from the historical image t0 and the real-time image t1 to obtain a corresponding FPN historical feature map f0 and a FPN real-time feature map f1; A feature fusion unit, used for fusing features of the FPN historical feature map f0 and the FPN real-time feature map f1 using a fusion feature module to obtain a fusion feature map; The segmentation unit is used to obtain the changed segmentation result, i.e., the changed region mask, by using the segmentation network to fuse the feature map.

15. A storage medium, characterized in that The storage medium contains a computer program, and when the computer program is executed by one or more processors, the obstacle detection method according to any one of claims 1 to 7 is implemented.