Obstacle detection method, device, vehicle, storage medium and program product
By fusing camera and radar data to generate spatiotemporal features and combining them with map information for obstacle detection, the problem of high computational cost and poor real-time performance in existing technologies has been solved, achieving efficient and accurate obstacle detection and improving the safety of autonomous driving.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- BEIJING JINGDONG QIANSHITECHNOLOGY CO LTD
- Filing Date
- 2026-02-09
- Publication Date
- 2026-07-03
AI Technical Summary
In existing autonomous driving technologies, obstacle detection suffers from high computational costs, poor real-time performance, and low detection accuracy, which affects vehicle driving safety.
By fusing image data collected by camera devices and point cloud data collected by radar devices, spatiotemporal feature data and spatial location codes are generated. Feature extraction and classification models are used to determine the category and location of obstacles. Combined with map information, the results of obstacle detection are generated.
It reduces the computational power consumption of model inference operations, improves the response speed and accuracy of obstacle detection, reduces computing costs, avoids redundancy and missed detections in detection results, and improves vehicle driving safety.
Smart Images
Figure CN122336709A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of autonomous driving technology, and in particular to an obstacle detection method, device, vehicle, storage medium, and computer program product. Background Technology
[0002] With the continuous development of autonomous driving technology, autonomous vehicles are being widely used. In the field of autonomous driving technology, obstacle detection typically employs multi-sensor fusion technology. This involves acquiring various perceptual data, such as point clouds collected by LiDAR and high-definition images captured by cameras, and inputting this multi-sensory information into a deep learning model to detect obstacles. However, the large volume of point cloud data from LiDAR and image data from cameras results in significant computational costs for inference operations on deep learning models based on multi-source perceptual data. Furthermore, obstacle detection suffers from latency, slow response, poor real-time performance, and low accuracy, posing potential safety hazards to vehicle operation. Summary of the Invention
[0003] This disclosure provides an obstacle detection method, apparatus, vehicle, storage medium, and computer program product.
[0004] According to a first aspect of this disclosure, an obstacle detection method is provided, comprising: fusing image data acquired by a camera device and point cloud data acquired by a radar device to generate spatiotemporal feature data and a spatial location code corresponding to the spatiotemporal feature data; determining information of a first obstacle based on the spatiotemporal feature data and the spatial location code, wherein the first obstacle has a corresponding obstacle category; determining information of a second obstacle based on the spatiotemporal feature data, the spatial location code, map information, and driving path information, wherein the second obstacle includes a third obstacle that does not have a corresponding obstacle category and road surface facilities; and fusing the information of the first obstacle and the information of the second obstacle to obtain an obstacle detection result.
[0005] Optionally, determining the information of the first obstacle based on the spatiotemporal feature data and the spatial location code includes: generating bird's-eye view feature data based on the spatiotemporal feature data; and determining the information of the first obstacle based on the bird's-eye view feature data and the spatial location code.
[0006] Optionally, generating bird's-eye view feature data based on the spatiotemporal feature data includes: using a first feature extraction model to obtain first region spatiotemporal feature data corresponding to a preset region based on the spatiotemporal feature data; and performing pooling processing on the first region spatiotemporal feature data along a specified axis to generate the bird's-eye view feature data.
[0007] Optionally, determining the information of the first obstacle based on the bird's-eye view feature data and the spatial location code includes: using a first decoder and a query vector corresponding to the first obstacle to decode the bird's-eye view feature data and the spatial location code to obtain first decoded data; and using a first classification model to classify the first decoded data to obtain the information of the first obstacle.
[0008] Optionally, determining the information of the second obstacle based on the spatiotemporal feature data, the spatial location code, map information, and driving path information includes: using a second feature extraction model to obtain second region spatiotemporal feature data corresponding to a preset region based on the spatiotemporal feature data; using a second decoder and a query vector corresponding to the third obstacle to decode the second region spatiotemporal feature data and the spatial location code to obtain second decoded data; and using a second classification model to classify the second decoded data to obtain the information of the third obstacle.
[0009] Optionally, determining the information of the second obstacle based on the spatiotemporal feature data, the spatial location code, the map information, and the driving path information includes: encoding the map information to obtain a map code; encoding the driving path information to obtain a path code; decoding the spatiotemporal feature data of the second area, the spatial location code, the map code, and the path code using a third decoder and a query vector corresponding to the road surface facilities to obtain third decoded data; and classifying the third decoded data using a third classification model to obtain the information of the road surface facilities.
[0010] Optionally, the information of the first obstacle includes: confidence score, obstacle category, first 3D location range information, and status information; the obstacle category includes at least one type of pedestrian, vehicle, and traffic light; the information of the third obstacle includes: second 3D location range information, existence probability, and status information; the information of the road surface facility includes: facility type and location information of the road surface facility; the facility type includes at least one type of lane, boundary, stop line, and speed limit sign.
[0011] Optionally, the step of fusing the information of the first obstacle and the information of the second obstacle to obtain the obstacle detection result includes: determining a third obstacle contained within the spatial range of the first obstacle as an associated obstacle based on the first 3D location range information and the second 3D location range information; removing all associated obstacle information from the information of all third obstacles to obtain information of non-associated obstacles; and obtaining the obstacle detection result, wherein the obstacle detection result includes the information of the first obstacle, the information of the non-associated obstacles, and the information of the road surface facilities.
[0012] Optionally, the step of fusing image data acquired by the camera device and point cloud data acquired by the radar device to generate spatiotemporal feature data and spatial location codes corresponding to the spatiotemporal feature data includes: using a spatiotemporal feature generation model to obtain the spatiotemporal feature data and corresponding spatial location codes based on the image data and the point cloud data; wherein the image data includes the current image frame and multiple historical image frames; and the point cloud data includes the current point cloud frame and multiple historical point cloud frames.
[0013] According to a third aspect of this disclosure, an obstacle detection device is provided, comprising: a feature generation module, configured to fuse image data acquired by a camera device and point cloud data acquired by a radar device to generate spatiotemporal feature data and a spatial location code corresponding to the spatiotemporal feature data; a first obstacle determination module, configured to determine information of a first obstacle based on the spatiotemporal feature data and the spatial location code, wherein the first obstacle has a corresponding obstacle category; a second obstacle determination module, configured to determine information of a second obstacle based on the spatiotemporal feature data, the spatial location code, map information, and driving path information, wherein the second obstacle includes a third obstacle that does not have a corresponding obstacle category and road surface facilities; and a fusion processing module, configured to fuse the information of the first obstacle and the information of the second obstacle to obtain an obstacle detection result.
[0014] According to a third aspect of this disclosure, an obstacle detection device is provided, comprising: a memory; and a processor coupled to the memory, the processor being configured to perform the method described above based on instructions stored in the memory.
[0015] According to a fourth aspect of this disclosure, a vehicle is provided, including: the obstacle detection device as described above.
[0016] According to a fifth aspect of this disclosure, a computer-readable storage medium is provided that stores computer instructions, which, when executed by a processor, implement the steps of the method described above.
[0017] According to a sixth aspect of this disclosure, a computer program product is provided, comprising a computer program that, when executed by a processor, implements the steps of the method described above.
[0018] The obstacle detection method, apparatus, vehicle, storage medium, and computer program product disclosed herein can reduce the computational power consumption of model inference operations, lower computational costs, and provide a fast response to obstacle detection, improving real-time detection and obstacle recognition accuracy. By detecting a first obstacle with an obstacle category, a third obstacle without an obstacle category, and road surface facilities, and performing fusion processing, obstacle recognition accuracy can be improved, effectively avoiding redundancy and missed detections in the detection results, and improving vehicle driving safety. Attached Figure Description
[0019] The above and other objects, features, and advantages of this disclosure will become more apparent from the more detailed description of the embodiments thereof in conjunction with the accompanying drawings. The accompanying drawings are provided to further illustrate the embodiments of this disclosure and form part of the specification. They are used together with the embodiments of this disclosure to explain the disclosure and do not constitute a limitation thereof. The above and other objects and advantages of this disclosure will be further described below with reference to specific embodiments and the accompanying drawings. In the drawings, the same or corresponding technical features or components will be represented by the same or corresponding reference numerals.
[0020] Figure 1 The following is a flowchart illustrating some embodiments of the obstacle detection method according to the present disclosure;
[0021] Figure 2 This is a schematic diagram illustrating the generation of spatiotemporal feature data in some embodiments of the obstacle detection method according to this disclosure;
[0022] Figure 3 This is a schematic diagram of the process for determining a first obstacle in some embodiments of the obstacle detection method according to the present disclosure;
[0023] Figure 4 This is a schematic flowchart illustrating the determination of a first obstacle in some other embodiments of the obstacle detection method according to this disclosure;
[0024] Figure 5 This is a schematic diagram illustrating the generation of information about a first obstacle in some embodiments of the obstacle detection method according to the present disclosure;
[0025] Figure 6 This is a schematic diagram of the process for determining a third obstacle in some embodiments of the obstacle detection method according to this disclosure;
[0026] Figure 7This is a schematic flowchart illustrating the determination of information about road surface facilities in some embodiments of the obstacle detection method according to this disclosure;
[0027] Figure 8 This is a schematic diagram illustrating the generation of information about a second obstacle in some embodiments of the obstacle detection method according to this disclosure;
[0028] Figure 9 This is a schematic flowchart illustrating the process of obtaining fused obstacle detection results in some embodiments of the obstacle detection method according to this disclosure;
[0029] Figure 10A A schematic diagram of modules according to some embodiments of the obstacle detection device according to this disclosure; Figure 10B A schematic diagram of a first obstacle determination module in some embodiments of the obstacle detection apparatus according to the present disclosure; Figure 10C A schematic diagram of a second obstacle determination module in some embodiments of the obstacle detection apparatus according to the present disclosure;
[0030] Figure 11 This is a schematic diagram of modules according to some other embodiments of the obstacle detection device according to the present disclosure. Detailed Implementation
[0031] Exemplary embodiments of the present disclosure will be described below with reference to the accompanying drawings. For clarity and brevity, not all features of the embodiments are described in the specification. However, it should be understood that many implementation-specific settings must be made in carrying out the embodiments to achieve the developer's specific goals, such as complying with constraints related to the device and business, and these constraints may vary depending on the implementation. Furthermore, it should be understood that while development work can be very complex and time-consuming, such development work is merely a routine task for those skilled in the art who benefit from the present disclosure.
[0032] It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values of the components and steps set forth in these embodiments do not limit the scope of this disclosure.
[0033] Those skilled in the art will understand that the terms "first," "second," etc., in the embodiments of this disclosure are only used to distinguish different steps, devices, or modules, and do not represent any specific technical meaning, nor do they indicate a necessary logical order between them.
[0034] It should also be understood that in the embodiments disclosed herein, "a plurality of" may refer to two or more, and "at least one" may refer to one, two or more.
[0035] It should also be understood that any component, data or structure mentioned in the embodiments of this disclosure can generally be understood as one or more unless expressly defined or given to the contrary in the context.
[0036] Furthermore, the term "and / or" in this disclosure is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this disclosure generally indicates that the preceding and following related objects have an "or" relationship.
[0037] It should also be understood that the description of the various embodiments in this disclosure emphasizes the differences between the various embodiments, and the similarities or similarities can be referred to each other. For the sake of brevity, they will not be described in detail.
[0038] At the same time, it should be understood that, for ease of description, the dimensions of the various parts shown in the accompanying drawings are not drawn according to actual scale.
[0039] The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit this disclosure or its application or use.
[0040] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and equipment should be considered part of the specification.
[0041] It should be noted that similar labels and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be discussed further in subsequent figures.
[0042] Furthermore, to avoid obscuring this disclosure with unnecessary detail, only processing steps and / or apparatus structures closely related to at least the solutions according to this disclosure are shown in the accompanying drawings, while other details less relevant to this disclosure are omitted. It should also be noted that similar reference numerals and letters in the drawings indicate similar items, and therefore once an item is defined in one drawing, it need not be discussed again in subsequent drawings.
[0043] Figure 1 This is a schematic flowchart illustrating some embodiments of the obstacle detection method according to this disclosure. Figure 1 As shown, the obstacle detection method includes steps S101-S104.
[0044] Step S101: The image data collected by the camera device and the point cloud data collected by the radar device are fused to generate spatiotemporal feature data and spatial location codes corresponding to the spatiotemporal feature data.
[0045] Cameras and radar devices can be installed on vehicles such as driverless cars and unmanned delivery vehicles. The camera devices can be various types of cameras or webcams, and the radar devices can be lidar or similar devices.
[0046] Spatiotemporal feature data is 4D spatiotemporal feature data that integrates three-dimensional spatial and temporal information and contains spatiotemporal dynamic information. Spatiotemporal feature data can be high-dimensional feature vectors at the kilobyte level. Spatiotemporal feature data can characterize the state change patterns of objects or scenes in continuous spatiotemporal space and can support the perception and prediction of complex patterns such as motion trends and interactive behaviors.
[0047] The spatial location code corresponding to the spatiotemporal feature data can be a 4D world location code, which is used to characterize the three-dimensional spatial location information of an object or scene in a continuous spatiotemporal space in the world coordinate system; the spatial location code can be a high-dimensional location vector.
[0048] Step S102: Based on the spatiotemporal feature data and spatial location coding, determine the information of the first obstacle, which has a corresponding obstacle category.
[0049] The first obstacle is an obstacle with a corresponding obstacle category, that is, a whitelisted obstacle. The obstacle category can include at least one of the following: vehicles, pedestrians, cyclists, traffic lights, traffic signs, etc. Traffic lights can be red and green lights, etc. For example, the first obstacle (whitelisted obstacle) includes pedestrians, vehicles, traffic lights, etc.
[0050] The information about the first obstacle includes: confidence score, obstacle category, first 3D position range information, and status information. The first 3D position range information of the first obstacle can be the 3D bounding box parameters of the first obstacle and the three-dimensional coordinates of the center of the 3D bounding box. The 3D bounding box parameters can be the length, width, and height values corresponding to the first obstacle. Status information includes one or more of the following: whether it is moving, speed, direction of movement, and acceleration.
[0051] In subsequent navigation and other processing, the confidence score of the first obstacle can be compared with a score threshold. If the confidence score is greater than the score threshold, the first obstacle can be retained; if the confidence score is less than or equal to the score threshold, the first obstacle can be removed. Alternatively, the first obstacle can be tracked and its motion predicted based on its state information.
[0052] Step S103: Based on spatiotemporal feature data, spatial location coding, map information and driving path information, determine the information of the second obstacle, wherein the second obstacle includes a third obstacle that does not have a corresponding obstacle category and road surface facilities.
[0053] The third obstacle can be a general obstacle, which does not have a corresponding obstacle category. For example, general obstacles include road surface debris and traffic barriers. The information of the third obstacle includes its second 3D location range, probability of existence, and state information. The second 3D location range information of the third obstacle can be its 3D bounding box parameters and the three-dimensional coordinates of the bounding box center. The 3D bounding box parameters can be the numerical values of length, width, and height corresponding to the third obstacle. State information includes one or more of the following: whether it is moving, its speed, direction of movement, and acceleration.
[0054] In subsequent navigation and other processing, the probability of a third obstacle's existence can be compared with a probability threshold. If the probability is greater than the threshold, the third obstacle can be retained; if the probability is less than or equal to the threshold, it can be removed. Alternatively, third obstacles can be tracked and their motion predicted based on state information.
[0055] Road surface facilities include at least one type of feature drawn on the road surface, such as lanes, boundaries, stop lines, and speed limit signs. Information about road surface facilities includes the facility type and its location; the location information may include the 3D coordinates of the road surface facility in a world coordinate system.
[0056] Step S104: The information of the first obstacle and the information of the second obstacle are fused to obtain the obstacle detection result.
[0057] The obstacle detection method disclosed herein utilizes spatiotemporal feature data and corresponding spatial location codes for obstacle detection. The spatiotemporal feature data and spatial location codes are small in size and have spatiotemporal correlation, which reduces the computational power consumption of model inference operations, lowers computational costs, and provides a fast response time for obstacle detection, improving real-time performance and the accuracy of obstacle recognition, as well as the accuracy of recognizing obstacle positions and motion states. By detecting first obstacles with obstacle categories, third obstacles without obstacle categories, and road surface facilities, and performing fusion processing, the obstacle recognition accuracy can be improved, and redundancy and missed detections in the detection results can be effectively avoided, enhancing driving safety and making it applicable to complex scenarios.
[0058] In some embodiments, various methods can be used to generate spatiotemporal feature data and spatial location codes. For example, a spatiotemporal feature generation model can be used to obtain spatiotemporal feature data and corresponding spatial location codes based on image data and point cloud data. Image data may include the current image frame and multiple historical image frames; point cloud data may include the current point cloud frame and multiple historical point cloud frames.
[0059] For example, image data can consist of current image frames and multiple historical image frames acquired by one or more camera devices, while point cloud data can consist of current point cloud frames and multiple historical point cloud frames acquired by one or more lidar sensors. Preprocessing techniques can be applied to point cloud data, such as removing rain and fog noise and selecting the region of interest; similar preprocessing techniques can be applied to image data, such as distortion correction.
[0060] Image data and point cloud data can be time-stamped using various methods, and the image coordinate system and point cloud coordinate system can be aligned using a calibration matrix, enabling data from both sensors to be processed in the same coordinate system.
[0061] like Figure 2 As shown, the spatiotemporal feature generation model 201 can be a variety of models. For example, the spatiotemporal feature generation model 201 includes feature extraction models and feature encoding models, etc.; the feature extraction model can be a variety of trained neural network models, and the feature encoding model can be a variety of trained Transformer models, etc.
[0062] Image data and point cloud data are input into the spatiotemporal feature generation model 201 to obtain spatiotemporal feature data and corresponding spatial location codes output by the spatiotemporal feature generation model 201; both spatiotemporal feature data and spatial location codes can be vectors, and the spatiotemporal feature data is 4D spatiotemporal feature data.
[0063] The spatiotemporal feature generation model 201 can perform feature extraction and encoding operations on multi-source data to generate spatiotemporal feature data and spatial location codes. This can reduce the computational power consumption of subsequent model inference operations, lower the computational cost, improve the response to obstacle detection, enhance the real-time performance of detection, and improve the accuracy of obstacle recognition.
[0064] Information about the first obstacle can be determined using a variety of methods. Figure 3 This is a schematic diagram of the process for determining a first obstacle according to some embodiments of the obstacle detection method of this disclosure, such as... Figure 3 As shown:
[0065] Step S301: Generate bird's-eye view feature data based on spatiotemporal feature data.
[0066] Bird's-eye view (BEV) feature data is feature data that compresses spatiotemporal feature data into a two-dimensional plane (usually the XY plane) while retaining a predetermined height (Z-axis). Based on bird's-eye view feature data, the computational complexity of first obstacle recognition can be reduced while ensuring recognition accuracy.
[0067] Step S302: Determine the information of the first obstacle based on the bird's-eye view feature data and spatial location coding.
[0068] By combining bird's-eye view feature data with spatial location coding for first obstacle identification, global spatial information from the bird's-eye view perspective can be utilized, and spatial location coding can be used to determine the spatial information of obstacles in different spatiotemporal scenarios, thereby improving the accuracy of first obstacle classification and detection.
[0069] Figure 4 This is a schematic diagram of the process for determining a first obstacle according to some other embodiments of the obstacle detection method of this disclosure, such as... Figure 4 As shown:
[0070] Step S401: Using the first feature extraction model, obtain the first region spatiotemporal feature data corresponding to the preset region based on the spatiotemporal feature data.
[0071] The preset region can be a preset region of interest, etc.; by obtaining the first region spatiotemporal feature data of the preset region through the first feature extraction model, the interference of irrelevant data on the detection of the first obstacle can be reduced, and the detection efficiency and accuracy can be improved.
[0072] Step S402: Perform pooling processing on the spatiotemporal feature data of the first region along a specified axis to generate bird's-eye view feature data.
[0073] The specified axis can be the Z-axis, etc.; pooling processing along the Z-axis can be performed on the spatiotemporal feature data of the first region, which can retain the key features of obstacle height while compressing redundant information in the height dimension, thus improving the accuracy of detection.
[0074] Step S403: Using the first decoder and the query vector corresponding to the first obstacle, the bird's-eye view feature data and spatial location encoding are decoded to obtain the first decoded data.
[0075] Multiple methods can be used to determine the query vector corresponding to the first obstacle. The query vector can be used to extract relevant features of the first obstacle in a targeted manner, avoiding interference from irrelevant features and improving decoding efficiency and accuracy.
[0076] Step S404: Classify the first decoded data using the first classification model to obtain information about the first obstacle.
[0077] In some embodiments, such as Figure 5 As shown, the first feature extraction model 501 can be a variety of pre-trained neural network models. Spatiotemporal feature data is input into the first feature extraction model 501, and the first feature extraction model 501 outputs first region spatiotemporal feature data corresponding to a preset region; the preset region can be an area within 0-30 meters of the vehicle's perimeter, etc.
[0078] Z-axis pooling module 502 can be used for various trained neural network models for Z-axis pooling. Z-axis pooling is performed on the spatiotemporal feature data of the first region along the Z-axis, and max pooling is performed on the Z-axis (height) dimension of the spatiotemporal features of the first region to retain object height information (e.g., 2 meters) and generate bird's-eye view feature data.
[0079] The first decoder 503 can be a Transformer decoder or other decoders. The query vector corresponding to the first obstacle can be a variety of query vectors, such as query vectors corresponding to pedestrians, vehicles, traffic lights, etc.
[0080] The bird's-eye view feature data, spatial location encoding, and query vector corresponding to the first obstacle are input into the first decoder 503. The first decoder 503 uses a multi-head attention mechanism to match the query vector with the bird's-eye view feature data, spatial location encoding, etc., and performs decoding processing to output the first decoded data.
[0081] The first classification model 504 can be a multi-task MLP (Multi-Layer Perceptron) model or other models. The first classification model 504 performs classification, regression, and other processing on the first decoded data after decoding by the first decoder 503 to obtain information about the first obstacle. The first obstacle can be a whitelisted obstacle such as a pedestrian, vehicle, or traffic light. The information about the first obstacle includes confidence score, obstacle category, first 3D location range information, and state information.
[0082] For example, the first obstacle is a pedestrian, the pedestrian's confidence score is 0.88, the pedestrian's obstacle category is "pedestrian", the pedestrian's first 3D position range information is the 3D bounding box parameters "0.5m×0.5m×1.7m" and the three-dimensional coordinates of the center of the 3D bounding box, and the pedestrian's state information is a speed of 2.3m / s, moving eastward, and acceleration of 0, etc.
[0083] The first obstacle is a traffic light. The confidence score of the traffic light is 0.88. The obstacle category of the traffic light is "traffic light". The first 3D position range information of the traffic light is the 3D bounding box parameter "1m×2m×0.5m" and the three-dimensional coordinates of the center of the 3D bounding box. The status information of the traffic light is: the traffic light is facing east and the current light is green.
[0084] Information about third obstacles can be determined using a variety of methods. Figure 6 This is a schematic diagram of the process for determining a third obstacle according to some embodiments of the obstacle detection method of this disclosure, such as... Figure 6 As shown:
[0085] Step S601: Using the second feature extraction model, obtain the second region spatiotemporal feature data corresponding to the preset region based on the spatiotemporal feature data.
[0086] The preset region can be a preset region of interest, etc.; by obtaining the second spatiotemporal feature data of the preset region through the second feature extraction model, the interference of irrelevant data on the detection of the third obstacle can be reduced, and the detection efficiency and accuracy can be improved.
[0087] Step S602: Using the second decoder and the query vector corresponding to the third obstacle, the spatiotemporal feature data and spatial location encoding of the second region are decoded to obtain the second decoded data.
[0088] Multiple methods can be used to determine the query vector corresponding to the third obstacle. The query vector can be used to extract relevant features of the third obstacle in a targeted manner, avoiding interference from irrelevant features and improving decoding efficiency and accuracy.
[0089] Step S603: Use the second classification model to classify the second decoded data to obtain the third obstacle information.
[0090] Information about road surface facilities can be determined using a variety of methods. Figure 7 This is a schematic flowchart illustrating the process of determining information about road surface facilities according to some embodiments of the obstacle detection method of this disclosure, such as... Figure 7 As shown:
[0091] Step S701: Encode the map information to obtain the map code.
[0092] Step S702: Encode the driving route information to obtain the route code.
[0093] Step S703: Using the third decoder and the query vector corresponding to the path encoding, the spatiotemporal feature data, spatial location encoding, map encoding and path encoding of the second region are decoded to obtain the third decoded data.
[0094] Multiple methods can be used to determine the query vector corresponding to the path encoding. The query vector can be used to extract relevant features of road surface facilities in a targeted manner, avoid interference from irrelevant features, and improve decoding efficiency and accuracy.
[0095] Step S704: Use the third classification model to classify the third decoded data to obtain information about road surface facilities.
[0096] In some embodiments, such as Figure 8As shown, the second feature extraction model 801 can be a variety of pre-trained neural network models. Spatiotemporal feature data is input into the second feature extraction model 801, and the second feature extraction model 801 outputs second-region spatiotemporal feature data corresponding to a preset region. The preset region can be an area within 0-30 meters of the vehicle's perimeter, etc.
[0097] The second decoder 802 can be a Transformer decoder or other decoders. The query vector corresponding to the third obstacle can be of various types, such as query vectors corresponding to general obstacles. The second region's spatiotemporal feature data, spatial location encoding, and the query vector corresponding to the third obstacle are input into the second decoder 802. The second decoder 802 uses a multi-head attention mechanism to match the query vector with the second region's spatiotemporal feature data and spatial location encoding, performs decoding processing, and outputs the second decoded data.
[0098] The second classification model 803 can be a multi-task MLP (Multi-Layer Perceptron) model or other models. The second classification model 803 performs classification, regression, and other processing on the second decoded data from the second decoder 802 to obtain information about the third obstacle. The third obstacle can be a common obstacle such as road surface debris, and its information includes second 3D location range information, existence probability, and state information.
[0099] For example, the information of the third obstacle includes: the probability of its existence is 88%, the second 3D location range information is the 3D bounding box parameters as "0.1m×0.5m×1m", and the three-dimensional coordinates of the center of the 3D bounding box, and the state information as: stationary.
[0100] Map information can be high-precision map information, and may include elements such as lanes, boundaries, and stop lines drawn on the road surface. Driving route information includes the planned driving route for the user's current trip and empirically determined routes based on the user's historical driving data.
[0101] The first encoder 804 and the second encoder 805 can be various pre-trained neural network models. Map information is input into the first encoder 804 to obtain the map code output by the first encoder 804; the map code can be of various types. Driving route information is input into the second encoder 805 to obtain the path code output by the second encoder 805; the path code can be of various types.
[0102] The query model 806 can be any of the trained neural network models. The path code output by the second encoder 805 is input into the query model 806 to obtain the query vector output by the query model 806 corresponding to the path code. The query vector corresponding to the path code is the query vector corresponding to the road surface facilities.
[0103] The third decoder 807 can be a Transformer decoder or other decoders. The second region's spatiotemporal feature data, spatial location code, map code, and query vector corresponding to the path code are input into the third decoder 807. The third decoder 807 uses a multi-head attention mechanism to match and decode the query vector with the second region's spatiotemporal feature data, spatial location code, and map code, and outputs the third decoded data.
[0104] The third classification model 808 can be a multi-task MLP (Multi-Layer Perceptron) model or other models. It performs classification, regression, and other processing on the third decoded data after decoding by the third decoder 807 to obtain information about road surface facilities. This information includes the facility type and its location. The facility type includes at least one of the following: lane, boundary, stop line, and speed limit sign. For example, if the road surface facility is a speed limit sign, the information for the speed limit sign includes the facility type "speed limit sign" and its 3D coordinates.
[0105] Multiple methods can be used to obtain the fused obstacle detection results. Figure 9 This is a schematic flowchart illustrating the process of obtaining fused obstacle detection results in some embodiments of the obstacle detection method according to this disclosure, such as... Figure 9 As shown:
[0106] Step S901: Based on the first 3D position range information and the second 3D position range information, determine the third obstacle contained within the spatial range of the first obstacle, and use it as the associated obstacle.
[0107] Step S902: Remove all associated obstacle information from all third obstacle information to obtain non-associated obstacle information.
[0108] Step S903: Obtain obstacle detection results, which include information on the first obstacle, information on unrelated obstacles, and information on road surface facilities.
[0109] For example, the first obstacle is a truck, which is a whitelisted obstacle, and there is only one first obstacle in total. The confidence score of the first obstacle (truck) is 0.96, the obstacle category is "vehicle", and the state information is stationary; the first 3D position range information of the truck is the 3D bounding box parameters and the three-dimensional coordinates of the center of the 3D bounding box.
[0110] There are two obstacles: Obstacle 1 and Obstacle 2. Obstacle 1 has an existence probability of 0.88 and is stationary. Its second 3D position range information includes the 3D bounding box parameters corresponding to Obstacle 1 and the 3D coordinates of the bounding box center. Obstacle 2 has an existence probability of 0.78 and is stationary. Its second 3D position range information includes the 3D bounding box parameters corresponding to Obstacle 2 and the 3D coordinates of the bounding box center.
[0111] Based on the first 3D location range information in the information of the first obstacle (truck) and the second 3D location range information in the information of obstacle 1, if obstacle 1 is determined to be included within the spatial range of the first obstacle (truck), then obstacle 1 is determined to be an associated obstacle.
[0112] Based on the first 3D location range information in the information of the first obstacle (truck) and the second 3D location range information in the information of obstacle 2, it is determined that obstacle 2 is not included in the spatial range of the first obstacle (truck), and therefore obstacle 2 is determined to be an unrelated obstacle.
[0113] From the information of all third obstacles (obstacle 1 and obstacle 2), all information related to the obstacle (obstacle 1) is removed, and information about the unrelated obstacle (obstacle 2) is obtained. The obstacle detection results are then obtained, including information about the first obstacle (truck), unrelated obstacles, and road surface facilities. Autonomous vehicles can use these obstacle detection results to ensure driving safety.
[0114] By using the 3D location range information of the first and third obstacles, associated obstacles can be accurately identified, thereby removing redundant information and reducing data processing pressure. It can also ensure that the identification results of whitelisted obstacles are not covered by unknown obstacles, giving whitelisted obstacles a high priority and improving driving safety.
[0115] The obstacle detection method disclosed herein uses spatiotemporal feature data with spatiotemporal information for model inference calculations. The spatiotemporal feature data has a small data volume, which can reduce the consumption of computing power and lower the computational cost. It has a fast response time for obstacle detection, which improves real-time performance and detection accuracy. Furthermore, by integrating the detection results of general obstacles, whitelisted obstacles, and road surface facilities, the final detection result is determined, which can improve the obstacle recognition accuracy. This method solves the problems of high computing power consumption, increased computational cost, poor real-time performance, and low accuracy in related technologies.
[0116] In some embodiments, such as Figure 10A As shown, this disclosure provides an obstacle detection device, which includes a feature generation module 1001, a first obstacle determination module 1002, a second obstacle determination module 1003, and a fusion processing module 1004, etc.
[0117] The feature generation module 1001 fuses image data acquired by the camera device and point cloud data acquired by the radar device to generate spatiotemporal feature data and spatial location codes corresponding to the spatiotemporal feature data. The first obstacle determination module 1002 determines the information of the first obstacle based on the spatiotemporal feature data and spatial location codes, wherein the first obstacle has a corresponding obstacle category.
[0118] The second obstacle determination module 1003 determines the information of the second obstacle based on spatiotemporal feature data, spatial location encoding, map information, and driving path information. The second obstacle includes third obstacles that do not have a corresponding obstacle category and road surface facilities. The fusion processing module 1004 fuses the information of the first obstacle and the information of the second obstacle to obtain the obstacle detection result.
[0119] In some embodiments, the feature generation module 1001 uses a spatiotemporal feature generation model to obtain spatiotemporal feature data and corresponding spatial location codes based on image data and point cloud data; wherein, the image data includes the current image frame and multiple historical image frames; and the point cloud data includes the current point cloud frame and multiple historical point cloud frames.
[0120] like Figure 10B As shown, the first obstacle determination module 1002 includes a bird's-eye view feature generation module 10021 and a first determination module 10022. The bird's-eye view feature generation module 10021 generates bird's-eye view feature data based on spatiotemporal feature data; the first determination module 10022 determines the information of the first obstacle based on the bird's-eye view feature data and spatial location encoding.
[0121] For example, the bird's-eye view feature generation module 10021 uses a first feature extraction model to obtain first region spatiotemporal feature data corresponding to a preset region based on spatiotemporal feature data; the bird's-eye view feature generation module 10021 performs pooling processing on the first region spatiotemporal feature data along a specified axis to generate bird's-eye view feature data.
[0122] The first determining module 10022 uses the first decoder and the query vector corresponding to the first obstacle to decode the bird's-eye view feature data and spatial location encoding to obtain the first decoded data; the first determining module 10022 uses the first classification model to classify the first decoded data to obtain the information of the first obstacle.
[0123] like Figure 10C As shown, the second obstacle determination module 1003 includes a second determination module 10031 and a third determination module 10032. The second determination module 10031 uses a second feature extraction model to obtain second region spatiotemporal feature data corresponding to a preset region based on spatiotemporal feature data. The second determination module 10031 uses a second decoder and a query vector corresponding to the third obstacle to decode the second region spatiotemporal feature data and spatial location encoding to obtain second decoded data. The second determination module 10031 uses a second classification model to classify the second decoded data to obtain information about the third obstacle.
[0124] The third determining module 10032 encodes the map information to obtain a map code; the third determining module 10032 encodes the driving route information to obtain a route code; the third determining module 10032 uses a third decoder and a query vector corresponding to the route code to decode the second region's spatiotemporal feature data, spatial location code, map code, and route code to obtain third decoded data; the third determining module 10032 uses a third classification model to classify the third decoded data to obtain information about road surface facilities.
[0125] In some embodiments, the fusion processing module 1004 determines a third obstacle contained within the spatial range of the first obstacle as an associated obstacle based on the first 3D location range information and the second 3D location range information; the fusion processing module 1004 removes all associated obstacle information from the information of all third obstacles to obtain information of non-associated obstacles; the fusion processing module 1004 obtains obstacle detection results, wherein the obstacle detection results include information of the first obstacle, information of non-associated obstacles, and information of road surface facilities.
[0126] like Figure 11As shown, the obstacle detection device may include a memory 1101, a processor 1102, a communication interface 1103, and a bus 1104. The memory 1101 is used to store instructions, and the processor 1102 is coupled to the memory 1101. The processor 1102 is configured to execute the obstacle detection method described above based on the instructions stored in the memory 1101.
[0127] The memory 1101 can be a high-speed RAM, non-volatile memory, or a memory array. The memory 1101 may also be divided into blocks, and these blocks can be combined into virtual volumes according to certain rules. The processor 1102 can be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the obstacle detection method of this disclosure.
[0128] In some embodiments, a vehicle is provided according to this disclosure, including the obstacle detection device as described in any of the above embodiments; the vehicle includes an autonomous vehicle, an unmanned delivery vehicle, etc.
[0129] In some embodiments, this disclosure provides a computer-readable storage medium storing computer instructions that, when executed by a processor, implement the obstacle detection method as described in any of the foregoing embodiments.
[0130] Computer-readable storage media may take the form of any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may, for example, include, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any combination thereof. More specific examples (not an exhaustive list) of readable storage media may include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.
[0131] Embodiments of this disclosure may also be computer program products, including computer program instructions that, when executed by a processor, cause the processor to perform the steps in the methods according to various embodiments of this disclosure described in the "Exemplary Methods" section above.
[0132] The obstacle detection method, device, vehicle, storage medium, and computer program product in the above embodiments can reduce computing power consumption and lower computing costs. They can respond quickly to obstacle detection, improve real-time performance, and improve detection accuracy. Furthermore, by integrating the detection results of general obstacles, whitelisted obstacles, and road surface facilities, the final detection result is determined, which improves vehicle driving safety and enhances the user experience.
[0133] The basic principles of this disclosure have been described above with reference to specific embodiments. However, it should be noted that the advantages, benefits, and effects mentioned in this disclosure are merely examples and not limitations, and should not be considered as essential features of each embodiment of this disclosure. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the scope of this disclosure to the necessity of employing the aforementioned specific details for implementation.
[0134] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For system embodiments, since they largely correspond to method embodiments, the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.
[0135] The block diagrams of devices, apparatuses, devices, and systems disclosed herein are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, devices, and systems can be connected, arranged, and configured in any manner. Words such as “comprising,” “including,” “having,” etc., are open-ended terms meaning “including but not limited to,” and are used interchangeably with them. The terms “or” and “and” as used herein refer to the terms “and / or,” and are used interchangeably with them unless the context clearly indicates otherwise. The term “such as” as used herein refers to the phrase “such as but not limited to,” and is used interchangeably with it.
[0136] It should also be noted that in the apparatus, devices, and methods of this disclosure, the components or steps can be disassembled and / or recombined. These disassemblies and / or recombinations should be considered as equivalent solutions to this disclosure.
[0137] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use this disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein can be applied to other aspects without departing from the scope of this disclosure. Therefore, this disclosure is not intended to be limited to the aspects shown herein, but rather to be carried out within the widest scope consistent with the principles and novel features disclosed herein.
[0138] The above description has been given for illustrative and descriptive purposes. Furthermore, this description is not intended to limit the embodiments of this disclosure to the forms disclosed herein. Although several exemplary aspects and embodiments have been discussed above, those skilled in the art will understand that the above embodiments are illustrative only and do not limit the scope of this disclosure. Those skilled in the art will understand that the above embodiments can be combined, modified, or replaced without departing from the scope and spirit of this disclosure.
Claims
1. An obstacle detection method, comprising: Image data acquired by a camera device and point cloud data acquired by a radar device are fused together to generate spatiotemporal feature data and spatial location codes corresponding to the spatiotemporal feature data. Based on the spatiotemporal feature data and the spatial location code, information about the first obstacle is determined, wherein the first obstacle has a corresponding obstacle category; Based on the spatiotemporal feature data, the spatial location code, map information, and driving path information, the information of the second obstacle is determined, wherein the second obstacle includes a third obstacle that does not have a corresponding obstacle category and road surface facilities; The information of the first obstacle and the information of the second obstacle are fused to obtain the obstacle detection result.
2. The method of claim 1, wherein, The step of determining the information of the first obstacle based on the spatiotemporal feature data and the spatial location code includes: Based on the spatiotemporal feature data, generate bird's-eye view feature data; Information about the first obstacle is determined based on the bird's-eye view feature data and the spatial location code.
3. The method of claim 2, wherein, The step of generating bird's-eye view feature data based on the spatiotemporal feature data includes: Using the first feature extraction model, spatiotemporal feature data of the first region corresponding to the preset region is obtained based on the spatiotemporal feature data; The spatiotemporal feature data of the first region is pooled along a specified axis to generate the bird's-eye view feature data.
4. The method of claim 2, wherein, The step of determining the information of the first obstacle based on the bird's-eye view feature data and the spatial location code includes: Using a first decoder and a query vector corresponding to the first obstacle, the bird's-eye view feature data and the spatial location encoding are decoded to obtain first decoded data; The first classification model is used to classify the first decoded data to obtain information about the first obstacle.
5. The method of claim 1, wherein, The information for determining the second obstacle based on the spatiotemporal feature data, the spatial location code, map information, and driving path information includes: Using the second feature extraction model, based on the spatiotemporal feature data, the spatiotemporal feature data of the second region corresponding to the preset region is obtained; Using the second decoder and the query vector corresponding to the third obstacle, the spatiotemporal feature data of the second region and the spatial location encoding are decoded to obtain the second decoded data. The second decoded data is classified using a second classification model to obtain the third obstacle information.
6. The method of claim 5, wherein, The information for determining the second obstacle based on the spatiotemporal feature data, the spatial location code, map information, and driving path information includes: The map information is encoded to obtain the map code; The driving route information is encoded to obtain a route code; Using a third decoder and a query vector corresponding to the road surface facilities, the spatiotemporal feature data of the second area, the spatial location code, the map code, and the path code are decoded to obtain the third decoded data; The third classification model is used to classify the third decoded data to obtain information about the road surface facilities.
7. The method of claim 1, wherein, The information of the first obstacle includes: confidence score, obstacle category, first 3D location range information, and status information; the obstacle category includes at least one type among pedestrians, vehicles, and traffic lights; The information of the third obstacle includes: second 3D location range information, existence probability, and status information; The information on the road surface facilities includes: facility type and location information of the road surface facilities; the facility type includes at least one of lanes, boundaries, stop lines, and speed limit signs.
8. The method of claim 7, wherein, The step of fusing the information of the first obstacle and the information of the second obstacle to obtain the obstacle detection result includes: Based on the first 3D location range information and the second 3D location range information, a third obstacle contained within the spatial range of the first obstacle is determined as an associated obstacle. Remove all associated obstacle information from all third obstacle information to obtain information on non-associated obstacles; Obtain the obstacle detection results, wherein the obstacle detection results include information about the first obstacle, information about the unrelated obstacles, and information about the road surface facilities.
9. The method of any one of claims 1 to 8, wherein, The process of fusing image data acquired by the camera device and point cloud data acquired by the radar device to generate spatiotemporal feature data and spatial location codes corresponding to the spatiotemporal feature data includes: Using a spatiotemporal feature generation model, the spatiotemporal feature data and the corresponding spatial location code are obtained based on the image data and the point cloud data; The image data includes the current image frame and multiple historical image frames; the point cloud data includes the current point cloud frame and multiple historical point cloud frames.
10. An obstacle detection device, comprising: The feature generation module is used to fuse image data acquired by the camera device and point cloud data acquired by the radar device to generate spatiotemporal feature data and spatial location codes corresponding to the spatiotemporal feature data. The first obstacle determination module is used to determine the information of the first obstacle based on the spatiotemporal feature data and the spatial location code, wherein the first obstacle has a corresponding obstacle category; The second obstacle determination module is used to determine the information of the second obstacle based on the spatiotemporal feature data, the spatial location code, map information and driving path information, wherein the second obstacle includes a third obstacle that does not have a corresponding obstacle category and road surface facilities; The fusion processing module is used to fuse the information of the first obstacle and the information of the second obstacle to obtain the obstacle detection result.
11. An obstacle detection device, comprising: Memory; And a processor coupled to the memory, the processor being configured to perform the method as described in any one of claims 1 to 9 based on instructions stored in the memory.
12. A vehicle comprising: The obstacle detection device as described in claim 10 or 11.
13. A computer-readable storage medium storing computer instructions that, when executed by a processor, implement the steps of the method as described in any one of claims 1 to 9.
14. A computer program product comprising a computer program which, when executed by a processor, implements the steps of the method of any one of claims 1 to 9.