Obstacle determination method and apparatus, electronic device, and storage medium

Abstract

This disclosure provides an obstacle determination method, apparatus, electronic device, and storage medium. This disclosure relates to the field of artificial intelligence technology, particularly to multi-sensor data fusion, obstacle detection and confirmation, and can be used in obstacle tracking and detection scenarios in autonomous driving. The specific implementation scheme is as follows: First probability values ​​and second probability values ​​corresponding to multiple first obstacles are obtained, the first obstacles being determined based on ultrasonic data; multiple second obstacles are obtained, the second obstacles being determined based on fisheye camera data; a target second obstacle is determined for each first obstacle based on the multiple second obstacles; the existence probability of each first obstacle is determined based on the intersection-union ratio (IUU) of each first obstacle and its corresponding target second obstacle, as well as the first and second probability values ​​corresponding to each first obstacle; first obstacles with an existence probability greater than a preset threshold are identified as target first obstacles.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, particularly to the fields of multi-sensor data fusion, obstacle detection and confirmation, and can be used in scenarios such as obstacle tracking and detection in autonomous driving. Specifically, it relates to an obstacle determination method, device, electronic device and storage medium. Background Technology

[0002] In autonomous driving, obstacle detection is mostly achieved using ultrasonic radar to determine the type of obstacle and its distance from the vehicle. However, ultrasonic radar is susceptible to interference, and the accuracy of obstacle detection still needs improvement. Summary of the Invention

[0003] This disclosure provides an obstacle determination method, apparatus, electronic device, and storage medium.

[0004] According to one aspect of this disclosure, an obstacle determination method is provided, the method comprising:

[0005] Obtain the first probability value and second probability value corresponding to each of the multiple first obstacles, wherein the first obstacles are determined based on ultrasonic data;

[0006] Multiple second obstacles are acquired, which are determined based on fisheye camera data;

[0007] A target second obstacle is determined for each first obstacle based on multiple second obstacles;

[0008] Based on the intersection-union ratio of each first obstacle and its corresponding target second obstacle, as well as the first probability value and the second probability value corresponding to each first obstacle, the existence probability of each first obstacle is determined;

[0009] The first obstacle whose existence probability is greater than a preset threshold is identified as the target first obstacle.

[0010] According to another aspect of this disclosure, an obstacle determination device is provided, the device comprising:

[0011] The first acquisition unit is used to acquire the first probability value and the second probability value corresponding to each of the multiple first obstacles, wherein the first obstacles are determined based on ultrasonic data;

[0012] The second acquisition unit is used to acquire multiple second obstacles, which are determined based on fisheye camera data;

[0013] The first determining unit is configured to determine a target second obstacle for each first obstacle based on multiple second obstacles;

[0014] The second determining unit is used to determine the existence probability of each first obstacle based on the intersection-union ratio of each first obstacle with its corresponding target second obstacle, and the first probability value and the second probability value corresponding to each first obstacle;

[0015] The third determining unit is used to determine the first obstacle with an existence probability greater than a preset threshold as the target first obstacle.

[0016] According to another aspect of this disclosure, an electronic device is provided, comprising:

[0017] At least one processor; and

[0018] The memory is communicatively connected to the at least one processor; wherein,

[0019] The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform any of the methods described in the present disclosure.

[0020] According to another aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions, wherein the computer instructions are used to cause the computer to perform any of the methods according to embodiments of this disclosure.

[0021] According to another aspect of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements any of the methods according to embodiments of this disclosure.

[0022] The solution disclosed herein can improve the accuracy and reliability of obstacle identification, thereby improving the safety of vehicle driving.

[0023] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0024] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0025] Figure 1 This is a schematic flowchart of an obstacle determination method according to an embodiment of the present disclosure;

[0026] Figure 2 This is an architecture diagram of an obstacle determination system according to an embodiment of the present disclosure;

[0027] Figure 3 This is a schematic diagram illustrating the relationship between obstacles, measured distance, and ellipse according to an embodiment of the present disclosure;

[0028] Figure 4 This is a schematic diagram of an obstacle and its echo according to an embodiment of the present disclosure;

[0029] Figure 5 This is a schematic diagram illustrating the inference, tracking, association, and updating of obstacles according to embodiments of this disclosure;

[0030] Figure 6 This is a schematic diagram of the obstacle determination device according to an embodiment of the present disclosure;

[0031] Figure 7 This is a scene illustration of the obstacle determination method according to an embodiment of the present disclosure:

[0032] Figure 8 This is a schematic diagram of the structure of an electronic device used to implement the obstacle determination method of the embodiments of this disclosure. Detailed Implementation

[0033] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0034] In this document, the term "and / or" merely describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent three cases: A alone, A and B simultaneously, and B alone. The term "at least one" in this document indicates any combination of at least two of a plurality of elements. For example, including at least one of A, B, and C can mean including any one or more elements selected from the set consisting of A, B, and C. The terms "first" and "second" in this document refer to and distinguish between multiple similar technical terms, not to restrict the order or to limit there to only two. For example, "first feature" and "second feature" refer to two categories / two features; the first feature can be one or more, and the second feature can also be one or more.

[0035] Furthermore, to better illustrate this disclosure, numerous specific details are set forth in the following detailed description. Those skilled in the art will understand that this disclosure can be practiced without certain specific details. In some instances, methods, means, components, and circuits well known to those skilled in the art have not been described in detail in order to highlight the main points of this disclosure.

[0036] Before introducing the technical solutions of the embodiments of this disclosure, the technical terms that may be used in this disclosure will be further explained:

[0037] Ultrasonic sensor: A sensor that measures distance by emitting and receiving ultrasonic waves, often used in obstacle detection and distance measurement in autonomous driving.

[0038] Fisheye camera: A camera with an ultra-wide field of view, whose lens design is similar to that of a fisheye, enabling it to capture a wider range of environmental information.

[0039] Intersection over Union (IoU): An index that evaluates the degree of overlap between two target regions. It is calculated as the ratio of the area of ​​intersection to the area of ​​union of the two regions.

[0040] Obstacle recognition: The process of extracting and recognizing information such as the position, size, and shape of obstacles based on data from target echo combinations.

[0041] In related technologies, traditional ultrasonic sensing systems suffer from drawbacks such as low accuracy, high false alarm rate, and inability to detect obstacles in certain areas. While fisheye cameras can provide rich semantic information, they cannot provide accurate depth information.

[0042] To at least partially address one or more of the aforementioned problems and other potential issues, this disclosure proposes an innovative obstacle determination method that utilizes ultrasonic technology for more accurate obstacle detection and tracking. Simultaneously, it efficiently fuses the ultrasonic detection results with those from a fisheye camera to fully leverage the fisheye camera's advantage in near-field vision, thereby achieving near-range blind spot compensation. This fusion strategy not only improves the accuracy of obstacle detection but also expands the perception capabilities of autonomous driving systems in complex environments, providing a more reliable guarantee for safe vehicle operation.

[0043] This disclosure provides a method for determining obstacles. Figure 1 This is a flowchart illustrating an obstacle determination method according to an embodiment of the present disclosure. This obstacle determination method can be applied to an obstacle determination device. The obstacle determination device is located in an electronic device. The electronic device includes, but is not limited to, fixed devices and / or mobile devices. For example, fixed devices include, but are not limited to, servers, which can be cloud servers or ordinary servers. Mobile devices include, but are not limited to, mobile phones, tablets, and vehicle-mounted terminals. In some possible implementations, the obstacle determination method can also be implemented by a processor calling computer-readable instructions stored in memory. Figure 1 As shown, the obstacle determination method includes:

[0044] S101. Obtain the first probability value and the second probability value corresponding to each of the multiple first obstacles, wherein the first obstacle is determined based on ultrasonic data;

[0045] S102. Obtain multiple second obstacles, which are determined based on fisheye camera data;

[0046] S103. Determine the target second obstacle for each first obstacle based on multiple second obstacles;

[0047] S104. Based on the intersection-union ratio of each first obstacle and its corresponding target second obstacle, and the first probability value and the second probability value corresponding to each first obstacle, determine the existence probability of each first obstacle;

[0048] S105. The first obstacle whose existence probability is greater than a preset threshold is identified as the target first obstacle.

[0049] In this embodiment of the disclosure, the ultrasonic data is raw data about the surrounding environment collected by the ultrasonic sensor, which typically includes information such as the distance to obstacles and the intensity of reflection.

[0050] In this embodiment of the disclosure, the first probability value can be the probability value of the existence of a first obstacle calculated based on a first given condition and sensor data; the second probability value can be the probability value of the existence of a first obstacle calculated based on a second given condition and sensor data. The first probability value and the second probability value can also be the probability of the obstacle's existence calculated based on the recognition result and a preset probability model.

[0051] In some implementations, an ultrasonic sensor is used to scan the surrounding environment, acquire ultrasonic data, and identify multiple first obstacles based on the ultrasonic data. The probability of the existence of a first obstacle is determined using a first given condition, resulting in a first probability value for the first obstacle; a second given condition is used to determine the probability of the existence of the first obstacle, resulting in a second probability value for the first obstacle. Here, the first given condition is a first set of rules or criteria used to determine the probability of the existence of the first obstacle, such as based on the sensor's raw data and feature extraction results. The second given condition is a second set of rules or criteria used to determine the probability of the existence of the first obstacle, such as based on image recognition results and feature analysis of image data. In practical applications, the first and second given conditions can be set or adjusted according to requirements.

[0052] In some implementations, a fisheye camera is used to capture images of the surrounding environment, and image processing algorithms are used to identify multiple secondary obstacles. Fisheye cameras, due to their wide field of view, are suitable for capturing close-up and wide-range environmental information.

[0053] In some implementations, for each first obstacle, a possible corresponding target second obstacle is selected from the second obstacles based on its position and size in space. This step can be implemented using spatial matching algorithms, such as nearest neighbor search or shape-based matching.

[0054] In some implementations, the Intersection over Union (IoU) between each first obstacle and its corresponding target second obstacle is calculated. IoU is an indicator of the degree of overlap between two obstacles; a higher IoU value indicates that the two obstacles are closer or overlap in space.

[0055] In some implementations, a weighted fusion algorithm is used to calculate the existence probability of each first obstacle by combining the first probability value, the second probability value, and the IoU value between the first obstacle and its corresponding target second obstacle. Here, the weighted fusion algorithm can assign different weights to each input factor based on the characteristics of different sensors (such as the distance accuracy of an ultrasonic sensor, the image resolution of a fisheye camera, etc.) and the confidence level of the IoU value.

[0056] In some implementations, a preset threshold is set, and obstacles with a probability of existence greater than this threshold are identified as target first obstacles. These target first obstacles are considered to be real obstacles, requiring the autonomous driving system to perform corresponding obstacle avoidance operations.

[0057] It should be noted that this disclosure does not restrict the execution order of S101 and S102. They can be executed simultaneously, and S102 can be executed before or after S101.

[0058] For example, suppose during autonomous driving, an ultrasonic sensor detects obstacle A 5 meters ahead and gives it a probability of existence of 0.8. Simultaneously, a fisheye camera also identifies obstacle B (possibly the same obstacle) ahead and gives it a probability of existence of 0.7. The IoU value between obstacle A and obstacle B is calculated to be 0.6 (indicating some spatial overlap). Based on a weighted fusion algorithm, the probability of obstacle A's existence is ultimately determined to be 0.85 (higher than the preset threshold of 0.8). Therefore, obstacle A is identified as the primary obstacle, and the autonomous driving system will take appropriate obstacle avoidance measures.

[0059] The technical solution of this disclosure, combining data from ultrasonic sensors and fisheye cameras, can more comprehensively capture information about the surrounding environment, reduce false alarms or missed alarms that may occur with a single sensor, improve the accuracy and reliability of obstacle recognition, and thus facilitate timely obstacle avoidance decisions, contributing to improved driving safety. In complex and ever-changing road conditions, by fusing data from multiple sensors, various challenges such as changes in lighting and obstructions can be addressed more effectively, enhancing the safety of autonomous driving.

[0060] In some embodiments, the obstacle determination method may further include: grouping ultrasonic data to form a plurality of echo combinations; determining a target echo combination from the plurality of echo combinations; and determining a plurality of first obstacles based on the target echo combination.

[0061] In this embodiment of the disclosure, the echo combination is a combination formed by grouping ultrasonic data according to specific rules, and each combination contains a set of related ultrasonic data.

[0062] In this embodiment of the disclosure, the target echo combination is the data combination most likely to correspond to the real obstacle selected from multiple echo combinations. The target echo combination typically contains the most stable and consistent intersection data, and therefore is most likely to correspond to the real obstacle.

[0063] In this embodiment of the disclosure, the first obstacle is an obstacle identified based on the combination of target echoes, which typically includes information such as its location, size, and shape.

[0064] In some implementations, the collected ultrasonic data is grouped according to specific rules to form multiple echo combinations. These grouping rules may be based on factors such as time, spatial location, and reflection intensity. For example, ultrasonic data from different directions within the same time period can be grouped together, or data with similar reflection intensities can be grouped together.

[0065] In some implementations, determining a target echo combination from multiple echo combinations includes: extracting key features from each echo combination, such as average distance and maximum reflection intensity; and determining the target echo combination from multiple echo combinations based on the extracted features and preset filtering conditions (such as distance range and reflection intensity threshold).

[0066] In some implementations, determining the first obstacle based on the target echo combination includes: analyzing the data in the target echo combination to identify information such as the location, size, and shape of the obstacle; and calculating the existence probability of each obstacle, such as a first probability value and a second probability value, based on the identification results and a preset probability model (such as a Bayesian network or a random forest).

[0067] For example, suppose that during autonomous driving, ultrasonic sensors collect a series of data about obstacles ahead. This data is grouped into three groups: near-range, medium-range, and far-range. Then, based on characteristics such as average distance and reflection intensity, target echo combinations are selected from these three groups. Finally, the data in the target echo combinations is analyzed, identifying a pedestrian obstacle 5 meters ahead and calculating its probability of existence as 0.9.

[0068] Thus, by grouping ultrasonic data, key information can be extracted and utilized more effectively, leading to more accurate obstacle identification. By filtering target echo combinations, false alarms and missed alarms caused by noise, interference, and other factors can be reduced. This method can adapt to changes in different environments and road conditions, improving the autonomous driving system's ability to identify and respond to obstacles and enhancing system robustness.

[0069] In some embodiments, determining a target echo combination from a plurality of echo combinations includes:

[0070] Determine the sum of the standard deviations of the intersection points for each echo combination in the multiple echo combinations;

[0071] The echo combination with the smallest sum of the standard deviations of the intersection points is determined as the target echo combination.

[0072] In this embodiment of the disclosure, the sum of the standard deviations of the intersection points refers to the sum of the standard deviations calculated for each intersection point of the echo combination and all the standard deviations added together. It reflects the dispersion of the intersection points, that is, the stability and consistency of the intersection points.

[0073] In some implementations, determining the sum of the standard deviations of the intersection points corresponding to each echo combination in a plurality of echo combinations includes: first, preprocessing the collected ultrasonic data, including noise reduction and filtering steps, to improve the accuracy and reliability of the data; for each echo combination, calculating the intersection points between each ultrasonic data point within it. These intersection points may represent the location of an obstacle in space; and for the intersection points of each echo combination, calculating the sum of their standard deviations.

[0074] In some implementations, determining the echo combination with the smallest sum of intersection standard deviations as the target echo combination includes: comparing the sum of the intersection standard deviations of multiple echo combinations and finding the smallest one. The echo combination with the smallest sum of intersection standard deviations is then determined as the target echo combination.

[0075] Suppose that during autonomous driving, ultrasonic sensors collect a series of data about obstacles ahead, forming three echo combinations. By calculating the sum of the standard deviations of the intersection points of each echo combination, it is found that the first echo combination has the smallest sum of standard deviations, and therefore it is identified as the target echo combination. Next, the data in the target echo combination is analyzed, identifying a vehicle obstacle 3 meters ahead, and calculating its probability of existence as 0.85.

[0076] Thus, by comparing the sum of the standard deviations of the intersection points, the most stable and consistent echo combination can be selected. From multiple echo combinations, the target echo combination most likely corresponding to the real obstacle can be determined, thereby improving the accuracy of obstacle recognition. Since the selection of the target echo combination is based on the stability and consistency of the data, false alarms and missed alarms caused by noise, interference, and other factors can be reduced.

[0077] In some embodiments, determining a target echo combination from a plurality of echo combinations includes:

[0078] Determine the number of intersection points for each echo combination in the multiple echo combinations;

[0079] The echo combination with the most intersection points is determined as the target echo combination.

[0080] In this embodiment of the disclosure, the total number of intersections formed between various data points in the ultrasonic data reflects the spatial distribution of the ultrasonic data and the number of potential obstacles.

[0081] In this embodiment of the disclosure, the target echo combination is the echo combination with the most intersection points selected from multiple echo combinations. It usually contains the most intersection point data and is therefore most likely to correspond to the real obstacle.

[0082] In some implementations, the collected ultrasonic data is preprocessed, including noise reduction and filtering, to improve the accuracy and reliability of the data. For each echo combination, the intersections between the individual ultrasonic data points within it are calculated. These intersections may represent the potential location of an obstacle in space. The total number of intersections in each echo combination is counted, i.e., the total number of intersections formed between all ultrasonic data points within that combination.

[0083] In some implementations, the echo combination with the most intersection points is determined as the target echo combination. This includes comparing the number of intersection points among multiple echo combinations and finding the one with the largest number of intersection points; the echo combination with the most intersection points is then determined as the target echo combination. This combination typically contains the most intersection point data and is therefore most likely to correspond to a real obstacle.

[0084] Suppose that during autonomous driving, ultrasonic sensors collect data about obstacles ahead, forming three echo combinations. By calculating the number of intersections in each echo combination, the second echo combination is found to have the most intersections and is therefore identified as the target echo combination. Next, analysis of the data in the target echo combination identifies a pedestrian obstacle 5 meters ahead, and its probability of existence is calculated to be 0.9. Based on the obstacle's probability and the autonomous driving system's safety strategy, the system formulates an appropriate obstacle avoidance decision, successfully avoiding a collision with the pedestrian.

[0085] Thus, by comparing the number of intersection points, the echo combination containing the most intersection point data can be selected, thereby improving the accuracy of obstacle recognition. Since the selection of target echo combinations is based on the number of intersection points, false alarms and missed alarms caused by noise, interference, and other factors can be reduced.

[0086] In some embodiments, in each echo combination, the intersection points obtained based on the current frame have a higher priority than the intersection points obtained based on historical frames.

[0087] In this embodiment of the disclosure, the current frame refers to the data frame acquired by the radar or ultrasonic sensor at the current moment. It reflects the latest environmental information.

[0088] In this embodiment of the disclosure, a historical frame refers to a data frame collected by a radar or ultrasonic sensor at a past moment. It provides background information on environmental changes.

[0089] In this embodiment of the disclosure, the intersection point is a point formed by the intersection of different data points (such as echo signals) in radar or ultrasonic data. It typically represents the potential location of an obstacle.

[0090] In this embodiment of the disclosure, priority is the processing order set according to factors such as the importance or timeliness of the data when processing multiple data. For example, the priority of the current frame intersection is higher than that of the historical frame intersection.

[0091] In some implementations, raw data from radar or ultrasonic sensors is acquired, including multiple echo signals from the current frame and historical frames. The raw data is preprocessed, such as denoising, filtering, and calibration, to improve data quality. For each echo combination, the intersection point in the current frame and historical frames is calculated separately. The intersection points are categorized into two types: current frame intersection points (calculated from current frame data) and historical frame intersection points (calculated from historical frame data).

[0092] In some implementations, current frame intersections are given higher priority because they reflect the latest environmental information, which is crucial for real-time obstacle avoidance and decision-making. Historical frame intersections are given lower priority; while they provide background information on environmental changes, they may not be as accurate or timely as the current frame data. Based on priority, current frame intersections are used first for obstacle identification and localization. Historical frame intersections are considered as a supplement only when the current frame intersections are insufficient to determine the obstacle's position or shape. Obstacle identification is performed based on the filtered and fused intersection data, including the extraction of information such as position, size, shape, and speed. Based on the identification results and the safety strategy of the autonomous driving system, corresponding obstacle avoidance decisions are made.

[0093] Suppose that during autonomous driving, radar sensors collect a series of data about obstacles ahead. At a certain moment, the system simultaneously receives multiple echo combinations from the current frame and historical frames. By calculating intersection points and assigning priorities, the system detects an intersection point in the current frame that closely matches the position of the vehicle ahead, and therefore identifies it as an obstacle. Subsequently, based on the obstacle's position and speed information, the system formulates an appropriate obstacle avoidance decision, successfully avoiding a collision with the vehicle ahead.

[0094] Thus, by prioritizing the use of intersection points from the current frame, the system can respond more quickly to environmental changes, improving the real-time performance of the autonomous driving system. Current frame data is typically more accurate and timely than historical frame data, thereby reducing false alarms and missed alarms and improving obstacle recognition accuracy. In complex and changing environments, the fusion of intersection points from the current and historical frames provides more comprehensive information, enhancing the robustness of the autonomous driving system.

[0095] In some embodiments, determining a plurality of first obstacles based on target echo combinations includes:

[0096] When there is an intersection in the target echo combination, the intersection is taken as the obstacle observation result, and the obstacle observation result is correlated with the first historical trajectory;

[0097] In the case where there is no intersection of the target echo combination, the single arc is associated with the second historical trajectory. After the second historical trajectory is continuously associated with the single arc a certain number of times, it will no longer be associated with the single arc.

[0098] The obstacles obtained from the target echo combination are filtered to identify multiple first obstacles.

[0099] In this embodiment of the disclosure, the first historical trajectory refers to the trajectory of an obstacle that has been previously identified and tracked.

[0100] In this embodiment of the disclosure, the second historical trajectory refers to the historical trajectory of a previously observed arc-shaped signal, which may be associated with a single-arc echo.

[0101] In this embodiment of the disclosure, filtering refers to the process of denoising and smoothing the observation results to improve data quality and recognition accuracy.

[0102] In some implementations, when intersections exist in the target echo sequence, these intersections are likely to represent the locations of actual obstacles. Therefore, these intersections are used as obstacle observations, and attempts are made to correlate them with the first historical trajectory.

[0103] In some implementations, all intersections are extracted from the target echo combination. The extracted intersections are then matched against a first historical trajectory (i.e., previously identified obstacle trajectories) to determine if these intersections belong to known obstacles. If an intersection matches a historical trajectory, the intersection is associated with that trajectory, and the trajectory information (such as position, velocity, etc.) is updated. If the match fails, it may be a newly encountered obstacle, requiring the initialization of a new trajectory.

[0104] In some implementations, when there are no intersections in the target echo combination, a single arc-shaped echo signal may be received. These signals may represent a moving obstacle, but are insufficient to determine its specific location. The process involves identifying single-arc echoes in the target echo combination; associating the identified single-arc echoes with a second historical trajectory (which may be a historical trajectory of similar arc-shaped signals previously observed); if a single-arc echo is continuously associated with a historical trajectory multiple times (e.g., similar arc-shaped signals are observed in several consecutive frames), then the single-arc echo is considered to indeed represent an obstacle. However, if the number of consecutive associations reaches a certain threshold and no new single-arc echoes are associated with that trajectory, it may mean that the obstacle has left the detection range or no longer generates arc-shaped signals, in which case the association with that trajectory should be terminated.

[0105] In some implementations, the obstacles obtained from the target echo combination are filtered to determine multiple first obstacles. This includes: filtering noise to remove false obstacle observations caused by sensor noise, environmental interference, etc.; smoothing the already associated obstacle trajectories to reduce trajectory fluctuations caused by measurement errors, etc.; and confirming the final list of first obstacles based on the filtered observation results and trajectory information. These first obstacles should be stable, well-defined, and physically consistent entities.

[0106] For example, suppose that during autonomous driving, radar sensors collect a series of data about obstacles ahead, forming a combination of target echoes. At one point, the system detects an intersection in the target echo combination that matches a previous historical trajectory (representing a moving vehicle), and therefore associates this intersection with that trajectory and updates the trajectory information. At another point, the system receives a single-arc echo signal that has been continuously associated with multiple frames of data from a previous historical trajectory (representing a turning vehicle), thus confirming that this single-arc echo represents a real obstacle. Finally, the system filters all observations to remove noise and false observations, obtaining the final first list of obstacles.

[0107] Thus, by correlating intersection points with historical trajectories and processing single-arc echoes, real obstacles can be identified more accurately, reducing false alarms and missed alarms. Filtering removes false observations and noise interference, reducing the amount of data and computational complexity in subsequent processing and improving resource utilization efficiency.

[0108] In some embodiments, obtaining the first probability value and the second probability value corresponding to each of the plurality of first obstacles includes:

[0109] The probability value is the sum of the standard deviations of the intersection points of all subgroups in the echo combination with the smallest sum of the standard deviations of the intersection points, after normalization by the mean.

[0110] The probability obtained by normalizing the sum of the standard deviations of the intersection points of all subgroups in the echo combination with the most intersection points is taken as the second probability value.

[0111] In this embodiment of the disclosure, the standard deviation of the intersection point is a statistic used to measure the dispersion of the intersection point distribution. The smaller the standard deviation, the more concentrated the intersection point distribution.

[0112] In this embodiment of the disclosure, mean normalization is the process of converting the original data into probability values ​​between 0 and 1, which is usually used for data standardization.

[0113] In this embodiment of the disclosure, the echo combination is a set of multiple subgroups, each containing intersection information about obstacles.

[0114] In this embodiment of the disclosure, a subgroup is the basic unit in the echo combination and may contain specific intersection information about obstacles.

[0115] In some implementations, data from multiple echo combinations are collected, each containing multiple subgroups with intersection information about obstacles. The standard deviation of the intersections for each subgroup is calculated, which is an important indicator for assessing the dispersion of the intersection distribution.

[0116] In some implementations, for each obstacle, the echo combination with the smallest sum of standard deviations of intersection points is found; the sum of standard deviations of intersection points of all subgroups in the echo combination is calculated and normalized to obtain the normalized probability value, i.e., the first probability value.

[0117] In some implementations, for each obstacle, the echo combination with the most intersection points is identified; the sum of the standard deviations of the intersection points of all subgroups in the echo combination is calculated and normalized to obtain the normalized probability value, i.e., the second probability value.

[0118] Here, the purpose of mean normalization is to convert the original data into probability values ​​between 0 and 1, which facilitates subsequent comparison and analysis.

[0119] Suppose there are three obstacles A, B, and C, each corresponding to one of three echo combinations:

[0120] Obstacle A: Echo combination 1 (subgroup 1: intersection standard deviation = 2, subgroup 2: intersection standard deviation = 3), Echo combination 2 (subgroup 1: intersection standard deviation = 1, subgroup 2: intersection standard deviation = 4).

[0121] Obstacle B: Echo combination 1 (subgroup 1: intersection standard deviation = 2, subgroup 2: intersection standard deviation = 2), echo combination 2 (subgroup 1: intersection standard deviation = 3, subgroup 2: intersection standard deviation = 3).

[0122] Obstacle C: Echo combination 1 (subgroup 1: intersection standard deviation = 1, subgroup 2: intersection standard deviation = 1, subgroup 3: intersection standard deviation = 1), echo combination 2 (subgroup 1: intersection standard deviation = 2, subgroup 2: intersection standard deviation = 2).

[0123] For obstacle A: The echo combination with the smallest sum of standard deviations of the intersection points is echo combination 2 (2+4=6), and the first probability value after mean normalization is 6 / (2+4+1+5)=0.6.

[0124] The echo combinations with the most intersection points are echo combination 1 and echo combination 2 (both have 2 intersection points), but echo combination 2 with the smaller sum of the standard deviations of the intersection points is selected for calculation. The second probability value after mean normalization is also 0.6 (note that here, to simplify the calculation, it is assumed that the number of intersection points is the same, so the combination with the smaller sum of the standard deviations is selected).

[0125] The same steps can be followed to calculate obstacles B and C.

[0126] Thus, by introducing a first probability value and a second probability value, the dispersion and number of intersection points can be comprehensively considered, thereby improving the accuracy of obstacle identification. Mean normalization simplifies the subsequent calculation process and improves computational efficiency.

[0127] In some embodiments, determining a target second obstacle for each first obstacle based on a plurality of second obstacles includes:

[0128] Convert the three-dimensional position of each first obstacle into a two-dimensional projected position;

[0129] Traverse the detection boxes of multiple second obstacles, and for each first obstacle, determine the second obstacle with the largest intersection-union ratio with its two-dimensional projection position as the target second obstacle.

[0130] In this embodiment of the disclosure, the three-dimensional position is the position information of the obstacle in three-dimensional space, which usually includes three coordinate values: X, Y, and Z.

[0131] In this embodiment of the disclosure, the two-dimensional projection position is the projection position information of the obstacle on the two-dimensional image, which usually includes two coordinate values, X and Y.

[0132] In this embodiment of the disclosure, the detection box is a rectangular box used to represent the position and size of an obstacle in a two-dimensional image.

[0133] In some implementations, converting a 3D position to a 2D projected position includes: acquiring the 3D position information of each first obstacle. Simultaneously, acquiring the camera's intrinsic and extrinsic parameters, as well as distortion parameters, are crucial for converting the 3D position to a 2D projected position. Methods such as perspective projection or orthographic projection are used to convert the 3D position of each first obstacle to a 2D projected position. Perspective projection is generally more suitable for simulating human visual effects, while orthographic projection is more suitable for scenarios requiring the preservation of shape and size invariance. During the conversion process, factors such as the camera's angle of view, focal length, and optical axis direction need to be considered to ensure the accuracy of the conversion result.

[0134] In some implementations, traversing the detection boxes and determining the target second obstacle includes: traversing the detection boxes of all second obstacles, and for each first obstacle, calculating the intersection-over-union (IoU) ratio between its two-dimensional projected position and the detection box of each second obstacle. For each first obstacle, the second obstacle with the largest IoU ratio with its two-dimensional projected position is selected as the target second obstacle. This means that the two-dimensional projected positions of the target second obstacle and the first obstacle have the highest degree of overlap in the two-dimensional image.

[0135] For example, suppose there are two first obstacles, A and B, and three second obstacles, C, D, and E. The position and bounding box information for each obstacle are as follows:

[0136] Obstacle A (3D position): (X1, Y1, Z1), 2D projected position: (x1, y1)

[0137] Obstacle B (3D position): (X2, Y2, Z2), 2D projected position: (x2, y2)

[0138] Obstacle C (detection box): (xcl, yc1, wc, hc)

[0139] Obstacle D (detection box): (xd1, ydl, wd, hd)

[0140] Obstacle E (detection box): (xe1, yel, we, he)

[0141] For obstacle A: Calculate its intersection-union ratios (IoU_AC, IoU_AD, and IoU_AE) with obstacles C, D, and E, respectively. Assuming IoU_AD is the largest, select obstacle D as the second target obstacle for obstacle A. The same steps can be followed for calculating and selecting obstacle B.

[0142] Thus, by calculating the intersection-over-union ratio (IoU) and selecting the second obstacle with the highest IoU as the target second obstacle, the accuracy of the matching results can be ensured. By traversing the detection boxes and calculating the IoU, the matching process is simplified and computational efficiency is improved.

[0143] In some embodiments, the existence probability of each first obstacle is determined based on the intersection-union ratio of each first obstacle to its corresponding target second obstacle, and the first probability value and the second probability value corresponding to each first obstacle, including:

[0144] A preset fusion algorithm, such as Dempster-Shafer evidence theory (also known as DST evidence theory or DS evidence theory), is used to fuse the intersection-union ratio of each first obstacle with its corresponding target second obstacle, as well as the first probability value and the second probability value corresponding to each first obstacle, to obtain the existence probability of each first obstacle.

[0145] In this embodiment of the disclosure, DS evidence theory is a reasoning method for handling uncertain information, which arrives at a final conclusion by constructing an identification framework, assigning basic probabilities, and synthesizing evidence.

[0146] In this embodiment of the disclosure, the Basic Probability Assignment (BPA) is a probability value assigned to each proposition in the identification framework in the DS evidence theory.

[0147] In this embodiment of the disclosure, the Dempster synthesis rule is a core rule in the DS evidence theory, used to synthesize information from multiple evidence sources.

[0148] In some implementations, an identification framework encompassing all possible answers (i.e., the existence states of the first obstacle) is constructed based on the specific needs of the problem. For example, the identification framework can be defined as {existent, non-existent}. A basic probability value regarding the existence of each first obstacle is assigned based on the intersection-union ratio (IU). For example, a higher IU results in a larger probability value assigned to the "existent" state. The first probability value and the second probability value (BPA) are assigned: the first probability value and the second probability value of each first obstacle are treated as two independent sources of evidence and assigned to the "existent" state in the identification framework. Using Dempster's synthesis rule, the IU, the first probability value, and the second probability value are synthesized to obtain a new basic probability assignment function. The synthesized basic probability assignment function is normalized to ensure that the sum of all probability values ​​is 1. From the synthesized basic probability assignment function, the probability value corresponding to the "existent" state for each first obstacle is extracted as the existence probability of that obstacle.

[0149] For example, suppose there are two first obstacles, A and B, and their corresponding target second obstacles have the following intersection-union ratio, first probability value, and second probability value as follows:

[0150] obstacle IoU First probability value Second probability value A 0.8 0.7 0.6 B 0.5 0.5 0.4

[0151] For obstacle A, the BPA for assigning the intersection-union ratio is {existence: 0.8, non-existence: 0.2}, the BPA for the first probability value is {existence: 0.7, non-existence: 0.3}, and the BPA for the second probability value is {existence: 0.6, non-existence: 0.4}. For obstacle B, the BPA for assigning the intersection-union ratio is {existence: 0.5, non-existence: 0.5}, the BPA for the first probability value is {existence: 0.5, non-existence: 0.5}, and the BPA for the second probability value is {existence: 0.4, non-existence: 0.6}. Using Dempster's composition rule, the three BPAs are combined to obtain a new basic probability assignment function. From the combined basic probability assignment function, the probability values ​​of obstacles A and B corresponding to the "existence" state are extracted as their existence probabilities. For example, the existence probability of obstacle A might be 0.9 (an assumed value; actual calculations involve more complex mathematical operations).

[0152] Thus, by fusing information from multiple evidence sources using Dempster's evidence theory, the accuracy of determining the existence of the first obstacle to the target is improved. Furthermore, evidence fusion using Dempster's synthesis rules simplifies the calculation process and improves computational efficiency.

[0153] This disclosure provides an architecture diagram of an obstacle determination system, such as... Figure 2As shown, the system includes an echo grouping submodule, a selection submodule, an obstacle tracking submodule, a post-processing submodule, and a fusion submodule. The echo grouping submodule is responsible for grouping ultrasonic data and removing illogical combinations. The selection submodule calculates all possible intersections and selects the optimal combination as the obstacle detection result. The obstacle tracking submodule tracks and filters obstacles to form stable obstacle trajectories. The post-processing submodule reduces the confidence of a single arc, for example, by disabling association of the arc after a certain number of associations. The fusion submodule fuses the data with the detection results from the fisheye camera.

[0154] In some embodiments, obstacles can be identified via S301 to S307.

[0155] S301: Traverse and calculate all echo intersections as candidate obstacles.

[0156] Figure 3 A schematic diagram showing the relationship between obstacles, measured distance, and ellipse is provided. Figure 3 In the diagram, T represents the transmitter, and R1 and R2 represent the receivers. The measured distance received by R1 is 11+1, and the measured distance received by R2 is 12+1. The obstacle is located on an ellipse with R1 and T as foci and 11+1 as the major axis, and also on an ellipse with R2 and T as foci and 12+1 as the major axis; that is, the obstacle is located at the intersection of these two ellipses.

[0157] S302: Traverse the echo data and group it.

[0158] Assuming the maximum number of obstacles to be detected is 2, all echoes are divided into two groups to ensure that echoes from the same probe are not in the same group.

[0159] Figure 4 A schematic diagram of the obstacle and its echo is shown, such as Figure 4 As shown, traversing through all possible combinations yields:

[0160] The first echo combination (1): subgroups 4-1 and 5-1, subgroups 4-2 and 5-2, subgroup 4-3;

[0161] The second echo combination (2): subgroups 4-1 and 5-2, subgroups 4-2 and 5-1, subgroup 4-3.

[0162] S303: Assign an identification (ID) to each subgroup.

[0163] When a subgroup is empty, its ID is set to -1;

[0164] When a subgroup has only one echo and cannot find its intersection with historical frames, set its ID to 3.

[0165] If there are pairs of echoes in a subgroup whose intersection points cannot be determined, set their ID to 1.

[0166] When the standard deviation of the intersection of all pairwise echoes in a subgroup is greater than a certain threshold, its ID is set to 2.

[0167] In other cases, its ID is set to 0, indicating that the subgroup can find valid obstacles.

[0168] Based on the above ID assignment method Figure 4 The IDs of the first echo combination (1) are 1, 1 and 3; the IDs of the second echo combination (2) are 1, 0 and 3.

[0169] S304: Select the best echo combination (i.e., target echo grouping).

[0170] First, based on the IDs assigned to the subgroups, some echo groups (denoted as groups) are deleted. Specifically, if any subgroup in a group has an ID of 1 or 2, the group is deleted; if all subgroups in a group have IDs of -1 or 3, the group is deleted.

[0171] Secondly, these groups are selected as the target echo groups in order of priority:

[0172] The group with the smallest sum of standard deviations of the intersection points of all subgroups; the sum of standard deviations is a_n, and after mean normalization, it is denoted as probability a_n_normalized.

[0173] The group with the most pairs of echo intersections among all subgroups; the sum of standard deviations is b_n, which is denoted as probability b_n_normalized after mean normalization.

[0174] S305: Track and process obstacles in the target echo group.

[0175] Figure 5 A schematic diagram is shown illustrating the inference, tracking, association, and updating of obstacles.

[0176] The prediction step includes: first, determining the position of the obstacle in the previous frame; since the obstacle is moving, its new position in the current frame needs to be predicted based on its direction and speed of movement; by considering the time difference and the obstacle's motion characteristics (such as speed, direction, etc.), the predicted position of the obstacle in the current frame is calculated.

[0177] The association step includes: grouping the echo data in the current frame and calculating the focal position of each group of echoes; matching the predicted position with each focal position to find the closest focal position; if a focal position is very close to the predicted position (i.e., within a certain error range), then the focal position is considered to be the same object as the obstacle in the previous frame, and the obstacle has been successfully tracked.

[0178] The update step includes: updating the position information of successfully tracked obstacles to the focus position in the current frame; and saving the updated position information as the starting point for inference in the next frame.

[0179] When new echo data is received, the steps of inference, tracking and updating are repeated to continuously track the movement trajectory of the obstacle.

[0180] The three steps of inference, tracking association, and updating constitute a complete tracking loop, enabling continuous tracking and position updates of moving obstacles. In practical applications, these steps may require the combination of specific algorithms and data processing techniques.

[0181] S306: After associating a single-arc echo a certain number of times, the single-arc echo will no longer be associated, in order to reduce the confidence level of the single-arc echo.

[0182] S307: The existence probability of ultrasonic targets and fisheye camera targets is fused using the DS evidence theory.

[0183] First, a two-dimensional (2D) projection of the ultrasonic target needs to be obtained. This can be achieved using the following formula:

[0184] O ultrasonic_2D =P ultrasonic ×O ultrasonic_3D ;

[0185] Among them, O ultrasonic_2D P represents the projection of the target detected by the ultrasonic sensor onto a 2D plane; ultrasonic The projection matrix represents how an ultrasonic sensor projects the position of a three-dimensional (3D) target onto a 2D plane; O ultrasonic_3D This represents the 3D position of the target detected by the ultrasonic sensor.

[0186] Thus, the projection matrix of the ultrasonic sensor can be used to convert the 3D target position into a projection on a 2D plane.

[0187] Next, target matching will be performed, including:

[0188] 1. Obtaining Fisheye Detected Targets: All 2D targets are detected using a fisheye camera. These targets are typically represented as bounding boxes (bboxes), denoted as O. fisheye .

[0189] 2. Calculate IoU: For each 2D target O detected by the fisheye, fisheye Calculate its relationship with the ultrasonic projection target O ultrasonic_2D The IoU between two target regions. IoU is an indicator that measures the degree of overlap between two target regions, and its value ranges from 0 to 1.

[0190] 3. Identify the matching target: Find the target O that matches the ultrasonic projection. ultrasonic _ 2D Fisheye-detected targets with the largest IoU are considered to be matched. If the largest IoU value is still 0 (i.e., no fisheye-detected target overlaps with the ultrasonic projection target), then the ultrasonic target is considered not to have matched any target in the current fisheye image, and its existence probability is 0.

[0191] 4. Processing the matching results: Based on the matching results, subsequent processing can be performed, such as updating the target status and fusing sensor information.

[0192] It should be noted that in practical applications, the matching process may contain certain errors due to factors such as sensor noise, target occlusion, and detection errors. Therefore, more complex algorithms or strategies can be used to improve the accuracy and robustness of the matching.

[0193] Furthermore, if the fusion of multiple ultrasonic sensors and multiple fisheye cameras needs to be considered, the above method needs to be further extended to handle the data correlation and fusion between multiple sensors.

[0194] In the previous steps, the target was detected using both the ultrasonic sensor and the fisheye camera, and the IoU between them and possible normalization parameters (such as a_n_normalized and b_n_normalized, which may represent normalization values ​​related to detection confidence or target attributes) were calculated. Now, DS evidence theory will be used to fuse this information to derive the final existence probability.

[0195] DS evidence theory is a mathematical framework for dealing with uncertain information. It allows information from multiple sources to be fused to arrive at a more accurate and reliable result. During the fusion process, each information source assigns a confidence level (or basic confidence assignment) to the events it deems possible.

[0196] For the application scenario, each information source (such as an ultrasonic sensor and a fisheye camera) can be regarded as a "piece of evidence" that provides credibility. Then, the fusion rule of DS evidence theory is used to calculate the final probability of existence.

[0197] The specific steps are as follows:

[0198] 1. Define the event set: First, define a set Θ containing all possible events. In the scenario, this set may include events such as "target exists" and "target does not exist".

[0199] 2. Calculate the basic confidence assignment: For each information source, calculate its basic confidence assignment for each event. For example, an ultrasonic sensor might assign confidence based on the signal strength it detects and the target distance; a fisheye camera might assign confidence based on the target bounding box size it detects and the IoU value. Furthermore, normalization parameters such as a_n_normalized and b_n_normalized can be taken into consideration.

[0200] 3. Applying the DS fusion rule: The fusion rule of the DS evidence theory is used to calculate the final existence probability. This rule can be expressed as:

[0201] (If and only if A = X)

[0202] However, this formula can be overly complex when applied directly, as it requires considering all possible combinations of events. In practice, simpler methods, such as the Dempster-Shafer combination rule, are typically used to progressively fuse information from multiple sources.

[0203] 4. Set a threshold and output the result: Set a threshold to determine if the final probability of existence is high enough to confirm the existence of the target. If m(A) is greater than or equal to this threshold, notify the perception module of the result; otherwise, filter out the detection as a false detection.

[0204] It is important to note that while the DS evidence theory provides a powerful method for fusing information from multiple sources, it still requires careful handling in practical applications. For example, it is necessary to ensure that the trust level allocation for each information source is reasonable, and the fusion rules and thresholds need to be carefully selected to avoid over-fusion or false positives. Furthermore, since the DS evidence theory can be computationally complex, optimization or approximation may be required for real-time applications.

[0205] Thus, by fusing ultrasonic waves and the 2D projection of the target from a fisheye camera using DS evidence theory, false alarms can be effectively filtered out, improving the accuracy of obstacle identification. Fusion of ultrasonic waves with camera perception for obstacle detection and tracking in autonomous driving can improve the accuracy and reliability of obstacle detection, reduce false alarm rates, and enhance anti-interference capabilities.

[0206] This disclosure provides an obstacle determination device, such as... Figure 6 As shown, the device may include: a first acquisition unit 601, used to acquire a first probability value and a second probability value corresponding to each of a plurality of first obstacles, wherein the first obstacles are determined based on ultrasonic data; a second acquisition unit 602, used to acquire a plurality of second obstacles, wherein the second obstacles are determined based on fisheye camera data; a first determination unit 603, used to determine a target second obstacle for each first obstacle based on the plurality of second obstacles; a second determination unit 604, used to determine the existence probability of each first obstacle based on the intersection-union ratio of each first obstacle and its corresponding target second obstacle, and the first probability value and the second probability value corresponding to each first obstacle; and a third determination unit 605, used to determine the first obstacles whose existence probability is greater than a preset threshold as target first obstacles.

[0207] In some embodiments, the device further includes:

[0208] Grouping unit, used to group ultrasonic data to form multiple echo combinations;

[0209] The fourth determining unit is used to determine the target echo combination from multiple echo combinations;

[0210] The fifth determining unit is used to determine multiple first obstacles based on the combination of target echoes.

[0211] In some embodiments, the fourth determining unit is configured to:

[0212] Determine the sum of the standard deviations of the intersection points for each echo combination in the multiple echo combinations;

[0213] The echo combination with the smallest sum of the standard deviations of the intersection points is determined as the target echo combination.

[0214] In some embodiments, the fourth determining unit is configured to:

[0215] Determine the number of intersection points for each echo combination in the multiple echo combinations;

[0216] The echo combination with the most intersection points is determined as the target echo combination.

[0217] In some embodiments, in each echo combination, the intersection points obtained based on the current frame have a higher priority than the intersection points obtained based on historical frames.

[0218] In some embodiments, the first determining unit 603 includes:

[0219] The first association subunit is used to take the intersection point as the obstacle observation result when there is an intersection point in the target echo combination, and to associate the obstacle observation result with the first historical trajectory.

[0220] The second association subunit is used to associate a single arc with the second historical trajectory when there is no intersection of the target echo combination. After the second historical trajectory is continuously associated with the single arc a certain number of times, it will no longer be associated with the single arc.

[0221] The filtering subunit is used to filter the obstacles obtained from the target echo combination in order to identify multiple first obstacles.

[0222] In some embodiments, the first acquisition unit 601 is specifically used for:

[0223] The probability of the sum of the standard deviations of the intersection points of all subgroups in the echo combination with the smallest sum of the standard deviations of the intersection points, after normalizing the sum of the standard deviations of the intersection points, is taken as the first probability value.

[0224] The probability of the sum of the standard deviations of the intersection points of all subgroups in the echo combination with the most intersection points, after normalization to the mean, is used as the second probability value.

[0225] In some embodiments, the first determining unit 603 includes:

[0226] The projection subunit is used to convert the three-dimensional position of each first obstacle into a two-dimensional projected position.

[0227] A sub-unit is determined to traverse the detection boxes of multiple second obstacles and identify the second obstacle with the maximum intersection-union ratio with its two-dimensional projection position as the target second obstacle for each first obstacle.

[0228] In some embodiments, the third determining unit 605 is configured to:

[0229] A preset fusion algorithm, such as DS evidence theory, is used to fuse the intersection-union ratio of each first obstacle with its corresponding target second obstacle, as well as the first probability value and the second probability value corresponding to each first obstacle, to obtain the existence probability of each first obstacle.

[0230] The specific functions and examples of each unit and subunit of the apparatus in this disclosure embodiment can be found in the relevant descriptions of the corresponding steps in the above method embodiments, and will not be repeated here.

[0231] The obstacle determination device of this disclosure can improve the accuracy and reliability of obstacle detection, thereby helping to improve the safety of vehicle driving.

[0232] This disclosure provides a scenario illustration of an obstacle determination method, such as... Figure 7 As shown.

[0233] As previously described, the obstacle determination method provided in this disclosure is applied to electronic devices. Electronic devices are intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. Electronic devices can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices.

[0234] Specifically, the electronic device may perform the following operations:

[0235] Obtain the first probability value and second probability value corresponding to each of the multiple first obstacles, wherein the first obstacles are determined based on ultrasonic data;

[0236] Multiple secondary obstacles are acquired, and these obstacles are determined based on fisheye camera data;

[0237] Based on the plurality of second obstacles, a target second obstacle is determined for each first obstacle;

[0238] Based on the intersection-union ratio of each first obstacle and its corresponding target second obstacle, as well as the first probability value and the second probability value corresponding to each first obstacle, the existence probability of each first obstacle is determined;

[0239] The first obstacle whose existence probability is greater than a preset threshold is identified as the target first obstacle.

[0240] It should be understood that Figure 7 The scene diagrams shown are merely illustrative and not restrictive; those skilled in the art can interpret them based on... Figure 7 Even with various obvious changes and / or substitutions to the examples, the resulting technical solutions still fall within the scope of this disclosure.

[0241] The acquisition, storage, and application of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0242] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0243] Figure 8A schematic block diagram of an example electronic device 800 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0244] like Figure 8 As shown, device 800 includes a computing unit 801, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 802 or a computer program loaded from storage unit 808 into random access memory (RAM) 803. RAM 803 may also store various programs and data required for the operation of device 800. The computing unit 801, ROM 802, and RAM 803 are interconnected via bus 804. Input / output (I / O) interface 805 is also connected to bus 804.

[0245] Multiple components in device 800 are connected to I / O interface 805, including: input unit 806, such as keyboard, mouse, etc.; output unit 807, such as various types of monitors, speakers, etc.; storage unit 808, such as disk, optical disk, etc.; and communication unit 809, such as network card, modem, wireless transceiver, etc. Communication unit 809 allows device 800 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0246] The computing unit 801 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, CPUs, graphics processing units (GPUs), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any suitable processors, controllers, microcontrollers, etc. The computing unit 801 performs the various methods and processes described above, such as obstacle determination methods. For example, in some embodiments, the obstacle determination method may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 808. In some embodiments, part or all of the computer program may be loaded and / or installed on device 800 via ROM 802 and / or communication unit 809. When the computer program is loaded into RAM 803 and executed by the computing unit 801, one or more steps of the obstacle determination method described above may be performed. Alternatively, in other embodiments, the computing unit 801 may be configured to perform an obstacle determination method by any other suitable means (e.g., by means of firmware).

[0247] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0248] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0249] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory, read-only memory, erasable programmable read-only memory (EPROM), flash memory, optical fiber, compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0250] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a cathode ray tube (CRT) or liquid crystal display (LCD) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0251] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0252] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0253] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0254] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A method for determining obstacles, comprising: The method involves obtaining a first probability value and a second probability value corresponding to each of multiple first obstacles, wherein the first obstacles are determined based on ultrasonic data; wherein obtaining the first probability value and the second probability value corresponding to each of the multiple first obstacles includes: taking the probability of the sum of the standard deviations of the intersection points of all subgroups in the echo combination with the smallest sum of the standard deviations of the intersection points as the first probability value; and taking the probability of the sum of the standard deviations of the intersection points of all subgroups in the echo combination with the most intersection points as the second probability value. Multiple second obstacles are acquired, which are determined based on fisheye camera data; Based on the plurality of second obstacles, a target second obstacle is determined for each first obstacle; Based on the intersection-union ratio of each first obstacle and its corresponding target second obstacle, as well as the first probability value and the second probability value corresponding to each first obstacle, the existence probability of each first obstacle is determined; The first obstacle whose existence probability is greater than a preset threshold is identified as the target first obstacle.

2. The method according to claim 1, further comprising: The ultrasonic data is grouped and processed to form multiple echo combinations; Determine the target echo combination from the plurality of echo combinations; The plurality of first obstacles are determined based on the target echo combination.

3. The method according to claim 2, wherein, Determining the target echo combination from the plurality of echo combinations includes: Determine the sum of the standard deviations of the intersection points for each of the plurality of echo combinations; The echo combination with the smallest sum of the standard deviations of the intersection points is determined as the target echo combination.

4. The method according to claim 2, wherein, Determining the target echo combination from the plurality of echo combinations includes: Determine the number of intersection points corresponding to each of the plurality of echo combinations; The echo combination with the most intersection points is determined as the target echo combination.

5. The method according to claim 3 or 4, wherein, In each echo combination, the intersection points obtained based on the current frame have a higher priority than the intersection points obtained based on historical frames.

6. The method according to claim 2, wherein, The determination of the plurality of first obstacles based on the target echo combination includes: If the target echo combinations have an intersection, the intersection is taken as the obstacle observation result, and the obstacle observation result is associated with the first historical trajectory; In the case where there is no intersection of the target echo combination, the single arc is associated with the second historical trajectory, wherein the second historical trajectory is no longer associated with the single arc after being continuously associated with the target a certain number of times; The obstacles obtained from the target echo combination are filtered to determine the plurality of first obstacles.

7. The method according to claim 1, wherein, The step of determining a target second obstacle for each first obstacle based on the plurality of second obstacles includes: Convert the three-dimensional position of each first obstacle into a two-dimensional projected position; The detection frames of the plurality of second obstacles are traversed, and the second obstacle with the largest intersection-union ratio with its two-dimensional projection position is determined as the target second obstacle for each first obstacle.

8. The method according to claim 1, wherein, The determination of the existence probability of each first obstacle based on the intersection-union ratio of each first obstacle and its corresponding target second obstacle, and the first probability value and second probability value corresponding to each first obstacle, includes: A preset fusion algorithm is used to fuse the intersection-union ratio of each first obstacle and its corresponding target second obstacle, as well as the first probability value and the second probability value corresponding to each first obstacle, to obtain the existence probability of each first obstacle.

9. An obstacle detection device, comprising: The first acquisition unit is used to acquire a first probability value and a second probability value corresponding to each of the multiple first obstacles, wherein the first obstacles are determined based on ultrasonic data; wherein the first acquisition unit is used to: take the probability of the sum of the standard deviations of the intersection points of all subgroups in the echo combination with the smallest sum of the standard deviations of the intersection points after normalizing the mean as the first probability value; and take the probability of the sum of the standard deviations of the intersection points of all subgroups in the echo combination with the most intersection points after normalizing the mean as the second probability value; The second acquisition unit is used to acquire multiple second obstacles, which are determined based on fisheye camera data; The first determining unit is configured to determine a target second obstacle for each of the plurality of second obstacles; The second determining unit is used to determine the existence probability of each first obstacle based on the intersection-union ratio of each first obstacle with its corresponding target second obstacle, and the first probability value and the second probability value corresponding to each first obstacle; The third determining unit is used to determine the first obstacle whose existence probability is greater than a preset threshold as the target first obstacle.

10. The apparatus according to claim 9, further comprising: A grouping unit is used to group the ultrasonic data to form multiple echo combinations; The fourth determining unit is used to determine the target echo combination from the plurality of echo combinations; The fifth determining unit is used to determine the plurality of first obstacles based on the target echo combination.

11. The apparatus according to claim 10, wherein, The fourth determining unit is used for: Determine the sum of the standard deviations of the intersection points for each of the plurality of echo combinations; The echo combination with the smallest sum of the standard deviations of the intersection points is determined as the target echo combination.

12. The apparatus according to claim 10, wherein, The fourth determining unit is used for: Determine the number of intersection points corresponding to each of the plurality of echo combinations; The echo combination with the most intersection points is determined as the target echo combination.

13. The apparatus according to claim 11 or 12, wherein, In each echo combination, the intersection points obtained based on the current frame have a higher priority than the intersection points obtained based on historical frames.

14. The apparatus according to claim 10, wherein, The first determining unit includes: The first association subunit is used to, when there is an intersection point in the target echo combination, take the intersection point as the obstacle observation result and associate the obstacle observation result with the first historical trajectory; The second association subunit is used to associate a single arc with a second historical trajectory when there is no intersection of the target echo combination. The second historical trajectory is no longer associated with the single arc after being continuously associated with the target a certain number of times. A filtering subunit is used to filter the obstacles obtained from the target echo combination in order to determine the plurality of first obstacles.

15. The apparatus according to claim 9, wherein, The first determining unit includes: The projection subunit is used to convert the three-dimensional position of each first obstacle into a two-dimensional projected position. A sub-unit is defined to traverse the detection boxes of the plurality of second obstacles and determine the second obstacle with the maximum intersection-union ratio with its two-dimensional projection position for each first obstacle as the target second obstacle.

16. The apparatus according to claim 9, wherein, The third determining unit is used for: A preset fusion algorithm is used to fuse the intersection-union ratio of each first obstacle and its corresponding target second obstacle, as well as the first probability value and the second probability value corresponding to each first obstacle, to obtain the existence probability of each first obstacle.

17. An electronic device comprising: At least one processor; as well as A memory that is communicatively connected to at least one processor; wherein, The memory stores instructions that can be executed by at least one processor to enable the at least one processor to perform the method of any one of claims 1-8.

18. A non-transitory computer-readable storage medium storing computer instructions, wherein, Computer instructions are used to cause a computer to perform the method according to any one of claims 1-8.

19. A computer program product comprising a computer program stored on a storage medium, wherein the computer program, when executed by a processor, implements the method according to any one of claims 1-8.

Images