Obstacle determination method and device, electronic equipment and storage medium

Abstract

The invention provides an obstacle determination method and device, electronic equipment and a storage medium. The invention relates to the technical field of artificial intelligence, in particular to the fields of multi-sensor data fusion, obstacle detection and confirmation and the like, and can be used for scenes such as obstacle tracking and detection in automatic driving. According to the specific implementation scheme, first probability values and second probability values corresponding to a plurality of first obstacles are obtained, wherein the first obstacles are determined based on ultrasonic data; acquiring a plurality of second obstacles, wherein the second obstacles are determined based on the fisheye camera data; determining a target second obstacle for each first obstacle based on a plurality of second obstacles; determining the existence probability of each first obstacle based on the intersection-to-union ratio of each first obstacle and the corresponding target second obstacle, and the first probability value and the second probability value corresponding to each first obstacle; and determining the first obstacle of which the existence probability is greater than a preset threshold value as a target first obstacle.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, and in particular to the fields of multi-sensor data fusion, obstacle detection and confirmation, and can be used in scenarios such as obstacle tracking and detection in autonomous driving, and specifically relates to an obstacle determination method, device, electronic device and storage medium. Background Art

[0002] In autonomous driving, ultrasonic radar is mostly used to detect obstacles, thereby determining the type of obstacle and the distance between the obstacle and the vehicle. However, ultrasonic radar is susceptible to interference, and the accuracy of obstacle detection still needs to be improved. Summary of the invention

[0003] The present disclosure provides an obstacle determination method, device, electronic device and storage medium.

[0004] According to one aspect of the present disclosure, there is provided a method for determining an obstacle, the method comprising:

[0005] Acquire a first probability value and a second probability value corresponding to each of a plurality of first obstacles, where the first obstacles are determined based on the ultrasonic data;

[0006] Acquire a plurality of second obstacles, where the second obstacles are determined based on the fisheye camera data;

[0007] determining a target second obstacle for each first obstacle based on the plurality of second obstacles;

[0008] Determine the existence probability of each first obstacle based on the intersection-and-joint ratio of each first obstacle and its corresponding target second obstacle, and the first probability value and the second probability value corresponding to each first obstacle;

[0009] A first obstacle whose existence probability is greater than a preset threshold is determined as a target first obstacle.

[0010] According to another aspect of the present disclosure, there is provided an obstacle determination device, the device comprising:

[0011] A first acquisition unit, configured to acquire a first probability value and a second probability value corresponding to each of a plurality of first obstacles, wherein the first obstacles are determined based on ultrasonic data;

[0012] A second acquisition unit, configured to acquire a plurality of second obstacles, wherein the second obstacles are determined based on the fisheye camera data;

[0013] a first determining unit, configured to determine a target second obstacle for each first obstacle based on the plurality of second obstacles;

[0014] a second determining unit, configured to determine the existence probability of each first obstacle based on an intersection-and-joint ratio of each first obstacle and its corresponding target second obstacle, and a first probability value and a second probability value corresponding to each first obstacle;

[0015] The third determining unit is configured to determine a first obstacle whose existence probability is greater than a preset threshold as a target first obstacle.

[0016] According to another aspect of the present disclosure, there is provided an electronic device, comprising:

[0017] at least one processor; and

[0018] a memory communicatively connected to the at least one processor; wherein,

[0019] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute any method in the embodiments of the present disclosure.

[0020] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute any method according to the embodiments of the present disclosure.

[0021] According to another aspect of the present disclosure, a computer program product is provided, including a computer program, which implements any method according to the embodiments of the present disclosure when executed by a processor.

[0022] The solution disclosed herein can improve the accuracy and reliability of obstacle determination, thereby improving the safety of vehicle driving.

[0023] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0025] Figure 1 is a flowchart of an obstacle determination method according to an embodiment of the present disclosure;

[0026] Figure 2 is an architectural diagram of an obstacle determination system according to an embodiment of the present disclosure;

[0027] Figure 3 is a schematic diagram of the relationship between obstacles, measurement distances and ellipses according to an embodiment of the present disclosure;

[0028] Figure 4 is a schematic diagram of an obstacle and its echo according to an embodiment of the present disclosure;

[0029] Figure 5 is a schematic diagram of inferring, tracking, associating and updating obstacles according to an embodiment of the present disclosure;

[0030] Figure 6 is a schematic structural diagram of an obstacle determination device according to an embodiment of the present disclosure;

[0031] Figure 7 : is a schematic diagram of a scenario of an obstacle determination method according to an embodiment of the present disclosure:

[0032] Figure 8 It is a schematic diagram of the structure of an electronic device used to implement the obstacle determination method of the embodiment of the present disclosure. DETAILED DESCRIPTION

[0033] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0034] The term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there may be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. The term "at least one" in this article means any combination of at least two of any one or more of a plurality of. For example, including at least one of A, B, and C can mean including any one or more elements selected from the set consisting of A, B, and C. The terms "first" and "second" in this article refer to multiple similar technical terms and distinguish them. They do not mean to limit the order or to limit them to only two. For example, the first feature and the second feature refer to two types / two features. The first feature can be one or more, and the second feature can also be one or more.

[0035] In addition, in order to better illustrate the present disclosure, numerous specific details are given in the following specific embodiments. It should be understood by those skilled in the art that the present disclosure can also be implemented without certain specific details. In some examples, methods, means, components and circuits well known to those skilled in the art are not described in detail in order to highlight the subject matter of the present disclosure.

[0036] Before introducing the technical solutions of the embodiments of the present disclosure, the technical terms that may be used in the present disclosure are further explained:

[0037] Ultrasonic sensor: A sensor that measures distance by emitting and receiving ultrasonic waves. It is commonly used for obstacle detection and distance measurement in autonomous driving.

[0038] Fisheye camera: A camera with an ultra-wide-angle field of view whose lens design is similar to that of a fish eye, capable of capturing a wider range of environmental information.

[0039] Intersection over Union (IoU): An indicator for evaluating the degree of overlap between two target regions. It is calculated as the ratio of the intersection area to the union area of ​​the two regions.

[0040] Obstacle identification: The process of extracting and identifying the location, size, shape and other information of obstacles based on the data in the target echo combination.

[0041] In the related technologies, the traditional ultrasonic sensing system has the defects of low precision, high false alarm rate, and inability to detect obstacles in certain areas. Although fisheye cameras can provide rich semantic information, they cannot give accurate depth of field positions.

[0042] In order to at least partially solve one or more of the above-mentioned problems and other potential problems, the present disclosure proposes an innovative obstacle determination method, which can use ultrasonic technology to achieve more accurate obstacle detection and tracking. At the same time, the ultrasonic detection results are efficiently fused with the detection results of the fisheye camera to give full play to the advantages of the fisheye camera in the close-range field of view, thereby achieving the effect of blind spot filling at close range. This fusion strategy not only improves the accuracy of obstacle detection, but also expands the perception ability of the autonomous driving system in complex environments, providing a more reliable guarantee for the safe driving of the vehicle.

[0043] The present disclosure provides a method for determining an obstacle. Figure 1 : is a flow chart of an obstacle determination method according to an embodiment of the present disclosure, and the obstacle determination method can be applied to an obstacle determination device. The obstacle determination device is located in an electronic device. The electronic device includes but is not limited to a fixed device and / or a mobile device. For example, a fixed device includes but is not limited to a server, and the server can be a cloud server or an ordinary server. For example, a mobile device includes but is not limited to: a mobile phone, a tablet computer, and a vehicle-mounted terminal. In some possible implementations, the obstacle determination method can also be implemented by a processor calling a computer-readable instruction stored in a memory. For example Figure 1 As shown, the obstacle determination method includes:

[0044] S101, obtaining a first probability value and a second probability value corresponding to each of a plurality of first obstacles, where the first obstacles are determined based on ultrasonic data;

[0045] S102, acquiring a plurality of second obstacles, where the second obstacles are determined based on fisheye camera data;

[0046] S103, determining a target second obstacle for each first obstacle based on the plurality of second obstacles;

[0047] S104, determining the existence probability of each first obstacle based on an intersection-and-joint ratio between each first obstacle and its corresponding target second obstacle, and a first probability value and a second probability value corresponding to each first obstacle;

[0048] S105: Determine a first obstacle whose existence probability is greater than a preset threshold as a target first obstacle.

[0049] In the embodiments of the present disclosure, ultrasonic data is raw data about the surrounding environment collected by an ultrasonic sensor, and generally includes information such as the distance to obstacles and reflection intensity.

[0050] In the disclosed embodiment, the first probability value may be a probability value of the existence of the first obstacle calculated based on the first given condition and the sensor data; the second probability value may be a probability value of the existence of the first obstacle calculated based on the second given condition and the sensor data. The first probability value and the second probability value may also be the probability of the existence of the obstacle calculated based on the recognition result and the preset probability model.

[0051] In some embodiments, an ultrasonic sensor is used to scan the surrounding environment, obtain ultrasonic data, and determine multiple first obstacles based on the ultrasonic data. The possibility of the existence of the first obstacle is determined using the first given condition to obtain a first probability value corresponding to the first obstacle; the possibility of the existence of the first obstacle is determined using the second given condition to obtain a second probability value of the first obstacle. Here, the first given condition is a first set of rules or standards for determining the possibility of the existence of the first obstacle, such as based on the original data and feature extraction results of the sensor. The second given condition is a second set of rules or standards for determining the possibility of the existence of the first obstacle, such as based on image recognition results and feature analysis of image data. In actual applications, the first given condition and the second given condition can be set or adjusted according to needs.

[0052] In some embodiments, a fisheye camera is used to capture images of the surrounding environment, and multiple second obstacles are identified through an image processing algorithm. The fisheye camera is suitable for capturing close-range and wide-range environmental information due to its wide-angle field of view.

[0053] In some implementations, for each first obstacle, based on its position and size in space, a target second obstacle that may correspond to it is screened out from the second obstacles. This step can be implemented by a spatial matching algorithm, such as nearest neighbor search or shape-based matching.

[0054] In some implementations, the IoU between each first obstacle and its corresponding target second obstacle is calculated. IoU is an indicator for evaluating the degree of overlap between two obstacles, and a higher value indicates that the two obstacles are closer or overlapped in space.

[0055] In some embodiments, the existence probability of each first obstacle is calculated using a weighted fusion algorithm in combination with the first probability value and the second probability value corresponding to each first obstacle, and the IoU value between the first obstacle and its corresponding target second obstacle. Here, the weighted fusion algorithm can assign different weights to each input factor according to the characteristics of different sensors (such as the distance accuracy of the ultrasonic sensor, the image resolution of the fisheye camera, etc.) and the confidence of the IoU value.

[0056] In some embodiments, a preset threshold is set, and the first obstacles whose existence probability is greater than the threshold are determined as target first obstacles. These target first obstacles are considered to be real obstacles, and the automatic driving system needs to perform corresponding obstacle avoidance operations.

[0057] It should be noted that the present disclosure does not limit the execution order of S101 and S102. The two can be executed at the same time, and S102 can be executed before S101 or after S101.

[0058] For example, suppose that during the autonomous driving process, the ultrasonic sensor detects an obstacle A 5 meters ahead and gives its existence probability as 0.8. At the same time, the fisheye camera also recognizes that there is an obstacle B in front (possibly the same obstacle) and gives its existence probability as 0.7. The calculated IoU value of obstacles A and B is 0.6 (indicating that the two overlap in space to a certain extent). According to the weighted fusion algorithm, the probability of the existence of obstacle A is finally determined to be 0.85 (higher than the preset threshold of 0.8), so obstacle A is determined as the first target obstacle, and the autonomous driving system will take corresponding obstacle avoidance measures.

[0059] The technical solution of the disclosed embodiment, combined with the data of ultrasonic sensors and fisheye cameras, can more comprehensively capture the surrounding environment information, reduce the false alarms or missed alarms that may be caused by a single sensor, improve the accuracy and reliability of obstacle recognition, and thus facilitate timely obstacle avoidance decisions, which helps to reduce driving safety. Under complex and changeable road conditions, by integrating data from multiple sensors, various challenges such as lighting changes, obstructions, etc. can be more effectively responded to, thereby improving the safety of autonomous driving.

[0060] In some embodiments, the obstacle determination method may further include: grouping the ultrasonic data to form a plurality of echo combinations; determining a target echo combination from the plurality of echo combinations; and determining a plurality of first obstacles based on the target echo combination.

[0061] In the embodiment of the present disclosure, the echo combination is a combination formed by grouping the ultrasonic data according to a specific rule, and each combination includes a group of related ultrasonic data.

[0062] In the embodiment of the present disclosure, the target echo combination is a data combination that is most likely to correspond to a real obstacle selected from multiple echo combinations. The target echo combination usually contains the most stable and consistent intersection data, and therefore is most likely to correspond to a real obstacle.

[0063] In the embodiment of the present disclosure, the first obstacle is an obstacle identified based on the target echo combination, and generally includes information such as its position, size, and shape.

[0064] In some embodiments, the collected ultrasonic data are grouped and processed according to specific rules to form multiple echo combinations. The grouping rules may be based on factors such as time, spatial position, reflection intensity, etc. For example, ultrasonic data from different directions in the same time period can be grouped together, or data with similar reflection intensity can be grouped together.

[0065] In some embodiments, determining a target echo combination from a plurality of echo combinations includes: extracting key features from each echo combination, such as average distance, maximum reflection intensity, etc. Based on the extracted features and preset screening conditions (such as distance range, reflection intensity threshold, etc.), determining a target echo combination from a plurality of echo combinations.

[0066] In some embodiments, determining the first obstacle based on the target echo combination includes: analyzing the data in the target echo combination to identify the location, size, shape and other information of the obstacle; and calculating the existence probability of each obstacle, such as the first probability value and the second probability value, based on the identification result and a preset probability model (such as a Bayesian network, a random forest, etc.).

[0067] For example, suppose that during autonomous driving, the ultrasonic sensor collects a series of data about obstacles ahead. Through group processing, these data are divided into three groups: short-range group, medium-range group, and long-range group. Then, based on characteristics such as average distance and reflection intensity, the target echo combination is screened out from these three groups of data. Finally, the data in the target echo combination is analyzed to identify a pedestrian obstacle 5 meters ahead, and its probability of existence is calculated to be 0.9.

[0068] In this way, by grouping and processing ultrasonic data, key information in the data can be extracted and utilized more effectively, thereby more accurately identifying obstacles. By screening the target echo combination, false positives and false negatives caused by noise, interference and other factors can be reduced. This method can adapt to changes in different environments and road conditions, improve the ability of the autonomous driving system to identify and respond to obstacles, and enhance the robustness of the system.

[0069] In some embodiments, determining a target echo combination from a plurality of echo combinations comprises:

[0070] respectively determining the sum of the standard deviations of the intersection points corresponding to each echo combination in the plurality of echo combinations;

[0071] The echo combination with the smallest sum of intersection standard deviations is determined as the target echo combination.

[0072] In the disclosed embodiment, the sum of intersection standard deviations refers to the sum obtained by calculating the standard deviation of the intersection of each echo combination and adding all the standard deviations, which reflects the degree of discreteness of the intersection, that is, the stability and consistency of the intersection.

[0073] In some embodiments, determining the sum of standard deviations of intersection points corresponding to each echo combination in a plurality of echo combinations includes: first, preprocessing the collected ultrasonic data, including steps such as denoising and filtering, to improve the accuracy and reliability of the data. For each echo combination, calculating the intersection points between each ultrasonic data point within it. These intersection points may represent the position of an obstacle in space. For the intersection points of each echo combination, calculating the sum of their standard deviations.

[0074] In some embodiments, determining the echo combination with the smallest sum of intersection standard deviations as the target echo combination includes: comparing the sums of intersection standard deviations of multiple echo combinations to find the smallest one. Determining the echo combination with the smallest sum of intersection standard deviations as the target echo combination.

[0075] Assume that during the autonomous driving process, the ultrasonic sensor collects a series of data about the obstacle ahead and forms three echo combinations. By calculating the sum of the standard deviations of the intersection points of each echo combination, it is found that the sum of the standard deviations of the intersection points of the first echo combination is the smallest, so it is determined as the target echo combination. Then, the data in the target echo combination is analyzed, and a vehicle obstacle 3 meters ahead is identified, and its probability of existence is calculated to be 0.85.

[0076] In this way, by comparing the sum of the intersection standard deviations, the most stable and consistent echo combination can be selected, and the target echo combination that is most likely to correspond to the real obstacle can be determined from multiple echo combinations, thereby improving the accuracy of obstacle identification. Since the selection of the target echo combination is based on the stability and consistency of the data, it can reduce false alarms and missed alarms caused by noise, interference and other factors.

[0077] In some embodiments, determining a target echo combination from a plurality of echo combinations comprises:

[0078] respectively determining the number of intersection points corresponding to each echo combination in the plurality of echo combinations;

[0079] The echo combination with the largest number of intersection points is determined as the target echo combination.

[0080] In the embodiment of the present disclosure, the total number of intersections formed between various data points in the ultrasonic data reflects the distribution of the ultrasonic data in space and the number of potential obstacles.

[0081] In the disclosed embodiment, the target echo combination is the echo combination with the largest number of intersections selected from multiple echo combinations, and usually contains the most intersection data, and therefore is most likely to correspond to a real obstacle.

[0082] In some embodiments, the collected ultrasonic data is preprocessed, including steps such as denoising and filtering, to improve the accuracy and reliability of the data. For each echo combination, the intersections between the ultrasonic data points within it are calculated. These intersections may represent potential locations of obstacles in space. The number of intersections in each echo combination is counted, that is, the total number of intersections formed between all ultrasonic data points in the combination.

[0083] In some implementations, determining the echo combination with the largest number of intersections as the target echo combination includes: comparing the number of intersections of multiple echo combinations to find the largest one; and determining the echo combination with the largest number of intersections as the target echo combination. This combination usually contains the most intersection data and is therefore most likely to correspond to a real obstacle.

[0084] Assume that during the autonomous driving process, the ultrasonic sensor collects a series of data about the obstacle ahead and forms three echo combinations. By calculating the number of intersections of each echo combination, it is found that the second echo combination has the largest number of intersections, so it is determined as the target echo combination. Then, the data in the target echo combination is analyzed, and a pedestrian obstacle 5 meters ahead is identified, and its probability of existence is calculated to be 0.9. Based on the probability of the obstacle's existence and the safety strategy of the autonomous driving system, the system makes corresponding obstacle avoidance decisions and successfully avoids collision with pedestrians.

[0085] In this way, by comparing the number of intersections, the echo combination containing the most intersection data can be selected, thereby improving the accuracy of obstacle recognition. Since the selection of the target echo combination is based on the number of intersections, false alarms and missed alarms caused by factors such as noise and interference can be reduced.

[0086] In some embodiments, in each echo combination, the priority of the intersection point obtained based on the current frame is greater than the priority of the intersection point obtained based on the historical frame.

[0087] In the disclosed embodiment, the current frame refers to the data frame collected by the radar or ultrasonic sensor at the current moment, which reflects the latest environmental information.

[0088] In the embodiment of the present disclosure, the historical frame refers to the data frame collected by the radar or ultrasonic sensor at the past time, which provides the background information of the environmental change.

[0089] In the disclosed embodiment, the intersection point is a point formed by the intersection of different data points (such as echo signals) in radar or ultrasonic data, which usually represents the potential location of an obstacle.

[0090] In the disclosed embodiment, the priority is a processing order set according to the importance or timeliness of the data when processing multiple data. For example, the priority of the current frame intersection is higher than the historical frame intersection.

[0091] In some embodiments, raw data from a radar or ultrasonic sensor is collected, and the data includes multiple echo signals of a current frame and a historical frame. The raw data is preprocessed, such as denoising, filtering, and calibration, to improve data quality. For each echo combination, the intersection points in the current frame and the historical frame are calculated respectively. The intersection points are divided into two categories: the intersection points of the current frame (calculated from the current frame data) and the intersection points of the historical frame (calculated from the historical frame data).

[0092] In some embodiments, a higher priority is set for the current frame intersections because they reflect the latest environmental information and are crucial for real-time obstacle avoidance and decision-making. A lower priority is set for the historical frame intersections. Although they provide background information about environmental changes, they may not be as accurate and timely as the current frame data. According to the priority, the current frame intersections are used for obstacle identification and positioning. When the current frame intersections are not sufficient to determine the position or shape of the obstacle, consider fusing the historical frame intersections for supplementation. Obstacle identification is performed based on the screened and fused intersection data, including the extraction of information such as position, size, shape, and speed. According to the identification results and the safety strategy of the autonomous driving system, corresponding obstacle avoidance decisions are made.

[0093] Assume that during the autonomous driving process, the radar sensor collects a series of data about the obstacle ahead. At a certain moment, the system receives multiple echo combinations of the current frame and the historical frame at the same time. By calculating the intersection points and setting the priority, the system finds that there is an intersection point in the current frame that is highly consistent with the position of the vehicle ahead, so it is prioritized as an obstacle. Subsequently, the system makes corresponding obstacle avoidance decisions based on the position and speed information of the obstacle, successfully avoiding a collision with the vehicle ahead.

[0094] In this way, by giving priority to the intersection of the current frame, it is possible to respond to environmental changes more quickly and improve the real-time performance of the autonomous driving system. The current frame data is usually more accurate and timely than the historical frame data, so it can reduce false positives and false negatives and improve the accuracy of obstacle recognition. In complex and changing environments, the intersection fusion of the current frame and the historical frame can provide more comprehensive information and enhance the robustness of the autonomous driving system.

[0095] In some embodiments, determining a plurality of first obstacles based on a combination of target echoes includes:

[0096] When there is an intersection in the target echo combination, the intersection is used as an obstacle observation result, and the obstacle observation result is associated with the first historical trajectory;

[0097] When there is no intersection in the target echo combination, the single arc is associated with the second historical trajectory, wherein the second historical trajectory is no longer associated with the single arc after being continuously associated with the single arc for the target times;

[0098] Obstacles obtained from the target echo combination are filtered to determine a plurality of first obstacles.

[0099] In the embodiment of the present disclosure, the first historical trajectory refers to an obstacle trajectory that has been previously identified and tracked.

[0100] In the disclosed embodiment, the second historical trajectory refers to a historical trajectory of a similar arc-shaped signal observed previously, which may be associated with a single arc echo.

[0101] In the disclosed embodiments, filtering processing refers to the process of performing denoising, smoothing, and other processing on the observation results to improve data quality and recognition accuracy.

[0102] In some embodiments, when there are intersections in the target echo combination, these intersections are likely to represent the positions of real obstacles. Therefore, these intersections are used as obstacle observation results, and an attempt is made to associate them with the first historical trajectory.

[0103] In some embodiments, all intersections are extracted from the target echo combination. The extracted intersections are matched with the first historical trajectory (i.e., the previously identified obstacle trajectory) to determine whether these intersections belong to known obstacles. If the intersection successfully matches a historical trajectory, the intersection is associated with the trajectory and the trajectory information (such as position, speed, etc.) is updated. If the match is unsuccessful, it may be a newly appeared obstacle and a new trajectory needs to be initialized.

[0104] In some embodiments, when there is no intersection in the target echo combination, a single arc echo signal may be received. These signals may represent moving obstacles, but are not sufficient to determine their specific location. Identify the single arc echo in the target echo combination; associate the identified single arc echo with a second historical trajectory (which may be a historical trajectory of a similar arc signal previously observed); if the single arc echo is continuously associated with a historical trajectory for multiple times (such as similar arc signals are observed for several consecutive frames), it is considered that the single arc echo does represent an obstacle. However, if the number of consecutive associations reaches a certain threshold, and no new single arc echoes are associated with the trajectory, it may mean that the obstacle has left the detection range or no longer generates an arc signal, and the association with the trajectory should be terminated at this time.

[0105] In some embodiments, the obstacles obtained from the target echo combination are filtered to determine multiple first obstacles, including: filtering the noise to remove false obstacle observation results caused by sensor noise, environmental interference and other factors; smoothing the associated obstacle trajectory to reduce trajectory fluctuations caused by measurement errors and other reasons; and confirming the final first obstacle list based on the filtered observation results and trajectory information. These first obstacles should be entities that exist stably, have clear positions and conform to physical laws.

[0106] For example, suppose that during the autonomous driving process, the radar sensor collects a series of data about the obstacles ahead and forms a target echo combination. At a certain moment, the system finds that there is an intersection in the target echo combination, and the intersection successfully matches a previous historical trajectory (representing a moving vehicle), so the intersection is associated with the trajectory and the trajectory information is updated. At another moment, the system receives a single arc echo signal, which is continuously associated with a previous historical trajectory (representing a turning vehicle) for multiple frames of data, so it is confirmed that the single arc echo represents a real obstacle. Finally, the system filters all observations, removes noise and false observations, and obtains the final first obstacle list.

[0107] In this way, by associating the intersection with the historical trajectory and processing the single arc echo, the real obstacle can be identified more accurately, reducing false alarms and missed alarms. False observations and noise interference are removed through filtering, which reduces the amount of data and computational complexity of subsequent processing and improves resource utilization efficiency.

[0108] In some embodiments, obtaining first probability values ​​and second probability values ​​corresponding to each of the plurality of first obstacles includes:

[0109] The probability after the sum of the intersection standard deviations of all subgroups in the echo combination with the smallest sum of the intersection standard deviations is normalized by the mean is taken as the first probability value;

[0110] The probability obtained by normalizing the sum of the standard deviations of the intersection points of all subgroups in the echo combination with the largest number of intersection points is used as the second probability value.

[0111] In the disclosed embodiment, the intersection standard deviation is a statistic used to measure the degree of dispersion of the intersection distribution. The smaller the standard deviation, the more concentrated the intersection distribution is.

[0112] In the disclosed embodiment, mean normalization is a process of converting raw data into probability values ​​between 0 and 1, and is usually used for data standardization.

[0113] In the embodiment of the present disclosure, the echo combination is a set including multiple subgroups, each subgroup including intersection information about obstacles.

[0114] In the embodiment of the present disclosure, a subgroup is a basic unit in an echo combination and may include specific intersection information about an obstacle.

[0115] In some embodiments, data of multiple echo combinations are collected, each echo combination includes multiple subgroups, and each subgroup includes intersection information about obstacles. The intersection standard deviation of each subgroup is calculated, which is an important indicator for evaluating the degree of discreteness of intersection distribution.

[0116] In some implementations, for each obstacle, find the echo combination with the smallest sum of intersection standard deviations; calculate the sum of intersection standard deviations of all subgroups in the echo combination, and perform mean normalization to obtain a normalized probability value, namely, a first probability value.

[0117] In some implementations, for each obstacle, find the echo combination with the largest number of intersections; calculate the sum of the standard deviations of the intersections of all subgroups in the echo combination, and perform mean normalization to obtain a normalized probability value, ie, a second probability value.

[0118] Here, the purpose of mean normalization is to convert the original data into probability values ​​between 0 and 1 to facilitate subsequent comparison and analysis.

[0119] Assume there are three obstacles A, B, and C, corresponding to three echo combinations:

[0120] Obstacle A: echo combination 1 (subgroup 1: intersection standard deviation = 2, subgroup 2: intersection standard deviation = 3), echo combination 2 (subgroup 1: intersection standard deviation = 1, subgroup 2: intersection standard deviation = 4).

[0121] Obstacle B: echo combination 1 (subgroup 1: intersection standard deviation = 2, subgroup 2: intersection standard deviation = 2), echo combination 2 (subgroup 1: intersection standard deviation = 3, subgroup 2: intersection standard deviation = 3).

[0122] Obstacle C: echo combination 1 (subgroup 1: intersection standard deviation = 1, subgroup 2: intersection standard deviation = 1, subgroup 3: intersection standard deviation = 1), echo combination 2 (subgroup 1: intersection standard deviation = 2, subgroup 2: intersection standard deviation = 2).

[0123] For obstacle A: the echo combination with the smallest sum of intersection standard deviations is echo combination 2 (2+4=6), and the first probability value after mean normalization is 6 / (2+4+1+5)=0.6.

[0124] The echo combinations with the largest number of intersections are echo combination 1 and echo combination 2 (both have 2 intersections), but echo combination 2 with a smaller sum of intersection standard deviations is selected for calculation, and the second probability value after mean normalization is also 0.6 (note that in order to simplify the calculation here, assuming that the number of intersections is the same, the combination with a smaller sum of standard deviations is selected).

[0125] For obstacles B and C, the calculation can be performed according to the above steps.

[0126] Thus, by introducing the first probability value and the second probability value, the discrete degree and quantity of the intersection points can be comprehensively considered, thereby improving the accuracy of obstacle recognition. By the mean normalization process, the subsequent calculation process is simplified and the calculation efficiency is improved.

[0127] In some embodiments, determining a target second obstacle for each first obstacle based on a plurality of second obstacles includes:

[0128] Converting the three-dimensional position of each first obstacle into a two-dimensional projection position;

[0129] The detection frames of the plurality of second obstacles are traversed, and for each first obstacle, a second obstacle having a maximum intersection-over-union ratio with its two-dimensional projection position is determined as a target second obstacle.

[0130] In the disclosed embodiment, the three-dimensional position is the position information of the obstacle in the three-dimensional space, and generally includes three coordinate values ​​of X, Y, and Z.

[0131] In the embodiment of the present disclosure, the two-dimensional projection position is the projection position information of the obstacle on the two-dimensional image, which usually includes two coordinate values, X and Y.

[0132] In the disclosed embodiment, the detection frame is a rectangular frame used to represent the position and size of the obstacle in the two-dimensional image.

[0133] In some embodiments, converting a three-dimensional position into a two-dimensional projection position includes: obtaining the three-dimensional position information of each first obstacle. At the same time, obtaining the internal and external parameters and distortion parameters of the camera, which are crucial for converting the three-dimensional position into a two-dimensional projection position. Using methods such as perspective projection or orthographic projection, the three-dimensional position of each first obstacle is converted into a two-dimensional projection position. Perspective projection is generally more suitable for simulating the visual effect of the human eye, while orthographic projection is more suitable for scenes that need to maintain shape and size invariance. During the conversion process, factors such as the camera's viewing angle, focal length, and optical axis direction need to be considered to ensure the accuracy of the conversion result.

[0134] In some embodiments, traversing the detection frames and determining the target second obstacle includes: traversing the detection frames of all second obstacles, and for each first obstacle, calculating the intersection-and-union ratio of its two-dimensional projection position with each second obstacle detection frame. For each first obstacle, selecting the second obstacle with the largest intersection-and-union ratio with its two-dimensional projection position as the target second obstacle. This means that the target second obstacle has the highest degree of overlap with the two-dimensional projection position of the first obstacle on the two-dimensional image.

[0135] For example, suppose there are two first obstacles A and B, and three second obstacles C, D, and E. The position and detection frame information of each obstacle are as follows:

[0136] Obstacle A (3D position): (X1, Y1, Z1), 2D projection position: (x1, y1)

[0137] Obstacle B (3D position): (X2, Y2, Z2), 2D projection position: (x2, y2)

[0138] Obstacle C (detection box): (xcl, yc1, wc, hc)

[0139] Obstacle D (detection box): (xd1, ydl, wd, hd)

[0140] Obstacle E (detection box): (xe1, yel, we, he)

[0141] For obstacle A: calculate its intersection over union ratio with obstacles C, D, and E, which are IoU_AC, IoU_AD, and IoU_AE respectively. Assuming that IoU_AD is the largest, obstacle D is selected as the second target obstacle of obstacle A. For obstacle B, the above steps can also be followed for calculation and selection.

[0142] In this way, by calculating the intersection-and-union ratio and selecting the second obstacle with the largest intersection-and-union ratio as the target second obstacle, the accuracy of the matching result can be ensured. By traversing the detection frame and calculating the intersection-and-union ratio, the matching process is simplified and the calculation efficiency is improved.

[0143] In some embodiments, determining the existence probability of each first obstacle based on the intersection-and-union ratio of each first obstacle and its corresponding target second obstacle, and the first probability value and the second probability value corresponding to each first obstacle, includes:

[0144] A preset fusion algorithm such as the Dempster-Shafer evidence theory (also known as the DST evidence theory or the DS evidence theory) is used to fuse the intersection ratio of each first obstacle and its corresponding target second obstacle, as well as the first probability value and the second probability value corresponding to each first obstacle to obtain the existence probability of each first obstacle.

[0145] In the disclosed embodiment, DS evidence theory is a reasoning method for processing uncertain information, which draws a final conclusion by constructing a recognition framework, assigning basic probabilities and synthesizing evidence.

[0146] In the disclosed embodiment, Basic Probability Assignment (BPA) is a probability value assigned to each proposition in the recognition framework in DS evidence theory.

[0147] In the disclosed embodiment, the Dempster synthesis rule is a core rule in the DS evidence theory, which is used to synthesize information from multiple evidence sources.

[0148] In some embodiments, according to the specific requirements of the problem, a recognition framework containing all possible answers (i.e., the existence status of the first obstacle) is constructed. For example, the recognition framework can be defined as {existence, non-existence}. According to the size of the intersection-and-union ratio, a basic probability value regarding its existence is assigned to each first obstacle. For example, the higher the intersection-and-union ratio, the greater the probability value assigned to the "existence" state. Assigning the BPA of the first probability value and the second probability value: The first probability value and the second probability value of each first obstacle are respectively used as two independent sources of evidence and assigned to the "existence" state in the recognition framework. Using the Dempster synthesis rule, the BPA of the intersection-and-union ratio, the BPA of the first probability value, and the BPA of the second probability value are synthesized to obtain a new basic probability allocation function. The synthesized basic probability allocation function is normalized to ensure that the sum of all probability values ​​is 1. From the synthesized basic probability allocation function, the probability value of each first obstacle corresponding to the "existence" state is extracted as the existence probability of the obstacle.

[0149] For example, assuming there are two first obstacles A and B, the corresponding intersection-over-combination ratios, first probability values, and second probability values ​​of the target second obstacles are as follows:

[0150] obstacle IoU The first probability value The second probability value A 0.8 0.7 0.6 B 0.5 0.5 0.4

[0151] For obstacle A, the BPA for the intersection and union ratio is {existence: 0.8, non-existence: 0.2}, the BPA for the first probability value is {existence: 0.7, non-existence: 0.3}, and the BPA for the second probability value is {existence: 0.6, non-existence: 0.4}. For obstacle B, the BPA for the intersection and union ratio is {existence: 0.5, non-existence: 0.5}, the BPA for the first probability value is {existence: 0.5, non-existence: 0.5}, and the BPA for the second probability value is {existence: 0.4, non-existence: 0.6}. Using the Dempster synthesis rule, the three BPAs are synthesized to obtain a new basic probability distribution function. From the synthesized basic probability distribution function, the probability values ​​of obstacles A and B corresponding to the "existence" state are extracted as their existence probabilities. For example, the existence probability of obstacle A may be 0.9 (hypothetical value, the actual calculation will involve more complex mathematical operations).

[0152] In this way, the DS evidence theory is used to fuse the information of multiple evidence sources, which improves the accuracy of the judgment of the existence of the first obstacle of the target. The Dempster synthesis rule is used to fuse the evidence, which simplifies the calculation process and improves the calculation efficiency.

[0153] The present disclosure provides an architectural diagram of an obstacle determination system, such as Figure 2As shown, it includes an echo grouping submodule, a selection submodule, an obstacle tracking submodule, a post-processing submodule and a fusion submodule; wherein the echo grouping submodule is responsible for grouping ultrasonic data and deleting illogical combinations; the selection submodule is responsible for calculating all possible intersections and selecting the optimal combination as the obstacle detection result; the obstacle tracking submodule is responsible for tracking and filtering obstacles to form a stable obstacle trajectory; the post-processing submodule is responsible for reducing the confidence of a single arc, for example, no longer associating a single arc after associating it a certain number of times; the fusion submodule is responsible for fusing with the detection results of the fisheye camera.

[0154] In some embodiments, obstacles may be determined through S301 to S307 .

[0155] S301: traverse and calculate all echo intersection points as candidate obstacles.

[0156] Figure 3 The diagram shows the relationship between obstacles, measured distance and ellipse. Figure 3 In , T represents the transmitter, R1 and R2 represent the receiving heads, the measured distance received by R1 is 11+1, and the measured distance received by R2 is 12+1. The obstacle is located on the ellipse with R1 and T as the focus and 11+1 as the major axis length, and also on the ellipse with R2 and T as the focus and 12+1 as the major axis length, that is, the position of the obstacle is the intersection of the two ellipses.

[0157] S302: Traverse the echo data and group them.

[0158] Assuming that the maximum number of detected obstacles is 2, all echoes are divided into two groups to ensure that echoes from the same probe are not in the same group.

[0159] Figure 4 A schematic diagram of obstacles and their echoes is shown, such as Figure 4 As shown, traverse all possible combinations:

[0160] The first echo combination (1): subgroups 4-1 and 5-1, subgroups 4-2 and 5-2, and subgroup 4-3;

[0161] The second echo combination (2): subgroup 4-1 and 5-2, subgroup 4-2 and 5-1, subgroup 4-3.

[0162] S303: assigning an identification (ID) to each subgroup (denoted as subgroup).

[0163] When the subgroup is empty, set its ID to -1;

[0164] When a subgroup has only one echo and cannot find an intersection with the historical frame, its ID is set to 3;

[0165] When there are two echoes in a subgroup that cannot find the intersection point, set its ID to 1;

[0166] When the standard deviation of the intersection points of all two echoes in the subgroup is greater than a certain threshold, its ID is set to 2;

[0167] In other cases, its ID is set to 0, indicating that the subgroup can find valid obstacles.

[0168] Based on the above ID assignment method, Figure 4 The IDs of the first echo combination (1) are: 1, 1 and 3; the IDs of the second echo combination (2) are: 1, 0 and 3.

[0169] S304: Select the best echo combination (ie, target echo grouping).

[0170] First, some echo groups (referred to as group) are deleted according to the IDs assigned to the subgroups. Specifically, if any subgroup in the group has an ID of 1 or 2, the group is deleted; if all subgroups in the group have IDs of -1 or 3, the group is deleted.

[0171] Secondly, these groups are selected as target echo groups in turn:

[0172] The group with the smallest sum of standard deviations of the intersection points of all subgroups; the sum of standard deviations is a_n, which is normalized to the same value and recorded as probability a_n_normalized.

[0173] The group with the largest number of pairwise echo intersections among all subgroups; the sum of the standard deviations is b_n, which is normalized to the mean and recorded as the probability b_n_normalized.

[0174] S305: Tracking the obstacles in the target echo group.

[0175] Figure 5 A schematic diagram showing the inference, tracking, association and update of obstacles.

[0176] The Predict step includes: first, determining the position of the obstacle in the previous frame; since the obstacle is moving, it is necessary to infer its new position in the current frame based on the obstacle's movement direction and speed; by considering the time difference and the obstacle's movement characteristics (such as speed, direction, etc.), the predicted position of the obstacle in the current frame is calculated.

[0177] The tracking (Association) step includes: grouping the echo data in the current frame and calculating the focal position of each group of echoes; matching the predicted position with each focal position to find the closest focal point; if a focal point is very close to the predicted position (i.e. within a certain error range), it is considered that the focal point and the obstacle in the previous frame are the same object, that is, the obstacle is successfully tracked.

[0178] The update step includes: updating the position information of the successfully tracked obstacle to the focus position in the current frame; and saving the updated position information as the starting point for the next frame speculation.

[0179] When new echo data is received, the steps of inference, tracking and updating are repeated to continuously track the movement trajectory of the obstacle.

[0180] The three steps of inference, tracking association and updating constitute a complete tracking cycle, which can realize the continuous tracking and position update of moving obstacles. In practical applications, these steps may need to be implemented in combination with specific algorithms and data processing technologies.

[0181] S306: After associating the single arc echo a certain number of times, the single arc echo is no longer associated to reduce the confidence of the single arc echo.

[0182] S307: The ultrasonic target and the fisheye camera target are fused with existence probability by using DS evidence theory.

[0183] First, it is necessary to obtain the two-dimensional (2D) projection of the ultrasonic target. This can be done using the following formula:

[0184] O ultrasonic_2D =P ultrasonic ×O ultrasonic_3D ;

[0185] Among them, O ultrasonic_2D Represents the projection of the target detected by the ultrasonic sensor on the 2D plane; P ultrasonic Represents the projection matrix of the ultrasonic sensor projecting the three-dimensional (3D) target position onto the 2D plane; ultrasonic_3D Represents the 3D position of a target detected by the ultrasonic sensor.

[0186] In this way, the 3D target position can be converted into a projection on a 2D plane through the projection matrix of the ultrasonic sensor.

[0187] Next, target matching is performed, including:

[0188] 1. Get fisheye detection targets: Use the fisheye camera to detect all 2D targets, which are usually represented in the form of bounding boxes (denoted as bbox), denoted as O fisheye .

[0189] 2. Calculate IoU: For each 2D target O detected by the fisheye fisheye , calculate its difference with the ultrasonic projection target O ultrasonic_2D IoU is an indicator that measures the degree of overlap between two target areas, and its value ranges from 0 to 1.

[0190] 3. Determine the matching target: Find the target O that matches the ultrasonic projection target ultrasonic _ 2D The fisheye detected targets with the largest IoU are considered to be matched. If the maximum IoU value is still 0 (that is, no fisheye detected target overlaps with the ultrasonic projection target), it is considered that the ultrasonic target does not match any target in the current fisheye image, and its existence probability is 0.

[0191] 4. Process the matching results: Based on the matching results, subsequent processing can be performed, such as updating the target status, fusing sensor information, etc.

[0192] It should be noted that in practical applications, there may be certain errors in the matching process due to factors such as sensor noise, target occlusion, detection error, etc. Therefore, more complex algorithms or strategies can be used to improve the accuracy and robustness of matching.

[0193] In addition, if the fusion problem of multiple ultrasonic sensors and multiple fisheye cameras needs to be considered, the above method needs to be further extended to handle the data association and fusion between multiple sensors.

[0194] In the previous steps, the targets have been detected by the ultrasonic sensor and the fisheye camera respectively, and the IoU between them and possible normalization parameters (such as a_n_normalized and b_n_normalized, which may represent normalized values ​​related to detection confidence or target attributes) have been calculated. Now, DS evidence theory will be used to fuse this information to derive the final existence probability.

[0195] DS evidence theory is a mathematical framework for processing uncertain information, which allows information from multiple sources to be fused to produce a more accurate and reliable result. During the fusion process, each information source assigns a degree of confidence (or basic confidence assignment) to the event it considers possible.

[0196] For application scenarios, each information source (such as ultrasonic sensors and fisheye cameras) can be regarded as a "evidence" that provides confidence. Then, the fusion rule of DS evidence theory is used to calculate the final existence probability.

[0197] The specific steps are as follows:

[0198] 1. Define the event set: First, define a set Θ containing all possible events. In the scenario, this set may include events such as "target exists" and "target does not exist".

[0199] 2. Calculate basic trust assignment: For each information source, calculate its basic trust assignment for each event. For example, an ultrasonic sensor may assign trust based on the signal strength and target distance it detects; a fisheye camera may assign trust based on the size of the target bounding box and the IoU value it detects. In addition, normalization parameters such as a_n_normalized and b_n_normalized can also be taken into account.

[0200] 3. Apply DS fusion rule: Use the fusion rule of DS evidence theory to calculate the final existence probability. This rule can be expressed as:

[0201] (If and only if A=X)

[0202] However, this formula may be too complex when applied directly, because all possible event combinations need to be considered. In practical applications, more simplified methods such as the Dempster-Shafer combination rule are often used to gradually fuse information from multiple sources.

[0203] 4. Set a threshold and output the result: Set a threshold to determine whether the final existence probability is high enough to confirm the existence of the target. If m(A) is greater than or equal to this threshold, the result is notified to the perception module; otherwise, the detection is filtered out as a false positive.

[0204] It should be noted that although DS evidence theory provides a powerful method to fuse information from multiple sources, it still needs to be handled with caution in practical applications. For example, it is necessary to ensure that the trust distribution of each information source is reasonable, and the fusion rules and thresholds need to be carefully selected to avoid over-fusion or false detection. In addition, since DS evidence theory can be computationally complex, it may be optimized or approximated as needed in real-time applications.

[0205] In this way, by fusing the 2D projection of the ultrasonic and fisheye camera targets through the DS evidence theory, false detections can be effectively filtered out and the accuracy of obstacle determination can be improved. Fusion of ultrasonic and camera perception for obstacle detection and tracking in autonomous driving can improve the accuracy and reliability of obstacle detection, reduce the false alarm rate, and improve anti-interference capabilities.

[0206] The present disclosure provides an obstacle determination device, such as Figure 6 As shown, the device may include: a first acquisition unit 601, used to acquire a first probability value and a second probability value corresponding to each of a plurality of first obstacles, the first obstacle being determined based on ultrasonic data; a second acquisition unit 602, used to acquire a plurality of second obstacles, the second obstacle being determined based on fisheye camera data; a first determination unit 603, used to determine a target second obstacle for each first obstacle based on the plurality of second obstacles; a second determination unit 604, used to determine the existence probability of each first obstacle based on an intersection-and-union ratio of each first obstacle to its corresponding target second obstacle, and the first probability value and the second probability value corresponding to each first obstacle; and a third determination unit 605, used to determine a first obstacle whose existence probability is greater than a preset threshold as a target first obstacle.

[0207] In some embodiments, the apparatus further comprises:

[0208] A grouping unit, used for grouping and processing the ultrasonic data to form a plurality of echo combinations;

[0209] a fourth determining unit, configured to determine a target echo combination from the plurality of echo combinations;

[0210] The fifth determining unit is configured to determine a plurality of first obstacles based on the target echo combination.

[0211] In some embodiments, the fourth determining unit is configured to:

[0212] respectively determining the sum of the standard deviations of the intersection points corresponding to each echo combination in the plurality of echo combinations;

[0213] The echo combination with the smallest sum of intersection standard deviations is determined as the target echo combination.

[0214] In some embodiments, the fourth determining unit is configured to:

[0215] respectively determining the number of intersection points corresponding to each echo combination in the plurality of echo combinations;

[0216] The echo combination with the largest number of intersection points is determined as the target echo combination.

[0217] In some embodiments, in each echo combination, the priority of the intersection point obtained based on the current frame is greater than the priority of the intersection point obtained based on the historical frame.

[0218] In some embodiments, the first determining unit 603 includes:

[0219] A first associating subunit is used for, when there is an intersection in the target echo combination, taking the intersection as an obstacle observation result, and associating the obstacle observation result with the first historical trajectory;

[0220] A second association subunit is used to associate the single arc with the second historical trajectory when there is no intersection in the target echo combination, wherein the second historical trajectory is no longer associated with the single arc after being continuously associated with the single arc for the target number of times;

[0221] The filtering subunit is used to perform filtering processing on obstacles obtained from the target echo combination to determine a plurality of first obstacles.

[0222] In some embodiments, the first acquisition unit 601 is specifically used to:

[0223] The probability after the sum of the intersection point standard deviations of all subgroups in the echo combination with the smallest sum of the intersection point standard deviations is normalized by the mean is taken as the first probability value;

[0224] The probability after the sum of the standard deviations of the intersection points of all subgroups in the echo combination with the largest number of intersection points is normalized by the mean is taken as the second probability value.

[0225] In some embodiments, the first determining unit 603 includes:

[0226] A projection subunit, used to convert the three-dimensional position of each first obstacle into a two-dimensional projection position;

[0227] The determination subunit is used to traverse the detection frames of multiple second obstacles, and determine, for each first obstacle, a second obstacle having a maximum intersection-over-union ratio with its two-dimensional projection position as a target second obstacle.

[0228] In some embodiments, the third determining unit 605 is configured to:

[0229] A preset fusion algorithm such as DS evidence theory is used to fuse the intersection ratio of each first obstacle and its corresponding target second obstacle, as well as the first probability value and the second probability value corresponding to each first obstacle to obtain the existence probability of each first obstacle.

[0230] For the description of specific functions and examples of each unit and sub-unit of the device in the embodiment of the present disclosure, reference can be made to the relevant description of the corresponding steps in the above method embodiment, which will not be repeated here.

[0231] The obstacle determination device of the embodiment of the present disclosure can improve the accuracy and reliability of obstacle detection, thereby helping to improve the safety of vehicle driving.

[0232] The present disclosure provides a scene diagram of an obstacle determination method, such as Figure 7 shown.

[0233] As mentioned above, the obstacle determination method provided by the embodiment of the present disclosure is applied to electronic devices. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, wearable devices, and other similar computing devices.

[0234] Specifically, the electronic device can perform the following operations:

[0235] Obtaining first probability values ​​and second probability values ​​corresponding to each of a plurality of first obstacles, wherein the first obstacles are determined based on ultrasonic data;

[0236] Acquire a plurality of second obstacles, where the second obstacles are determined based on the fisheye camera data;

[0237] determining a target second obstacle for each first obstacle based on the plurality of second obstacles;

[0238] Determine the existence probability of each first obstacle based on the intersection-and-joint ratio of each first obstacle and its corresponding target second obstacle, and the first probability value and the second probability value corresponding to each first obstacle;

[0239] A first obstacle whose existence probability is greater than a preset threshold is determined as a target first obstacle.

[0240] It should be understood that Figure 7 The scene diagram shown is only illustrative and not restrictive. Those skilled in the art can Figure 7 Various obvious changes and / or substitutions can be made to the examples, and the resulting technical solutions still fall within the scope of the disclosure of the embodiments of the present disclosure.

[0241] In the technical solution disclosed herein, the acquisition, storage and application of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0242] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0243] Figure 8A schematic block diagram of an example electronic device 800 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0244] like Figure 8 As shown, the device 800 includes a computing unit 801, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 802 or a computer program loaded from a storage unit 808 to a random access memory (RAM) 803. In the RAM 803, various programs and data required for the operation of the device 800 can also be stored. The computing unit 801, the ROM 802, and the RAM 803 are connected to each other via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0245] A number of components in the device 800 are connected to the I / O interface 805, including: an input unit 806, such as a keyboard, a mouse, etc.; an output unit 807, such as various types of displays, speakers, etc.; a storage unit 808, such as a disk, an optical disk, etc.; and a communication unit 809, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 809 allows the device 800 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0246] The computing unit 801 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a CPU, a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The computing unit 801 performs the various methods and processes described above, such as the obstacle determination method. For example, in some embodiments, the obstacle determination method may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 808. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 800 via the ROM 802 and / or the communication unit 809. When the computer program is loaded into the RAM 803 and executed by the computing unit 801, one or more steps of the obstacle determination method described above may be performed. Alternatively, in other embodiments, the computing unit 801 may be configured to execute the obstacle determination method in any other appropriate manner (eg, by means of firmware).

[0247] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0248] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0249] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory, a read-only memory, an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0250] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a cathode ray tube (CRT) or a liquid crystal display (LCD) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0251] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: Local Area Networks (LANs), Wide Area Networks (WANs), and the Internet.

[0252] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0253] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0254] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the principles of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A method for determining an obstacle, comprising: Acquire a first probability value and a second probability value corresponding to each of a plurality of first obstacles, wherein the first obstacles are determined based on ultrasonic data; Acquire a plurality of second obstacles, wherein the second obstacles are determined based on fisheye camera data; determining a target second obstacle for each first obstacle based on the plurality of second obstacles; Determine the existence probability of each first obstacle based on the intersection-and-joint ratio of each first obstacle and its corresponding target second obstacle, and the first probability value and the second probability value corresponding to each first obstacle; The first obstacle whose existence probability is greater than a preset threshold is determined as a target first obstacle.

2. The method according to claim 1, further comprising: performing grouping processing on the ultrasonic data to form a plurality of echo combinations; determining a target echo combination from the plurality of echo combinations; The plurality of first obstacles are determined based on the target echo combination.

3. The method according to claim 2, wherein: Determining a target echo combination from the multiple echo combinations includes: Respectively determining the sum of the standard deviations of the intersection points corresponding to each echo combination in the multiple echo combinations; The echo combination with the smallest sum of intersection standard deviations is determined as the target echo combination.

4. The method according to claim 2, wherein: Determining a target echo combination from the multiple echo combinations includes: Respectively determining the number of intersection points corresponding to each echo combination in the multiple echo combinations; The echo combination with the largest number of intersection points is determined as the target echo combination.

5. The method according to claim 3 or 4, wherein: In each echo combination, the priority of the intersection point obtained based on the current frame is greater than the priority of the intersection point obtained based on the historical frame.

6. The method according to claim 2, wherein: The determining the plurality of first obstacles based on the target echo combination comprises: In the case where there is an intersection in the target echo combination, taking the intersection as an obstacle observation result, and associating the obstacle observation result with the first historical trajectory; When there is no intersection in the target echo combination, the single arc is associated with the second historical trajectory, wherein the second historical trajectory is no longer associated with the single arc after being continuously associated with the single arc for a target number of times; Obstacles obtained from the target echo combination are filtered to determine the multiple first obstacles.

7. The method according to claim 1, wherein: The obtaining of first probability values ​​and second probability values ​​corresponding to each of the plurality of first obstacles comprises: The probability after the sum of the intersection point standard deviations of all subgroups in the echo combination with the smallest sum of the intersection point standard deviations is normalized by the mean is taken as the first probability value; The probability obtained by normalizing the sum of the standard deviations of the intersection points of all subgroups in the echo combination with the largest number of intersection points is used as the second probability value.

8. The method according to claim 1, wherein: The step of determining a target second obstacle for each first obstacle based on the plurality of second obstacles comprises: Converting the three-dimensional position of each first obstacle into a two-dimensional projection position; The detection frames of the plurality of second obstacles are traversed, and for each first obstacle, a second obstacle having a maximum intersection-over-union ratio with its two-dimensional projection position is determined as the target second obstacle.

9. The method according to claim 1, wherein: The determining the existence probability of each first obstacle based on the intersection-and-joint ratio of each first obstacle and its corresponding target second obstacle, and the first probability value and the second probability value corresponding to each first obstacle, comprises: A preset fusion algorithm is used to fuse the intersection-and-joint ratio of each first obstacle and its corresponding target second obstacle, as well as the first probability value and the second probability value corresponding to each first obstacle, to obtain the existence probability of each first obstacle.

10. An obstacle determination device, comprising: A first acquisition unit, configured to acquire a first probability value and a second probability value corresponding to each of a plurality of first obstacles, wherein the first obstacles are determined based on ultrasonic data; A second acquisition unit, configured to acquire a plurality of second obstacles, wherein the second obstacles are determined based on the fisheye camera data; a first determining unit, configured to determine a target second obstacle for each first obstacle based on the plurality of second obstacles; a second determining unit, configured to determine the existence probability of each first obstacle based on an intersection-and-joint ratio of each first obstacle and its corresponding target second obstacle, and a first probability value and a second probability value corresponding to each first obstacle; The third determining unit is configured to determine the first obstacle whose existence probability is greater than a preset threshold as a target first obstacle.

11. The apparatus according to claim 10, further comprising: A grouping unit, used for grouping the ultrasonic data to form a plurality of echo combinations; a fourth determining unit, configured to determine a target echo combination from the plurality of echo combinations; A fifth determining unit is configured to determine the plurality of first obstacles based on the target echo combination.

12. The device according to claim 11, wherein The fourth determining unit is configured to: Respectively determining the sum of the standard deviations of the intersection points corresponding to each echo combination in the multiple echo combinations; The echo combination with the smallest sum of intersection standard deviations is determined as the target echo combination.

13. The device according to claim 11, wherein: The fourth determining unit is configured to: Respectively determining the number of intersection points corresponding to each echo combination in the multiple echo combinations; The echo combination with the largest number of intersection points is determined as the target echo combination.

14. The device according to claim 12 or 13, wherein: In each echo combination, the priority of the intersection point obtained based on the current frame is greater than the priority of the intersection point obtained based on the historical frame.

15. The device according to claim 11, wherein The first determining unit includes: A first associating subunit is used for, when there is an intersection point in the target echo combination, taking the intersection point as an obstacle observation result, and associating the obstacle observation result with the first historical trajectory; A second association subunit is used to associate the single arc with the second historical trajectory when there is no intersection in the target echo combination, wherein the second historical trajectory is no longer associated with the single arc after being continuously associated with the single arc for a target number of times; The filtering subunit is used to perform filtering processing on the obstacles obtained from the target echo combination to determine the multiple first obstacles.

16. The device according to claim 10, wherein: The first acquiring unit is used to: The probability after the sum of the intersection point standard deviations of all subgroups in the echo combination with the smallest sum of the intersection point standard deviations is normalized by the mean is taken as the first probability value; The probability obtained by normalizing the sum of the standard deviations of the intersection points of all subgroups in the echo combination with the largest number of intersection points is used as the second probability value.

17. The device according to claim 10, wherein: The first determining unit includes: A projection subunit, used to convert the three-dimensional position of each first obstacle into a two-dimensional projection position; The determination subunit is used to traverse the detection frames of the plurality of second obstacles, and determine, for each first obstacle, a second obstacle having a maximum intersection-over-union ratio with its two-dimensional projection position as the target second obstacle.

18. The device according to claim 10, wherein: The third determining unit is used to: A preset fusion algorithm is used to fuse the intersection-and-joint ratio of each first obstacle and its corresponding target second obstacle, as well as the first probability value and the second probability value corresponding to each first obstacle, to obtain the existence probability of each first obstacle.

19. An electronic device comprising: at least one processor; as well as a memory communicatively connected to at least one processor; wherein, The memory stores instructions that can be executed by at least one processor, and the instructions are executed by at least one processor to enable the at least one processor to perform the method of any one of claims 1 to 9.

20. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are for causing a computer to execute a method according to any one of claims 1-9.

21. A computer program product comprising a computer program stored on a storage medium, the computer program implementing the method according to any one of claims 1 to 9 when executed by a processor.

Images