Obstacle detection methods, equipment and computer-readable storage media

Abstract

This invention discloses an obstacle detection method, comprising: inputting a scene image acquired by an image capturing device into a first preset model to obtain a first image and a first probability that each pixel in the scene image is an unexpected obstacle; inputting the scene image into a second preset model to obtain a second image; obtaining the uncertainty of each pixel in the scene image based on the second preset model and the second image; determining a region of interest in the scene image based on the first image; obtaining a second probability that each pixel in the region of interest is an unexpected obstacle based on the uncertainty of each pixel in the region of interest; and obtaining a third probability that each pixel in the region of interest is an unexpected obstacle based on the first probability and the second probability. This invention improves the detection performance for distant obstacles and unexpected obstacles through probability fusion.

Description

Technical Field

[0001] This invention relates to the field of obstacle detection, and more particularly to an obstacle detection method, device, and computer-readable storage medium. Background Technology

[0002] Road traffic safety hazards are a global social security problem. Of all traffic accidents, those involving automobiles account for more than two-thirds and are the most serious. The vast majority of traffic accidents originate from collisions between vehicles and obstacles. The dangers posed by obstacles to traffic safety are easily overlooked; they are highly likely to cause accidents and have a significant impact on people's property and even lives.

[0003] Unexpected obstacles on the road (such as lost goods, scattered stones, etc.) are easily overlooked by drivers due to their small size and complex shape, and accurate and effective obstacle detection is a challenging task for general detection systems. Currently, the automotive environmental perception field typically uses machine vision and radar solutions. Among these, machine vision solutions are closer to human driving behavior and have enormous development potential. Machine vision solutions use cameras as sensors, and the images captured by cameras are most similar to the real world perceived by the human eye. The rich information contained within can be used to complete various tasks in the field of environmental perception. Furthermore, machine vision solutions are low-cost and easier to commercialize.

[0004] In machine vision solutions, stereo vision and deep learning are two main detection methods. Stereo vision methods primarily utilize differences in the geometric structure of objects to identify image content. However, stereo vision technology has certain limitations, being greatly affected by distance and performing poorly in detecting obstacles at a distance. Traditional supervised deep learning systems can only identify obstacle types present in the training set, failing to detect unexpected obstacles and exhibiting a high false positive rate. Summary of the Invention

[0005] The main objective of this invention is to provide an obstacle detection method, device, and computer-readable storage medium, aiming to solve the technical problems of poor detection performance and high false positive rate of distant and accidental obstacles in machine vision.

[0006] To achieve the above objectives, the present invention provides an obstacle detection method, which is applied to a terminal device having an image capturing device, and the obstacle detection method includes the following steps:

[0007] The scene image acquired by the image capturing device is input into the first preset model to obtain the first image and the first probability that each pixel in the scene image is an unexpected obstacle;

[0008] The scene image is input into a second preset model to obtain a second image. Based on the second preset model and the second image, the uncertainty of each pixel in the scene image is obtained.

[0009] Based on the first image, a region of interest is determined in the scene image, and based on the uncertainty of each pixel in the region of interest, a second probability is obtained that each pixel in the region of interest is an unexpected obstacle;

[0010] Based on the first probability and the second probability, a third probability is obtained that each pixel in the region of interest is an unexpected obstacle.

[0011] Preferably, the step of obtaining a third probability that each pixel in the region of interest is an unexpected obstacle based on the first probability and the second probability includes:

[0012] The preset prior probability, the first probability, and the second probability are input into the Bayesian framework to obtain the third probability corresponding to each pixel in the region of interest.

[0013] Preferably, the first image includes a free space region and an unexpected obstacle region. The step of determining a region of interest in the scene image based on the first image, and obtaining a second probability that each pixel in the region of interest is an unexpected obstacle based on the uncertainty of each pixel in the region of interest, includes:

[0014] The free space region and the unexpected obstacle region in the first image are taken as the first region;

[0015] The first region is expanded by a preset number of pixels in both length and width to obtain the second region;

[0016] The projection area of ​​the second region in the scene image is taken as the region of interest.

[0017] The second probability is obtained based on the uncertainty of each pixel in the region of interest.

[0018] Preferably, the step of obtaining the second probability based on the uncertainty of each pixel in the region of interest includes:

[0019] A frequency distribution histogram is obtained based on the uncertainty of each pixel in the region of interest, wherein the horizontal axis of the frequency distribution histogram represents the uncertainty, and the vertical axis represents the frequency of occurrence of the pixel corresponding to the uncertainty.

[0020] Determine the left peak interval and the right peak interval of the frequency in the frequency distribution histogram, and between the left peak interval and the right peak interval, determine the uncertainty corresponding to the midpoint of the valley interval of the frequency as the threshold point;

[0021] The second probability is obtained based on the uncertainty of each pixel in the region of interest and the threshold point.

[0022] Preferably, the step of inputting the scene image into a second preset model to obtain a second image, and obtaining the uncertainty of each pixel in the scene image based on the second preset model and the second image, includes:

[0023] The scene image is semantically segmented based on the second preset model, and the second image is output, wherein the second image includes several category regions;

[0024] Based on the Softmax layer in the second preset model, the predicted probability of each pixel in the scene image belonging to each category region in the second image is determined;

[0025] The uncertainty is obtained based on the predicted probability.

[0026] Preferably, the step of obtaining the uncertainty based on the predicted probability includes:

[0027] Based on the predicted probability, the information entropy of each pixel in the scene image is obtained;

[0028] The information entropy is used as the uncertainty.

[0029] Preferably, the step of inputting the scene image acquired by the image capturing device into a first preset model to obtain a first image and a first probability that each pixel in the scene image is an unexpected obstacle includes:

[0030] Based on the first preset model, the scene image is semantically segmented and the first image is output.

[0031] Based on the Softmax layer in the first preset model, the first probability that each pixel in the scene image is an unexpected obstacle is obtained.

[0032] Preferably, before the step of inputting the scene image acquired by the image capturing device into the first preset model to obtain the first image and the first probability that each pixel in the scene image is an unexpected obstacle, the method further includes:

[0033] A pre-defined semantic segmentation model is constructed based on dilated convolution with a pre-defined dilation rate.

[0034] The preset semantic segmentation model is trained based on a preset training set;

[0035] The first preset model is obtained based on the semantic information corresponding to the preset training set.

[0036] In addition, to achieve the above objectives, the present invention also provides an obstacle detection device, the obstacle detection device comprising: a memory, a processor, and an obstacle detection program stored in the memory and executable on the processor, wherein the obstacle detection program, when executed by the processor, implements the steps of the obstacle detection method as described above.

[0037] In addition, to achieve the above objectives, the present invention also provides a computer-readable storage medium storing an obstacle detection program, which, when executed by a processor, implements the steps of the obstacle detection method as described above.

[0038] The obstacle detection method proposed in this invention involves inputting a scene image acquired by an image capturing device into a first preset model to obtain a first image and a first probability that each pixel in the scene image is an unexpected obstacle; inputting the scene image into a second preset model to obtain a second image; obtaining the uncertainty of each pixel in the scene image based on the second preset model and the second image; determining a region of interest (ROI) in the scene image based on the first image; obtaining a second probability that each pixel in the ROI is an unexpected obstacle based on the uncertainty of each pixel in the ROI; and obtaining a third probability that each pixel in the ROI is an unexpected obstacle based on the first and second probabilities. By obtaining the probabilities of each pixel in the ROI of the scene image as an unexpected obstacle from multiple sources and fusing these probabilities to obtain a new probability, the method more accurately determines whether each pixel in the ROI is an unexpected obstacle, improving the accuracy of unexpected obstacle detection, enhancing the detection effect for distant obstacles and unexpected obstacles, and reducing the false positive rate of obstacle detection. Attached Figure Description

[0039] Figure 1 This is a schematic diagram of the obstacle detection device in the hardware operating environment involved in the embodiments of the present invention;

[0040] Figure 2 This is a flowchart illustrating the first embodiment of the obstacle detection method of the present invention;

[0041] Figure 3 This is a schematic diagram of a semantic segmentation model incorporating dilated convolution in one embodiment of the obstacle detection method of the present invention;

[0042] Figure 4This is a schematic diagram of the frequency distribution histogram in one embodiment of the obstacle detection method of the present invention.

[0043] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0044] It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0045] like Figure 1 As shown, Figure 1 This is a schematic diagram of the obstacle detection device in the hardware operating environment involved in the embodiments of the present invention.

[0046] The obstacle detection device in this invention can be a PC, or a terminal device with an image capturing device such as a smartphone, tablet computer, or portable computer.

[0047] like Figure 1 As shown, the obstacle detection device may include: a processor 1001, such as a CPU; a network interface 1004; a user interface 1003; a memory 1005; and a communication bus 1002. The communication bus 1002 is used to enable communication between these components. The user interface 1003 may include a display screen or an input unit such as a keyboard; optionally, the user interface 1003 may also include a standard wired interface or a wireless interface. The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface). The memory 1005 may be high-speed RAM or non-volatile memory, such as a disk drive. Optionally, the memory 1005 may also be a storage device independent of the aforementioned processor 1001.

[0048] Optionally, the obstacle detection device may also include a camera, RF (Radio Frequency) circuitry, sensors, audio circuitry, a WiFi module, and so on. Sensors such as light sensors, motion sensors, and other sensors will not be elaborated upon here.

[0049] Those skilled in the art will understand that Figure 1 The terminal structure shown does not constitute a limitation on the obstacle detection device and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0050] like Figure 1 As shown, the memory 1005, which serves as a computer storage medium, may include an operating system, a network communication module, a user interface module, and an obstacle detection program.

[0051] exist Figure 1 In the simulation test device shown, the network interface 1004 is mainly used to connect to the backend server and communicate with the backend server; the user interface 1003 is mainly used to connect to the client (user terminal) and communicate with the client; and the processor 1001 can be used to call the obstacle detection program stored in the memory 1005.

[0052] In this embodiment, the obstacle detection device includes: a memory 1005, a processor 1001, and an obstacle detection program stored in the memory 1005 and executable on the processor 1001. When the processor 1001 calls the obstacle detection program stored in the memory 1005, it executes the steps of the obstacle detection methods in the following embodiments.

[0053] The present invention also provides a method, referring to Figure 2 , Figure 2 This is a flowchart illustrating the first embodiment of the obstacle detection method of the present invention.

[0054] In this embodiment, the obstacle detection method includes the following steps:

[0055] Step S101: Input the scene image acquired by the image capturing device into the first preset model to obtain the first image and the first probability that each pixel in the scene image is an unexpected obstacle.

[0056] In this embodiment, after acquiring a scene image through an image capturing device, the scene image is input into a first preset model. The first preset model is used to extract features from the scene image, thereby outputting a first image. At the same time, the first probability that each pixel in the scene image is an unexpected obstacle can be obtained based on the first preset model.

[0057] For example, the first preset model is a preset first semantic segmentation model. A scene image is acquired and input into this semantic segmentation model. This model can divide the scene image into three categories: free space, unexpected obstacles, and background. The first image is the scene classification image output by this semantic segmentation model. In this scene classification image, the background region includes pixel regions unrelated to unexpected obstacles (sky, buildings, etc.) and standard obstacles in a road scene (cars, pedestrians, etc.). The free space region is the pixel region of the free space where vehicles can drive, and the unexpected obstacle region represents the pixel region corresponding to unexpected obstacles. The free space region and the unexpected obstacle region are marked with different colors; the free space region can be purple, and the unexpected obstacle region... The field is red, while the background area is uncolored. In the semantic segmentation model, the Softmax layer can obtain the predicted probability of each pixel in the scene image belonging to one of the three categories. Based on the maximum value of the three predicted probabilities for each pixel, the category to which the pixel belongs can be determined. For example, if the predicted probability of any pixel corresponding to an unexpected obstacle is the maximum value among the three predicted probabilities, then the pixel is determined to belong to an unexpected obstacle and is marked in red. Finally, the category to which each pixel in the scene image belongs can be determined and marked accordingly, thereby outputting the scene classification image, i.e., the first image. In this process, the Softmax layer of the semantic segmentation model can obtain the predicted probability of each pixel belonging to an unexpected obstacle, which is used as the first probability.

[0058] It should be noted that the color markings of each category region in the scene classification image are not fixed. It is only necessary to be able to distinguish each category region. For example, free space can be covered with orange, accidental obstacle regions can be covered with green, and the background region is not uncolored, but can be covered with blue.

[0059] Step S102: Input the scene image into the second preset model to obtain the second image; based on the second preset model and the second image, obtain the uncertainty of each pixel in the scene image.

[0060] In this embodiment, the above-mentioned scene image is input into the second preset model, and the scene image is feature extracted according to the second preset model to obtain the second image. Then, the uncertainty of each pixel in the scene image is obtained according to the second preset model and the second image.

[0061] For example, the second preset model is a preset semantic segmentation model that is different from the first semantic segmentation model mentioned above. This second semantic segmentation model can divide the scene image into several regular category regions. The second image is a scene classification image divided according to the regular categories, including several regular category regions, such as free space, standard obstacles, and background regions. In the second image output by the second semantic segmentation model, each regular category region is marked with a different color. From the Softmax layer of the second semantic segmentation model, the predicted probability of each pixel in the scene image belonging to each regular category can be obtained, and the information entropy corresponding to each pixel can be determined according to the predicted probability. Finally, the information entropy is used as uncertainty to obtain the second probability that each pixel in the scene image is an unexpected obstacle.

[0062] Step S103: Based on the first image, determine the region of interest in the scene image, and based on the uncertainty of each pixel in the region of interest, obtain a second probability that each pixel in the region of interest is an unexpected obstacle;

[0063] In this embodiment, based on the category regions divided by the first preset model in the first image, the ROI (Region of Interest) can be determined in the scene image. Then, based on the uncertainty of each pixel in the ROI of the scene image, the second probability that each pixel is an unexpected obstacle is obtained.

[0064] For example, the first image includes a free space region, an unexpected obstacle region, and a background region, while the second image includes several conventional category regions such as a free space region, a standard obstacle region, and a background region. In order to filter out the interference caused by the uncertainty of the boundaries of each category region in the second image, the region of interest is directly determined in the scene image based on the free space region and the unexpected obstacle region of the first image. The free space region and the unexpected obstacle region are extended by 3 pixels in both length and width, and the extended region is taken as the region of interest. Then, based on the uncertainty of each pixel in the scene image, the uncertainty of each pixel in the region of interest is obtained. A frequency distribution histogram is then constructed based on the uncertainty. The threshold point of the uncertainty of each pixel in the region of interest is determined based on the frequency distribution histogram. Finally, the second probability that each pixel in the region of interest is an unexpected obstacle is determined based on the deviation of each uncertainty from the threshold point.

[0065] In another embodiment, since the free space region involves the free space in which the vehicle can drive, and the unexpected obstacle region is itself determined to be an unexpected obstacle by the first semantic segmentation model, the region of interest can be obtained by expanding only the pixels of the free space region. In addition, the expansion of pixels in the length and width dimensions to form the region of interest is not limited to 3 pixels, but can also be 4 pixels, 5 pixels, or other custom reasonable pixel values.

[0066] In another embodiment, the free space region and the unexpected obstacle region can be selected as the region of interest first, and then the region of interest can be expanded by a preset number of pixels in the length and width dimensions.

[0067] Step S104: Based on the first probability and the second probability, obtain a third probability that each pixel in the region of interest is an unexpected obstacle.

[0068] In this embodiment, based on the first probability that each pixel in the scene image is an unexpected obstacle, and the second probability that each pixel in the region of interest in the scene image is an unexpected obstacle, a third probability that each pixel in the region of interest is an unexpected obstacle can be obtained. For example, the first probability and the second probability that each pixel in the region of interest is an unexpected obstacle can be input into a preset Bayesian framework for inference, thereby obtaining the third probability of each pixel in the region of interest. This third probability is the final probability that each pixel in the region of interest is an unexpected obstacle.

[0069] In another embodiment, other multi-source data probability fusion methods can also be used to fuse the first probability and the second probability to obtain a third probability that each pixel is an unexpected obstacle.

[0070] The obstacle detection method proposed in this embodiment involves inputting a scene image acquired by an image capturing device into a first preset model to obtain a first image and a first probability that each pixel in the scene image is an unexpected obstacle; inputting the scene image into a second preset model to obtain a second image; and obtaining the uncertainty of each pixel in the scene image based on the second preset model and the second image. Then, based on the first image, a region of interest (ROI) is determined in the scene image, and based on the uncertainty of each pixel in the ROI, a second probability that each pixel in the ROI is an unexpected obstacle is obtained. Finally, based on the first probability and the second probability, a third probability that each pixel in the ROI is an unexpected obstacle is obtained. By obtaining the probabilities of each pixel in the ROI of the scene image as an unexpected obstacle from multiple sources and fusing these probabilities to obtain a new probability, the method more accurately determines whether each pixel in the ROI is an unexpected obstacle, improving the accuracy of unexpected obstacle detection, enhancing the detection effect for distant obstacles and unexpected obstacles, and reducing the false positive rate of obstacle detection.

[0071] Based on the first embodiment, a second embodiment of the obstacle detection method of the present invention is proposed. In this embodiment, step S104 includes:

[0072] Step S201: Input the preset prior probability, the first probability, and the second probability into the Bayesian framework to obtain the third probability corresponding to each pixel in the region of interest.

[0073] In this embodiment, based on the Bayesian framework, the first probability and the second probability that each pixel in the region of interest is an unexpected obstacle are input, along with a preset prior probability. Probability fusion is then performed to obtain the third probability that each pixel in the region of interest is an unexpected obstacle. For example, the preset formula corresponding to the Bayesian framework is as follows:

[0074]

[0075] Where, p pr Let p be the prior probability that any pixel is an unexpected obstacle. SS p represents the first probability that the pixel is an unexpected obstacle. Un p(o) represents the second probability that the pixel is an unexpected obstacle, and p(o) represents the third probability that the pixel is an unexpected obstacle.

[0076] The obstacle detection method proposed in this embodiment obtains a third probability corresponding to each pixel in the region of interest by inputting a preset prior probability, a first probability, and a second probability into a Bayesian framework. Joint decision-making is performed through probability fusion to obtain the probability that each pixel in the region of interest is an unexpected obstacle. This allows the second probability to supplement the first probability. Based on the fused probabilities, the method more accurately determines whether each pixel in the region of interest is an unexpected obstacle, improving the accuracy of unexpected obstacle detection, enhancing the detection performance for both distant and unexpected obstacles, and reducing the false positive rate.

[0077] Based on the first embodiment, a third embodiment of the obstacle detection method of the present invention is proposed. In this embodiment, the first image includes a free space region and an unexpected obstacle region. Step S103 includes:

[0078] Step S301: The free space region and the unexpected obstacle region in the first image are taken as the first region;

[0079] Step S302: Expand the first region by a preset number of pixels in both length and width to obtain the second region;

[0080] Step S303: The projection area of ​​the second region in the scene image is taken as the region of interest;

[0081] Step S304: Obtain the second probability based on the uncertainty of each pixel in the region of interest.

[0082] In this embodiment, the scene image is input into a first preset model, and the output first image includes a free space region and an unexpected obstacle region. The free space region and the unexpected obstacle region are used as the first region. The first region is expanded by a preset number of pixels, and the expanded region is used as the second region. Then, the projection of the second region onto the scene image is used as the region of interest. Finally, based on the uncertainty of each pixel in the region of interest, a second probability is obtained that each pixel in the region of interest is an unexpected obstacle. For example, the first preset model is a first semantic segmentation model, which extracts features from the scene image and outputs the first image, which includes a free space region and unexpected obstacles. The system defines an object region and a background region. The first region comprises all pixels within the free space region and the unexpected obstacle region. This first region is then expanded by a preset number of pixels in both length and width (e.g., 3 pixels). The expanded region is then designated as the second region. This second region is projected onto the scene image, and the area covered by the projection is the region of interest (ROI). Specifically, when the first image and the scene image overlap, the corresponding area of ​​the second region in the scene image is the ROI. Finally, based on the uncertainty of each pixel within the ROI, a second probability is obtained that each pixel within the ROI is an unexpected obstacle.

[0083] The obstacle detection method proposed in this embodiment uses the free space region and the unexpected obstacle region in the first image as the first region; then, the first region is expanded by a preset number of pixels in length and width to obtain a second region; the projection area of ​​the second region in the scene image is then used as the region of interest; finally, the second probability is obtained based on the uncertainty of each pixel in the region of interest. By using pixel expansion, the uncertainty of the background region and the boundaries of each category region is filtered out as much as possible to eliminate interference from the acquisition of the second probability, making the second probability more accurate as a supplement to the first probability, reducing the false positive rate of obstacle detection, improving the accuracy of the subsequently obtained third probability, and improving the detection effect for distant obstacles and unexpected obstacles.

[0084] Based on the third embodiment, a fourth embodiment of the obstacle detection method of the present invention is proposed. In this embodiment, step S304 includes:

[0085] Step S401: Obtain the corresponding frequency distribution histogram based on the uncertainty of each pixel in the region of interest, wherein the horizontal axis of the frequency distribution histogram represents the uncertainty and the vertical axis represents the frequency of occurrence of the pixel corresponding to the uncertainty.

[0086] Step S402: Determine the left peak interval and the right peak interval of the frequency in the frequency distribution histogram; between the left peak interval and the right peak interval, determine the uncertainty corresponding to the midpoint of the valley interval of the frequency as the threshold point.

[0087] Step S403: Based on the uncertainty of each pixel in the region of interest and the threshold point, obtain the second probability.

[0088] In this embodiment, firstly, a frequency distribution histogram is obtained based on the uncertainty of each pixel in the region of interest. The horizontal axis of this frequency distribution histogram represents the uncertainty, and the vertical axis represents the frequency of occurrence of the pixel corresponding to each uncertainty. Then, based on the first peak interval on the left and the first peak interval on the right in the frequency distribution histogram, the uncertainty corresponding to the midpoint of the valley interval between the two peak intervals is determined as the threshold point. Based on the deviation between the uncertainty and the threshold point, the second probability that each pixel in the region of interest is an unexpected obstacle can be obtained. For example, when using semantic segmentation methods to detect unexpected obstacles, since the output probability of the unknown class (unexpected obstacle) is more dispersed across multiple category regions, it has higher uncertainty compared to the known class (regular category). Therefore, uncertainty can be used to distinguish between the known class and the unknown class, thereby detecting unexpected obstacles. Figure 4 , Figure 4 The frequency distribution histogram shown has uncertainty (u) on the horizontal axis and the frequency (f) of pixels with a certain uncertainty on the vertical axis. This histogram reflects the general distribution of uncertainty. According to the histogram, the known class is more numerous and mainly distributed on the left, while the unknown class is fewer and mainly distributed on the right. If we consider the known and unknown classes as two different sets, then the point with the loosest coupling between the two sets should be selected as the threshold point for segmentation. Therefore, the leftmost and rightmost frequency peak intervals are determined in the frequency distribution histogram. Then, the uncertainty corresponding to the midpoint of the valley interval between these two peak intervals is used as the threshold point. Finally, the second probability is obtained based on the deviation between the uncertainty of each pixel in the region of interest and the threshold point. Assuming the probability of the threshold point is 50%, the formula for the second probability is as follows:

[0089]

[0090] Among them, u t The uncertainty at the threshold point, u i Let be the uncertainty of pixel i.

[0091] The obstacle detection method proposed in this embodiment obtains a frequency distribution histogram based on the uncertainty of each pixel in the region of interest (ROI). The horizontal axis of the frequency distribution histogram represents the uncertainty, and the vertical axis represents the frequency of the pixel corresponding to the uncertainty. Then, the left and right peak intervals of the frequency in the frequency distribution histogram are determined. Between these intervals, the midpoint of the frequency trough interval is identified as the uncertainty threshold point. Finally, based on the uncertainty of each pixel in the ROI and the threshold point, a second probability is obtained. Accidental obstacles are segmented according to the threshold point of the uncertainty of each pixel in the ROI to obtain the second probability. This second probability supplements the first probability, addressing the shortcomings of the first probability obtained by the first semantic segmentation model in detecting distant obstacles and improving the accuracy of accidental obstacle detection.

[0092] Based on the first embodiment, a fifth embodiment of the obstacle detection method of the present invention is proposed. In this embodiment, step S102 includes:

[0093] Step S501: Perform semantic segmentation on the scene image based on the second preset model and output the second image, wherein the second image includes several category regions;

[0094] Step S502: Based on the Softmax layer in the second preset model, determine the predicted probability that each pixel in the scene image belongs to each category region in the second image;

[0095] Step S503: Obtain the uncertainty based on the predicted probability.

[0096] In this embodiment, the second preset model is a preset second semantic segmentation model. After inputting a scene image, the second semantic segmentation model extracts features from the scene image, which can divide the scene image into several regular category regions and output them as a second image. According to the Softmax layer in the second semantic segmentation model, the predicted probability of each pixel in the scene image belonging to each regular category region in the second image can be obtained, and the uncertainty of each pixel can be obtained according to the predicted probability. For example, the DeepLabV3+ model is selected, and the Cityscapes dataset is used as the model training set to construct the second semantic segmentation model for segmenting the regular categories in the scene image. According to the Softmax layer of the second semantic segmentation model, the predicted probability of each pixel in the scene image belonging to each regular category can be obtained, and the uncertainty of each pixel can be obtained according to the predicted probability.

[0097] In another embodiment, the Cityscapes dataset used by the second semantic segmentation model can complement the Lost and Found dataset used by the first semantic segmentation model to some extent, thereby improving the accuracy of unexpected obstacle detection.

[0098] The obstacle detection method proposed in this embodiment performs semantic segmentation on the scene image based on a second preset model and outputs a second image, wherein the second image includes several category regions. Based on the Softmax layer in the second preset model, it determines the predicted probability that each pixel in the scene image belongs to each category region in the second image; and thus obtains the uncertainty based on the predicted probability. By constructing a second semantic segmentation model to perform semantic segmentation on the regular categories of the scene image, the uncertainty of each pixel is obtained, allowing the second probability obtained based on the uncertainty to complement the first probability, improving the accuracy of unexpected obstacle detection and enhancing the detection effect of distant and unexpected obstacles.

[0099] Based on the fifth embodiment, a sixth embodiment of the obstacle detection method of the present invention is proposed. In this embodiment, step S503 includes:

[0100] Step S601: Based on the predicted probability, obtain the information entropy of each pixel in the scene image;

[0101] Step S602: Use the information entropy as the uncertainty.

[0102] In this embodiment, based on the predicted probability of each pixel in the scene image belonging to each conventional category in the second semantic segmentation model described above, the information entropy of each pixel can be obtained, and the information entropy of any pixel can be used as the uncertainty of that pixel.

[0103] Specifically, information entropy can describe the uncertainty of various possible events in an information source. Therefore, this method uses information entropy as a metric for pixel uncertainty. For example, the information entropy of each pixel is obtained based on the predicted probability of each pixel in the scene image, and the information entropy is used as the uncertainty of each pixel. The formula is as follows:

[0104]

[0105] Where N is the total number of categories, c is the category index, and logp i,c The base is e by default, p i,c Let u represent the predicted probability that the i-th pixel in the image belongs to category c. This predicted probability can be obtained from the Softmax layer of the semantic segmentation model. Therefore, the uncertainty of each pixel is u. i .

[0106] The obstacle detection method proposed in this embodiment obtains the information entropy of each pixel in the scene image based on the predicted probability; and uses the information entropy as the uncertainty. By using the information entropy as a metric for the uncertainty of each pixel, the uncertainty of each pixel in the scene image is obtained. This allows for the subsequent determination of a second probability that each pixel in the region of interest is an unexpected obstacle based on the uncertainty, improving the accuracy of the second probability. Furthermore, the second probability can supplement the first probability, improving the detection performance for distant obstacles and unexpected obstacles.

[0107] Based on the first embodiment, a seventh embodiment of the obstacle detection method of the present invention is proposed. In this embodiment, step S101 includes:

[0108] Step S701: Perform semantic segmentation on the scene image based on the first preset model and output the first image;

[0109] Step S702: Based on the Softmax layer in the first preset model, obtain the first probability that each pixel in the scene image is an unexpected obstacle.

[0110] In this embodiment, the first preset model is a preset first semantic segmentation model. The scene image is input into the first semantic segmentation model for feature extraction, thereby outputting a first image. Furthermore, based on the Softmax layer in the first semantic segmentation model, the first probability that each pixel in the scene image is an unexpected obstacle can be directly obtained. For example, the first semantic segmentation model is a semantic segmentation network trained using the Lost and Found dataset. This dataset contains highly complex road scenes and includes various unexpected obstacles. This dataset can complement the Cityscapes dataset used by the second semantic segmentation model to a certain extent. Through the first semantic segmentation model, feature extraction is performed on the scene image for three categories: free space, unexpected obstacles, and background. The first image is a scene classification image output by this semantic segmentation model. In this scene classification image, the background region includes pixel regions unrelated to unexpected obstacles (sky, buildings, etc.) and standard obstacles in the road scene (cars, pedestrians, etc.), while the free space region is the pixel region of the free space where vehicles can drive, and the unexpected obstacle region represents the pixel region corresponding to the unexpected obstacle. In the first image, the free space region and the unexpected obstacle region are marked with different colors, such as purple for the free space region and red for the unexpected obstacle region. The background region is uncolored. In the semantic segmentation model, the Softmax layer obtains the predicted probability of each pixel in the scene image belonging to one of the three categories. Based on the maximum value of the three predicted probabilities for each pixel, the category to which the pixel belongs can be determined. For example, if the predicted probability of any pixel belonging to an unexpected obstacle is the maximum value among the three predicted probabilities, then the pixel is determined to belong to an unexpected obstacle and is covered in red in the first image. The Softmax layer uses the predicted probability of each pixel belonging to an unexpected obstacle as the first probability. The Softmax function can be specifically expressed as the following formula:

[0111]

[0112] Among them, Z j This represents the output of category j after passing through the layer preceding the Softmax layer, where c is the category index and N is the total number of categories. Based on this Softmax function, the first probability p... SS This can be expressed as the following formula:

[0113]

[0114] Z1, Z2, and Z3 represent the outputs of the background category, free space category, and unexpected obstacle category, respectively, after passing through the layer preceding the Softmax layer.

[0115] The obstacle detection method proposed in this embodiment performs semantic segmentation on the scene image based on the first preset model and outputs the first image; and obtains the first probability that each pixel in the scene image is an unexpected obstacle based on the Softmax layer in the first preset model. Semantic segmentation of the scene image based on the first preset model and outputting the first image; by reasonably defining the pixel categories of the model training set, the characteristics of unexpected obstacles in the scene image are highlighted, thereby achieving the detection of unexpected obstacles, improving the accuracy of the first probability, and making the subsequent third probability obtained through probability fusion more accurate, thus improving the detection effect for distant obstacles and unexpected obstacles.

[0116] Based on the above embodiments, an eighth embodiment of the obstacle detection method of the present invention is proposed. In this embodiment, before step S101, the method further includes:

[0117] Step S801: Construct a preset semantic segmentation model based on dilated convolution with a preset dilation rate;

[0118] Step S802: Train the preset semantic segmentation model based on the preset training set;

[0119] Step S803: Obtain the first preset model based on the semantic information corresponding to the preset training set.

[0120] In this embodiment, the first preset model is a preset first semantic segmentation model, which is a preset semantic segmentation model trained on a preset training set. This semantic segmentation model introduces dilated convolutions with a preset dilation rate. For example, DeepLabV3+ is selected as the preset semantic segmentation model, and ResNet-50 is used as the backbone feature extraction network. This preset semantic segmentation model, combined with an encoder-decoder structure and ASPP, can effectively recover the edge information of the image and learn multi-scale features. Figure 3 This pre-defined semantic segmentation model introduces dilated convolution, which, compared to standard convolution, increases the receptive field of the model and extracts richer spatial information about the target. For two-dimensional signals such as images, the mapping formula between the input x and output y of this dilated convolution is as follows:

[0121]

[0122] Where i is the index of a single element in the feature map; d is the dilation rate of the dilated convolution; and ω[k] represents a convolution kernel of size k.

[0123] Furthermore, by changing the dilated convolutions with original dilation rates of 6, 12, and 18 in the preset semantic segmentation model to 4, 8, and 12 respectively, low-resolution feature maps can be effectively extracted. Finally, the preset semantic segmentation model is trained using the Lost and Found dataset as the preset training set to obtain the first semantic segmentation model, which serves as the first preset model.

[0124] The obstacle detection method proposed in this embodiment constructs a preset semantic segmentation model using dilated convolution with a preset dilation rate; then trains the preset semantic segmentation model using a preset training set; and obtains the first preset model based on the semantic information corresponding to the preset training set. By introducing dilated convolution to construct the preset semantic segmentation model, the interference of large background regions on feature extraction is effectively reduced. Simultaneously, the feature maps in the model retain the boundary information of the target as much as possible without adding excessive computational parameters. Training the preset semantic segmentation model using a preset training set including various unexpected obstacles to obtain the first model improves the accuracy of subsequent unexpected obstacle detection and makes the third probability obtained through probability fusion more accurate, thus improving the detection effect for unexpected obstacles.

[0125] Furthermore, this embodiment of the invention also proposes an obstacle detection device, which includes: a memory, a processor, and an obstacle detection program stored in the memory and executable on the processor. When the obstacle detection program is executed by the processor, it implements the steps of the obstacle detection method as described above.

[0126] Furthermore, the present invention also provides a computer-readable storage medium, preferably a computer-readable storage medium, on which an obstacle detection program is stored, wherein the obstacle detection program, when executed by a processor, implements the steps of the obstacle detection method as described above.

[0127] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or system. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or system that includes that element.

[0128] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0129] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0130] The above are merely preferred embodiments of the present invention and do not limit the patent scope of the present invention. Any equivalent structural or procedural transformations made based on the content of the present invention's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of the present invention.

Claims

1. An obstacle detection method, characterized in that, The obstacle detection method is applied to a terminal device with an image capturing device, and the obstacle detection method includes the following steps: The scene image acquired by the image capturing device is input into the first preset model to obtain the first image and the first probability that each pixel in the scene image is an unexpected obstacle; The scene image is input into a second preset model to obtain a second image. Based on the second preset model and the second image, the uncertainty of each pixel in the scene image is obtained. Based on the first image, a region of interest (ROI) is determined in the scene image. Based on the uncertainty of each pixel in the ROI, a second probability is obtained that each pixel in the ROI is an unexpected obstacle. The first image includes a free space region and an unexpected obstacle region. This step includes: taking the free space region and the unexpected obstacle region in the first image as a first region; expanding the first region by a preset number of pixels in length and width to obtain a second region; taking the projection region of the second region in the scene image as the ROI; and obtaining the second probability based on the uncertainty of each pixel in the ROI. Based on the first probability and the second probability, a third probability is obtained that each pixel in the region of interest is an unexpected obstacle; The step of obtaining the second probability based on the uncertainty of each pixel in the region of interest includes: A frequency distribution histogram is obtained based on the uncertainty of each pixel in the region of interest, wherein the horizontal axis of the frequency distribution histogram represents the uncertainty, and the vertical axis represents the frequency of occurrence of the pixel corresponding to the uncertainty. Determine the left peak interval and the right peak interval of the frequency in the frequency distribution histogram, and determine the uncertainty corresponding to the midpoint of the valley interval between the left peak interval and the right peak interval as the threshold point; The second probability is obtained based on the uncertainty of each pixel in the region of interest and the threshold point.

2. The obstacle detection method as described in claim 1, characterized in that, The step of obtaining a third probability that each pixel in the region of interest is an unexpected obstacle based on the first probability and the second probability includes: The preset prior probability, the first probability, and the second probability are input into the Bayesian framework to obtain the third probability corresponding to each pixel in the region of interest.

3. The obstacle detection method as described in claim 1, characterized in that, The step of inputting the scene image into a second preset model to obtain a second image, and obtaining the uncertainty of each pixel in the scene image based on the second preset model and the second image, includes: The scene image is semantically segmented based on the second preset model, and the second image is output, wherein the second image includes several category regions; Based on the Softmax layer in the second preset model, the predicted probability of each pixel in the scene image belonging to each category region in the second image is determined; The uncertainty is obtained based on the predicted probability.

4. The obstacle detection method as described in claim 3, characterized in that, The step of obtaining the uncertainty based on the predicted probability includes: Based on the predicted probability, the information entropy of each pixel in the scene image is obtained; The information entropy is used as the uncertainty.

5. The obstacle detection method as described in claim 1, characterized in that, The step of inputting the scene image acquired by the image capturing device into the first preset model to obtain the first image and the first probability that each pixel in the scene image is an unexpected obstacle includes: Based on the first preset model, the scene image is semantically segmented and the first image is output. Based on the Softmax layer in the first preset model, the first probability that each pixel in the scene image is an unexpected obstacle is obtained.

6. The obstacle detection method as described in claim 5, characterized in that, Before the step of inputting the scene image acquired by the image capturing device into the first preset model to obtain the first image and the first probability that each pixel in the scene image is an unexpected obstacle, the method further includes: A pre-defined semantic segmentation model is constructed based on dilated convolution with a pre-defined dilation rate. The preset semantic segmentation model is trained based on a preset training set; The first preset model is obtained based on the semantic information corresponding to the preset training set.

7. An obstacle detection device, characterized in that, The obstacle detection device includes: a memory, a processor, and an obstacle detection program stored in the memory and executable on the processor, wherein the obstacle detection program, when executed by the processor, implements the steps of the obstacle detection method as described in any one of claims 1 to 6.

8. A computer-readable storage medium, characterized in that, The readable storage medium stores an obstacle detection program, which, when executed by a processor, implements the steps of the obstacle detection method as described in any one of claims 1 to 6.

Images