Obstacle detection method, device, equipment, storage medium and program product

Abstract

The present application discloses an obstacle detection method, apparatus, device, storage medium and program product. The method includes: obtaining an RGB image and a first depth image of a target detection area; performing object detection on the RGB image to obtain one or more suspected obstacle information in the target detection area; when it is determined that a certain suspected obstacle information is light interference obstacle information, obtaining depth information of the light interference obstacle information based on the first depth image, setting the depth information in the first depth image to a specified threshold, and generating a second depth image from which the depth information of the light interference obstacle information is removed; based on the second depth image and the RGB image, generating point cloud data corresponding to each target obstacle information. The present application classifies highlight areas to eliminate false detections caused by light interference, thereby effectively solving the problem of high false detection rate of RGBD cameras when encountering strong light interference and improving the efficiency of obstacle detection.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of image data processing, and in particular to an obstacle detection method, apparatus, device, storage medium, and program product. Background Art

[0002] Currently, one of the robot obstacle detection methods is to measure distance through RGBD cameras and use binocular stereo matching to obtain the depth information of pixel points. Together with the three-dimensional point cloud information calculated with the camera's intrinsic parameters, it can be used to build maps, plan obstacle avoidance, etc.

[0003] However, when using this method to detect obstacles outdoors, the binocular stereo matching algorithm will fail in certain scene areas if it encounters light interference. The point cloud formed in this area is likely to have false detections in terms of distance, orientation, and semantics, seriously affecting the obstacle avoidance effect. Summary of the Invention

[0004] The present application provides an obstacle detection method, apparatus, device, storage medium and program product to overcome the problem of serious false detection in specific scenarios when detecting obstacles based on RGBD cameras in current technology. It processes false detections caused by strong light interference, filters out strong light interference, and thus improves detection efficiency.

[0005] In a first aspect, an embodiment of the present application provides an obstacle detection method, the method comprising:

[0006] Acquire an RGB image and a first depth image of the target detection area;

[0007] Performing object detection on the RGB image to obtain information of one or more suspected obstacles in the target detection area;

[0008] Determining whether each piece of suspected obstacle information is light interference obstacle information;

[0009] If the suspected obstacle information is light interference obstacle information, obtaining depth information of the light interference obstacle information based on the first depth image, setting the depth information to a specified threshold in the first depth image, and generating a second depth image from which the depth information of the light interference obstacle information is removed;

[0010] Based on the second depth image and the RGB image, point cloud data corresponding to each target obstacle information is generated, wherein the target obstacle information is obstacle information determined from the suspected obstacle information remaining after removing the light interference obstacle information.

[0011] In a second aspect, an embodiment of the present application provides an obstacle detection device, the device comprising:

[0012] An image acquisition module is used to acquire an RGB image and a first depth image of the target detection area;

[0013] a suspected obstacle information acquisition module, configured to perform object detection on the RGB image to obtain information of one or more suspected obstacles in the target detection area;

[0014] A light interference obstacle information recognition and judgment module is used to judge whether each piece of suspected obstacle information is light interference obstacle information;

[0015] a second depth image generating module configured to, when suspected obstacle information is light interference obstacle information, obtain depth information of the light interference obstacle information based on the first depth image, set the depth information to a specified threshold in the first depth image, and generate a second depth image from which the depth information of the light interference obstacle information is removed;

[0016] A point cloud data generation module is used to generate point cloud data corresponding to each target obstacle information based on the second depth image and the RGB image, wherein the target obstacle information is obstacle information determined from the suspected obstacle information remaining after removing the light interference obstacle information.

[0017] In a third aspect, an embodiment of the present application further provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned method when executing the program.

[0018] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium on which a computer program is stored, and the program implements the above-mentioned method when executed by a processor.

[0019] In a fifth aspect, an embodiment of the present application further provides a computer program product, comprising computer executable instructions, which are used to implement the above method when executed.

[0020] This application has the following beneficial effects:

[0021] By acquiring an RGB image and a first depth image of a target detection area, object detection is performed on the RGB image to obtain information on one or more suspected obstacles in the target detection area. It is then determined whether each piece of suspected obstacle information is light interference obstacle information. If a piece of suspected obstacle information is light interference obstacle information, depth information of the light interference obstacle information is acquired based on the first depth image. The depth information is set to a specified threshold in the first depth image to generate a second depth image after removing the depth information of the light interference obstacle information. Based on the second depth image and the RGB image, point cloud data corresponding to each piece of target obstacle information is generated. The target obstacle information is obstacle information determined from the suspected obstacle information remaining after removing the light interference obstacle information. This embodiment classifies highlight areas to eliminate false detections caused by light interference, effectively solving the problem of high false detection rates of RGBD cameras when encountering strong light interference and improving the efficiency of obstacle detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 This is a flowchart of an obstacle detection method embodiment provided in Example 1 of the present application;

[0023] Figure 2 This is a schematic diagram of a CenterNet network framework provided in Example 1 of the present application;

[0024] Figure 3 This is a schematic diagram of a classification network structure provided in Example 1 of the present application;

[0025] Figure 4 This is a schematic diagram of an obstacle point cloud projection provided in Example 1 of the present application;

[0026] Figure 5 This is a schematic diagram of an obstacle detection process provided in Example 1 of the present application;

[0027] Figure 6 This is a structural block diagram of an obstacle detection device embodiment provided in Example 2 of the present application;

[0028] Figure 7 This is a structural diagram of an electronic device provided in Example 3 of the present application. DETAILED DESCRIPTION

[0029] The present application will be further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are intended only to illustrate the present application and are not intended to limit the present application. It should also be noted that, for ease of description, the accompanying drawings only illustrate portions of the present application, not all of the structures.

[0030] Example 1

[0031] Figure 1 This is a flowchart of an obstacle detection method embodiment provided in Example 1 of the present application.

[0032] The embodiments of the present application are mainly aimed at situations where, in an environment with light interference, for example, strong sunlight is reflected on a smooth ground or water surface outdoors, and the light just enters the field of view of the binocular camera. The binocular camera is very sensitive to ambient light, and the distance and orientation of the point cloud formed in this light interference area may be misdetected.

[0033] This embodiment addresses false detections caused by light interference. After acquiring RGB images and depth images, the images are processed to obtain semantic information and depth information of obstacles. Combined with the reflection and refraction characteristics of light, areas of light interference are identified in the RGB image. By matching the RGB image with the depth image, false detections in the depth image are filtered out, thereby ensuring that the subsequently obtained point cloud data is more accurate and reducing the false detection rate.

[0034] like Figure 1 As shown, this embodiment may include the following steps:

[0035] Step 110: Acquire an RGB image and a first depth image of the target detection area;

[0036] In this step, an RGB image and a first depth image of the target detection area are acquired by an RGBD camera. The depth image is an image channel containing information or information related to the surface distance of scene objects from the viewpoint. The pixel value of each pixel in the depth image is the distance from the image collector to each point in the scene, where the distance here is also called depth.

[0037] For RGB images, each image has multiple color pixel groups, each of which is a three-value group. These three values ​​correspond to the red, green, and blue components of the RGB image at a specific spatial location. In other words, RGB images are obtained by varying the red, green, and blue color channels and superimposing them to create a variety of colors.

[0038] When the RGB image and the first depth image of the target detection area are obtained by the RGBD camera, they can be real-time scene images of the target detection area. After obtaining the RGB image of the target area, obstacles in the image can be determined through deep neural network learning, such as the specific name and category of the obstacle. Combined with the depth information in the first depth image, the specific location information and semantic information of the obstacle can be obtained.

[0039] Step 120: Perform object detection on the RGB image to obtain information of one or more suspected obstacles in the target detection area.

[0040] In this step, the RGB image can be input into the image target detection network to identify obstacles in the RGB image and obtain RGB image information such as the specific semantic information and location information of each obstacle through deep learning of the target detection network.

[0041] In a specific implementation, the RGB image can be input into the CenterNet network (target detection network), such as Figure 2 As shown in the CenterNet network framework, the RGB image is resized and input into the backbone network for feature extraction. After obtaining the feature map, it is input into three convolutional neural networks for three convolution calculations to obtain the heat map, center point offset, width and height of the target respectively. The center point coordinates are obtained from the heat map, and thus the center point position, width and height of each obstacle in the image are obtained.

[0042] The CenterNet architecture can use commonly used backbone networks, such as Resnet-18, ResNet-101, DLA-34, Hourglass-104, etc. As an embodiment of the present application, a CenterNet network with Resnet-18 as the backbone network is used. The number 18 here refers to the total of 18 convolutional layers and fully connected layers, and Resnet is the name of the network.

[0043] Deep neural networks are generally difficult to train. The residual learning framework can alleviate this difficulty. Residual networks are also easier to optimize, and increasing the number of network layers helps improve accuracy. This allows portions of the original input to be directly transferred to the next layer, bypassing matrix multiplication and nonlinear transformations. This is like building numerous information highways within a deep network. The ResNet backbone network changes the learning objective from learning the residuals rather than the full output. This addresses the information loss and attrition issues associated with traditional convolutional or fully connected layers. By directly transferring information from input to output, information integrity is protected to a certain extent.

[0044] In this embodiment, the image is resized to 512*512, the size of the feature map obtained by Resnet-18 is 128*128*256, the predicted heat map size is 128*128*80 (representing 80 categories), the predicted center point offset size is 128*128*2 (2 represents x, y), and the predicted width and height sizes are 128*128*2 (2 represents width and height).

[0045] In addition, to improve the efficiency of obstacle recognition, we can fully utilize the performance of the Nvidia TX2 hardware platform. The CenterNet network inference based on the Nvidia TX2 hardware platform can improve the speed of obstacle detection and classification in RGB images, achieving real-time recognition.

[0046] Step 130 : Determine whether each piece of suspected obstacle information is light interference obstacle information.

[0047] In this step, the obstacle information obtained through the inference of the target detection network such as the CenterNet network may be misdetected due to interference from light, such as strong light. Therefore, the obstacle information obtained after the RGB image is processed by the target detection network can be first regarded as suspected obstacle information. From the suspected obstacle information, it is determined whether there is interference from strong light, that is, whether there is light interference obstacle information, so that the obstacle information used later can be more accurate.

[0048] In one embodiment, the suspected obstacle information includes location information, and step 130 includes the following steps:

[0049] Step 130 - 1 , converting the RGB image into a grayscale image.

[0050] In this step, grayscale images represent each image point using varying degrees of black saturation, with each pixel having a grayscale value. Converting the RGB values ​​of an RGB image to grayscale values ​​effectively converts the perception of color into a perception of brightness. The conversion process involves sequentially reading the R, G, and B values ​​of each pixel, calculating the grayscale value, and assigning it to the corresponding position in the image. This conversion is completed after all pixels have been traversed. The calculation formula is: Gray (Grayscale value) = 0.299*R + 0.587*G + 0.114*B.

[0051] Step 130 - 2 : Match the location information with the grayscale image to determine the grayscale area of ​​each suspected obstacle in the grayscale image.

[0052] In this step, because the pixels of the RGB image correspond one-to-one to the pixels of the grayscale image, the grayscale area of ​​each suspected obstacle in the grayscale image can be determined in the grayscale image based on the position information of the suspected obstacles identified from the RGB image.

[0053] Step 130 - 3 , selecting a target grayscale region from the grayscale regions, and inputting the target grayscale region into a pre-generated classification model.

[0054] In this step, since a larger grayscale value for a pixel in the grayscale image indicates a brighter pixel, and vice versa, a smaller grayscale value indicates a darker pixel, the grayscale value of a region consisting of multiple pixels can be determined by taking the average grayscale value of all pixels in the region. Since light interference obstacles are generally caused by reflection or refraction of strong light, the average grayscale value of the grayscale region of a light interference obstacle will be higher than the average grayscale values ​​of other obstacle grayscale regions. The purpose of selecting a target grayscale region from the grayscale region is to pre-select a number of suspected obstacle grayscale regions with a higher probability of being light interference obstacles from the suspected obstacles using the average grayscale value, thereby enabling faster subsequent identification of light interference obstacles.

[0055] Several target grayscale areas can be sequentially loaded into a pre-generated classification network, which can be a binary classification network, that is, the output classification results can be two results: the grayscale area of ​​the light interference obstacle and the grayscale area of ​​the non-light interference obstacle.

[0056] In the specific implementation, the characteristics and training methods of the classification network are:

[0057] Sample set: Prepare two types of sample sets: Class 0 is normal obstacles without interference, such as trash cans, stones, etc. Class 1 is light interference obstacles, such as outdoor water surface reflections, indoor light shining on smooth mirror reflections, etc.

[0058] The network structure can be referenced Figure 3 The schematic diagram of the classification network structure shown includes:

[0059] Input layer: Set the input image to 48*48*1;

[0060] Hidden layer:

[0061] The first layer consists of a convolutional layer, a normalization layer, an activation function layer, and a maximum pooling layer. Features are extracted through a 32*3*3*3 convolution with a stride of 1 and a pad of 1. Batch Normalization is then used to normalize the features. The features are nonlinearly mapped using the ReLU activation function (linear rectification function). Noise is filtered and downsampled using a 2*2 maximum pooling layer with a stride of 2 and a pad of 0, and the downsampling is halved.

[0062] The second layer includes a convolution layer, a normalization layer, an activation function layer, and a maximum pooling layer. Features are extracted through a 64*32*3*3 convolution with a stride of 1 and a pad of 1. BatchNormalize is used for normalization, and the features are nonlinearly mapped using the ReLU activation function. Noise is filtered and downsampled through a 2*2 maximum pooling layer with a stride of 2 and a pad of 0.

[0063] The third layer includes a convolution layer, a normalization layer, an activation function layer, and a maximum pooling layer. Features are extracted through a 128*64*3*3 convolution with a stride of 1 and a pad of 1. BatchNormalize is used for normalization, and the features are nonlinearly mapped using the ReLU activation function. Noise is filtered and downsampled through a 2*2 maximum pooling layer with a stride of 2 and a pad of 0.

[0064] The fourth layer includes a convolution layer, a normalization layer, an activation function layer, and a maximum pooling layer. Features are extracted through a 256*128*3*3 convolution layer with a stride of 1 and a pad of 1. Batch Normalization is then used for normalization. The features are nonlinearly mapped using the ReLU activation function. Noise is filtered and downsampled using a 2*2 maximum pooling layer with a stride of 2 and a pad of 0.

[0065] The fifth layer includes a convolution layer, a normalization layer, and an activation function layer. It extracts features through a 512*256*3*3 convolution with a stride of 1 and a pad of 0. It is then normalized using BatchNormalize and nonlinearly mapped using the ReLU activation function.

[0066] The sixth layer: includes a convolution layer, a normalization layer, and an activation function layer; classification is performed through a 2*512*1*1 convolution with a stride of 1 and a pad of 0, and then normalized through BatchNormalize. The features are nonlinearly mapped through the Relu activation function to obtain the final classification result;

[0067] Output layer: A binary classifier that outputs two categories: 0 indicates that the input image is an obstacle, and 1 indicates that the input image is a light interference obstacle. False detections caused by light interference can be confirmed based on the classifier output.

[0068] It should be noted that the input image mentioned in the classification network can refer to the target grayscale area that is loaded into the classification network in sequence.

[0069] Step 130-4, obtaining the obstacle type of each target grayscale area output by the classification model;

[0070] In this step, the obstacle type to which each input target grayscale area belongs is determined through the output result of the classification model output layer, that is, it can be confirmed whether it is a light interference obstacle.

[0071] Step 130 - 5 : determining light interference obstacle information from the suspected obstacle information according to the obstacle type.

[0072] In this step, based on the target grayscale area output as Class 1 in the classification network, the specific position of the light interference obstacle can be determined by corresponding the position of the target grayscale area in the grayscale image to the position in the RGB image, thereby determining the light interference obstacle from one or more suspected obstacles, and determining the light interference obstacle information from the suspected obstacle information.

[0073] In one embodiment, selecting a target grayscale region from grayscale regions in step 130-3 includes the following steps:

[0074] Get the grayscale statistics of each grayscale area;

[0075] The grayscale regions are sorted according to grayscale statistical data, and several grayscale regions that are ranked first are used as target grayscale regions.

[0076] In this process, the grayscale statistics of all suspected obstacle grayscale areas can be determined first. The grayscale statistics can include the average grayscale value of each grayscale area. The average grayscale value is counted and sorted from large to small. The suspected obstacle grayscale areas with the top average grayscale values ​​are selected as target grayscale areas. For example, the top three suspected obstacle grayscale areas with the highest average grayscale values ​​are selected, and the selected suspected obstacle grayscale areas are used as target grayscale areas.

[0077] Step 140: If the suspected obstacle information is light interference obstacle information, depth information of the light interference obstacle information is obtained based on the first depth image, and the depth information is set to a specified threshold in the first depth image to generate a second depth image after removing the depth information of the light interference obstacle information.

[0078] In this step, after determining that a piece of suspected obstacle information is light interference obstacle information, the depth information corresponding to the light interference obstacle information is determined in the first depth image based on the light interference obstacle information. The depth information corresponding to the light interference obstacle information in the first depth image can be set to a specified threshold, for example, the depth information can be set to 0. This ensures that when the first depth image is converted into point cloud data, point cloud data for the light interference obstacle area is not generated, thereby eliminating light interference. The depth image obtained by setting the depth information corresponding to the light interference obstacle information in the first depth image to the specified threshold is the second depth image.

[0079] In one embodiment, the step 140 of acquiring depth information of light interference obstacle information based on the first depth image includes the following steps:

[0080] Step 140 - 1 : unify the first depth image and the RGB image into the same coordinate system and perform position alignment.

[0081] In this step, since the spatial coordinate systems of the RGB image and the depth image are different, the origin of the RGB image is the RGB camera, while the origin of the depth image is the infrared camera, there will be corresponding errors between the two. By unifying the pixels of the depth image and the RGB image into the same coordinate system and aligning the pixel positions of the depth image with the pixel positions of the RGB image, a one-to-one correspondence can be achieved. That is, when the position information of a pixel in one type of image is known, the position information of the corresponding pixel in the other type of image can also be determined accordingly.

[0082] The specific process of unifying the first depth image and the RGB image into the same coordinate system can be: the two-dimensional pixel points on the first depth map are originally in the depth coordinate system, the depth points in the depth coordinate system are converted to the world coordinate system, and then the depth points in the world coordinate system are converted to the color coordinate system (that is, the coordinate system where the RGB image pixel points are located), and the depth points in the color coordinate system are mapped to the plane of Z=1, that is, the pixel points of the first depth map can be corresponded to the pixel points of the RGB image.

[0083] Step 140 - 2 : searching the first depth image for matching pixels of each pixel of the light interference obstacle information, and using the pixel values ​​of the matching pixels as the depth information of the corresponding pixel points of the light interference obstacle information.

[0084] In this step, once the coordinate system of the first depth image and the RGB image is unified, the pixel positions of each light interference obstacle area in the RGB image can be determined based on the light interference obstacle information. Then, the corresponding matching pixel is searched in the first depth image, that is, the pixel point of the depth image at the same position in the unified coordinate system. The depth information of the corresponding pixel point of the light interference obstacle information is the pixel value of the matching pixel in the first depth image, that is, the depth value.

[0085] Step 150 : generating point cloud data corresponding to each target obstacle information based on the second depth image and the RGB image, wherein the target obstacle information is obstacle information determined from the suspected obstacle information remaining after removing the light interference obstacle information.

[0086] In this step, camera calibration can also be performed during the process of generating point cloud data. Since the degree of distortion of each lens is different, camera calibration can correct this lens distortion. It can be understood as solving the camera intrinsic parameters and distortion parameters through calibration, which is equivalent to a camera calibration. The calibration and calibration process can refer to the following process: determine the center point (Cx, Cy) of the depth image, the focal length of the camera (Fx, Fy), and obtain the depth value D of a certain pixel point from the depth image. Taking the pixel point n (xn, yn) converted into a 3D point cloud as an example, the X direction: Xd = (xn-Cx) / Fx*Dn, the Y direction: Yd = (yn-Cy) / Fy*Dn, the Z direction: Zd = Dn, and obtain the 3D point (Xd, Yd, Zd).

[0087] In addition, reference Figure 4 The obstacle point cloud projection diagram is used to project the point cloud data corresponding to the target obstacle information finally generated onto the map for use in the subsequent obstacle avoidance of the automatic driving equipment. Figure 4 In the diagram, an obstacle may be composed of multiple points on the schematic diagram, and a single point among them may be a point cloud data containing multiple points of the obstacle, that is, the point cloud data projected on the map may be the point cloud data after the point cloud data of the obstacle is sparsely operated. For example, the framed area a in the figure may indicate that there is an obstacle in area a and the obstacle is represented by multiple point cloud data in the figure. For example, a table is composed of hundreds of point cloud data, and the point cloud data representing the table after the point cloud data is sparsely operated and projected onto the map may be a few points. The target obstacle information may also include semantic information. The point cloud data may be matched with the semantic information of the obstacle and projected onto the map of the automatic driving device together with the semantic information.

[0088] In one embodiment, step 150 includes the following steps:

[0089] Step 150-1, determining depth information of each pixel in the RGB image based on the second depth image and the RGB image;

[0090] Step 150-2, generating point cloud data of each pixel in the RGB image based on the coordinate information and depth information of each pixel in the RGB image;

[0091] Step 150 - 3 , matching the point cloud data corresponding to the remaining suspected obstacle information after removing the light interference obstacle information from the point cloud data;

[0092] Step 150 - 4 : Identify ground point cloud data representing the ground from the remaining point cloud data corresponding to the suspected obstacle information, delete the ground point cloud data, and obtain point cloud data corresponding to the target obstacle information.

[0093] In this process, the second depth image is matched with the RGB image to determine the depth value corresponding to each pixel in the RGB image. Since the RGB image provides the x and y coordinates of the pixel in the RGB image coordinate system, combined with the depth value of the pixel, point cloud data can be generated. Because the depth information corresponding to the light interference obstacle area in the second depth image has been set to a threshold that cannot generate point cloud data, the point cloud data generated by matching the second depth image with the RGB image is able to eliminate light interference.

[0094] According to the suspected obstacle information and the pixel position of the suspected obstacle, the point cloud data corresponding to the suspected obstacle information is matched from a large number of point cloud data.

[0095] Among them, even the point cloud data corresponding to the remaining suspected obstacle information after removing the light interference obstacle information may still be accompanied by some ground point cloud data. For example, for a trash can on the road, since the bottom of the trash can is directly connected to the ground, when determining the point cloud data of the trash can, it is very likely that some ground point cloud data will also be classified as the point cloud data that constitutes the trash can. Therefore, in order to make the point cloud data of the obstacle more accurate, it is necessary to identify and delete some of the ground point cloud data that may be contained in the point cloud data corresponding to the remaining suspected obstacle information. After removing the ground point cloud data from the point cloud data corresponding to the remaining suspected obstacle information after removing the light interference obstacle information, the obtained point cloud data can be regarded as the point cloud data corresponding to the target obstacle information.

[0096] Furthermore, operations such as statistical filtering and radius filtering can be performed on the point cloud data corresponding to the target obstacle information to remove noise points and obtain more accurate obstacle point cloud data.

[0097] In one embodiment, step 150-4 includes the following steps:

[0098] The RANSAC algorithm is used to fit the point cloud data of each pixel in the RGB image to determine all the ground point cloud data in the point cloud data of each pixel in the RGB image;

[0099] The ground point cloud data corresponding to the remaining suspected obstacle information is determined based on all the ground point cloud data.

[0100] In this step, the RANSAC (Random Sample Consensus) algorithm in the PCL library can be used to determine the ground point cloud data.

[0101] The PCL library is an open source toolkit that implements a large number of general point cloud-related algorithms and efficient data structures, involving point cloud acquisition, filtering, segmentation, registration, retrieval, feature extraction, recognition, tracking, surface reconstruction, and visualization.

[0102] The basic principle of the RANSAC algorithm is to assume that the sample contains both correct data, that is, data that can be described by the model, and abnormal data, that is, data that deviates far from the normal range and cannot be fit into the mathematical model. At the same time, RANSAC also assumes that, given a set of correct data, there is a method to calculate the model parameters that fit this data.

[0103] When fitting the point cloud data of each pixel in the RGB image using the RANSAC algorithm, when fitting the ground, the fitted ground can be regarded as a fitted plane. Small potholes on the ground, that is, small concave and convex points on the plane, can be regarded as valid data; while obstacles on the ground, deep pits on the ground, etc., that is, large concave and convex points on the plane, these with excessive offsets can be regarded as invalid data. The RANSAC fitting plane is to fit the valid data, manually set a threshold, and the points whose distance from the fitting plane exceeds the threshold are judged as invalid data. At the same time, multiple planes can be randomly fitted, and the plane with the most data points in the plane, or the plane with the least invalid data, is selected as the fitting result.

[0104] By performing a plane fit on the point cloud data for each pixel in the RGB image using the RANSAC algorithm, all ground point cloud data within the point cloud data for each pixel in the RGB image can be determined. All ground point cloud data within the point cloud data for each pixel in the RGB image can be cleared. By matching the point cloud data for each pixel in the RGB image after clearing all ground point cloud data with the point cloud data corresponding to the remaining suspected obstacle information after removing the light interference obstacle information, the ground point cloud data contained in the remaining point cloud data corresponding to the suspected obstacle information can be identified.

[0105] In addition, GPU cuda acceleration can be used when processing point cloud data to achieve real-time detection.

[0106] In one specific implementation, point cloud data corresponding to target obstacle information can also be displayed in the map displayed on the client, corresponding to the point cloud location of the point cloud data. In addition to directly providing the point cloud data corresponding to the target obstacle information to the autonomous driving device for obstacle avoidance processing, the point cloud data can also be provided to the display screen of the autonomous driving device, such as the robot itself, to display the relevant point cloud data of obstacles when displaying the map. The point cloud data can also be provided to the management client of the autonomous driving device to inform the user of the management client of the current obstacle situation.

[0107] When understanding this embodiment, you can refer to Figure 5 Schematic diagram of obstacle detection process, Figure 5 The steps, operations, processes and objects in each detection process are briefly summarized to assist in understanding this embodiment.

[0108] In this embodiment, an RGB image and a first depth image of a target detection area are acquired, and object detection is performed on the RGB image to obtain information on one or more suspected obstacles in the target detection area. It is then determined whether each piece of suspected obstacle information is light interference obstacle information. If a piece of suspected obstacle information is light interference obstacle information, depth information of the light interference obstacle information is acquired based on the first depth image. The depth information in the first depth image is set to a specified threshold to generate a second depth image after removing the depth information of the light interference obstacle information. Based on the second depth image and the RGB image, point cloud data corresponding to each piece of target obstacle information is generated. The target obstacle information is the obstacle information determined from the suspected obstacle information remaining after removing the light interference obstacle information. This embodiment classifies highlight areas to eliminate false detections caused by light interference, effectively solving the problem of high false detection rates of RGBD cameras when encountering strong light interference and improving the efficiency of obstacle detection.

[0109] Example 2

[0110] Figure 6 This is a structural block diagram of an embodiment of an obstacle detection device provided in Embodiment 2 of the present invention, wherein the device includes:

[0111] An image acquisition module 610 is configured to acquire an RGB image and a first depth image of a target detection area;

[0112] a suspected obstacle information acquisition module 620, configured to perform object detection on the RGB image to obtain information of one or more suspected obstacles in the target detection area;

[0113] The light interference obstacle information determination module 630 is configured to determine whether each piece of suspected obstacle information is light interference obstacle information;

[0114] A second depth image generating module 640 is configured to, when suspected obstacle information is light interference obstacle information, obtain depth information of the light interference obstacle information based on the first depth image, set the depth information to a specified threshold in the first depth image, and generate a second depth image from which the depth information of the light interference obstacle information is removed;

[0115] The point cloud data generation module 650 is used to generate point cloud data corresponding to each target obstacle information based on the second depth image and the RGB image, wherein the target obstacle information is obstacle information determined from the suspected obstacle information remaining after removing the light interference obstacle information.

[0116] In one embodiment, the suspected obstacle information includes location information, and the light interference obstacle information determination module 630 includes the following submodules:

[0117] A grayscale image conversion submodule, used for converting the RGB image into a grayscale image;

[0118] a grayscale region determination submodule, configured to match the position information with the grayscale image to determine the grayscale region of each suspected obstacle in the grayscale image;

[0119] A classification model input submodule, configured to select a target grayscale region from the grayscale regions and input the target grayscale region into a pre-generated classification model;

[0120] The obstacle type acquisition submodule is used to obtain the obstacle type of each target grayscale area output by the classification model;

[0121] The light interference obstacle information determination submodule is configured to determine light interference obstacle information from the suspected obstacle information according to the obstacle type.

[0122] In one embodiment, the classification model input submodule includes the following units for selecting a target grayscale region from the grayscale regions:

[0123] a grayscale statistical data obtaining unit, configured to obtain grayscale statistical data of each grayscale area;

[0124] The target grayscale region determining unit is configured to sort the grayscale regions according to the grayscale statistical data, and use the grayscale regions that are ranked first as target grayscale regions.

[0125] In one embodiment, the second depth image generation module 640 acquires the depth information of the light interference obstacle information based on the first depth image, including the following submodules:

[0126] A position alignment submodule, configured to unify the first depth image and the RGB image into the same coordinate system and perform position alignment;

[0127] The light interference obstacle depth information determination submodule is configured to search the first depth image for matching pixels of each pixel of the light interference obstacle information, and use the pixel values ​​of the matching pixels as the depth information of the corresponding pixel points of the light interference obstacle information.

[0128] In one embodiment, the point cloud data generation module 650 includes the following submodules:

[0129] a submodule for determining depth information of each pixel, configured to determine depth information of each pixel in the RGB image based on the second depth image and the RGB image;

[0130] a point cloud data generation submodule, configured to generate point cloud data of each pixel in the RGB image based on the coordinate information of each pixel in the RGB image and the depth information;

[0131] a point cloud data matching submodule, configured to match, from the point cloud data, point cloud data corresponding to the remaining suspected obstacle information after removing the light interference obstacle information;

[0132] The ground point cloud data deletion submodule is used to identify the ground point cloud data used to represent the ground from the point cloud data corresponding to the remaining suspected obstacle information, and delete the ground point cloud data to obtain the point cloud data corresponding to the target obstacle information.

[0133] In one embodiment, the ground point cloud data deletion submodule further includes the following units:

[0134] a point cloud data fitting unit, configured to fit the point cloud data of each pixel in the RGB image using a RANSAC algorithm to determine all ground point cloud data in the point cloud data of each pixel in the RGB image;

[0135] The ground point cloud data determining unit is configured to determine the ground point cloud data in the point cloud data corresponding to the remaining suspected obstacle information based on all the ground point cloud data.

[0136] It is noted that the obstacle detection device provided in the embodiment of the present application can execute the obstacle detection method provided in the first embodiment of the present application, and has the corresponding functional modules and beneficial effects of the execution method.

[0137] Example 3

[0138] Figure 7 This is a structural diagram of an electronic device provided in Example 3 of the present application, such as Figure 7 As shown, the electronic device includes a processor 710, a memory 720, an input device 730, and an output device 740; the number of processors 710 in the electronic device can be one or more. Figure 7 In the figure, a processor 710 is used as an example; the processor 710, memory 720, input device 730 and output device 740 in the electronic device can be connected via a bus or other means. Figure 7 The bus connection is taken as an example.

[0139] The memory 720, as a computer-readable storage medium, can be used to store software programs, computer-executable programs, and modules, such as the program instruction modules corresponding to the method embodiments in the embodiments of the present application. The processor 710 executes the software programs, instructions, and modules stored in the memory 720 to execute various functional applications and data processing of the electronic device, thereby implementing the above-mentioned method.

[0140] The memory 720 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system and applications required for at least one function; the data storage area may store data created based on the use of the terminal, etc. In addition, the memory 720 may include a high-speed random access memory and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some instances, the memory 720 may further include a memory remotely located relative to the processor 710, and these remote memories may be connected to the electronic device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0141] The input device 730 may be used to receive input digital or character information and generate key signal input related to user settings and function control of an electronic device. The output device 740 may include a display device such as a display screen.

[0142] Example 4

[0143] Embodiment 4 of the present application further provides a storage medium comprising computer-executable instructions, which are used to execute the method in the method embodiment when executed by a computer processor.

[0144] Through the above description of the implementation methods, those skilled in the art can clearly understand that the present application can be implemented with the help of software and necessary general-purpose hardware, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the current technology can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as a computer floppy disk, read-only memory (ROM), random access memory (RAM), flash memory (FLASH), hard disk or optical disk, etc., including a number of instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0145] It is worth noting that in the embodiment of the above-mentioned device, the various modules and modules included are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application.

[0146] Example 5

[0147] The fifth embodiment of the present application further provides a computer program product, which includes computer-executable instructions. When the computer-executable instructions are executed by a computer processor, they are used to execute the method in the above-mentioned method embodiment 1.

[0148] Of course, the computer program product provided in the embodiment of the present application, whose computer executable instructions are not limited to the method operations described above, can also execute related operations in the method provided in any embodiment of the present application.

[0149] Note that the above are only preferred embodiments of the present application and the technical principles employed. Those skilled in the art will understand that the present application is not limited to the specific embodiments described herein, and that various obvious changes, readjustments, and substitutions can be made by those skilled in the art without departing from the scope of protection of the present application. Therefore, although the present application has been described in more detail through the above embodiments, the present application is not limited to the above embodiments and may include many other equivalent embodiments without departing from the scope of the present application. The scope of the present application is determined by the scope of the appended claims.

Claims

1. An obstacle detection method, characterized in that: The method comprises: Acquire an RGB image and a first depth image of the target detection area; Performing object detection on the RGB image to obtain information of one or more suspected obstacles in the target detection area; Determining whether each piece of suspected obstacle information is light interference obstacle information; If the suspected obstacle information is light interference obstacle information, obtaining depth information of the light interference obstacle information based on the first depth image, setting the depth information to a specified threshold in the first depth image, and generating a second depth image from which the depth information of the light interference obstacle information is removed; generating point cloud data corresponding to each target obstacle information based on the second depth image and the RGB image, wherein the target obstacle information is obstacle information determined from the suspected obstacle information remaining after removing the light interference obstacle information; The suspected obstacle information includes location information, and determining whether each piece of suspected obstacle information is light interference obstacle information includes: Converting the RGB image into a grayscale image; Matching the location information with the grayscale image to determine the grayscale area of ​​each suspected obstacle in the grayscale image; Selecting a target grayscale region from the grayscale regions, and inputting the target grayscale region into a pre-generated classification model; Obtaining the obstacle type of each target grayscale area output by the classification model; determining light interference obstacle information from the suspected obstacle information according to the obstacle type; The selecting a target grayscale area from the grayscale areas includes: Obtaining grayscale statistical data of each grayscale area; sorting the grayscale regions according to the grayscale statistical data, and taking the grayscale regions ranked first as target grayscale regions; The classification model includes a binary classification network; the binary classification network outputs classification results including a light interference obstacle grayscale area and a non-light interference obstacle grayscale area.

2. The method according to claim 1, characterized in that The acquiring the depth information of the light interference obstacle information based on the first depth image includes: Unifying the first depth image and the RGB image into the same coordinate system and performing position alignment; Searching for matching pixels of each pixel of the light interference obstacle information in the first depth image, and using pixel values ​​of the matching pixels as depth information of corresponding pixel points of the light interference obstacle information.

3. The method according to claim 2, characterized in that Generating point cloud data corresponding to each target obstacle information based on the second depth image and the RGB image includes: Determining depth information of each pixel in the RGB image based on the second depth image and the RGB image; generating point cloud data of each pixel in the RGB image based on the coordinate information of each pixel in the RGB image and the depth information; Matching point cloud data corresponding to the remaining suspected obstacle information after removing the light interference obstacle information from the point cloud data; Ground point cloud data representing the ground is identified from the remaining point cloud data corresponding to the suspected obstacle information, and the ground point cloud data is deleted to obtain point cloud data corresponding to the target obstacle information.

4. The method according to claim 3, characterized in that The step of identifying ground point cloud data representing the ground from the remaining point cloud data corresponding to the suspected obstacle information includes: The RANSAC algorithm is used to fit the point cloud data of each pixel in the RGB image to determine all ground point cloud data in the point cloud data of each pixel in the RGB image; The ground point cloud data in the point cloud data corresponding to the remaining suspected obstacle information is determined according to all the ground point cloud data.

5. An obstacle detection device, characterized in that: The device comprises: An image acquisition module is used to acquire an RGB image and a first depth image of the target detection area; a suspected obstacle information acquisition module, configured to perform object detection on the RGB image to obtain information of one or more suspected obstacles in the target detection area; A light interference obstacle information judgment module is used to judge whether each piece of suspected obstacle information is light interference obstacle information; a second depth image generating module configured to, when suspected obstacle information is light interference obstacle information, obtain depth information of the light interference obstacle information based on the first depth image, set the depth information to a specified threshold in the first depth image, and generate a second depth image from which the depth information of the light interference obstacle information is removed; a point cloud data generation module, configured to generate point cloud data corresponding to each target obstacle information based on the second depth image and the RGB image, wherein the target obstacle information is obstacle information determined from the suspected obstacle information remaining after removing the light interference obstacle information; The suspected obstacle information includes location information, and the light interference obstacle information judgment module includes the following submodules: A grayscale image conversion submodule, used for converting the RGB image into a grayscale image; a grayscale region determination submodule, configured to match the position information with the grayscale image to determine the grayscale region of each suspected obstacle in the grayscale image; A classification model input submodule, configured to select a target grayscale region from the grayscale regions and input the target grayscale region into a pre-generated classification model; The obstacle type acquisition submodule is used to obtain the obstacle type of each target grayscale area output by the classification model; a light interference obstacle information determination submodule, configured to determine light interference obstacle information from the suspected obstacle information according to the obstacle type; The classification model input submodule includes the following units for selecting a target grayscale area from the grayscale areas: a grayscale statistical data obtaining unit, configured to obtain grayscale statistical data of each grayscale area; a target grayscale region determining unit, configured to sort the grayscale regions according to the grayscale statistical data, and select the grayscale regions ranked first as target grayscale regions; The classification model includes a binary classification network; the binary classification network outputs classification results including a light interference obstacle grayscale area and a non-light interference obstacle grayscale area.

6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the method according to any one of claims 1 to 4 is implemented.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 4 is implemented.

8. A computer program product, comprising computer executable instructions, wherein the computer executable instructions are used to implement the method according to any one of claims 1 to 4 when executed.

Images