Passable area detection method and device, equipment and storage medium

Abstract

The invention provides a passable area detection method and device and a storage medium, and the method comprises the steps: obtaining a binary image of obstacle and non-obstacle dichotomy based on an all-round stitching image of a vehicle; respectively calculating the gradient of each pixel in the binary image to obtain a gradient distribution map; and determining a passable area in the look-around spliced graph based on the gradient distribution graph. According to the invention, all passable areas and obstacle areas in the look-around spliced graph can be identified. According to the method, the surround-view spliced graph is comprehensively analyzed, a passable area with a larger range is output, obstacles in the passable area can be detected, all boundary areas of the obstacles and the passable area are reserved, the boundary of the outermost layer of the closed passable area can be calculated, and attributes such as the position, the category and the contour of the obstacles can be calculated; the environment perception capability of the automatic driving system can be improved, and the robustness of the automatic driving function is improved. And the calculation process is simple, the detection efficiency of the passable area is improved, and calculation resources are saved.

Description

Technical Field

[0001] The present application belongs to the field of artificial intelligence technology, and specifically relates to a traversable area detection method, device, equipment and storage medium. Background Art

[0002] In the field of autonomous driving, identifying the traversable area in the environment is a critical task. In related technologies, deep learning networks are usually used to predict the boundary of the traversable area on the image. The boundary consists of the pixels of the nearest obstacle ground point in each column on the image.

[0003] Since the boundary of the passable area in the related art is composed of the pixels of the nearest obstacle grounding point, the area blocked by the nearest obstacle cannot be identified, and there may be passable areas in the blocked area, resulting in incomplete and low accuracy detection of the passable area. In addition, the deep learning network is used, which has a large amount of calculation and occupies more computing resources. Summary of the invention

[0004] The present application proposes a traversable area detection method, device, equipment and storage medium, which can alleviate the technical problems of incomplete and low accuracy of traversable area detection in related technologies.

[0005] The first embodiment of the present application provides a traversable area detection method, including:

[0006] Based on the vehicle's surround view mosaic image, a binary image for classification of obstacles and non-obstacles is obtained;

[0007] Calculating the gradient of each pixel in the binary image respectively to obtain a gradient distribution map;

[0008] Based on the gradient distribution map, a traversable area in the surround view spliced ​​map is determined.

[0009] In some embodiments of the present application, the step of obtaining a binary image for classifying obstacles and non-obstacles based on the vehicle's surround view mosaic image includes:

[0010] Identify the obstacle category to which each pixel in the vehicle's surround view mosaic image belongs respectively, and obtain a category distribution map;

[0011] The category distribution map is converted into a binary map, and pixels in the binary map belong to obstacles or non-obstacles.

[0012] In some embodiments of the present application, respectively calculating the gradient of each pixel in the binary image includes:

[0013] Calculate the grayscale difference between a first pixel and each second pixel respectively, the first pixel is any pixel in the binary image except a boundary pixel, and the second pixel is a pixel adjacent to the first pixel;

[0014] An average value of each of the grayscale differences is calculated, and the average value is used as the gradient of the first pixel.

[0015] In some embodiments of the present application, respectively calculating the gradient of each pixel in the binary image includes:

[0016] For the boundary pixels in the binary image, based on the gray value of the boundary pixels being 0, the gradient of the boundary pixels is assigned a preset value, and the preset value is greater than 0; or

[0017] Based on the fact that the grayscale value of the boundary pixel is not 0, the gradient of the boundary pixel is assigned 0.

[0018] In some embodiments of the present application, determining the traversable area in the surround view spliced ​​map based on the gradient distribution map includes:

[0019] Searching for a connected domain with a gradient greater than 0 in the gradient distribution graph;

[0020] Count the row and column coordinates of each connected domain;

[0021] Based on the row coordinates and the column coordinates of each connected domain, a connected domain surrounding the vehicle is determined from each connected domain as the outermost boundary of the traversable area.

[0022] In some embodiments of the present application, the step of determining, based on the row coordinates and the column coordinates of each connected domain, a connected domain surrounding the vehicle as the outermost boundary of the traversable area from each connected domain includes:

[0023] If the minimum value of the row coordinates of the first connected domain is less than half of the width of the surround view mosaic graph, the maximum value of the row coordinates is greater than half of the width, the minimum value of the column coordinates of the first connected domain is less than half of the height of the surround view mosaic graph, and the maximum value of the column coordinates is greater than half of the height, then the first connected domain is determined as the outermost boundary of the traversable area surrounding the vehicle;

[0024] The first connected domain is any connected domain in each of the connected domains.

[0025] In some embodiments of the present application, the method further includes:

[0026] A connected domain that does not surround the vehicle in each connected domain is determined as an obstacle contour.

[0027] In some embodiments of the present application, the method further includes:

[0028] Counting the obstacle categories to which each pixel in the first obstacle outline belongs; the first obstacle outline is any obstacle outline;

[0029] The obstacle category that appears most frequently is determined as the obstacle category corresponding to the first obstacle contour.

[0030] The embodiment of the second aspect of the present application provides a traversable area detection device, including:

[0031] An acquisition module is used to obtain a binary image for classification of obstacles and non-obstacles based on the vehicle's surround view mosaic image;

[0032] A gradient calculation module, used to calculate the gradient of each pixel in the binary image respectively to obtain a gradient distribution map;

[0033] A determination module is used to determine a traversable area in the surround view spliced ​​image based on the gradient distribution map.

[0034] An embodiment of the third aspect of the present application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method described in the first aspect above.

[0035] An embodiment of the fourth aspect of the present application provides a computer-readable storage medium on which a computer program is stored, and the program is executed by a processor to implement the method described in the first aspect above.

[0036] The technical solution provided in the embodiments of the present application has at least the following technical effects or advantages:

[0037] In the embodiment of the present application, the surround view mosaic image is processed into a binary image for classification of obstacles and non-obstacles, and a gradient distribution image is obtained based on the binary image, and then the passable area is identified based on the gradient distribution image. Among them, the gradient can characterize the degree of difference between the pixel and the surrounding pixels in the categories of obstacles and non-obstacles. Based on the gradient distribution image, the passable area is determined, and all the passable areas and all the obstacle areas in the surround view mosaic image can be identified. The obstacle category can also be determined for the obstacle, and the surround view mosaic image is analyzed and detected very comprehensively, and a larger passable area is output. At the same time, obstacles inside the passable area are detected, and all boundary areas between obstacles and passable areas are retained. Not only can the outermost boundary of the closed passable area be calculated, but also the location, category, contour and other attributes of the obstacle can be calculated, which can improve the environmental perception ability of the autonomous driving system and improve the robustness of the autonomous driving function. The passable area detection is more comprehensive and accurate. Moreover, it can be achieved only through simple image processing, without the need to use a deep learning network, which simplifies the calculation process, improves the efficiency of passable area detection, and saves computing resources.

[0038] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] By reading the detailed description of the preferred embodiment below, various other advantages and benefits will become clear to those of ordinary skill in the art. The accompanying drawings are only used for the purpose of illustrating the preferred embodiment and are not considered to be limitations of the present application. In addition, the same reference symbols are used to represent the same components throughout the accompanying drawings.

[0040] In the attached picture:

[0041] Figure 1 A schematic diagram showing areas blocked by obstacles in two traversable area detection schemes in the related art is shown;

[0042] Figure 2 A flow chart of a traversable area detection method provided by an embodiment of the present application is shown;

[0043] Figure 3 A schematic diagram showing a category distribution diagram provided by an embodiment of the present application is shown;

[0044] Figure 4 A schematic diagram of calculating the gradient of a pixel in a binary image provided by an embodiment of the present application is shown;

[0045] Figure 5 A schematic diagram of a gradient distribution diagram provided by an embodiment of the present application is shown;

[0046] Figure 6 Another flowchart of a traversable area detection provided by an embodiment of the present application is shown;

[0047] Figure 7 A schematic structural diagram of a traversable area detection device provided in one embodiment of the present application is shown;

[0048] Figure 8 A schematic diagram of the structure of an electronic device provided by an embodiment of the present application is shown;

[0049] Fig. 9 A schematic diagram of a storage medium provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0050] The exemplary embodiments of the present application will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present application are shown in the accompanying drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments described herein. On the contrary, these embodiments are provided in order to enable a more thorough understanding of the present application and to fully convey the scope of the present application to those skilled in the art.

[0051] It should be noted that, unless otherwise specified, the technical terms or scientific terms used in this application should have the common meanings understood by technicians in the field to which this application belongs.

[0052] The following describes a passable area detection method, device, equipment and storage medium proposed according to an embodiment of the present application in conjunction with the accompanying drawings.

[0053] In the field of autonomous driving, identifying obstacles and traversable areas in the environment is a critical task. At present, the common way to identify obstacles such as vehicles, pedestrians, cones, etc. is three-dimensional object detection. However, there are still many obstacles of various shapes and categories that affect driving, such as walls, pillars, etc. These obstacles are not suitable for three-dimensional object detection technology. Therefore, a solution is needed that can identify all obstacles in the environment that may affect driving. Traversable area detection is one of them. Traversable area refers to the traversable area without obstacles during the vehicle's driving process.

[0054] Two types of passable area detection are proposed in the related technology. One is to predict the ground line through a model. Specifically, a deep learning network is used to predict the outline of the passable area on a two-dimensional image. The outline is composed of the pixels of the obstacle grounding point closest to each column of vehicles on the image. The outline has no three-dimensional position information and needs to be converted to the BEV (Bird's Eye View) perspective through an inverse perspective transformation algorithm before it can be output for use by the downstream path planning control module.

[0055] Another related art of passable area detection is to predict the boundary of the passable area through a deep learning network. Specifically, the deep learning network is used to predict the polar coordinates of the contour points of the passable area on the surround view mosaic graph, and the contour points are the intersection points of the polar coordinate rays of the vehicle and the boundary of the obstacle.

[0056] The two passable area detection solutions in the above-mentioned related technologies both predict the outline of the passable area through models, and both use the obstacle closest to the vehicle as the boundary of the passable area. The passable area blocked by obstacles cannot be obtained. Figure 1 As shown, there is an obstacle between the vehicle and a certain area. Figure 1 Figure (a) is a schematic diagram of predicting the obstacle grounding point as the boundary of the passable area in the related art. Figure 1 The middle (b) figure is a schematic diagram of using the intersection of the polar coordinate ray and the obstacle boundary as the boundary of the passable area. Figure 1 It can be seen that the area blocked by obstacles in the related technology (i.e. the black area in the figure) cannot be detected, resulting in incomplete detection of the passable area in the related technology and missing information about the blocked area. In addition, the related technology must be calculated through a deep learning network, which has a large amount of calculation and the calculation accuracy is limited by the accuracy of the deep learning network.

[0057] Based on this, an embodiment of the present application provides a method for detecting a passable area, which obtains a binary image for classifying obstacles and non-obstacles based on the vehicle's surround view mosaic image. The gradient of each pixel in the binary image is calculated separately to obtain a gradient distribution map. Based on the gradient distribution map, the passable area in the surround view mosaic image is determined. The method calculates the gradient of each pixel in the binary image, and the gradient can characterize the degree of difference between the pixel and the surrounding pixels in the categories of obstacles and non-obstacles. The passable area is determined based on the gradient distribution map, and all passable areas and all obstacle areas in the surround view mosaic image can be identified. The passable area detection is more comprehensive and accurate. Moreover, it can be achieved only through simple image processing, without the need to use a deep learning network, which simplifies the calculation process, improves the efficiency of passable area detection, and saves computing resources.

[0058] See also Figure 2 , the method specifically comprises the following steps:

[0059] Step 101: Based on the surround view mosaic image of the vehicle, a binary image for classifying obstacles and non-obstacles is obtained.

[0060] The execution subject of the embodiment of the present application may be a vehicle, specifically, a vehicle-mounted terminal on the vehicle. The vehicle is equipped with one or more cameras for capturing images of the surrounding environment. As an example, cameras may be installed in the front, back, left, and right directions of the vehicle, respectively, for capturing images of the environment in front of, behind, on the left, and on the right of the vehicle. The camera installed on the vehicle may be a color camera, a fisheye camera, etc.

[0061] After the vehicle obtains the images taken by the cameras installed on itself at the same time, the images taken can be stitched together according to the installation orientation of the cameras to obtain a surround view stitching map of the vehicle. For example, the images taken by the cameras at the front, back, left, and right of the vehicle at the same time can be stitched together to obtain a surround view stitching map.

[0062] After the vehicle obtains the current surround view mosaic image, it generates a binary image corresponding to the surround view mosaic image. The pixels in the binary image are divided into two categories, one belonging to obstacles and the other belonging to non-obstacles. In the binary image, the grayscale values ​​of the pixels belonging to obstacles are assigned values ​​such as 1 or 255, and the grayscale values ​​of the pixels belonging to non-obstacles are assigned values ​​such as 0.

[0063] The process of generating the above binary image includes respectively identifying the obstacle category to which each pixel in the surround view mosaic image of the vehicle belongs to, and obtaining a category distribution map; converting the category distribution map into a binary image, and the pixels in the binary image belong to obstacles or non-obstacles.

[0064] In some embodiments of the present application, the obstacle category to which each pixel in the surround stitching image belongs is identified by a pre-trained semantic segmentation network, and the semantic segmentation network may include deeplabv3plus, FCN (Fully Convolutional Networks), UNet, etc. Obstacle categories may include vehicles, pedestrians, cones, walls, pillars, etc. Different obstacle categories are represented by different numerical numbers, such as 1 for vehicles, 2 for pedestrians, 3 for cones, 4 for walls, 5 for pillars, 0 for non-obstacles, and so on.

[0065] The size of the category distribution map obtained by the semantic segmentation network is the same as the size of the surround view mosaic map. Each pixel in the category distribution map corresponds to the pixel at the same coordinate in the surround view mosaic map. The pixel value of the pixel in the category distribution map is a numerical serial number used to indicate the category to which the pixel at the same coordinate in the surround view mosaic map belongs. For example, assuming that the pixel value of the pixel at the coordinate (x, y) in the category distribution map is 2, it means that the pixel at the coordinate (x, y) in the surround view mosaic map belongs to a pedestrian.

[0066] like Figure 3A schematic diagram of a category distribution diagram is shown, in which the number 0 indicates that the pixel here is a non-obstacle, a number not equal to 0 indicates that the pixel is an obstacle, and different numbers among the non-zero numbers indicate different obstacles.

[0067] The training process of the above-mentioned semantic segmentation network may include obtaining a large number of images of the vehicle driving environment, marking the digital numbers of the pixels of various obstacles and the digital numbers of the pixels of non-obstacles in these images, inputting the marked images into the semantic segmentation network for training and learning, and finally obtaining a trained semantic segmentation network.

[0068] The semantic segmentation network is used to identify the category to which each pixel in the surround mosaic belongs, forming a category distribution map. The identified categories include non-obstacle categories and a variety of specific obstacle categories, which can identify obstacles of various forms and categories, with higher and more comprehensive obstacle identification accuracy.

[0069] After obtaining the category distribution map corresponding to the surround mosaic map in the above manner, the category distribution map is converted into a binary map. Specifically, the grayscale value of the pixel with a digital sequence number of 0 in the category distribution map is assigned to 0, and the grayscale value of the pixel with a digital sequence number other than 0 is assigned to a non-zero value such as 1 or 255. After the assignment operation, a binary map of obstacle and non-obstacle classification is obtained.

[0070] In the above-mentioned embodiment, the semantic segmentation network classifies each pixel in the surround view mosaic image into two or more categories. In other embodiments, the semantic segmentation network may also only classify each pixel in the surround view mosaic image into obstacles and non-obstacles, such as using 1 to represent obstacles and 0 to represent non-obstacles. In this way, the category distribution map output by the semantic segmentation network is a binary map of the obstacle and non-obstacle classification.

[0071] The binary image corresponding to the surround stitching image is obtained in the above manner, and the pixels in the image are simplified into two categories. The binary image can intuitively distinguish the distribution of obstacles and non-obstacles in the image. The binary image is used for subsequent detection of passable areas, which also achieves the effect of simplifying data and helps reduce the amount of calculation in subsequent processes.

[0072] Step 102: Calculate the gradient of each pixel in the binary image respectively to obtain a gradient distribution map.

[0073] The process of calculating the gradient for each pixel in the binary image except the boundary pixels is the same. The embodiment of the present application takes any pixel in the binary image except the boundary pixels as the first pixel as an example to explain the gradient calculation process of non-boundary pixels in detail. Figure 4 Pixels on the edge of the strip.

[0074] Specifically, the grayscale differences between the first pixel and each second pixel are calculated respectively, where the second pixel is a pixel adjacent to the first pixel, and the average value of each grayscale difference is calculated, and the average value is used as the gradient of the first pixel.

[0075] The gradient calculation formula is, Where g represents the gradient of the first pixel, v is the grayscale value of the first pixel, and v i Represents the grayscale of the adjacent i-th second pixel.

[0076] like Figure 4 As shown, the gray-filled pixel is the first pixel, and the eight adjacent pixels around the first pixel are the second pixels. v represents the gray value of the pixel. The difference between the gray value v of the first pixel and the gray values ​​v1, v2, ..., v8 of each second pixel is calculated to obtain eight gray difference values. The average value of these eight gray difference values ​​is calculated, and the obtained average value is used as the gradient of the first pixel.

[0077] The gradient of the pixel calculated in the above manner can reflect the grayscale change rate between the pixel and the surrounding adjacent pixels, and can also characterize the degree of category difference between the pixel and the surrounding adjacent pixels in the classification of obstacles and non-obstacles.

[0078] In the gradient calculated in the above manner, if the gradient of a pixel is greater than 0, it means that the category of the pixel is not exactly the same as that of its surrounding pixels, so it indicates that the pixel is located at the boundary between the obstacle and the passable area.

[0079] Since the number of adjacent pixels of the pixels on the boundary of the binary image is small, if the gradient of the boundary pixels is calculated using the above method, the gradient of the boundary pixels will be inaccurate, thereby affecting the accuracy of subsequent identification of passable areas. Based on this, some embodiments of the present application use an assignment method to determine the gradient of the boundary pixels of the binary image.

[0080] Specifically, for boundary pixels in a binary image, based on the grayscale value of the boundary pixels being 0, the gradient of the boundary pixels is assigned a preset value, which is greater than 0; or, based on the grayscale value of the boundary pixels being not 0, the gradient of the boundary pixels is assigned 0.

[0081] The above preset value may be 1 or 2. The gray value of the boundary pixel is 0, indicating that the boundary pixel belongs to the non-obstacle category, that is, the boundary pixel belongs to the pixel of the passable area. The gradient of the boundary pixel is assigned a preset value greater than 0, which helps to identify the boundary pixel as the boundary of the passable area in the subsequent process of identifying the passable area.

[0082] If the grayscale value of the boundary pixel is not 0, it means that the boundary pixel belongs to the obstacle category, that is, the boundary pixel is a pixel in a certain obstacle. Therefore, the gradient of the boundary pixel is assigned to 0. In this way, in the subsequent recognition process, the boundary pixel will be identified as a pixel inside a certain obstacle or a pixel inside a certain passable area. In short, the boundary pixel will not be identified as a pixel on the critical point between the obstacle and the passable area, thereby avoiding the generation of a path passing through the boundary pixel during subsequent path planning, thereby improving the accuracy of passable area recognition.

[0083] Since the gradient of boundary pixels and non-boundary pixels is determined differently, before obtaining the gradient of a pixel in the above manner, it is necessary to first identify whether the pixel is a boundary pixel or a non-boundary pixel. The specific identification method is to obtain the coordinates of the pixel in the binary image. If the row coordinate or column coordinate of the pixel is 0, or the row coordinate is the width of the binary image or the column coordinate is the height of the binary image, then the pixel is determined to be a boundary pixel of the binary image. Otherwise, the pixel is determined to be a non-boundary pixel.

[0084] Through the operation of this step, the gradient of each pixel in the binary image can be quickly and accurately obtained to obtain a gradient distribution map, and the degree of category difference between pixels in obstacle and non-obstacle classification can be quantified through the gradient.

[0085] like Figure 5 A schematic diagram showing a gradient distribution map is shown. Figure 5 The medium gray pixels with a gradient of 1 are all pixels with a gradient greater than 0, that is, pixels at the boundary between obstacles and passable areas. The white pixels with a gradient of 0 are pixels inside obstacles or passable areas.

[0086] Step 103: Determine a traversable area in the surround stitching map based on the gradient distribution map.

[0087] Specifically, a connected domain with a gradient greater than 0 is searched in the gradient distribution map, the row coordinates and column coordinates of each connected domain are counted, and based on the row coordinates and column coordinates of each connected domain, a connected domain surrounding the vehicle is determined from each connected domain as the outermost boundary of the passable area.

[0088] In the embodiment of the present application, a depth-first algorithm may be used to search the gradient distribution map, and the search may be started from a corner (such as the upper left corner) of the gradient distribution map, in a certain search order (such as from left to right and from top to bottom). As an example, Figure 5For the gradient distribution graph shown, the search process can be as follows: start the search from the upper left corner of the gradient distribution graph, and search to the current pixel whose gradient is 0, then continue to search to the right for the second pixel whose gradient is also 0, and continue to search to the right for the third pixel. If the gradient of the third pixel is 1, then record the third pixel, and then search all the adjacent pixels of the third pixel. The adjacent pixels can be searched in a clockwise order. For example, first search for the pixel adjacent to the right of the third pixel whose gradient is 1, and record the pixel adjacent to the right of the third pixel. Then search for the pixel adjacent to the lower right of the third pixel whose gradient is 0, continue to search for the pixel adjacent to the lower part of the third pixel whose gradient is 1, and record the pixel adjacent to the lower part. In this way, search the pixels adjacent to the third pixel in order, and record the pixels with a gradient of 1. After searching the pixels adjacent to the third pixel, continue to search from left to right and from top to bottom until the entire gradient distribution graph is searched and all pixels with a gradient greater than 0 are obtained.

[0089] If the gradients of two pixels are both greater than 0 and the two pixels are adjacent pixels, then the two pixels are connected. In the above manner, multiple connected domains can be determined, and the gradients of the pixels in each connected domain are all greater than 0, and the pixels in the same connected domain are continuous regions that are adjacent in position.

[0090] After identifying each connected domain in the gradient distribution map in the above manner, the row coordinates and column coordinates of each connected domain are counted. Specifically, the maximum and minimum values ​​of the row coordinates and the maximum and minimum values ​​of the column coordinates of each connected domain are counted.

[0091] Based on the maximum and minimum values ​​of the row coordinates and the maximum and minimum values ​​of the column coordinates of each connected domain, the connected domain surrounding the vehicle is screened out from all the identified connected domains. If the image coordinate system in the gradient distribution map is based on the upper left vertex as the coordinate origin, then any connected domain among all the connected domains is taken as the first connected domain as an example to describe in detail the process of identifying the outermost boundary of the passable area.

[0092] Specifically, if the minimum value of the row coordinates of the first connected domain is less than half the width of the surround view mosaic map, the maximum value of the row coordinates is greater than half the width, the minimum value of the column coordinates of the first connected domain is less than half the height of the surround view mosaic map, and the maximum value of the column coordinates is greater than half the height, then the first connected domain is determined as the outermost boundary of the passable area surrounding the vehicle.

[0093] If the minimum and maximum values ​​of the row and column coordinates of the first connected domain are represented as top, left, bottom, and right, respectively. <height / 2;left<width / 2;bottom> height / 2; right>width / 2. Wherein, height and width represent the pixel height and pixel width of the surround view mosaic map respectively. If the coordinates of the first connected domain meet the above conditions, it means that the center of the surround view mosaic map is located inside the circumscribed rectangle of the first connected domain outline, that is, the outline of the first connected domain surrounds the center of the vehicle, and the first connected domain is the outermost boundary of the passable area.

[0094] For each identified connected domain, the above method is used to determine whether each connected domain is the outermost boundary of the passable area. The above method can identify the outermost boundaries of all possible passable areas in the surround mosaic map, and the passable area identification is more comprehensive. The passable area in the area blocked by the obstacle closest to the vehicle can also be accurately identified, thereby improving the accuracy of passable area identification.

[0095] In some embodiments of the present application, for the passable area that is identified as not being the outermost boundary of the passable area through the above-mentioned identification process, that is, the connected domain that does not surround the vehicle, the connected domain that does not surround the vehicle in each connected domain is determined as an obstacle contour.

[0096] In some embodiments of the present application, for each determined obstacle profile, the obstacle category to which the obstacle profile belongs may also be determined. Since the process of determining the category to which each obstacle profile belongs is the same, any obstacle profile in each obstacle profile is used as the first obstacle profile to describe the process of determining the obstacle category in detail.

[0097] Specifically, if the specific and detailed obstacle category to which each pixel belongs is identified in the process of generating the category distribution map through the semantic segmentation network in step 101, the obstacle category to which each pixel in the first obstacle contour belongs can be counted, and the obstacle category that appears most times in the first obstacle contour is determined as the category of the obstacle corresponding to the first obstacle contour.

[0098] If in step 101, only whether a pixel belongs to an obstacle or a non-obstacle is identified in the category distribution map generated by the semantic segmentation network, an obstacle image corresponding to the first obstacle contour can be cut out from the surround stitching map based on the position information of the minimum circumscribed rectangle of the first obstacle contour, and the obstacle image is input into a semantic segmentation network capable of identifying multiple refined obstacle categories, and the category to which each pixel in the obstacle image belongs is output, and the obstacle category that appears most times in the obstacle image is used as the category of the obstacle corresponding to the first obstacle contour.

[0099] For each other obstacle contour, the category of the obstacle corresponding to each other obstacle contour may be determined respectively in the above manner.

[0100] Through the above method, based on the gradient distribution map of the surround view mosaic map, the outermost boundary of the passable area, as well as the outlines of each obstacle in the surround view mosaic map and the position and obstacle category of each obstacle outline can be identified. It is possible to determine whether each pixel in the surround view mosaic map belongs to the passable area or an obstacle, and the obstacle category can be determined for the obstacle. The surround view mosaic map is analyzed and detected very comprehensively, and a larger passable area is output. At the same time, obstacles inside the passable area are detected, and all boundary areas between obstacles and the passable area are retained. Not only can the outermost boundary of the closed passable area be calculated, but also the location, category, outline and other attributes of the obstacle can be calculated, which can improve the environmental perception ability of the autonomous driving system and improve the robustness of the autonomous driving function.

[0101] The embodiments of the present application can be applied to automatic parking scenarios or automatic driving scenarios. In automatic parking scenarios, the embodiments of the present application can detect parking spaces that are blocked by obstacles, thereby improving the parking success rate.

[0102] In order to facilitate understanding of the specific process of the embodiment of the present application, Figure 6 For explanation. Figure 6 As shown in the figure, for a surround view mosaic of the vehicle, it is first processed by the semantic segmentation model to finally obtain a binary image for obstacle and non-obstacle classification. Then the contour of the binary image is extracted, and the pixels with a gradient greater than 0 in the binary image are the pixels on the contour boundary between obstacles and non-obstacles. Then, through connected domain calculation and obstacle classification, the traversable area, obstacle area and obstacle category are marked in the final feature map.

[0103] In the embodiment of the present application, the surround view mosaic image is processed into a binary image for classification of obstacles and non-obstacles, and a gradient distribution image is obtained based on the binary image, and then the passable area is identified based on the gradient distribution image. Among them, the gradient can characterize the degree of difference between the pixel and the surrounding pixels in the categories of obstacles and non-obstacles. Based on the gradient distribution image, the passable area is determined, and all the passable areas and all the obstacle areas in the surround view mosaic image can be identified. The obstacle category can also be determined for the obstacle, and the surround view mosaic image is analyzed and detected very comprehensively, and a larger passable area is output. At the same time, obstacles inside the passable area are detected, and all boundary areas between obstacles and passable areas are retained. Not only can the outermost boundary of the closed passable area be calculated, but also the location, category, contour and other attributes of the obstacle can be calculated, which can improve the environmental perception ability of the autonomous driving system and improve the robustness of the autonomous driving function. The passable area detection is more comprehensive and accurate. Moreover, it can be achieved only through simple image processing, without the need to use a deep learning network, which simplifies the calculation process, improves the efficiency of passable area detection, and saves computing resources.

[0104] The present application also provides a traversable area detection device, which is used to execute the traversable area detection method provided in any of the above embodiments. Figure 7 As shown, the device comprises:

[0105] An acquisition module 201 is used to obtain a binary image for classifying obstacles and non-obstacles based on a surround view mosaic image of the vehicle;

[0106] A gradient calculation module 202 is used to calculate the gradient of each pixel in the binary image to obtain a gradient distribution map;

[0107] The determination module 203 is used to determine the traversable area in the surround stitching map based on the gradient distribution map.

[0108] The acquisition module 201 is used to respectively identify the obstacle category to which each pixel in the surround view mosaic image of the vehicle belongs, and obtain a category distribution map; the category distribution map is converted into a binary map, and the pixels in the binary map belong to obstacles or non-obstacles.

[0109] The gradient calculation module 202 is used to respectively calculate the grayscale difference between the first pixel and each second pixel, where the first pixel is any pixel in the binary image except the boundary pixel, and the second pixel is a pixel adjacent to the first pixel; calculate the average value of each grayscale difference, and use the average value as the gradient of the first pixel.

[0110] The gradient calculation module 202 is used to assign the gradient of the boundary pixels in the binary image to a preset value based on the gray value of the boundary pixels being 0, and the preset value being greater than 0; or, based on the gray value of the boundary pixels being not 0, assigning the gradient of the boundary pixels to 0.

[0111] The determination module 203 is used to search for connected domains with a gradient greater than 0 in the gradient distribution map; count the row coordinates and column coordinates of each connected domain; and based on the row coordinates and column coordinates of each connected domain, determine the connected domain surrounding the vehicle from each connected domain as the outermost boundary of the passable area.

[0112] The determination module 203 is used to determine the first connected domain as the outermost boundary of the passable area surrounding the vehicle if the minimum value of the row coordinates of the first connected domain is less than half of the width of the surround view mosaic graph, the maximum value of the row coordinates is greater than half of the width, the minimum value of the column coordinates of the first connected domain is less than half of the height of the surround view mosaic graph, and the maximum value of the column coordinates is greater than half of the height; wherein the first connected domain is any connected domain in each connected domain.

[0113] The device also includes an obstacle recognition module, which is used to determine the connected domain that does not surround the vehicle in each connected domain as an obstacle outline.

[0114] The obstacle recognition module is further used to count the obstacle categories to which each pixel in the first obstacle outline belongs; the first obstacle outline is any obstacle outline; and the obstacle category with the largest number of occurrences is determined as the obstacle category corresponding to the first obstacle outline.

[0115] The traversable area detection device provided in the embodiment of the present application and the traversable area detection method provided in the embodiment of the present application are based on the same inventive concept and have the same beneficial effects as the methods adopted, operated or implemented therein.

[0116] The present application also provides an electronic device to perform the above-mentioned passable area detection method. Figure 8 It shows a schematic diagram of an electronic device provided by some embodiments of the present application. Figure 8 As shown, the electronic device 8 includes: a processor 800, a memory 801, a bus 802 and a communication interface 803, wherein the processor 800, the communication interface 803 and the memory 801 are connected via the bus 802; the memory 801 stores a computer program that can be run on the processor 800, and when the processor 800 runs the computer program, the passable area detection method provided in any of the aforementioned embodiments of the present application is executed.

[0117] The memory 801 may include a high-speed random access memory (RAM), and may also include a non-volatile memory, such as at least one disk storage. The communication connection between the device network element and at least one other network element is realized through at least one communication interface 803 (which may be wired or wireless), and the Internet, wide area network, local area network, metropolitan area network, etc. may be used.

[0118] The bus 802 may be an ISA bus, a PCI bus, or an EISA bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. The memory 801 is used to store programs, and the processor 800 executes the programs after receiving execution instructions. The passable area detection method disclosed in any implementation of the above-mentioned embodiment of the present application may be applied to the processor 800, or implemented by the processor 800.

[0119] The processor 800 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit or software instructions in the processor 800. The above processor 800 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a readily available programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the embodiments of the present application can be directly embodied as a hardware decoding processor to be executed, or the hardware and software modules in the decoding processor can be executed. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 801, and the processor 800 reads the information in the memory 801 and completes the steps of the above method in combination with its hardware.

[0120] The electronic device provided in the embodiment of the present application and the passable area detection method provided in the embodiment of the present application are based on the same inventive concept and have the same beneficial effects as the methods adopted, operated or implemented therein.

[0121] The present application also provides a computer-readable storage medium corresponding to the passable area detection method provided in the above embodiment. Fig. 9 The computer-readable storage medium shown is a CD 30 on which a computer program (ie, a program product) is stored. When the computer program is run by a processor, the passable area detection method provided in any of the aforementioned embodiments will be executed.

[0122] It should be noted that examples of the computer-readable storage medium may also include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other optical or magnetic storage media, which are not listed here one by one.

[0123] The computer-readable storage medium provided in the above-mentioned embodiments of the present application and the passable area detection method provided in the embodiments of the present application are based on the same inventive concept and have the same beneficial effects as the methods adopted, run or implemented by the application programs stored therein.

[0124] It should be noted that:

[0125] In the description provided herein, a large number of specific details are described. However, it is understood that the embodiments of the present application can be practiced without these specific details. In some instances, well-known structures and technologies are not shown in detail so as not to obscure the understanding of this description.

[0126] Similarly, it should be understood that in order to streamline the present application and help understand one or more of the various inventive aspects, in the above description of the exemplary embodiments of the present application, the various features of the present application are sometimes grouped together into a single embodiment, figure, or description thereof. However, the disclosed method should not be interpreted as reflecting the following schematic diagram: the claimed application requires more features than the features clearly stated in each claim. More specifically, as reflected in the claims below, the inventive aspects are less than all the features of the single embodiment disclosed above. Therefore, the claims following the specific embodiment are hereby expressly incorporated into the specific embodiment, wherein each claim itself serves as a separate embodiment of the present application.

[0127] In addition, those skilled in the art will appreciate that, although some embodiments described herein include certain features included in other embodiments but not other features, the combination of features of different embodiments is meant to be within the scope of the present application and form different embodiments. For example, in the claims below, any one of the claimed embodiments may be used in any combination.

[0128] The above is only a preferred specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A method for detecting a traversable area, characterized in that: include: Based on the vehicle's surround view mosaic image, a binary image for classification of obstacles and non-obstacles is obtained; Calculating the gradient of each pixel in the binary image respectively to obtain a gradient distribution map; Based on the gradient distribution map, a traversable area in the surround view spliced ​​map is determined.

2. The method according to claim 1, characterized in that The vehicle-based surround view mosaic image is used to obtain a binary image of obstacles and non-obstacles, including: Identify the obstacle category to which each pixel in the vehicle's surround view mosaic image belongs respectively, and obtain a category distribution map; The category distribution map is converted into a binary map, and pixels in the binary map belong to obstacles or non-obstacles.

3. The method according to claim 1, characterized in that The respectively calculating the gradient of each pixel in the binary image comprises: Calculate the grayscale difference between a first pixel and each second pixel respectively, the first pixel is any pixel in the binary image except a boundary pixel, and the second pixel is a pixel adjacent to the first pixel; An average value of each of the grayscale differences is calculated, and the average value is used as the gradient of the first pixel.

4. The method according to claim 1 or 3, characterized in that: The respectively calculating the gradient of each pixel in the binary image comprises: For the boundary pixels in the binary image, based on the gray value of the boundary pixels being 0, the gradient of the boundary pixels is assigned a preset value, and the preset value is greater than 0; or Based on the fact that the grayscale value of the boundary pixel is not 0, the gradient of the boundary pixel is assigned 0.

5. The method according to claim 1, characterized in that The determining, based on the gradient distribution map, a traversable area in the surround view spliced ​​map comprises: Searching for a connected domain with a gradient greater than 0 in the gradient distribution graph; Count the row and column coordinates of each connected domain; Based on the row coordinates and the column coordinates of each connected domain, a connected domain surrounding the vehicle is determined from each connected domain as the outermost boundary of the traversable area.

6. The method according to claim 5, characterized in that The step of determining, based on the row coordinates and the column coordinates of each connected domain, a connected domain surrounding the vehicle as the outermost boundary of a traversable area from each connected domain comprises: If the minimum value of the row coordinates of the first connected domain is less than half of the width of the surround view mosaic graph, the maximum value of the row coordinates is greater than half of the width, the minimum value of the column coordinates of the first connected domain is less than half of the height of the surround view mosaic graph, and the maximum value of the column coordinates is greater than half of the height, then the first connected domain is determined as the outermost boundary of the traversable area surrounding the vehicle; The first connected domain is any connected domain in each of the connected domains.

7. The method according to claim 5 or 6, characterized in that: The method further comprises: A connected domain that does not surround the vehicle in each connected domain is determined as an obstacle contour.

8. The method according to claim 7, characterized in that The method further comprises: Counting the obstacle categories to which each pixel in the first obstacle outline belongs; the first obstacle outline is any obstacle outline; The obstacle category that appears most frequently is determined as the obstacle category corresponding to the first obstacle contour.

9. A traversable area detection device, characterized in that: include: An acquisition module is used to obtain a binary image for classification of obstacles and non-obstacles based on the vehicle's surround view mosaic image; A gradient calculation module, used to calculate the gradient of each pixel in the binary image respectively to obtain a gradient distribution map; A determination module is used to determine a traversable area in the surround view spliced ​​image based on the gradient distribution map.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: The processor runs the computer program to implement the method according to any one of claims 1 to 8.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that: The program is executed by a processor to implement the method according to any one of claims 1 to 8.

Images