Road obstacle detection method, device, equipment and storage medium

Abstract

Embodiments of the present application provide a road obstacle detection method, apparatus, device, and storage medium, wherein the method includes: acquiring a road image; determining a drivable area and an ideal drivable area in the road image based on the road image; and determining an obstacle area on the road based on the drivable area and the ideal drivable area.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of intelligent driving technology, and are related to, but not limited to, a road obstacle detection method, device, equipment, and storage medium. Background Art

[0002] In autonomous driving systems, target detection is usually used to identify obstacles. Since the shape and location of obstacles are unknown, a large amount of sample data is required, making obstacle detection difficult and inaccurate. Summary of the Invention

[0003] The embodiment of the present application provides a technical solution for detecting road obstacles.

[0004] The technical solution of the embodiment of the present application is implemented as follows:

[0005] An embodiment of the present application provides a road obstacle detection method, the method comprising: acquiring a road image; determining a drivable area and an ideal drivable area in the road image based on the road image; and determining an obstacle area on the road based on the drivable area and the ideal drivable area.

[0006] In some embodiments, determining a drivable area in the road image based on the road image includes: identifying an image region in the road image where the similarity between the texture information and the information representing the road texture exceeds a preset similarity threshold; and determining the image region as the drivable area. Thus, by identifying an image region in the road image where the similarity between the image information and the information representing the road texture is high as the drivable area, the complexity of identifying the drivable area is reduced and easy to implement.

[0007] In some embodiments, determining the ideal driving area from the road image includes: detecting the road image to determine a road edge in the road image; and determining an area enclosed by the road edge as the ideal driving area. Using the road area enclosed by the road edge as the ideal driving area can make the coverage of the determined ideal driving area more comprehensive.

[0008] In some embodiments, determining the ideal driving area based on the road image includes determining the ideal driving area based on the drivable area. Thus, by determining the ideal driving area based on the drivable area, the obstacle area determined by comparing the ideal driving area with the drivable area is more accurately determined.

[0009] In some embodiments, determining the ideal driving area based on the drivable area includes: determining at least one outline of the drivable area; determining, within the at least one outline, a target outline that encloses the largest area; and determining the area enclosed by the target outline as the ideal driving area. Thus, by identifying multiple outlines of the drivable area and selecting the area enclosed by the outermost target outline as the ideal driving area, the process is simplified and the accuracy of determining the ideal driving area is high.

[0010] In some embodiments, before determining the area enclosed by the target outline as the ideal driving area, the method further includes: determining that the area enclosed by the target outline includes two road edges of the road. Then, the ideal driving area is determined based on the target outline and the two road edges, so that the ideal driving area is more reasonable.

[0011] In some embodiments, determining the ideal driving area based on the road image includes: detecting the road image to obtain multiple lanes in the road image; and determining the ideal driving area based on the multiple lanes and the traffic regulations corresponding to the multiple lanes. This ensures that the ideal driving area fully encompasses the coverage areas of each lane, thereby improving the accuracy of determining the ideal driving area.

[0012] In some embodiments, determining the drivable area in the road image based on the road image includes: determining an image region where the similarity between the texture in the road image and the information representing the road texture exceeds a preset similarity threshold; and determining the image region as the drivable area. In this manner, by segmenting the road image into the drivable area and the ideal road area, region segmentation can be achieved quickly and accurately.

[0013] In some embodiments, determining an obstacle area on the road based on the drivable area and the ideal drivable area includes: determining an area within the ideal drivable area that does not overlap with the drivable area; and determining the non-overlapping area as the obstacle area. Using the non-overlapping area as the obstacle area allows for accurate detection of obstacles that may affect vehicle travel and improves the stability of obstacle recognition.

[0014] In some embodiments, after determining the obstacle area on the road, the method further includes: determining a driving path for the vehicle based on the obstacle area; and controlling the vehicle based on the driving path. This makes the generated driving path more accurate, thereby enabling precise control of the vehicle based on the driving path.

[0015] An embodiment of the present application provides a road obstacle detection device, comprising: an image acquisition module for acquiring a road image; a driving area determination module for determining a drivable area and an ideal driving area in the road image based on the road image; and an obstacle area determination module for determining an obstacle area on the road based on the drivable area and the ideal driving area.

[0016] Correspondingly, an embodiment of the present application provides a computer storage medium, on which computer-executable instructions are stored. After the computer-executable instructions are executed, the method steps described above can be implemented.

[0017] An embodiment of the present application provides a computer device, which includes a memory and a processor. The memory stores computer-executable instructions, and the processor can implement the above-described method steps when executing the computer-executable instructions on the memory.

[0018] Embodiments of the present application provide a road obstacle detection method, apparatus, device, and storage medium. By detecting an acquired road image, a drivable area and an ideal drivable area in the road image are identified. By comparing the drivable area and the ideal drivable area, the obstacle area where obstacles of any shape on the road are located can be effectively determined. In this way, by comparing the drivable area and the ideal drivable area, the obstacle area on the road can be quickly identified, and the stability of identifying obstacles of different shapes can also be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 This is a schematic diagram of the implementation process of the road obstacle detection method according to an embodiment of the present application;

[0020] Figure 2 A schematic diagram of another implementation flow of the road obstacle detection method provided in an embodiment of the present application;

[0021] Figure 3 A schematic diagram of another implementation flow of the road obstacle detection method provided in an embodiment of the present application;

[0022] Figure 4 A schematic diagram of an application scenario of the road obstacle detection method provided in an embodiment of the present application;

[0023] Figure 5 A schematic diagram of another application scenario of the road obstacle detection method provided in an embodiment of the present application;

[0024] Figure 6 A schematic diagram of another application scenario of the road obstacle detection method provided in an embodiment of the present application;

[0025] Figure 7A network structure diagram of the road obstacle detection method provided in an embodiment of the present application;

[0026] Figure 8 This is a schematic diagram of the structure of a road obstacle detection device according to an embodiment of the present application;

[0027] Figure 9 This is a schematic diagram of the composition structure of the computer device according to an embodiment of the present application. DETAILED DESCRIPTION

[0028] To make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the specific technical solutions of the invention will be further described in detail below in conjunction with the drawings in the embodiments of the present application. The following embodiments are used to illustrate the present application but are not intended to limit the scope of the present application.

[0029] In the following description, reference is made to “some embodiments”, which describes a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0030] In the following description, the terms "first\second\third" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described in some embodiments can be implemented in an order other than that illustrated or described in some embodiments.

[0031] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this application pertains. The terms used herein are for the purpose of describing the embodiments of this application only and are not intended to limit this application.

[0032] Before further describing the embodiments of the present application in detail, the nouns and terms involved in the embodiments of the present application are explained. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations.

[0033] 1) Deep Neural Networks (DNNs), a deep learning framework, are neural networks with at least one hidden layer. Based on the location of different layers within a DNN, the neural network layers can be divided into three categories: input layer, hidden layer, and output layer. As shown in the figure below, the first layer is generally the input layer, the last layer is the output layer, and all layers in between are hidden layers. In the embodiments of this application, deep neural networks can be used to implement tasks such as semantic segmentation and object detection.

[0034] 2) Ego vehicle: A vehicle that includes sensors for sensing its surroundings. The vehicle coordinate system is fixed to the ego vehicle, with the x-axis pointing in the direction of travel, the y-axis pointing to the left of the vehicle's direction of travel, and the z-axis pointing upward, perpendicular to the ground. This corresponds to a right-handed coordinate system. The origin of the coordinate system is located on the ground below the midpoint of the rear axle.

[0035] The following describes exemplary applications of the road obstacle detection method provided in the embodiments of this application applied to electronic devices. The electronic devices provided in the embodiments of this application can be vehicle-mounted devices, cloud platforms, or other computer devices. For example, the vehicle-mounted devices can be thin clients, thick clients, microprocessor-based systems, small computer systems, etc. installed on the vehicle, and the cloud platforms can be distributed cloud computing technology environments including small computer systems or large computer systems, etc. The following describes exemplary applications of the electronic devices implemented as terminals or servers.

[0036] The method can be applied to a computer device, and the functions implemented by the method can be implemented by calling program codes by a processor in the computer device. Of course, the program codes can be stored in a computer storage medium. It can be seen that the computer device at least includes a processor and a storage medium.

[0037] Figure 1 This is a schematic diagram of the implementation process of the road obstacle detection method of the present application embodiment, as shown in FIG. Figure 1 As shown, combined with Figure 1 The following steps are explained:

[0038] Step S101: Acquire a road image.

[0039] In some embodiments, the road image may be an image captured from any road, and may include an image with complex content or an image with simple content, for example, a road image captured by an image capture device on a vehicle.

[0040] In some embodiments, the image acquisition device can be installed on a vehicle-mounted device or can be independent of the vehicle-mounted device. The vehicle-mounted device can be communicatively connected with the vehicle's sensors, positioning devices, etc., and the vehicle-mounted device can obtain data collected by the vehicle's sensors and geographic location information reported by the positioning devices through the communication connection. Exemplarily, the vehicle's sensor can be at least one of a millimeter-wave radar, a lidar, a camera, and the like; the positioning device can be a device for providing positioning services based on at least one of the following positioning systems: Global Positioning System (GPS), Beidou Satellite Navigation System, or Galileo Satellite Navigation System.

[0041] In some embodiments, the on-board device may be an Advanced Driving Assistance System (ADAS), which is installed on the vehicle. The ADAS may obtain the vehicle's real-time location information from the vehicle's positioning device, and / or the ADAS may obtain image data, radar data, etc. representing information about the vehicle's surrounding environment from the vehicle's sensors. Optionally, the ADAS may send vehicle driving data including the vehicle's real-time location information to a cloud platform, so that the cloud platform may receive the vehicle's real-time location information and / or image data, radar data, etc. representing information about the vehicle's surrounding environment.

[0042] The road image is obtained by an image acquisition device (i.e., a sensor, such as a camera) mounted on the vehicle. The image acquisition device captures images of the vehicle's surroundings in real time as the vehicle moves, thereby obtaining the road image. In some possible implementations, the camera mounted on the vehicle may capture images of the road and surrounding environment while the vehicle is in motion, thereby obtaining the road image.

[0043] Step S102: determining a drivable area and an ideal drivable area in the road image based on the road image.

[0044] In some embodiments, the vehicle-related road may be the road the vehicle is currently traveling on. The drivable area and the ideal drivable area may be determined independently, or the ideal drivable area may be determined based on the drivable area. Taking the independent determination of the drivable area and the ideal drivable area as an example, a road image is input into a trained neural network, and semantic segmentation is performed on the road image using different branches of the neural network to obtain the drivable area and the ideal drivable area on the road. The neural network may be a neural network for semantic segmentation. The neural network detects the drivable area in the road image; the drivable area (freespace) represents the road area where the vehicle is drivable. In addition to the current vehicle, the road image typically includes other vehicles, pedestrians, trees, road edges, etc. For example, areas containing other vehicles, pedestrians, trees, and road edges are all areas where the current vehicle is not drivable. Therefore, by performing semantic segmentation on the road image, the areas containing other vehicles, pedestrians, trees, and road edges are removed from the road image to obtain the drivable area for the vehicle. The ideal drivable area is the actual road area where the vehicle is drivable, assuming there are no vehicles, pedestrians, or objects on the road. In this way, the coverage area of ​​the ideal driving area is larger than the drivable area; or, the ideal driving area includes the drivable area on the road.

[0045] Step S103: determining an obstacle area on the road based on the drivable area and the ideal drivable area.

[0046] In some embodiments, the obstacle area of ​​the road is an area where any type of obstacle that may affect the normal driving of the vehicle is located; for example, an area where other vehicles, pedestrians, trees, etc. other than the vehicle are located on the road.

[0047] In some possible implementations, after identifying a drivable area and an ideal area of ​​the road on which the vehicle is located in a road image, the overlapping and non-overlapping portions between the two areas are compared. The overlapping area between the drivable area and the ideal drivable area, i.e., the road area that exists in both the drivable area and the ideal drivable area, indicates that the overlapping area is the actual road area of ​​the road. The overlapping area is the area on which the road vehicle can drivable, which further indicates that there are no obstacles in the overlapping area. The non-overlapping area between the drivable area and the ideal drivable area is the area that exists in the ideal drivable area but not in the drivable area, which indicates that the non-overlapping area is the area where the vehicle cannot drivable, which further indicates that the non-overlapping area is the area where obstacles exist, i.e., the obstacle area on the road.

[0048] In an embodiment of the present application, by detecting the acquired road image, the vehicle-related drivable area and the ideal driving area in the road image are identified; by comparing the drivable area and the ideal driving area, the area where obstacles of any shape on the road are located can be effectively determined; in this way, by comparing the drivable area and the ideal driving area, the obstacle area on the road can be quickly identified, and the stability of identifying obstacles of different shapes can also be improved.

[0049] In some embodiments, not only can the drivable area be obtained by inputting the road image into the first network, but the drivable area in the road image can also be identified by comparing the image information of the road image with the image information of a preset road image. That is, the determination of the drivable area in the road image based on the road image in step S101 can be achieved by the following steps S111 to S113 (not shown):

[0050] Step S111 : determining an image region where the similarity between the texture in the road image and the information representing the road texture is greater than a preset similarity threshold.

[0051] In some embodiments, the road texture information may be a road surface image obtained by capturing images of a road on which a vehicle can travel, or may be an image region of the road extracted from a road image.

[0052] In some possible implementations, texture features of a road image are extracted, texture features having a high similarity to information representing the road texture are searched among the extracted texture features, and an image region corresponding to the texture feature is determined.

[0053] Step S112: determining the image area as the drivable area.

[0054] In some embodiments, by identifying image regions in a road image that have a high degree of similarity to the image information of a preset road image, it is indicated that the image within that image region has a high degree of similarity to the image information of the preset road image, and furthermore, the image within that image region is a road surface, i.e., the image region is a drivable area for a vehicle. Thus, by identifying image regions in a road image that have a high degree of similarity to the image information of the preset road image as drivable areas, the identification of drivable areas is simplified and less complex.

[0055] In some embodiments, not only can the ideal driving area be obtained by inputting the road image into a neural network (e.g., the first network), but the ideal driving area can also be determined by identifying the road edges of the vehicle-related road and dividing the area surrounded by the road edges. That is, the ideal driving area is determined based on the road image in step S101, which can be determined by the following steps S113 and S114 (not shown):

[0056] Step S113: Detect the road image to determine the road edge in the road image.

[0057] In some embodiments, a third network is used to identify the road edges of the road and determine the road edges in the road image. The third network can be any network used for road edge recognition, such as a residual network, a super-resolution Visual Geometry Group (VGG) network, etc. The road image is input into the third network, and features are extracted from the road image. Based on the extracted image features, the road edges are identified to identify the road edges of the vehicle-related road.

[0058] Step S114: determining the area surrounded by the road edge as the ideal driving area.

[0059] In some embodiments, the identified road edge can be understood as the outermost contour of the road, so that the area surrounded by the outermost contour is the real road area without any type of vehicles, pedestrians or objects on the road; thereby, the road area surrounded by the road edge is used as the ideal driving area, which can make the coverage of the determined ideal driving area more comprehensive.

[0060] In some embodiments, the ideal driving area may be determined based on the drivable area. That is, the determination of the ideal driving area based on the drivable area in step S101 may be achieved through the following process:

[0061] Based on the drivable area, the ideal driving area is determined.

[0062] Here, after identifying the vehicle's drivable area in the road image using any of the methods described in the above embodiments, the drivable area is expanded on the road within the drivable area to determine the ideal drivable area. Thus, by determining the ideal drivable area based on the drivable area, the ideal drivable area can include the drivable area, thereby more accurately determining the obstacle area by comparing the ideal drivable area with the drivable area.

[0063] In some embodiments, determining the ideal driving area based on the drivable area can be achieved by the following steps:

[0064] In the first step, at least one outline of the drivable area is determined.

[0065] In some embodiments, since the drivable area is a road surface area obtained by removing other vehicles, objects, etc. on the entire road surface, the drivable area includes multiple holes, so the drivable area has multiple outlines.

[0066] In the second step, among the at least one contour, determine the target contour with the largest enclosed area.

[0067] In some embodiments, the area of ​​the region enclosed by each of the at least one contour is determined, and the contour with the largest area is selected as the target contour. The target contour having the largest enclosed region may also be understood as the outermost contour of the drivable region of the at least one contour.

[0068] In the third step, the area surrounded by the target outline is determined as the ideal driving area.

[0069] In some embodiments, since the target contour can be the outermost contour of the drivable area, it is reasonable to determine the area surrounded by the target contour as the ideal driving area. In this way, by identifying multiple contours of the drivable area, the area surrounded by the outermost target contour is used as the ideal driving area. The implementation process is simple and the accuracy of determining the ideal driving area is high.

[0070] In some embodiments, before determining the area enclosed by the target outline as the ideal driving area, it is also possible to first analyze whether the area enclosed by the target outline includes two road edges, that is:

[0071] Before determining the area surrounded by the target contour as the ideal driving area, the method further includes:

[0072] It is determined that the area enclosed by the target outline includes two road edges of the road.

[0073] In this way, when the enclosed area of ​​the target contour includes the two road edges of the road, the enclosed area of ​​the target contour is determined as the ideal driving area.

[0074] In some embodiments, if the enclosed area of ​​the target contour includes two road edges, it means that the enclosed area of ​​the target contour covers the entire road surface, thus making the obtained ideal driving area more reasonable.

[0075] In some embodiments, the above step S101 can also be implemented in the following two ways:

[0076] In a first approach, the ideal driving area can also be determined by detecting multiple lanes on a road and analyzing traffic rules corresponding to the lanes. That is, the determination of the ideal driving area based on the road image in step S102 can also be achieved by the following steps:

[0077] The first step is to detect the road image to obtain multiple lanes in the road image.

[0078] In some embodiments, a detector can be used to detect lane lines in a road image. Based on the detected multiple vehicle lines, the road area between two adjacent vehicle lines is treated as a single vehicle, thereby generating multiple lanes. Alternatively, the road image can be input into a trained lane detection network to directly output the multiple lanes on the road.

[0079] In a second step, the ideal driving area is determined based on the multiple lanes and traffic rules corresponding to the multiple lanes.

[0080] In some embodiments, after detecting multiple lanes on a road, road surface signals (e.g., arrows, solid lines, or dashed lines) are acquired for each lane. Traffic regulations can be used to determine the meaning of these signals. For example, a solid white line indicates a prohibited crossing of the same-direction lane divider. Thus, by determining the area covered by each lane and the traffic regulations corresponding to the road surface signals for each lane, the area covered by each lane can be determined. By combining the areas covered by multiple vehicles, the actual road area within which vehicles can travel without obstacles can be determined, i.e., the ideal driving area. Thus, by determining the area enclosed by lanes as the ideal driving area through lane detection, not only can the method for determining the ideal driving area be enriched, but also the ideal driving area can be made to fully encompass the areas covered by each lane, thereby improving the accuracy of determining the ideal driving area.

[0081] Method 2: Semantically segment the actual road area and the drivable area of ​​the road in the road image using different network branches in the same neural network to obtain the ideal drivable area and the drivable area. That is, the above step S101 can be implemented by the following process:

[0082] Semantic segmentation is performed on the road image to obtain the drivable area and the ideal drivable area in the road image.

[0083] In some embodiments, a first network is first used to semantically segment the road area in the road image where the vehicle can travel, thereby obtaining the drivable area. Then, a second network is used to semantically segment the actual road area of ​​the road associated with the vehicle, thereby obtaining the ideal drivable area. The first network can be any type of neural network that has been trained to semantically segment drivable and drivable areas on the road. The first network can be a network branch of any neural network; for example, the neural network can be a fully convolutional neural network, a mask region convolutional neural network (Mask R-CNN), a deep neural network, etc. By inputting the road image into the first network and extracting features from the road image, semantic segmentation is performed on the drivable and drivable areas of the road image based on the extracted image features. In this way, the drivable areas in the road image can be quickly and accurately identified.

[0084] The second network and the first network can be different branches of the same neural network. Alternatively, the second network can be any type of pre-trained neural network capable of semantically segmenting the road's real road and non-road areas. By inputting a road image into the second network, the second network performs semantic segmentation on the road image's real road and non-road areas, using the segmented real road area as the ideal driving area. The real road area is the road area where a vehicle can actually drive if no objects are present.

[0085] The above process can be performed simultaneously in different branches of a neural network. A road image is input into the neural network for feature extraction. The extracted features are then fed into the first and second networks, respectively. The first branch of the neural network performs semantic segmentation on the drivable road area in the road image to obtain the drivable area. Simultaneously, the second branch of the neural network performs semantic segmentation on the actual road area in the road image to obtain the ideal drivable area. In this way, segmenting the drivable area and the ideal road area in the road image using different branches of the same neural network allows for rapid and accurate area segmentation.

[0086] In the embodiment of the present application, the drivable area and the ideal driving area are determined in a variety of ways, so that the drivable area and the ideal driving area of ​​the vehicle can be identified with high accuracy under any circumstances.

[0087] In some embodiments, the obstacle area of ​​the road is determined by comparing the non-overlapping area between the drivable area and the ideal drivable area, that is, the above step S103 can be performed by Figure 2 The steps shown achieve:

[0088] Step S201: Determine an area in the ideal driving area that does not overlap with the drivable area.

[0089] In some embodiments, the ideal driving area is compared with the drivable area to determine overlapping areas between the two areas and non-overlapping areas between the two areas. The overlapping area is the actual road area and may be equal to the drivable area. The non-overlapping area between the two areas, i.e., the area that exists in the ideal driving area but not in the drivable area, indicates that the non-overlapping area is an area where the vehicle is currently not drivable.

[0090] Step S202: Determine the non-overlapping area as the obstacle area.

[0091] In some embodiments, areas where a vehicle is currently not drivable (i.e., non-overlapping areas) indicate the presence of obstacles that could affect the vehicle's normal travel. These obstacles could be other vehicles, pedestrians, trees, or other objects. By using non-overlapping areas as obstacle areas, the system can accurately detect obstacles that could affect the vehicle's travel, regardless of their location or shape, thus improving the stability of obstacle recognition.

[0092] In some embodiments, the vehicle path is re-formulated by detecting the obstacle area to more effectively control the vehicle's driving, that is, after step S103, the following steps are also included: Figure 3 Steps shown:

[0093] Step S301: Determine the driving path of the vehicle based on the obstacle area.

[0094] In some embodiments, the driving path includes the vehicle's speed and direction. By analyzing the location of obstacles on the road and the positional relationship between the obstacle and the vehicle, the vehicle's driving path on the road can be redefined. For example, if the distance between the vehicle and the obstacle is close, the vehicle can be controlled to prepare to brake or move away from the obstacle.

[0095] Step S302: Control the driving of the vehicle based on the driving path.

[0096] In some embodiments, the vehicle's autonomous driving system can determine the vehicle's driving path and then control the vehicle to drive according to the driving path.

[0097] Through the above steps S301 and S302, after the obstacle area is identified, the subsequent driving path is generated in combination with the road information; in this way, the generated driving path is more accurate, and based on this, precise control of the vehicle can be achieved by using the driving path.

[0098] Below, an exemplary application of an embodiment of the present application in an actual application scenario will be described, taking the use of a deep neural network to determine the turning direction of a vehicle at a road intersection based on road signs as an example.

[0099] The field of autonomous driving, primarily based on deep learning, has made significant progress, including in image segmentation and object detection. However, as a holistic system, the output of the perception module serves subsequent modules. For example, perception results not only indicate the presence of an object ahead but also provide relevant logical outputs for subsequent modules, providing control signals and logic signals for autonomous driving.

[0100] In an embodiment of the present application, a road obstacle detection method is provided. From the perspective of semantic segmentation, obstacle detection is performed by determining the non-overlapping areas of the ideal driving area and the drivable area. Obstacles of irregular shapes and uncertain positions can be effectively identified, and the distance to the obstacles can be effectively assessed, which can provide effective information for the subsequent decision-making planning control module of the autonomous driving system.

[0101] An embodiment of the present application provides a road obstacle detection method, which uses a deep learning method to obtain the drivable area and ideal driving area of ​​an autonomous driving vehicle, and uses the drivable area and ideal driving area to obtain general obstacle detection results on the road surface, providing an effective signal for subsequent autonomous driving path planning.

[0102] After semantic segmentation of the collected road image, the drivable area and ideal driving area in the image are determined, such as Figure 4 As shown, the road image 401 is semantically segmented to obtain the drivable area and the ideal drivable area of ​​the road in the image; Figure 5 and Figure 6 As shown, Figure 5 The area 501 in the figure represents the drivable area. Figure 6 Region 601 in the figure represents the ideal driving area. Comparing regions 501 and 601, we can see that the drivable area excludes vehicles, pedestrians, and some irregular obstacles. The ideal driving area, on the other hand, identifies the area where the vehicle can ideally drive. The main difference between the ideal driving area and the drivable area lies in the identification of obstacles.

[0103] In some embodiments, during the autonomous driving process, the vehicle performs perception output based on the information provided on the road and synthesizes the results of the model output. Figure 7 As shown, Figure 7 This is a diagram of the road perception signal output network structure provided in an embodiment of the present application, wherein:

[0104] Image input module 701, used for inputting road images;

[0105] The backbone network 702 is used to extract features from the input road image.

[0106] Here, the backbone network can be a residual network, a VGG network, a GoogleNet network, or a ResNet network, etc.

[0107] The drivable area branch network 703 is used to implement the segmentation task and perform semantic segmentation based on the extracted image features to output the drivable area.

[0108] In some possible implementations, the drivable area is segmented by inputting the image features extracted by the backbone network 702 into the drivable area branch network 703 that performs the semantic segmentation task, so as to output the drivable area.

[0109] In some possible implementations, since roads all have similar textures, the drivable area branch network 703 searches for objects similar to roads as drivable areas. Therefore, the drivable area branch network 703 can generate a drivable area that excludes obstacles. In this way, the drivable area branch network 403 can robustly detect drivable areas in road areas.

[0110] The ideal driving area branch network 704 is used to implement the segmentation task and perform semantic segmentation based on the extracted image features to output the ideal driving area.

[0111] In some possible implementations, the image features extracted by the backbone network 702 are input into the ideal driving area branch network 704 that performs semantic segmentation to segment the ideal driving area and output the ideal driving area. The ideal driving area branch network 704 identifies which areas of the road area are real road areas (even if there are obstacles on the road surface), and which areas are not real road areas.

[0112] In some possible implementations, the area within the outermost contour of the image area input to the ideal driving area branch network 704 is determined as the ideal area.

[0113] In some possible implementations, the ideal driving area may be determined based on the drivable area. For example, the outermost contour of the drivable area may be determined as the ideal driving area.

[0114] In some possible implementations, lane detection is performed on a road image to determine that the area enclosed by the lane is an ideal driving area. In this way, the ideal driving area can be determined based on the detected lane area and the lane driving rules of the road.

[0115] In some embodiments, after determining the drivable area and ideal drivable area respectively through the drivable area branch network 703 and the ideal drivable area branch network 704, the results of the two areas are compared to classify the overlapping areas as real road areas. The areas that do not overlap (in most cases, the area of ​​the ideal drivable area is larger than the area of ​​the drivable area), that is, areas that exist in the ideal driving area but not in the drivable area, are identified as obstacle areas and output. In this way, vehicles, pedestrians, non-motor vehicles, and some obstacles of irregular shapes can be effectively identified.

[0116] In an embodiment of the present application, the drivable area and the ideal driving area can be effectively identified based on a deep learning model, and the two learning tasks can be concentrated into one network, with two branches and one identical backbone network, which can not only accelerate the model processing speed, but also effectively reduce the pressure on the operating speed.

[0117] The present invention provides a road obstacle detection device. Figure 8 This is a schematic diagram of the structure of the road obstacle detection device according to the embodiment of the present application. Figure 8 As shown, the road obstacle detection device 800 includes:

[0118] An image acquisition module 801 is used to acquire a road image;

[0119] A driving area determination module 802 is configured to determine a drivable area and an ideal driving area in the road image based on the road image;

[0120] The obstacle area determination module 803 is configured to determine an obstacle area on the road based on the drivable area and the ideal drivable area.

[0121] In some embodiments, the driving area determination module 802 includes:

[0122] An image region identification submodule, configured to determine an image region in which the similarity between the texture in the road image and the information representing the road texture is greater than a preset similarity threshold;

[0123] The drivable area determination submodule is configured to determine the image area as the drivable area.

[0124] In some embodiments, the driving area determination module 802 includes:

[0125] a road edge recognition submodule, configured to detect the road image and determine the road edge in the road image;

[0126] The first ideal driving area determination submodule is configured to determine the area surrounded by the road edge as the ideal driving area.

[0127] In some embodiments, the driving area determination module 802 includes:

[0128] The second ideal driving area determination submodule is configured to determine the ideal driving area based on the drivable area.

[0129] In some embodiments, the second ideal driving area determination submodule includes:

[0130] a contour determining unit, configured to determine at least one contour of the drivable area;

[0131] a target contour determining unit, configured to determine, among the at least one contour, a target contour having a largest enclosed area;

[0132] The ideal driving area determining unit is configured to determine an area surrounded by the target outline as the ideal driving area.

[0133] In some embodiments, the second ideal driving area determination submodule is further configured to:

[0134] It is determined that the area enclosed by the target outline includes two road edges of the road.

[0135] In some embodiments, the driving area determination module 802 includes:

[0136] a lane detection submodule, configured to detect the road image and obtain a plurality of lanes in the road image;

[0137] The fourth ideal driving area determination submodule is configured to determine the ideal driving area based on the multiple lanes and traffic rules corresponding to the multiple lanes.

[0138] In some embodiments, the driving area determination module 802 is further configured to:

[0139] Semantic segmentation is performed on the road image to obtain the drivable area and the ideal drivable area in the road image.

[0140] In some embodiments, the obstacle area determination module 803 includes:

[0141] A non-overlapping area determination submodule, configured to determine an area in the ideal driving area that does not overlap with the drivable area;

[0142] The obstacle area determination submodule is configured to determine the non-overlapping area as the obstacle area.

[0143] In some embodiments, the apparatus further comprises:

[0144] A driving path determination module, configured to determine a driving path of the vehicle based on the obstacle area;

[0145] A vehicle control module is used to control the driving of the vehicle based on the driving path.

[0146] It should be noted that the description of the above device embodiment is similar to the description of the above method embodiment and has similar beneficial effects as the method embodiment. For technical details not disclosed in the device embodiment of this application, please refer to the description of the method embodiment of this application for understanding.

[0147] It should be noted that in the embodiments of the present application, if the above-mentioned road obstacle detection method is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiments of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a terminal, server, etc.) to execute all or part of the methods described in each embodiment of the present application. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a sports hard disk, a read-only memory (ROM), a magnetic disk or an optical disk. In this way, the embodiments of the present application are not limited to any specific combination of hardware and software.

[0148] Correspondingly, an embodiment of the present application further provides a computer program product, which includes computer-executable instructions. When the computer-executable instructions are executed, the steps of the road obstacle detection method provided in the embodiment of the present application can be implemented. Accordingly, an embodiment of the present application further provides a computer storage medium, which stores computer-executable instructions. When the computer-executable instructions are executed by a processor, the steps of the road obstacle detection method provided in the above embodiment can be implemented. Accordingly, an embodiment of the present application provides a computer device, Figure 9 This is a schematic diagram of the structure of the computer device according to the embodiment of the present application. Figure 9 As shown, computer device 900 includes a processor 901, at least one communication bus, a communication interface 902, at least one external communication interface, and memory 903. Communication interface 902 is configured to facilitate communication between these components. Communication interface 902 may include a display screen, and external communication interfaces may include standard wired and wireless interfaces. Processor 901 is configured to execute a road obstacle detection program in memory to implement the steps of the road obstacle detection method provided in the above embodiment.

[0149] The descriptions of the above embodiments of the road obstacle detection device, computer device, and storage medium are similar to those of the above-mentioned method embodiments, and offer similar technical descriptions and beneficial effects as the corresponding method embodiments. Due to space limitations, they can be incorporated into the description of the above-mentioned method embodiments and are therefore not further elaborated here. For technical details not disclosed in the embodiments of the road obstacle detection device, computer device, and storage medium of this application, please refer to the description of the method embodiments of this application for an understanding. It should be understood that references to "one embodiment" or "an embodiment" throughout this specification mean that specific features, structures, or characteristics associated with an embodiment are included in at least one embodiment of this application. Therefore, the appearance of "in one embodiment" or "in an embodiment" throughout this specification does not necessarily refer to the same embodiment. Furthermore, these specific features, structures, or characteristics may be combined in any suitable manner in one or more embodiments. It should be understood that in the various embodiments of this application, the order of the sequence numbers of the above-mentioned processes does not necessarily indicate a specific order of execution. The order of execution of each process is determined by its function and inherent logic and does not constitute any limitation on the implementation of the embodiments of this application. The sequence numbers of the above-mentioned embodiments of this application are for descriptive purposes only and do not represent the superiority or inferiority of each embodiment.

[0150] It should be noted that, in this document, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or apparatus comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or apparatus comprising the element.

[0151] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0152] The units described above as separate components may or may not be physically separate, and the components displayed as units may or may not be physical units; they may be located in one place or distributed across multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, the functional units in the various embodiments of the present application may all be integrated into one processing unit, or each unit may be a separate unit, or two or more units may be integrated into one unit; the above-mentioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units. Those skilled in the art will understand that all or part of the steps of the above-mentioned method embodiments may be completed by hardware related to program instructions, and the aforementioned program may be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above-mentioned method embodiments; and the aforementioned storage medium includes various media that can store program codes, such as mobile storage devices, read-only memories (ROMs), magnetic disks or optical disks.

[0153] Alternatively, if the above-mentioned integrated unit of the present application is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for making a computer device (which can be a personal computer, a server, or a network device, etc.) execute all or part of the methods described in each embodiment of the present application. The aforementioned storage medium includes various media that can store program codes, such as mobile storage devices, ROMs, magnetic disks, or optical disks. The above is only a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any person skilled in the art can easily think of changes or replacements within the technical scope disclosed in the present application, which should be included within the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

Claims

1. An obstacle detection method, characterized in that: The method comprises: Acquire road images; determining a drivable area and an ideal drivable area in the road image based on the road image; Determining an obstacle area on the road based on the drivable area and the ideal drivable area, Wherein, determining the obstacle area on the road based on the drivable area and the ideal drivable area includes: determining an area in the ideal drivable area that does not overlap with the drivable area; The non-overlapping area is determined as the obstacle area.

2. The method according to claim 1, characterized in that Determining a drivable area in the road image based on the road image includes: Determining an image region where the similarity between the texture in the road image and the information representing the road texture is greater than a preset similarity threshold; The image area is determined as the drivable area.

3. The method according to claim 1 or 2, characterized in that Determining the ideal driving area based on the road image includes: detecting the road image to determine a road edge in the road image; The area surrounded by the road edge is determined as the ideal driving area.

4. The method according to claim 1, wherein Determining the ideal driving area based on the road image includes: The ideal driving area is determined based on the drivable area.

5. The method according to claim 4, characterized in that The determining the ideal driving area based on the drivable area includes: determining at least one contour of the drivable area; Determining, among the at least one contour, a target contour having a largest enclosed area; An area surrounded by the target contour is determined as the ideal driving area.

6. The method according to claim 5, characterized in that Before determining the area surrounded by the target contour as the ideal driving area, the method further includes: It is determined that the area enclosed by the target outline includes two road edges of the road.

7. The method according to claim 1, characterized in that Determining the ideal driving area based on the road image includes: Detecting the road image to obtain a plurality of lanes in the road image; The ideal driving area is determined based on the multiple lanes and traffic regulations corresponding to the multiple lanes.

8. The method according to claim 1, characterized in that The determining, based on the road image, a drivable area and an ideal drivable area in the road image includes: Semantic segmentation is performed on the road image to obtain the drivable area and the ideal drivable area in the road image.

9. The method according to any one of claims 1 to 8, characterized in that After determining the obstacle area on the road, the method further includes: Determining a driving path of the vehicle based on the obstacle area; Based on the driving path, the driving of the vehicle is controlled.

10. A road obstacle detection device, characterized in that: The device comprises: An image acquisition module, used for acquiring road images; a driving area determination module, configured to determine a drivable area and an ideal driving area in the road image based on the road image; an obstacle area determination module, configured to determine an obstacle area on the road based on the drivable area and the ideal drivable area, Wherein, the obstacle area determination module includes: A non-overlapping area determination submodule, configured to determine an area in the ideal driving area that does not overlap with the drivable area; The obstacle area determination submodule is configured to determine the non-overlapping area as the obstacle area.

11. A computer storage medium, characterized in that The computer storage medium stores computer-executable instructions, which, after being executed, can implement the method steps described in any one of claims 1 to 9.

12. A computer device, characterized in that: The computer device includes a memory and a processor, the memory stores computer-executable instructions, and the processor can implement the method steps described in any one of claims 1 to 9 when running the computer-executable instructions on the memory.

Images