Road obstacle detection method, device, equipment, and storage medium

Abstract

A method, device, equipment, and storage medium for detecting road obstacles, the method includes a step of acquiring a road image (S101), a step of determining a drivable area and an ideal driving area in the road image based on the road image (S102), and a step of determining an obstacle area on the road based on the drivable area and the ideal driving area (S103). In this way, by comparing the drivable area with the ideal driving area, it is possible to quickly recognize an obstacle area on the road, and further to improve the stability of recognizing obstacles of different shapes.

Description

[Technical field]

[0001] (CROSS-REFERENCE TO RELATED APPLICATIONS) This application claims priority to a Chinese patent application filed with the China Patent Office on March 24, 2022, bearing application number 202210303623.0 and entitled "Method, device, equipment and storage medium for detecting road obstacles," the entire contents of which are incorporated herein by reference.

[0002] The present application relates to the technical field of intelligent driving, and in particular, but not limited to, a method, device, apparatus and storage medium for detecting road obstacles. [Background technology]

[0003] Autonomous driving systems usually adopt target detection methods to recognize obstacles, but because the shape and location of the obstacle are unknown, a large amount of sample data is required, making it difficult to detect the obstacle and the accuracy is low. Summary of the Invention

[0004] The embodiments of the present disclosure provide a technical solution for road obstacle detection.

[0005] The technical solutions of the embodiments of the present disclosure are realized as follows.

[0006] An embodiment of the present disclosure provides a method for detecting road obstacles, the method including the steps of acquiring a road image, determining a driveable area and an ideal driving area in the road image based on the road image, and determining an obstacle area on the road based on the driveable area and the ideal driving area.

[0007] In some embodiments, the step of determining a drivable area in the road image based on the road image includes the steps of determining an image area in which a similarity between a texture in the road image and information representing the road texture is greater than a predetermined similarity threshold, and determining the image area as the drivable area. In this way, by recognizing an image area in the road image in which a similarity between screen information and information representing the road texture is high as the drivable area, the recognition complexity of the drivable area is reduced and it is easy to realize.

[0008] In some embodiments, the step of determining the ideal driving area based on the road image includes the steps of detecting the road image and determining road edges in the road image, and determining an area surrounded by the road edges as the ideal driving area. In this way, by setting the area surrounded by the road edges as the ideal driving area, the coverage of the determined ideal driving area can be made more comprehensive.

[0009] In some embodiments, determining the ideal driving area based on the road image includes determining the ideal driving area based on the drivable area. By determining the ideal driving area based on the drivable area in this manner, the obstacle area determined by comparing the ideal driving area with the drivable area becomes more accurate.

[0010] In some embodiments, the step of determining the ideal driving area based on the driveable area includes the steps of determining at least one contour of the driveable area, determining a target contour having the largest area surrounded by the at least one contour, and determining the area surrounded by the target contour as the ideal driving area. In this way, by recognizing multiple contours of the driveable area and determining the area surrounded by the outermost target contour as the ideal driving area, the realization process is simplified and the accuracy of the determined ideal driving area is increased.

[0011] In some embodiments, before determining the area surrounded by the target contour as the ideal driving area, the road obstacle detection method further includes determining that the area surrounded by the target contour includes two road edges of the road. Then, the ideal driving area is jointly determined based on the target contour and the two road edges, so that the obtained ideal driving area is more reasonable.

[0012] In some embodiments, the step of determining the ideal driving area based on the road image includes the steps of: detecting the road image to obtain a plurality of roadways in the road image; and determining the ideal driving area based on the plurality of roadways and the traffic rules corresponding to the plurality of roadways. In this way, the ideal driving area can be made to completely include the coverage area of ​​each roadway, and the accuracy of determining the ideal driving area can be improved.

[0013] In some embodiments, the step of determining a drivable area in the road image based on the road image includes the steps of determining an image area in which a similarity between a texture in the road image and information representing a road texture is greater than a predetermined similarity threshold, and determining the image area as the drivable area. In this way, by dividing the road image into the drivable area and the ideal road area, it is possible to quickly and accurately realize area division.

[0014] In some embodiments, the step of determining an obstacle area on a road based on the driveable area and the ideal driveable area includes the steps of determining an area in the ideal driveable area that does not overlap with the driveable area, and determining the non-overlapping area as the obstacle area. By determining the non-overlapping area as the obstacle area in this manner, not only can the obstacle area that affects vehicle drive be accurately detected, but also the stability of obstacle recognition can be improved.

[0015] In some embodiments, after determining the obstacle area on the road, the road obstacle detection method further includes: determining a driving path of the vehicle based on the obstacle area; and controlling the driving of the vehicle based on the driving path. In this way, the generated driving path is more accurate, and based on this, accurate control of the vehicle can be realized according to the driving path.

[0016] An embodiment of the present disclosure provides a road obstacle detection device, the device comprising: an image acquisition portion configured to acquire a road image; a driving area determination portion configured to determine a driveable area and an ideal driving area in the road image based on the road image; and an obstacle area determination portion configured to determine an obstacle area on a road based on the driveable area and the ideal driving area.

[0017] Correspondingly, an embodiment of the present disclosure provides a computer storage medium having computer-executable instructions stored thereon, which, after being executed, can realize steps of the above-mentioned road obstacle detection method.

[0018] An embodiment of the present disclosure provides a computing device, the computing device comprising: a memory having computer-executable instructions stored therein; and a processor capable of implementing steps of the above-mentioned road obstacle detection method when executing the computer-executable instructions on the memory.

[0019] An embodiment of the present disclosure further provides a computer program product including a computer program or instructions which, when executed on an electronic device, causes the electronic device to perform steps in any possible embodiment of the first aspect above.

[0020] The embodiments of the present disclosure provide a method, device, equipment, and storage medium for detecting road obstacles, which detect an acquired road image, recognize a drivable area and an ideal driving area in the road image, and effectively determine an obstacle area on the road where an obstacle of any shape exists by comparing the drivable area with the ideal driving area. In this way, by comparing the drivable area with the ideal driving area, the obstacle area on the road can be quickly recognized, and the stability of the recognition of obstacles of different shapes can be improved.

[0021] In order to make the above objectives, features and advantages of the embodiments of the present disclosure more comprehensible, preferred embodiments accompanied with drawings are described in detail below. [Brief description of the drawings]

[0022] [Figure 1A] FIG. 1 is a schematic diagram of a system architecture to which the road obstacle detection method of the embodiment of the present disclosure can be applied. [Figure 1B] 1 is an implementation flowchart of a method for detecting road obstacles according to an embodiment of the present disclosure. [Diagram 2] 4 is a flowchart illustrating another implementation of a method for detecting road obstacles according to an embodiment of the present disclosure. [Diagram 3] 13 is a flowchart illustrating yet another implementation of a method for detecting road obstacles according to an embodiment of the present disclosure. [Figure 4] 1 is a schematic diagram of an application scene of a road obstacle detection method according to an embodiment of the present disclosure; [Diagram 5] FIG. 2 is a schematic diagram of another application scene of the road obstacle detection method according to an embodiment of the present disclosure; [Figure 6] FIG. 2 is a schematic diagram of yet another application scene of the road obstacle detection method according to an embodiment of the present disclosure; [Figure 7] FIG. 2 is a network structure diagram of a method for detecting road obstacles according to an embodiment of the present disclosure. [Figure 8] 1 is a schematic diagram showing a configuration of a road obstacle detection device according to an embodiment of the present disclosure. [Figure 9]FIG. 1 is a schematic diagram illustrating a configuration of a computing device according to an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0023] In order to more clearly explain the technical solutions of the embodiments of the present disclosure, the drawings necessary for the description of the embodiments are briefly described above. The drawings herein are incorporated in and constitute a part of this specification, and these drawings show embodiments consistent with the present disclosure, and are used together with the specification to explain the technical solutions of the present disclosure. It should be understood that the above drawings only show some of the embodiments of the present disclosure, and should not be considered as limiting the scope of protection, and those skilled in the art can also obtain other related drawings based on these drawings without creative efforts.

[0024] In order to make the objectives, technical solutions and advantages of the embodiments of the present disclosure clearer, the specific technical solutions of the present invention will be described in more detail below with reference to the drawings of the embodiments of the present disclosure. The following embodiments are for illustrating the present disclosure, and are not intended to limit the scope of the present disclosure.

[0025] In the following, references to "some embodiments" describe a subset of all possible embodiments, but it is understood that "some embodiments" may be the same or different subsets of all possible embodiments, and may be combined with each other without inconsistency.

[0026] In the following, terms such as "first / second / third" are not intended to limit a particular order but to distinguish between similar objects. It is understood that "first / second / third" can be used to convert a particular order or order when appropriate, so that the embodiments of the present disclosure described in some embodiments can be implemented in an order other than that shown or described in some embodiments.

[0027] Unless otherwise specified, all technical and scientific terms used in this disclosure have the same meaning as commonly understood by those skilled in the art of this disclosure. The terms used in this disclosure are only used to describe the embodiments of this disclosure and are not intended to limit the disclosure.

[0028] Before describing the embodiments of the present disclosure in detail, nouns and terms related to the embodiments of the present disclosure will be explained. The nouns and terms related to the embodiments of the present disclosure will be explained as follows.

[0029] 1) Deep Neural Networks (DNN): A type of deep learning framework, which is a neural network with at least one hidden layer. According to the positions of different layers of the DNN, the neural network layers in the DNN can be divided into three: input layer, hidden layer, and output layer. As shown in the figure below, the first layer is generally the input layer, the last layer is the output layer, and the middle layer is the hidden layer. In the embodiment of the present disclosure, the deep neural network can be used to realize semantic segmentation tasks, target detection tasks, etc.

[0030] 2) Ego vehicle: A vehicle equipped with sensors to sense the surrounding environment. The vehicle coordinate system is fixedly connected to the ego vehicle, where the x-axis is the direction the vehicle is moving, the y-axis points to the left of the direction the vehicle is moving, and the z-axis is perpendicular to the ground and points upwards, conforming to a right-handed coordinate system. The origin of the coordinate system is located on the ground below the rear axle midpoint.

[0031] In the following, an application example of the road obstacle detection method according to the embodiment of the present disclosure to an electronic device will be described. The electronic device according to the embodiment of the present disclosure may be an in-vehicle device, a cloud platform, or other computer device. Exemplarily, the in-vehicle device may be a thin client, a fat client, a microprocessor-based system, a small computer system, etc. mounted on a vehicle, and the cloud platform may be a distributed cloud computing technology environment including a small computer system or a large computer system. Next, an application example in which the electronic device is implemented as a terminal or a server will be described.

[0032] 1A is a schematic diagram of a system architecture for applying a road obstacle detection method according to an embodiment of the present disclosure, and as shown in FIG. 1A, the system architecture includes an image capture device 11, a network 12, and an in-vehicle control terminal 13. To realize and support an exemplary application, the image capture device 11 and the in-vehicle control terminal 13 establish a communication connection via the network 12, and firstly, the image capture device 11 reports the captured road image to the in-vehicle control terminal 13 via the network 12, and the in-vehicle control terminal 13 recognizes a drivable area and an ideal driving area in the road image, and determines an obstacle area on the road according to the two areas.

[0033] As an example, the image capture device 11 may include a visual processing device having visual information processing capabilities. The network 12 may employ a wired or wireless connection method. Here, when the image capture device 11 is a visual processing device, the vehicle control terminal 13 may be communicatively connected to the visual processing device by a wired connection method such as data communication via a bus.

[0034] Or, in some scenes, the image capture device 11 may be a visual processing device with a video collection module, or a host computer equipped with a camera. In this case, the road obstacle detection method of the embodiment of the present disclosure may be executed by the image capture device 11, and the above system architecture may not include the network 12 and the in-vehicle control terminal 13.

[0035] In order to make the objectives, technical solutions and advantages of the embodiments of the present disclosure clearer, the specific technical solutions of the present invention will be described in more detail below with reference to the drawings of the embodiments of the present disclosure. The following embodiments are intended to illustrate the present disclosure, but are not intended to limit the scope of the present disclosure.

[0036] In the following, references to "some embodiments" describe a subset of all possible embodiments, but it is understood that "some embodiments" may be the same or different subsets of all possible embodiments, and may be combined with each other without inconsistency.

[0037] The road obstacle detection method can be applied to a computer device, and the functions realized by the method can be realized by a processor in the computer device calling a program code, and of course, the program code may be stored in a computer storage medium, and it is understood that the computer device includes at least a processor and a storage medium.

[0038] FIG. 1B is a flowchart illustrating an implementation of a method for detecting road obstacles according to an embodiment of the present disclosure, which will be described in conjunction with the steps shown in FIG. 1B as shown in FIG.

[0039] In step S101, a road image is acquired.

[0040] In some embodiments, the road image may be an image collected on any road, may be an image with complex screen content, or may be an image with simple screen content, such as a road image collected by an image collection device on a vehicle.

[0041] In some embodiments, the image collection device may be mounted on an on-board device of a vehicle or may be independent of the on-board device. The on-board device may be communicatively connected to a sensor of the vehicle, a positioning device, etc., and the on-board device may acquire data collected by the sensor of the vehicle, geolocation information reported by the positioning device, etc., through the communicative connection. Exemplarily, the sensor of the vehicle may be at least one of a millimeter wave radar, a laser radar, a camera, etc., and the positioning device may be a device for providing a positioning service based on at least one of a Global Positioning System (GPS), a Beidou satellite navigation system, a Galileo satellite navigation system, etc.

[0042] In some embodiments, the in-vehicle device may be an Advanced Driving Assistant System (ADAS), where the ADAS is installed in a vehicle, and the ADAS can acquire real-time position information of the vehicle from a positioning device of the vehicle, and / or the ADAS can acquire image data, radar data, etc. representing vehicle surrounding environment information from a sensor of the vehicle. Here, optionally, the ADAS can transmit vehicle driving data including the real-time position information of the vehicle to a cloud platform. In this manner, the cloud platform can receive the image data, radar data, etc. representing the real-time position information of the vehicle and / or the vehicle surrounding environment information.

[0043] Road images are acquired by an image collecting device (i.e., a sensor such as a camera) installed in a vehicle, and the image collecting device collects images of the surroundings of the vehicle in real time as the vehicle moves to obtain the road images. In some possible implementation forms, a camera mounted on the vehicle may collect images of the road on which the vehicle is traveling and the surrounding environment while the vehicle is traveling to obtain the road images.

[0044] In step S102, a driveable area and an ideal drive area in the road image are determined based on the road image.

[0045] In some embodiments, the vehicle-related road may be a road on which the vehicle is currently traveling. The drivable area and the ideal driving area may be determined independently, or the ideal driving area may be determined based on the drivable area. Taking the case where the drivable area and the ideal driving area are determined independently as an example, a road image is input to a trained neural network, and semantic segmentation is performed on the road image using different branch networks of the neural network to obtain the drivable area and the ideal driving area on the road. The neural network may be a neural network for performing semantic segmentation. A drivable area in the road image is detected by the neural network, and the drivable area (free space) represents the drivable road area of ​​the vehicle. In addition to the current vehicle, the road image usually includes other vehicles, pedestrians, trees, road edges, etc., and the area where the other vehicles, pedestrians, trees, road edges, etc. are located is an area where the vehicle cannot currently travel. Therefore, by performing semantic segmentation on the road image, the area where the other vehicles, pedestrians, trees, road edges, etc. are located in the road image is removed to obtain the drivable area of ​​the vehicle. The ideal driving area is the actual road area that can be driven on the road surface, assuming that there are no vehicles, pedestrians or objects on the road surface. In this way, the coverage area of ​​the ideal driving area is larger than the drivable area, or the ideal driving area includes the drivable area on the road.

[0046] In step S103, an obstacle area on the road is determined based on the travelable area and the ideal travel area.

[0047] In some embodiments, an obstacle area of ​​a road is an area where any type of obstacle that affects the normal running of the vehicle is located, such as other vehicles on the road, pedestrians, trees, etc.

[0048] In some possible implementations, after recognizing the drivable area and the ideal area of ​​the road for the road on which the vehicle is located in the road image, the overlapping and non-overlapping areas between the two areas are determined. The overlapping area between the drivable area and the ideal driving area is a road area that exists in the drivable area and also exists in the ideal driving area, which indicates that the overlapping area is the actual road area of ​​the road. The overlapping area is an area where the road vehicle can travel, and furthermore, the overlapping area does not have any obstacles. The non-overlapping area between the drivable area and the ideal driving area is an area that exists in the ideal driving area and does not exist in the drivable area, which indicates that the non-overlapping area is an area where the vehicle cannot travel, and furthermore, the non-overlapping area is an area where an obstacle exists, i.e., an obstacle area on the road.

[0049] In the embodiment of the present disclosure, the acquired road image is detected, the vehicle-related drivable area and the ideal driving area in the road image are recognized, and the drivable area and the ideal driving area are compared to effectively determine the obstacle area in which an obstacle of any shape is located on the road. In this way, by comparing the drivable area with the ideal driving area, the obstacle area on the road can be quickly recognized, and the stability of the recognition of obstacles of different shapes can be improved.

[0050] In some embodiments, not only can a drivable area be obtained by inputting a road image into the first network, but also a drivable area in the road image can be recognized by comparing screen information of the road image with screen information of a predetermined road image. That is, in the above step S101, the step of determining the drivable area in the road image based on the road image can be realized by the following steps S111 to S113 (not shown).

[0051] In step S111, an image area in which the similarity between the texture in the road image and the information representing the road texture is greater than a predetermined similarity threshold is determined.

[0052] In some embodiments, the road texture information may be a road surface image obtained by collecting roads on which the vehicle can travel, or may be an image area cropped from the road image where the screen shows the road.

[0053] In some possible implementations, texture features of a road image are extracted, and among the extracted texture features, texture features that have a high similarity to information representing the road texture are found, and the image regions corresponding to the texture features are determined.

[0054] In step S112, the image area is determined as the drivable area.

[0055] In some embodiments, an image area in a road image that has a high similarity to screen information of a predetermined road image means that the screen in the image area is an area that has a high similarity to the screen of the predetermined road image, and further means that the screen in the image area is a road surface, i.e., that the image area is an area where a vehicle can drive. In this way, by recognizing an image area in a road image that has a high similarity to the screen information of the predetermined road image as a drivable area, the recognition complexity of the drivable area is reduced and realization is made easier.

[0056] In some embodiments, not only can an ideal driving area be obtained by inputting a road image into a neural network (e.g., a first network), but also the ideal driving area can be determined by recognizing road edges of a vehicle-related road and dividing an area surrounded by the road edges. That is, in the above step S101, the step of determining the ideal driving area based on the road image can be realized by the following steps S113 and S114 (not shown).

[0057] In step S113, the road image is detected to determine road edges in the road image.

[0058] In some embodiments, a third network is employed to recognize road edges of the roads and determine road edges in the road image. The third network may be any network for performing road edge recognition, such as a residual network, a Visual Geometry Group (VGG) network, etc. A road image is input to the third network, feature extraction is performed on the road image, and the road edges are recognized based on the extracted image features to recognize the road edges of the vehicle-related roads.

[0059] In step S114, the area surrounded by the road edges is determined as the ideal driving area.

[0060] In some embodiments, the recognized road edge can be understood as the outermost contour of the road, and thus the area surrounded by the outermost contour is the actual road area without any type of vehicle, pedestrian or object on the road, so that the area surrounded by the road edge can be taken as the ideal driving area, thereby making the coverage of the determined ideal driving area more comprehensive.

[0061] In some embodiments, the ideal driving area may be determined based on the drivable area. That is, in the above step S101, the step of determining the ideal driving area based on the drivable area may be realized by the following process.

[0062] The ideal driving area is determined based on the driving possible area.

[0063] Here, after recognizing the vehicle's driveable area in the road image by any of the above-mentioned embodiments, the driveable area is expanded on the road on which the driveable area is located based on the driveable area to determine the ideal driveable area. In this way, by determining the ideal driveable area based on the driveable area, the ideal driveable area can be made to include the driveable area, and thus the obstacle area determined by comparing the ideal driveable area with the driveable area becomes more accurate.

[0064] In some embodiments, the step of determining the ideal driving area based on the drivable area can be realized by the following steps.

[0065] In a first step, at least one contour of the drivable area is determined.

[0066] In some embodiments, the drivable area includes multiple cavities because the drivable area is the area of ​​the road surface obtained by removing other vehicles, objects, etc. on the road from the entire road surface. In this way, the drivable area has multiple contours.

[0067] In a second step, a target contour having the largest area of ​​an enclosed region is determined from among the at least one contour.

[0068] In some embodiments, the area of ​​an area enclosed by each of the at least one contour is determined, and the contour with the largest area is selected as the target contour, which may be understood to be the outermost contour of the drivable area of ​​the at least one contour.

[0069] In a third step, the area surrounded by the target contour is determined as the ideal traveling area.

[0070] In some embodiments, the target contour may be the outermost contour of the drivable area, so it is reasonable to determine the area surrounded by the target contour as the ideal drivable area. In this way, by recognizing multiple contours of the drivable area and determining the area surrounded by the outermost target contour as the ideal drivable area, the realization process is simplified and the accuracy of the determined ideal drivable area is increased.

[0071] In some embodiments, before determining the area surrounded by the target contour as the ideal driving area, it is also possible to first analyze whether the area surrounded by the target contour includes two road edges of the road. That is, Before determining the area surrounded by the target contour as the ideal driving area, the method for detecting road obstacles includes: The method further includes determining that a region bounded by the target contour includes two road edges of the road.

[0072] In this manner, when the area enclosed by the target contour includes two road edges of the road, the area enclosed by the target contour is determined as the ideal driving area.

[0073] In some embodiments, when the enclosed area of ​​the target contour includes two road edges, it means that the enclosed area of ​​the target contour covers the entire road surface, so the obtained ideal driving area is more reasonable.

[0074] In some embodiments, the above step S101 can also be realized in the following two ways.

[0075] Method 1: The ideal driving area can also be determined by detecting multiple roadways on the road and analyzing the traffic rules corresponding to the roadways. That is, in the above step S102, the step of determining the ideal driving area based on the road image can also be realized by the following steps.

[0076] In a first step, the road image is detected to obtain a plurality of roadways in the road image.

[0077] In some embodiments, a detector can be employed to detect lanes in a road image. Based on the detected lanes, the road area between two adjacent lanes is treated as one roadway to obtain the roadways. The road image can also be input to a trained lane detection network, which directly outputs the roadways on the road.

[0078] In a second step, the ideal driving area is determined based on the plurality of roadways and traffic rules corresponding to the plurality of roadways.

[0079] In some embodiments, after detecting multiple lanes of a road, road surface signals (e.g., arrows, solid or broken lines, etc. on the road surface) on each road can be obtained, and the meaning represented by the road surface signals can be determined according to traffic rules. For example, a solid white line is a boundary line that prohibits crossing of roads in the same direction. In this way, the area covered by each road can be determined by determining the area covered by each road and the traffic rules corresponding to the road surface signals on each road, and the area covered by multiple roads can be combined to determine the actual road area in which a road vehicle can travel when there are no obstacles, that is, the ideal driving area. In this way, by determining the area surrounded by the roads as the ideal driving area by roadway detection, not only can the method of determining the ideal driving area be enriched, but also the ideal driving area can be made to completely include the covered area of ​​each roadway, and the accuracy of determining the ideal driving area can be improved.

[0080] Method 2: The ideal driving area and the drivable area are obtained by performing semantic segmentation on the actual road area and the vehicle drivable area of ​​the road in the road image through different network branches in the same neural network. That is, the above step S101 can be realized by the following process.

[0081] Semantic segmentation is performed on the road image to obtain the driveable area and the ideal drive area within the road image.

[0082] In some embodiments, a first network is first employed to perform semantic segmentation on the vehicle's drivable road area in the road in the road image to obtain the drivable area, and then a second network is employed to perform semantic segmentation on the actual road area of ​​the vehicle-related road in the road image to obtain the ideal drivable area. The first network may be any type of neural network trained to perform semantic segmentation on the vehicle's drivable road area and the vehicle's non-drivable road area on the road, and the first network may be, for example, a network branch of any neural network, such as a fully convolutional neural network, a mask region convolutional neural network (Mask R-CNN), a deep neural network, etc. A road image is input to the first network to perform feature extraction on the road image, and based on the extracted image features, semantic segmentation is performed on the vehicle's drivable road area and the vehicle's non-drivable area in the road image, thereby quickly and accurately recognizing the drivable area in the road image.

[0083] The second network and the first network may be different network branches of the same neural network, and the second network may be any type of neural network trained to perform semantic segmentation on the actual road and non-road regions on the road. By inputting a road image into the second network, the second network performs semantic segmentation on the actual road and non-road regions of the road in the road image, and the road region obtained by segmentation is the ideal driving region. The actual road region is a road region where a vehicle can actually drive when there are no objects of any type on the road.

[0084] The above process can be performed simultaneously through different network branches in a neural network. A road image is input to the neural network to perform feature extraction, and the extracted features are input to a first network and a second network, respectively, and a semantic segmentation is performed on the drivable road area of ​​the vehicle in the road image through the first network branch of the neural network to obtain a drivable area, and at the same time, a semantic segmentation is performed on the actual road area of ​​the road in the road image through the second network branch of the neural network to obtain an ideal drivable area. In this way, the drivable area and the ideal road area are divided from the road image through different network branches in the same neural network, thereby realizing quick and accurate area division.

[0085] In the embodiments of the present disclosure, the driveable area and ideal driveable area are determined using a plurality of methods, so that the driveable area and ideal driveable area of ​​the vehicle can be recognized with high accuracy in all cases.

[0086] In some embodiments, the obstacle area of ​​the road is determined by determining a non-overlapping area between the travelable area and the ideal travel area. That is, the above step S103 can be realized by the steps shown in FIG.

[0087] In step S201, a region within the ideal driving region that does not overlap with the driving possible region is determined.

[0088] In some embodiments, the ideal driving area is compared with the drivable area to determine an overlap area between the two areas and a non-overlapping area between the two areas. The overlap area is an actual road area and may be equal to the drivable area. The non-overlapping area between the two areas is an area that is in the ideal driving area but not in the drivable area, which means that the non-overlapping area is an area where the vehicle cannot currently travel.

[0089] In step S202, the non-overlapping region is determined as the obstacle region.

[0090] In some embodiments, an area in which the vehicle cannot currently travel (i.e., a non-overlapping area) means that an obstacle that affects the normal travel of the vehicle exists in the area, and the obstacle may be another vehicle, a pedestrian, a tree, or another object, etc. In this way, by determining the non-overlapping area as an obstacle area, it is possible to accurately detect an obstacle area that affects the travel of the vehicle regardless of the position or shape of the obstacle, and the stability of obstacle recognition is improved.

[0091] In some embodiments, the route of the vehicle is re-planned according to the detected obstacle area, so as to more effectively control the vehicle travel, that is, after step S103, the method further includes the steps shown in FIG.

[0092] In step S301, a driving route for the vehicle is determined based on the obstacle area.

[0093] In some embodiments, the driving route includes the vehicle's driving speed and driving direction, etc. By analyzing the position of the obstacle area on the road and the positional relationship between the obstacle area and the vehicle, the driving route for the vehicle to drive on the road is re-planned. For example, when the distance between the vehicle and the obstacle area is close, the vehicle is controlled to prepare for braking, or the vehicle is controlled to move away from the obstacle area.

[0094] In step S302, the driving of the vehicle is controlled based on the driving route.

[0095] In some embodiments, the autonomous vehicle driving system can determine a driving route for the vehicle and further control the vehicle to drive according to the driving route.

[0096] After the obstacle area is recognized through the above steps S301 and S302, a subsequent driving route is generated in combination with the road information, so that the generated driving route is more accurate, and the vehicle can be controlled accurately according to the driving route.

[0097] In the following, an exemplary application of the embodiments of the present disclosure in a practical application scenario is described by taking as an example adopting a deep neural network on road markings to determine vehicle turning directions at road intersections.

[0098] Great progress has been made in the field of autonomous driving, mainly focusing on deep learning, including the fields of image segmentation and target detection. However, autonomous driving is a holistic system, and the output of the sensing module serves the following modules, for example, the sensing result not only needs to provide whether an object exists ahead, but also to provide the corresponding logic output for the following modules, and provide certain control and logic signals for autonomous driving.

[0099] An embodiment of the present disclosure provides a method for detecting road obstacles, and from the perspective of the field of semantic segmentation, determines a non-overlapping area between an ideal driving area and a driving area, and performs obstacle detection, thereby effectively recognizing obstacles with irregular shapes and indefinite positions, effectively evaluating the distance of the obstacles, and providing useful information to subsequent decision-making, planning, and control modules of an autonomous driving system.

[0100] An embodiment of the present disclosure provides a method for detecting road obstacles, using a deep learning method to obtain a drivable area and an ideal driving area for an autonomous vehicle, and using the drivable area and the ideal driving area to obtain a general-purpose obstacle detection result on the road surface, thereby providing a valid signal for subsequent autonomous driving route planning.

[0101] After performing semantic segmentation on the collected road image, the drivable area and ideal driving area in the image are determined. As shown in FIG. 4, semantic segmentation is performed on the road image 401 to obtain the drivable area and the ideal driving area in the road of the image, where, as shown in FIG. 5 and FIG. 6, the area 501 in FIG. 5 represents the drivable area, and the area 601 in FIG. 6 represents the ideal driving area. Comparing the area 501 and the area 601, it can be seen that the drivable area excludes vehicles, pedestrians, and some irregular obstacles, whereas the ideal driving area recognizes all areas where a vehicle can drive in an ideal state. This shows that the main difference between the ideal driving area and the drivable area is the recognition of obstacles.

[0102] In some embodiments, during autonomous driving, the vehicle performs sensing output according to information provided on the road, and performs information integration on the results output from the model. As shown in FIG. 7, FIG. 7 is a structural diagram of a road sensing signal output network according to an embodiment of the present disclosure, where: The image input module 701 is configured to input a road image; The backbone network 702 is configured to perform feature extraction on the input road image.

[0103] Here, the backbone network may be a residual network or a VGG network, a GoogleNet network, a ResNet network, etc.

[0104] The drivable area branching network 703 is configured to realize the segmentation task, and performs semantic segmentation based on the extracted image features and outputs the drivable area.

[0105] In some possible implementations, the image features extracted by the backbone network 702 are input to a drivable area branching network 703 that performs a semantic segmentation task, segmenting the drivable area, and outputting the drivable area.

[0106] In some possible implementations, the drivable area branching network 703 finds similar road objects as drivable areas because roads have similar textures, and thus the drivable area branching network 703 can generate a drivable area that excludes obstacles. In this way, the drivable area branching network 403 can robustly detect drivable areas in road areas.

[0107] The ideal driving region branching network 704 is configured to realize the segmentation task, and performs semantic segmentation based on the extracted image features and outputs the ideal driving region.

[0108] In some possible implementations, the image features extracted by the backbone network 702 are input to an ideal driving area branching network 704 that performs a semantic segmentation task, thereby performing segmentation into an ideal driving area and outputting the ideal driving area. The ideal driving area branching network 704 recognizes which areas of the road area are actual road areas (which need to be recognized even if there are obstacles on the road surface) and which areas are not actual road areas.

[0109] In some possible implementations, the ideal running region branching network 704 determines the region within the outermost contour of the input image region as the ideal region.

[0110] In some possible implementations, the ideal driving area can be determined based on the driveable area, for example, the outermost contour of the driveable area is determined as the ideal driving area.

[0111] In some possible implementations, the area surrounded by the road is determined as the ideal driving area by performing roadway detection on the road image. In this way, the ideal driving area can be determined based on the detected roadway area and the roadway driving rules of the road.

[0112] In some embodiments, after the drivable area and the ideal driving area are determined by the drivable area branching network 703 and the ideal driving area branching network 704, the results of the two areas are compared to determine the overlapping area as the actual road area, and the area where the two do not overlap (in most cases, the area of ​​the ideal driving area is larger than the area of ​​the drivable area), i.e., the area that exists in the ideal driving area but does not exist in the drivable area, is recognized and output as an obstacle area. In this way, vehicles, pedestrians, non-automobiles, and obstacles whose shapes are not fixed can be effectively recognized.

[0113] In the embodiment of the present disclosure, based on the deep learning model, the driving possible area and the ideal driving area are effectively recognized, and the two learning tasks are concentrated in one network, and the two branches are integrated into one and the same backbone network, which not only accelerates the model processing speed, but also effectively reduces the pressure on the operating speed.

[0114] Those skilled in the art can understand that in the above method of the specific embodiment, the described order of each step does not limit the implementation process in a strict execution order, and the specific execution order of each step should be determined by its function and possible internal logic.

[0115] Based on the same technical concept, the embodiments of the present disclosure further provide a movement intention determination device corresponding to the movement intention determination method, and the principle for solving the problems of the device in the embodiments of the present disclosure is similar to the above-mentioned movement intention determination method in the embodiments of the present disclosure, so the implementation of the device can refer to the above-mentioned method embodiments.

[0116] An embodiment of the present disclosure provides a road obstacle detection device. FIG. 8 is a schematic diagram showing a configuration of a road obstacle detection device according to an embodiment of the present disclosure. As shown in FIG. 8, the road obstacle detection device 800 includes: an image capture portion 801 configured to capture road images; A driving area determination portion 802 configured to determine a driveable area and an ideal driving area in the road image based on the road image; and an obstacle area determination portion 803 configured to determine an obstacle area on a road based on the driveable area and the ideal drive area.

[0117] In some embodiments, the travel area determination portion 802 includes: an image region recognition subpart configured to determine image regions in which a similarity between a texture in the road image and information representative of the road texture is greater than a predefined similarity threshold; and a drivable area determination subportion configured to determine the image area as the drivable area.

[0118] In some embodiments, the travel area determination portion 802 includes: a road edge recognition subpart configured to detect the road image and determine road edges in the road image; and a first ideal driving area determination sub-portion configured to determine an area surrounded by the road edges as the ideal driving area.

[0119] In some embodiments, the travel area determination portion 802 includes: The vehicle further includes a second ideal driving area determination sub-portion configured to determine the ideal driving area based on the driveable area.

[0120] In some embodiments, the second ideal driving region determination submodule: a contour determining portion configured to determine at least one contour of the drivable area; a target contour determining portion configured to determine a target contour having a largest area of ​​an enclosed region among the at least one contour; and an ideal traveling area determining portion configured to determine an area surrounded by the target contour as the ideal traveling area.

[0121] In some embodiments, the second ideal driving region determination submodule further comprises: The method is configured to determine that a region bounded by the target contour includes two road edges of the road.

[0122] In some embodiments, the travel area determination portion 802 includes: a roadway sub-detection unit configured to detect the road image to obtain a plurality of roadways in the road image; and a fourth ideal driving area determination subpart configured to determine the ideal driving area based on the plurality of roadways and traffic rules corresponding to the plurality of roadways.

[0123] In some embodiments, the travel area determination portion 802 further comprises: The system is configured to perform semantic segmentation on the road image to obtain the driveable area and the ideal drive area within the road image.

[0124] In some embodiments, the obstacle region determination portion 803 includes: a non-overlapping area determination sub-portion configured to determine an area in the ideal driving area that does not overlap with the driving area; and an obstacle region determining subportion configured to determine the non-overlapping region as the obstacle region.

[0125] In some embodiments, the road obstacle detection device comprises: A driving route determination portion configured to determine a driving route of the vehicle based on the obstacle area; and a vehicle control portion configured to control driving of the vehicle based on the driving route.

[0126] It should be noted that the above description of the apparatus embodiment is similar to the above description of the method embodiment, and has similar beneficial effects as the method embodiment. The technical details not disclosed in the apparatus embodiment of the present disclosure can be understood by referring to the description of the method embodiment of the present disclosure.

[0127] In this and other embodiments of the present disclosure, a "module" may be a circuit, a processor, a program, or software, and may of course be a unit or a non-module.

[0128] It should be noted that in the embodiments of the present disclosure, the above-mentioned road obstacle detection method may be implemented as a software function module, and may be stored in a computer-readable storage medium when sold or used as an independent product. Based on this understanding, the essential part of the technical solution of the embodiments of the present disclosure, or the part that contributes to the prior art, may be embodied in the form of a software product, and the computer software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a terminal, a server, etc.) to execute all or part of the method described in each embodiment of the present disclosure. The aforementioned storage medium includes various media that can store program code, such as a U disk, a mobile hard disk, a read only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk. In this way, the embodiments of the present disclosure are not limited to a specific combination of hardware and software.

[0129] Correspondingly, the embodiment of the present disclosure further provides a computer program product having computer executable instructions stored therein, which can realize the steps of the road obstacle detection method provided by the embodiment of the present disclosure after being executed. Correspondingly, the embodiment of the present disclosure further provides a computer storage medium having computer executable instructions stored therein, which, when executed by a processor, realizes the steps of the road obstacle detection method provided by the above embodiment. Correspondingly, the embodiment of the present disclosure provides a computer device, and FIG. 9 is a schematic diagram showing the configuration of a computer device according to the embodiment of the present disclosure. As shown in FIG. 9, the computer device 900 includes one processor 901, at least one communication bus, a communication interface 902, at least one external communication interface, and a memory 903. Here, the communication interface 902 realizes the connection communication between these components. Here, the communication interface 902 may include a display, and the external communication interface may include a standard wired interface and a wireless interface. Here, the processor 901 executes a road obstacle detection program in the memory to realize the steps of the road obstacle detection method provided by the above embodiment.

[0130] The above description of the road obstacle detection device, computer device, and storage medium embodiments is similar to the description of the above method embodiments, and has similar technical descriptions and beneficial effects to the method embodiments. Due to space limitations, reference can be made to the description of the above method embodiments, and will not be repeated here. Technical details not disclosed in the road obstacle detection device, computer device, and storage medium embodiments of the present disclosure can be understood by referring to the description of the method embodiments of the present disclosure. It should be understood that in this specification, "one embodiment" or "one embodiment" means that a particular feature, structure, or characteristic associated with an embodiment is included in at least one embodiment of the present disclosure. Therefore, "one embodiment" or "one embodiment" in the entire specification does not necessarily refer to the same embodiment. Furthermore, these particular features, structures, or characteristics can be incorporated in one or more embodiments in any suitable manner. It should be understood that in each embodiment of the present disclosure, the magnitude of the sequence number of each of the above processes does not mean the order of execution, and the execution order of each process should be determined by its function and internal logic, and should not constitute any restriction on the implementation process of the embodiment of the present disclosure. The sequence numbers of the above embodiments of the present disclosure are for illustrative purposes only and do not represent the superiority or inferiority of the embodiments.

[0131] It should be explained that, as used herein, the terms "comprising," "including," or any other variation thereof, are intended to cover a non-exclusive inclusion, whereby a process, method, article, or apparatus that comprises a set of elements includes not only those elements, but also other elements not expressly listed or that are inherent to such process, method, article, or apparatus. Unless specifically limited, an element defined with the phrase "comprising..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes the element.

[0132] It should be understood that in some embodiments provided in the present disclosure, the disclosed devices and methods can be realized in other ways. The above-described device embodiments are merely illustrative, for example, the division of the units is merely a division of logical functions, and in actual implementation, there may be other division methods, for example, multiple units or components may be combined, integrated into another system, or some features may be ignored or not implemented. Furthermore, the mutual coupling or direct coupling or communication connection between each component shown or described may be an indirect coupling or communication connection via some interfaces, devices or units, and may be in an electrical, mechanical or other form.

[0133] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, and may be located in one place or distributed among multiple network units. Some or all of the units may be selected according to actual needs to achieve the objectives of the technical solutions of the present embodiment. In addition, the functional units in each embodiment of the present disclosure may all be integrated into one second processing unit, each unit may be used individually as one unit, or two or more units may be integrated into one unit. The integrated units may be embodied in the form of hardware or in the form of a combination of hardware and software functional units. Those skilled in the art will understand that all or some of the steps for implementing the above embodiments may be performed by hardware associated with program instructions, and the above-mentioned programs may be stored in a computer-readable storage medium, and when the programs are executed, the steps including the above-mentioned method embodiments are performed, and the above-mentioned storage medium includes various media capable of storing program code, such as removable storage, read-only memory (ROM), magnetic disk, or optical disk.

[0134] Alternatively, the above integrated units of the present disclosure may be realized in the form of a software functional module and stored in one computer-readable storage medium when sold or used as an independent product. Based on this understanding, the essential part of the technical solution of the embodiments of the present disclosure or the part that contributes to the prior art may be embodied in the form of a software product, and the computer software product is stored in one storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the method described in each embodiment of the present disclosure. The above storage medium includes various media that can store program code, such as removable storage, ROM, magnetic disk, or optical disk. The above content is merely an embodiment of the present disclosure, and the protection scope of the present disclosure is not limited thereto. Any modifications or replacements that a person skilled in the art can easily think of within the technical scope disclosed in this disclosure should be included in the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure shall be subject to the protection scope of the claims. [Industrial Applicability]

[0135] An embodiment of the present disclosure provides a method, apparatus, device, and storage medium for detecting road obstacles, the method including the steps of acquiring a road image, determining a driveable area and an ideal driving area in the road image based on the road image, and determining an obstacle area on the road based on the driveable area and the ideal driving area.

Claims

1. A method for detecting an obstacle, the method comprising: acquiring a road image; determining a driveable area and an ideal drive area in the road image based on the road image; determining an obstacle area on a road based on the travelable area and the ideal travel area.

2. The step of determining a drivable area in the road image based on the road image includes: determining an image area in which a similarity between a texture in the road image and information representing the road texture is greater than a predetermined similarity threshold; determining the image area as the drivable area; The method for detecting an obstacle according to claim 1 .

3. The step of determining the ideal driving area based on the road image includes: detecting said road image to determine road edges within said road image; determining an area surrounded by the road edges as the ideal driving area; The method for detecting an obstacle according to claim 1 or 2.

4. The step of determining the ideal driving area based on the road image includes: determining the ideal driving area based on the driving possible area, The method for detecting an obstacle according to claim 1 .

5. The step of determining the ideal traveling area based on the travelable area includes: determining at least one contour of the drivable area; determining a target contour having a largest area of ​​an enclosed region among the at least one contour; determining an area surrounded by the target contour as the ideal traveling area; The method for detecting an obstacle according to claim 4.

6. Before determining the area surrounded by the target contour as the ideal traveling area, the obstacle detection method includes: determining that an area enclosed by the target contour includes two road edges of the road; The method for detecting an obstacle according to claim 5.

7. The step of determining the ideal driving area based on the road image includes: detecting the road image to obtain a plurality of roadways in the road image; determining the ideal driving area based on the plurality of roadways and traffic rules corresponding to the plurality of roadways; The method for detecting an obstacle according to claim 1 .

8. The step of determining a driveable area and an ideal drive area in the road image based on the road image includes: The method includes a step of performing semantic segmentation on the road image to obtain the driveable area and the ideal drive area in the road image. The method for detecting an obstacle according to claim 1 .

9. The step of determining an obstacle area on a road based on the travelable area and the ideal travel area includes: determining an area within the ideal driving area that does not overlap with the driving possible area; determining the non-overlapping region as the obstacle region. The method for detecting an obstacle according to any one of claims 1 to 8.

10. After determining the obstacle area on the road, the obstacle detection method includes: determining a travel route for the vehicle based on the obstacle area; and controlling the travel of the vehicle based on the travel route. The method for detecting an obstacle according to any one of claims 1 to 9.

11. A road obstacle detection device, comprising: an image capture portion configured to capture road images; a driving area determination unit configured to determine a driveable area and an ideal driving area in the road image based on the road image; and an obstacle area determining portion configured to determine an obstacle area on a road based on the travelable area and the ideal travel area.

12. A computer storage medium having stored thereon computer executable instructions which, when executed, are capable of implementing the steps of the method for detecting obstacles according to any one of claims 1 to 10.

13. A computing device comprising: a memory having computer-executable instructions stored therein; and a processor that, when executing the computer-executable instructions on the memory, implements the steps of the method for detecting an obstacle according to any one of claims 1 to 10.

14. A computer program product comprising a computer program or instructions which, when executed on an electronic device, causes said electronic device to carry out the method for detecting obstacles according to any one of claims 1 to 10.

Images