Road obstacle detection method, road obstacle detection device, computer storage medium, computer device, and computer program

Abstract

A method, device, equipment, and storage medium for detecting road obstacles, the method includes a step of acquiring a road image (S101), a step of determining a drivable area and an ideal driving area in the road image based on the road image (S102), and a step of determining an obstacle area on the road based on the drivable area and the ideal driving area (S103). In this way, by comparing the drivable area with the ideal driving area, it is possible to quickly recognize an obstacle area on the road, and further to improve the stability of recognizing obstacles of different shapes.

Description

[Technical Field]

[0001] (CROSS-REFERENCE TO RELATED APPLICATIONS) This application claims priority to a Chinese patent application filed with the China Patent Office on March 24, 2022, bearing application number 202210303623.0 and entitled "Road Obstacle Detection Method, Apparatus, Device and Storage Medium," the entire contents of which are incorporated herein by reference.

[0002] The present application relates to the technical field of intelligent driving, and in particular, but not exclusively, to a method, device, apparatus and storage medium for detecting road obstacles. [Background technology]

[0003] Autonomous driving systems usually use target detection methods to recognize obstacles, but because the shape and location of the obstacles are unknown, a large amount of sample data is required, making obstacle detection difficult and not very accurate. Summary of the Invention

[0004] The embodiments of the present disclosure provide a technical solution for road obstacle detection.

[0005] The technical solutions of the embodiments of the present disclosure are realized as follows:

[0006] An embodiment of the present disclosure provides a method for detecting road obstacles, the method including the steps of acquiring a road image, determining a drivable area and an ideal driving area in the road image based on the road image, and determining an obstacle area on the road based on the drivable area and the ideal driving area.

[0007] In some embodiments, the step of determining a drivable area in the road image based on the road image includes the steps of determining an image area in which a similarity between a texture in the road image and information representing a road texture is greater than a predetermined similarity threshold, and determining the image area as the drivable area. In this way, by recognizing an image area in the road image in which a similarity between screen information and information representing a road texture is high as the drivable area, the complexity of recognizing the drivable area is reduced and implementation is easier.

[0008] In some embodiments, the step of determining the ideal driving area based on the road image includes the steps of detecting the road image, determining road edges in the road image, and determining an area surrounded by the road edges as the ideal driving area. In this way, by setting the area surrounded by the road edges as the ideal driving area, the coverage of the determined ideal driving area can be made more comprehensive.

[0009] In some embodiments, determining the ideal driving area based on the road image includes determining the ideal driving area based on the drivable area. By determining the ideal driving area based on the drivable area in this manner, the obstacle area determined by comparing the ideal driving area with the drivable area becomes more accurate.

[0010] In some embodiments, the step of determining the ideal driving area based on the drivable area includes the steps of determining at least one contour of the drivable area, determining a target contour among the at least one contour that has the largest area enclosed by the target contour, and determining the area enclosed by the target contour as the ideal driving area. In this way, by recognizing multiple contours of the drivable area and determining the area enclosed by the outermost target contour as the ideal driving area, the implementation process is simplified and the accuracy of the determined ideal driving area is increased.

[0011] In some embodiments, before determining the area surrounded by the target contour as the ideal driving area, the road obstacle detection method further includes determining that the area surrounded by the target contour includes two road edges of the road. Then, the ideal driving area is jointly determined based on the target contour and the two road edges, making the obtained ideal driving area more reasonable.

[0012] In some embodiments, determining the ideal driving area based on the road image includes detecting the road image to obtain a plurality of roadways in the road image, and determining the ideal driving area based on the plurality of roadways and traffic rules corresponding to the plurality of roadways. In this way, the ideal driving area can be made to completely include the coverage area of ​​each roadway, and the accuracy of determining the ideal driving area can be improved.

[0013] In some embodiments, the step of determining a drivable area in the road image based on the road image includes the steps of determining an image area in which a similarity between a texture in the road image and information representing the road texture is greater than a predetermined similarity threshold, and determining the image area as the drivable area. In this way, by dividing the road image into a drivable area and an ideal road area, it is possible to quickly and accurately achieve area division.

[0014] In some embodiments, the step of determining an obstacle area on the road based on the drivable area and the ideal driving area includes the steps of determining an area in the ideal driving area that does not overlap with the drivable area, and determining the non-overlapping area as the obstacle area. By determining the non-overlapping area as the obstacle area in this way, it is possible to accurately detect obstacle areas that affect vehicle driving, as well as to improve the stability of obstacle recognition.

[0015] In some embodiments, after determining the obstacle area on the road, the road obstacle detection method further includes determining a driving path of the vehicle based on the obstacle area, and controlling the driving of the vehicle based on the driving path. In this way, the generated driving path becomes more accurate, and based on this, accurate control of the vehicle can be realized according to the driving path.

[0016] An embodiment of the present disclosure provides a road obstacle detection device, the device comprising: an image acquisition portion configured to acquire a road image; a driving area determination portion configured to determine a drivable area and an ideal driving area in the road image based on the road image; and an obstacle area determination portion configured to determine an obstacle area on a road based on the drivable area and the ideal driving area.

[0017] Correspondingly, an embodiment of the present disclosure provides a computer storage medium having computer-executable instructions stored thereon, which, after being executed, can realize the steps of the above-mentioned road obstacle detection method.

[0018] An embodiment of the present disclosure provides a computer device, the computer device including: a memory having computer-executable instructions stored therein; and a processor capable of implementing the steps of the above-described road obstacle detection method when executing the computer-executable instructions on the memory.

[0019] An embodiment of the present disclosure further provides a computer program product including a computer program or instructions, which, when executed on an electronic device, causes the electronic device to perform the steps in any possible embodiment of the first aspect above.

[0020] The embodiments of the present disclosure provide a road obstacle detection method, device, equipment, and storage medium, which detect an acquired road image, recognize a drivable area and an ideal drivable area in the road image, and effectively determine an obstacle area on the road where an obstacle of any shape exists by comparing the drivable area with the ideal drivable area. In this way, by comparing the drivable area with the ideal drivable area, it is possible to quickly recognize an obstacle area on the road and further improve the stability of obstacle recognition for different shapes.

[0021] In order to make the above objects, features and advantages of the embodiments of the present disclosure more comprehensible, preferred embodiments accompanied with drawings are described in detail below. [Brief explanation of the drawings]

[0022] [Figure 1A] 1 is a schematic diagram of a system architecture to which the road obstacle detection method of the present disclosure can be applied; [Figure 1B] 1 is an implementation flowchart of a method for detecting road obstacles according to an embodiment of the present disclosure. [Figure 2] 1 is a flowchart illustrating another implementation of a method for detecting road obstacles according to an embodiment of the present disclosure. [Figure 3] 10 is a flowchart illustrating yet another implementation of a method for detecting road obstacles according to an embodiment of the present disclosure. [Figure 4] 1 is a schematic diagram of an application scene of a road obstacle detection method according to an embodiment of the present disclosure; [Figure 5] FIG. 2 is a schematic diagram of another application scene of the road obstacle detection method according to an embodiment of the present disclosure; [Figure 6] FIG. 10 is a schematic diagram of yet another application scene of the road obstacle detection method according to an embodiment of the present disclosure; [Figure 7] FIG. 1 is a network structure diagram of a road obstacle detection method according to an embodiment of the present disclosure. [Figure 8] 1 is a schematic diagram illustrating the configuration of a road obstacle detection device according to an embodiment of the present disclosure. [Figure 9]FIG. 1 is a schematic diagram illustrating a configuration of a computer device according to an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0023] In order to more clearly explain the technical solutions of the embodiments of the present disclosure, the drawings necessary for explaining the embodiments have been briefly described above. The drawings herein are incorporated into and constitute a part of this specification, and these drawings illustrate embodiments consistent with the present disclosure and are used together with the specification to explain the technical solutions of the present disclosure. It should be understood that the above drawings only illustrate some of the embodiments of the present disclosure and should not be considered as limiting the scope of protection; those skilled in the art can also obtain other related drawings based on these drawings without any creative efforts.

[0024] In order to make the objectives, technical solutions and advantages of the embodiments of the present disclosure clearer, the specific technical solutions of the present invention will be described in more detail below with reference to the drawings of the embodiments of the present disclosure. The following examples are for illustrating the present disclosure, but are not intended to limit the scope of the present disclosure.

[0025] In the following, references to "some embodiments" describe a subset of all possible embodiments, but it is understood that "some embodiments" may be the same or different subsets of all possible embodiments and may be combined with each other without contradiction.

[0026] In the following, terms such as "first / second / third" do not limit a particular order but distinguish between similar objects. It is understood that "first / second / third" can be used to translate a particular order or order when appropriate, so that the embodiments of the present disclosure described in some examples can be implemented in an order other than that shown or described in some examples.

[0027] Unless otherwise specified, all technical and scientific terms used in this disclosure have the same meaning as commonly understood by those skilled in the art to which this disclosure pertains. The terms used in this disclosure are used only to describe the embodiments of this disclosure and are not intended to limit the disclosure.

[0028] Before describing the embodiments of the present disclosure in detail, nouns and terms related to the embodiments of the present disclosure will be explained. The explanation of nouns and terms related to the embodiments of the present disclosure is as follows.

[0029] 1) Deep Neural Networks (DNNs): A type of deep learning framework, a neural network with at least one hidden layer. According to the positions of different layers in a DNN, the neural network layers within a DNN can be divided into three: an input layer, a hidden layer, and an output layer. As shown in the figure below, the first layer is generally the input layer, the last layer is the output layer, and the middle layer is a hidden layer. In the embodiments of the present disclosure, deep neural networks can be used to realize tasks such as semantic segmentation or target detection.

[0030] 2) Ego vehicle: A vehicle equipped with sensors for sensing the surrounding environment. The vehicle coordinate system is fixedly connected to the ego vehicle, where the x-axis is the direction the vehicle is moving, the y-axis points to the left of the direction the vehicle is moving, and the z-axis is perpendicular to the ground and points upward, conforming to a right-handed coordinate system. The origin of the coordinate system is located on the ground below the midpoint of the rear axle.

[0031] An example of application of the road obstacle detection method according to the embodiment of the present disclosure to an electronic device will be described below. The electronic device according to the embodiment of the present disclosure may be an in-vehicle device, a cloud platform, or other computer device. For example, the in-vehicle device may be a thin client, a fat client, a microprocessor-based system, a small computer system, or the like installed in a vehicle, and the cloud platform may be a distributed cloud computing technology environment including a small computer system or a large computer system. Next, an example of application when the electronic device is implemented as a terminal or a server will be described.

[0032] 1A is a schematic diagram of a system architecture for applying a road obstacle detection method according to an embodiment of the present disclosure. As shown in FIG. 1A, the system architecture includes an image capture device 11, a network 12, and an in-vehicle control terminal 13. To realize and support an exemplary application, the image capture device 11 and the in-vehicle control terminal 13 establish a communication connection via the network 12. First, the image capture device 11 reports captured road images to the in-vehicle control terminal 13 via the network 12. The in-vehicle control terminal 13 recognizes a drivable area and an ideal driving area in the road image, and determines an obstacle area on the road based on these two areas.

[0033] For example, the image capture device 11 may include a visual processing device having visual information processing capabilities. The network 12 may employ a wired or wireless connection method. If the image capture device 11 is a visual processing device, the vehicle control terminal 13 may be connected to the visual processing device via a wired connection method such as data communication via a bus.

[0034] Or, in some scenarios, the image capture device 11 may be a vision processing device with a video collection module or a host computer equipped with a camera, in which case the road obstacle detection method of the embodiment of the present disclosure may be executed by the image capture device 11, and the above system architecture may not include the network 12 and the in-vehicle control terminal 13.

[0035] In order to make the objectives, technical solutions and advantages of the embodiments of the present disclosure clearer, the specific technical solutions of the present invention will be described in more detail below with reference to the drawings of the embodiments of the present disclosure. The following examples are intended to illustrate the present disclosure but not to limit the scope of the present disclosure.

[0036] In the following, references to "some embodiments" describe a subset of all possible embodiments, but it is understood that "some embodiments" may be the same or different subsets of all possible embodiments and may be combined with each other without contradiction.

[0037] The road obstacle detection method can be applied to a computer device, and the functions realized by the method can be realized by a processor in the computer device calling a program code; of course, the program code may be stored in a computer storage medium, and it can be understood that the computer device includes at least a processor and a storage medium.

[0038] FIG. 1B is a flowchart illustrating an implementation of a road obstacle detection method according to an embodiment of the present disclosure, which will be described in conjunction with the steps shown in FIG. 1B as shown in FIG.

[0039] In step S101, a road image is acquired.

[0040] In some embodiments, the road image may be an image collected on any road, an image with complex screen content, or an image with simple screen content, such as a road image collected by an image collection device on a vehicle.

[0041] In some embodiments, the image collection device may be mounted on an on-board device of a vehicle or may be independent of the on-board device. The on-board device may be communicatively connected to a sensor, a positioning device, etc. of the vehicle, and the on-board device may acquire data collected by the sensor, geolocation information reported by the positioning device, etc. through the communicative connection. Exemplarily, the sensor of the vehicle may be at least one of a millimeter-wave radar, a laser radar, a camera, etc., and the positioning device may be a device for providing positioning services based on at least one of a Global Positioning System (GPS), a Beidou satellite navigation system, a Galileo satellite navigation system, etc.

[0042] In some embodiments, the in-vehicle device may be an Advanced Driving Assistant System (ADAS), which is installed in a vehicle. The ADAS may acquire real-time position information of the vehicle from a positioning device of the vehicle, and / or the ADAS may acquire image data, radar data, etc. representing vehicle surrounding environment information from a sensor of the vehicle. Here, optionally, the ADAS may transmit vehicle driving data including the real-time position information of the vehicle to a cloud platform. In this way, the cloud platform may receive the image data, radar data, etc. representing the real-time position information of the vehicle and / or the vehicle surrounding environment information.

[0043] The road image is acquired by an image collecting device (i.e., a sensor such as a camera) installed in the vehicle, and the image collecting device collects images of the surroundings of the vehicle in real time as the vehicle moves, thereby obtaining the road image. In some possible implementation forms, a camera mounted on the vehicle may collect images of the road on which the vehicle is traveling and the surrounding environment while the vehicle is traveling, thereby obtaining the road image.

[0044] In step S102, a drivable area and an ideal driving area within the road image are determined based on the road image.

[0045] In some embodiments, the vehicle-related road may be the road on which the vehicle is currently traveling. The drivable area and the ideal drivable area may be determined independently, or the ideal drivable area may be determined based on the drivable area. For example, when the drivable area and the ideal drivable area are determined independently, a road image is input to a trained neural network, and semantic segmentation is performed on the road image using different branching networks of the neural network to obtain the drivable area and the ideal drivable area on the road. The neural network may be a neural network for performing semantic segmentation. The drivable area in the road image is detected using the neural network, and the drivable area (free space) represents the drivable road area for the vehicle. In addition to the current vehicle, a road image typically includes other vehicles, pedestrians, trees, road edges, etc., and areas where other vehicles, pedestrians, trees, road edges, etc. are located are areas where the current vehicle cannot travel. Therefore, by performing semantic segmentation on the road image, areas where other vehicles, pedestrians, trees, road edges, etc. are located are removed from the road image to obtain the drivable area for the vehicle. The ideal driving area is the actual road area that can be driven on the road surface, assuming that there are no vehicles, pedestrians, or objects on the road surface. Thus, the coverage area of ​​the ideal driving area is larger than the drivable area, or the ideal driving area includes the drivable area on the road.

[0046] In step S103, an obstacle area on the road is determined based on the travelable area and the ideal travel area.

[0047] In some embodiments, an obstacle area of ​​a road is an area where any type of obstacle that affects the normal running of the vehicle is located, such as an area where other vehicles other than the vehicle on the road, pedestrians, trees, etc. are located.

[0048] In some possible implementations, after recognizing the drivable area and the ideal area of ​​the road for the road on which the vehicle is located in the road image, the overlapping and non-overlapping areas between the two areas are determined. The overlapping area between the drivable area and the ideal area is a road area that exists both in the drivable area and in the ideal area, and indicates that the overlapping area is the actual road area of ​​the road. The overlapping area is an area on the road where a vehicle can travel and further indicates that no obstacles exist in the overlapping area. The non-overlapping area between the drivable area and the ideal area is an area that exists in the ideal area but not in the drivable area, and indicates that the non-overlapping area is an area where an obstacle exists, i.e., an obstacle area on the road.

[0049] In the embodiment of the present disclosure, an acquired road image is detected, a vehicle-related drivable area and an ideal drivable area are recognized in the road image, and the drivable area and the ideal drivable area are compared to effectively determine an obstacle area where an obstacle of any shape is located on the road. In this way, by comparing the drivable area with the ideal drivable area, an obstacle area on the road can be quickly recognized and the stability of obstacle recognition of different shapes can be improved.

[0050] In some embodiments, not only can a drivable area be obtained by inputting a road image into the first network, but also a drivable area in the road image can be recognized by comparing screen information of the road image with screen information of a predetermined road image. That is, in the above step S101, the step of determining the drivable area in the road image based on the road image can be realized by the following steps S111 to S113 (not shown).

[0051] In step S111, an image area is determined in which the similarity between the texture in the road image and the information representing the road texture is greater than a predetermined similarity threshold.

[0052] In some embodiments, the road texture information may be a road surface image obtained by collecting roads that the vehicle can drive on, or may be an image area cropped from the road image where the image shows the road.

[0053] In some possible implementations, texture features of a road image are extracted, and among the extracted texture features, texture features that have a high similarity to information representing the road texture are found, and the image area corresponding to the texture feature is determined.

[0054] In step S112, the image area is determined as the drivable area.

[0055] In some embodiments, an image area in a road image that has a high similarity to screen information of a predetermined road image means that the screen in the image area has a high similarity to the screen information of the predetermined road image, and further means that the screen in the image area is a road surface, i.e., that the image area is a drivable area for a vehicle. In this way, by recognizing an image area in a road image that has a high similarity to the screen information of the predetermined road image as a drivable area, the recognition complexity of the drivable area is reduced and implementation is easier.

[0056] In some embodiments, the ideal driving area can be obtained not only by inputting a road image into a neural network (e.g., the first network), but also by recognizing road edges of a vehicle-related road and dividing the area surrounded by the road edges. That is, in the above step S101, the step of determining the ideal driving area based on the road image can be realized by the following steps S113 and S114 (not shown).

[0057] In step S113, the road image is detected to determine road edges in the road image.

[0058] In some embodiments, a third network is employed to recognize road edges of the roads and determine road edges in the road image. The third network may be any network for road edge recognition, such as a residual network or a VGG (Visual Geometry Group) network. A road image is input to the third network, feature extraction is performed on the road image, and the road edges are recognized based on the extracted image features to recognize the road edges of vehicle-related roads.

[0059] In step S114, the area surrounded by the road edges is determined as the ideal driving area.

[0060] In some embodiments, the recognized road edge can be understood as the outermost contour of the road, and therefore the area surrounded by the outermost contour is the actual road area without any type of vehicle, pedestrian, or object on the road, thereby making the area surrounded by the road edge the ideal driving area, thereby making the coverage of the determined ideal driving area more comprehensive.

[0061] In some embodiments, the ideal driving area may be determined based on the drivable area. That is, in step S101, the step of determining the ideal driving area based on the drivable area may be realized by the following process.

[0062] The ideal driving area is determined based on the driving-capable area.

[0063] Here, after recognizing the vehicle's drivable area in the road image using any of the above-described embodiments, the drivable area is expanded on the road on which the drivable area is located based on the drivable area to determine the ideal drivable area. By determining the ideal drivable area based on the drivable area in this way, the ideal drivable area can be made to include the drivable area, thereby making it possible to more accurately determine obstacle areas by comparing the ideal drivable area with the drivable area.

[0064] In some embodiments, the step of determining the ideal driving area based on the drivable area can be realized by the following steps.

[0065] In a first step, at least one contour of the drivable area is determined.

[0066] In some embodiments, the drivable area includes multiple cavities because it is the area of ​​the road surface obtained by removing other vehicles, objects, etc. from the entire road surface. In this way, the drivable area has multiple contours.

[0067] In the second step, a target contour having the largest area enclosed by the at least one contour is determined.

[0068] In some embodiments, the area of ​​an area enclosed by each of at least one contour is determined, and the contour with the largest area is selected as the target contour, which may be understood as the outermost contour of the drivable area of ​​the at least one contour.

[0069] In the third step, the area surrounded by the target contour is determined as the ideal running area.

[0070] In some embodiments, the target contour may be the outermost contour of the drivable area, so it is reasonable to determine the area surrounded by the target contour as the ideal drivable area. In this way, by recognizing multiple contours of the drivable area and determining the area surrounded by the outermost target contour as the ideal drivable area, the implementation process is simplified and the accuracy of the determined ideal drivable area is increased.

[0071] In some embodiments, before determining the area surrounded by the target contour as the ideal driving area, it may be first analyzed whether the area surrounded by the target contour includes two road edges of the road. Before determining the area surrounded by the target contour as the ideal driving area, the road obstacle detection method includes: The method further includes determining that two road edges of the road are included in an area enclosed by the target contour.

[0072] In this way, if the area enclosed by the target contour includes two road edges of the road, the area enclosed by the target contour is determined as the ideal driving area.

[0073] In some embodiments, if the enclosed area of ​​the target contour includes two road edges, it means that the enclosed area of ​​the target contour covers the entire road surface, making the resulting ideal driving area more reasonable.

[0074] In some embodiments, the above step S101 can also be realized in the following two ways.

[0075] Method 1: The ideal driving area can also be determined by detecting multiple roadways on the road and analyzing the traffic rules corresponding to the roadways. That is, in the above step S102, the step of determining the ideal driving area based on the road image can also be realized by the following steps:

[0076] In a first step, the road image is detected to obtain a plurality of roadways in the road image.

[0077] In some embodiments, a detector can be employed to detect lanes in a road image. Based on the detected lanes, multiple roadways can be obtained by treating the road area between two adjacent lanes as one roadway. Alternatively, the road image can be input to a trained lane detection network, which can directly output multiple roadways on the road.

[0078] In a second step, the ideal driving area is determined based on the plurality of roadways and traffic rules corresponding to the plurality of roadways.

[0079] In some embodiments, after detecting multiple roadways on a road, road surface signals (e.g., arrows, solid or dashed lines, etc.) on each roadway can be obtained and the meaning of the road surface signals can be determined according to traffic rules. For example, a solid white line is a boundary line that prohibits crossing roads in the same direction. In this way, by determining the area covered by each roadway and the traffic rules corresponding to the road surface signals on each roadway, the area covered by each roadway can be determined. By combining the areas covered by multiple roadways, the actual road area on which road vehicles can travel when there are no obstacles, i.e., the ideal driving area, can be determined. In this way, determining the area surrounded by roadways as the ideal driving area through roadway detection not only enriches the method for determining the ideal driving area, but also enables the ideal driving area to completely include the area covered by each roadway, further improving the accuracy of determining the ideal driving area.

[0080] Method 2: The ideal driving area and the drivable area are obtained by performing semantic segmentation on the actual road area and the drivable area of ​​the road in the road image through different network branches in the same neural network. That is, the above step S101 can be realized by the following process.

[0081] Semantic segmentation is performed on the road image to obtain the drivable area and the ideal driving area within the road image.

[0082] In some embodiments, a first network is first employed to perform semantic segmentation of the vehicle-drivable road regions within the road in the road image to obtain the drivable region, and a second network is then employed to perform semantic segmentation of the actual road region of the vehicle-related road in the road image to obtain the ideal drivable region. The first network may be any type of neural network trained to perform semantic segmentation of the vehicle-drivable road regions and the vehicle-undrivable road regions on the road. For example, the first network may be a branch of any neural network, such as a fully convolutional neural network, a mask region convolutional neural network (Mask R-CNN), or a deep neural network. A road image is input to the first network to perform feature extraction on the road image, and then semantic segmentation of the vehicle-drivable road regions and the vehicle-undrivable regions within the road image is performed based on the extracted image features, thereby quickly and accurately recognizing the drivable region in the road image.

[0083] The second network and the first network may be different branches of the same neural network, and the second network may be any type of neural network trained to perform semantic segmentation on actual road and non-road regions on a road. By inputting a road image into the second network, the second network performs semantic segmentation on the actual road and non-road regions of the road in the road image, and the road region obtained by segmentation is the ideal driving region. The actual road region is a road region on which a vehicle can actually drive if there are no objects of any type on the road.

[0084] The above process can be performed simultaneously through different network branches within a neural network. A road image is input to the neural network for feature extraction, and the extracted features are input to a first network and a second network, respectively. The first network branch of the neural network performs semantic segmentation on the drivable road area of ​​the vehicle in the road image to obtain the drivable area. At the same time, the second network branch of the neural network performs semantic segmentation on the actual road area of ​​the road in the road image to obtain the ideal drivable area. In this way, by dividing the drivable area and the ideal road area from the road image through different network branches within the same neural network, region segmentation can be achieved quickly and accurately.

[0085] In the embodiments of the present disclosure, the drivable area and ideal driving area are determined using a plurality of methods, and in any case, the drivable area and ideal driving area of ​​the vehicle can be recognized with high accuracy.

[0086] In some embodiments, the obstacle area of ​​the road is determined by determining the non-overlapping area between the drivable area and the ideal driving area. That is, the above step S103 can be realized by the steps shown in FIG. 2.

[0087] In step S201, an area within the ideal driving area that does not overlap with the driving-enabled area is determined.

[0088] In some embodiments, the ideal driving area and the drivable area are compared to determine an overlapping area between the two areas and a non-overlapping area between the two areas. The overlapping area is an actual road area and may be equal to the drivable area. The non-overlapping area between the two areas is an area that exists within the ideal driving area but does not exist within the drivable area, and means that the non-overlapping area is an area where the vehicle cannot currently travel.

[0089] In step S202, the non-overlapping area is determined as the obstacle area.

[0090] In some embodiments, an area where the vehicle cannot currently travel (i.e., a non-overlapping area) means that there is an obstacle in that area that affects the normal travel of the vehicle, and the obstacle may be another vehicle, a pedestrian, a tree, or another object. In this way, by determining the non-overlapping area as an obstacle area, it is possible to accurately detect an obstacle area that affects the travel of the vehicle regardless of the position or shape of the obstacle, thereby improving the stability of obstacle recognition.

[0091] In some embodiments, the route of the vehicle is re-planned according to the detected obstacle area, thereby more effectively controlling the vehicle travel, i.e., after step S103, the method further includes the steps shown in FIG.

[0092] In step S301, a travel route for the vehicle is determined based on the obstacle area.

[0093] In some embodiments, the driving path includes the vehicle's driving speed and driving direction, etc. By analyzing the position of the obstacle area on the road and the positional relationship between the obstacle area and the vehicle, the driving path of the vehicle traveling on the road is re-planned. For example, if the distance between the vehicle and the obstacle area is close, the vehicle is controlled to prepare for braking, or the vehicle is controlled to move away from the obstacle area.

[0094] In step S302, the driving of the vehicle is controlled based on the driving route.

[0095] In some embodiments, the automated driving system of the vehicle can determine a route for the vehicle and further controls the vehicle to follow the route.

[0096] After the obstacle area is recognized through the above steps S301 and S302, a subsequent driving route is generated in combination with the road information, so that the generated driving route becomes more accurate, and based on this, the vehicle can be controlled accurately according to the driving route.

[0097] In the following, an exemplary application of the embodiments of the present disclosure in a practical application scenario is described by taking as an example adopting a deep neural network on road markings to determine vehicle turning directions at road intersections.

[0098] Significant progress has been made in the field of autonomous driving, primarily through deep learning, including in the fields of image segmentation and target detection. However, autonomous driving is a holistic system, and the output of a sensing module serves subsequent modules. For example, the sensing result not only indicates whether an object is present ahead, but also provides related logic outputs for subsequent modules, and provides certain control and logic signals for autonomous driving.

[0099] An embodiment of the present disclosure provides a road obstacle detection method, which, from the perspective of semantic segmentation, determines the non-overlapping area between the ideal driving area and the drivable area, and performs obstacle detection, thereby effectively recognizing obstacles with irregular shapes and indeterminate positions, effectively evaluating the distance to the obstacle, and providing useful information to subsequent decision-making, planning, and control modules of an autonomous driving system.

[0100] An embodiment of the present disclosure provides a road obstacle detection method, which uses a deep learning method to obtain a drivable area and an ideal driving area for an autonomous vehicle, and uses the drivable area and the ideal driving area to obtain a general obstacle detection result on the road surface, thereby providing an effective signal for subsequent autonomous driving route planning.

[0101] After performing semantic segmentation on the collected road images, the drivable area and ideal driving area within the image are determined. As shown in FIG. 4, semantic segmentation is performed on road image 401 to obtain the drivable area and ideal driving area within the road in the image. Here, as shown in FIGS. 5 and 6, area 501 in FIG. 5 represents the drivable area, and area 601 in FIG. 6 represents the ideal driving area. Comparing area 501 and area 601, it can be seen that the drivable area excludes vehicles, pedestrians, and some irregular obstacles, while the ideal driving area recognizes the entire area in which a vehicle can drive under ideal conditions. This shows that the main difference between the ideal driving area and the drivable area is the recognition of obstacles.

[0102] In some embodiments, during autonomous driving, the vehicle performs sensing output according to information provided on the road, and performs information integration on the results output from the model. As shown in Figure 7, Figure 7 is a structural diagram of a road sensing signal output network according to an embodiment of the present disclosure, where: The image input module 701 is configured to input a road image; The backbone network 702 is configured to perform feature extraction on the input road image.

[0103] Here, the backbone network may be a residual network or a VGG network, a GoogleNet network, a ResNet network, etc.

[0104] The drivable area branching network 703 is configured to realize the segmentation task, and performs semantic segmentation based on the extracted image features to output the drivable area.

[0105] In some possible implementations, the image features extracted by the backbone network 702 are input to a drivable area branching network 703 that performs a semantic segmentation task, dividing the drivable area and outputting the drivable area.

[0106] In some possible implementations, because roads have similar textures, the drivable area branching network 703 finds objects on similar roads as drivable areas, and therefore the drivable area branching network 703 can generate a drivable area that excludes obstacles. In this way, the drivable area branching network 403 can robustly detect drivable areas in road areas.

[0107] The ideal driving region branching network 704 is configured to realize the segmentation task, and performs semantic segmentation based on the extracted image features to output the ideal driving region.

[0108] In some possible implementations, the image features extracted by the backbone network 702 are input to an ideal driving area branching network 704, which performs a semantic segmentation task, to perform segmentation into ideal driving areas and output the ideal driving areas. The ideal driving area branching network 704 recognizes which areas of the road area are actual road areas (which need to be recognized even if there are obstacles on the road surface) and which areas are not actual road areas.

[0109] In some possible implementations, the ideal running region branching network 704 determines the region within the outermost contour of the input image region as the ideal region.

[0110] In some possible implementations, the ideal driving area can be determined based on the driveable area, for example, the outermost contour of the driveable area is determined as the ideal driving area.

[0111] In some possible implementations, roadway detection is performed on a road image to determine an area surrounded by the roadway as an ideal driving area. In this way, the ideal driving area can be determined based on the detected roadway area and the roadway driving rules of the road.

[0112] In some embodiments, the drivable area branching network 703 and the ideal driving area branching network 704 determine the drivable area and the ideal driving area, respectively, and then compare the results of the two areas to determine the overlapping area as the actual road area, and the area where the two do not overlap (in most cases, the area of ​​the ideal driving area is larger than the area of ​​the drivable area), i.e., the area that exists in the ideal driving area but not in the drivable area, is recognized and output as an obstacle area. In this way, vehicles, pedestrians, non-automobiles, and obstacles with non-fixed shapes can be effectively recognized.

[0113] In the embodiments of the present disclosure, based on a deep learning model, the drivable area and the ideal driving area are effectively recognized, and the two learning tasks are concentrated in one network, and the two branches are integrated into one and the same backbone network, which not only accelerates the model processing speed but also effectively reduces the pressure on the operating speed.

[0114] Those skilled in the art can understand that in the above method of the specific embodiment, the described order of each step does not limit the implementation process in a strict execution order, and the specific execution order of each step should be determined by its function and possible internal logic.

[0115] Based on the same technical concept, the embodiments of the present disclosure further provide a movement intention determination device corresponding to the movement intention determination method, and the principles for solving the problems of the device in the embodiments of the present disclosure are similar to the above-mentioned movement intention determination method in the embodiments of the present disclosure, so the implementation of the device can refer to the above-mentioned method embodiments.

[0116] An embodiment of the present disclosure provides a road obstacle detection device. FIG. 8 is a schematic diagram showing the configuration of a road obstacle detection device according to an embodiment of the present disclosure. As shown in FIG. 8, the road obstacle detection device 800 includes: an image capture portion 801 configured to capture road images; a driving area determination unit 802 configured to determine a driveable area and an ideal driving area in the road image based on the road image; and an obstacle area determination unit 803 configured to determine an obstacle area on a road based on the travelable area and the ideal travel area.

[0117] In some embodiments, the travel area determination portion 802 includes: an image region recognition subpart configured to determine image regions in which a similarity between a texture in the road image and information representing the road texture is greater than a predetermined similarity threshold; and a drivable area determination subportion configured to determine the image area as the drivable area.

[0118] In some embodiments, the travel area determination portion 802 includes: a road edge recognition subpart configured to detect the road image and determine road edges within the road image; and a first ideal driving area determination sub-portion configured to determine an area surrounded by the road edges as the ideal driving area.

[0119] In some embodiments, the travel area determination portion 802 includes: The vehicle further includes a second ideal driving area determination sub-portion configured to determine the ideal driving area based on the driveable area.

[0120] In some embodiments, the second ideal driving area determination submodule: a contour determination portion configured to determine at least one contour of the drivable area; a target contour determining section configured to determine a target contour having the largest area of ​​an enclosed region among the at least one contour; and an ideal traveling area determining section configured to determine an area surrounded by the target contour as the ideal traveling area.

[0121] In some embodiments, the second ideal driving area determination sub-module further comprises: The method is configured to determine that an area enclosed by the target contour includes two road edges of the road.

[0122] In some embodiments, the travel area determination portion 802 includes: a roadway sub-detector configured to detect the road image to obtain a plurality of roadways in the road image; and a fourth ideal driving area determination sub-part configured to determine the ideal driving area based on the plurality of roadways and traffic rules corresponding to the plurality of roadways.

[0123] In some embodiments, the travel area determination portion 802 further comprises: The system is configured to perform semantic segmentation on the road image to obtain the drivable area and the ideal driving area within the road image.

[0124] In some embodiments, the obstacle region determination portion 803 comprises: a non-overlapping area determination sub-portion configured to determine an area within the ideal driving area that does not overlap with the drivable area; and an obstacle region determining subportion configured to determine the non-overlapping region as the obstacle region.

[0125] In some embodiments, the road obstacle detection device comprises: a travel route determination portion configured to determine a travel route of the vehicle based on the obstacle region; The vehicle control unit is further configured to control the driving of the vehicle based on the driving route.

[0126] It should be noted that the above description of the apparatus embodiment is similar to the description of the method embodiment above, and has similar beneficial effects as the method embodiment. The technical details not disclosed in the apparatus embodiment of the present disclosure can be understood by referring to the description of the method embodiment of the present disclosure.

[0127] In this and other embodiments of the present disclosure, a "module" may be a circuit, a processor, a program, software, or the like, and may of course be a unit or a non-module.

[0128] It should be noted that in the embodiments of the present disclosure, the above-described road obstacle detection method may be implemented as a software functional module and stored in a computer-readable storage medium when sold or used as an independent product. Based on this understanding, essential parts of the technical solutions of the embodiments of the present disclosure or parts that contribute to the prior art may be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for causing a computer device (which may be a terminal, a server, etc.) to execute all or part of the methods described in each embodiment of the present disclosure. The aforementioned storage medium includes various media capable of storing program code, such as a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk. Thus, the embodiments of the present disclosure are not limited to a specific combination of hardware and software.

[0129] Correspondingly, embodiments of the present disclosure further provide a computer program product having computer-executable instructions stored therein, which, after being executed, can realize the steps of the road obstacle detection method provided by the embodiments of the present disclosure. Correspondingly, embodiments of the present disclosure further provide a computer storage medium having computer-executable instructions stored therein, which, when executed by a processor, realize the steps of the road obstacle detection method provided by the above embodiments. Correspondingly, embodiments of the present disclosure provide a computer device. FIG. 9 is a schematic diagram showing the configuration of a computer device according to an embodiment of the present disclosure. As shown in FIG. 9, the computer device 900 includes one processor 901, at least one communication bus, a communication interface 902, at least one external communication interface, and a memory 903. Here, the communication interface 902 realizes communication connection between these components. Here, the communication interface 902 may include a display, and the external communication interface may include a standard wired interface and a wireless interface. Here, the processor 901 executes a road obstacle detection program in the memory to implement the steps of the road obstacle detection method provided by the above embodiment.

[0130] The above description of the road obstacle detection device, computer device, and storage medium embodiments is similar to the description of the above method embodiments, and has similar technical details and beneficial effects. Due to space limitations, reference can be made to the description of the above method embodiments, and the description will not be repeated here. Technical details not disclosed in the road obstacle detection device, computer device, and storage medium embodiments of the present disclosure can be understood by referring to the description of the method embodiments of the present disclosure. It should be understood that in this specification, "one embodiment" or "one embodiment" means that a particular feature, structure, or characteristic associated with an embodiment is included in at least one embodiment of the present disclosure. Therefore, "one embodiment" or "one embodiment" throughout the specification does not necessarily refer to the same embodiment. Furthermore, these particular features, structures, or characteristics may be incorporated in one or more embodiments in any appropriate manner. It should be understood that in each embodiment of the present disclosure, the magnitude of the sequence numbers of the above processes does not imply an order of execution. The execution order of each process should be determined by its function and internal logic and should not constitute any limitation on the implementation process of the embodiments of the present disclosure. The sequence numbers in the above embodiments of the present disclosure are for illustrative purposes only and do not represent the superiority or inferiority of the embodiments.

[0131] It should be explained that, as used herein, the terms "comprises," "includes," or any other variation thereof, are intended to cover a non-exclusive inclusion, whereby a process, method, article, or apparatus comprising a set of elements includes not only those elements but also other elements not expressly listed or inherent in such process, method, article, or apparatus. Unless otherwise limited, an element defined with the phrase "comprises" does not exclude the presence of other identical elements in the process, method, article, or apparatus that comprises the element.

[0132] It should be understood that in some embodiments provided in the present disclosure, the disclosed devices and methods can be realized in other ways. The device embodiments described above are merely illustrative. For example, the division of the units is merely a division of logical functions. In actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not implemented. Furthermore, the mutual coupling or direct coupling or communication connection between each component shown or described may be an indirect coupling or communication connection via some interfaces, devices, or units, which may be electrical, mechanical, or other forms.

[0133] The units described as separate components may or may not be physically separated. Components represented as units may or may not be physical units, located in one location, or distributed across multiple network units. Some or all of these units may be selected according to actual needs to achieve the objectives of the technical solutions of the present embodiments. Furthermore, the functional units in the embodiments of the present disclosure may all be integrated into a single second processing unit, or each unit may be used individually as a single unit, or two or more units may be integrated into a single unit. The integrated unit may be embodied in the form of hardware or a combination of hardware and software functional units. Those skilled in the art will understand that all or some of the steps for implementing the above embodiments can be performed by hardware associated with program instructions, and the above-described program may be stored in a computer-readable storage medium, which, when executed, performs the steps including the above-described method embodiments. The above-described storage medium may include various media capable of storing program code, such as removable storage, read-only memory (ROM), magnetic disk, or optical disk.

[0134] Alternatively, the above-described integrated units of the present disclosure may be realized in the form of software functional modules and stored in a computer-readable storage medium when sold or used as an independent product. Based on this understanding, the essential parts of the technical solutions of the embodiments of the present disclosure or the parts that contribute to the prior art may be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, a network device, etc.) to execute all or part of the methods described in each embodiment of the present disclosure. The aforementioned storage medium includes various media capable of storing program code, such as removable storage, ROM, magnetic disk, or optical disk. The above content is merely an embodiment of the present disclosure, and the scope of protection of the present disclosure is not limited thereto. Any modifications or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present disclosure should be included in the scope of protection of the present disclosure. Therefore, the scope of protection of the present disclosure shall be governed by the scope of protection of the claims. [Industrial Applicability]

[0135] An embodiment of the present disclosure provides a method, apparatus, device, and storage medium for detecting road obstacles, the method including the steps of acquiring a road image, determining a drivable area and an ideal driving area in the road image based on the road image, and determining an obstacle area on the road based on the drivable area and the ideal driving area.

Claims

1. A method for detecting road obstacles, implemented by an electronic device, comprising: acquiring a road image; determining a drivable area and an ideal driving area within the road image based on the road image; determining an obstacle area on a road based on the travelable area and the ideal travel area; The step of determining the ideal driving area based on the road image includes: determining the ideal driving area based on the drivable area; The step of determining the ideal traveling area based on the travelable area includes: determining at least one contour of the drivable area; determining a target contour having the largest area of ​​an enclosed region among the at least one contour; and determining an area surrounded by the target contour as the ideal traveling area. How to detect road obstacles.

2. The step of determining a drivable area in the road image based on the road image includes: determining an image area in which a similarity between a texture in the road image and information representing the road texture is greater than a predetermined similarity threshold; determining the image area as the drivable area; The method for detecting a road obstacle according to claim 1 .

3. Before determining the area surrounded by the target contour as the ideal driving area, the road obstacle detection method includes: further comprising determining that an area enclosed by the target contour includes two road edges of the road; The method for detecting a road obstacle according to claim 1 .

4. The step of determining an obstacle area on a road based on the travelable area and the ideal travel area includes: determining an area within the ideal driving area that does not overlap with the drivable area; determining the non-overlapping region as the obstacle region; The method for detecting a road obstacle according to claim 1 .

5. After determining the obstacle area on the road, the road obstacle detection method includes: determining a travel route for the vehicle based on the obstacle area; and controlling the travel of the vehicle based on the travel route. The method for detecting a road obstacle according to claim 1 .

6. A road obstacle detection device, an image capture portion configured to capture road images; a driving area determination unit configured to determine a driveable area and an ideal driving area within the road image based on the road image; an obstacle area determination unit configured to determine an obstacle area on a road based on the travelable area and the ideal travel area, A driving area determination section that determines the ideal driving area based on the road image, a second ideal driving area determination sub-section that determines the ideal driving area based on the travelable area; the second ideal driving area determination sub-section determining the ideal driving area based on the travelable area, a contour determining portion for determining at least one contour of the drivable area; a target contour determining section that determines a target contour having the largest area of ​​an enclosed region among the at least one contour; and an ideal traveling area determination unit that determines an area surrounded by the target contour as the ideal traveling area. Road obstacle detection device.

7. A computer storage medium having stored thereon computer-executable instructions for causing a computer to carry out the steps of the method for detecting road obstacles according to any one of claims 1 to 5.

8. A computer device comprising: a memory in which computer-executable instructions are stored; and a processor that, when executing the computer-executable instructions on the memory, implements the steps of the method for detecting road obstacles according to any one of claims 1 to 5.

9. A computer program for causing an electronic device to execute the road obstacle detection method according to any one of claims 1 to 5.

Images