Image processing method, device, equipment, and storage medium

Abstract

The image processing method, device, equipment, and storage medium acquire a road image collected by an image collecting device mounted on a vehicle (S101), detect a plurality of road boundaries in the road image based on the road image (S102), and determine a target road boundary that is dangerous to the vehicle from among the plurality of road boundaries (S103). In this way, the traveling of the vehicle can be more accurately controlled based on the target road boundary.

Description

[Technical field]

[0001] (CROSS-REFERENCE TO RELATED APPLICATIONS) This application claims priority to a Chinese patent application filed with the China Patent Office on March 24, 2022, bearing application number 202210303731.8 and entitled "Image processing method, apparatus, device and storage medium," the entire contents of which are incorporated herein by reference.

[0002] The present application relates to the technical field of autonomous driving, and in particular, to an image processing method, device, equipment, and storage medium, but is not limited thereto. [Background technology]

[0003] In recent years, the field of autonomous driving, which mainly uses deep learning, has made great progress, including the fields of image segmentation and target detection. As an overall system, autonomous driving uses the output of a sensing module to provide services to subsequent modules. In related technologies, the correlation between the sensing module and the subsequent control signal is often lost, which affects the control accuracy and reliability of the system. Summary of the Invention

[0004] The embodiments of the present disclosure provide a technical solution for image processing.

[0005] The technical solutions of the embodiments of the present disclosure are realized as follows.

[0006] An embodiment of the present disclosure provides an image processing method, the method including the steps of acquiring a road image collected by an image collection device mounted on a vehicle, detecting a plurality of road boundaries in the road image based on the road image, and determining a target road boundary among the plurality of road boundaries that is dangerous to the vehicle.

[0007] In some embodiments, detecting a plurality of road boundaries in the road image based on the road image comprises detecting the road image to determine a plurality of road boundaries associated with the vehicle, in this manner allowing for quick and accurate recognition of a plurality of road boundaries in the road image.

[0008] In some embodiments, the step of detecting a plurality of road boundaries in the road image based on the road image includes the steps of: detecting the road image to obtain a plurality of roadways in the road image; and connecting the ends of each roadway in the plurality of roadways to obtain the plurality of road boundaries. In this manner, by connecting the end edges of each roadway, the plurality of road boundaries in the road image can be more simply recognized.

[0009] In some embodiments, the step of detecting a plurality of road boundaries in the road image based on the road image includes the steps of: performing semantic segmentation on the road image to obtain a drivable area in the road image; and determining the plurality of road boundaries based on a contour of the drivable area. In this manner, by dividing the drivable area of ​​the road in the road image, the plurality of road boundaries in the road image can be accurately recognized.

[0010] In some embodiments, the step of determining a target road boundary that is dangerous to the vehicle among the plurality of road boundaries includes at least one of the following steps: determining, as the target road boundary, a road boundary that is adjacent to a roadway on which the vehicle is located among the plurality of road boundaries, the step of determining, as the target road boundary, a road boundary that is less than a first predetermined distance from the vehicle among the plurality of road boundaries, the step of determining, as the target road boundary, a road boundary that is less than a predetermined road space between the vehicle among the plurality of road boundaries, and the step of determining, as the target road boundary, a target road boundary that is dangerous to the vehicle among the plurality of road boundaries based on road information determined by the road image, the road information including at least one of road signals, lanes, stop line areas, driving direction signs, and obstacle information in the road image. In this way, by determining the target road boundary in a plurality of ways and comprehensively considering a plurality of pieces of information on the road surface, it is possible to accurately detect a target road boundary that is dangerous to the vehicle.

[0011] In some embodiments, the step of determining a target road boundary that is dangerous for the vehicle among the plurality of road boundaries based on road information determined by the road image includes the steps of determining an actual road area and an unknown area that is not recognizable to the vehicle based on the road information, determining a road boundary that is not visible to the vehicle based on the actual road area and the unknown area, and determining the road boundary that is not visible to the vehicle as the target road boundary. In this way, by comparing the actual road area and the unknown area of ​​the vehicle-related road, it is possible to accurately recognize a target road boundary that is dangerous for the vehicle.

[0012] In some embodiments, the step of determining road boundaries invisible to the vehicle based on the actual road area and the unknown area includes the steps of: converting the collection viewpoints of the actual road area and the unknown area into a bird's-eye view to obtain a transformed actual road area and a transformed unknown area, respectively; determining an overlapping area between the transformed actual road area and the transformed unknown area; and determining the road boundaries in the overlapping area as road boundaries invisible to the vehicle. In this way, by analyzing the overlapping area between the transformed actual road area and the transformed unknown area in the bird's-eye view, the road boundaries invisible to the vehicle can be effectively recognized with a small network resource, which facilitates the planning of the driving route of the following vehicle.

[0013] In some embodiments, the step of determining the overlapping area between the transformed actual road area and the transformed unknown area includes the steps of fitting lanes, stop line areas, and driving direction signs in the transformed actual road area to obtain first fitting information, fitting lanes, stop line areas, and driving direction signs in the transformed unknown area to obtain second fitting information, and determining the overlapping area between the transformed actual road area and the transformed unknown area based on the first fitting information and the second fitting information. In this way, by comprehensively considering multiple pieces of information on the road, it is possible to more accurately determine the target road boundary in the overlapping area.

[0014] In some embodiments, after determining the target road boundary, the image processing method further includes: determining a driving route for the vehicle based on the target road boundary and / or the road information; and controlling the driving of the vehicle based on the driving route. In this way, after recognizing the target road boundary, a more accurate driving route can be generated in combination with abundant road information, and accurate control of the vehicle can be realized.

[0015] In some embodiments, the step of determining a driving route of the vehicle based on the road information includes the steps of: determining a turning direction and a turning position of the vehicle based on a road surface signal and a driving direction sign in the road information; and determining a driving route of the vehicle based on the turning direction and the turning position. In this way, by following the road surface signal in the road information, the turning direction and the turning position of the vehicle at a future time can be accurately predicted, and the turning of the vehicle can be accurately controlled.

[0016] In some embodiments, the step of controlling the travel of the vehicle based on the travel route includes the steps of updating the travel route based on obstacle information in the road information to obtain an updated route, and controlling the travel of the vehicle based on the updated route. In this manner, by integrating position information of obstacles in the road information to update the travel route, more information can be provided to the autonomous vehicle when making decisions.

[0017] In some embodiments, determining a driving route for the vehicle based on the target road boundary includes: updating map data for a location of the vehicle based on the target road boundary to obtain an updated map; and determining a driving route for the vehicle based on the updated map. In this way, a driving route for controlling the driving of the vehicle is generated according to the updated map, and the safety of the driving route is improved.

[0018] In some embodiments, after determining the target road boundary, the image processing method further comprises controlling the vehicle based on a relationship between the target road boundary and a driving state of the vehicle. In this way, after recognizing the target road boundary, the relationship between the target road boundary and the driving state can be analyzed to effectively control the vehicle to drive safely.

[0019] In some embodiments, the relationship between the target road boundary and the driving state of the vehicle includes at least one of: a distance between the overlapping area in which the target road boundary is located and a road intersection ahead of the vehicle is less than a second predetermined distance; a distance between the overlapping area and the vehicle's location is less than a third predetermined distance; an angle between the vehicle's driving direction and the target road boundary is less than a predetermined angle; and the target road boundary is adjacent to the roadway on which the vehicle is located.

[0020] In some embodiments, controlling the vehicle includes controlling the vehicle to enter a braking state from a driving state or to move away from the target road boundary. In this way, when the target road boundary affects the driving of the vehicle, the driving safety of the vehicle can be further improved by controlling the vehicle to enter a braking state or by controlling the vehicle to move away from the target road boundary.

[0021] In some embodiments, after determining the target road boundary, the image processing method further comprises the steps of: setting a region of interest based on the target road boundary and acquiring an image corresponding to the region of interest based on a first resolution, the road image being acquired according to a second resolution, the second resolution being smaller than the first resolution, and / or acquiring an image corresponding to the region of interest based on a first frame rate, the road image being acquired based on a second frame rate, the second frame rate being smaller than the first frame rate, thus facilitating object recognition for images corresponding to a subsequent region of interest.

[0022] In some embodiments, after determining the target road boundary, the image processing method further includes the steps of: collecting road environment information around the target road boundary, generating notification information based on the road environment information, and transmitting the notification information to a rear vehicle of the vehicle, the rear vehicle being located on the same roadway as the vehicle and traveling in the same direction. In this way, by timely notifying the rear vehicle that a target road boundary exists ahead, the rear vehicle can timely adjust its traveling route.

[0023] An embodiment of the present disclosure provides an image processing device, the device comprising: an image acquisition portion configured to acquire a road image collected by an image collection device mounted on a vehicle; a road boundary detection portion configured to detect a plurality of road boundaries in the road image based on the road image; and a target road boundary determination portion configured to determine a target road boundary among the plurality of road boundaries that is dangerous to the vehicle.

[0024] Correspondingly, an embodiment of the present disclosure provides a computer storage medium having computer-executable instructions stored thereon, which, after being executed, can realize the steps of the above image processing method.

[0025] An embodiment of the present disclosure provides a computing device comprising: a memory having computer-executable instructions stored therein; and a processor capable of implementing the steps of the above-mentioned image processing method when executing the computer-executable instructions on the memory.

[0026] An embodiment of the present disclosure further provides a computer program product including a computer program or instructions which, when executed on an electronic device, causes the electronic device to perform steps in any possible embodiment of the first aspect above.

[0027] Embodiments of the present disclosure provide an image processing method, device, equipment, and storage medium that detect an acquired road image, recognize multiple road boundaries in the road image, and select a target road boundary that is dangerous to the vehicle from the multiple road boundaries, thereby more accurately controlling the vehicle's driving based on the target road boundary.

[0028] In order to make the above objectives, features and advantages of the embodiments of the present disclosure more comprehensible, preferred embodiments accompanied with drawings are described in detail below. [Brief description of the drawings]

[0029] [Figure 1A] FIG. 1 is a schematic diagram of a system architecture to which an image processing method according to an embodiment of the present disclosure can be applied. [Figure 1B] 1 is an implementation flowchart of an image processing method according to an embodiment of the present disclosure. [Diagram 2] 1 is a flowchart illustrating another implementation of an image processing method according to an embodiment of the present disclosure. [Diagram 3] 13 is a flowchart illustrating yet another implementation of an image processing method according to an embodiment of the present disclosure. [Figure 4] FIG. 2 is a network structure diagram of an image processing method according to an embodiment of the present disclosure. [Figure 5A] FIG. 1 is a schematic diagram of an application scene of an image processing method according to an embodiment of the present disclosure. [Figure 5B] FIG. 2 is a schematic diagram of another application scene of the image processing method according to an embodiment of the present disclosure. [Figure 6A] FIG. 1 is a schematic diagram of an application scene of an image processing method according to an embodiment of the present disclosure. [Figure 6B] FIG. 2 is a schematic diagram of another application scene of the image processing method according to an embodiment of the present disclosure. [Figure 7] FIG. 13 is a schematic diagram of yet another application scene of the image processing method according to an embodiment of the present disclosure. [Figure 8] FIG. 13 is a schematic diagram of yet another application scene of the image processing method according to an embodiment of the present disclosure. [Figure 9]FIG. 2 is a schematic diagram of another application scene of the image processing method according to an embodiment of the present disclosure. [Figure 10] FIG. 1 is a schematic diagram illustrating a configuration of an image processing device according to an embodiment of the present disclosure. [Figure 11] FIG. 2 is a schematic diagram illustrating a configuration of a computing device according to an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0030] In order to more clearly explain the technical solutions of the embodiments of the present disclosure, the drawings necessary for the description of the embodiments are briefly described above. The drawings herein are incorporated in and constitute a part of this specification, and these drawings show embodiments consistent with the present disclosure, and are used together with the specification to explain the technical solutions of the present disclosure. It should be understood that the above drawings only show some of the embodiments of the present disclosure, and should not be considered as limiting the scope of protection, and those skilled in the art can also obtain other related drawings based on these drawings without creative efforts.

[0031] In order to make the objectives, technical solutions and advantages of the embodiments of the present disclosure clearer, the specific technical solutions of the present invention will be described in more detail below with reference to the drawings of the embodiments of the present disclosure. The following embodiments are for illustrating the present disclosure, and are not intended to limit the scope of the present disclosure.

[0032] In the following, references to "some embodiments" describe a subset of all possible embodiments, but it is understood that "some embodiments" may be the same or different subsets of all possible embodiments, and may be combined with each other without inconsistency.

[0033] In the following, terms such as "first / second / third" are not intended to limit a particular order but to distinguish between similar objects. It is understood that "first / second / third" can be used to convert a particular order or order when appropriate, so that the embodiments of the present disclosure described in some embodiments can be implemented in an order other than that shown or described in some embodiments.

[0034] Unless otherwise specified, all technical and scientific terms used in this disclosure have the same meaning as commonly understood by those skilled in the art of this disclosure. The terms used in this disclosure are only used to describe the embodiments of this disclosure and are not intended to limit the disclosure.

[0035] Before describing the embodiments of the present disclosure in detail, nouns and terms related to the embodiments of the present disclosure will be explained. The nouns and terms related to the embodiments of the present disclosure will be explained as follows.

[0036] 1) Convolutional Neural Networks (CNN): A type of feedforward neural network (FNN) with a deep structure that includes convolutional calculations. It has a representation learning function and can classify input information into translationally invariant classes according to its hierarchical structure.

[0037] 2) Ego vehicle: A vehicle equipped with sensors to sense the surrounding environment. The vehicle coordinate system is fixed to the ego vehicle, where the x-axis is the direction the vehicle is moving, the y-axis points to the left of the direction the vehicle is moving, and the z-axis is perpendicular to the ground and points upward, conforming to a right-handed coordinate system. The origin of the coordinate system is located on the ground below the midpoint of the rear axle.

[0038] In the following, an application example of the image processing method according to the embodiment of the present disclosure to an electronic device will be described. The electronic device according to the embodiment of the present disclosure may be an in-vehicle device, a cloud platform, or other computer device. Exemplarily, the in-vehicle device may be a thin client, a fat client, a microprocessor-based system, a small computer system, etc. installed in a vehicle, and the cloud platform may be a distributed cloud computing technology environment including a small computer system or a large computer system. Next, an application example in which the electronic device is implemented as a terminal or a server will be described.

[0039] 1A is a schematic diagram of a system architecture for applying an image processing method according to an embodiment of the present disclosure, and as shown in FIG. 1A, the system architecture includes an image capture device 11, a network 12, and an in-vehicle control terminal 13. To realize an exemplary application, the image capture device 11 and the in-vehicle control terminal 13 establish a communication connection via the network 12, and firstly, the image capture device 11 reports the captured road image to the in-vehicle control terminal 13 via the network 12, and the in-vehicle control terminal 13 performs road boundary detection on the road image, thereby detecting target road boundaries that are dangerous to the vehicle.

[0040] As an example, the image capture device 11 may include a visual processing device having visual information processing capabilities. The network 12 may employ a wired or wireless connection method. Here, when the image capture device 11 is a visual processing device, the vehicle control terminal 13 may be communicatively connected to the visual processing device by a wired connection method such as data communication via a bus.

[0041] Or, in some scenes, the image capture device 11 may be a visual processing device with a video collection module, or a host computer with a camera, in which case the image processing method of the embodiment of the present disclosure may be executed by the image capture device 11, and the above system architecture may not include the network 12 and the in-vehicle control terminal 13.

[0042] In order to make the objectives, technical solutions and advantages of the embodiments of the present disclosure clearer, the specific technical solutions of the present invention will be described in more detail below with reference to the drawings of the embodiments of the present disclosure. The following embodiments are intended to illustrate the present disclosure, but are not intended to limit the scope of the present disclosure.

[0043] In the following, references to "some embodiments" describe a subset of all possible embodiments, but it is understood that "some embodiments" may be the same or different subsets of all possible embodiments, and may be combined with each other without inconsistency.

[0044] The image processing method may be applied to a computer device, and the functions realized by the method may be realized by a processor in the computer device calling a program code, and of course, the program code may be stored in a computer storage medium, and it is understood that the computer device includes at least a processor and a storage medium.

[0045] FIG. 1B is a flowchart showing an implementation of an image processing method according to an embodiment of the present disclosure, which will be described in conjunction with the steps shown in FIG. 1B as shown in FIG.

[0046] In step S101, road images are acquired by an image collecting device mounted on a vehicle.

[0047] In some embodiments, the road image may be an image collected on any road, may be an image with complex screen content, or may be an image with simple screen content, such as a road image collected by an on-board device on a vehicle.

[0048] In some embodiments, the image collection device may be mounted on an on-board device of a vehicle or may be independent of the on-board device. The on-board device may be communicatively connected to a sensor, a positioning device, etc. of the vehicle, and the on-board device may acquire data collected by the sensor of the vehicle, geolocation information reported by the positioning device, etc. through the communicative connection. Exemplarily, the sensor of the vehicle may be at least one of a millimeter wave radar, a laser radar, a camera, etc., and the positioning device may be a device for providing a positioning service based on at least one of a Global Positioning System (GPS), a Beidou satellite navigation system, a Galileo satellite navigation system, etc.

[0049] In some embodiments, the in-vehicle device may be an Advanced Driving Assistant System (ADAS), where the ADAS is installed in a vehicle, and the ADAS can acquire real-time position information of the vehicle from a positioning device of the vehicle, and / or the ADAS can acquire image data, radar data, etc. representing vehicle surrounding environment information from a sensor of the vehicle. Here, optionally, the ADAS can transmit vehicle driving data including the real-time position information of the vehicle to a cloud platform. In this manner, the cloud platform can receive the image data, radar data, etc. representing the real-time position information of the vehicle and / or the vehicle surrounding environment information.

[0050] Road images are acquired by an image acquisition device (i.e., a sensor such as a camera) installed in a vehicle, and the image acquisition device acquires images of the surroundings of the vehicle in real time as the vehicle moves to obtain the road images. In some possible implementation forms, a camera mounted on the vehicle may acquire images of the road on which the vehicle is traveling and the surrounding environment while the vehicle is traveling to obtain the road images. In this way, the road images can be detected to recognize the multiple road boundaries.

[0051] In step S102, a plurality of road boundaries in the road image are detected based on the road image.

[0052] In some embodiments, a detection network is employed to detect road boundaries in the road image, and the vehicle in the road image may be any vehicle traveling on a road.

[0053] In some possible implementations, edge detection can be performed on a road image to detect edges of roads in the road image and obtain multiple road boundaries, for example, multiple roadway lanes can be detected in the road image and the multiple road boundaries can be obtained by connecting the end edges of the lanes, or the road image can be input to a trained edge detection network to output multiple road boundaries in the road image, etc.

[0054] In step S103, a target road boundary that is dangerous for the vehicle is determined from among the plurality of road boundaries.

[0055] In some embodiments, the target road boundary may be a road boundary that is invisible to the vehicle, a road boundary that is recognizable to the vehicle but is close to the vehicle, or a road boundary that is dangerous to the vehicle as determined by analyzing road information in a road image, such as a road boundary that is occluded by an obstacle, or a road boundary that is too far away from the vehicle, or a road boundary that is in the blind spot of the vehicle, or a road boundary that is too close to the vehicle and prevents the vehicle from traveling normally.

[0056] In some possible implementations, detecting the positional relationship between an obstacle and a road boundary can determine whether the obstacle is blocking the road boundary, thereby determining whether the road boundary is invisible, i.e., whether it is a target road boundary; detecting the distance between the vehicle and the road boundary can determine whether the road boundary is too far away from the vehicle, thereby determining whether the road boundary is a target road boundary; detecting the positional relationship between the vehicle and the road boundary can determine whether the road boundary is in the blind spot of the vehicle, thereby determining whether the road boundary is a target road boundary. In this way, by recognizing a target road boundary that is dangerous to the vehicle among the detected road boundaries, the subsequent driving route can be planned more accurately.

[0057] In an embodiment of the present disclosure, an acquired road image is detected, multiple road boundaries within the road image are recognized, and target road boundaries that are dangerous to the vehicle are selected from the multiple road boundaries, thereby enabling more accurate control of vehicle driving.

[0058] In some embodiments, the acquired road image can be detected to recognize a plurality of road boundaries associated with the vehicle in the road image, that is, the above step S102 can be realized in a number of ways:

[0059] Method 1: Detect the road image and determine a number of road boundaries associated with the vehicle.

[0060] In the first method, a first network is used to detect the road image to determine a plurality of road boundaries associated with the vehicle. The plurality of road boundaries associated with the vehicle may be road boundaries of each roadway of the road where the vehicle is located or road boundaries of a plurality of roadways of the road where the vehicle is located. In a specific implementation, since the vehicle can reach any roadway on the road where the vehicle is located by lane changing or U-turning, the road boundaries of any roadway on the road where the vehicle is located can be considered as the road boundaries associated with the vehicle. For example, if the road where the vehicle is located includes four roadways, the plurality of road boundaries associated with the vehicle include the road boundaries of each of the four roadways. The first network may be a Deep Neural Network (DNN), for example, any network capable of image detection. In some possible implementations, the first network may be a residual network, a Visual Geometry Group (VGG) network, etc. The first network is a network trained to detect road boundaries, and is capable of inputting a road image to the first network, extracting features from the road image, and recognizing multiple road boundaries associated with the vehicle based on the extracted image features. In this way, multiple road boundaries in a road image can be quickly and accurately recognized.

[0061] In addition, in a specific implementation process, the road boundary associated with the vehicle can be determined by determining an overlapping portion between the drivable area and the road boundary in the road image, that is, the overlapping portion between the detected drivable area and the detected road boundary is determined as the road boundary associated with the vehicle. Here, the drivable area can be detected by adopting a DNN, but is not limited thereto.

[0062] Method 2: Determine multiple road boundaries by detecting roadways in the road image. This step can be realized by the following steps:

[0063] In a first step, the road image is detected to obtain a plurality of roadways in the road image.

[0064] In some embodiments, a roadway is detected in the road image to obtain a plurality of roadways. A second network is used to detect a roadway in the road image to obtain a plurality of roadways. The second network may be the same as or different from the first network. A second network is used to detect a plurality of roadways in the road image, i.e., a plurality of lanes are detected in the road image. The road image is processed by the second network to obtain the lanes in the road image, i.e., the plurality of roadways. In other embodiments, other image detection methods may be adopted to detect a plurality of roadways in the road image. In some possible implementations, a grayscale process is first performed on the road image, and roadway edges in the road image after the grayscale process are detected, for example, an edge detection operator is adopted to perform edge detection, and the processed image is further subjected to a binarization process to obtain the lanes in the road image.

[0065] In a second step, the ends of each roadway in the plurality of roadways are connected to obtain the plurality of road boundaries.

[0066] In some embodiments, the multiple road boundaries associated with the vehicle can be obtained by connecting the end edges of the lanes of each roadway. For example, the road boundaries perpendicular to the road on which the vehicle is located can be obtained by connecting the end edges of the lanes of the vehicle on both the left and right sides of the vehicle. In this way, the multiple road boundaries in the road image can be more simply recognized by connecting the end edges of each roadway.

[0067] Method 3: Determine the drivable area of ​​the road where the vehicle is located by performing semantic segmentation on the road image. This step can be realized by the following steps.

[0068] In a first step, semantic segmentation is performed on the road image to obtain the drivable area within the road image.

[0069] In some embodiments, a third network is used to perform semantic segmentation on the road image to obtain a drivable area of ​​the road in the road image. The third network may be a neural network for performing semantic segmentation, such as a fully convolutional neural network or a mask region convolutional neural network (Mask R-CNN). The third network detects a drivable area in the road image, and the drivable area (free space) may also be called a passable area and represents an area in which the vehicle can be driven. In addition to the current vehicle, the road image usually includes other vehicles, pedestrians, trees, road edges, etc., and the area where the other vehicles, pedestrians, trees, road edges, etc. are located is an area in which the current vehicle cannot be driven. Therefore, the third network performs semantic segmentation on the road image to obtain a drivable area of ​​the vehicle by removing areas in the road image, such as areas where other vehicles, pedestrians, trees, road edges, etc. are located.

[0070] In a second step, the plurality of road boundaries are determined based on the contour of the drivable area.

[0071] In some embodiments, the contour of a drivable area is recognized to obtain road boundaries of the road on which the drivable area is located. For example, the contour of the drivable area is set as the road boundary of the road on which the drivable area is located. In this manner, by dividing the drivable area of ​​a road in a road image, multiple road boundaries in the road image can be accurately recognized.

[0072] In some embodiments, by recognizing road information of the vehicle-related road or analyzing the relationship between the road boundaries and the vehicle, it is possible to accurately select a target road boundary that is dangerous to the vehicle among a plurality of road boundaries. That is, the above step S103 can be realized in the following manners:

[0073] Method 1: Among the plurality of road boundaries, a road boundary adjacent to the roadway on which the vehicle is located is determined as the target road boundary.

[0074] In method 1, the road boundary adjacent to the roadway on which the vehicle is located may be a road boundary that borders the roadway on which the vehicle is located, and since the road boundary that borders the roadway on which the vehicle is located is in the blind spot of the vehicle's field of vision, the road boundary is not visible to the vehicle, i.e., the road boundary is a target road boundary.

[0075] Method 2: Among the plurality of road boundaries, a road boundary whose distance from the vehicle is less than a first predetermined distance is determined as the target road boundary.

[0076] In the second method, the first predetermined distance can be set by measuring the blind spot range of the vehicle, for example, the first predetermined distance can be set to be equal to or less than the maximum diameter of the blind spot range. The distance between the road boundary and the vehicle is the distance between each point on the road boundary and the vehicle, and if the distance from the point to the vehicle is less than the first predetermined distance, it means that the vehicle cannot see the point. In this way, by analyzing whether the distance between a plurality of points and the vehicle is less than the first predetermined distance, it can be determined whether the road boundary consisting of the plurality of points is the target road boundary.

[0077] In some possible implementations, for any road boundary, points on the road boundary are sampled according to a certain length interval, and a distance between the sampling point and the vehicle is less than a first predetermined distance, thereby determining whether the road boundary is a target road boundary. For example, the first sampling point whose distance to the vehicle is less than the first predetermined distance is taken as a start point, and the last sampling point whose distance to the vehicle is less than the first predetermined distance is taken as an end point. In this way, the road boundary between the start point and the end point is the target road boundary.

[0078] Method 3: Among the plurality of road boundaries, a road boundary whose road space between the vehicle and the road boundary is less than a predetermined space is determined as the target road boundary.

[0079] In the third method, the road space between the road boundary and the vehicle may be the width of the road area from the vehicle to the road boundary. The predetermined space may be determined based on the width of the roadway and the width of the vehicle that can travel on the roadway, for example, the predetermined space is set to be larger than the width of the vehicle that can travel on the roadway and smaller than the width of the roadway. If the width between the road boundary and the vehicle is smaller than the predetermined space, it means that an oncoming vehicle cannot travel between the road boundary and the vehicle, i.e., the space between the road boundary and the vehicle is small, and furthermore, it means that the road boundary may be dangerous to the normal travel of the vehicle, so such a road boundary is set as the target road boundary. If the width between the road boundary and the vehicle is equal to or larger than the predetermined space, it means that an oncoming vehicle can travel between the road boundary and the vehicle, i.e., there is a sufficient space between the road boundary and the vehicle, and furthermore, it means that the road boundary is not dangerous to the normal travel of the vehicle, so such a road boundary is not set as the target road boundary.

[0080] Method 4: Among the multiple road boundaries, a target road boundary that is dangerous to the vehicle is determined based on road information determined from the road image.

[0081] In the fourth method, image detection is performed on a road image to recognize road information of a road where a vehicle is located in the road image, and a target road boundary dangerous to the vehicle in a plurality of road boundaries can be recognized based on the road information. The road information of a vehicle-related road is used to represent a plurality of detectable information on the road, for example, the road information includes at least one of road surface signals, lanes, stop line areas, direction signs, and obstacle information in the road image. The direction signs may be located at a direction change edge of the road. By comprehensively considering a plurality of road surface information, a target road boundary dangerous to the vehicle can be accurately detected.

[0082] In some possible implementations, the road information can be obtained by the following steps.

[0083] In a first step, a road surface signal of a road associated with the vehicle in the road image is determined.

[0084] In some embodiments, a detector in a deep neural network is adopted to perform image feature extraction on a road image, and based on the extracted image features, detection of road surface signals in the road image can be realized. The detected road surface signals include multiple types of road surface arrow information, such as going straight, turning left, turning right, going straight and turning left, going straight and turning right, a U-turn, turning left, going straight and turning right, etc.

[0085] In a second step, the lanes of the road are divided to obtain a plurality of lanes.

[0086] In some embodiments, a semantic segmentation branch in a deep neural network is employed to perform segmentation on the lanes of the road, outputting a number of lanes with class labels, where lanes of different classes can be represented by employing different class labels, e.g., the left lane is represented by class 1, the right lane is represented by class 2, and the background is represented by class 0.

[0087] In a third step, a stop line of the road is detected to obtain a stop line area.

[0088] In some embodiments, a stop line segmentation branch in the deep neural network is adopted to perform two-class segmentation on the stop lines on the road, and the obtained segmentation result is that the stop line region is represented by 1 and the background region is represented by 0, thereby achieving segmentation on the stop lines.

[0089] In the fourth step, the intersection signposts of the road are recognized to obtain multi-class signposts.

[0090] In some embodiments, an intersection direction output branch in the deep neural network is employed to perform semantic segmentation on the road intersection direction signs to obtain multi-class direction signs, for example, three classes are defined for the direction signs from left to right, in which the left direction sign is class 1, the forward direction sign is class 2, the right direction sign is class 3, and the background class is 0.

[0091] In a fifth step, obstacles on the road are detected and object information of the obstacles is obtained.

[0092] In some embodiments, an obstacle detection branch in the deep neural network is employed to detect obstacles on the road, and the obstacles are taken as the foreground of target detection, and non-obstacles are taken as the background, where the obstacles may be any object other than a vehicle or a pedestrian, and the obstacle information includes the position and size information of the obstacle, etc.

[0093] The above first to fifth steps may be executed simultaneously by different branches in the same network.

[0094] In a sixth step, at least one of the road signals, the plurality of lanes, the stop line areas, the multi-class directional signs, and the object information is determined as the road information.

[0095] In the sixth step, the road sign information, lane information, intersection travel direction information, etc. obtained in the first to fifth steps are used as road information. In this way, the road signal detection, lane detection, and intersection travel direction detection tasks are fused into the same deep learning network for joint learning, and the output road information is obtained, thereby enriching the content of the road information. This provides the vehicle with rich information, enabling it to generate effective control signals.

[0096] In some possible implementations, the road information can be analyzed to recognize actual road areas and unknown areas, so as to detect road boundaries that are not visible to the vehicle. As shown in Figure 2, Figure 2 is another implementation flowchart of an image processing method according to an embodiment of the present disclosure, which will be described below in conjunction with the steps shown in Figure 2.

[0097] In step S201, an actual road area and an unknown area that cannot be recognized by the vehicle are determined based on the road information.

[0098] In some embodiments, the actual road area of ​​the road can be obtained by analyzing the road surface signals, the multiple lanes, the stop line area, the multi-class driving direction signs, and the object information in the road information. For example, when there are no objects or pedestrians on the road, the road surface area of ​​the road is the actual road area. The unknown area that cannot be recognized by the vehicle may be an area on the road or an area outside the road. For example, the unknown area may be an area in the blind spot of the vehicle, an area blocked by an obstacle, or an area that cannot be recognized due to a long distance.

[0099] For example, in the crossroads scene shown in Figures 5A and 5B, there is a building (top, bottom, left and right of the image correspond to north, south, west and east, respectively) at the southwest corner of the intersection, and the building usually blocks the viewpoint of a vehicle traveling from south to north, so that the driver or sensor in the vehicle cannot obtain information of a part of the area blocked by the building, as shown in Figure 5B, and this part of the area can be called an unknown area, and is shown as area 522 in Figure 5B. A road boundary that is blocked by a building or cannot be sensed by a driver or sensor in the vehicle due to other reasons (e.g., due to too far distance) is called an invisible road boundary.

[0100] In step S202, road boundaries invisible to the vehicle are determined based on the actual road area and the unknown area.

[0101] In some embodiments, the actual road area and unknown area can be transformed from the current collection viewpoint to the actual road area and unknown area in the overhead viewpoint to determine overlapping road boundaries between the two areas in the overhead viewpoint, the road boundaries being road boundaries that are not visible to the vehicle.

[0102] In step S203, the road boundary that is not visible to the vehicle is determined as the target road boundary.

[0103] In some embodiments, the road boundary that is not visible to the vehicle is a target road boundary that is not recognizable to the vehicle among the multiple road boundaries, and thus the target road boundary can be accurately recognized by comparing the actual road area of ​​the vehicle-related road with the unknown area.

[0104] The above steps S201 to S203 provide a method for determining a target road boundary, in which a road boundary that is not visible to a vehicle is determined as a target road boundary that is dangerous to the vehicle. In this way, potential dangers to the vehicle can be effectively determined, and the driving safety of the autonomous vehicle can be improved.

[0105] In another implementation, target road boundaries that are dangerous for the vehicle can also be determined by analyzing road boundaries visible to the vehicle.

[0106] In some embodiments, the road boundaries that are not visible to the vehicle can be determined by converting both the actual road area and the unknown area into a bird's-eye view and analyzing the road boundaries that overlap between the two areas in the bird's-eye view. That is, the above step S202 can be realized by the following steps S221 to S223 (not shown).

[0107] In step S221, the collection viewpoints of the actual road area and the unknown area are converted into a bird's-eye view, respectively, to obtain a converted actual road area and a converted unknown area.

[0108] In some embodiments, the actual road area and the unknown area are respectively transformed to a bird's-eye view by adopting homography, so that the transformed actual road area and the transformed unknown area in the bird's-eye view can be obtained. In this way, the road information in the actual road area, for example, the road surface signal, the multiple lanes, the stop line area, the multi-class driving direction sign, and the object information in the actual road area are all transformed to the road surface signal, the multiple lanes, the stop line area, the multi-class driving direction sign, and the object information in the bird's-eye view. For example, the position of the object information in the actual road area is transformed to the position in the transformed road area in the bird's-eye view. Similarly, the road information in the unknown area is also transformed to the road information in the bird's-eye view at the same time.

[0109] In step S222, the overlapping area between the transformed actual road area and the transformed unknown area is determined.

[0110] In some embodiments, road information can be fitted in a transformed actual road region in an overhead view, road information can be fitted in a transformed unknown region, and the overlapping portion, i.e., overlapping region, between the two regions can be determined based on the fitted information.

[0111] In step S223, road boundaries within the overlapping region are determined as road boundaries that are not visible to the vehicle.

[0112] In some embodiments, since the unknown area is an area that cannot be recognized by the vehicle, the transformed unknown area is still an area that cannot be recognized by the vehicle, and based on this, the overlapping area between the transformed actual road area and the transformed unknown area is an actual road area that cannot be recognized by the vehicle, and the road boundary within the said area is also obviously not recognizable by the vehicle, i.e., the target road boundary.

[0113] In an embodiment of the present disclosure, by analyzing the overlapping area between the transformed actual road area and the transformed unknown area in a bird's-eye view, road boundaries that are invisible to the vehicle can be effectively recognized with limited network resources, making it easier to plan driving routes for following vehicles.

[0114] In some embodiments, the road information in the transformed actual road area in the bird's-eye view and the road information in the transformed unknown area in the bird's-eye view are fitted together, and the fitting result can obtain the overlapping road boundary between the two areas. That is, the above step S222 can be realized by the following steps.

[0115] In a first step, lanes, stop line areas and driving direction signs in the transformed actual road area are fitted to obtain first fitting information.

[0116] In some embodiments, a first fitting information is obtained by fitting a plurality of lanes, stop line regions, and multi-class driving direction signs in the transformed actual road area by a matrix transformation manner, the first fitting information including the fitted lanes, stop line regions, and multi-class driving direction signs in the transformed actual road area.

[0117] In a second step, the lanes, stop line regions and driving direction signs in the transformed unknown region are fitted to obtain second fitting information.

[0118] In some embodiments, second fitting information is obtained by fitting the multiple lanes, stop line regions, and multi-class signposts in the transformed unknown region by a method of matrix transformation, the second fitting information including the fitted lanes, stop line regions, and multi-class signposts in the transformed unknown region.

[0119] In a third step, an overlapping area between the transformed actual road area and the transformed unknown area is determined based on the first fitting information and the second fitting information.

[0120] In some embodiments, according to the fitted lanes, stop line areas, and multi-class driving direction signs in the transformed actual road area and the fitted lanes, stop line areas, and multi-class driving direction signs in the transformed unknown area, the overlapping lanes, stop line areas, and driving direction signs between the two areas can be determined, and the overlapping area between the two areas can be obtained.

[0121] In the embodiment of the present disclosure, the fitting results of the different road information can be obtained by fitting the road information in a bird's-eye view. In this way, the target road boundary in the overlapping area can be more accurately determined by comprehensively considering multiple pieces of information on the road.

[0122] In some embodiments, after recognizing the target road boundary that is dangerous to the vehicle, at least one of the target road boundary and road information is analyzed to generate a driving path for controlling vehicle driving, thereby controlling the automatic driving of the vehicle, that is, after step S103, the method further includes the steps shown in FIG.

[0123] In step S301, a driving route for the vehicle is determined based on the target road boundary and / or the road information.

[0124] In some embodiments, the driving route of the vehicle may be determined based on the target road boundary to control the driving of the vehicle, the driving route of the vehicle may be determined based on road information to control the driving of the vehicle, or the target road boundary and road information may be combined to determine the driving route of the vehicle to control the driving of the vehicle. The target road boundary may be analyzed and the location of the target road boundary may be notified to the vehicle to generate a driving route of the vehicle and control the vehicle to move away from the target road boundary, thereby reducing possible dangers during the driving of the vehicle; or the road information of the road where the vehicle is located, i.e., the road surface signals, the multiple lanes, the stop line areas, the multi-class driving direction signs and the object information, etc. of the road where the vehicle is located, may be analyzed to predict the driving route of the vehicle at a future time and control the driving of the vehicle; or the target road boundary may be combined with the road information to generate a more accurate driving route and control the driving of the vehicle more accurately.

[0125] The driving route is a route plan for the vehicle to travel at a future time, and includes the vehicle's driving direction, driving speed, driving route, etc. The driving route of the vehicle may be determined based on a target road boundary, or based on road information, or the driving route of the vehicle may be determined by combining the target road boundary and the road information.

[0126] In some possible implementations, a driving route of the vehicle is determined based on the road information. This step can be realized by the following steps.

[0127] In a first step, a driving intention of the vehicle is determined based on the road information.

[0128] In some embodiments, the vehicle's driving intention is determined based on at least some information of road information, such as multiple lanes, stop line areas, and multi-class driving direction signs, and the driving intention is used to represent the vehicle's driving manner in the future period, such as the driving speed and driving direction in the next minute.

[0129] In a second step, a driving route for the vehicle is determined based on the driving intention.

[0130] In some embodiments, the driving route is determined by specifying a driving route for the vehicle during a predetermined time period in the future according to the driving intention of the vehicle during the predetermined time period. For example, if the driving intention is to go straight, a route for the vehicle to go straight during the predetermined time period is determined.

[0131] In step S302, the driving of the vehicle is controlled based on the driving route.

[0132] In some embodiments, the electronic device can determine a driving route for the vehicle based on the road boundary that the vehicle can enter, and can control the vehicle to drive according to the driving route. In this way, by comprehensively considering the target road boundary and the road information, it is possible to realize effective control of the vehicle.

[0133] In some embodiments, that is, the "step of determining a driving route for the vehicle based on the road information" in the above step S301 can be realized by the following steps S311 and S312 (not shown).

[0134] In step S311, the turning direction and turning position of the vehicle are determined based on the road surface signals and driving direction signs in the road information.

[0135] In some embodiments, it can be determined according to the road surface signal in the road information whether the vehicle's travel direction sign is straight ahead, left turn, right turn, straight ahead left turn, straight ahead right turn, U-turn, left turn straight ahead right turn, the turning position represents the turning point where the vehicle enters the turning road when turning, and the turning direction represents the driving direction of the vehicle when entering the turning road from the current position. In this way, the turning direction may be the driving direction continuously maintained by the vehicle during the turning process.

[0136] In step S312, a turning travel route of the vehicle is determined based on the turning direction and the turning position.

[0137] In some embodiments, the turning path of the vehicle is predicted according to the traveling direction when the vehicle turns, which is indicated by the turning direction, and the turning point when the vehicle turns, so that the vehicle can accurately turn based on the turning path. In this way, by following the road surface signal in the road information, the turning direction and turning position of the vehicle at a future time can be accurately predicted, and the turning of the vehicle can be accurately controlled.

[0138] In the above steps S311 and S312, the road surface signals and travel direction signs in the road information are obtained from the road information, and a vehicle travel route is generated according to the road surface signals and travel direction signs, thereby improving the accuracy of the travel route.

[0139] In some embodiments, the driving route is updated by detecting obstacle information in the road image, thereby effectively controlling the driving of the vehicle. That is, the above step S302 can be realized by the following steps S321 and S322 (not shown).

[0140] In step S321, the driving route is updated based on the obstacle information in the road information to obtain an updated route.

[0141] In some embodiments, when object information of an obstacle is detected in the road information, i.e., when an obstacle exists on the road, the generated driving route is updated according to the object information of the obstacle in the road information, for example, the route that passes through the obstacle position in the original driving route is updated according to the position information and size information of the obstacle, so that the updated route avoids the obstacle.

[0142] In step S322, the driving of the vehicle is controlled based on the updated route.

[0143] In some embodiments, the vehicle can be controlled to travel along the updated route, thereby avoiding obstacles while the vehicle is traveling, thereby improving the safety of the vehicle's travel.

[0144] In the above steps S321 and S322, the position information of obstacles in the road information is combined to update the driving route, thereby controlling the vehicle's driving according to the updated route and providing more information to the autonomous vehicle when making decisions.

[0145] In the embodiment of the present disclosure, after recognizing the target road boundary, a subsequent driving path is generated in combination with abundant road information, so that the generated driving path is more accurate, and based on this, the vehicle can be controlled accurately according to the driving path.

[0146] In some embodiments, the vehicle driving route is generated by analyzing the target road boundary and updating the map of the vehicle's location. That is, in the above step S301, the step of determining the vehicle's driving route based on the target road boundary can be realized by the following steps:

[0147] In a first step, map data of the vehicle's location is updated based on the target road boundary to obtain an updated map.

[0148] In some embodiments, map data of the vehicle's location is obtained, which may be a third-party map, road information collected by a positioning system in the vehicle, and traffic signs (e.g., traffic sign lights, traffic sign boards), etc. The target road boundary is marked on the map data of the vehicle's location to obtain the updated map. In this way, the target road boundary is carried in the updated map, and the vehicle can be informed of where the invisible road boundary exists.

[0149] In a second step, a driving route for the vehicle is determined based on the updated map.

[0150] In some embodiments, a driving route is planned that moves away from the target road boundary according to the target road boundary marked on the updated map, thereby avoiding contact with the target road boundary when the vehicle travels along the driving route.

[0151] In an embodiment of the present disclosure, a map that takes into account road hazards is created according to the detected target road boundaries, and a driving route that controls vehicle driving is generated according to the updated map, thereby improving the safety of the driving route.

[0152] In some embodiments, after recognizing the target road boundary, the relationship between the target road boundary and the driving state is analyzed to effectively control the driving of the vehicle. This step can be realized by the following process.

[0153] The vehicle is controlled based on the relationship between the target road boundary and the running state of the vehicle.

[0154] Here, the relationship between the target road boundary and the vehicle's driving state is used to represent the influence of the target road boundary on the driving state, and includes the magnitude of the angle between the target road boundary and the vehicle's driving direction, the magnitude of the distance between the target road boundary and the driving vehicle, etc.

[0155] In some possible implementations, after recognizing the target road boundary, the vehicle can be controlled to enter a braking state, i.e. controlling the vehicle may be controlling the vehicle from a running state to enter a braking state or controlling the vehicle to move away from the target road boundary.

[0156] In this way, after the target road boundary is determined, braking instruction information is generated to cause the vehicle to enter a braking state. In this way, when the target road boundary is recognized, the vehicle is controlled to prepare for braking, thereby improving the safety of vehicle driving. After the electronic device determines the target road boundary, the electronic device generates braking instruction information and feeds back the braking instruction information to the automatic driving system of the vehicle, and the automatic driving system of the vehicle controls the vehicle to enter a braking state in response to the braking instruction information. When it is determined that there is a danger, the vehicle can be controlled to enter a braking state from a running state or the vehicle can be controlled to move away from the target road boundary. For example, after the target road boundary is recognized, the vehicle is controlled to enter a braking state from a running state or the vehicle is controlled to move away from the target road boundary. In this way, after the target road boundary is detected, braking instruction information is generated to control the vehicle to enter a braking state, thereby improving the driving safety of the vehicle.

[0157] In some possible implementations, the relationship between the target road boundary and the vehicle's driving state includes at least one of the following cases:

[0158] Case 1: The relationship between the target road boundary and the traveling state of the vehicle may be such that the distance between an overlapping area in which the target road boundary is located and a road intersection ahead of the vehicle is less than a second predetermined distance.

[0159] Here, the intersection is an intersection along the vehicle's traveling direction on the roadway where the vehicle is located, i.e., an intersection ahead of the roadway where the vehicle is located. The distance from the intersection to the overlapping area may be the minimum distance between the intersection and the overlapping area, or may be the average value of the maximum distance and the minimum distance between the intersection and the overlapping area. The second predetermined distance may be the same as or different from the first predetermined distance, and may be set based on measuring the blind spot range of the vehicle, or may be set independently by the user. If the distance between the intersection and the overlapping area is less than the second predetermined distance, it means that the invisible overlapping area affects vehicles passing through the intersection, and in this case, braking instruction information is generated to control the vehicle to enter a braking state in order to improve the safety of vehicle driving.

[0160] Case 2: The relationship between the target road boundary and the traveling state of the vehicle may be such that the distance between the overlapping area and a location of the vehicle is less than a third predetermined distance.

[0161] In some embodiments, the distance between the overlapping area and the vehicle location may be the minimum distance between the overlapping area and the vehicle location, or may be the average value of the maximum distance and the minimum distance between the overlapping area and the vehicle location, and the third predetermined distance may be the distance between the vehicle location and the road edge when the vehicle is running normally. If the distance between the overlapping area and the vehicle location is less than the third predetermined distance, it means that the overlapping area affects the normal running of the vehicle, and in this case, to improve the safety of the vehicle running, braking instruction information is generated to control the vehicle to enter a braking state.

[0162] Case 3: The relationship between the target road boundary and the traveling state of the vehicle may be such that an angle between a traveling direction of the vehicle and the target road boundary is less than a predetermined angle.

[0163] Here, the predetermined angle may be set based on the minimum angle between the driving direction of the vehicle and the road boundary when the vehicle is driving normally, for example, the minimum angle between the driving direction of the vehicle and the road boundary on the premise that the vehicle can turn normally. If the angle between the driving direction of the vehicle and the target road boundary is less than the predetermined angle, it means that the target road boundary affects the normal driving of the vehicle, and in this case, the vehicle can be controlled to enter a braking state from a driving state or the vehicle can be controlled to move away from the target road boundary, thereby improving the safety of the vehicle driving.

[0164] Case 4: The relationship between the target road boundary and the traveling state of the vehicle may be that the target road boundary is in contact with a roadway on which the vehicle is located.

[0165] In this case, if the target road boundary contacts the roadway where the vehicle is located, the vehicle will contact the target road boundary if it continues to travel on the roadway according to the current travel direction. Since the danger of the target road boundary is unpredictable, when the target road boundary contacts the roadway where the vehicle is located, the vehicle can be controlled to enter a braking state from a traveling state or the vehicle can be controlled to move away from the target road boundary, thereby effectively reducing the potential danger of the vehicle traveling.

[0166] In an embodiment of the present disclosure, by analyzing the relationship between the target road boundary and the driving state of the vehicle, if the target road boundary affects the normal driving of the vehicle, braking instruction information for controlling the vehicle to enter a braking state or controlling the vehicle to move away from the target road boundary can be generated, thereby further improving the driving safety of the vehicle.

[0167] In some embodiments, after determining the target road boundary, the object recognition in the region of interest can be made more accurate by acquiring the region of interest and road images according to different resolutions or frame rates. This step can be realized in the following manner.

[0168] Method 1: A region of interest is set based on the target road boundary, and an image corresponding to the region of interest is obtained based on a first resolution.

[0169] In manner one, the road image is acquired according to a second resolution, the second resolution being smaller than the first resolution.

[0170] Method 2: Acquire an image corresponding to the region of interest based on a first frame rate.

[0171] In the second method, the road images are captured based on a second frame rate, and the second frame rate is smaller than the first frame rate.

[0172] Here, the electronic device sets a Region of Interest (ROI) based on a road boundary where the vehicle can enter. The electronic device can adopt a second resolution (also called a low resolution) to acquire road images for the road environment, while the region of interest can be acquired by adopting a first resolution (also called a high resolution) higher than the second resolution. In this way, higher quality images can be collected for the region of interest, facilitating object recognition for the images corresponding to the subsequent region of interest.

[0173] Alternatively, the electronic device can adopt a second frame rate (also called a low frame rate) to acquire road images for a road environment, while the electronic device can adopt a first frame rate (also called a high frame rate) higher than the second frame rate to acquire the region of interest, thus facilitating object recognition for the images corresponding to the subsequent region of interest.

[0174] In some embodiments, after detecting that the vehicle has departed from the target road boundary, send risk prediction notification information to a vehicle behind the vehicle to inform the vehicle behind to pay attention to the target road boundary. This step can be realized by the following process.

[0175] First, road environment information around the target road boundary is collected.

[0176] In some embodiments, when it is detected that the vehicle has left the target road boundary, road environment information around the target road boundary is collected. Since the target road boundary is not visible, the vehicle cannot predict the risk that may exist at the target road boundary. Therefore, after it is detected that the vehicle has passed the target road boundary, a camera in the vehicle can recognize the target road boundary and collect road environment information around the target road boundary by the camera. The road environment information includes the length, position, obstacle information, road surface signals, etc. of the target road boundary.

[0177] Next, notification information is generated based on the road environment information.

[0178] In some embodiments, the notification information is transmitted to a vehicle behind the vehicle based on road environment information of the target road boundary being included in the notification information.

[0179] Finally, the notification information is transmitted to a vehicle behind the vehicle.

[0180] In some embodiments, notification information including road environment information is sent to the autonomous driving system of the rear vehicle or to a terminal that communicates with the autonomous driving system, so that the rear vehicle can plan an appropriate driving route based on the road environment information in the notification information.

[0181] In an embodiment of the present disclosure, after it is detected that a vehicle has passed a target road boundary, road environment information around the target road boundary is transmitted to a rear vehicle in the form of notification information, so as to timely inform the rear vehicle that a target road boundary exists ahead, and thus the rear vehicle can timely adjust its driving route.

[0182] In the following, an illustrative application example of the embodiment of the present disclosure in a practical application scenario is described, taking as an example employing a deep neural network to recognize road marks and determine whether a vehicle should turn at a road intersection.

[0183] As an overall system, autonomous driving requires that the output of the sensing module provides services to the following modules, for example, the sensing result not only provides whether an object exists ahead, but also provides the corresponding logic output to the following modules, and provides certain control and logic signals for autonomous driving. However, commercially available sensing modules do not effectively combine all the sensing information. Thus, many problems are introduced in the application, namely, the purpose of sensing is only to judge whether the target exists, and there is no concern about the reliability and accuracy of the following control signals.

[0184] In view of this, an embodiment of the present disclosure provides a road intersection driving direction selection method based on road signs, and adopts road sign information, lane information, and intersection driving direction information, etc., to provide effective automatic driving signals to lower modules.

[0185] An embodiment of the present disclosure provides an image processing method, which detects road boundaries to obtain road sensing information, converts the obtained road sensing information into a bird's-eye view, and integrates the road sensing information in the bird's-eye view to determine a turn at a road intersection. The method can be realized by the following steps:

[0186] In the first step, road boundaries are detected in a road image and road sensing information of the road is extracted.

[0187] In some embodiments, road boundary detection can be achieved by the following two methods.

[0188] Method 1: Detect road boundaries directly using the detection model. This step can be achieved by the following process.

[0189] During autonomous driving, the vehicle performs sensing output according to the information provided on the road, and performs information integration on the results output from the model. As shown in FIG. 4, FIG. 4 is a network structure diagram of an image processing method according to an embodiment of the present disclosure, and the network architecture includes an image input module 401, a backbone network 402, a road surface signal detection branching network 41, a lane segmentation branching network 42, a stop line segmentation branching network 43, an intersection travel direction output branching network 44, and an obstacle detection output branching network 45. Wherein, The image input module 401 is configured to input a road image; The backbone network 402 is configured to perform feature extraction on the input road image.

[0190] The backbone network may be a VGG network, a GoogleNet network, or a residual (ResNet) network, etc.

[0191] The road signal detection branching network 41 is configured to perform a detection task and performs road signal detection based on the extracted image features.

[0192] Here, the road surface signal detection branch network 41 may be realized by a detector such as a two-stage detector or a one-stage detector, etc. The road surface signal detection branch network 41 may be a classification branch for classifying the detected road surface signals, where the classification includes: go straight, turn left, turn right, go straight and turn left, go straight and turn right, U-turn, turn left, go straight and turn right, etc.

[0193] The lane segmentation branching network 42 is configured to perform segmentation on lanes in the road image based on the extracted image features.

[0194] Here, taking lane labeling of "3 roads, 4 lanes" as an example, the labeled labels include the left lane of the roadway where the vehicle is located (i.e., the left lane), the left lane of the left roadway where the vehicle is located (i.e., the left-hand lane), the right lane of the roadway where the vehicle is located (the right-hand lane), and the right lane of the right roadway where the vehicle is located (the right-hand lane). The lane detection task is defined as semantic segmentation. That is, the left-hand lane is class 1, the left lane is class 2, the right lane is class 3, the right lane is class 4, and the background class is class 0.

[0195] The stop line segmentation branching network 43 is configured to perform segmentation on stop lines in the road image based on the extracted image features.

[0196] Here, a two-class segmentation method can be adopted to perform stop line detection, and the stop line region can be set as 1 and the background class as 0.

[0197] The intersection heading output branching network 44 is configured to employ semantic segmentation methods to recognize intersection heading changing edges.

[0198] Here, from left to right, the intersection travel direction is defined into three classes, with the left travel direction changing edge being class 1, the forward travel direction changing edge being class 2, the right travel direction changing edge being class 3, and the background class being 0.

[0199] The obstacle detection output branching network 45 is configured to recognize obstacles on the road surface, and performs the obstacle recognition by a detection method.

[0200] Here, an obstacle detection output branching network 45 is adopted to recognize obstacles on the road surface, and the obstacles are taken as the foreground of target detection, and non-obstacles are taken as the background. As shown in Fig. 5A, in the collected vehicle-mounted camera image 511, road boundary detection is performed, so that obstacles 512 and road boundaries 513 in the image 511 can be recognized.

[0201] Method 2: Use a detection model to detect other road information, and estimate road boundaries based on the other road information. Here, the method of method 1 can be adopted to detect other road information.

[0202] There are two methods for estimating road boundaries based on other road information:

[0203] 1. Connect the ends of each road to determine the road boundaries.

[0204] 2. Use semantic segmentation to determine the drivable area and use the contours of the drivable area to determine road boundaries.

[0205] In a second step, road boundaries that are not visible to the ego vehicle are determined based on the road sensing information.

[0206] In some embodiments, road boundaries that are not visible to the ego vehicle can be determined by the following steps.

[0207] In step 1, road information is recognized from images collected by an on-board camera.

[0208] Here, the road information includes object information and lanes on the road. The network architecture shown in Figure 4 can recognize road information for collected images.

[0209] In step 2, an unknown area that is not visible from the host vehicle is determined.

[0210] Here, the unknown area may be an area that is blocked by an obstruction.

[0211] In step 3, the actual road area is estimated based on the road information.

[0212] In step 4, the actual road area and unknown area are transformed into a bird's-eye view.

[0213] Here, image 511 of FIG. 5A may be converted into an image from a bird's-eye view such as image 521 shown in FIG. 5B, where in image 521, unknown area 522 is an area not visible to host vehicle 523, actual area 524 is the actual road area, boundaries 525, 526 and 527 are road boundaries that are foreseeable to host vehicle 523, boundary 528 is a road boundary that is not visible to host vehicle 523, and obstacle 529 is an obstacle.

[0214] In step 5, if the actual road area in the bird's-eye view overlaps with the unknown area, the road boundary of the actual area that overlaps with the unknown area is determined as the road boundary that is not visible from the host vehicle.

[0215] In some possible implementations, the road sensing information obtained in the first step is transformed into a bird's-eye view by homography. That is, the road sensing information in the forward view of the vehicle is transformed into road sensing information in a bird's-eye view by a method of matrix transformation, and the road sensing information in the bird's-eye view is fitted. That is, the lanes, stop lines, and turning edges in the bird's-eye view are fitted to obtain a fitting result. As shown in FIG. 6A, FIG. 6A is a schematic diagram of an application scene of the image processing method according to an embodiment of the present disclosure, and as can be seen from FIG. 6A, road sensing information in the forward view, that is, stop lines 51, 52, and 53, turning edges 54, 55, and 56, lanes 501 and 502, and obstacles 503 in the forward view are shown. The stop lines, direction-changing edges, lanes and obstacles in Figure 6A are converted to a bird's-eye view, and as shown in Figure 6B, stop lines 51, 52 and 53 are converted to stop lines 61 and 62 in the bird's-eye view, direction-changing edges 54, 55 and 56 are converted to direction-changing edges 63, 64 and 65 in the bird's-eye view, and obstacle 503 is converted to obstacle 601 in the bird's-eye view.

[0216] Also, as can be seen from FIG. 6B, the host vehicle 605 can detect semantic information based on the road surface signal, and therefore can know that the host vehicle can turn right. Therefore, the host vehicle selects the right direction-changing edge, obtains the turning direction and turning position, etc., generates a subsequent route plan, and transmits a control signal to the host vehicle to control the turning. Similarly, the host vehicle can generate other straight-line commands such as left turn and straight-line based on the road surface signal, and can generate more stable signals by aligning with the signals on the map. At the same time, the host vehicle considers the position information of obstacles on the road. That is, when a certain direction-changing edge is blocked by an obstacle, it feeds back that the road boundary in that direction cannot be accurately recognized, thereby providing more information for the autonomous vehicle to make decisions.

[0217] In the third step, after the invisible road boundary is recognized, braking preparation is performed to control the vehicle to move away from the invisible road boundary.

[0218] In some possible implementations, after determining the overlapping area between the unknown area and the actual road area, prepare to brake if the distance between the intersection and the overlapping area is within a predetermined range, or if the distance between the overlapping area and an unknown area that the vehicle cannot recognize is within a predetermined range, prepare to brake if the angle between the driving direction of the vehicle and the invisible road boundary is less than a predetermined value, prepare to brake if the invisible road boundary borders the vehicle's roadway, and at the invisible road boundary, control the vehicle to move away from the road boundary bordering the vehicle's roadway.

[0219] In the embodiment of the present disclosure, a deep neural network is used to predict road markings, lanes and road intersection driving direction information to obtain accurate road structure information and driving direction. Based on the above sensing information, the sensing information in the forward perspective is converted into a bird's-eye view to determine the vehicle's driving direction information at the road intersection. In this way, the road markings, lane detection and intersection driving direction detection tasks are fused into the same deep learning network for joint learning, and a final sensing output is obtained, providing an effective signal for subsequent directional control; and, by fusing and learning multiple tasks into a hybrid network, network resources can be effectively saved.

[0220] In the embodiment of the present disclosure, when a vehicle is at a crossroads, a necessary road boundary can be selected by detecting multiple road boundaries. As shown in FIG. 7, when a vehicle 71 is at a crossroads, a necessary road boundary is selected by detecting multiple road boundaries, for example, when the vehicle 71 is traveling on the left road, the road boundary of the left road is selected as a reachable road. The road boundaries detected by the vehicle 71 include road boundaries 81-88 as shown in FIG. 8, and reachable boundaries and unreachable boundaries are determined among the road boundaries 81-88. As shown in FIG. 9, boundaries 91, 93, 95, and 98 are reachable boundaries, and boundaries 92, 94, 96, and 97 are unreachable boundaries.

[0221] In the embodiments of the present disclosure, richer shape kernels and control information can be provided for an autonomous vehicle based on whether road boundaries are visible based on obstacle detection. From the perspective of model design and learning, road markings, lane detection, and intersection direction detection tasks are fused into the same deep learning network to perform joint learning, and a final sensing output can be obtained, which not only effectively saves network resources, but also provides effective signals for subsequent directional control.

[0222] Those skilled in the art can understand that in the above method of the specific embodiment, the described order of each step does not limit the implementation process in a strict execution order, and the specific execution order of each step should be determined by its function and possible internal logic.

[0223] Based on the same technical concept, the embodiments of the present disclosure further provide a movement intention determination device corresponding to the movement intention determination method, and the principle for solving the problems of the device in the embodiments of the present disclosure is similar to the above-mentioned movement intention determination method in the embodiments of the present disclosure, so the implementation of the device can refer to the above-mentioned method embodiments.

[0224] An embodiment of the present disclosure provides an image processing device. FIG. 10 is a schematic diagram showing a configuration of an image processing device according to an embodiment of the present disclosure. As shown in FIG. 10, the image processing device 1000 includes: an image capture portion configured to capture road images collected by an image capture device mounted on the vehicle; a road boundary detection portion configured to detect a plurality of road boundaries in the road image based on the road image; a target road boundary determining portion configured to determine, among the plurality of road boundaries, a target road boundary that is dangerous for the vehicle.

[0225] In some embodiments, the road boundary detection portion 1002 further comprises: The road image is configured to detect and determine a plurality of road boundaries associated with the vehicle.

[0226] In some embodiments, the road boundary detection portion 1002 includes: a roadway detection subpart configured to detect the road image to obtain a plurality of roadways in the road image; a first road boundary determination sub-portion configured to connect ends of each roadway in the plurality of roadways to obtain the plurality of road boundaries.

[0227] In some embodiments, the road boundary detection portion 1002 includes: a drivable area segmentation sub-part configured to perform semantic segmentation on the road image to obtain a drivable area in the road image; and a second road boundary determination sub-portion configured to determine the plurality of road boundaries based on a contour of the drivable area.

[0228] In some embodiments, the target road boundary determination portion 1003 includes: a first target road boundary determination sub-portion configured to determine, among the plurality of road boundaries, a road boundary adjacent to a roadway on which the vehicle is located as the target road boundary; a second target road boundary determination sub-portion configured to determine, among the plurality of road boundaries, a road boundary whose distance from the vehicle is less than a first predetermined distance as the target road boundary; a third target road boundary determination sub-part configured to determine, among the plurality of road boundaries, a road boundary whose road space between the vehicle and the target road boundary is less than a predetermined space as the target road boundary; A fourth target road boundary determination sub-part configured to determine, among the multiple road boundaries, a target road boundary that is dangerous to the vehicle based on road information determined by the road image, the road information including at least one of road surface signals, lanes, stop line areas, directional signs, and obstacle information in the road image.

[0229] In some embodiments, the fourth target road boundary determination subpart comprises: an unknown road area determining portion configured to determine an actual road area and an unknown area that is not recognizable to the vehicle based on the road information; a road boundary determination portion configured to determine road boundaries that are not visible to the vehicle based on the actual road region and the unknown region; and a target road boundary determining portion configured to determine a road boundary that is not visible to the vehicle as the target road boundary.

[0230] In some embodiments, the road boundary determination unit comprises: an area viewpoint transformation subpart configured to transform the collection viewpoints of the actual road area and the unknown area into a bird's-eye viewpoint, respectively, to obtain a transformed actual road area and a transformed unknown area; an overlap area determining subpart configured to determine an overlap area between the transformed actual road area and the transformed unknown area; and a road boundary determination subportion configured to determine road boundaries within the overlapping region as road boundaries that are not visible to the vehicle.

[0231] In some embodiments, the overlap area determination subpart is further configured to fit lanes, stop line areas, and directional signs in the transformed actual road area to obtain first fitting information, fit a plurality of lanes, stop line areas, and directional signs in the transformed unknown area to obtain second fitting information, and determine an overlap area between the transformed actual road area and the transformed unknown area based on the first fitting information and the second fitting information.

[0232] In some embodiments, the image processor comprises: A driving route determination portion configured to determine a driving route of the vehicle based on the target road boundary and / or the road information; and a vehicle driving control portion configured to control driving of the vehicle based on the driving route.

[0233] In some embodiments, the driving route determination module: a turn determination sub-portion configured to determine a turn direction and a turn position of the vehicle based on road surface signals and driving direction signs in the road information; and a turn driving path determination subportion configured to determine a driving path of the vehicle based on the turn direction and the turn position.

[0234] In some embodiments, the vehicle cruise control module further comprises: a driving route update subpart configured to update the driving route based on object information of an obstacle in the second road information to obtain an updated route; and a vehicle driving control subportion configured to control driving of the vehicle based on the updated route.

[0235] In some embodiments, the driving route determination module: a map data updating subpart configured to update map data of the vehicle's location based on the target road boundary to obtain an updated map; and a driving route determination subportion configured to determine a driving route for the vehicle based on the updated map.

[0236] In some embodiments, the image processor comprises: The vehicle control portion further includes a vehicle control portion configured to control the vehicle based on a relationship between the target road boundary and a driving state of the vehicle.

[0237] In some embodiments, the relationship between the target road boundary and the vehicle's driving state is The distance between the overlapping area in which the target road boundary is located and the road intersection in front of the vehicle is less than a second predetermined distance; the distance between the overlapping area and the location of the vehicle is less than a third predetermined distance; The angle between the vehicle's traveling direction and the target road boundary is less than a predetermined angle; The target road boundary borders a roadway on which the vehicle is located.

[0238] In some embodiments, the vehicle control module is further configured to perform the steps of controlling the vehicle to enter a braking state from a driving state or controlling the vehicle to move away from the target road boundary.

[0239] In some embodiments, the image processor comprises: a first region of interest determination section configured to set a region of interest based on the target road boundary and to acquire an image corresponding to the region of interest based on a first resolution, the road image being acquired according to a second resolution, the second resolution being smaller than the first resolution; and / or The system further comprises a second region of interest determination portion configured to acquire an image corresponding to the region of interest based on a first frame rate, the road image being acquired based on a second frame rate, the second frame rate being smaller than the first frame rate.

[0240] In some embodiments, the image processor comprises: A road environment information collection portion configured to collect road environment information around the target road boundary; A notification information generating unit configured to generate notification information based on the road environment information; The notification information portion is configured to transmit the notification information to a vehicle behind the vehicle, the vehicle being located on the same roadway as the vehicle and traveling in the same direction.

[0241] It should be noted that the above description of the apparatus embodiment is similar to the above description of the method embodiment, and has similar beneficial effects as the method embodiment. The technical details not disclosed in the apparatus embodiment of the present disclosure can be understood by referring to the description of the method embodiment of the present disclosure.

[0242] In this and other embodiments of the present disclosure, a "module" may be a circuit, a processor, a program, or software, and may of course be a unit or a non-module.

[0243] It should be noted that in the embodiments of the present disclosure, the above-mentioned road obstacle detection method can be implemented as a software function module and stored in a computer-readable storage medium when sold or used as an independent product. Based on this understanding, the essential part of the technical solution of the embodiments of the present disclosure or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a terminal, a server, etc.) to execute all or part of the method described in each embodiment of the present disclosure. The aforementioned storage medium includes various media that can store program code, such as a U disk, a mobile hard disk, a read only memory (ROM), a magnetic disk, or an optical disk. In this way, the embodiments of the present disclosure are not limited to a specific combination of hardware and software.

[0244] Correspondingly, the embodiment of the present disclosure further provides a computer program product having computer executable instructions stored therein, which can realize the steps of the image processing method provided by the embodiment of the present disclosure after being executed. Correspondingly, the embodiment of the present disclosure further provides a computer storage medium having computer executable instructions stored therein, which, when executed by a processor, realizes the steps of the image processing method provided by the above embodiment. Correspondingly, the embodiment of the present disclosure provides a computer device, and FIG. 11 is a schematic diagram showing the configuration of a computer device according to the embodiment of the present disclosure. As shown in FIG. 11, the computer device 1100 includes one processor 1101, at least one communication bus, a communication interface 1102, at least one external communication interface, and a memory 1103. Here, the communication interface 1102 realizes the connection communication between these components. Here, the communication interface 1102 may include a display, and the external communication interface may include a standard wired interface and a wireless interface. Here, the processor 1101 executes an image processing program in the memory to realize the steps of the image processing method provided by the above embodiment.

[0245] The above description of the image processing device, computer device, and storage medium embodiments is similar to the description of the above method embodiments, and has similar technical descriptions and beneficial effects to the method embodiments. Due to space limitations, reference can be made to the description of the above method embodiments, and will not be repeated here. Technical details not disclosed in the image processing device, computer device, and storage medium embodiments of the present disclosure can be understood by referring to the description of the method embodiments of the present disclosure. It should be understood that in this specification, "one embodiment" or "one embodiment" means that a particular feature, structure, or characteristic associated with an embodiment is included in at least one embodiment of the present disclosure. Therefore, "one embodiment" or "one embodiment" in the entire specification does not necessarily refer to the same embodiment. Furthermore, these particular features, structures, or characteristics can be incorporated in one or more embodiments in any suitable manner. It should be understood that in each embodiment of the present disclosure, the magnitude of the sequence number of each of the above processes does not mean the order of execution, and the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present disclosure. The sequence numbers of the above embodiments of the present disclosure are for illustrative purposes only and do not represent the superiority or inferiority of the embodiments.

[0246] It should be explained that, as used herein, the terms "comprising," "including," or any other variation thereof, are intended to cover a non-exclusive inclusion, whereby a process, method, article, or apparatus that comprises a set of elements includes not only those elements, but also other elements not expressly listed or that are inherent to such process, method, article, or apparatus. Unless specifically limited, an element defined with the phrase "comprising..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes the element.

[0247] It should be understood that in some embodiments provided in the present disclosure, the disclosed devices and methods can be realized in other ways. The above-described device embodiments are merely illustrative, for example, the division of the units is merely a division of logical functions, and in actual implementation, there may be other division methods, for example, multiple units or components may be combined, integrated into another system, or some features may be ignored or not implemented. Furthermore, the mutual coupling or direct coupling or communication connection between each component shown or described may be an indirect coupling or communication connection via some interfaces, devices or units, and may be in an electrical, mechanical or other form.

[0248] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, and may be located in one place or distributed among multiple network units. Some or all of the units may be selected according to actual needs to achieve the objectives of the technical solutions of the present embodiment. In addition, the functional units in each embodiment of the present disclosure may all be integrated into one second processing unit, each unit may be used individually as one unit, or two or more units may be integrated into one unit. The integrated units may be embodied in the form of hardware or in the form of a combination of hardware and software functional units. Those skilled in the art will understand that all or some of the steps for implementing the above embodiments may be performed by hardware associated with program instructions, and the above-mentioned programs may be stored in a computer-readable storage medium, and when the programs are executed, the steps including the above-mentioned method embodiments are performed, and the above-mentioned storage medium includes various media capable of storing program code, such as removable storage, read-only memory (ROM), magnetic disk, or optical disk.

[0249] Alternatively, the above integrated units of the present disclosure may be realized in the form of a software functional module and stored in one computer-readable storage medium when sold or used as an independent product. Based on this understanding, the essential part of the technical solution of the embodiments of the present disclosure or the part that contributes to the prior art may be embodied in the form of a software product, and the computer software product is stored in one storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the method described in each embodiment of the present disclosure. The above storage medium includes various media that can store program code, such as removable storage, ROM, magnetic disk, or optical disk. The above content is merely an embodiment of the present disclosure, and the protection scope of the present disclosure is not limited thereto. Any modifications or replacements that a person skilled in the art can easily think of within the technical scope disclosed in this disclosure should be included in the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure shall be subject to the protection scope of the claims. [Industrial Applicability]

[0250] An embodiment of the present disclosure provides an image processing method, apparatus, device, and storage medium, in which a road image collected by an image collection device mounted on a vehicle is acquired, a plurality of road boundaries in the road image are detected based on the road image, and a target road boundary that is dangerous to the vehicle is determined among the plurality of road boundaries.

Claims

1. An image processing method executed by an electronic device, comprising: acquiring road images collected by an image collection device mounted on a vehicle; detecting a plurality of road boundaries in the road image based on the road image; determining a target road boundary among said plurality of road boundaries that is dangerous to said vehicle.

2. The step of detecting a plurality of road boundaries in the road image based on the road image includes: detecting the road image to determine a plurality of road boundaries associated with the vehicle; The image processing method according to claim 1 .

3. The step of detecting a plurality of road boundaries in the road image based on the road image includes: detecting the road image to obtain a plurality of roadways in the road image; and connecting ends of each roadway in the plurality of roadways to obtain the plurality of road boundaries.

3. The image processing method according to claim 1.

4. The step of detecting a plurality of road boundaries in the road image based on the road image includes: performing semantic segmentation on the road image to obtain a drivable area within the road image; determining the plurality of road boundaries based on a contour of the drivable area; The image processing method according to any one of claims 1 to 3.

5. The step of determining a target road boundary that is dangerous for the vehicle among the plurality of road boundaries includes: determining, from among the plurality of road boundaries, a road boundary adjacent to a roadway on which the vehicle is located as the target road boundary; determining, among the plurality of road boundaries, a road boundary whose distance from the vehicle is less than a first predetermined distance as the target road boundary; determining, as the target road boundary, a road boundary among the plurality of road boundaries, the road boundary having a road space between the vehicle and the plurality of road boundaries that is less than a predetermined space; determining a target road boundary that is dangerous for the vehicle among the plurality of road boundaries based on road information determined by the road image, the road information including at least one of road signals, lanes, stop line areas, turn-by-turn signs, and obstacle information in the road image; The image processing method according to any one of claims 1 to 4.

6. The step of determining a target road boundary that is dangerous for the vehicle among the plurality of road boundaries based on road information determined from the road image includes: determining an actual road area and an unknown area that is not recognizable to the vehicle based on the road information; determining road boundaries that are not visible to the vehicle based on the actual road region and the unknown region; determining a road boundary that is not visible to the vehicle as the target road boundary. The image processing method according to claim 5.

7. The step of determining road boundaries not visible to the vehicle based on the actual road region and the unknown region comprises: Transforming the collection viewpoints of the actual road area and the unknown area into a bird's-eye view, respectively, to obtain a transformed actual road area and a transformed unknown area; determining an overlap area between the transformed actual road area and the transformed unknown area; determining road boundaries within the overlapping region as road boundaries that are not visible to the vehicle; The image processing method according to claim 6.

8. The step of determining an overlap area between the transformed actual road area and the transformed unknown area includes: fitting lanes, stop line areas, and driving direction signs in the transformed actual road area to obtain first fitting information; fitting lanes, stop line regions, and driving direction signs in the transformed unknown area to obtain second fitting information; determining an overlap area between the transformed actual road area and the transformed unknown area based on the first fitting information and the second fitting information; The image processing method according to claim 7.

9. After determining the target road boundary, the image processing method includes: determining a driving route for the vehicle based on the target road boundary and / or the road information; and controlling the travel of the vehicle based on the travel route. The image processing method according to any one of claims 5 to 8.

10. The step of determining a travel route of the vehicle based on the road information includes: determining a turning direction and a turning position of the vehicle based on road surface signals and driving direction signs in the road information; determining a driving route for the vehicle based on the turning direction and the turning position; The image processing method according to claim 9.

11. The step of controlling the travel of the vehicle based on the travel route includes: updating the driving route based on obstacle information in the road information to obtain an updated route; and controlling travel of the vehicle based on the updated route. The image processing method according to claim 9.

12. The step of determining a driving route of the vehicle based on the target road boundary includes: updating map data of the vehicle's location based on the target road boundary to obtain an updated map; determining a route for the vehicle based on the updated map; The image processing method according to claim 9.

13. After determining the target road boundary, the image processing method includes: The method further includes controlling the vehicle based on a relationship between the target road boundary and a running state of the vehicle.

9. The image processing method according to claim 7 or 8.

14. The relationship between the target road boundary and the running state of the vehicle is as follows: The distance between the overlapping area in which the target road boundary is located and the road intersection ahead of the vehicle is less than a second predetermined distance; the distance between the overlapping area and the location of the vehicle is less than a third predetermined distance; The angle between the vehicle's traveling direction and the target road boundary is less than a predetermined angle; and the target road boundary is adjacent to a roadway on which the vehicle is located; The image processing method according to claim 13.

15. The step of controlling the vehicle includes: controlling the vehicle to enter a braking state from a traveling state or to move away from the target road boundary. The image processing method according to claim 13.

16. After determining the target road boundary, the image processing method includes: setting a region of interest based on the target road boundary and acquiring an image corresponding to the region of interest based on a first resolution, the road image being acquired according to a second resolution, the second resolution being smaller than the first resolution; and / or acquiring an image corresponding to the region of interest based on a first frame rate, the road image being acquired based on a second frame rate, the second frame rate being smaller than the first frame rate; The image processing method according to any one of claims 1 to 4.

17. After determining the target road boundary, the image processing method includes: collecting road environment information around the target road boundary; generating notification information based on the road environment information; and transmitting the notification information to a vehicle behind the vehicle, the vehicle being located on the same roadway as the vehicle and traveling in the same direction. The image processing method according to any one of claims 1 to 4.

18. An image processing device, an image capture portion configured to capture road images collected by an image capture device mounted on the vehicle; a road boundary detection portion configured to detect a plurality of road boundaries in the road image based on the road image; a target road boundary determining portion configured to determine a target road boundary that is dangerous for the vehicle among the plurality of road boundaries.

19. A computer storage medium having stored thereon computer executable instructions which, when executed, are capable of implementing the image processing method of any one of claims 1 to 17.

20. A computing device comprising: a memory having computer-executable instructions stored therein; and a processor capable of implementing the image processing method according to any one of claims 1 to 17, when executing the computer-executable instructions on the memory.

21. A computer program product comprising a computer program or instructions which, when executed on an electronic device, causes said electronic device to carry out the image processing method according to any one of claims 1 to 17.

Images