Image processing methods, apparatus, devices, and storage media

Abstract

The image processing method, device, equipment, and storage medium acquire a road image collected by an image collecting device mounted on a vehicle (S101), detect a plurality of road boundaries in the road image based on the road image (S102), and determine a target road boundary that is dangerous to the vehicle from among the plurality of road boundaries (S103). In this way, the traveling of the vehicle can be more accurately controlled based on the target road boundary.

Description

Technical Field

[0007] , , ,

[0005] , , , ,

[0006] ,

[0001] (Cross - reference to related applications) This application claims the priority of a Chinese patent application with an application number of 202210303731.8 and an invention title of "Image Processing Method, Apparatus, Device and Storage Medium", which was filed with the Chinese Patent Office on March 24, 2022, and all of its content is incorporated herein by reference.

[0002] This application relates to the technical field of autonomous driving, and particularly, but not limited to, an image processing method, apparatus, device and storage medium.

Background Art

[0003] In recent years, the field of autonomous driving mainly based on deep learning has made great progress, including the fields of image segmentation and target detection. Autonomous driving is an overall system, and the output of the perception module provides services to subsequent modules. In related technologies, there is often a loss of correlation between the perception module and subsequent control signals, which affects the control accuracy and reliability of the system.

Summary of the Invention

[0004] Embodiments of the present disclosure provide a technical solution for image processing.

[0005] The technical solution of the embodiments of the present disclosure is realized as follows.

[0006] Embodiments of the present disclosure provide an image processing method, the method includes: obtaining a road image collected by an image collection device mounted on a vehicle; detecting a plurality of road boundaries in the road image based on the road image; and determining a target road boundary dangerous to the vehicle among the plurality of road boundaries.

[0007] In some embodiments, the step of detecting multiple road boundaries within a road image based on the road image includes detecting the road image and determining multiple road boundaries associated with the vehicle. In this way, multiple road boundaries within a road image can be recognized quickly and accurately.

[0008] In some embodiments, the step of detecting multiple road boundaries within a road image based on the road image includes the steps of detecting the road image to obtain multiple roadways within the road image, and connecting the ends of each roadway within the multiple roadways to obtain the multiple road boundaries. In this way, by connecting the end edges of each roadway, multiple road boundaries within a road image can be recognized more simply.

[0009] In some embodiments, the step of detecting multiple road boundaries within a road image based on the road image includes the steps of performing semantic segmentation on the road image to obtain a drivable area within the road image, and determining the multiple road boundaries based on the contour lines of the drivable area. In this way, by dividing the drivable area of ​​the road within the road image, multiple road boundaries within the road image can be accurately recognized.

[0010] In some embodiments, the step of determining a target road boundary that is dangerous to the vehicle among the plurality of road boundaries includes at least one of the following steps: determining the road boundary adjacent to the roadway on which the vehicle is located as the target road boundary among the plurality of road boundaries; determining the road boundary whose distance from the vehicle is less than a first predetermined distance as the target road boundary among the plurality of road boundaries; determining the road boundary whose road space between the vehicle and the road boundary is less than a predetermined space as the target road boundary among the plurality of road boundaries; and determining a target road boundary that is dangerous to the vehicle among the plurality of road boundaries based on road information determined by the road image, wherein the road information includes at least one of road surface signals, lanes, stop line areas, direction signs, and obstacle information in the road image. In this way, by determining the target road boundary in multiple ways and comprehensively considering multiple pieces of information on the road surface, it is possible to accurately detect a target road boundary that is dangerous to the vehicle.

[0011] In some embodiments, the step of determining a target road boundary that is dangerous to the vehicle among the plurality of road boundaries, based on road information determined by the road image, includes the steps of determining the actual road area and the unknown area that is not recognizable to the vehicle, based on the road information; determining a road boundary that is not visible to the vehicle, based on the actual road area and the unknown area; and determining the road boundary that is not visible to the vehicle as the target road boundary. In this way, by comparing the actual road area and the unknown area of ​​the road related to the vehicle, the target road boundary that is dangerous to the vehicle can be accurately recognized.

[0012] In some embodiments, the step of determining road boundaries invisible to the vehicle based on the actual road area and the unknown area includes the steps of: converting the collection viewpoints of the actual road area and the unknown area to an overhead viewpoint to obtain the converted actual road area and the converted unknown area; determining the overlapping area between the converted actual road area and the converted unknown area; and determining the road boundaries within the overlapping area as road boundaries invisible to the vehicle. By analyzing the overlapping area between the converted actual road area and the converted unknown area in the overhead viewpoint, road boundaries invisible to the vehicle can be effectively recognized with fewer network resources, facilitating the planning of the travel routes of subsequent vehicles.

[0013] In some embodiments, the step of determining the overlapping area between the converted actual road area and the converted unknown area includes the steps of: obtaining first fitting information by fitting lanes, stop line areas, and direction signs within the converted actual road area; obtaining second fitting information by fitting lanes, stop line areas, and direction signs within the converted unknown area; and determining the overlapping area between the converted actual road area and the converted unknown area based on the first and second fitting information. In this way, by comprehensively considering multiple pieces of information on the road, the target road boundary within the overlapping area can be determined more accurately.

[0014] In some embodiments, after determining the target road boundary, the image processing method further includes the steps of determining the vehicle's travel path based on the target road boundary and / or the road information, and controlling the vehicle's travel based on the travel path. In this way, after recognizing the target road boundary, a more accurate travel path can be generated in conjunction with rich road information, enabling precise control of the vehicle.

[0015] In some embodiments, the step of determining the vehicle's travel path based on the road information includes the steps of determining the direction and location of the vehicle's turn based on road signals and direction markers in the road information, and determining the vehicle's travel path based on the direction and location of the turn. In this way, by following the road signals in the road information, the direction and location of the vehicle's turn at a future time can be accurately predicted, and the vehicle's turn can be accurately controlled.

[0016] In some embodiments, the step of controlling the vehicle's movement based on the travel path includes the steps of updating the travel path based on obstacle information in the road information to obtain an updated path, and controlling the vehicle's movement based on the updated path. By integrating the location information of obstacles in the road information and updating the travel path in this way, more information can be provided to the autonomous vehicle when making decisions.

[0017] In some embodiments, the step of determining the vehicle's travel path based on the target road boundary includes the steps of updating map data of the vehicle's location based on the target road boundary to obtain an updated map, and determining the vehicle's travel path based on the updated map. In this way, a travel path is generated that controls the vehicle's movement according to the updated map, improving the safety of the travel path.

[0018] In some embodiments, after determining the target road boundary, the image processing method further includes a step of controlling the vehicle based on the relationship between the target road boundary and the vehicle's driving state. In this way, after recognizing the target road boundary, the relationship between the target road boundary and the driving state can be analyzed to effectively control the vehicle for safe driving.

[0019] In some embodiments, the relationship between the target road boundary and the vehicle's driving state includes at least one of the following: the distance between the overlapping area where the target road boundary is located and the road intersection in front of the vehicle is less than a second predetermined distance; the distance between the overlapping area and the vehicle's location is less than a third predetermined distance; the angle between the vehicle's driving direction and the target road boundary is less than a predetermined angle; and the target road boundary is adjacent to the roadway where the vehicle is located.

[0020] In some embodiments, the steps of controlling the vehicle include controlling the vehicle to enter a braking state from a driving state, or controlling the vehicle to move away from the target road boundary. In this way, if the target road boundary affects the vehicle's driving, the vehicle's driving safety can be further improved by controlling the vehicle to enter a braking state or to move away from the target road boundary.

[0021] In some embodiments, after determining the target road boundary, the image processing method further includes the steps of setting a region of interest based on the target road boundary, acquiring an image corresponding to the region of interest based on a first resolution, wherein the road image is acquired according to a second resolution, the second resolution being smaller than the first resolution, and / or acquiring an image corresponding to the region of interest based on a first frame rate, wherein the road image is acquired according to a second frame rate, the second frame rate being smaller than the first frame rate. In this way, object recognition for subsequent images corresponding to the region of interest is facilitated.

[0022] In some embodiments, after determining the target road boundary, the image processing method further includes the steps of: collecting road environment information around the target road boundary; generating notification information based on the road environment information; and transmitting the notification information to a vehicle following the vehicle, wherein the following vehicle is located on the same roadway as the vehicle and is traveling in the same direction. In this way, by informing the following vehicle in a timely manner that a target road boundary exists ahead, the following vehicle can adjust its travel path in a timely manner.

[0023] Embodiments of the present disclosure provide an image processing apparatus comprising: an image acquisition portion configured to acquire a road image collected by an image acquisition device mounted on a vehicle; a road boundary detection portion configured to detect a plurality of road boundaries within the road image based on the road image; and a target road boundary determination portion configured to determine a target road boundary that is dangerous to the vehicle among the plurality of road boundaries.

[0024] In response to this, embodiments of the present disclosure provide a computer storage medium in which computer executable instructions are stored, and after the computer executable instructions are executed, the steps of the above-described image processing method can be realized.

[0025] Embodiments of the present disclosure provide a computer device comprising a memory storing computer executable instructions and a processor capable of implementing the steps of the above-described image processing method when executing computer executable instructions in the memory.

[0026] Embodiments of the present disclosure further provide a computer program product including a computer program or instruction, which, when executed on an electronic device, causes the electronic device to perform the steps in any possible embodiment of the first embodiment described above.

[0027] Embodiments of the present disclosure provide an image processing method, apparatus, device, and storage medium. By detecting an acquired road image, recognizing a plurality of road boundaries in the road image, and selecting a target road boundary that is dangerous for a vehicle from the plurality of road boundaries, the driving of the vehicle can be controlled more accurately based on the target road boundary.

[0028] To make the above objects, features, and advantages of the embodiments of the present disclosure easier to understand, preferred embodiments will be described in detail below in conjunction with the drawings.

Brief Description of the Drawings

[0029] [Figure 1A] It is a schematic diagram of a system architecture to which the image processing method of the embodiments of the present disclosure is applicable. [Figure 1B] It is a flowchart of the implementation of the image processing method according to the embodiments of the present disclosure. [Figure 2] It is another flowchart of the implementation of the image processing method according to the embodiments of the present disclosure. [Figure 3] It is yet another flowchart of the implementation of the image processing method according to the embodiments of the present disclosure. [Figure 4] It is a network structure diagram of the image processing method according to the embodiments of the present disclosure. [Figure 5A] It is a schematic diagram of an application scenario of the image processing method according to the embodiments of the present disclosure. [Figure 5B] It is a schematic diagram of another application scenario of the image processing method according to the embodiments of the present disclosure. [Figure 6A] It is a schematic diagram of an application scenario of the image processing method according to the embodiments of the present disclosure. [Figure 6B] It is a schematic diagram of another application scenario of the image processing method according to the embodiments of the present disclosure. [Figure 7] It is a schematic diagram of yet another application scenario of the image processing method according to the embodiments of the present disclosure. [Figure 8] It is a schematic diagram of yet another application scenario of the image processing method according to the embodiments of the present disclosure. [Figure 9]This is a schematic diagram of another application scenario of the image processing method according to the embodiments of this disclosure. [Figure 10] This is a schematic diagram showing the configuration of an image processing apparatus according to an embodiment of the present disclosure. [Figure 11] This is a schematic diagram showing the configuration of a computer device according to an embodiment of the present disclosure. [Modes for carrying out the invention]

[0030] To more clearly illustrate the technical solutions of the embodiments of this disclosure, the drawings necessary for describing the embodiments are briefly described above. These drawings are incorporated herein and constitute part of this specification, and these drawings illustrate embodiments consistent with this disclosure and are used together with the specification to illustrate the technical solutions of this disclosure. It should be understood that these drawings only illustrate a portion of the embodiments of this disclosure and should not be considered to limit the scope of protection, and a person skilled in the art can obtain other relevant drawings based on these drawings without any creative effort.

[0031] To further clarify the purpose, technical solutions, and advantages of the embodiments of this disclosure, specific technical solutions of the present invention will be described in more detail below with reference to the drawings of the embodiments of this disclosure. The following embodiments are for illustrative purposes only and are not intended to limit the scope of this disclosure.

[0032] In the following, the phrase "several embodiments" describes a subset of all possible embodiments, but understandably, "several embodiments" may be the same subset or a different subset of all possible embodiments, and can be combined with each other without contradiction.

[0033] In the following, terms such as "1st / 2nd / 3rd" do not limit a specific order, but rather distinguish similar objects. Understandably, "1st / 2nd / 3rd" can be used to change a specific order or sequence where appropriate, so the embodiments of this disclosure described in some embodiments may be carried out in an order other than those illustrated or described in some embodiments.

[0034] Unless otherwise specified, all technical and scientific terms used in this disclosure have the same meaning as those commonly understood by those skilled in the art. The terms used in this disclosure are adopted solely to illustrate the embodiments of this disclosure and are not intended to limit this disclosure.

[0035] Before describing the embodiments of this disclosure in detail, the nouns and terms used in the embodiments of this disclosure will be explained below.

[0036] 1) Convolutional Neural Networks (CNNs): A type of feedforward neural network (FNN) with a deep structure that includes convolutional computations. It possesses representation learning capabilities and can classify input information translationally invariantly according to its hierarchical structure.

[0037] 2) Self-vehicle (ego vehicle): This is a vehicle equipped with sensors to perceive the surrounding environment. The vehicle coordinate system is fixed to the self-vehicle, where the x-axis points in the direction the vehicle is moving, the y-axis points to the left in the direction the vehicle is moving, and the z-axis points upward perpendicular to the ground, fitting a right-handed coordinate system. The origin of the coordinate system is located on the ground below the midpoint of the rear axle.

[0038] The following describes examples of applying the image processing method according to the embodiments of this disclosure to electronic devices. The electronic devices according to the embodiments of this disclosure may be in-vehicle devices, cloud platforms, or other computer devices. For example, in-vehicle devices may be thin clients, fat clients, microprocessor-based systems, or small computer systems installed in a vehicle, and cloud platforms may be distributed cloud computing technology environments including small computer systems or large computer systems. Next, examples of applications when the electronic devices are implemented as terminals or servers will be described.

[0039] Figure 1A is a schematic diagram of a system architecture to which the image processing method according to an embodiment of the present disclosure is applied. As shown in Figure 1A, the system architecture includes an image acquisition device 11, a network 12, and an in-vehicle control terminal 13. To realize one exemplary application, the image acquisition device 11 and the in-vehicle control terminal 13 establish a communication connection via the network 12. First, the image acquisition device 11 reports the acquired road image to the in-vehicle control terminal 13 via the network 12. The in-vehicle control terminal 13 then performs road boundary detection on the road image, thereby detecting a target road boundary that is dangerous to the vehicle.

[0040] For example, the image acquisition device 11 may include a visual processing device having visual information processing capabilities. The network 12 can employ a wired or wireless connection method. Here, if the image acquisition device 11 is a visual processing device, the in-vehicle control terminal 13 can communicate with the visual processing device by a wired connection method, such as data communication via a bus.

[0041] Alternatively, in some scenarios, the image acquisition device 11 may be a visual processing device with a video acquisition module, or a host computer equipped with a camera. In this case, the image processing method of the embodiment of this disclosure may be performed by the image acquisition device 11, and the system architecture described above may not include the network 12 and the in-vehicle control terminal 13.

[0042] To further clarify the purpose, technical solutions, and advantages of the embodiments of this disclosure, specific technical solutions of the present invention will be described in more detail below with reference to the drawings of the embodiments of this disclosure. The following embodiments are for illustrative purposes only and are not intended to limit the scope of this disclosure.

[0043] In the following, the phrase "several embodiments" describes a subset of all possible embodiments, but understandably, "several embodiments" may be the same subset or a different subset of all possible embodiments, and can be combined with each other without contradiction.

[0044] The image processing method described above can be applied to computer equipment, and the functions realized by the method can be realized by a processor in the computer equipment calling program code, and of course the program code may be stored in a computer storage medium, and it is understood that the computer equipment includes at least a processor and a storage medium.

[0045] Figure 1B is a flowchart illustrating the implementation of an image processing method according to an embodiment of this disclosure, and will be explained in conjunction with the steps shown in Figure 1B, as shown in Figure 1B.

[0046] In step S101, road images are acquired by an image acquisition device mounted on the vehicle.

[0047] In some embodiments, the road image may be an image collected on any road, and may contain complex screen content or simple screen content. For example, it may be a road image collected by an in-vehicle device on a vehicle.

[0048] In some embodiments, the image acquisition device may be mounted on the vehicle's onboard equipment or may be independent of the onboard equipment. The onboard equipment can communicate with the vehicle's sensors, positioning devices, etc., and through this communication connection, the onboard equipment can acquire data collected by the vehicle's sensors, geolocation information reported by the positioning device, etc. Exemplarily, the vehicle's sensors may be at least one of devices such as millimeter-wave radar, laser radar, and cameras, and the positioning device may be a device that provides positioning services based on at least one of the Global Positioning System (GPS), Beidou satellite navigation system, or Galileo satellite navigation system.

[0049] In some embodiments, the in-vehicle equipment may be an Advanced Driving Assistant System (ADAS), which is installed in the vehicle and can acquire real-time vehicle location information from the vehicle's positioning device and / or can acquire image data, radar data, etc., representing information about the vehicle's surrounding environment from the vehicle's sensors. Here, optionally, the ADAS can transmit vehicle driving data, including the vehicle's real-time location information, to a cloud platform. In this way, the cloud platform can receive the vehicle's real-time location information and / or image data, radar data, etc., representing information about the vehicle's surrounding environment.

[0050] Road images are acquired by an image acquisition device (i.e., a sensor such as a camera) installed on the vehicle, and the image acquisition device collects images of the area around the vehicle in real time as the vehicle moves to obtain the road images. In some possible implementations, a camera mounted on the vehicle may collect images of the road the vehicle is traveling on and the surrounding environment while the vehicle is in motion to obtain the road images. By detecting the road images in this way, the multiple road boundaries can be recognized.

[0051] In step S102, multiple road boundaries within the road image are detected based on the road image.

[0052] In some embodiments, a detection network is employed to detect multiple road boundaries within the road image. The vehicles in the road image may be any vehicles traveling on the road.

[0053] In several possible implementations, edge detection can be performed on a road image to detect the edges of roads within the image and obtain multiple road boundaries. For example, multiple road boundaries can be obtained by detecting multiple road lanes within a road image and connecting the end edges of the lanes, or by inputting the road image into a trained edge detection network to output multiple road boundaries within the road image.

[0054] In step S103, among the plurality of road boundaries, a target road boundary that is dangerous to the vehicle is determined.

[0055] In some embodiments, the target road boundary may be a road boundary that is not visible to the vehicle, a road boundary that is recognizable to the vehicle but is at a short distance from the vehicle, or a road boundary that is dangerous to the vehicle, determined by analyzing road information in the road image. For example, a road boundary that is obscured by an obstacle, or a road boundary that is too far from the vehicle, or a road boundary that is in the vehicle's blind spot, or a road boundary that is too close to the vehicle to allow it to travel normally.

[0056] In several possible implementations, by detecting the positional relationship between an obstacle and the road boundary, it is possible to determine whether the obstacle obstructs the road boundary, thereby determining whether the road boundary is not visible, i.e., whether it is a target road boundary. By detecting the distance between the vehicle and the road boundary, it is possible to determine whether the road boundary is too far from the vehicle, thereby determining whether the road boundary is a target road boundary. By detecting the positional relationship between the vehicle and the road boundary, it is possible to determine whether the road boundary is in the vehicle's blind spot, thereby determining whether the road boundary is a target road boundary. In this way, by recognizing the target road boundary that is dangerous to the vehicle among the multiple detected road boundaries, the subsequent driving route can be planned more accurately.

[0057] In the embodiments of this disclosure, the vehicle's movement can be controlled more accurately by detecting the acquired road image, recognizing multiple road boundaries within the road image, and selecting a target road boundary that is dangerous to the vehicle from among the multiple road boundaries.

[0058] In some embodiments, the acquired road image can be detected to recognize multiple road boundaries related to a vehicle within the road image. That is, step S102 described above can be implemented by one of the following methods.

[0059] Method 1: The road image is detected and multiple road boundaries related to the vehicle are determined.

[0060] In the above-described method 1, the road image is detected using the first network to determine a plurality of road boundaries associated with the vehicle. The plurality of road boundaries associated with the vehicle may be the road boundaries of each carriageway on the road where the vehicle is located or the road boundaries of multiple carriageways on the road where the vehicle is located. In specific implementations, since a vehicle can reach any carriageway on the road where the vehicle is located by changing lanes or making a U-turn, the road boundaries of any carriageway on the road where the vehicle is located can be considered as road boundaries associated with the vehicle. For example, if the road where the vehicle is located includes four carriageways, the plurality of road boundaries associated with the vehicle include the road boundaries of each of the four carriageways. The first network may be a deep neural network (DNN), for example, any network capable of image detection. In some possible implementations, the first network may be a residual network, a VGG (Visual Geometry Group) network, etc. The first network is a network trained to detect road boundaries. By inputting a road image into the first network, it performs feature extraction on the road image and recognizes multiple road boundaries related to the vehicle based on the extracted image features. In this way, multiple road boundaries within a road image can be recognized quickly and accurately.

[0061] Furthermore, in the specific implementation process, the road boundary related to a vehicle can be determined by determining the overlapping portion between the drivable area and the road boundary in the road image; that is, the overlapping portion between the detected drivable area and the detected road boundary is determined as the road boundary related to a vehicle. Here, the drivable area can be detected using a DNN, but is not limited to this.

[0062] Method 2: Determine multiple road boundaries by detecting roadways within the road image. This step can be achieved by the following steps.

[0063] In the first step, the road image is detected to obtain multiple roadways within the road image.

[0064] In some embodiments, roadways are detected in the road image to obtain multiple roadways. A second network is used to detect roadways in the road image to obtain multiple roadways. The second network may be the same as or different from the first network. The second network detects multiple roadways in the road image, i.e., multiple lanes in the road image. The second network processes the road image to obtain the lanes in the road image, i.e., the multiple roadways. In other embodiments, other image detection methods may be employed to detect multiple roadways in the road image. In some possible implementations, first, the road image is subjected to grayscale processing, roadway edges in the grayscale-processed road image are detected, edge detection is performed using, for example, an edge detection operator, and then the processed image is binarized to obtain the lanes in the road image.

[0065] In the second step, the ends of each of the roadways within the plurality of roadways are connected to obtain the plurality of road boundaries.

[0066] In some embodiments, multiple road boundaries related to a vehicle are obtained by connecting the end edges of the lanes of each roadway. For example, by connecting the end edges under the vehicle on both the left and right sides, a road boundary perpendicular to the road on which the vehicle is located can be obtained. In this way, by connecting the end edges of each roadway, multiple road boundaries within a road image can be recognized more concisely.

[0067] Method 3: Semantic segmentation is performed on the road image to determine the drivable area of ​​the road where the vehicle is located. This step can be achieved by the following steps.

[0068] In the first step, semantic segmentation is performed on the road image to obtain the drivable area within the road image.

[0069] In some embodiments, a third network is used to perform semantic segmentation on the road image to obtain the drivable area of ​​the road within the road image. The third network may be a neural network for performing semantic segmentation, such as a fully convolutional neural network or a Mask Region Convolutional Neural Network (Mask R-CNN). The third network detects the drivable area within the road image, which may also be called the passable area and represents the area where a vehicle can travel. In addition to the current vehicle, the road image usually includes other vehicles, pedestrians, trees, road edges, etc., and the areas where other vehicles, pedestrians, trees, road edges, etc. are located are currently areas where vehicles cannot travel. Therefore, the third network performs semantic segmentation on the road image and removes areas where, for example, other vehicles, pedestrians, trees, road edges, etc. are located within the road image to obtain the drivable area for vehicles.

[0070] In the second step, the plurality of road boundaries are determined based on the contour lines of the drivable area.

[0071] In some embodiments, the road boundary of the road where the drivable area is located is obtained by recognizing the contour line of the drivable area. For example, the contour line of the drivable area is used as the road boundary of the road where the drivable area is located. In this way, by dividing the drivable area of ​​a road within a road image, multiple road boundaries within the road image can be accurately recognized.

[0072] In some embodiments, by recognizing road information of the vehicle-related road or analyzing the relationship between the road boundary and the vehicle, it is possible to accurately select a target road boundary that is dangerous to the vehicle from among multiple road boundaries. That is, step S103 described above can be implemented by one of the following methods.

[0073] Method 1: Among the multiple road boundaries, the road boundary adjacent to the roadway where the vehicle is located is determined as the target road boundary.

[0074] In Method 1, the road boundary adjacent to the roadway on which the vehicle is located may be the road boundary adjacent to the roadway on which the vehicle is located, and since the road boundary adjacent to the roadway on which the vehicle is located is in the vehicle's blind spot, the road boundary is not visible to the vehicle, that is, the road boundary is the target road boundary.

[0075] Method 2: Among the multiple road boundaries, the road boundary in which the distance to the vehicle is less than the first predetermined distance is determined as the target road boundary.

[0076] In method two, the first predetermined distance can be set by measuring the blind spot range of the vehicle. For example, the first predetermined distance can be set to be less than or equal to the maximum diameter of the blind spot range. The distance between the road boundary and the vehicle is the distance between each point on the road boundary and the vehicle. If the distance from a point to the vehicle is less than the first predetermined distance, it means that the point is not visible to the vehicle. In this way, by analyzing whether the distance between a plurality of points and the vehicle is less than the first predetermined distance, it is possible to determine whether the road boundary consisting of the plurality of points is the target road boundary.

[0077] In some possible implementations, for any road boundary, sampling is performed at points on the road boundary according to a fixed length interval, and it can be determined whether the road boundary is a target road boundary by determining whether the distance between the sampling point and the vehicle is less than a first predetermined distance. For example, the first sampling point where the distance from the vehicle is less than the first predetermined distance is taken as the starting point, and the last sampling point where the distance from the vehicle is less than the first predetermined distance is taken as the ending point. In this way, the road boundary between the starting point and the ending point becomes the target road boundary.

[0078] Method 3: Among the multiple road boundaries, the road boundary in which the road space between the vehicle and the boundary is less than a predetermined space is determined as the target road boundary.

[0079] In method three, the road space between the road boundary and the vehicle may be the width of the road area from the vehicle to the road boundary. The predetermined space may be determined based on the width of the carriageway and the width of a vehicle that can travel on the carriageway. For example, the predetermined space may be set to be greater than the width of a vehicle that can travel on the carriageway and smaller than the width of the carriageway. If the width between the road boundary and the vehicle is less than the predetermined space, an oncoming vehicle cannot travel between the road boundary and the vehicle, meaning that the space between the road boundary and the vehicle is small, and furthermore, that the road boundary may be dangerous to the normal driving of a vehicle. Therefore, such a road boundary is designated as the target road boundary. If the width between the road boundary and the vehicle is greater than or equal to the predetermined space, an oncoming vehicle can travel between the road boundary and the vehicle, meaning that there is sufficient space between the road boundary and the vehicle, and furthermore, that the road boundary is not dangerous to the normal driving of a vehicle. Therefore, such a road boundary is not designated as the target road boundary.

[0080] Method 4: Among the multiple road boundaries, a target road boundary that is dangerous to the vehicle is determined based on the road information determined by the road image.

[0081] In method four, by performing image detection on a road image, road information of the road where the vehicle is located within the road image can be recognized, and based on the road information, target road boundaries that are dangerous to the vehicle within multiple road boundaries can be recognized. The road information of the vehicle-related road is used to represent multiple detectable pieces of information on the road, for example, the road information includes at least one of the following in the road image: road surface signals, lanes, stop line areas, direction signs, and obstacle information. The direction signs may also be located at the road's direction change edge. By comprehensively considering multiple pieces of road surface information, target road boundaries that are dangerous to the vehicle can be accurately detected.

[0082] In some possible implementations, the road information can be obtained by the following steps.

[0083] In the first step, the road surface signals related to the vehicle in the road image are determined.

[0084] In some embodiments, a detector within a deep neural network is used to extract image features from road images, and based on the extracted image features, road surface signals in the road images can be detected. The detected road surface signals include multiple types of road surface arrow information, such as straight, left turn, right turn, straight left turn, straight right turn, U-turn, left turn, straight right turn, etc.

[0085] In the second step, the road lanes are divided to obtain multiple lanes.

[0086] In some embodiments, semantic segmentation branching within a deep neural network is employed to divide the road lanes, resulting in the output of multiple lanes with class labels. Here, lanes of different classes can be represented using different class labels; for example, the left lane is displayed as class 1, the right lane as class 2, and the background as class 0.

[0087] In the third step, the stop line of the road is detected to obtain the stop line area.

[0088] In some embodiments, a stop line segmentation branch within the deep neural network is employed to perform two-class segmentation on the road's stop line. The resulting segmentation is such that the stop line region is represented by 1 and the background region by 0, thereby achieving segmentation for the stop line.

[0089] In the fourth step, the direction of travel signs at the intersection of the road are recognized, and a multi-class direction of travel sign is obtained.

[0090] In some embodiments, the intersection direction output branching within the deep neural network is used to perform semantic segmentation on road intersection direction signs to obtain multi-class direction signs. For example, three classes are defined for direction signs from left to right, with the left-direction sign being class 1, the forward-direction sign being class 2, and the right-direction sign being class 3, while the background class is 0.

[0091] In the fifth step, obstacles on the road are detected, and object information of the obstacles is obtained.

[0092] In some embodiments, an obstacle detection branch within the deep neural network is employed to detect obstacles on the road, with the obstacles used as the foreground for target detection and the non-obstacle objects as the background. Here, the obstacles may be any object other than a vehicle or pedestrians. The obstacle information includes the position and size information of the obstacles, etc.

[0093] Steps 1 through 5 described above may be executed simultaneously by different branches within the same network.

[0094] In the sixth step, at least one of the road signals, the multiple lanes, the stop line area, the multi-class direction of travel signs, and the object information is determined to be the road information.

[0095] In the sixth step, the road information obtained in steps 1 through 5, including road sign information, lane information, and intersection direction information, is used as road information. In this way, the tasks of road signal detection, lane detection, and intersection direction detection are merged into the same deep learning network for collaborative learning, and the output road information is acquired to enrich the content of the road information. This provides the vehicle with rich information, enabling it to generate effective control signals.

[0096] In some possible implementations, the road information can be analyzed to recognize the actual road area and unknown areas, thereby enabling the detection of road boundaries that are not visible from the vehicle. As shown in Figure 2, Figure 2 is a flowchart of another implementation of the image processing method according to an embodiment of the present disclosure, and will be described below in conjunction with the steps shown in Figure 2.

[0097] In step S201, the actual road area and the unknown area that is not recognizable by the vehicle are determined based on the road information.

[0098] In some embodiments, the actual road area of ​​the road can be obtained by analyzing road surface signals, the multiple lanes, the stop line area, the multi-class direction of travel signs, and the object information within the road information. For example, if there are no objects or pedestrians on the road, the road surface area of ​​the road is considered the actual road area. An unknown area that is unrecognizable by a vehicle may be an area on the road or an area outside the road. For example, an unknown area may be an area in the vehicle's blind spot, an area obscured by an obstacle, or an area that cannot be recognized due to distance.

[0099] For example, in the intersection scene shown in Figures 5A and 5B, there is a building at the southwest corner of the intersection (the top, bottom, left, and right of the image correspond to north, south, west, and east, respectively). Typically, this building obstructs the view of a vehicle traveling from south to north. As shown in Figure 5B, the driver or sensors inside the vehicle cannot obtain information about part of the area obscured by the building. This area can be called an unknown area and is shown as area 522 in Figure 5B. A road boundary that cannot be detected by the driver or sensors inside the vehicle due to obstruction by a building or other reasons (e.g., being too far away) is called an invisible road boundary.

[0100] In step S202, the road boundary that is not visible from the vehicle is determined based on the actual road area and the unknown area.

[0101] In some embodiments, by converting the actual road area and unknown area from the current collection viewpoint to the actual road area and unknown area from an overhead viewpoint, it is possible to determine the overlapping road boundary between the two areas in the overhead viewpoint, and this road boundary is a road boundary that is not visible from the vehicle.

[0102] In step S203, the road boundary that is not visible from the vehicle is determined as the target road boundary.

[0103] In some embodiments, a road boundary invisible to the vehicle is a target road boundary that is unrecognizable to the vehicle among multiple road boundaries. By comparing the actual road area and the unknown area of ​​the vehicle-related road in this way, the target road boundary can be accurately recognized.

[0104] Steps S201 to S203 described above provide a method for determining a target road boundary, in which a road boundary not visible to the vehicle is designated as a target road boundary that is dangerous to the vehicle. In this way, potential hazards to the vehicle can be effectively determined, and the driving safety of the autonomous vehicle can be improved.

[0105] In other implementations, dangerous target road boundaries for vehicles can also be determined by analyzing road boundaries that are recognizable to the vehicle.

[0106] In some embodiments, the road boundary that is not visible from the vehicle can be determined by converting both the actual road area and the unknown area into an overhead view, and by analyzing the road boundary that overlaps between the two areas in the overhead view. That is, step S202 above can be realized by the following steps S221 to S223 (not shown).

[0107] In step S221, the collection viewpoints for the actual road area and the unknown area are converted to an overhead viewpoint, thereby obtaining the converted actual road area and the converted unknown area.

[0108] In some embodiments, homography is employed to convert the actual road area and the unknown area into an overhead view, thereby obtaining the converted actual road area and the converted unknown area in the overhead view. In this way, road information within the actual road area, such as road signals, the multiple lanes, the stop line area, the multi-class directional signs, and object information within the actual road area, is converted into road signals, the multiple lanes, the stop line area, the multi-class directional signs, and object information in the overhead view. For example, the position of object information in the actual road area is converted to its position in the converted road area in the overhead view. Similarly, road information within the unknown area is simultaneously converted into road information in the overhead view.

[0109] In step S222, the area overlapping between the converted actual road area and the converted unknown area is determined.

[0110] In some embodiments, road information within the converted actual road area in an overhead view can be fitted, road information within the converted unknown area can be fitted, and based on the fitted information, the overlapping portion between the two areas, i.e., the overlapping area, can be determined.

[0111] In step S223, the road boundary within the overlapping area is determined to be a road boundary that is not visible from the vehicle.

[0112] In some embodiments, the unknown region is an area unrecognizable to the vehicle, and therefore the converted unknown region remains unrecognizable to the vehicle. Based on this, the overlapping area between the converted actual road region and the converted unknown region is an actual road region unrecognizable to the vehicle, and the road boundary within this region is also clearly unrecognizable to the vehicle, i.e., it is the target road boundary.

[0113] In the embodiments of this disclosure, by analyzing the overlapping area between the transformed actual road area and the transformed unknown area in an overhead view, road boundaries invisible to vehicles can be effectively recognized with fewer network resources, facilitating the planning of the travel route for subsequent vehicles.

[0114] In some embodiments, road information within the transformed actual road area in an overhead view and road information within the transformed unknown area in an overhead view are fitted together, and the overlapping road boundary between the two areas can be obtained as a result of the fitting. That is, step S222 described above can be realized by the following steps.

[0115] In the first step, the lane markings, stop line markings, and direction-of-travel signs within the converted actual road area are fitted to obtain first fitting information.

[0116] In some embodiments, first fitting information is obtained by fitting multiple lanes, stop line areas, and multi-class directional signs within a transformed actual road area using a matrix transformation method. The first fitting information includes fitted lanes, stop line areas, and multi-class directional signs within the transformed actual road area.

[0117] In the second step, the lanes, stop line areas, and direction of travel signs within the converted unknown area are fitted to obtain second fitting information.

[0118] In some embodiments, a second fitting information is obtained by fitting multiple lanes, stop line areas, and multi-class directional signs within a transformed unknown region using a matrix transformation method. The second fitting information includes the fitted lanes, stop line areas, and multi-class directional signs within the transformed unknown region.

[0119] In the third step, the overlapping area between the converted actual road area and the converted unknown area is determined based on the first fitting information and the second fitting information.

[0120] In some embodiments, the overlapping lanes, stop lines, and direction signs between two regions can be determined according to the fitted lanes, stop lines, and direction signs within the converted actual road region and the fitted lanes, stop lines, and direction signs within the converted unknown region, and furthermore, the overlapping region between the two regions can be obtained.

[0121] In the embodiments of this disclosure, fitting results for each road information can be obtained by fitting different road information from an overhead perspective. In this way, by comprehensively considering multiple pieces of information on the road, the target road boundary within the overlapping area can be determined more accurately.

[0122] In some embodiments, after recognizing a target road boundary that is dangerous to the vehicle, a driving path for controlling vehicle movement is generated by analyzing at least one of the target road boundary and road information, thereby controlling the vehicle's autonomous driving. That is, after step S103, the steps shown in Figure 3 are further included and will be described below in conjunction with Figure 3.

[0123] In step S301, the vehicle's travel route is determined based on the target road boundary and / or the road information.

[0124] In some embodiments, the vehicle's travel path may be determined and controlled based on a target road boundary, the vehicle's travel path may be determined and controlled based on road information, or the vehicle's travel path may be determined and controlled by combining the target road boundary and road information. By analyzing the target road boundary and informing the vehicle of its location, the vehicle's travel path can be generated and controlled to move the vehicle away from the target road boundary, thereby reducing potential hazards during vehicle travel. Alternatively, by analyzing road information of the road on which the vehicle is located, i.e., road signals, multiple lanes, stop line areas, multi-class directional signs, and object information, the vehicle's travel path at a future time can be predicted and the vehicle's travel can be controlled. Alternatively, by combining the target road boundary and road information, a more accurate travel path can be generated and the vehicle's travel can be controlled more accurately.

[0125] The travel route is a route plan for the vehicle to travel at a future time, and includes the vehicle's direction of travel, speed, and route. The vehicle's travel route may be determined based on a target road boundary, based on road information, or by combining the target road boundary and road information.

[0126] In some possible implementations, the vehicle's travel route is determined based on the aforementioned road information. This step can be achieved by the following steps.

[0127] In the first step, the driving intention of the vehicle is determined based on the road information.

[0128] In some embodiments, the vehicle's driving intention is determined based on at least some road information. For example, the vehicle's driving intention is determined based on multiple lanes, stop line areas, and multi-class directional signs. The driving intention is used to represent the vehicle's driving pattern over a future period, such as the speed and direction of travel for the next minute.

[0129] In the second step, the vehicle's travel route is determined based on the intended driving route.

[0130] In some embodiments, a driving route is obtained by specifying the driving route of the vehicle within a predetermined period in the future, according to the vehicle's intended driving intention during that period. For example, if the intended driving intention is to drive straight, a route in which the vehicle drives straight within the predetermined period is formulated.

[0131] In step S302, the vehicle's movement is controlled based on the aforementioned travel path.

[0132] In some embodiments, electronic equipment can determine a vehicle's travel path relative to road boundaries in which the vehicle can enter, and further control the vehicle to travel along that path. By comprehensively considering target road boundaries and road information in this way, effective control of the vehicle can be achieved.

[0133] In some embodiments, that is, the step of "determining the vehicle's travel route based on the road information" in step S301 above can be achieved by the following steps S311 and S312 (not shown).

[0134] In step S311, the direction and location of the vehicle's turn are determined based on the road signals and direction signs in the road information.

[0135] In some embodiments, the direction of travel indicator for a vehicle can be determined according to road signals in the road information, whether it is going straight, turning left, turning right, going straight and turning left, going straight and turning right, making a U-turn, or turning left, going straight, and turning right. The turning position represents the turning point where the vehicle enters the turning lane when turning, and the turning direction represents the direction of travel of the vehicle when entering the turning lane from its current position. Thus, the turning direction may be the direction of travel that the vehicle continuously maintains during the turning process.

[0136] In step S312, the vehicle's turning path is determined based on the direction of the turn and the position of the turn.

[0137] In some embodiments, the vehicle's turning path can be predicted according to the direction of travel when the vehicle turns, as indicated by the turning direction, and the turning point of the vehicle when the vehicle turns, thereby enabling the vehicle to turn precisely based on the turning path. In this way, by following the road surface signals in the road information, the direction and position of the vehicle's turn at a future time can be accurately predicted, and the vehicle's turning can be precisely controlled.

[0138] In steps S311 and S312 described above, road signals and direction markers are obtained from road information, and the accuracy of the driving route can be improved by generating the vehicle's driving route according to the road signals and direction markers.

[0139] In some embodiments, the driving path is updated by detecting obstacle information in the road image, thereby effectively controlling the vehicle's movement. That is, step S302 described above can be achieved by the following steps S321 and S322 (not shown).

[0140] In step S321, the travel route is updated based on the obstacle information in the road information to obtain the updated route.

[0141] In some embodiments, when object information for an obstacle is detected in the road information, that is, when an obstacle exists on the road, the generated travel route is updated according to the object information for the obstacle in the road information. For example, by updating the route that passes through the location of the obstacle within the original travel route according to the location and size information of the obstacle, the updated route avoids the obstacle.

[0142] In step S322, the vehicle's movement is controlled based on the updated route.

[0143] In some embodiments, controlling a vehicle to travel along an updated route can improve the safety of vehicle travel by allowing the vehicle to avoid obstacles while in motion.

[0144] In steps S321 and S322 described above, the location information of obstacles in the road information is combined and the driving route is updated, thereby controlling the vehicle's movement according to the updated route and providing more information when the autonomous vehicle makes decisions.

[0145] In the embodiments of this disclosure, after recognizing a target road boundary, a subsequent driving path is generated in conjunction with rich road information. In this way, the generated driving path is more accurate, and based on this, precise control of the vehicle can be achieved by the driving path.

[0146] In some embodiments, the vehicle's travel path is generated by analyzing the target road boundary and updating the map of the vehicle's location. That is, in step S301 above, the step of determining the vehicle's travel path based on the target road boundary can be achieved by the following steps.

[0147] In the first step, the map data of the vehicle's location is updated based on the target road boundary to obtain an updated map.

[0148] In some embodiments, map data of the vehicle's location is acquired, which may be a third-party map, road information and traffic signs (e.g., traffic lights, traffic signs) collected by a positioning system in the vehicle's equipment. The target road boundary is marked on the map data of the vehicle's location to obtain the updated map. In this way, the updated map carries the target road boundary, allowing the vehicle to be informed of the location of any invisible road boundaries.

[0149] In the second step, the vehicle's route is determined based on the updated map.

[0150] In some embodiments, a driving path is formulated that moves away from the target road boundary, according to the target road boundary marked on an updated map, thereby preventing the vehicle from contacting the target road boundary when driving along the said driving path.

[0151] In the embodiments of this disclosure, a road hazard-aware map is created according to the detected target road boundary, thereby generating a driving path that controls vehicle movement according to the updated map, and improving the safety of the driving path.

[0152] In some embodiments, after recognizing a target road boundary, the relationship between the target road boundary and the driving state is analyzed to effectively control the vehicle's movement. This step can be achieved through the following process.

[0153] The vehicle is controlled based on the relationship between the target road boundary and the vehicle's driving state.

[0154] Here, the relationship between the target road boundary and the vehicle's driving state is used to represent the influence the target road boundary has on the driving state, and includes the magnitude of the angle between the target road boundary and the vehicle's direction of travel, and the magnitude of the distance between the target road boundary and the vehicle in motion.

[0155] In some possible implementations, after recognizing a target road boundary, the vehicle can be controlled to enter a braking state. That is, controlling the vehicle may involve controlling the vehicle to move from a driving state to a braking state, or controlling the vehicle to move away from the target road boundary.

[0156] In this way, after determining the target road boundary, braking instruction information is generated to cause the vehicle to enter a braking state. In this way, when the target road boundary is recognized, the vehicle can be controlled to prepare for braking, thereby improving the safety of vehicle driving. After determining the target road boundary, the electronic equipment generates braking instruction information and feeds this braking instruction information back to the vehicle's automated driving system. The vehicle's automated driving system responds to the braking instruction information and controls the vehicle to enter a braking state. If danger is detected, the vehicle can be controlled to move from a driving state to a braking state, or the vehicle can be controlled to move away from the target road boundary. For example, after recognizing the target road boundary, the vehicle can be controlled to move from a driving state to a braking state, or the vehicle can be controlled to move away from the target road boundary. In this way, after detecting the target road boundary, braking instruction information is generated to control the vehicle to enter a braking state, thereby improving the safety of vehicle driving.

[0157] In some possible implementations, the relationship between the target road boundary and the vehicle's driving state includes at least one of the following cases:

[0158] Case 1: The relationship between the target road boundary and the vehicle's driving state may be such that the distance between the overlapping area where the target road boundary is located and the road intersection in front of the vehicle is less than the second predetermined distance.

[0159] Here, the intersection is the intersection in the direction of travel of the vehicle on the roadway where the vehicle is located, that is, the intersection ahead of the roadway where the vehicle is located. The distance from the intersection to the overlapping area may be the minimum distance between the intersection and the overlapping area, or it may be the average of the maximum and minimum distances between the intersection and the overlapping area. The second predetermined distance may be the same as or different from the first predetermined distance, may be set based on measuring the vehicle's blind spot range, or may be set arbitrarily by the user. If the distance between the intersection and the overlapping area is less than the second predetermined distance, it means that the invisible overlapping area will affect vehicles passing through the intersection, and in this case, braking instruction information is generated to control the vehicle to enter a braking state in order to improve the safety of vehicle driving.

[0160] Case 2: The relationship between the target road boundary and the vehicle's driving state may be such that the distance between the overlapping area and the vehicle's location is less than the third predetermined distance.

[0161] In some embodiments, the distance between the overlapping area and the vehicle's location may be the minimum distance between the overlapping area and the vehicle's location, or the average of the maximum and minimum distances between the overlapping area and the vehicle's location, and the third predetermined distance may be the distance between the vehicle's location and the road edge when the vehicle is traveling normally. If the distance between the overlapping area and the vehicle's location is less than the third predetermined distance, it means that the overlapping area affects the vehicle's normal driving, and in this case, braking instruction information is generated to control the vehicle to enter a braking state in order to improve the safety of vehicle driving.

[0162] Case 3: The relationship between the target road boundary and the vehicle's driving state may be such that the angle between the vehicle's driving direction and the target road boundary is less than a predetermined angle.

[0163] Here, the predetermined angle may be set based on the minimum angle between the direction of travel and the road boundary when the vehicle is traveling normally, for example, the minimum angle between the direction of travel and the road boundary assuming the vehicle can turn normally. If the angle between the direction of travel and the target road boundary is less than the predetermined angle, it means that the target road boundary will affect the normal driving of the vehicle. In this case, the safety of vehicle driving can be improved by controlling the vehicle to move from a driving state to a braking state, or by controlling the vehicle to move away from the target road boundary.

[0164] Case 4: The relationship between the target road boundary and the vehicle's driving state may be such that the target road boundary is adjacent to the roadway in which the vehicle is located.

[0165] Here, if the target road boundary is adjacent to the roadway where the vehicle is located, the vehicle will come into contact with the target road boundary if it continues to travel along the roadway in its current direction of travel. Since the danger of the target road boundary is unpredictable, if the target road boundary is adjacent to the roadway where the vehicle is located, the potential danger to the vehicle's movement can be effectively reduced by controlling the vehicle to move from a driving state to a braking state, or by controlling the vehicle to move away from the target road boundary.

[0166] In the embodiments of this disclosure, by analyzing the relationship between the target road boundary and the vehicle's driving state, if the target road boundary affects the vehicle's normal driving, braking instruction information can be generated to control the vehicle to enter a braking state, or the vehicle can be controlled to move away from the target road boundary, thereby further improving the vehicle's driving safety.

[0167] In some embodiments, after determining the target road boundary, object recognition in the region of interest can be made more accurate by acquiring images of the region of interest and the road according to different resolutions or frame rates. This step can be achieved in the following way.

[0168] Method 1: A region of interest is set based on the target road boundary, and an image corresponding to the region of interest is acquired based on a first resolution.

[0169] In method 1, the road image is acquired according to a second resolution, the second resolution being smaller than the first resolution.

[0170] Method 2: An image corresponding to the region of interest is acquired based on the first frame rate.

[0171] In method two, the road image is acquired based on a second frame rate, the second frame rate being smaller than the first frame rate.

[0172] Here, the electronic device sets a Region of Interest (ROI) based on the road boundary in which the vehicle can enter. When acquiring road images of the road environment, the electronic device can use a second resolution (also called low resolution), while the Region of Interest can be acquired using a first resolution (also called high resolution), which is higher than the second resolution. In this way, higher quality images are collected for the Region of Interest, facilitating object recognition in subsequent images corresponding to the Region of Interest.

[0173] Alternatively, when an electronic device acquires road images of the road environment, it can use a second frame rate (also called a low frame rate) to acquire them, while the region of interest can be acquired using a first frame rate (also called a high frame rate) that is higher than the second frame rate. In this way, object recognition in the image corresponding to the subsequent region of interest is facilitated.

[0174] In some embodiments, after detecting that a vehicle has moved away from the target road boundary, a hazard prediction notification is sent to a vehicle following the vehicle to alert the following vehicle to pay attention to the target road boundary. This step can be achieved by the following process.

[0175] First, road environment information around the aforementioned target road boundary is collected.

[0176] In some embodiments, when it is detected that the vehicle has moved away from the target road boundary, road environment information around the target road boundary is collected. Because the target road boundary is not visible, the vehicle cannot predict the risks that may exist at the target road boundary. Therefore, after it is detected that the vehicle has passed the target road boundary, a camera inside the vehicle recognizes the target road boundary and can collect road environment information around the target road boundary using the camera. The road environment information includes the length, position, obstacle information, and road signals of the target road boundary.

[0177] Next, notification information is generated based on the aforementioned road environment information.

[0178] In some embodiments, based on the fact that the road environment information of the target road boundary is included in the notification information, the notification information is transmitted to a vehicle following the vehicle.

[0179] Finally, the notification information is transmitted to the vehicle following the vehicle.

[0180] In some embodiments, notification information including road environment information is transmitted to the autonomous driving system of a following vehicle, or to a terminal that communicates with the autonomous driving system, so that the following vehicle can formulate an appropriate driving route based on the road environment information in the notification information.

[0181] In the embodiments of this disclosure, after detecting that a vehicle has passed a target road boundary, road environment information around the target road boundary is transmitted to the following vehicle in the form of notification information to inform the following vehicle in a timely manner that a target road boundary exists ahead, so that the following vehicle can adjust its driving path in a timely manner.

[0182] In the following, an exemplary application example of the embodiments of this disclosure in a real-world application scenario will be described, using a deep neural network to recognize road signs and determine whether a vehicle should turn at a road intersection.

[0183] In autonomous driving, as a whole system, the output of a sensing module serves subsequent modules. For example, the sensing result not only indicates whether or not an object is present ahead, but also provides logical outputs relevant to the subsequent modules, and needs to provide certain control signals and logical signals for autonomous driving. However, commercially available sensing modules do not effectively combine all sensing information. Thus, many problems are introduced in applications, namely, the purpose of sensing is only to determine the presence or absence of a target, and there is no concern for the reliability and accuracy of subsequent control signals.

[0184] In view of this, the embodiments of this disclosure provide a method for selecting the direction of travel at a road intersection based on road markings, and employ road marking information, lane information, and intersection direction of travel information to provide an effective automated driving signal to a lower-level module.

[0185] Embodiments of this disclosure provide an image processing method that detects road boundaries, acquires road sensing information, converts the acquired road sensing information into an overhead view, and integrates the road sensing information in the overhead view to determine a change in the direction of travel at a road intersection. The method can be implemented by the following steps.

[0186] In the first step, road boundaries are detected in the road image, and road sensing information for the road is extracted.

[0187] In some embodiments, road boundary detection can be achieved by the following two methods.

[0188] Method 1: Directly detect road boundaries using a detection model. This step can be achieved through the following process.

[0189] During autonomous driving, the vehicle senses and outputs information according to the information provided on the road and integrates the information with the results output from the model. As shown in Figure 4, Figure 4 is a network structure diagram of an image processing method according to an embodiment of the present disclosure, and the network architecture comprises an image input module 401, a core network 402, a road surface signal detection branch network 41, a lane segmentation branch network 42, a stop line segmentation branch network 43, an intersection direction of travel output branch network 44, and an obstacle detection output branch network 45. Here, The image input module 401 is configured to input road images. The core network 402 is configured to perform feature extraction on the input road images.

[0190] The aforementioned core network may be a VGG network, a GoogleNet network, or a residual (ResNet) network, etc.

[0191] The road surface signal detection branch network 41 is configured to perform a detection task and performs road surface signal detection based on extracted image features.

[0192] Here, the road signal detection branching network 41 may be implemented by detectors such as a two-stage detector or a one-stage detector. The road signal detection branching network 41 may also be a classification branching network for classifying the detected road signals, where the classification includes going straight, turning left, turning right, going straight and turning left, going straight and turning right, U-turn, turning left and going straight and turning right, etc.

[0193] The lane segmentation branching network 42 is configured to perform segmentation on lanes within a road image based on extracted image features.

[0194] Here, using lane labeling for a "3-lane, 4-line" road as an example, the labeled lanes include the left lane of the road in which the vehicle is located (i.e., the left-side lane), the left lane of the road to the left of the road in which the vehicle is located (i.e., the left-left lane), the right lane of the road in which the vehicle is located (the right-side lane), and the right lane of the road to the right of the road in which the vehicle is located (the right-right lane). The lane detection task is defined as semantic segmentation. Specifically, the left-left lane is designated as Class 1, the left lane as Class 2, the right lane as Class 3, the right-right lane as Class 4, and the background class as Class 0.

[0195] The stop line segmentation branching network 43 is configured to perform segmentation on stop lines within a road image based on extracted image features.

[0196] Here, a two-class segmentation method can be used to detect the stop line, setting the stop line region to class 1 and the background class to class 0.

[0197] The intersection direction output branching network 44 is configured to recognize intersection direction change edges by employing a semantic segmentation method.

[0198] Here, we define the direction of travel at an intersection into three classes, from left to right: the left-side direction change edge is Class 1, the forward direction change edge is Class 2, the right-side direction change edge is Class 3, and the background class is 0.

[0199] The obstacle detection output branching network 45 is configured to recognize obstacles on the road surface and performs obstacle recognition according to the detection method.

[0200] Here, an obstacle detection output branching network 45 is employed to recognize obstacles on the road surface, using the obstacles as the foreground for target detection and the non-obstacle objects as the background. As shown in Figure 5A, by performing road boundary detection on the collected in-vehicle camera image 511, obstacles 512 and road boundaries 513 in the image 511 can be recognized.

[0201] Method 2: Use a detection model to detect other road information and estimate road boundaries based on that information. Here, the method of Method 1 can be used to detect other road information.

[0202] There are two methods for estimating road boundaries based on other road information:

[0203] 1. Determine the road boundaries by connecting the ends of each road.

[0204] 2. Semantic segmentation is used to determine the drivable area, and the contours of the drivable area are used to determine the road boundaries.

[0205] In the second step, the road boundaries that are not visible from the vehicle are determined based on road sensing information.

[0206] In some embodiments, the road boundary, which is not visible from the vehicle, can be determined by the following steps.

[0207] In Step 1, road information is recognized from images collected by the in-vehicle camera.

[0208] Here, the road information includes information about objects on the road and lane markings. The network architecture shown in Figure 4 allows for the recognition of road information from the collected images.

[0209] In step 2, determine the unknown areas that are not visible from your vehicle.

[0210] Here, the unknown region may be a region that is shielded by an obstruction.

[0211] In step 3, the actual road area is estimated based on road information.

[0212] In step 4, the actual road area and unknown areas are converted to an overhead view.

[0213] Here, the image 511 in Figure 5A may be converted to an overhead view image like the image 521 shown in Figure 5B, in which the unknown area 522 is an area not visible from the vehicle 523, the actual area 524 is the actual road area, the boundary lines 525, 526 and 527 are road boundaries foreseeable by the vehicle 523, the boundary line 528 is a road boundary not visible from the vehicle 523, and the obstacle 529 is the obstacle.

[0214] In Step 5, if the actual road area and the unknown area overlap in the overhead view, the road boundary of the actual area that overlaps with the unknown area is determined as the road boundary that is not visible from the vehicle.

[0215] In several possible implementations, the road sensing information acquired in the first step is converted to an overhead view by homography. That is, the road sensing information in the forward view of the vehicle is converted to road sensing information in the overhead view by a matrix transformation method, and the road sensing information in the overhead view is fitted. That is, lanes, stop lines, and direction change edges in the overhead view are fitted to obtain the fitting result. As shown in Figure 6A, Figure 6A is a schematic diagram of an application scene of the image processing method according to an embodiment of this disclosure, and as can be seen from Figure 6A, road sensing information in the forward view, i.e., stop lines 51, 52 and 53, direction change edges 54, 55 and 56, lanes 501 and 502, and obstacles 503 in the forward view are shown. The stop lines, direction change edges, lanes, and obstacles in Figure 6A are converted to an overhead view, and as shown in Figure 6B, the stop lines 51, 52, and 53 are converted to stop lines 61 and 62 in the overhead view, the direction change edges 54, 55, and 56 are converted to direction change edges 63, 64, and 65 in the overhead view, and obstacle 503 is converted to obstacle 601 in the overhead view.

[0216] Furthermore, as can be seen from Figure 6B, the vehicle 605 can detect semantic information based on road signals, and therefore it can determine that it is able to turn right. Accordingly, it selects the right-hand direction change edge, obtains the direction and position of the turn, generates a subsequent route plan, and sends a control signal to the vehicle to perform the turn. Similarly, the vehicle can generate other straight-ahead commands such as left turn or straight-ahead based on road signals, and can generate more stable signals by aligning them with signals on the map. At the same time, the vehicle considers the positional information of obstacles on the road. That is, if a direction change edge is blocked by an obstacle, it provides feedback that the road boundary in that direction cannot be accurately recognized, thereby providing more information when the autonomous vehicle makes a decision.

[0217] In the third step, after recognizing the invisible road boundary, the system prepares to brake and controls the vehicle to move away from the invisible road boundary.

[0218] In some possible implementations, after determining the overlapping area between the unknown area and the actual road area, braking preparations are made if the distance to the overlapping area with a crossroads is within a predetermined range, or if the distance to the overlapping area with an unknown area that the vehicle cannot perceive is within a predetermined range, braking preparations are made if the angle between the vehicle's direction of travel and the invisible road boundary is less than a predetermined value, braking preparations are made if the invisible road boundary is in contact with the vehicle's roadway, and at the invisible road boundary, the vehicle is controlled to move away from the road boundary that is in contact with the vehicle's roadway.

[0219] In the embodiments of this disclosure, a deep neural network is used to predict road markings, lane markings, and road intersection direction information to obtain accurate road structure information and direction of travel. Based on the above sensing information, the sensing information in the forward view is converted to an overhead view to determine the direction of travel information of the vehicle at the road intersection. In this way, the tasks of road marking, lane detection, and intersection direction detection are merged into the same deep learning network for collaborative learning, the final sensing output is obtained, and an effective signal is provided for subsequent direction control. Furthermore, by merging and learning multiple tasks in a hybrid network, network resources can be effectively saved.

[0220] In embodiments of this disclosure, when a vehicle is at an intersection, the required road boundary can be selected by detecting multiple road boundaries. As shown in Figure 7, when vehicle 71 is at an intersection, the required road boundary is selected by detecting multiple road boundaries. For example, if vehicle 71 is traveling on the left lane, the road boundary of the left lane is selected as the reachable road. The road boundaries detected by vehicle 71 include road boundaries 81 to 88, as shown in Figure 8, and reachable and unreachable boundaries are determined from among road boundaries 81 to 88. As shown in Figure 9, boundaries 91, 93, 95, and 98 are reachable boundaries, and boundaries 92, 94, 96, and 97 are unreachable boundaries.

[0221] In the embodiments of this disclosure, based on whether or not road boundaries are visible based on obstacle detection, an autonomous vehicle can be provided with richer shape and control information. From the standpoint of model design and learning, road marking, lane detection, and intersection direction detection tasks can be integrated into the same deep learning network for collaborative learning, obtaining the final sensing output, which not only effectively saves network resources but also provides an effective signal for subsequent direction control.

[0222] Those skilled in the art will understand that in the above-described method of a specific embodiment, the order in which each step is described does not restrict the implementation process to a strict execution order, and the specific execution order of each step should be determined by its function and possible internal logic.

[0223] Based on the same technical concept, embodiments of the present disclosure further provide a motion intention determination device corresponding to a motion intention determination method, and since the principle for solving the problems of the device in embodiments of the present disclosure is similar to the motion intention determination method described above in embodiments of the present disclosure, the implementation of the device can refer to embodiments of the above method.

[0224] The embodiments of this disclosure provide an image processing apparatus, and Figure 10 is a schematic diagram showing the configuration of the image processing apparatus according to the embodiments of this disclosure, and as shown in Figure 10, the image processing apparatus 1000 is An image acquisition unit configured to acquire road images collected by an image acquisition device mounted on a vehicle, A road boundary detection unit configured to detect multiple road boundaries within the road image based on the road image, The system includes a target road boundary determination section configured to determine a target road boundary that is dangerous to the vehicle among the plurality of road boundaries.

[0225] In some embodiments, the road boundary detection portion 1002 further, The system is configured to detect the aforementioned road image and determine a plurality of road boundaries related to the vehicle.

[0226] In some embodiments, the road boundary detection portion 1002 is A roadway detection sub-part is configured to detect the aforementioned road image and obtain multiple roadways within the road image, The system includes a first road boundary determination sub-part configured to connect the ends of each roadway within the plurality of roadways to obtain the plurality of road boundaries.

[0227] In some embodiments, the road boundary detection portion 1002 is A sub-part for dividing the drivable area is configured to perform semantic segmentation on the road image to obtain the drivable area within the road image, The system includes a second road boundary determination sub-part configured to determine the plurality of road boundaries based on the contour lines of the drivable area.

[0228] In some embodiments, the target road boundary determination portion 1003 is A first target road boundary determination sub-part is configured to determine, among the plurality of road boundaries, the road boundary adjacent to the roadway where the vehicle is located as the target road boundary. A second target road boundary determination sub-part is configured to determine, among the plurality of road boundaries, the road boundary in which the distance to the vehicle is less than a first predetermined distance as the target road boundary. A third target road boundary determination sub-part is configured to determine, among the plurality of road boundaries, the road boundary in which the road space between the vehicle and the boundary is less than a predetermined space as the target road boundary. A fourth target road boundary determination sub-part is configured to determine a target road boundary that is dangerous to the vehicle, based on road information determined by the road image, wherein the road information includes at least one of the following in the road image: road signals, lanes, stop line areas, direction signs, and obstacle information.

[0229] In some embodiments, the fourth target road boundary determination sub-part is An unknown road area determination unit configured to determine the actual road area and the unknown area that is unrecognizable by the vehicle based on the aforementioned road information, A road boundary determination unit configured to determine a road boundary that is not visible from the vehicle, based on the actual road area and the unknown area, The system includes a target road boundary determination portion configured to determine a road boundary that is not visible from the vehicle as the target road boundary.

[0230] In some embodiments, the road boundary determination unit is A sub-component of area viewpoint conversion is configured to convert the collection viewpoints of the actual road area and the unknown area into an overhead viewpoint, thereby obtaining the converted actual road area and the converted unknown area. An overlapping area determination sub-part configured to determine the overlapping area between the converted actual road area and the converted unknown area, The system includes a road boundary determination sub-part configured to determine the road boundary within the overlapping area as a road boundary invisible to the vehicle.

[0231] In some embodiments, the overlapping area determination sub-component is further configured to obtain first fitting information by fitting lanes, stop line areas, and direction signs within the converted actual road area, obtain second fitting information by fitting multiple lanes, stop line areas, and direction signs within the converted unknown area, and determine the overlapping area between the converted actual road area and the converted unknown area based on the first fitting information and the second fitting information.

[0232] In some embodiments, the image processing apparatus is A route determination unit configured to determine the vehicle's route based on the target road boundary and / or the road information, The system further includes a vehicle driving control unit configured to control the driving of the vehicle based on the aforementioned driving path.

[0233] In some embodiments, the travel path determination module is: A turning determination sub-component is configured to determine the direction and position of the vehicle's turn based on road signals and direction-of-travel signs within the aforementioned road information, The system includes a turning path determination sub-part configured to determine the vehicle's travel path based on the direction of the turn and the position of the turn.

[0234] In some embodiments, the vehicle driving control module is A sub-part for updating the travel route is configured to update the travel route based on object information of obstacles in the second road information to obtain an updated route, The system further includes a vehicle driving control sub-component configured to control the driving of the vehicle based on the updated route.

[0235] In some embodiments, the travel path determination module is: A map data update sub-part is configured to update the map data of the vehicle's location based on the aforementioned target road boundary to obtain an updated map, The system includes a route determination sub-component configured to determine the vehicle's route based on the updated map.

[0236] In some embodiments, the image processing apparatus is The system further includes a vehicle control unit configured to control the vehicle based on the relationship between the target road boundary and the vehicle's driving state.

[0237] In some embodiments, the relationship between the target road boundary and the vehicle's driving state is: The distance between the overlapping area where the target road boundary is located and the road intersection in front of the vehicle is less than the second predetermined distance. The distance between the overlapping area and the location of the vehicle is less than the third predetermined distance. The angle between the direction of travel of the vehicle and the target road boundary is less than a predetermined angle. The target road boundary includes at least one of the following: the target road boundary is adjacent to the roadway on which the vehicle is located.

[0238] In some embodiments, the vehicle control module is further configured to perform the steps of controlling the vehicle to move from a driving state to a braking state, or to control the vehicle to move away from the target road boundary.

[0239] In some embodiments, the image processing apparatus is A first region of interest determination unit is configured to set a region of interest based on the target road boundary and acquire an image corresponding to the region of interest based on a first resolution, wherein the road image is acquired according to a second resolution, and / or the first region of interest determination unit is smaller than the first resolution. A second region of interest determination component is configured to acquire an image corresponding to the region of interest based on a first frame rate, wherein the road image is acquired based on a second frame rate, and the second frame rate is smaller than the first frame rate.

[0240] In some embodiments, the image processing apparatus is A road environment information collection unit configured to collect road environment information around the aforementioned target road boundary, A notification information generation unit configured to generate notification information based on the aforementioned road environment information, A notification information portion configured to transmit the aforementioned notification information to a vehicle following the vehicle, wherein the following vehicle is located on the same roadway as the vehicle and travels in the same direction, further comprising a notification information portion.

[0241] It should be noted that the above description of the apparatus embodiments is similar to the description of the method embodiments and has similar beneficial effects. Technical details not disclosed in the apparatus embodiments of this disclosure can be understood by referring to the description of the method embodiments of this disclosure.

[0242] In the embodiments of this disclosure and other embodiments, “module” may be a circuit, processor, program or software, and of course it may be a unit, or it may not be a module at all.

[0243] It should be noted that, in embodiments of this disclosure, if the above-described road obstacle detection method is implemented as a software functional module and sold or used as an independent product, it may also be stored on a single computer-readable storage medium. Based on this understanding, essential parts of the technical solutions of embodiments of this disclosure, or parts that contribute to the prior art, may be embodied in the form of a software product, which may be stored on a single storage medium and include several instructions for causing a single computer device (which may be a terminal, server, etc.) to execute all or part of the methods of each embodiment of this disclosure. The aforementioned storage mediums include various media capable of storing program code, such as U disks, mobile hard disks, read-only memory (ROM), magnetic disks, or optical disks. Thus, embodiments of this disclosure are not limited to any particular combination of hardware and software.

[0244] In response to this, embodiments of the present disclosure further provide a computer program product storing computer executable instructions, which, after execution, can realize steps of an image processing method provided by embodiments of the present disclosure. In response to this, embodiments of the present disclosure further provide a computer storage medium storing computer executable instructions, which, when executed by a processor, realize steps of an image processing method provided by the above embodiments. In response to this, embodiments of the present disclosure provide a computer device, and Figure 11 is a schematic diagram showing the configuration of a computer device according to an embodiment of the present disclosure, as shown in Figure 11, the computer device 1100 comprises one processor 1101, at least one communication bus, a communication interface 1102, at least one external communication interface, and memory 1103. Here, the communication interface 1102 enables connection communication between these components. Here, the communication interface 1102 may include a display, and the external communication interface may include a standard wired interface and a wireless interface. Here, the processor 1101 executes an image processing program in memory to realize the steps of the image processing method provided by the above embodiment.

[0245] The above descriptions of the embodiments of the image processing apparatus, computer equipment, and storage medium are similar to the descriptions of the embodiments of the methods described above, and have similar technical descriptions and beneficial effects. Due to space limitations, the above descriptions of the embodiments of the methods can be referenced and will not be repeated here. Technical details not disclosed in the embodiments of the image processing apparatus, computer equipment, and storage medium of this disclosure can be understood by referring to the descriptions of the embodiments of the methods of this disclosure. It should be understood that in this specification, “one embodiment” or “one example” means that a particular feature, structure, or characteristic related to the embodiment is included in at least one embodiment of this disclosure. Therefore, “one embodiment” or “one example” throughout the specification does not necessarily refer to the same embodiment. Furthermore, these particular features, structures, or characteristics can be incorporated into one or more embodiments in any suitable manner. It should be understood that in each embodiment of this disclosure, the magnitude of the sequence number of each process described above does not mean the order of execution, and the execution order of each process should be determined by its function and internal logic and should not constitute any limitation on the implementation process of the embodiments of this disclosure. The sequence numbers in the embodiments described above are for illustrative purposes only and do not indicate any superiority or inferiority among the embodiments.

[0246] It should be noted that in this specification, the terms “includes,” “contains,” or any other variation thereof are intended to encompass non-exclusive inclusion, thereby including not only those elements but also other elements not expressly enumerated, or elements specific to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the expression “contains” does not preclude the presence of another identical element in a process, method, article, or apparatus containing that element.

[0247] It should be understood that, in some embodiments provided in this disclosure, the disclosed devices and methods can be implemented in other ways. The embodiments of the devices described above are illustrative only, and for example, the division of the units is merely a division of logical functions, and in actual implementation, there may be other methods of division, for example, multiple units or components may be combined, integrated into another system, and some features may be ignored or not implemented. Furthermore, the interconnections, direct connections or communication connections between each illustrated or described component may be indirect connections or communication connections via some interfaces, devices or units, and may be in the form of electrical, mechanical or other.

[0248] The units described as separation members may or may not be physically separated, the members referred to as units may or may not be physical units, they may be located in one place or distributed across multiple network units, and some or all of these units may be selected as needed to realize the objectives of the technical proposal of this embodiment. Furthermore, each functional unit in each embodiment of this disclosure may be integrated into a single second processing unit, each unit may be used individually as a single unit, or two or more units may be integrated into a single unit. The integrated unit may be embodied in the form of hardware or in the form of a combination of hardware and software functional units. Those skilled in the art will know that all or some of the steps for realizing the above embodiments can be performed by hardware related to program instructions, the aforementioned program may be stored in a computer-readable storage medium, and when the program is executed, steps including embodiments of the above method are performed, the aforementioned storage medium includes various media capable of storing program code, such as removable storage, read-only memory (ROM), magnetic disks or optical disks.

[0249] Alternatively, the integrated units of the present disclosure may be implemented in the form of software function modules and, if sold or used as independent products, stored on a single computer-readable storage medium. Based on this understanding, essential parts of the technical solutions of the embodiments of the present disclosure, or parts that contribute to the prior art, may be embodied in the form of a software product, which is stored on a single storage medium and includes several instructions for causing a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods described in each embodiment of the present disclosure. The aforementioned storage mediums include various media capable of storing program code, such as removable storage, ROM, magnetic disks, or optical disks. The above is merely an embodiment of the present disclosure, and the scope of protection of the present disclosure is not limited thereto. Any modifications or substitutions that are easily conceivable by a person skilled in the art within the technical scope disclosed in the present disclosure should be included within the scope of protection of the present disclosure. Accordingly, the scope of protection of the present disclosure shall be subject to the scope of protection of the claims. [Industrial applicability]

[0250] Embodiments of this disclosure provide an image processing method, apparatus, device, and storage medium, wherein a road image is acquired by an image acquisition device mounted on a vehicle, a plurality of road boundaries are detected within the road image based on the road image, and a target road boundary that is dangerous to the vehicle is determined among the plurality of road boundaries.

Claims

1. An image processing method performed by an electronic device, The steps include acquiring road images collected by an image acquisition device mounted on a vehicle, Based on the road image, the steps include detecting multiple road boundaries within the road image, The step of determining a target road boundary that is dangerous to the vehicle among the plurality of road boundaries, The step of determining a target road boundary that is dangerous to the vehicle among the aforementioned multiple road boundaries is: The steps include determining the actual road area and the unknown area that is unrecognizable by the vehicle based on the road information determined by the aforementioned road image, The steps include converting the collection viewpoints of the actual road area and the unknown area to an overhead viewpoint to obtain the converted actual road area and the converted unknown area, The steps include determining the overlapping area between the converted actual road area and the converted unknown area, The steps include determining the road boundary within the overlapping area as a road boundary that is not visible from the vehicle, The step of determining the road boundary that is not visible from the vehicle as the target road boundary, Image processing methods, including those mentioned above.

2. The step of detecting multiple road boundaries within the road image based on the road image is: The step includes detecting the road image and determining a plurality of road boundaries related to the vehicle, The image processing method according to claim 1.

3. The step of detecting multiple road boundaries within the road image based on the road image is: The steps include detecting the road image and obtaining multiple roadways within the road image, The step of obtaining the multiple road boundaries by connecting the ends of each roadway within the multiple roadways, The image processing method according to claim 1.

4. The step of detecting multiple road boundaries within the road image based on the road image is: The steps include performing semantic segmentation on the road image to obtain the drivable area within the road image, The step of determining the plurality of road boundaries based on the contour lines of the drivable area includes: The image processing method according to claim 1.

5. The step of determining a target road boundary that is dangerous to the vehicle among the aforementioned multiple road boundaries is further: Among the aforementioned multiple road boundaries, the step of determining the road boundary adjacent to the roadway where the vehicle is located as the target road boundary, A step of determining the target road boundary among the plurality of road boundaries, where the distance to the vehicle is less than a first predetermined distance. The step of determining, among the plurality of road boundaries, a road boundary in which the road space between the vehicle and the boundary is less than a predetermined space is defined as the target road boundary, includes at least one of the following steps: The image processing method according to claim 1.

6. The road information includes at least one of the following in the road image: road surface signals, lanes, stop line areas, direction signs, and obstacle information. The image processing method according to claim 1.

7. The step of determining the overlapping area between the converted actual road area and the converted unknown area is: The steps include: fitting the converted lane lines, stop line areas, and direction signs within the actual road area to obtain first fitting information; The steps include: fitting the lane lines, stop line areas, and direction of travel signs within the converted unknown area to obtain second fitting information; The process includes the step of determining the overlapping area between the converted actual road area and the converted unknown area based on the first fitting information and the second fitting information, The image processing method according to claim 1.

8. After determining the target road boundary, the image processing method is A step of determining the vehicle's travel route based on the aforementioned target road boundary and / or road information, The further step includes controlling the movement of the vehicle based on the aforementioned travel path, The image processing method according to claim 5.

9. The step of determining the vehicle's travel route based on the aforementioned road information is: The steps include determining the direction and location of the vehicle's turn based on the road signals and direction-of-travel signs in the aforementioned road information, The step of determining the vehicle's travel path based on the direction of the turn and the position of the turn, The image processing method according to claim 8.

10. The step of controlling the movement of the vehicle based on the aforementioned travel path is: The steps include updating the travel route based on the obstacle information in the aforementioned road information to obtain the updated route, The steps include controlling the vehicle's movement based on the updated route, The image processing method according to claim 8.

11. The step of determining the vehicle's travel path based on the aforementioned target road boundary is: The steps include updating the map data of the vehicle's location based on the aforementioned target road boundary to obtain an updated map, The steps include determining the vehicle's route based on the updated map, The image processing method according to claim 8.

12. After determining the target road boundary, the image processing method is The step further includes controlling the vehicle based on the relationship between the target road boundary and the vehicle's driving state, The image processing method according to claim 1.

13. The relationship between the target road boundary and the vehicle's driving state is: The distance between the overlapping area where the target road boundary is located and the road intersection in front of the vehicle is less than the second predetermined distance. The distance between the overlapping area and the location of the vehicle is less than the third predetermined distance. The angle between the direction of travel of the vehicle and the target road boundary is less than a predetermined angle, and The target road boundary includes at least one of the following: the target road boundary is adjacent to the roadway on which the vehicle is located. The image processing method according to claim 12.

14. The step of controlling the vehicle is: The steps include controlling the vehicle to move from a driving state to a braking state, or controlling the vehicle to move away from the target road boundary. The image processing method according to claim 12.

15. After determining the target road boundary, the image processing method is A step of setting a region of interest based on the target road boundary, and acquiring an image corresponding to the region of interest based on a first resolution, wherein the road image is acquired according to a second resolution, the second resolution being smaller than the first resolution, and / or A step of acquiring an image corresponding to the region of interest based on a first frame rate, further comprising the step of acquiring the road image based on a second frame rate, wherein the second frame rate is smaller than the first frame rate. The image processing method according to claim 1.

16. After determining the target road boundary, the image processing method is The steps include: collecting road environment information around the aforementioned target road boundary, The steps include generating notification information based on the aforementioned road environment information, The step of transmitting the notification information to a vehicle following the vehicle, wherein the following vehicle is located on the same roadway as the vehicle and is traveling in the same direction, further comprising: The image processing method according to claim 1.

17. An image processing device, An image acquisition unit configured to acquire road images collected by an image acquisition device mounted on a vehicle, A road boundary detection unit configured to detect multiple road boundaries within the road image based on the road image, The system includes a target road boundary determination section configured to determine a target road boundary that is dangerous to the vehicle among the plurality of road boundaries, The aforementioned target road boundary determination portion is, Based on the road information determined from the aforementioned road image, the actual road area and the unknown area that is unrecognizable by the vehicle are determined. The collection viewpoints for the actual road area and the unknown area are converted to an overhead view, thereby obtaining the converted actual road area and the converted unknown area. Determine the overlapping area between the converted actual road area and the converted unknown area. The road boundary within the overlapping area is determined to be a road boundary that is not visible from the vehicle. The road boundary that is not visible from the vehicle is determined to be the target road boundary. Image processing device.

18. A computer storage medium storing computer-executable instructions that cause a computer to perform the image processing method described in any one of claims 1 to 16.

19. A computer device comprising: a memory storing computer executable instructions; and a processor capable of implementing the image processing method described in any one of claims 1 to 16 when executing the computer executable instructions in the memory.

20. A computer program for causing an electronic device to perform the image processing method described in any one of claims 1 to 16.

Images