Road image detection method, device, electronic device and storage medium

By using feature extraction and multiple edge detection modules in the processing model to detect road images, the problem of poor detection in the prior art is solved, and more comprehensive road deconstruction information acquisition and more efficient detection performance are achieved.

CN116434173BActive Publication Date: 2024-11-26TENCENT TECHNOLOGY (SHENZHEN) CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202310404721.8
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2023-04-12
Publication Date
2024-11-26
Estimated Expiration
2043-04-12

AI Technical Summary

Technical Problem

The existing technology has too simple road image detection methods, resulting in incomplete road deconstruction information, and it is necessary to improve the degree of detection refinement.

Method used

Using a processing model, the target road image is extracted through the feature extraction module, and multiple edge detection modules are used to map the target image features, thereby extracting different types of road edges and enriching the types of road deconstruction information.

Benefits of technology

The refinement of road image detection is improved, the detection efficiency is improved, and the detection performance of the processing model is improved through the correlation between multiple edge detection modules.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN116434173B_ABST
    Figure CN116434173B_ABST
Patent Text Reader

Abstract

The embodiments of the present application disclose a road image detection method, device, electronic device and storage medium. By introducing a processing model, the detection of road edges of a target road image can be realized based on the processing model, and multiple types of road edges can be detected, thereby enriching the types of road deconstruction information obtained by detection, and thus improving the refinement of road image detection. In addition, by integrating multiple edge detection modules into one processing model, on the one hand, the detection efficiency of road images can be effectively improved, and on the other hand, since multiple edge detection modules share target image features, the learning between multiple edge detection modules can be correlated, thereby improving the detection performance of the processing model as a whole, and can be widely used in cloud technology, intelligent transportation, intelligent driving, maps, navigation and other technical fields.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a road image detection method, device, electronic device and storage medium. Background Art

[0002] As a basic geographic data, road network plays an important role in driving navigation, assisted driving, map drawing and updating. Correspondingly, road deconstruction information is a key data in practical applications. At present, road images are usually detected to obtain road deconstruction information. However, the relevant technology is too simple to detect road images, resulting in incomplete road deconstruction information. Therefore, it is necessary to improve the refinement of road image detection. Summary of the invention

[0003] The following is a summary of the subject matter described in detail herein. This summary is not intended to limit the scope of the claims.

[0004] The embodiments of the present application provide a road image detection method, device, electronic device and storage medium, which can improve the refinement of road image detection.

[0005] On the one hand, an embodiment of the present application provides a road image detection method, including:

[0006] Acquire a target road image, input the target road image into a processing model, and extract features of the target road image based on a feature extraction module of the processing model to obtain target image features of the target road image;

[0007] Based on the multiple edge detection modules of the processing model, the target image features are mapped respectively to obtain the detection results of the target road edges output by the edge detection modules, wherein the target road edges are the edges of the road surfaces in the target road images, and the edge types of the target road edges are different from each other;

[0008] The target road edge is extracted from the target road image according to the detection results of the plurality of target road edges.

[0009] On the other hand, an embodiment of the present application further provides a road image detection device, including:

[0010] A target image feature determination module is used to obtain a target road image, input the target road image into a processing model, and extract features of the target road image based on a feature extraction module of the processing model to obtain target image features of the target road image;

[0011] a target road edge determination module, configured to map the target image features respectively based on the multiple edge detection modules of the processing model, and obtain the detection results of the target road edges output by the respective edge detection modules, wherein the target road edges are edges of the road surface in the target road image, and the edge types of the respective target road edges are different from each other;

[0012] The target road edge extraction module is used to extract the target road edge from the target road image according to the detection results of the plurality of target road edges.

[0013] Furthermore, the target road edge determination module is specifically used for:

[0014] Mapping the target image features based on an edge detection module for detecting the edge of a road section in the processing model to obtain a detection result of the edge of the road section;

[0015] Mapping the target image features based on an edge detection module for detecting the edge of the intersection surface in the processing model to obtain a detection result of the edge of the intersection surface;

[0016] The target image features are mapped based on an edge detection module in the processing model for detecting closed edges of road sections to obtain detection results of closed edges of road sections.

[0017] Furthermore, the target road edge determination module is specifically used for:

[0018] Based on the multiple edge detection modules of the processing model, the target image features are mapped respectively to obtain the detection results of the candidate edges output by each edge detection module;

[0019] Mapping the target image features based on the instance classification module of the processing model to obtain target classification results of various instances in the target road image output by the instance classification module;

[0020] The detection result of the target road edge is determined according to the target classification result and the detection result of the candidate edge.

[0021] Furthermore, the target road edge determination module is specifically used for:

[0022] Based on the instance classification module of the processing model, the target image feature is fully connected to obtain a first fully connected feature;

[0023] The first fully connected feature is normalized based on the instance classification module to obtain target classification results of various instances in the target road image output by the instance classification module.

[0024] Furthermore, the road image detection device further includes a training module, and the training module is specifically used for:

[0025] Acquire a sample road image and road edge mask images of multiple different edge types corresponding to the sample road image;

[0026] determining a plurality of sample road edges of the sample road image having different edge types based on the processing model;

[0027] Determining edge detection loss according to the sample road edge and the corresponding road edge mask image;

[0028] The processing model is trained according to the edge detection loss.

[0029] Furthermore, the sample road image is annotated with category labels of various instances, and the above training module is specifically used for:

[0030] Determine sample classification results of various instances in the sample road image based on the processing model, and determine classification loss according to the sample classification results and the category label;

[0031] The processing model is trained according to the edge detection loss and the classification loss.

[0032] Furthermore, the target image feature determination module is specifically used for:

[0033] Performing down-sampling processing on the target road image based on the feature extraction module of the processing model to obtain down-sampled image features of the target road image;

[0034] The target road image is up-sampled based on the feature extraction module to obtain the target image feature of the target road image.

[0035] Furthermore, the first road image detection device further includes a post-processing module, which is specifically used for:

[0036] The road surface detection module based on the processing model maps the target image features to obtain a detection result of the target road surface output by the road surface detection module;

[0037] Extracting the target road surface from the target road image according to the detection result of the target road surface, performing contour extraction on the target road surface, and obtaining a road surface contour of the target road surface;

[0038] The target road edge is corrected according to the road surface contour.

[0039] Furthermore, the above post-processing module is specifically used for:

[0040] Determine a difference set between the road surface contour and each of the target road edges to obtain a first edge set that is not successfully detected by the edge detection module;

[0041] Determine the intersection between the road surface contour and each of the target road edges to obtain a second edge set successfully detected by the edge detection module;

[0042] Traversing first road edges in the first edge set, and determining at least one second road edge adjacent to the first road edge from the second edge set;

[0043] The edge type of the first road edge is re-determined according to the edge type of the second road edge, and the target road edge is corrected according to the first road edge after the edge type is re-determined.

[0044] Furthermore, the number of the second road edges is two, and the post-processing module is specifically used for:

[0045] comparing edge types of two edges of the second road;

[0046] When the comparison result indicates that the edge types of the two second road edges are the same, the edge type of the second road edge is used as the edge type of the first road edge; or, when the comparison result indicates that the edge types of the two second road edges are different, the edge type of the first road edge is determined as a road segment edge.

[0047] Furthermore, the above post-processing module is specifically used for:

[0048] A road surface detection module based on the processing model performs full connection processing on the target image feature to obtain a second fully connected feature;

[0049] Performing activation processing on the second fully connected feature based on the road surface detection module to obtain an activated feature;

[0050] Performing full connection processing on the activated feature based on the road surface detection module to obtain a third fully connected feature;

[0051] The third fully connected feature is normalized based on the road surface detection module to obtain a detection result of the target road surface output by the road surface detection module.

[0052] On the other hand, an embodiment of the present application further provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the above-mentioned road image detection method when executing the computer program.

[0053] On the other hand, an embodiment of the present application further provides a computer-readable storage medium, wherein the storage medium stores a computer program, and the computer program is executed by a processor to implement the above-mentioned road image detection method.

[0054] On the other hand, the embodiment of the present application further provides a computer program product, which includes a computer program stored in a computer-readable storage medium. A processor of a computer device reads the computer program from the computer-readable storage medium, and the processor executes the computer program, so that the computer device executes the above-mentioned road image detection method.

[0055] The embodiment of the present application includes at least the following beneficial effects: by introducing a processing model, a feature extraction module based on the processing model performs feature extraction on a target road image to obtain target image features of the target road image, and then multiple edge detection modules based on the processing model respectively map the target image features, so that different information can be learned from the target image features according to multiple edge detection modules, and then the detection results of target road edges of different edge types output by each edge detection module can be obtained, and the target road edges can be extracted from the target road image. Therefore, based on the processing model, the detection of road edges of the target road image can be realized, and multiple types of road edges can be detected, thereby enriching the types of road deconstruction information obtained by the detection, and then the refinement of the detection of road images can be improved. In addition, by integrating multiple edge detection modules into one processing model, on the one hand, the detection efficiency of road images can be effectively improved, and on the other hand, since multiple edge detection modules share target image features, the learning between multiple edge detection modules can be correlated, thereby improving the detection performance of the processing model as a whole.

[0056] Other features and advantages of the present application will be set forth in the following description, and in part will become apparent from the description, or may be understood by practicing the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] The accompanying drawings are used to provide further understanding of the technical solution of the present application and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the technical solution of the present application and do not constitute a limitation on the technical solution of the present application.

[0058] Figure 1 A schematic diagram of an optional implementation environment provided for an embodiment of the present application;

[0059] Figure 2 A schematic diagram of another optional implementation environment provided for an embodiment of the present application;

[0060] Figure 3An optional flowchart of the road image detection method provided in an embodiment of the present application;

[0061] Figure 4 An optional structural diagram of a processing model provided in an embodiment of the present application;

[0062] Figure 5 An optional structural diagram of an edge detection module provided in an embodiment of the present application;

[0063] Figure 6 A schematic diagram of another optional structure of the edge detection module provided in the embodiment of the present application;

[0064] Figure 7 Another optional structural diagram of the processing model provided in the embodiment of the present application;

[0065] Figure 8 Another optional structural diagram of the processing model provided in the embodiment of the present application;

[0066] Fig. 9 An optional structural diagram of an instance classification module provided in an embodiment of the present application;

[0067] Fig.10 An optional schematic diagram of a sample road image provided in an embodiment of the present application;

[0068] Fig.11 An optional schematic diagram of a road edge mask image corresponding to a road segment edge provided in an embodiment of the present application;

[0069] Fig.12 An optional schematic diagram of a road edge mask image corresponding to a closed edge of a road section provided in an embodiment of the present application;

[0070] Fig.13 An optional schematic diagram of a road edge mask image corresponding to an intersection surface edge provided in an embodiment of the present application;

[0071] Fig.14 An optional schematic diagram of a road surface mask image corresponding to a road surface provided in an embodiment of the present application;

[0072] Fig.15 A schematic diagram of an optional overall architecture of a road image detection method provided in an embodiment of the present application;

[0073] Fig.16 An optional structural diagram of a processing model provided in an embodiment of the present application;

[0074] Fig.17 A schematic diagram of an optional process for correcting the edge of a target road provided in an embodiment of the present application;

[0075] Fig.18 Another optional flow chart of the road image detection method provided in the embodiment of the present application;

[0076] Fig.19 Another optional schematic diagram of the road image detection method provided in the embodiment of the present application;

[0077] Fig. 20 A schematic diagram of an optional structure of a road image detection device provided in an embodiment of the present application;

[0078] Fig.21 A schematic diagram of another optional structure of the road image detection device provided in an embodiment of the present application;

[0079] Fig. 22 A partial structural block diagram of a terminal provided in an embodiment of the present application;

[0080] Fig.23 A partial structural block diagram of the server provided in an embodiment of the present application. DETAILED DESCRIPTION

[0081] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0082] It should be noted that in each specific embodiment of the present application, when it comes to the need to perform relevant processing based on data related to the characteristics of the target object such as target object attribute information or attribute information set, the permission or consent of the target object will be obtained first, and the collection, use and processing of these data will comply with relevant laws, regulations and standards. Among them, the target object can be a user. In addition, when the embodiment of the present application needs to obtain the target object attribute information, the target object's separate permission or separate consent will be obtained by means of a pop-up window or jumping to a confirmation page, etc., and after the separate permission or separate consent of the target object is clearly obtained, the necessary target object-related data for enabling the normal operation of the embodiment of the present application will be obtained.

[0083] To facilitate understanding of the technical solutions provided in the embodiments of the present application, some key terms used in the embodiments of the present application are explained here:

[0084] Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making.

[0085] Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. Basic artificial intelligence technologies generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operating / interactive systems, mechatronics and other technologies. Artificial intelligence software technologies mainly include computer vision technology, speech processing technology, natural language processing technology, as well as machine learning / deep learning, autonomous driving, smart transportation and other major directions.

[0086] Machine Learning (ML) is a multi-disciplinary subject that involves probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and other disciplines. It specializes in studying how computers simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications are spread across all areas of artificial intelligence. Machine learning and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and teaching learning.

[0087] Intelligent Traffic System (ITS), also known as Intelligent Transportation System, effectively and comprehensively applies advanced science and technology (information technology, computer technology, data communication technology, sensor technology, electronic control technology, automatic control theory, operations research, artificial intelligence, etc.) to transportation, service control and vehicle manufacturing, strengthens the connection between vehicles, roads and users, and thus forms a comprehensive transportation system that ensures safety, improves efficiency, improves the environment and saves energy.

[0088] Intelligent Vehicle Infrastructure Cooperative Systems (IVICS), referred to as IVICS, is a development direction of Intelligent Transportation Systems (ITS). IVICS uses advanced wireless communications and new generation Internet technologies to implement all-round dynamic real-time information interaction between vehicles and roads, and conducts active vehicle safety control and road cooperative management based on the collection and integration of dynamic traffic information in all time and space, fully realizing the effective coordination of people, vehicles and roads, ensuring traffic safety, and improving traffic efficiency, thus forming a safe, efficient and environmentally friendly road traffic system.

[0089] At present, the relevant technology's detection methods for road images are too crude, resulting in incomplete road deconstruction information. Therefore, it is necessary to improve the level of refinement of road image detection.

[0090] Based on this, the embodiments of the present application provide a road image detection method, device, electronic device and storage medium, which can improve the level of refinement of road image detection.

[0091] Reference Figure 1 , Figure 1 A schematic diagram of an optional implementation environment provided for an embodiment of the present application, the implementation environment includes a first terminal 101.

[0092] Exemplarily, the first terminal 101 can obtain a target road image, input the target road image into a processing model, perform feature extraction on the target road image based on a feature extraction module of the processing model to obtain target image features of the target road image, and map the target image features respectively based on multiple edge detection modules of the processing model to obtain detection results of the target road edges output by each edge detection module, extract the target road edge from the target road image based on the detection results of multiple target road edges, and the obtained target road edge can be used by the first terminal 101 to determine the passable area and the position and posture of the vehicle body relative to the road edge, thereby realizing functions such as vehicle navigation, lane departure warning, lane keeping and adaptive cruise.

[0093] In addition, refer to Figure 2 , Figure 2 A schematic diagram of another optional implementation environment provided for an embodiment of the present application, the implementation environment includes a second terminal 201 and a data processing server 202, wherein the second terminal 201 and the data processing server 202 are connected via a communication network.

[0094] Exemplarily, the data processing server 202 can obtain a target road image, perform feature extraction on the target road image based on a feature extraction module of a processing model, and obtain target image features of the target road image. Then, based on multiple edge detection modules of the processing model, the target image features are respectively mapped to obtain detection results of target road edges output by each edge detection module, wherein the target road edge is the edge of the road surface in the target road image, and the edge types of each target road edge are different. Then, based on the detection results of multiple target road edges, the target road edge is extracted from the target road image to obtain a road edge extraction result, and the road edge extraction result is sent to the second terminal 201, so that the second terminal 201 can determine the passable area and the position and posture of the vehicle body relative to the road edge based on the road edge extraction result, thereby realizing functions such as vehicle navigation, lane departure warning, lane keeping, and adaptive cruise.

[0095] By introducing a processing model, a feature extraction module based on the processing model extracts features of a target road image to obtain target image features of the target road image, and then multiple edge detection modules based on the processing model map the target image features respectively, so that different information can be learned from the target image features according to multiple edge detection modules respectively, and then the detection results of target road edges of different edge types output by each edge detection module can be obtained, and the target road edges can be extracted from the target road image. Therefore, based on the processing model, the detection of road edges of the target road image can be realized, and multiple types of road edges can be detected, thereby enriching the types of road deconstruction information obtained by the detection, and then the refinement of the detection of road images can be improved. In addition, by integrating multiple edge detection modules into one processing model, on the one hand, the detection efficiency of road images can be effectively improved, and on the other hand, since multiple edge detection modules share target image features, the learning between multiple edge detection modules can be correlated, thereby improving the detection performance of the processing model as a whole.

[0096] The first terminal 101 and the second terminal 201 can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, a vehicle terminal, etc., but are not limited thereto. The second terminal 201 and the server 202 can be directly or indirectly connected via wired or wireless communication, which is not limited in the embodiments of the present application.

[0097] Server 202 may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. In addition, server 202 may also be a node server in a blockchain network.

[0098] The method provided in the embodiments of the present application can be applied to various technical fields, including but not limited to cloud technology, intelligent transportation, intelligent driving, maps, navigation and other technical fields.

[0099] The principles of the road image detection method provided in the embodiments of the present application are described in detail below.

[0100] Reference Figure 3 , Figure 3 An optional flow chart of a road image detection method provided in an embodiment of the present application, the road image detection method can be executed by a server, or can also be executed by a terminal, or can also be executed by a server and a terminal in cooperation, in the embodiment of the present application, the method is executed by a server as an example for explanation. The road image detection method includes but is not limited to the following steps 301 to 303.

[0101] Step 301: Acquire a target road image, input the target road image into a processing model, perform feature extraction on the target road image based on a feature extraction module of the processing model, and obtain target image features of the target road image.

[0102] The target road image is an image to be detected, which is used to reflect the road and the scene around the road. For example, the target road image may include elements such as roads, buildings, pedestrians, and vehicles. In addition, the target road image may be an ordinary image or a remote sensing image. The ordinary image does not include coordinate information, while the remote sensing image includes coordinate information. Based on the high resolution of the remote sensing image and the characteristics of including coordinate information, the accuracy of detection can be effectively improved.

[0103] The processing model is used to detect and process the target road image. Figure 4 , Figure 4 An optional structural diagram of a processing model provided for an embodiment of the present application, wherein the processing model is provided with a feature extraction module and multiple edge detection modules, wherein the feature extraction module is used to extract features of a target road image; the target image features are used to characterize the features of the target road image, and the target image features extracted by the feature extraction module can be represented by vectors.

[0104] In a possible implementation, the target road image may be obtained by the server obtaining the target road image uploaded by the terminal, or by the server obtaining the target road image stored locally, or by one server obtaining the target road image transmitted by another server, and so on.

[0105] In one possible implementation, the target road image may be uploaded by the vehicle-mounted terminal, and accordingly, the target road image may be acquired by the vehicle's image acquisition device and uploaded by the vehicle-mounted terminal, wherein the image acquisition device may be a camera or a radar device such as a laser radar or a millimeter-wave radar; or, it may be acquired based on a map application running on the vehicle's vehicle-mounted terminal and uploaded by the vehicle-mounted terminal; or, it may be acquired by a map application running on an intelligent terminal (such as a smart phone) located on the vehicle and transmitted to the vehicle-mounted terminal and uploaded by the vehicle-mounted terminal; or, it may be acquired by a remote sensor provided on a remote sensing platform such as a remote sensing vehicle, ship or aircraft and transmitted to the vehicle-mounted terminal and uploaded by the vehicle-mounted terminal; or, it may be acquired by an intelligent vehicle-road cooperative system and transmitted to the vehicle-mounted terminal and uploaded by the vehicle-mounted terminal, and so on.

[0106] In one possible implementation, a feature extraction module based on a processing model performs feature extraction on a target road image to obtain target image features of the target road image. Specifically, the feature extraction module based on the processing model downsamples the target road image to obtain downsampled image features of the target road image, and then upsamples the downsampled image features based on the feature extraction module to obtain target image features of the target road image.

[0107] Among them, the downsampled image feature is a low-resolution feature extracted from the target road image, that is, an image feature with an image resolution less than a preset resolution. Correspondingly, the target image feature is used to indicate the high-resolution feature extracted from the downsampled image feature.

[0108] In an embodiment of the present application, by introducing a processing model, a feature extraction module based on the processing model performs down-sampling processing on the target road image, which can filter out redundant features in the target road image, thereby obtaining the down-sampled image features of the target road image; and based on the feature extraction module, up-sampling processing is performed on the down-sampled image features, which can amplify the down-sampled image features to make them into image features with higher resolution, thereby obtaining the target image features of the target road image. Therefore, based on the up-sampling and down-sampling processing of the feature extraction module, the feature extraction module can enhance its perception of texture details of different scales in the target road image, thereby more smoothly segmenting features of different resolutions in the target road image.

[0109] Step 302: Multiple edge detection modules based on the processing model respectively map the target image features to obtain detection results of the target road edges output by each edge detection module.

[0110] Among them, the edge detection module is used to perform edge detection on the target road image. The target road edge is the edge of the road surface in the target road image, that is, the target road edge is the boundary between the road surface and other elements in the target road image or the image edge of the target road image. The road surface is used to indicate the shape of the road in the target road image.

[0111] Among them, the edge types of each target road edge are different from each other, that is, target road edges of different edge types. For example, the target road edge can be a section edge of the road surface, and the section edge of the road surface is the edge of the main body of the road surface, or the target road edge can also be an intersection surface edge of the road surface, and the intersection surface edge is the edge of two roads in different directions in the road surface, or the target road edge can also be a section closed edge of the road surface, and the section closed edge is the part of the image edge of the target road image located in the road surface.

[0112] Correspondingly, the image processing functions of the multiple edge detection modules are also different from each other, for example, an edge detection module for detecting the edge of a road section, or an edge detection module for detecting the edge of a closed road section, or an edge detection module for detecting the edge of an intersection surface, etc. It can be understood that the multiple edge detection modules based on the processing model map the target image features respectively, and the multiple edge detection modules can map the target image features in parallel, thereby improving the efficiency of edge detection.

[0113] It is understandable that the number of edge detection modules is at least two, and accordingly, the detection result of the target road edge detected by the processing model can be at least two of the detection result of the road segment edge, the detection result of the intersection surface edge, and the detection result of the road segment closed edge. For example, the number of edge detection modules is three, then, in a possible implementation, the target image features are mapped based on multiple edge detection modules of the processing model respectively to obtain the detection result of the target road edge output by each edge detection module, specifically, the target image features are mapped based on the edge detection module used to detect the road segment edge in the processing model to obtain the detection result of the road segment edge, the target image features are mapped based on the edge detection module used to detect the intersection surface edge in the processing model to obtain the detection result of the intersection surface edge, and the target image features are mapped based on the edge detection module used to detect the road segment closed edge in the processing model to obtain the detection result of the road segment closed edge.

[0114] Among them, the processing model detects the detection results of the road section edge, the intersection surface edge and the road section closed edge respectively by setting an edge detection module for detecting the road section edge, an edge detection module for detecting the intersection surface edge and the road section closed edge. It can completely detect the edge of the road surface based on the three edges of the road section edge, the intersection surface edge and the road section closed edge, and subdivide them into the road section edge, the intersection surface edge and the road section closed edge. It can perform different local detections on the edge of the road surface in a refined manner, thereby effectively reducing the occurrence of edge missing.

[0115] In one possible implementation, referring to Figure 5 , Figure 5 An optional structural diagram of an edge detection module provided in an embodiment of the present application, wherein the edge detection module is provided with a fully connected layer, an activation function, a fully connected layer and a normalization layer, wherein the fully connected layer is used to perform fully connected processing on target image features or activation features, the activation feature is the feature obtained after activation processing using the activation function; the activation function is used to activate the features output by the fully connected layer; the normalization layer is used to normalize the features output by the fully connected layer.

[0116] In one possible implementation, referring to Figure 6 , Figure 6 Another optional structural diagram of an edge detection module provided in an embodiment of the present application, the edge detection module is provided with a fully connected layer, an activation function, a fully connected layer, an activation function, a fully connected layer and a normalization layer.

[0117] Among them, the detection result of the target road edge can be the classification result of various types of target road edges corresponding to each pixel point in the target road image, and the classification result of the target road edge can be a binary classification result, that is, whether a certain pixel point belongs to the target road edge detected by the corresponding edge detection module. The specific expression form of the detection result of the target road edge can be a matrix including the classification results of each pixel point.

[0118] Taking the edge detection module used to detect the edge of a road section as an example, the detection result it outputs is a matrix of whether each pixel point in a target road image belongs to the edge of a road section. Therefore, the pixel points belonging to the edge of the road section and the pixel points that do not belong to the edge of the road section in the target road image can be obtained. Then, based on the pixel points belonging to the edge of the road section, the edge of the road section in the target road image can be determined.

[0119] Step 303: extracting the target road edge from the target road image according to the detection results of the multiple target road edges.

[0120] In a possible implementation, the target road edge is actually a collection of corresponding pixel points. After the detection result of the target road edge is obtained, it is actually determined which type of target road edge each pixel point belongs to. Therefore, the pixel points corresponding to each target road edge can be further extracted, thereby obtaining target road edges of multiple edge types. In addition, the color of each corresponding pixel point can be changed according to different types of target road edges, and the target road image can be annotated to make the target road edge stand out in the target road image.

[0121] In the road image detection method including steps 301 to 303, by introducing a processing model, a feature extraction module based on the processing model performs feature extraction on the target road image to obtain target image features of the target road image, and then multiple edge detection modules based on the processing model respectively map the target image features, so that different information can be learned from the target image features according to the multiple edge detection modules respectively, and then target road edges of different edge types output by each edge detection module can be obtained. Therefore, based on the processing model, the target road image can be detected, and multiple types of road edges can be detected, thereby enriching the types of road deconstruction information obtained by the detection, and then the degree of refinement of the road image detection can be improved.

[0122] In addition, by integrating multiple edge detection modules into one processing model, on the one hand, the detection efficiency of road images can be effectively improved; on the other hand, since multiple edge detection modules share target image features, the learning between multiple edge detection modules can be correlated, thereby improving the detection performance of the processing model as a whole.

[0123] In one possible implementation, referring to Figure 7 , Figure 7 Another optional structural diagram of the processing model provided in the embodiment of the present application, the processing model may further include a road surface detection module, wherein the road surface detection module is used to perform road surface detection on the target road image.

[0124] In one possible implementation, the structure of the road surface detection module is similar to that of the edge detection module. The road surface detection module can also be provided with a fully connected layer, an activation function, a fully connected layer and a normalization layer, wherein the fully connected layer, the activation function, the fully connected layer and the normalization layer are connected in a fully connected manner; or, the road surface detection module can also be provided with a fully connected layer, an activation function, a fully connected layer, an activation function, a fully connected layer and a normalization layer, wherein the fully connected layer, the activation function, the fully connected layer, the activation function, the fully connected layer and the normalization layer are connected in a fully connected manner.

[0125] Based on this, the target image features can be mapped based on the road surface detection module of the processing model to obtain the detection result of the target road surface output by the road surface detection module. The detection result of the target road surface can be used to indicate the set of pixel points of the target road surface in the target road image, and the target road surface is the road surface in the target road image.

[0126] Among them, the road surface detection module based on the processing model maps the target image features to obtain the detection result of the target road surface output by the road surface detection module. Specifically, the road surface detection module based on the processing model performs full-connection processing on the target image features to obtain a second fully-connected feature, the road surface detection module performs activation processing on the second fully-connected feature to obtain an activated feature, the road surface detection module performs full-connection processing on the activated feature to obtain a third fully-connected feature, and the road surface detection module performs normalization processing on the third fully-connected feature to obtain the detection result of the target road surface output by the road surface detection module.

[0127] The road surface detection module based on the processing model performs full connection processing on the target image feature, that is, the road surface detection module performs full connection processing on the target image feature based on the fully connected layer; the road surface detection module performs activation processing on the second fully connected feature, that is, the activation function of the road surface detection module performs activation processing on the second fully connected feature; the road surface detection module performs full connection processing on the activated feature, that is, the road surface detection module performs full connection processing on the activated feature based on the fully connected layer; the road surface detection module performs normalization processing on the third fully connected feature, that is, the normalization layer of the road surface detection module performs normalization processing on the third fully connected feature. The activation function can be sigmoid, tanh or ReLU, etc.

[0128] In the embodiment of the present application, by introducing a fully connected layer, the loss of target image feature information can be reduced, and by introducing an activation function, the learning ability of the processing model can be enhanced and the detection efficiency of the processing model can be improved.

[0129] By introducing a road surface detection module to detect the target road surface, the target road image can be detected more finely, and by integrating the road surface detection module with multiple edge detection modules into one processing model, on the one hand, the detection efficiency of the road image can be effectively improved, and on the other hand, since the road surface detection module and the multiple edge detection modules share the target image features, the learning between the road surface detection module and the multiple edge detection modules can be further correlated, thereby improving the detection performance of the processing model as a whole.

[0130] In one possible implementation, referring to Figure 8 , Figure 8Another optional structural diagram of the processing model provided in the embodiment of the present application, the processing model may also include an instance classification module, the instance classification module is used to classify instances in the target road scene, the instances in the target road image are object elements in the target road image, such as roads, buildings, pedestrians, vehicles, etc., all of which belong to instances in the target road image.

[0131] Based on this, multiple edge detection modules based on the processing model map the target image features respectively to obtain the detection results of the target road edges output by each edge detection module. Specifically, the target image features can be mapped respectively based on the multiple edge detection modules of the processing model to obtain the detection results of the candidate edges output by each edge detection module, and then the target image features can be mapped based on the instance classification module of the processing model to obtain the target classification results of various instances in the target road image output by the instance classification module. Finally, the detection results of the target road edges are determined based on the target classification results and the detection results of the candidate edges.

[0132] It can be seen that the edge detection module can be reused, that is, the edge detection module can be used to perform edge detection on different instances in the target road image. Therefore, the same edge detection module can output edges of different instances.

[0133] For example, an edge detection module used to detect the edges of road sections can also be used to detect the edges of building bodies; or an edge detection module used to detect the edges of intersections on road surfaces can also be used to detect edges connecting different building bodies; an edge detection module used to detect the closed edges of road sections can also be used to detect the closed edges of buildings, and so on.

[0134] It can be seen that the candidate edges include edges of different instances of the same edge type in the target road image. The detection results of the candidate edges can be used to indicate the classification results of various types of candidate edges corresponding to each pixel point in the target road image. The classification result of the candidate edges can be whether a certain pixel point belongs to the candidate edge detected by the corresponding edge detection module.

[0135] Taking the edge detection module used to detect the edges of road sections and the main edges of buildings as an example, the detection result it outputs is a matrix of whether each pixel point in a target road image belongs to a candidate edge. Therefore, the pixel points in the target road image that belong to the candidate edges and the pixel points that do not belong to the candidate edges can be obtained. Based on the pixel points belonging to the candidate edges, the candidate edges in the target road image can be determined. At this time, the candidate edges include the edges of road sections and the main edges of buildings.

[0136] The target classification result of the instance can be used to indicate the category of various types of instances corresponding to each pixel point in the target road image. Since the candidate edges output by the same edge detection module include the edges of different instances, in an embodiment of the present application, the target image features are mapped by an instance classification module based on the processing model to obtain the target classification results of various instances in the target road image output by the instance classification module. The detection results of the candidate edges can be classified according to the target classification results, and then the detection results of the target road edges can be determined. When the edge detection module is reused, the detection results of the target road edges can be accurately determined from the detection results of the candidate edges.

[0137] Exemplarily, assuming that the real target road image includes building instances and road instances, the instance classification module can map the target image features to obtain the target classification results of each pixel belonging to the building instance and the target classification results of the road instance, and then map the target image features based on multiple edge detection modules respectively to obtain the detection results of the candidate edges output by each edge detection module. For one of the edge detection modules, the pixel points belonging to the main edge of the building and the edge of the road section can be detected at the same time. At this time, the target classification results of the building instance and the target classification results of the road instance obtained by the instance classification model for the pixel point can be further determined to determine whether the pixel point belongs to the main edge of the building or the edge of the road section.

[0138] In one possible implementation, an instance classification module based on a processing model maps target image features to obtain target classification results of various instances in a target road image output by the instance classification module. Specifically, the target image features may be fully connected based on the instance classification module of the processing model to obtain a first fully connected feature, and then the first fully connected feature may be normalized based on the instance classification module to obtain target classification results of various instances in the target road image output by the instance classification module.

[0139] Among them, refer to Fig. 9 , Fig. 9 An optional structural diagram of an instance classification module provided in an embodiment of the present application, the instance classification module may include a fully connected layer and a normalization layer, the fully connected layer and the normalization layer are connected in a fully connected manner, the instance classification module based on the processing model performs fully connected processing on the target image features, that is, the target image features are fully connected based on the fully connected layer; and the first fully connected feature is normalized based on the instance classification module, that is, the first fully connected feature is normalized based on the normalization layer (for example, it can be a softmax layer). In addition, the instance classification module may also include multiple groups of fully connected layers and normalization layers, which can be set according to actual conditions, and the embodiment of the present application does not make specific restrictions on this.

[0140] In an embodiment of the present application, by introducing an instance classification module and performing full-connection processing on the target image features based on the instance classification module, the target image features can be classified at the pixel level, and then the first fully-connected features are normalized based on the instance classification module, so that the probability of each pixel belonging to various instances can be obtained, thereby obtaining the target classification results of various instances.

[0141] In one possible implementation, the processing model may be pre-trained. When training the processing model, specifically, a sample road image and road edge mask images of multiple different edge types corresponding to the sample road image may be obtained, and then multiple sample road edges of different edge types of the sample road image may be determined based on the processing model. Then, the edge detection loss may be determined based on the sample road edges and the corresponding road edge mask images, and finally, the processing model may be trained based on the edge detection loss.

[0142] The sample road image is a sample image used as a processing model training image; the road edge mask image is an image used to block the road edge in the sample road image to control the area of ​​image processing. The road edge mask image can be a road edge mask image of edge types such as a road segment edge, a road segment closed edge, or a road intersection edge. For example, Figures 10 to 13 ,in, Fig.10 An optional schematic diagram of a sample road image provided in an embodiment of the present application, Fig.11 An optional schematic diagram of a road edge mask image corresponding to a road segment edge provided in an embodiment of the present application, Fig.12 An optional schematic diagram of a road edge mask image corresponding to a closed edge of a road section provided in an embodiment of the present application, Fig.13 This is an optional schematic diagram of a road edge mask image corresponding to the intersection surface edge provided in the embodiment of the present application. Fig.10 The road image shown in FIG. 1 is a road edge mask image corresponding to the road segment edge in the sample road image. Fig.11 As shown in the figure, the road edge mask image corresponding to the closed edge of the road segment in the sample road image is as follows: Fig.12 As shown in the figure, the road edge mask image corresponding to the intersection edge in the sample road image is as follows: Fig.13 As shown, Fig.11 The white lines in the figure represent the edge of the road segment. Fig.12 The white lines in the figure represent the closed edges of the road segments. Fig.13 The white lines in the figure represent the edges of the intersection faces.

[0143] The sample road edge is a set of pixel points of the edge of the road surface extracted from the sample road image, wherein the sample road edge can be a road segment edge, or a road intersection edge, or a road segment closed edge; the edge detection loss is used to measure the similarity between the sample road edge and the corresponding road edge mask image. Accordingly, the detection result of the sample road edge can be the classification result of each pixel point in the sample road image corresponding to various types of target road edges.

[0144] In an embodiment of the present application, by training a processing model based on edge detection loss, the negative impact caused by the imbalance between the number of pixels at the edge of the road and the number of other pixels in the training samples can be alleviated, and the degree of fineness of image segmentation can be optimized.

[0145] In one possible implementation, the edge detection loss is determined based on the sample road edge and the corresponding road edge mask image. Specifically, the number of identical pixels between the sample road edge and the road edge mask image is counted, recorded as the first number, and the sum of the number of pixels in the sample road edge and the number of pixels in the road edge mask image is calculated, recorded as the second number. A loss coefficient is obtained based on the first number and the second number. Finally, the difference between 1 and the loss coefficient is calculated to obtain the pixel loss, and the pixel loss is used as the edge detection loss.

[0146] On this basis, each pixel point in the sample road edge can also include an edge category label. After obtaining the detection result of the sample road edge, the cross entropy loss can be calculated based on the detection result of the sample road edge and the edge category label. The sum of the pixel loss and the cross entropy loss is used as the edge detection loss, thereby improving the accuracy of the edge detection loss.

[0147] In one possible implementation, different edge detection modules correspond to different edge detection losses. When training a processing model based on the edge detection loss, the edge detection losses corresponding to each edge detection module can be weighted to obtain a target loss, and then the processing model can be trained based on the target loss. Therefore, the learning between different edge detection modules can be correlated.

[0148] In one possible implementation, when the processing model includes a road surface detection module, when training the processing model, a road surface mask image corresponding to the sample road image can also be obtained, and then the sample road surface of the sample road image is determined based on the processing model, and then the road surface detection loss is determined based on the sample road surface and the corresponding road surface mask image, and finally the processing model is trained based on the road surface detection loss.

[0149] The road surface mask image is an image used to block the road surface in the sample road image to control the area of ​​image processing. Fig.14 , Fig.14 An optional schematic diagram of a road surface mask image corresponding to a road surface provided in an embodiment of the present application, wherein: Fig.14 The white area in represents the road surface. The sample road surface is used to indicate the set of pixel points of the road surface extracted from the sample road image; the road surface detection loss is used to measure the similarity between the sample road surface and the road surface mask image.

[0150] Similarly, in an embodiment of the present application, by training the processing model based on the road surface detection loss, the negative impact caused by the imbalance between the number of road surface pixels and the number of other pixels in the training samples can be alleviated, and the degree of fineness of image segmentation can be optimized.

[0151] In one possible implementation, the road surface detection loss is determined based on the sample road surface and the road surface mask image. Specifically, the number of identical pixels between the sample road surface and the road surface mask image is counted, recorded as the third number, and the sum of the number of pixels in the sample road surface and the number of pixels in the road surface mask image is calculated, recorded as the fourth number. The loss coefficient is obtained based on the third number and the fourth number, and finally the difference between 1 and the loss coefficient is calculated to obtain the pixel loss, and the pixel loss is used as the road surface detection loss.

[0152] On this basis, each pixel point in the sample road surface can also include a road surface category label. After obtaining the detection result of the sample road surface, the cross entropy loss can be calculated based on the detection result of the sample road surface and the road surface category label. The sum of the pixel loss and the cross entropy loss is used as the road surface detection loss, thereby improving the accuracy of the road surface detection loss.

[0153] In one possible implementation, the processing model can also be trained based on the road surface detection loss and the edge detection loss. For example, the road surface detection loss and the edge detection loss can be weighted to obtain a target loss value, and then the processing model can be trained based on the target loss. Therefore, the learning between the road surface detection module and different edge detection modules can be correlated.

[0154] In one possible implementation, when the processing model includes an instance classification module, the sample road image can be annotated with category labels of various instances, and the processing model can be trained based on the edge detection loss. Specifically, the sample classification results of various instances in the sample road image can be determined based on the processing model, the classification loss can be determined based on the sample classification results and the category labels, and then the processing model can be trained based on the edge detection loss and the classification loss.

[0155] Among them, the category label of the instance can be used to indicate the category of each instance in the sample road image; the sample classification result of the instance is used to indicate the category of each pixel in the sample road image corresponding to each type of instance; the classification loss can be used to measure the accuracy of classifying various instances in the sample road image. It can be understood that the smaller the value of the classification loss, the smaller the difference between the category label and the sample classification result, that is, the more accurate the classification of the instance; the larger the value of the classification loss, the greater the difference between the category label and the sample classification result, that is, the less accurate the classification of the instance.

[0156] In the embodiment of the present application, by training the processing model according to the edge detection loss, the negative impact caused by the imbalance between the number of road pixels and the number of non-road pixels in the training samples can be alleviated. By training the processing model with the classification loss, the classification effect of the processing model can be improved. Combining the edge detection loss and the classification loss to train the processing model can effectively improve the stability of the training process and the precision of the image segmentation.

[0157] In one possible implementation, the classification loss is determined based on the sample classification results and the category labels. Specifically, the sample classification results are normalized using a sigmoid function (or a softmax function) to obtain a predicted value, and then the number of sample classification results (that is, the number of categories of all instances in the sample road image) is determined. Then, the classification loss is calculated based on the number of sample classification results, the predicted value of the sample classification result, and the corresponding category label.

[0158] In one possible implementation, the classification loss and the edge detection loss may be weighted to obtain a target loss value, and then the processing model may be trained based on the target loss. Thus, the learning between the instance classification module and different edge detection modules may be correlated.

[0159] In one possible implementation, the processing model can also be trained based on the classification loss, road surface detection loss and edge detection loss. For example, the classification loss, road surface detection loss and edge detection loss can be weighted to obtain a target loss value, and then the processing model can be trained based on the target loss. Therefore, the learning between the instance classification module, the road surface detection module and the different edge detection modules can be correlated.

[0160] In one possible implementation, after extracting the target road edge, the target image features can be further mapped based on the road surface detection module of the processing model to obtain the detection result of the target road surface output by the road surface detection module, and then the target road surface is extracted from the target road image based on the detection result of the target road surface, and the contour of the target road surface is extracted to obtain the road surface contour of the target road surface, and finally the target road edge is corrected according to the road surface contour.

[0161] Among them, the road surface contour is the contour of the target road surface, which is equivalent to the complete edge of the target road surface. Similarly, the road surface contour is actually a collection of corresponding pixel points. Since the target road edge obtained by the edge detection module may be incomplete, and the road surface contour is equivalent to the complete edge of the target road surface, the target road edge can be corrected according to the road surface contour, thereby improving the integrity of the target road edge.

[0162] In one possible implementation, the target road edge is corrected according to the road surface contour, specifically, the difference between the road surface contour and each target road edge is determined to obtain a first edge set that is not successfully detected by the edge detection module, the intersection between the road surface contour and each target road edge is determined to obtain a second edge set that is successfully detected by the edge detection module, the first road edge in the first edge set is traversed, at least one second road edge adjacent to the first road edge is determined from the second edge set, the edge type of the first road edge is re-determined according to the edge type of the second road edge, and the target road edge is corrected according to the first road edge after the edge type is re-determined.

[0163] The first edge set includes edges that are not successfully detected by the edge detection module, and specifically records the pixel points involved in the edges that are not successfully detected by the edge detection module. The edge that is not successfully detected by the edge detection module can be a complete target road edge, or a part of a target road edge. Similarly, the second edge set includes edges that are successfully detected by the edge detection module, and specifically records the pixel points involved in the edges that are successfully detected by the edge detection module. The edge that is successfully detected by the edge detection module can be a complete target road edge, or a part of a target road edge.

[0164] Since the second edge set includes edges successfully detected by the edge detection module, the edge types of the edges in the first edge set may be re-determined according to the edge types of the edges in the second edge set.

[0165] In one possible implementation, at least one second road edge adjacent to the first road edge is determined from the second edge set, and the edge type of the first road edge is re-determined based on the edge type of the second road edge. Specifically, a second road edge adjacent to the first road edge may be determined from the second edge set, and the edge type of the first road edge may be re-determined based on the edge type of the second road edge.

[0166] Among them, a second road edge adjacent to the first road edge can be a second road edge that is closest to the first road edge, for example, it can be a second road edge that is closest to and adjacent to the endpoint of the first road edge; or, it can be a second road edge that is closest to and adjacent to the midpoint of the first road edge; or, it can be a second road edge that is closest to and adjacent to any point in the first road edge, and so on.

[0167] Among them, a second road edge adjacent to the first road edge is determined from the second edge set, and the edge type of the first road edge is re-determined according to the edge type of the second road edge. The edge type of the second road edge can be directly used as the edge type of the first road edge.

[0168] In one possible implementation, in order to improve the accuracy of detecting the edge type of the first road edge, two second road edges adjacent to the first road edge can be determined from the second edge set, and then the edge type of the first road edge can be re-determined based on the edge types of the two second road edges.

[0169] Among them, when the number of second road edges is two, the edge type of the first road edge is re-determined according to the edge type of the second road edges. Specifically, the edge types of the two second road edges may be compared. When the comparison result indicates that the edge types of the two second road edges are the same, the edge type of the second road edge is used as the edge type of the first road edge; or, when the comparison result indicates that the edge types of the two second road edges are different, the edge type of the first road edge is determined as a road segment edge.

[0170] For example, when the comparison result indicates that the edge types of the two second road edges are both segment closed edges, then the edge type of the first road edge is a segment closed edge; or, when the comparison result indicates that the edge types of the two second road edges are both intersection surface edges, then the edge type of the first road edge is an intersection surface edge; or, when the comparison result indicates that the edge types of the two second road edges are both segment edges, then the edge type of the first road edge is a segment edge, and so on.

[0171] It can be understood that when the comparison result indicates that the edge types of the two second road edges are different and the edge type of the first road edge is determined to be a segment edge, regardless of whether the edge types of the two second road edges are the same, the edge type of the first road edge is a segment edge.

[0172] In one possible implementation, when the comparison result indicates that the edge types of the two second road edges are different, the edge type of the first road edge can also be determined as a road segment closed edge, or the edge type of the first road edge can also be determined as an intersection surface edge, etc. The embodiment of the present application does not impose specific restrictions on this.

[0173] In one possible implementation, after obtaining the target road edge, map data can be generated based on the target road edge and stored. Since the road image detection method provided in the embodiment of the present application can finely detect a variety of different target road edges, the generated map data can contain more road information, thereby making the map data more refined, accurate and reliable.

[0174] In addition, after obtaining the target road edge, a lane-level road interface can also be rendered based on the target road edge, and the rendered lane-level road interface can be a navigation interface, or can also be an auxiliary driving interface, etc. Similarly, since the road image detection method provided in the embodiment of the present application can finely detect a variety of different target road edges, the generated lane-level road interface can contain more road information, so that the display elements of the lane-level road interface are more refined, more accurate and reliable.

[0175] The principle of the road image detection method provided in the embodiment of the present application is described in detail below.

[0176] Reference Fig.15 , Fig.15 An optional overall architecture diagram of the road image detection method provided in the embodiment of the present application. First, a target road image is acquired, and then the target road image is input into a processing model, and a feature extraction module based on the processing model performs downsampling processing on the target road image to obtain downsampled image features of the target road image, and then upsamples the downsampled image features based on the feature extraction module to obtain target image features of the target road image.

[0177] Next, the edge detection module for detecting the edge of the road section, the edge detection module for detecting the closed edge of the road section, and the edge detection module for detecting the edge of the intersection surface based on the processing model respectively map the target image features to obtain the detection results of the candidate edges output by each edge detection module. Then, the instance classification module based on the processing model performs full connection processing on the target image features to obtain the first fully connected feature, and the first fully connected feature is normalized based on the instance classification module to obtain the target classification results of various instances in the target road image output by the instance classification module. Then, the detection results of the road section edge, the detection results of the closed edge of the road section, and the detection results of the intersection surface edge are determined based on the target classification results and the detection results of the candidate edges.

[0178] Then, according to the detection results of the road segment edge, the detection results of the road segment closed edge and the detection results of the intersection surface edge, the road segment edge, the road segment closed edge and the intersection surface edge are extracted from the target road image.

[0179] Furthermore, the target image feature can be fully connected based on the road surface detection module of the processing model to obtain a second fully connected feature, the second fully connected feature can be activated based on the road surface detection module to obtain an activated feature, the activated feature can be fully connected based on the road surface detection module to obtain a third fully connected feature, and the third fully connected feature can be normalized based on the road surface detection module to obtain a detection result of the target road surface output by the road surface detection module. Then, the target road surface is extracted from the target road image according to the detection result of the target road surface, and the contour of the target road surface is extracted to obtain a road surface contour of the target road surface.

[0180] Next, determine the difference between the road surface contour and the road segment edge, the road segment closed edge and the intersection surface edge to obtain a first edge set that is not successfully detected by the edge detection module, determine the intersection between the road surface contour and the road segment edge, the road segment closed edge and the intersection surface edge to obtain a second edge set that is successfully detected by the edge detection module, traverse the first road edge in the first edge set, determine at least one second road edge adjacent to the first road edge from the second edge set, and re-determine the edge type of the first road edge according to the edge type of the second road edge, wherein, in the case where the number of second road edges is two, the edge types of the two second road edges can be compared, and when the comparison result indicates that the edge types of the two second road edges are the same, the edge type of the second road edge is used as the edge type of the first road edge; or, when the comparison result indicates that the edge types of the two second road edges are different, the edge type of the first road edge is determined as the road segment edge.

[0181] Finally, the segment edge, segment closed edge and intersection face edge are corrected according to the first road edge after the edge type is re-determined, so that the target road face, segment edge, segment closed edge and intersection face edge in the target road image can be completely obtained.

[0182] The following is an actual example to illustrate the overall structure of the processing model provided in the embodiment of the present application.

[0183] In one possible implementation, referring to Fig.16 , Fig.16 An optional structural diagram of a processing model provided for an embodiment of the present application, the processing model includes a feature extraction module and a decoder, wherein the feature extraction module is provided with a backbone network and a pixel decoder, the decoder includes an instance classification module, multiple edge detection modules and a road surface detection module, the multiple edge detection modules are respectively an edge detection module for detecting the edge of a road section, an edge detection module for detecting the closed edge of a road section, and an edge detection module for detecting the edge of an intersection surface.

[0184] Assume that the instance in the target road image is a road, and the road is composed of a road surface, a road segment edge, a road segment closed edge, and a road intersection surface edge. Based on this, firstly, the backbone network based on the feature extraction module can downsample the target road image to obtain the downsampled image features of the target road image, and then the pixel decoder based on the feature extraction module can upsample the downsampled image features to obtain the target image features of the target road image.

[0185] Next, the target image features are mapped based on multiple edge detection modules to obtain the detection results of candidate edges output by each edge detection module, and the target image features are mapped based on the instance classification module to obtain the target classification result of the road in the target road image output by the instance classification module, and the detection result of the target road edge is determined based on the target classification result and the detection result of the candidate edge, wherein the target road edge includes the road segment edge, the road segment closed edge and the intersection surface edge. At the same time, the target image features are mapped based on the road surface detection module to obtain the detection result of the target road surface output by the road surface detection module.

[0186] It can be understood that the above-mentioned backbone network can be Resnet or SwinTransformer, etc. In addition, each module in the above-mentioned decoder belongs to a different neural network, which can learn different information from the target image features, that is, the instance classification module can learn the category of each instance, and multiple edge detection modules can respectively learn the road section edge, the road section closed edge and the intersection surface edge, and the road surface detection module can learn the road surface. The embodiment of the present application does not impose specific restrictions on this.

[0187] The following is an actual example to illustrate the process of correcting the target road edge provided by an embodiment of the present application.

[0188] In one possible implementation, referring to Fig.17 , Fig.17 An optional flow chart of correcting the target road edge provided in the embodiment of the present application. First, multiple edge detection modules based on the processing model respectively map the target image features, obtain the detection results of the target road edge output by each edge detection module, and extract the target road edge from the target road image based on the detection results of the multiple target road edges, wherein these target road edges are the road segment edges Borders T , Borders C Borders E , specifically, the set of road segment edges can be expressed as:

[0189] Borders T = {Border0,…,Border t ,…,Border T},(0≤t≤T);

[0190] Among them, Border0,…,Border t ,…,Border T Both represent the edge of the road segment.

[0191] The set of intersection face edges can be expressed as:

[0192] Borders C = {Border0,…,Border c ,…,Border C},(0≤c≤C);

[0193] Among them, Border0,…,Border c ,…,Border C Both represent the edge of the intersection surface.

[0194] The set of closed edges of a road segment can be expressed as:

[0195] Borders E = {Border0,…,Border e ,…,Border E},(0≤e≤E);

[0196] Among them, Border0,…,Border e ,…,Border EBoth represent the closed edges of road segments.

[0197] Furthermore, the road surface detection module based on the processing model maps the target image features to obtain the detection result of the target road surface output by the road surface detection module, and extracts the target road surface from the target road image according to the detection result of the target road surface, extracts the contour of the target road surface, and obtains the road surface contour Border of the target road surface. R .

[0198] Then, the difference between the road surface contour and each target road edge is determined, that is, the road surface contour Border is determined. R Borders T , Borders C And the closed edges of the road segment Borders E The difference between them is used to obtain the first edge set NoRecallBorders that is not successfully detected by the edge detection module. R ,Right now:

[0199] NoRecallBorders R ={NoRecallBorder0,…,NoRecallBorder r ,…,NoRecallBorder R} (0≤r≤R);

[0200] Next, determine the intersection between the road surface contour and each target road edge, that is:

[0201] Determine the road surface border R Borders T The intersection between the road surface contour and the road segment edge is obtained to obtain the second edge set PrecisionBorders T ,Right now

[0202] PrecisionBorders T ={PrecisionBorders0,…,PrecisionBorders t ,…,PrecisionBorders T} (0≤t≤T);

[0203] Determine the road surface border R Borders with intersection edges C The intersection between them obtains the second edge set PrecisionBorders of the road surface contour and the intersection surface edge C ,Right now

[0204] PrecisionBorders C ={PrecisionBorder0,…,PrecisionBorder c ,…,PrecisionBorder C} (0≤c≤C);

[0205] Determine the road surface border R Borders with closed edges of road segments E The intersection between the road surface contour and the closed edge of the road segment is obtained to obtain the second edge set PrecisionBorders E ,Right now

[0206] PrecisionBorders E ={PrecisionBorder0,…,PrecisionBorder e ,…,PrecisionBorder E} (0≤e≤E);

[0207] Further, traverse the first edge set NoRecallBorders R The first road edge in the PrecisionBorders collection from the second edge T 、PrecisionBorders C and PrecisionBorders E In the process, two second road edges adjacent to the first road edge are determined, and the two second road edges are respectively recorded as NoRecallBorder m and NoRecallBorder n .

[0208] Compare the edge types of the two second road edges, that is, determine whether the edge types of the two second road edges are the same. If the comparison result indicates NoRecallBorder m and NoRecallBorder n The same edge type as the second road edge (ie NoRecallBorder m or NoRecallBorder n ) as the first edge set NoRecallBorders R The edge type of the first road edge in the comparison; when the comparison result indicates NoRecallBorder m and NoRecallBordern The edge types are not the same, the first edge set NoRecallBorders R The edge type of the first road edge in Borders is determined as the road segment edge T .

[0209] Finally, the target road edge is modified according to the first road edge after the edge type is re-determined, that is, the first edge set NoRecallBorders is merged R with the second edge set PrecisionBorders T 、PrecisionBorders C 、PrecisionBorders E .

[0210] After obtaining the corrected target road edge, map data may be generated according to the corrected target road edge and the map data may be stored, or a lane-level road interface may be obtained by rendering based on the target road edge.

[0211] It can be understood that in the road image detection method of the present application, by introducing a processing model, a feature extraction module based on the processing model extracts features of the target road image to obtain target image features of the target road image, and then the target image features are mapped based on multiple edge detection modules of the processing model, so that different information can be learned from the target image features according to multiple edge detection modules, and then the detection results of the target road edges of different edge types output by each edge detection module and the detection results of the target road surface output by the road surface detection module can be obtained under the target classification result of the instance classification module. Therefore, based on the processing model, the detection of the road edge of the target road image can be realized, and the road surface and multiple types of road edges can be detected, thereby enriching the types of road deconstruction information obtained by the detection, and thus the refinement of the road image detection can be improved. At the same time, the target image features can also be mapped based on the road surface detection module of the processing model, so that the road surface information can be learned from the target image features according to the road surface detection module, and then the road surface contour of the target road surface can be obtained, and the target road edge can be corrected according to the road surface contour, thereby improving the integrity of the target road edge. In addition, by integrating the instance classification model, the road surface detection module and multiple edge detection modules into one processing model, on the one hand, the detection efficiency of road images can be effectively improved; on the other hand, since the instance classification model, the road surface detection module and multiple edge detection modules share target image features, the learning between the instance classification model, the road surface detection module and multiple edge detection modules can be correlated, thereby improving the detection performance of the processing model as a whole.

[0212] Reference Fig.18 , Fig.18 Another optional flow chart of the road image detection method provided in the embodiment of the present application is described by taking the method executed by a terminal as an example. The road image detection method includes but is not limited to the following steps 1801 to 1802.

[0213] Step 1801: Acquire a target road image, and send the target road image to a server.

[0214] Step 1802: Receive the road edge extraction result of the target road image returned by the server.

[0215] Among them, the road edge extraction result is obtained by the server inputting the target road image into the processing model, and the feature extraction module based on the processing model extracts the feature of the target road image to obtain the target image feature of the target road image. The multiple edge detection modules based on the processing model respectively map the target image feature to obtain the detection result of the target road edge output by each edge detection module. The target road edge is obtained by extracting the target road edge from the target road image based on the detection results of the multiple target road edges. The target road edge is the edge of the road surface in the target road image, and the edge types of each target road edge are different.

[0216] The road edge extraction result may be a set of pixel points included in the target road edge, or the road edge extraction result may be a labeled image obtained by the server after labeling the target road image according to the extracted target road edge.

[0217] It is understandable that the detailed principle of how the server obtains the road edge extraction result can be found in the previous explanation and will not be repeated here.

[0218] In the road image detection method including steps 1801 to 1802, by introducing the processing model of the server, the feature extraction module based on the processing model extracts the feature of the target road image to obtain the target image feature of the target road image, and then the target image feature is mapped respectively based on the multiple edge detection modules of the processing model, so that different information can be learned from the target image feature respectively according to the multiple edge detection modules, and then the detection results of the target road edge of different edge types output by each edge detection module can be obtained, and the target road edge can be extracted from the target road image. Therefore, the detection of the road edge of the target road image can be realized based on the processing model, and multiple types of road edges can be detected, thereby enriching the types of road deconstruction information obtained by the detection, and then the refinement of the detection of the road image can be improved. In addition, by integrating multiple edge detection modules into one processing model, on the one hand, the detection efficiency of the road image can be effectively improved, and on the other hand, since multiple edge detection modules share the target image feature, the learning between the multiple edge detection modules can be associated, thereby improving the detection performance of the processing model as a whole.

[0219] For example, the road image detection method provided in the embodiment of the present application can be applied to scenarios such as vehicle navigation, assisted driving, map drawing and updating, etc. Fig.19 , Fig.19 Another optional schematic diagram of the road image detection method provided in the embodiment of the present application, the vehicle-mounted terminal of the intelligent vehicle is deployed with a processing model. In the vehicle navigation scenario, the intelligent vehicle can obtain a high-resolution remote sensing image (i.e., a target road image) containing a road through an image acquisition device, and input the remote sensing image into the processing model, and obtain a prediction result (i.e., a target road surface and multiple target road edges, wherein the multiple target road edges are respectively a road section closed edge, an intersection surface edge, and a road section edge) through the processing model, and then post-process the target road surface and the multiple target road edges, that is, perform contour extraction on the target road surface to obtain a road surface contour, and then determine the difference between the road surface contour and each target road edge to obtain a first edge set that is not successfully detected by the edge detection module, and determine the intersection between the road surface contour and each target road edge to obtain a second edge set that is successfully detected by the edge detection module, and finally update each target road edge according to the first edge set and the second edge set, and output the final result (i.e., multiple updated target road edges), wherein the multiple updated target road edges are respectively a road section closed edge, an intersection surface edge, and a road section edge. Through the updated target road edge, the intelligent vehicle can determine the passable area and the position and posture of the vehicle body relative to the road edge, thereby assisting vehicle navigation. It can also realize functions such as lane departure warning, lane keeping and adaptive cruise.

[0220] It is to be understood that, although the steps in the above-mentioned various flow charts are shown in sequence according to the indication of the arrows, these steps are not necessarily performed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in the present embodiment, the execution of these steps does not have a strict order restriction, and these steps can be performed in other orders. Moreover, at least a portion of the steps in the above-mentioned flow charts can include a plurality of steps or a plurality of stages, and these steps or stages are not necessarily performed at the same time, but can be performed at different times, and the execution order of these steps or stages is not necessarily performed in sequence, but can be performed in turn or alternately with at least a portion of the steps or stages in other steps or other steps.

[0221] Reference Fig. 20 , Fig. 20 This is an optional structural diagram of a road image detection device provided in an embodiment of the present application. The first road image detection device 2000 includes:

[0222] The target image feature determination module 2001 is used to obtain a target road image, input the target road image into a processing model, and extract features of the target road image based on a feature extraction module of the processing model to obtain target image features of the target road image;

[0223] A target road edge determination module 2002 is used to map target image features respectively based on multiple edge detection modules of the processing model to obtain detection results of target road edges output by each edge detection module, wherein the target road edge is an edge of a road surface in the target road image, and the edge types of each target road edge are different from each other;

[0224] The target road edge extraction module 2003 is used to extract the target road edge from the target road image according to the detection results of multiple target road edges.

[0225] Furthermore, the target road edge determination module 2002 is specifically used for:

[0226] Based on the multiple edge detection modules of the processing model, the target image features are mapped respectively to obtain the detection results of the candidate edges output by each edge detection module;

[0227] The instance classification module based on the processing model maps the target image features to obtain target classification results of various instances in the target road image output by the instance classification module;

[0228] The detection result of the target road edge is determined according to the target classification result and the detection result of the candidate edge.

[0229] Furthermore, the target road edge determination module 2002 is specifically used for:

[0230] The instance classification module based on the processing model performs full connection processing on the target image feature to obtain a first fully connected feature;

[0231] The first fully connected feature is normalized based on the instance classification module to obtain target classification results of various instances in the target road image output by the instance classification module.

[0232] Furthermore, the first road image detection device 2000 further includes a training module 2004, and the training module 2004 is specifically used for:

[0233] Acquire a sample road image and road edge mask images of multiple different edge types corresponding to the sample road image;

[0234] determining a plurality of sample road edges of different edge types in the sample road image based on the processing model;

[0235] determining an edge detection loss based on a sample road edge and a corresponding road edge mask image;

[0236] The processing model is trained based on the edge detection loss.

[0237] Furthermore, the sample road images are annotated with category labels of various instances. The above training module is specifically used for:

[0238] Determine sample classification results of various instances in the sample road image based on the processing model, and determine classification loss according to the sample classification results and the category labels;

[0239] The processing model is trained based on edge detection loss and classification loss.

[0240] Furthermore, the target image feature determination module 2001 is specifically used for:

[0241] The feature extraction module based on the processing model performs down-sampling processing on the target road image to obtain the down-sampled image features of the target road image;

[0242] The target road image is upsampled based on the feature extraction module to obtain the target image features of the target road image.

[0243] Furthermore, the first road image detection device 2000 further includes a post-processing module 2005, which is specifically used for:

[0244] The road surface detection module based on the processing model maps the target image features to obtain the detection result of the target road surface output by the road surface detection module;

[0245] Extracting the target road surface from the target road image according to the detection result of the target road surface, performing contour extraction on the target road surface, and obtaining the road surface contour of the target road surface;

[0246] The target road edge is corrected according to the road surface contour.

[0247] Furthermore, the post-processing module 2005 is specifically used for:

[0248] Determine a difference set between the road surface contour and each target road edge to obtain a first edge set that is not successfully detected by the edge detection module;

[0249] Determine the intersection between the road surface contour and each target road edge to obtain a second edge set successfully detected by the edge detection module;

[0250] Traversing the first road edge in the first edge set, and determining at least one second road edge adjacent to the first road edge from the second edge set;

[0251] The edge type of the first road edge is re-determined according to the edge type of the second road edge, and the target road edge is corrected according to the first road edge after the edge type is re-determined.

[0252] Furthermore, the number of the second road edges is two, and the post-processing module 2005 is specifically used for:

[0253] comparing edge types of two second road edges;

[0254] When the comparison result indicates that the edge types of the two second road edges are the same, the edge type of the second road edge is used as the edge type of the first road edge; or, when the comparison result indicates that the edge types of the two second road edges are different, the edge type of the first road edge is determined as a road segment edge.

[0255] Furthermore, the post-processing module 2005 is specifically used for:

[0256] The road surface detection module based on the processing model performs full connection processing on the target image features to obtain a second fully connected feature;

[0257] Activate the second fully connected feature based on the road surface detection module to obtain an activated feature;

[0258] Performing full connection processing on the activated features based on the road surface detection module to obtain a third fully connected feature;

[0259] The third fully connected feature is normalized based on the road surface detection module to obtain a detection result of the target road surface output by the road surface detection module.

[0260] The first road image detection device 2000 and the road image detection method are based on the same inventive concept, and by introducing a processing model, a feature extraction module based on the processing model extracts features of the target road image to obtain target image features of the target road image, and then multiple edge detection modules based on the processing model respectively map the target image features, so that different information can be learned from the target image features according to the multiple edge detection modules, and then the detection results of the target road edges of different edge types output by each edge detection module can be obtained, and the target road edges can be extracted from the target road image. Therefore, based on the processing model, the road edge detection of the target road image can be realized, and multiple types of road edges can be detected, thereby enriching the types of road deconstruction information obtained by the detection, and then the refinement of the road image detection can be improved. In addition, by integrating multiple edge detection modules into one processing model, on the one hand, the detection efficiency of the road image can be effectively improved, and on the other hand, since multiple edge detection modules share the target image features, the learning between the multiple edge detection modules can be correlated, thereby improving the detection performance of the processing model as a whole.

[0261] Reference Fig.21 , Fig.21 Another optional structural diagram of the road image detection device provided in the embodiment of the present application, the second road image detection device 2100 includes:

[0262] An acquisition module 2101 is used to acquire a target road image and send the target road image to a server;

[0263] A result receiving module 2102 is used to receive the road edge extraction result of the target road image returned by the server;

[0264] Among them, the road edge extraction result is obtained by the server inputting the target road image into the processing model, and the feature extraction module based on the processing model extracts the feature of the target road image to obtain the target image feature of the target road image. The multiple edge detection modules based on the processing model respectively map the target image feature to obtain the detection result of the target road edge output by each edge detection module. The target road edge is obtained by extracting the target road edge from the target road image based on the detection results of the multiple target road edges. The target road edge is the edge of the road surface in the target road image, and the edge types of each target road edge are different.

[0265] The second road image detection device 2100 and the road image detection method are based on the same inventive concept, and by introducing a processing model, a feature extraction module based on the processing model extracts features of the target road image to obtain target image features of the target road image, and then multiple edge detection modules based on the processing model map the target image features respectively, so that different information can be learned from the target image features according to the multiple edge detection modules, and then the detection results of the target road edges of different edge types output by each edge detection module can be obtained, and the target road edges can be extracted from the target road image. Therefore, based on the processing model, the road edge detection of the target road image can be realized, and multiple types of road edges can be detected, thereby enriching the types of road deconstruction information obtained by the detection, and then the refinement of the road image detection can be improved. In addition, by integrating multiple edge detection modules into one processing model, on the one hand, the detection efficiency of the road image can be effectively improved, and on the other hand, since multiple edge detection modules share the target image features, the learning between the multiple edge detection modules can be correlated, thereby improving the detection performance of the processing model as a whole.

[0266] The electronic device for executing the road image detection method provided in the embodiment of the present application may be a terminal. Fig. 22 , Fig. 22 This is a partial structural block diagram of a terminal provided in an embodiment of the present application. The terminal includes: a camera assembly 2210, a memory 2220, an input unit 2230, a display unit 2240, a sensor 2250, an audio circuit 2260, a wireless fidelity (WiFi) module 2270, a processor 2280, and a power supply 2290. Those skilled in the art can understand that Fig. 22 The terminal structure shown in the figure does not constitute a limitation on the terminal, and may include more or less components than shown in the figure, or combine certain components, or arrange the components differently.

[0267] The camera assembly 2210 can be used to capture images or videos. Optionally, the camera assembly 2210 includes a front camera and a rear camera. Typically, the front camera is disposed on the front panel of the terminal, and the rear camera is disposed on the back of the terminal. In some embodiments, there are at least two rear cameras, which are any one of a main camera, a depth of field camera, a wide-angle camera, and a telephoto camera, so as to realize the fusion of the main camera and the depth of field camera to realize the background blur function, the fusion of the main camera and the wide-angle camera to realize the panoramic shooting and VR (Virtual Reality) shooting function or other fusion shooting functions.

[0268] The memory 2220 may be used to store software programs and modules. The processor 2280 executes various functional applications and data processing of the terminal by running the software programs and modules stored in the memory 2220 .

[0269] The input unit 2230 may be used to receive input digital or character information and generate key signal input related to the terminal's settings and function control. Specifically, the input unit 2230 may include a touch panel 2231 and other input devices 2232 .

[0270] The display unit 2240 may be used to display input information or provided information and various menus of the terminal. The display unit 2240 may include a display panel 2241.

[0271] The audio circuit 2260, the speaker 2261, and the microphone 2262 may provide an audio interface.

[0272] The power source 2290 may be alternating current, direct current, a disposable battery, or a rechargeable battery.

[0273] The number of sensors 2250 may be one or more, and the one or more sensors 2250 include but are not limited to: acceleration sensors, gyroscope sensors, pressure sensors, optical sensors, etc. Among them:

[0274] The acceleration sensor can detect the magnitude of acceleration on the three coordinate axes of the coordinate system established by the terminal. For example, the acceleration sensor can be used to detect the components of gravity acceleration on the three coordinate axes. The processor 2280 can control the display unit 2240 to display the user interface in a horizontal view or a vertical view according to the gravity acceleration signal collected by the acceleration sensor. The acceleration sensor can also be used for collecting game or user motion data.

[0275] The gyroscope sensor can detect the body direction and rotation angle of the terminal, and the gyroscope sensor can cooperate with the acceleration sensor to collect the user's 3D actions on the terminal. The processor 2280 can implement the following functions based on the data collected by the gyroscope sensor: motion sensing (such as changing the UI according to the user's tilt operation), image stabilization during shooting, game control, and inertial navigation.

[0276] The pressure sensor can be set in the side frame of the terminal and / or the lower layer of the display unit 2240. When the pressure sensor is set in the side frame of the terminal, the user's holding signal of the terminal can be detected, and the processor 2280 performs left and right hand recognition or shortcut operation according to the holding signal collected by the pressure sensor. When the pressure sensor is set in the lower layer of the display unit 2240, the processor 2280 controls the operability controls on the UI interface according to the user's pressure operation on the display unit 2240. The operability control includes at least one of a button control, a scroll bar control, an icon control, and a menu control.

[0277] The optical sensor is used to collect the ambient light intensity. In one embodiment, the processor 2280 can control the display brightness of the display unit 2240 according to the ambient light intensity collected by the optical sensor. Specifically, when the ambient light intensity is high, the display brightness of the display unit 2240 is increased; when the ambient light intensity is low, the display brightness of the display unit 2240 is reduced. In another embodiment, the processor 2280 can also dynamically adjust the shooting parameters of the camera assembly 2210 according to the ambient light intensity collected by the optical sensor.

[0278] In this embodiment, the processor 2280 included in the terminal can execute the road image detection method of the previous embodiment.

[0279] The electronic device for executing the above road image detection method provided in the embodiment of the present application may also be a server. Fig.23 , Fig.23 Partial structural block diagram of the server provided in the embodiment of the present application, the server 2300 may have relatively large differences due to different configurations or performances, and may include one or more central processing units (CPU) 2322 (for example, one or more processors) and memory 2332, one or more storage media 2330 (for example, one or more mass storage devices) storing application programs 2342 or data 2344. Among them, the memory 2332 and the storage medium 2330 can be short-term storage or permanent storage. The program stored in the storage medium 2330 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations in the server 2300. Furthermore, the central processing unit 2322 can be configured to communicate with the storage medium 2330 and execute a series of instruction operations in the storage medium 2330 on the server 2300.

[0280] The server 2300 may also include one or more power supplies 2326, one or more wired or wireless network interfaces 2350, one or more input and output interfaces 2358, and / or one or more operating systems 2341, such as Windows ServerTM, Mac OS XTM, UnixTM, LinuxTM, FreeBSDTM, etc.

[0281] The processor in the server 2300 can be used to execute the road image detection method.

[0282] An embodiment of the present application further provides a computer-readable storage medium, which is used to store program codes, and the program codes are used to execute the road image detection methods of the aforementioned embodiments.

[0283] The embodiment of the present application also provides a computer program product, which includes a computer program stored in a computer-readable storage medium. A processor of a computer device reads the computer program from the computer-readable storage medium, and the processor executes the computer program, so that the computer device executes the above-mentioned road image detection method.

[0284] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate to describe the embodiments of the present application, for example, it can be implemented in an order other than those illustrated or described here. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0285] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0286] It should be understood that in the description of the embodiments of the present application, the meaning of multiple (or multiple items) is more than two, greater than, less than, exceed, etc. are understood to not include the number, and above, below, within, etc. are understood to include the number.

[0287] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0288] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0289] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0290] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store program codes.

[0291] It should also be understood that the various implementations provided in the embodiments of the present application can be combined arbitrarily to achieve different technical effects.

[0292] The above is a specific description of the preferred implementation of the present application, but the present application is not limited to the above implementation mode. Technical personnel familiar with the field can also make various equivalent modifications or substitutions under the shared conditions without violating the spirit of the present application. These equivalent modifications or substitutions are all included in the scope defined by the claims of the present application.

Claims

1. A road image detection method, characterized in that: include: Acquire a target road image, input the target road image into a processing model, and extract features of the target road image based on a feature extraction module of the processing model to obtain target image features of the target road image; Based on the edge detection module used to detect the edge of the road section in the processing model, the target image features are mapped to obtain the detection result of the edge of the road section; based on the edge detection module used to detect the edge of the intersection surface in the processing model, the target image features are mapped to obtain the detection result of the edge of the intersection surface; based on the edge detection module used to detect the closed edge of the road section in the processing model, the target image features are mapped to obtain the detection result of the closed edge of the road section, and the closed edge of the road section is the portion of the image edge of the target road image located in the road surface; Extracting a target road edge from the target road image according to the detection results of the road segment edge, the intersection surface edge and the road segment closed edge, wherein the target road edge includes the road segment edge, the intersection surface edge and the road segment closed edge; The road surface detection module based on the processing model maps the target image features to obtain a detection result of the target road surface output by the road surface detection module; Extracting the target road surface from the target road image according to the detection result of the target road surface, performing contour extraction on the target road surface, and obtaining a road surface contour of the target road surface; Determine a difference set between the road surface contour and each of the target road edges to obtain a first edge set that is not successfully detected by the edge detection module; Determine the intersection between the road surface contour and each of the target road edges to obtain a second edge set successfully detected by the edge detection module; Traversing first road edges in the first edge set, and determining at least one second road edge adjacent to the first road edge from the second edge set; The edge type of the first road edge is re-determined according to the edge type of the second road edge, and the target road edge is corrected according to the first road edge after the edge type is re-determined.

2. The road image detection method according to claim 1, characterized in that: The method includes mapping the target image features based on the edge detection module for detecting the edge of the road section in the processing model to obtain the detection result of the edge of the road section, mapping the target image features based on the edge detection module for detecting the edge of the intersection surface in the processing model to obtain the detection result of the edge of the intersection surface, and mapping the target image features based on the edge detection module for detecting the closed edge of the road section in the processing model to obtain the detection result of the closed edge of the road section, including: Based on the edge detection module for detecting the edge of the road section, the edge detection module for detecting the edge of the intersection surface, and the edge detection module for detecting the closed edge of the road section in the processing model, the target image features are mapped respectively to obtain the detection results of the candidate edges output by each of the edge detection modules; Mapping the target image features based on the instance classification module of the processing model to obtain target classification results of various instances in the target road image output by the instance classification module; The detection results of the road segment edge, the intersection surface edge and the road segment closed edge are determined respectively according to the target classification result and the detection result of the candidate edge.

3. The road image detection method according to claim 2, characterized in that: The instance classification module based on the processing model maps the target image features to obtain target classification results of various instances in the target road image output by the instance classification module, including: Based on the instance classification module of the processing model, the target image feature is fully connected to obtain a first fully connected feature; The first fully connected feature is normalized based on the instance classification module to obtain target classification results of various instances in the target road image output by the instance classification module.

4. The road image detection method according to claim 1, characterized in that: The processing model is trained by the following steps: Acquire a sample road image and road edge mask images of multiple different edge types corresponding to the sample road image; Determine, based on the processing model, detection results of a plurality of sample road edges of the sample road image having different edge types; Determining edge detection loss according to the detection result of the sample road edge and the corresponding road edge mask image; The processing model is trained according to the edge detection loss.

5. The road image detection method according to claim 4, characterized in that: The sample road image is annotated with category labels of various instances, and the processing model is trained according to the edge detection loss, including: Determine sample classification results of various instances in the sample road image based on the processing model, and determine classification loss according to the sample classification results and the category label; The processing model is trained according to the edge detection loss and the classification loss.

6. The road image detection method according to claim 1, characterized in that: The feature extraction module based on the processing model extracts features from the target road image to obtain target image features of the target road image, including: Performing down-sampling processing on the target road image based on the feature extraction module of the processing model to obtain down-sampled image features of the target road image; The downsampled image features are upsampled based on the feature extraction module to obtain target image features of the target road image.

7. The road image detection method according to claim 1, characterized in that: The number of the second road edges is two, and the re-determining the edge type of the first road edge according to the edge type of the second road edge includes: comparing edge types of two edges of the second road; When the comparison result indicates that the edge types of the two second road edges are the same, the edge type of the second road edge is used as the edge type of the first road edge; or, when the comparison result indicates that the edge types of the two second road edges are different, the edge type of the first road edge is determined as a road segment edge.

8. The road image detection method according to claim 1, characterized in that: The road surface detection module based on the processing model maps the target image features to obtain the detection result of the target road surface output by the road surface detection module, including: A road surface detection module based on the processing model performs full connection processing on the target image feature to obtain a second fully connected feature; Performing activation processing on the second fully connected feature based on the road surface detection module to obtain an activated feature; Performing full connection processing on the activated feature based on the road surface detection module to obtain a third fully connected feature; The third fully connected feature is normalized based on the road surface detection module to obtain a detection result of the target road surface output by the road surface detection module.

9. A road image detection device, characterized in that: include: a target image feature determination module, used for acquiring a target road image, inputting the target road image into a processing model, and extracting features of the target road image based on a feature extraction module of the processing model to obtain target image features of the target road image, wherein the target road image is a remote sensing image; a target road edge determination module, used to map the target image features based on an edge detection module for detecting road segment edges in the processing model to obtain a detection result of the road segment edge, map the target image features based on an edge detection module for detecting intersection surface edges in the processing model to obtain a detection result of the intersection surface edge, and map the target image features based on an edge detection module for detecting road segment closed edges in the processing model to obtain a detection result of the road segment closed edge, wherein the road segment closed edge is a portion of the image edge of the target road image located in the road surface; A target road edge extraction module is used to extract the target road edge from the target road image according to the detection results of the road segment edge, the intersection surface edge and the road segment closed edge, wherein the target road edge includes the road segment edge, the intersection surface edge and the road segment closed edge; A post-processing module, used for mapping the target image features based on the road surface detection module of the processing model to obtain a detection result of the target road surface output by the road surface detection module; Extracting the target road surface from the target road image according to the detection result of the target road surface, performing contour extraction on the target road surface, and obtaining a road surface contour of the target road surface; Determine a difference set between the road surface contour and each of the target road edges to obtain a first edge set that is not successfully detected by the edge detection module; Determine the intersection between the road surface contour and each of the target road edges to obtain a second edge set successfully detected by the edge detection module; Traversing first road edges in the first edge set, and determining at least one second road edge adjacent to the first road edge from the second edge set; The edge type of the first road edge is re-determined according to the edge type of the second road edge, and the target road edge is corrected according to the first road edge after the edge type is re-determined.

10. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the road image detection method according to any one of claims 1 to 8 is implemented.

11. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the road image detection method according to any one of claims 1 to 8 is implemented.