A system and method for extracting multi-level off-road road network from space-based / air-based images

Abstract

The present invention discloses a space-based / air-based multi-level off-road road network extraction system and method, the system comprising: an image acquisition module acquires a target image; a semantic extraction module performs adhesion coefficient semantic classification and road network extraction on the target image to obtain an adhesion coefficient classification map and a semantic road network map, and different adhesion coefficients represent different surface types; a morphological processing module generates a topological structure map of the target image using a morphological method; and a result output module outputs a semantic road network map, a topological structure map, and an adhesion coefficient classification map. The present invention relates to the field of artificial intelligence, and utilizes the advantages of strong interactivity and data fitting ability of a generative deep network to achieve efficient and automatic extraction of off-road road networks in unknown environments, improve extraction accuracy, and the hierarchical road network can reduce the discontinuity of the global reference road network caused by occlusion and recognition errors, and restrict the problem of traffic, provide passable guidance information for local planning, and improve autonomous capabilities under complex terrain.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a system and method for extracting multi-level off-road road networks from space-based / air-based maps. Background Art

[0002] High-precision maps can provide accurate route planning and driving guidance for unmanned vehicles, and have achieved remarkable results in the field of unmanned driving in urban environments. However, in off-road environments, the practicality of high-precision map navigation is limited due to factors such as complex terrain and incomplete map data. Therefore, efficient and accurate automatic extraction of road networks to provide vehicles with driving paths and driving strategies is a key link in off-road environment unmanned driving systems.

[0003] At present, although the method of manually extracting a large range of road networks can improve the extraction effect by relying on manual experience, manual extraction is time-consuming and inefficient, and the cost-benefit is not high. Specifically, on the one hand, the scope of the area to be labeled is large, and the roads are complicated, so it takes a lot of effort to manually label the road network; on the other hand, manual labeling of the road network will be affected by human factors, resulting in inaccurate punctuation and poor road curvature fitting.

[0004] Current automatic road network extraction methods can be roughly divided into two frameworks: multi-stage extraction and end-to-end extraction.

[0005] a) Multi-stage road network extraction algorithm

[0006] The essence of the multi-stage road network extraction method is to first extract the traversable area using a segmentation method, and then rely on morphological methods for post-processing to convert the binary image of the traversable area into a road network structure. In this way, the road network extraction task can be decomposed into three subtasks: extracting roads from binary images, extracting road skeletons, and extracting key path points. As shown in the figure, these tasks require three operations: image segmentation, image refinement, and corner point detection. Among them, the effect of image segmentation directly affects the accuracy of the entire road network extraction result. Once there is an obvious error in the image segmentation effect, errors will be accumulated in the subsequent morphological processing.

[0007] b) End-to-end road network extraction method

[0008] The end-to-end road network extraction method directly uses aerial photography / satellite images as model input and finally outputs the topological structure of the road network. Since the road network has its own topological properties, a graph-based network framework is usually selected when building an end-to-end model. With the entry of Transformer into the field of computer vision, some people have abandoned the graph-based iterative detection method and used the Transformer framework to build the network framework. The end-to-end road network extraction algorithm can be roughly divided into two categories according to the basic network framework: graph-based (local feature iterative detection) / non-graph-based (global feature decoding) methods.

[0009] There are also many problems with these two methods. For example, the multi-stage road network extraction method uses segmentation-morphological post-processing to convert the binary image of the traversable area into a road network structure. However, the effect of image segmentation directly affects the accuracy of the entire road network extraction result. Once there is an obvious error in the image segmentation effect, errors will be accumulated in the subsequent morphological processing. The end-to-end road network extraction method directly uses aerial / satellite images as model input and finally outputs the topological structure of the road network. Since the road network has its own topological properties, a graph-based network framework is usually selected when building an end-to-end model. The end-to-end method is currently a research direction that has received much attention, but this direction currently has problems such as low interpretability and data dependence. The existing off-road environment road network extraction technology has the following main defects:

[0010] 1) The annotation accuracy is low and cannot meet the driving requirements of unmanned vehicles in complex off-road terrain;

[0011] 2) The process is very complicated and relies on professional technicians, with a low degree of automation. This makes high-precision maps expensive, highly specialized, and inefficient to produce. The annotation speed is slow, the real-time performance is poor, and it is difficult to cope with the rapidly changing off-road environment;

[0012] 3) It is difficult to adapt to the diversity of different terrains and road conditions, and its versatility is poor.

[0013] 4) The storage method of existing high-precision maps is complex and also relies on a lot of manual work. The parsing method is complicated, which is not conducive to data analysis and visualization of unmanned vehicle path planning. Summary of the invention

[0014] In view of the deficiencies in the above-mentioned prior art, the present invention provides a system and method for extracting multi-level off-road road networks from space-based / air-based maps. By utilizing the advantages of strong interactivity and data fitting capabilities of generative deep networks, efficient and automatic extraction of off-road road networks in unknown environments can be achieved, while improving the extraction accuracy.

[0015] To achieve the above object, the present invention provides a space-based / air-based multi-level off-road road network extraction system, comprising:

[0016] An image acquisition module, used to acquire a target image of a target off-road area, wherein the target image is a sky-based image or an air-based image;

[0017] An image preprocessing module, used to preprocess the target image, thereby improving the quality and positioning accuracy of the target image;

[0018] A semantic extraction module based on adhesion coefficient is used to perform adhesion coefficient semantic classification and road network extraction on the target image to obtain an adhesion coefficient classification map and a semantic road network map, wherein different adhesion coefficients represent different surface types;

[0019] A morphological processing module, used for generating a topological structure diagram of the target image by using a morphological method based on the adhesion coefficient classification diagram;

[0020] A result output module is used to output the semantic road network diagram, the topological structure diagram and / or the adhesion coefficient classification diagram.

[0021] In one embodiment, the attachment coefficient-based semantic extraction module is specifically a visual generative deep network, and it includes a label;

[0022] The labels include space-based / air-based maps, adhesion coefficient classification map true values ​​and road network map true values. The adhesion coefficient classification map true values ​​include surface types represented by different adhesion coefficients, and the road network map true values ​​include binary raster maps of passable roads and non-passable roads.

[0023] In one embodiment, the training process of the semantic extraction module based on the attachment coefficient is:

[0024] First, the images are manually semantically annotated. That is, the image content is manually identified and labeled using the Labelme tool according to the adhesion coefficient semantic label and the binary labels of the passable area and the non-passable area, providing a complete data set for the training of the semantic extraction model based on the adhesion coefficient.

[0025] Secondly, based on the data set, model training, parameter adjustment and effect evaluation are performed to obtain a relatively optimal model;

[0026] Finally, the trained adhesion coefficient-based semantic extraction model is deployed for inference to obtain a semantic road network graph that contains adhesion coefficient labels, passable area labels, and non-passable area labels.

[0027] In one embodiment, the attachment coefficient-based semantic extraction module includes:

[0028] The embedding layer is used to introduce position encoding, which converts the two-dimensional image into sequential data by connecting the image slices through positional relationships, so that the model can understand the position information of pixels in the image and better extract global features;

[0029] The encoding layer is used to design a multi-layer generative deep network encoder. Each layer contains multiple self-attention heads and feedforward neural networks. The self-attention mechanism is used to capture the long-distance dependencies in the image and enhance the semantic connection based on the projection of the long-distance dependencies between image pixels in the passable area, ensuring the connectivity and accuracy of the final extracted passable area.

[0030] The decoding layer is used to reversely parse the encoder results, classify each pixel according to the label information, and represent the classification results in the form of a semantic road network.

[0031] In one embodiment, the specific operation process of the morphological processing module includes:

[0032] Firstly, the ZS algorithm is used to skeletonize the binary images of the passable area and the non-passable area of ​​the target image to obtain a road skeleton map;

[0033] Secondly, based on the road skeleton graph, an improved corner point detection method based on the Shi-Tomas algorithm is used to extract key road points;

[0034] Thirdly, the key road points are sparsely processed according to the road vector direction and position distance to reduce redundant data and enhance topology;

[0035] Finally, the road skeleton graph and the road key points are topologically fused to obtain a final topological structure graph.

[0036] In one of the embodiments, the preprocessing performed by the image preprocessing module on the target image includes image enhancement, cloud removal, geometric correction and / or registration;

[0037] The image enhancement includes rotation, scaling and / or flipping.

[0038] To achieve the above object, the present invention also provides a method for extracting multi-level off-road road networks from space-based / air-based maps, using the above system, comprising the following steps:

[0039] Step 1, obtaining a target image of a target off-road area, wherein the target image is a sky-based image or an air-based image;

[0040] Step 2, preprocessing the target image to improve the quality and positioning accuracy of the target image;

[0041] Step 3, performing adhesion coefficient semantic classification and road network extraction on the target image to obtain an adhesion coefficient classification map and a semantic road network map, wherein different adhesion coefficients represent different surface types;

[0042] Step 4, based on the adhesion coefficient classification map, using a morphological method to generate a topological structure map of the target image;

[0043] Step 5: output the semantic road network diagram, the topological structure diagram and / or the adhesion coefficient classification diagram.

[0044] Compared with the prior art, the present invention has the following beneficial technical effects:

[0045] 1. The invention can extract off-road road networks faster than manual extraction, with low annotation cost and less manual intervention. It can quickly generate unmanned driving reference road networks in unknown environments to cope with rapidly changing environments and improve the adaptability of non-high-precision map navigation in unknown environments.

[0046] 2. The hierarchical road network in the present invention can reduce the discontinuity of the global reference road network caused by occlusion and recognition errors, and restrict the passage of traffic, and can provide traversable guidance information for local planning, thereby improving the autonomous ability in complex terrain;

[0047] 3. The hierarchical road network in the present invention can meet the diversity of different environments, terrains and road conditions. It can be used not only in off-road environments, but also in road network extraction in urban environments, and has strong versatility. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the structures shown in these drawings without paying creative work.

[0049] Figure 1 This is a schematic diagram of the principle of the space-based / air-based multi-level off-road road network extraction system in Example 1 of the present invention;

[0050] Figure 2 is a structural block diagram of a semantic extraction module based on attachment coefficient in Embodiment 1 of the present invention;

[0051] Figure 3 This is a flow chart of the method for extracting multi-level off-road road networks from space-based / air-based maps in Example 2 of the present invention.

[0052] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0053] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0054] In addition, the technical solutions between the various embodiments of the present invention can be combined with each other, but it must be based on the fact that ordinary technicians in the field can implement it. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such combination of technical solutions does not exist and is not within the scope of protection required by the present invention.

[0055] Terminology explanation:

[0056] Space-based: refers to satellite images of off-road environments that can be used in unmanned driving;

[0057] Air-based: refers to drone aerial images of off-road environments that can be used in unmanned operations.

[0058] Example 1

[0059] This embodiment discloses a space-based / air-based multi-level off-road road network extraction system based on a generative deep network. By taking advantage of the strong interactivity and data fitting ability of the generative deep network, it can achieve efficient automatic extraction of off-road road networks in unknown environments and improve the extraction accuracy. It can be applied to fields such as unmanned driving and smart transportation, especially off-road scenes with unknown environments.

[0060] refer to Figure 1 The space-based / air-based multi-level off-road road network extraction system in this embodiment specifically includes an image acquisition module, an image preprocessing module, a semantic extraction module based on adhesion coefficient, a morphological processing module, and a result output module. Among them, the image acquisition module is used to acquire the target image of the target off-road area, and the target image is a space-based image or an air-based image; the image preprocessing module is used to preprocess the target image, so as to improve the quality and positioning accuracy of the target image; the semantic extraction module based on adhesion coefficient is used to perform adhesion coefficient semantic classification and road network extraction on the target image, and obtain an adhesion coefficient classification map and a semantic road network map, wherein different adhesion coefficients represent different surface types; the morphological processing module is used to generate a topological structure map of the target image using a morphological method based on the adhesion coefficient classification map; the result output module is used to output a semantic road network map, a topological structure map, and / or an adhesion coefficient classification map.

[0061] The adhesion coefficient classification map can characterize the surface type and improve the trafficability of unmanned driving in off-road environments. Since the traditional road network uses binary representation to represent the connection, it is easy to cause the road network to be interrupted due to occlusion, causing the unmanned driving global planning to find the wrong path. When the adhesion coefficient classification map is used, even if the road network is interrupted, the adhesion coefficient representation is continuous, and it is still possible to find some global paths with higher traffic efficiency as a guide based on the adhesion coefficient traffic rate.

[0062] In the specific implementation process, the image acquisition module can obtain the target image of the target off-road area through commercial satellite image services (such as Google Earth, DigitalGlobe, etc.) or drone aerial photos. The preprocessing of the target image by the image preprocessing module may include image enhancement, cloud removal, geometric correction and / or registration, etc., wherein the image enhancement may include rotation, scaling and / or flipping, thereby improving the quality and positioning accuracy of the target image. After preprocessing, the target image is divided into patches of fixed size (i.e., image patches that are subsequently input into the semantic extraction module based on the adhesion coefficient) to meet the input requirements of the semantic extraction module based on the adhesion coefficient, providing a basis for subsequent feature extraction and annotation.

[0063] In this embodiment, the semantic extraction module based on adhesion coefficient specifically uses a visual generative deep network (Vision Transformer) to perform adhesion coefficient semantic classification and road network extraction on the target image, and it includes labels, including space-based / air-based maps, adhesion coefficient classification map truth values ​​and road network map truth values. The adhesion coefficient classification map truth value includes the surface types represented by different adhesion coefficients, such as cement roads, gravel roads, mud and grass. The road network map truth value includes binary raster maps of passable roads and non-passable roads.

[0064] refer to Figure 2 The semantic extraction module based on the attachment coefficient mainly includes an embedding layer, an encoding layer, and a decoding layer. Among them, the embedding layer is used to introduce position encoding, and the image slices are connected through positional relationships to convert the two-dimensional image into sequential data, so that the model can understand the position information of the pixels in the image and better extract global features; the encoding layer is used to design a multi-layer generative deep network encoder, each layer contains multiple self-attention heads and feedforward neural networks, and the self-attention mechanism is used to capture the long-distance dependencies in the image, and the semantic connection is enhanced based on the projection of the long-distance dependencies between image pixels in the passable area, so as to ensure the connectivity and accuracy of the final extraction of the passable area; the decoding layer is used to reversely parse some of the encoder results, classify each pixel according to the label information, and represent the classification results in the form of a semantic road network.

[0065] For example, the process of processing the pre-processed target image by the semantic extraction module based on the attachment coefficient is as follows:

[0066] First, the preprocessed target image is input into the embedding layer, which slices the target image and converts the two-dimensional target image into sequential data according to the position relationship of the pixels in the target image, and then outputs it to the encoding layer.

[0067] Secondly, the encoding layer first performs different linear transformations on the encoding matrix to obtain different Q, V, and K values, and then performs multiple sets of self-attention processing on the original input sequence (see the following formula). Finally, after residual connection and layer normalization operations, it is output to a feedforward neural network with multiple fully connected layers. The connection layer uses the ReLu activation function.

[0068] ;

[0069] Finally, the decoding layer reversely parses the output of the encoding layer, classifies each pixel according to the label information, and represents and outputs the classification results in the form of a semantic road network.

[0070] In this embodiment, the training process of the semantic extraction module based on the attachment coefficient is:

[0071] First, the images are manually semantically annotated. That is, the image content is manually identified and labeled using the Labelme tool according to the adhesion coefficient semantic label and the binary labels of the passable area and the non-passable area, providing a complete data set for the training of the semantic extraction model based on the adhesion coefficient.

[0072] Secondly, based on the data set, model training, parameter adjustment and effect evaluation are performed to obtain a relatively optimal model;

[0073] Finally, the trained adhesion coefficient-based semantic extraction model is deployed for inference to obtain a semantic road network graph that contains adhesion coefficient labels, passable area labels, and non-passable area labels.

[0074] In the specific implementation process, the loss function in the training process of the semantic extraction model based on the attachment coefficient can be constructed as the cross entropy loss function ,for:

[0075] ;

[0076] Among them, m represents the total number of samples, n represents the total number of classifications of common terrain adhesion coefficients, Indicator function indicating that sample i belongs to class j, represents the predicted probability that i belongs to class j.

[0077] In this embodiment, the specific operation process of the morphological processing module specifically includes the following steps:

[0078] Firstly, the binary images of the passable area and the non-passable area of ​​the target image are skeletonized by using the ZS algorithm and the table lookup method to obtain the road skeleton map.

[0079] Secondly, based on the road skeleton graph, an improved corner point detection method based on the Shi-Tomas algorithm is used to extract key road points;

[0080] Thirdly, non-maximum suppression is used to perform sparse processing on key road points according to the road vector direction and position distance to reduce redundant data and enhance topology;

[0081] Finally, the road skeleton map and the key road points are topologically fused to obtain the final topological structure map.

[0082] Example 2

[0083] Based on the space-based / air-based multi-level off-road road network extraction system in Example 1, this example proposes a space-based / air-based multi-level off-road road network extraction method. By adopting the space-based / air-based multi-level off-road road network extraction system in Example 1, fast and accurate road network extraction of unknown environments such as off-road areas is achieved.

[0084] refer to Figure 3 The method for extracting multi-level off-road road networks from space-based / air-based maps in this embodiment specifically includes the following steps:

[0085] Step 1, obtaining a target image of a target off-road area, where the target image is a space-based image or an air-based image;

[0086] Step 2: preprocess the target image to improve the quality and positioning accuracy of the target image;

[0087] Step 3, performing adhesion coefficient semantic classification and road network extraction on the target image to obtain an adhesion coefficient classification map and a semantic road network map, where different adhesion coefficients represent different surface types;

[0088] Step 4, based on the adhesion coefficient classification map, a topological structure map of the target image is generated using a morphological method;

[0089] Step 5: output a semantic road network diagram, a topological structure diagram and / or an adhesion coefficient classification diagram.

[0090] Specifically, the specific implementation process of step 1 to step 5 is basically the same as that of embodiment 1, so this embodiment will not be repeated. Figure 3 The steps in the process are shown in sequence as indicated by the arrows, but the steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of the steps, and the steps can be executed in other orders. Figure 3 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0091] The above description is only a preferred embodiment of the present invention, and does not limit the patent scope of the present invention. All equivalent structural changes made by using the contents of the present invention specification and drawings under the inventive concept of the present invention, or directly / indirectly applied in other related technical fields are included in the patent protection scope of the present invention.

Claims

1. A space-based / air-based multi-level off-road road network extraction system, characterized in that: include: An image acquisition module, used to acquire a target image of a target off-road area, wherein the target image is a sky-based image or an air-based image; An image preprocessing module, used to preprocess the target image, thereby improving the quality and positioning accuracy of the target image; A semantic extraction module based on adhesion coefficient is used to perform adhesion coefficient semantic classification and road network extraction on the target image to obtain an adhesion coefficient classification map and a semantic road network map, wherein different adhesion coefficients represent different surface types; A morphological processing module, used for generating a topological structure diagram of the target image by using a morphological method based on the adhesion coefficient classification diagram; A result output module, used for outputting the semantic road network diagram, the topological structure diagram and / or the adhesion coefficient classification diagram; The semantic extraction module based on the attachment coefficient includes: The embedding layer is used to introduce position encoding, which converts the two-dimensional image into sequential data by connecting the image slices through positional relationships, so that the model can understand the position information of pixels in the image and better extract global features; The encoding layer is used to design a multi-layer generative deep network encoder. Each layer contains multiple self-attention heads and feedforward neural networks. The self-attention mechanism is used to capture the long-distance dependencies in the image and enhance the semantic connection based on the projection of the long-distance dependencies between image pixels in the passable area, ensuring the connectivity and accuracy of the final extracted passable area. The decoding layer is used to reversely parse the encoder results, classify each pixel according to the label information, and represent the classification results in the form of a semantic road network.

2. The space-based / air-based multi-level off-road road network extraction system according to claim 1 is characterized in that: The attachment coefficient-based semantic extraction module is specifically a visual generative deep network, and it includes a label; The labels include space-based / air-based maps, adhesion coefficient classification map true values ​​and road network map true values. The adhesion coefficient classification map true values ​​include surface types represented by different adhesion coefficients, and the road network map true values ​​include binary raster maps of passable roads and non-passable roads.

3. The space-based / air-based multi-level off-road road network extraction system according to claim 2 is characterized in that: The training process of the semantic extraction module based on the adhesion coefficient is as follows: First, the images are manually semantically annotated. That is, the image content is manually identified and labeled using the Labelme tool according to the adhesion coefficient semantic label and the binary labels of the passable area and the non-passable area, providing a complete data set for the training of the semantic extraction model based on the adhesion coefficient. Secondly, based on the data set, model training, parameter adjustment and effect evaluation are performed to obtain a relatively optimal model; Finally, the trained adhesion coefficient-based semantic extraction model is deployed for inference to obtain a semantic road network graph that contains adhesion coefficient labels, passable area labels, and non-passable area labels.

4. The space-based / air-based multi-level off-road road network extraction system according to any one of claims 1 to 3, characterized in that: The specific operation process of the morphological processing module includes: Firstly, the ZS algorithm is used to skeletonize the binary images of the passable area and the non-passable area of ​​the target image to obtain a road skeleton map; Secondly, based on the road skeleton graph, an improved corner point detection method based on the Shi-Tomas algorithm is used to extract key road points; Thirdly, the key road points are sparsely processed according to the road vector direction and position distance to reduce redundant data and enhance topology; Finally, the road skeleton graph and the road key points are topologically fused to obtain a final topological structure graph.

5. The space-based / air-based multi-level off-road road network extraction system according to any one of claims 1 to 3, characterized in that: The preprocessing performed on the target image by the image preprocessing module includes image enhancement, cloud removal, geometric correction and / or registration; The image enhancement includes rotation, scaling and / or flipping.

6. A method for extracting multi-level off-road road networks from space-based / air-based maps, characterized in that: The system according to any one of claims 1 to 5 comprises the following steps: Step 1, obtaining a target image of a target off-road area, wherein the target image is a sky-based image or an air-based image; Step 2, preprocessing the target image to improve the quality and positioning accuracy of the target image; Step 3, performing adhesion coefficient semantic classification and road network extraction on the target image to obtain an adhesion coefficient classification map and a semantic road network map, wherein different adhesion coefficients represent different surface types; Step 4, based on the adhesion coefficient classification map, using a morphological method to generate a topological structure map of the target image; Step 5: output the semantic road network diagram, the topological structure diagram and / or the adhesion coefficient classification diagram.

Images