Remote sensing image path planning method, system, equipment and medium

Abstract

The invention relates to a remote sensing image path planning method, system, device and medium, the remote sensing image path planning method comprises the following steps: combining a D-LinkNet network structure with an HED network structure to obtain a road extraction model of a full convolutional neural network, setting a side output layer, and also setting a fusion layer; constructing a new loss function, and replacing an original loss function, which is constructed by adding the cross entropy and the Dice coefficient loss, of the D-Link Net network structure with the new loss function; training the road extraction model based on a preset training set; using the trained road extraction model to extract the road of the selected research area to obtain a road membership map in a raster data form; and performing path planning on the road membership grid map based on an A * algorithm to generate a passage path. According to the method, the accuracy of image road extraction is improved, and a reasonable path planning scheme can be generated in the absence of road network data.

Description

Technical Field

[0001] The present invention relates to the technical field of path planning. In particular, it relates to a method, system, device and medium for path planning of remote sensing images. Background Technique

[0002] Traditional path planning methods mainly rely on vector data of roads to construct road networks, and then perform path planning based on the vector network. For areas where road vector data is missing but there are passable roads, path planning mainly relies on road network vectors or digital elevation model (DEM) data, etc.

[0003] The research on roadless path planning of high-resolution optical remote sensing data for complex scenes is an interdisciplinary and highly comprehensive research field, involving multiple aspects such as geographic information science, remote sensing science, and road traffic science. Currently, the research in this field mainly focuses on how to extract key information from high-resolution optical remote sensing data. Traditional methods for analyzing the passability of complex scenes based on remote sensing data generally first extract the road network after an event using remote sensing images, and then compare it with the road network before the event to judge the passability of the road; or first use the road network before the event, and perform road blockage discrimination along the road direction to further judge the passability of the road. In terms of efficiency, these methods generally involve a lot of spatial operations between vectors and rasters, with complex steps and low efficiency. In terms of accuracy, first, although the accuracy of the road extraction method for images is constantly improving, there are still problems such as discontinuous road networks, which cannot guarantee the effective update of the road network. Second, using the road extraction method for images cannot fully guarantee that the passability of the area is fully represented. For example, wastelands, farmlands, etc. cannot be recognized as passable areas under special conditions. At the same time, due to reasons such as the registration error between the image and the road and the simplification of the map-scale road, the road and the image cannot be fully matched, which limits the accuracy of the passability analysis results.

[0004] Deep learning algorithms play an important role in extracting road-related information such as road passability assessment and road blockage information identification. Deep learning-related technologies such as convolutional neural networks and attention mechanisms have performed well in tasks such as image recognition, classification, and segmentation, providing new solutions for extracting road information from remote sensing images. By training deep learning models, it is possible to accurately identify road blockage information in remote sensing images and accurately assess road passability, greatly improving processing efficiency and accuracy. With the continuous development of deep learning technology, some new methods and technologies have also been introduced into the research on remote sensing road blockage information extraction. For example, deep learning models based on attention mechanisms can improve the model's ability to recognize road features by learning the weight distribution of different features; technologies based on transfer learning can use pre-trained models to improve the learning efficiency of new tasks; technologies based on generative adversarial networks (GANs) can generate more realistic remote sensing images and provide more training data for road information extraction.

[0005] However, the research on remote sensing road-related information extraction based on deep learning still faces some challenges. First, the road features in remote sensing images are complex and changeable, and the road types, materials, and widths in different regions vary greatly, which brings certain difficulties to the extraction of road blockage information. Secondly, there may be a large amount of interference information in non-road areas in remote sensing images, such as buildings, trees, etc., which will also affect the extraction of road information. In response to these challenges, researchers have adopted a variety of methods to improve the accuracy of road information extraction. How to construct a high-resolution optical remote sensing roadless path planning model suitable for complex scenes to reduce interference information in non-road areas and improve the accuracy of road information extraction is an urgent problem to be solved. Summary of the invention

[0006] The present invention provides a remote sensing image path planning method, system, device and medium to solve the problems in the prior art involving a large number of spatial operations between vectors and grids, complex steps and low efficiency, untimely updating of road networks, incomplete representation of road passability and incomplete matching between roads and images.

[0007] To achieve the above object, in a first aspect, the present invention relates to a remote sensing image path planning method, comprising:

[0008] The D-LinkNet network structure is combined with the HED network structure to obtain a road extraction model of a fully convolutional neural network, wherein in the decoding area of ​​the D-LinkNet network structure, a side output layer is set after each feature layer of different scales, each of the side output layers outputs an edge map, and a fusion layer connected to the output end of the side output layer is also set to integrate the edge maps output by each side output layer;

[0009] Constructing a new loss function, replacing the original loss function of the D-LinkNet network structure constructed by adding the cross entropy and Dice coefficient losses with the new loss function, wherein the new loss function is the sum of the loss of the road extraction model output and the loss of each side output layer added according to their respective weights;

[0010] Training the road extraction model based on a preset training set;

[0011] The trained road extraction model is used to extract the roads in the selected study area to obtain a road membership map in the form of raster data;

[0012] Based on the A* algorithm, path planning is performed on the road membership grid map to generate a travel path.

[0013] Preferably, the side output layer is composed of a 1*1 convolution layer and a deconvolution layer to ensure that the edge map of the side output matches the size of the original image;

[0014] The fusion layer uses the convolutional layer to learn the optimal combination weights between the outputs of different side output layers.

[0015] Preferably, a new loss function is constructed to replace the original loss function of D-LinkNet constructed by adding the cross entropy and Dice coefficient losses, specifically:

[0016] Set the inter-class balance coefficient β to calculate the positive sample weight Pos_W of the cross entropy:

[0017] in, Count_Neg is the total number of pixels of negative samples in the road target image, and Count_Pos is the total number of pixels of positive samples in the road target image;

[0018] Set the distance weight Dis_W based on the distance from the pixel to the nearest road centerline. Effect_Dis is the preset maximum effective distance, which is the distance that contributes to the calculation of road features selected based on the actual study area, and Dis(i) is the pixel distance from the i-th pixel to the nearest road centerline;

[0019] Perform mathematical morphological thinning on the road target image to obtain the road centerline;

[0020] Find the pixel distance from the pixel to the nearest road centerline;

[0021] The pixel distance is converted into a spatial Euclidean distance in meters according to the image resolution, and is brought into the formula for distance weight calculation;

[0022] Get the distance weight Dis_W, multiply the pixel's Pos_W and Dis_W as the final weight calculation result, and input it into the weighted cross entropy function as the new loss function of the network;

[0023] Based on the new loss function, the D-LinkNet network structure is improved.

[0024] Preferably, the training of the road extraction model based on a preset training set includes:

[0025] Prepare training sample sets and validation sample sets, convert road network vectors into raster to obtain road samples as the input target layer of the neural network;

[0026] The target layer and the indicator layer both use a combination of open source data sets and collected and accumulated historical data, and the data structures of the training sample set and the verification sample set both use open source data sets;

[0027] Training the road extraction model based on the training samples in the training sample set;

[0028] The road extraction model is verified based on the verification sample to obtain a trained road extraction model.

[0029] Preferably, the path planning is performed on the road membership grid map based on the A* algorithm to generate a pass path, specifically:

[0030] The A* algorithm uses the evaluation function f(n) to select the next node to be expanded. At each expansion, the node with the smallest f(n) value is selected and inserted into the linked list of possible paths to generate a pass path. The evaluation function f(n) consists of two parts: f(n) = g(n) + h(n), where g(n) is the actual cost from the starting point to node n, and h(n) is the estimated cost from node n to the end point.

[0031] The A* algorithm defines h(n) as the sum of the straight-line distance from node n to the end point and the difficulty of passing through node n:

[0032] h(n)=Dist(n)+Diff(n)

[0033] Diff(n)=1-DoM(n)

[0034] Where Dist(n) is the straight-line distance from node n to the end point, Diff(n) is the difficulty of traveling at node n, which is in the interval [0,1], and DoM(n) is the road membership of node n.

[0035] Preferably, it includes: adjusting the numerical range of the road membership degree of node n, and the adjusted h(n) is defined as:

[0036] h(n) = Dist(n) + Diff(n) * Dist(s, e), where Dist(n) is the straight-line distance from node n to the end point, Diff(n) is the traffic difficulty of node n, and Dist(s, e) is the straight-line distance from the starting point to the end point.

[0037] Preferably, the path planning of the road membership raster map based on the A* algorithm to generate a traffic path further includes: magnifying the road membership map in an upscaling manner with multiples increasing from large to small until the difference between the path planning results of two adjacent scales is less than a set threshold, then stop the operation, and retain the result of the last path planning as the traffic path.

[0038] Preferably, when the difference between the path planning results of two adjacent scales is less than the set threshold, then stop the operation, specifically: resample the path planning result of n-fold upscaling to obtain a first path planning result, so that the first path planning result has the same resolution as the path planning result of n-fold upscaling, perform a morphological dilation operation with a 2-fold expansion on the path planning result of n-fold upscaling to obtain a second path planning result, when the ratio of the line width of the second path planning result to the line width of the first path planning result of 2n-fold upscaling after resampling is greater than the set threshold T, then stop the operation, where T ∈ (1, 0), and retain the result of the last path planning, where n is a positive integer.

[0039] To achieve the above object, in a second aspect, the present invention relates to a remote sensing image path planning system, including:

[0040] A model construction module, which is used to combine the D-LinkNet network structure and the HED network structure to obtain a road extraction model of a fully convolutional neural network. Among them, in the decoding area of the D-LinkNet network structure, a side output layer is set after each feature layer of different scales, and each side output layer will output an edge map, and a fusion layer connecting the output ends of the side output layers is also set to synthesize the edge maps output by each side output layer;

[0041] A loss function update module, which is used to construct a new loss function, and replace the loss function originally constructed by adding cross entropy and Dice coefficient loss in the D-LinkNet network structure with the new loss function. The new loss function calculates the loss output by the finally fused road extraction model and the loss of each side output layer, and adds them according to weights as the total loss;

[0042] A model training module, used for training the road extraction model based on a preset training set;

[0043] A road membership map acquisition module is used to extract roads in the selected study area using the trained road extraction model to obtain a road membership map in the form of raster data;

[0044] The path planning module is used to perform path planning on the road membership grid map based on the A* algorithm to generate a pass path.

[0045] To achieve the above-mentioned purpose, in a third aspect, the present invention also relates to an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned remote sensing image path planning method when executing the computer program.

[0046] To achieve the above objectives, in a fourth aspect, the present invention also relates to a computer-readable storage medium, in which instructions are stored, and when the instructions are executed, the above remote sensing image path planning method is executed.

[0047] The remote sensing image path planning method, system, device and medium disclosed in the present invention have the following beneficial effects compared with the prior art:

[0048] The present invention realizes direct extraction of path planning results in remote sensing images with complex scene conditions. The method is based on HEDLink-RoadNet neural network and adaptive A* algorithm, and can directly perform path planning using remote sensing images without relying on road vector data support. A high-resolution optical remote sensing non-road network path planning model suitable for complex scenes is constructed, thereby realizing path planning without pre-building a road network, and using neural networks to realize effective extraction and fusion of road condition feature information in complex scenes. Path planning information is obtained from remote sensing images using a path planning method based on a deep neural network, realizing an end-to-end information extraction, replacing the cumbersome and time-consuming road extraction and vectorization steps in traditional remote sensing path planning methods. This not only simplifies the processing process, reduces the technical threshold and computational complexity, but also significantly improves the efficiency and accuracy of path planning information acquisition. Based on this, the accuracy and efficiency of path planning are improved, and the stable operation of the path planning system in complex scenes is guaranteed.

[0049] This method has application value in many scenarios, such as disaster emergency response, geological survey route analysis, or path planning in remote areas without road infrastructure. This method not only improves the flexibility and adaptability of the path planning system, but also provides new solutions for scenarios that are difficult to cover with traditional path planning methods.

[0050] The D-LinkNet network structure is combined with the HED (Holistically-Nested Edge Detection) network structure: referring to the two network structures, a network structure more suitable for road extraction from remote sensing images is constructed; at the same time, a new loss function is proposed and used to replace the loss function originally constructed by adding the cross-entropy and Dice coefficient losses in D-LinkNet, improving the accuracy of image road extraction.

[0051] It can generate a reasonable path planning scheme in the absence of road network data. At the same time, for areas with road network data, this method can also get rid of the dependence on road network vector data, thus realizing path planning without vector calculation and road network update, and generating accurate path planning results that meet actual needs. Brief Description of the Drawings

[0052] Figure 1 is the method flow of a remote sensing image path planning method in Embodiment 1 of the present invention Figure 1 ;

[0053] Figure 2 is the method flow of a remote sensing image path planning method in Embodiment 1 of the present invention Figure 2 ;

[0054] Figure 3 is the network structure diagram of the road extraction model combining the D-LinkNet network structure and the HED network structure of a remote sensing image path planning method in Embodiment 1 of the present invention;

[0055] Figure 4 is the structural schematic diagram of a remote sensing image path planning system in Embodiment 2 of the present invention;

[0056] Figure 5 is the structural schematic diagram of an electronic device in Embodiment 3 of the present invention. Detailed Embodiments

[0057] The present invention will be further described in detail below with reference to the drawings and embodiments. It can be understood that the specific embodiments described herein are only used to explain the present invention, rather than limiting the present invention. Additionally, it should be noted that for the sake of description, only parts related to the present invention are shown in the drawings, not all structures.

[0058] Embodiment 1

[0059] A remote sensing image path planning method, please refer to Figures 1 - 5, high-resolution optical remote sensing data roadless path planning for complex scenarios. This method can be implemented using an intelligent electronic device with a central processing unit, such as an unmanned vehicle with an intelligent electronic device, a robotic dog with an intelligent electronic device, a personal computer, an intelligent terminal, a server, etc. As Figure 1 shown, it includes the following steps: S101 to S106.

[0060] S101 combines the D-LinkNet network structure and the HED network structure to obtain a road extraction model of a fully convolutional neural network. Among them, in the decoding area of the D-LinkNet network structure, a side output layer is set after each feature layer of different scales. Each side output layer will output an edge map, and a fusion layer connecting the output ends of the side output layers is also set to synthesize the edge maps output by each side output layer.

[0061] The specific combination method of the D-LinkNet network structure and the HED network structure is to delete the decoding area structure of the original D-LinkNet network structure and introduce the side output layer in the HED network structure after each feature layer of different scales. Each side output layer will output an edge map. The specific structure is as Figure 3 shown by the side output layers of different sizes in the decoding area. At the same time, the fusion layer used to connect all side output layers in the HED network structure is also introduced to synthesize the edge maps output by each side output layer. The specific structure is as Figure 3 shown by the structure after deconvolution in the decoding area.

[0062] Among them, the side output layer consists of a 1*1 convolutional layer and a deconvolution layer to ensure that the edge map output by the side matches the size of the original image. The fusion layer uses a convolutional layer to learn the optimal combination weights between different outputs. The specific learning method is to connect the 4 feature layers with a size of 1024*1024 output after deconvolution and the last 1 activation output layer using a convolutional kernel. This convolutional kernel can continuously adjust its own weights during the network learning process, so the result after its training is the optimal combination weight required for the fusion of the side output layer results.

[0063] Among them, the D-LinkNet network structure consists of three parts: a residual network encoding area, a central area, and a decoding area, following the Encoder-Decoder architecture design: the encoding area is responsible for accurately encoding road information into feature information; the decoding area maps this encoded road feature information to the spatial domain to achieve accurate road segmentation. The D-LinkNet network model introduces a dilated convolution layer in the central area without adding additional learning parameters, greatly simplifying the training process. Among them, in the construction of the encoding area, D-LinkNet uses 1 initial convolution module and 4 residual modules based on the pre-trained ResNet34 network structure. The residual modules improve the generalization representation ability of the regional blocks through skip connections. In the core part of the central area, D-LinkNet realizes a hybrid connection with both series and parallel connections. D-LinkNet receives a 1024×1024 image as input and uses ResNet34 pre-trained on the ImageNet dataset as its encoder. ResNet34 was originally designed for the classification of medium-resolution images with a size of 256×256. Using a pooling layer may reduce the resolution of the central feature map and lose spatial information. Therefore, D-LinkNet uses a dilated convolution layer to enhance the network's perception ability of road feature information. Compared with using a pooling layer, the dilated convolution layer can avoid the loss of spatial information while maintaining the high resolution of the feature map.

[0064] The design idea of the HED network structure is also applicable to road extraction. First, HED was originally designed for edge detection. Therefore, this network was initially defined as a pure binary classification problem, which does not require strong semantic information but needs to accurately identify the position of the edge, making it more applicable to the road recognition task. Second, the design concept of the HED network is that it is expected that each layer in the network can output useful information, and then the outputs of several layers are combined through the Ensemble method. The shallow network in the network is used to identify the edge contour of the object, while the deep network is more conducive to the recognition of semantic information such as object categories because it contains a larger perception field. Finally, the HED network structure designs a unique loss function to address the problem of class imbalance.

[0065] The present invention absorbs the design features of the D-LinkNet and HED network structures and proposes a network structure - HEDLink-RoadNet that is more targeted at the problem of image road extraction.

[0066] HEDLink-RoadNet integrates the advantages of the D-LinkNet and HED network structures mainly from two aspects: network structure design and loss function design:

[0067] In terms of network structure design, a key feature of the HED model design is the addition of side-output layers after convolutional layers at different levels. Each side-output layer outputs an edge map, and the edge maps output by each side-output layer are fused through a fusion layer to obtain the final edge detection result. This strategy effectively combines feature representations at different levels, greatly improving the accuracy and robustness of edge detection.

[0068] Referring to the design of the side-output layer, modify the D-LinkNet network structure by adding side-output layers after each feature layer at different scales: In the decoder part of D-LinkNet, each layer of feature maps usually corresponds to different scales of the original image. After each such feature layer, a side-output layer can be added, which usually consists of a convolutional layer (possibly a 1x1 convolution to adjust the number of channels) and a possible upsampling layer to ensure that the edge map output by the side-output layer matches the size of the original image or has a similar resolution. The goal of each side-output layer is to capture and emphasize the edge information at that feature level. This can be achieved by training these layers to output a probability map or an edge map, where each pixel value represents the probability that the position is an edge.

[0069] To generate the final edge detection result, a fusion layer needs to be designed to integrate the edge maps of each side-output layer. This fusion layer can be a simple weighted average layer or a more complex network structure, such as using a convolutional layer to learn the optimal combination weights between outputs at different levels. To improve the flexibility of the model, the fusion layer uses a convolutional layer, and the weights are obtained through learning during the training process rather than being set manually. In this way, the model can automatically adjust the contribution degrees of outputs at different levels to adapt to specific tasks and datasets.

[0070] S102 Construct a new loss function to replace the original loss function of the D-LinkNet network structure, which is constructed by adding the cross-entropy and Dice coefficient losses. The new loss function is to calculate the loss of the finally fused road extraction model output and the loss of each side-output layer, and add them together with their respective weights as the total loss.

[0071] To make full use of the information from all side output layers and promote their learning during training, a multi-scale loss function is designed. This function not only calculates the loss of the final fused output but also calculates the losses of each side output layer separately, and adds these losses with certain weights as the total loss. Considering that the importance of outputs at different levels may vary, their losses can be weighted. Generally, the weights of higher-level outputs (closer to the original image size) can be set higher because they contain richer detailed information. The specific network structure of HEDLink-RoadNet is as shown in Figure 2 shown below.

[0072] In this embodiment, the specific setting steps of the new loss function are as follows:

[0073] S1021 Set the inter-class balance coefficient β to calculate the positive sample weight Pos_W of the cross-entropy:

[0074] Where, Count_Neg is the total number of pixels of negative samples in the road target image, and Count_Pos is the total number of pixels of positive samples in the road target image;

[0075] S1022 Set the distance weight Dis_W calculated based on the distance from the pixel to the nearest road centerline, where Effect_Dis is the preset maximum effective distance, which is the distance contributing to the calculation of road features selected according to the actual study area, and Dis(i) is the pixel distance from the i-th pixel to the nearest road centerline;

[0076] S1023 Perform mathematical morphological thinning on the road target image to obtain the road centerline;

[0077] S1024 Calculate the pixel distance from the pixel to the nearest road centerline;

[0078] S1025 Convert the pixel distance to the spatial Euclidean distance in meters according to the image resolution, and substitute it into the formula for distance weight calculation;

[0079] S1026 Obtain the distance weight Dis_W, multiply the Pos_W and Dis_W of the pixel as the final weight calculation result, and input it into the cross-entropy function with weights as the new loss function of the network;

[0080] S1027 Improve the D-LinkNet network structure based on the new loss function.

[0081] S103 Train the road extraction model based on the preset training set.

[0082] In some embodiments, S103 includes:

[0083] S1031 prepares a training sample set and a verification sample set, converts the road network vector into a raster to obtain a road sample as an input target layer of the neural network.

[0084] S1032 The target layer and indicator layer both use a combination of open source data sets and accumulated historical data. The data structures of the training sample set and the validation sample set both use open source data sets.

[0085] Before conducting network training, we must first prepare training sample sets and verification sample sets. The target layer of the sample set can use the information provided by the existing road network, or use open source road network data such as OpenStreetMap (OSM). The data structure of the sample set refers to the open source data set, and is reasonably transformed and adjusted according to the input requirements of the model. It is ensured that the samples cover a variety of terrains and landforms, ensuring the adaptability of the model to various actual situations.

[0086] S1033 trains the road extraction model based on the training samples in the training sample set;

[0087] S1034 verifies the road extraction model based on the verification sample to obtain a trained road extraction model.

[0088] S104 uses the trained road extraction model to extract roads in the selected study area to obtain a road membership map in the form of raster data.

[0089] S105 performs path planning on the road membership grid map based on the A* algorithm to generate a travel path.

[0090] The difficulty map is generated based on the road membership result map, and the path planning is carried out according to the minimum difficulty principle to generate the path between the starting point and the end point. An adaptive A* path search algorithm is proposed to search for the optimal path on the difficulty map.

[0091] The A* path search algorithm is a widely used heuristic search algorithm, especially suitable for solving the problem of searching and finding the optimal path in a graph. It is widely used in many fields such as robot navigation, game development and map navigation. The A* algorithm calculates the cost of each node that can be reached at the current location, and then selects the node with the lowest cost to add to the search space. This new node added to the search space is used to generate more possible paths. The A* algorithm combines the advantages of the Dijkstra algorithm and the greedy best-first search algorithm, and guides the search process by introducing a heuristic function, thereby improving the search efficiency while ensuring that the optimal path is found.

[0092] In this embodiment, S105 is specifically:

[0093] The A* algorithm uses the evaluation function f(n) to select the next node to expand. At each step of the expansion, the node with the minimum f(n) value will be selected and inserted into the linked list of possible paths. The evaluation function f(n) consists of two parts: f(n) = g(n) + h(n), where g(n) is the actual cost from the starting point to node n, and h(n) is the estimated cost from node n to the end point;

[0094] The A* algorithm defines h(n) as the sum of the straight-line distance from node n to the end point and the traversability difficulty of node n:

[0095] h(n) = Dist(n) + Diff(n)

[0096] Diff(n) = 1 - DoM(n)

[0097] In the formula, Dist(n) is the straight-line distance from node n to the end point, Diff(n) is the traversability difficulty of node n, which is in the range of [0, 1], and DoM(n) is the road membership of node n.

[0098] The basic steps of the A* algorithm can be summarized as follows:

[0099] 1. Initialization: Create two lists. The open list (OPEN list) is used to store the nodes to be examined, and the closed list (CLOSE list) is used to store the nodes that have been examined. Add the starting point to the OPEN list and set its g value and f value.

[0100] 2. Select node: Select the node with the minimum f value from the OPEN list as the current node and move it from the OPEN list to the CLOSE list.

[0101] 3. Expand node: For each adjacent node of the current node, calculate its g value, h value, and f value. If the adjacent node is not in the OPEN list, add it to the OPEN list; if the adjacent node is already in the OPEN list, compare the f value of the adjacent node with the f value of the current node and keep the smaller f value.

[0102] 4. Repeat steps: Repeat steps 2 and 3 until the end point is found or the OPEN list is empty.

[0103] 5. Backtrack path: Starting from the end point, backtrack to the starting point according to the pointer of the parent node to form the shortest path.

[0104] The selection of the heuristic function h(n) has an important impact on the performance of the algorithm. It should be able to reflect an approximation of the actual cost from the current node to the end point and ensure the optimality of the algorithm. Guided by the heuristic function h(n), the A* algorithm can effectively reduce the search space, avoid unnecessary searches, and thus improve the search efficiency. The selection of the heuristic function h(n) has an important impact on the performance of the algorithm and needs to be designed according to the characteristics of specific problems to adapt to different application scenarios. In practical applications, it may be necessary to consider the complexity of the map and the limitations of computing resources to balance the search speed and search accuracy. The A* algorithm may encounter multiple nodes with the same f value during the search process. At this time, certain strategies (such as preferentially expanding nodes with smaller g values) need to be adopted to avoid falling into local optimal solutions.

[0105] Based on this, an adaptive A* algorithm is proposed, which can adapt to different distance dimensions and image resolutions, avoiding the influence of distance dimensions and image resolutions on the results. It is used to directly apply the road extraction results of the neural network to path planning. The road extraction results based on the neural network are raster-form road membership maps, which need to be converted into passage difficulties. The numerical range of the road membership of node n is adjusted. By adjusting the numerical range of the road membership of node n, the influence brought by the dimension can be avoided, and then it can automatically adapt to different distance dimensions. Specifically, the straight-line distance from the starting node to the end point is used as the magnification factor to adjust the numerical range of the passage difficulty.

[0106] By adopting the method of upscaling the road membership raster data results, these problems can be effectively alleviated, the efficiency of the algorithm can be improved, and the path planning time can be reduced. When the roads on the image are clear and obvious, the influence of different magnification factors of upscaling on path planning is not significant. However, for the hidden roads that are not clear on some mountainous area images, excessive upscaling may cause the road extraction results to break, and the influence of different magnification factors of upscaling on path planning is relatively large. To improve the efficiency of the A* algorithm, it also includes: magnifying the road membership map in descending order of magnification factor until the difference between the path planning results of two adjacent scales is less than the set threshold, then stop the operation, and retain the result of the last path planning as the passage path.

[0107] In this embodiment, the numerical range of the road membership of node n is adjusted, and the adjusted h(n) is defined as:

[0108] h(n) = Dist(n) + Diff(n) * Dist(s, e), where Dist(n) is the straight-line distance from node n to the end point, Diff(n) is the passage difficulty of node n, and Dist(s, e) is the straight-line distance from the starting point to the end point.

[0109] By adopting the method of upscaling the road membership raster data results, these problems can be effectively alleviated, the efficiency of the algorithm can be improved, and the time for path planning can be reduced. When the roads on the image are clear and obvious, different upscaling multiples have little effect on path planning. However, for some unclear hidden roads in mountainous areas, excessive upscaling may cause the road extraction results to be broken, and different upscaling multiples have a greater impact on path planning.

[0110] In this embodiment, in order to improve the efficiency of the A* algorithm, S105 also includes: amplifying the road membership graph in order from large to small multiples until the difference between the path planning results of two adjacent scales is less than a set threshold, then stopping the operation and retaining the result of the last path planning as the pass path.

[0111] The specific criteria for distinguishing between the two times are:

[0112] Specifically, the path planning result upscaled n times is resampled to obtain the first path planning result, so that the first path planning result and the path planning result upscaled n times have the same resolution, and the path planning result upscaled n times is expanded by 2 times by morphological dilation to obtain the second path planning result, so that the line width of the second path planning result is greater than the line width of the first path planning result upscaled 2n times after resampling. When the ratio is greater than the set threshold T, the operation is stopped, where T∈(1,0), and the result of the last path planning is retained, where n is a positive integer.

[0113] The ratio t of the line width of the second path planning result to the line width of the first path planning result after resampling and 2n times upscaling:

[0114]

[0115] Among them, I 2n is the result of 2n times upscaling, I n is n times the upscaling, ∩ is the intersection of the results, and ∪ is the union of the results. The values ​​of n are 4, 2, and 1. When the original image size is 1024*1024 pixels, the corresponding upscaled image sizes are 128*128 pixels, 256*256 pixels, and 512*512 pixels, respectively. When t is greater than the set threshold T, the operation is stopped and the result of the last path planning is retained. After testing and comparison, when T is set to 0.85, both efficiency and accuracy can be guaranteed.

[0116] Embodiment 2

[0117] A remote sensing image path planning system for high-resolution optical remote sensing data without road network path planning in complex scenarios. In this embodiment, the system can be implemented by an intelligent electronic device with a central processor, such as an unmanned vehicle with an intelligent electronic device, a robotic dog with an intelligent electronic device, a personal computer, an intelligent terminal, a server, etc. Please refer to Figure 4 , which includes a model construction module 61, a loss function update module 62, a model training module 63, a road membership map acquisition module 64, and a path planning module 65.

[0118] The model construction module 61 is used to combine the D-LinkNet network structure and the HED network structure to obtain a road extraction model of a fully convolutional neural network. Among them, in the decoding area of the D-LinkNet network structure, a side output layer is set after each feature layer of different scales, and each side output layer will output an edge map. A fusion layer connecting the output ends of the side output layers is also set to synthesize the edge maps output by each side output layer.

[0119] Among them, the side output layer is composed of a 1*1 convolutional layer and a deconvolutional layer to ensure that the edge map output by the side matches the size of the original image;

[0120] The fusion layer uses a convolutional layer to learn the optimal combination weights between the outputs of different side output layers.

[0121] The loss function update module 62 is used to construct a new loss function to replace the loss function originally constructed by adding cross-entropy and Dice coefficient loss in the D-LinkNet network structure. The new loss function calculates the loss of the output of the finally fused road extraction model and the loss of each side output layer, and adds them according to weights as the total loss.

[0122] The model training module 63 is used to train the road extraction model based on a preset training set;

[0123] The road membership map acquisition module is used to extract the roads in the selected study area by using the trained road extraction model to obtain a road membership map in the form of raster data.

[0124] The path planning module 64 is used to perform path planning on the road membership raster map based on the A* algorithm to generate a passing path.

[0125] In this embodiment, the loss function update module 62 is specifically used for:

[0126] Construct a new loss function to replace the loss function originally constructed by adding cross-entropy and Dice coefficient loss in D-LinkNet, specifically:

[0127] Set the inter-class balance coefficient β to calculate the positive sample weight Pos_W of the cross-entropy:

[0128] Where Count_Neg is the total number of pixels of negative samples in the road target image, and Count_Pos is the total number of pixels of positive samples in the road target image;

[0129] Set the distance weight Dis_W calculated based on the distance from the pixel to the nearest road centerline. Among them Effect_Dis is the preset maximum effective distance, which is the distance contributing to the calculation of road features selected according to the actual research area. Dis(i) is the pixel distance from the i-th pixel to the nearest road centerline;

[0130] Perform mathematical morphological thinning processing on the road target image to obtain the road centerline;

[0131] Find the pixel distance from the pixel to the nearest road centerline;

[0132] Convert the pixel distance to the spatial Euclidean distance in meters according to the image resolution, and substitute it into the formula for distance weight calculation;

[0133] Obtain the distance weight Dis_W, multiply the Pos_W and Dis_W of the pixel as the final weight calculation result, and input it into the cross-entropy function with weights as the new loss function of the network;

[0134] Improve the D-LinkNet network structure based on the new loss function.

[0135] In this embodiment, the path planning module 64 is specifically as follows: The A* algorithm uses the evaluation function f(n) to select the next node to be expanded, and at each expansion, the node with the minimum f(n) value will be selected and inserted into the linked list of possible paths to generate a traversable path. The evaluation function f(n) consists of two parts: f(n) = g(n) + h(n), where g(n) is the actual cost from the starting point to node n, usually the distance or cost, and h(n) is the estimated cost from node n to the end point (heuristic function);

[0136] The A* algorithm defines h(n) as the sum of the straight-line distance from node n to the end point and the traversability difficulty of node n:

[0137] h(n) = Dist(n) + Diff(n)

[0138] Diff(n) = 1 - DoM(n)

[0139] Wherein, Dist(n) is the straight-line distance from node n to the end point, Diff(n) is the difficulty of passing through node n, which is in the range of [0, 1], and DoM(n) is the road membership degree of node n.

[0140] The adjusted h(n) is defined as:

[0141] h(n) = Dist(n) + Diff(n) * Dist(s, e), where Dist(n) is the straight-line distance from node n to the end point, Diff(n) is the difficulty of passing through node n, and Dist(s, e) is the straight-line distance from the starting point to the end point.

[0142] To improve the efficiency of the A* algorithm, the path planning module 64 further includes: successively magnifying the road membership degree map in ascending order of magnification multiples until the difference between the path planning results of two adjacent scales is less than a set threshold, then stop the operation, and retain the result of the last path planning as the passing path.

[0143] Until the difference between the path planning results of two adjacent scales is less than the set threshold, then stop the operation. Specifically: resample the path planning result of n-fold upscaling to obtain the first path planning result, so that the first path planning result has the same resolution as the path planning result of n-fold upscaling, perform a morphological dilation operation of 2-fold expansion on the path planning result of n-fold upscaling to obtain the second path planning result, and stop the operation when the ratio of the line width of the second path planning result to the line width of the first path planning result of 2n-fold upscaling after resampling is greater than the set threshold T, where T ∈ (1, 0), and retain the result of the last path planning, where n is a positive integer.

[0144] The implementation process, method and effect of a remote sensing image path planning system in this embodiment are the same as those described in Embodiment 1, and will not be elaborated here.

[0145] Embodiment 3

[0146] Such as Figure 5As shown, this embodiment relates to an electronic device including at least one processor and a memory communicatively connected to the at least one processor. Among them, the memory stores a computer program that can be run by the at least one processor. The computer program is executed by the at least one processor so that the at least one processor can execute a remote sensing image path planning method of Embodiment 1 and achieve the corresponding beneficial effects of a remote sensing image path planning method, which will not be elaborated here. The electronic device provided in this embodiment can be a personal computer, such as a desktop computer, an all-in-one computer, a laptop computer, a tablet computer, etc., and can also be a terminal device such as a mobile phone, a wearable device, a personal digital assistant, etc. In this embodiment, the electronic device is a flight control computer. The electronic device is only an example and should not impose any limitations on the functions and usage scope of the embodiments of the present invention.

[0147] The components of the electronic device 3 may include, but are not limited to: the above-mentioned at least one processor 4, the above-mentioned at least one memory 5, and a bus 6 connecting different system components (including the memory 5 and the processor 4).

[0148] The bus 6 includes a data bus, an address bus, and a control bus.

[0149] The memory 5 may include a volatile memory, such as a random access memory (RAM) 51 and / or a cache memory 52, and may further include a read-only memory (ROM) 53.

[0150] The memory 5 may further include a program / utilities 55 having a set (at least one) of program modules 54. Such program modules 54 include, but are not limited to: an operating system, one or more application programs, other program modules, and program data. Each or some combination of these examples may include the implementation of a network environment.

[0151] The processor 4 executes various functional applications and data processing by running the computer program stored in the memory 5, such as the above-mentioned remote sensing image path planning method.

[0152] The electronic device 3 can also communicate with one or more external devices 7 (such as a keyboard, a pointing device, etc.). Such communication can be carried out through an input / output (I / O) interface 8. And, the electronic device 3 can also communicate with one or more networks (such as a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) through a network adapter 9. As Figure 5 shown, the network adapter 9 communicates with other modules of the electronic device 3 through the bus 6. It should be understood that although Figure 5Not shown in the figure, other hardware and / or software modules can be used in combination with the electronic device 3, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID (Redundant Array of Independent Disks) systems, tape drives, and data backup storage systems, etc.

[0153] It should be noted that although several units / modules or sub-units / modules of the electronic device are mentioned in the above detailed description, this division is merely exemplary and not mandatory. In fact, according to the embodiments of the present invention, the features and functions of two or more units / modules described above can be embodied in one unit / modules. Conversely, the features and functions of one unit / modules described above can be further divided and embodied by multiple units / modules.

[0154] Embodiment 4

[0155] The present invention relates to a computer-readable storage medium storing instructions, which when running, execute a remote sensing image path planning method according to Embodiment 1. The implementation process, method, and effect during its running are the same as those of the remote sensing image path planning method described in Embodiment 1, and will not be elaborated here.

[0156] It should be noted that in this article, the term "including", "comprising", or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article, or device including a series of elements not only includes those elements but also includes other elements not explicitly listed, or further includes elements inherent to such a process, method, article, or device. Without further limitation, an element defined by the statement "including one..." does not exclude the existence of another identical element in the process, method, article, or device including that element.

[0157] The above are only the preferred embodiments of the present invention, and do not limit the patent scope of the present invention accordingly. Any equivalent structural or equivalent process transformation made by using the specification and drawings of the present invention, or directly or indirectly applied to other related technical fields, shall be equally included in the patent protection scope of the present invention.

Claims

1. A remote sensing image path planning method, characterized in that: include: The D-LinkNet network structure is combined with the HED network structure to obtain a road extraction model of a fully convolutional neural network, wherein in the decoding area of ​​the D-LinkNet network structure, a side output layer is set after each feature layer of different scales, each of the side output layers outputs an edge map, and a fusion layer connected to the output end of the side output layer is also set to integrate the edge maps output by each side output layer; Constructing a new loss function, replacing the original loss function of the D-LinkNet network structure constructed by adding the cross entropy and Dice coefficient losses with the new loss function, wherein the new loss function is the sum of the loss of the road extraction model output and the loss of each side output layer added according to their respective weights; Training the road extraction model based on a preset training set; The trained road extraction model is used to extract the roads in the selected study area to obtain a road membership map in the form of raster data; Based on the A* algorithm, path planning is performed on the road membership grid map to generate a travel path.

2. A remote sensing image path planning method according to claim 1, characterized in that: The side output layer is composed of a 1*1 convolutional layer and a deconvolutional layer to ensure that the edge map of the side output matches the size of the original image; The fusion layer uses the convolutional layer to learn the optimal combination weights between the outputs of different side output layers.

3. A remote sensing image path planning method according to claim 2, characterized in that: A new loss function is constructed to replace the original loss function of D-LinkNet constructed by adding the cross entropy and Dice coefficient losses, specifically: Set the inter-class balance coefficient β to calculate the positive sample weight Pos_W of the cross entropy: in, Count_Neg is the total number of pixels of negative samples in the road target image, and Count_Pos is the total number of pixels of positive samples in the road target image; Set the distance weight Dis_W based on the distance from the pixel to the nearest road centerline, where Effect_Dis is the preset maximum effective distance, which is the distance that contributes to the calculation of road features selected based on the actual study area, and Dis(i) is the pixel distance from the i-th pixel to the nearest road centerline; Perform mathematical morphological thinning on the road target image to obtain the road centerline; Find the pixel distance from the pixel to the nearest road centerline; The pixel distance is converted into a spatial Euclidean distance in meters according to the image resolution, and is brought into the formula for distance weight calculation; Get the distance weight Dis_W, multiply the pixel's Pos_W and Dis_W as the final weight calculation result, and input it into the weighted cross entropy function as the new loss function of the network; Based on the new loss function, the D-LinkNet network structure is improved.

4. A remote sensing image path planning method according to claim 3, characterized in that: The training of the road extraction model based on a preset training set includes: Prepare training sample sets and validation sample sets, convert road network vectors into raster to obtain road samples as the input target layer of the neural network; The target layer and the indicator layer both use a combination of open source data sets and collected and accumulated historical data, and the data structures of the training sample set and the verification sample set both use open source data sets; Training the road extraction model based on the training samples in the training sample set; The road extraction model is verified based on the verification sample to obtain a trained road extraction model.

5. The remote sensing image path planning method according to claim 1, characterized in that: The path planning is performed on the road membership grid map based on the A* algorithm to generate a pass path, specifically: The A* algorithm uses the evaluation function f(n) to select the next node to be expanded. At each expansion, the node with the smallest f(n) value is selected and inserted into the linked list of possible paths to generate a pass path. The evaluation function f(n) consists of two parts: f(n) = g(n) + h(n), where g(n) is the actual cost from the starting point to node n, and h(n) is the estimated cost from node n to the end point. The A* algorithm defines h(n) as the sum of the straight-line distance from node n to the end point and the difficulty of traveling to node n: h(n)=Dist(n)+Diff(n) Diff(n)=1-DoM(n) Where Dist(n) is the straight-line distance from node n to the end point, Diff(n) is the difficulty of traveling at node n, which is in the interval [0,1], and DoM(n) is the road membership of node n.

6. A remote sensing image path planning method according to claim 5, characterized in that: include: The road membership of node n is adjusted to a numerical range, and the adjusted h(n) is defined as: h(n)=Dist(n)+Diff(n)*Dist(s,e), where Dist(n) is the straight-line distance from node n to the end point, Diff(n) is the difficulty of passing node n, and Dist(s,e) is the straight-line distance from the starting point to the end point.

7. A remote sensing image path planning method according to claim 5, characterized in that: The method of performing path planning on the road membership grid map based on the A* algorithm to generate a pass path also includes: sequentially amplifying the road membership map by multiples from large to small until the difference between the path planning results between two adjacent scales is less than a set threshold, then stopping the operation and retaining the result of the last path planning as the pass path.

8. A remote sensing image path planning method according to claim 7, characterized in that: The operation is stopped until the difference between the path planning results of two adjacent scales is less than a set threshold. Specifically, the path planning result upscaled n times is resampled to obtain a first path planning result, so that the first path planning result and the path planning result upscaled n times have the same resolution, and the path planning result upscaled n times is expanded by a 2-fold morphological dilation operation to obtain a second path planning result, so that the line width of the second path planning result is greater than the line width of the first path planning result upscaled 2n times after resampling. The operation is stopped, where T∈(1,0), and the result of the last path planning is retained, where n is a positive integer.

9. A remote sensing image path planning system, characterized in that: include: A model building module, for combining a D-LinkNet network structure with a HED network structure to obtain a road extraction model of a fully convolutional neural network, wherein in a decoding area of ​​the D-LinkNet network structure, a side output layer is provided after each feature layer of different scales, each of the side output layers outputs an edge map, and a fusion layer connected to the output end of the side output layer is provided to synthesize the edge maps output by each side output layer; A loss function update module, used to construct a new loss function, replacing the original loss function of the D-LinkNet network structure constructed by adding the cross entropy and Dice coefficient losses with the new loss function, wherein the new loss function calculates the loss of the final fused road extraction model output and the loss of each side output layer, and adds them according to the weights as the total loss; A model training module, used for training the road extraction model based on a preset training set; A road membership map acquisition module is used to extract roads in the selected study area using the trained road extraction model to obtain a road membership map in the form of raster data; The path planning module is used to perform path planning on the road membership grid map based on the A* algorithm to generate a travel path.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, a remote sensing image path planning method as described in any one of claims 1-8 is implemented.

11. A computer-readable storage medium, characterized in that: The storage medium stores instructions, which, when executed, execute a remote sensing image path planning method as described in any one of claims 1-8.

Images