A road extraction method and system based on dynamic routing neural network

Abstract

This paper discloses a road extraction method and system based on a dynamic routing neural network. The system uses a DR-Net network to downsample the original input image to obtain a feature map. Based on the scale distribution of real-time input data, the feature map is processed using a routing space, and data-dependent routing paths are selected to form a scale-adaptive dynamic neural network structure. The dynamic neural network structure is decoded using the DR-Net network to obtain the corresponding feature map. A loss function is used to correct errors in the dynamic routing neural network and upsample the feature map, achieving end-to-end road extraction. This system can effectively extract multi-scale roads in reality and, by taking computational constraints into account, can further reduce costs and improve network efficiency.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and in particular relates to a road extraction method and system based on a dynamic routing neural network. Background Art

[0002] In many fields such as unmanned driving, traffic management, smart cities, path planning, and object detection, the research on road information extraction has very practical and far-reaching significance. Since the 21st century, with the popularization of high-resolution remote sensing images, scholars at home and abroad have carried out in-depth research on road extraction based on high-resolution remote sensing images. The earliest method of road extraction from remote sensing images was for professionals to manually mark road areas through experience and computer interaction. Although this type of road target extracted by manual visual inspection is accurate and reliable, the marking personnel must be relevant professionals and have a certain foundation of remote sensing knowledge. When faced with massive data, this traditional method is often limited. In recent years, with the rapid development of deep learning methods, data-driven methods represented by CNN have been widely used in road extraction in remote sensing images due to their absolute advantages in deep expression of image features and automatic extraction in image processing. The existing road extraction algorithms mainly include:

[0003] 1. Before the emergence of convolutional neural networks, many scholars have continuously explored and researched methods for road extraction from remote sensing images. Based on manually determined features, they have proposed many classic road extraction methods, which can be mainly divided into five types: classification-based algorithms, knowledge-based algorithms, mathematical morphology algorithms, active contour model algorithms, and dynamic programming algorithms.

[0004] 2. Deep learning extraction algorithms (FCN, UNet, D-LinkNet, DeepLab series, HRNetV2, etc.) that use multi-scale sampling methods for road extraction.

[0005] Although existing algorithms have further improved their ability to extract roads, there is a lack of a universal network model for roads of multiple scales. In order to improve the universality of deep learning to extract road networks from remote sensing images, there is a lack of a multi-scale network model for extracting roads.

[0006] The existing methods for road extraction have the following problems:

[0007] 1. Traditional algorithms based on manually determined features. Although these algorithms improve the efficiency and accuracy of road extraction, they all rely on the characteristics of different remote sensing images and road scenes, and use manual experience to design reasonable classification or inference rules for road extraction. The entire process requires setting different parameters based on image characteristics or performing a large number of graphic and curve evolution operations. In addition, the applicability of different methods is relatively narrow and cannot be applied to large-scale automated road extraction work.

[0008] 2. Regarding the multi-scale deep learning network extraction method, although this type of algorithm can achieve automatic and efficient road extraction through deep learning, since this method expresses various road data of different scenarios and scales in a single network structure, it requires a large amount of representative road label data for training and learning to optimize parameters. This method has a high cost of manual labeling in the early data preparation stage. Summary of the Invention

[0009] The technical problem to be solved by the present invention is to address the deficiencies in the above-mentioned prior art and provide a road extraction method and system based on a dynamic routing neural network, so as to solve the technical problems that the existing methods have narrow applicability and cannot be applied to large-scale automated road extraction, as well as the high cost of manual labeling in the early data preparation stage.

[0010] The present invention adopts the following technical solutions:

[0011] A road extraction method based on a dynamic routing neural network comprises the following steps:

[0012] S1. Downsample the original input image through the DR-Net network to obtain the feature map;

[0013] S2. Based on the scale distribution of the real-time input data, the feature map obtained in step S1 is processed using the routing space, and a data-dependent routing path is selected to form a scale-adaptive dynamic neural network structure.

[0014] S3, decoding the dynamic neural network structure obtained in step S2 through the DR-Net network to obtain the corresponding feature map;

[0015] S4. Use the loss function to correct the error of the dynamic routing neural network and upsample the feature map obtained in step S3 to achieve end-to-end road extraction.

[0016] Specifically, in step S1, the original input image is downsampled by 1 / 4 through two scale-adaptive convolutions of size 3×3 to obtain a feature map with a size of 1 / 4 of the original input.

[0017] Furthermore, the scale regression layer initialization formula of scale adaptive convolution is:

[0018]

[0019] Among them, ω0 is the initialization convolution kernel of the scale regression layer, b0 is the initialization bias of the scale regression layer, a is the position in the convolution kernel, ε is the sample, N is the normal distribution, and σ is the standard deviation.

[0020] Specifically, in step S2, the data-dependent routing path is selected to form a scale-adaptive dynamic neural network structure as follows:

[0021] The routing space is divided into L layers and consists of several routing units. The number of routing units in the first three layers is 1, 2, and 3 respectively, and the number of routing units in each of the remaining layers does not exceed 4. The sampling ratio of adjacent routing units is 2, and the minimum scale of the feature map is 1 / 32 of the original input size. Routing units of the same level with the same scale support skip connections during the forward propagation process. The routing units of the previous and next layers support multi-path routing, that is, when the feature scale is transformed, the routing node selects at most 3 scale transformation paths.

[0022] Furthermore, each routing unit is divided into a feature fusion module and a switch module; the feature processing module is responsible for fusing and transforming input features of different scales; the switch module selects the routing path to connect to the next layer based on the input features of different scales. There are three options in total: 2x upsampling, maintaining the same resolution, and 2x downsampling. The switch module supports multi-path selection and supports cross-connection connection between routing units.

[0023] Furthermore, multi-path selection is specifically as follows:

[0024] When performing scale transformation, routing unit A performs 2x upsampling to connect to routing unit B, maintains resolution to connect to routing unit F, and downsamples twice to connect to routing unit C, selecting a total of three different paths. Routing unit E connects to routing units F and G through 2x upsampling and resolution-maintaining scale transformation operations, selecting a total of two different paths. Routing unit A maintains resolution and directly connects to routing unit F, thus realizing jump connections between routing units.

[0025] Specifically, in step S3, the decoding operation is as follows:

[0026] The 1 / 32 size feature map is upsampled by 2 times and superimposed on the 1 / 16 size feature map. The 1 / 16 size result generated by the superposition is upsampled by 2 times and superimposed on the 1 / 8 size feature map. The 1 / 8 size result generated by the superposition is upsampled by 2 times and superimposed on the 1 / 4 size feature map, and finally upsampled by 4 times.

[0027] Specifically, in step S4, the loss function Loss used by the dynamic neural network structure is:

[0028] Loss = λ1·Loss Net +λ2·Loss Cal

[0029] Among them, λ1 is the loss function coefficient of the network part, λ2 is the loss function coefficient of the calculation constraint part, and Loss Net is the loss function of the network part, Loss Cal is the loss function for computing constraints.

[0030] Furthermore, the loss function of the network part is Loss Net Including Dice coefficient loss function Loss Cal And binary cross entropy loss function Loss Cross , as follows:

[0031]

[0032]

[0033]

[0034] Among them, N is the total number of pixels in the input image, p i To represent the road probability of the i-th pixel in the prediction result, g i is the true value of the i-th pixel in the true image, smooth is 10-7 for smoothing, y is the true value label of the category, and a is the probability that the sample is a positive example;

[0035] Calculate the constraint loss function Loss Cal for:

[0036] Loss Cal =(C(Space) / C0-u 2 )

[0037] Among them, C(Space) is the overall computational cost of the routing space, C0 is the given computational constraint, and u is the coefficient of the budget constraint.

[0038] In a second aspect, an embodiment of the present invention provides a road extraction system based on a dynamic routing neural network, comprising:

[0039] The sampling module downsamples the original input image through the DR-Net network to obtain the feature map;

[0040] The processing module uses the routing space to process the feature maps obtained by the sampling module according to the scale distribution of the real-time input data, and selects data-dependent routing paths to form a scale-adaptive dynamic neural network structure;

[0041] The decoding module decodes the dynamic neural network structure obtained by the processing module through the DR-Net network to obtain the corresponding feature map;

[0042] The extraction module uses the loss function to correct the error of the dynamic routing neural network and upsamples the feature map obtained by the decoding module to achieve end-to-end road extraction.

[0043] Compared with the prior art, the present invention has at least the following beneficial effects:

[0044] A road extraction method based on a dynamic routing neural network first obtains a feature map of the data through downsampling and outputs the feature map to the routing space. Through processing in the routing space, data-dependent routing paths are selected according to the scale distribution of real-time input data to form a network structure. That is, the network structure changes with changes in the input data. At the same time, considering the computational cost constraints in real-world scenarios, DR-Net incorporates computational constraints into the loss function. When a computational budget is given, the computational cost is further reduced and the network efficiency is improved. The loss function solves the imbalanced distribution problem of foreground and background, which greatly enhances the robustness of the network structure and achieves better extraction results. DR-Net learns the multi-scale feature vectors of the road through a scale-adaptive lightweight convolution operation module. Then, through activation function calculation, the routing path probability of each scale transformation is obtained, and different routing paths are selected to form a network structure. Compared with the traditional attention mechanism, DR-Net is more capable of mining multi-scale information in the data. It aims to alleviate the problem of limited model expression ability when a single structure faces multi-scale data, thereby improving the efficiency of road extraction.

[0045] Furthermore, the original input is downsampled by 1 / 4 through two 3×3 scale-adaptive convolutions to obtain a feature map with a size of 1 / 4 of the original input. This can characterize the edge features of the image and reduce calculations. At the same time, the use of adaptive convolution can better mine multi-scale road features.

[0046] Furthermore, by setting the initialization formula of the scale regression layer of the scale-adaptive convolution, the iterative process can be better carried out through empirical values, so that the initial value is closer to the optimal value, the number of iterations and optimization time are reduced, and the computational efficiency is improved.

[0047] Furthermore, the setting of routing units can better perform data-driven routing space selection; for the skip connections, it can better utilize low-dimensional edge features and high-level abstract features, superimpose the two feature maps, and better mine the edges of the road and the pixels where they are located; supporting multiple routing options in the front and back layers can give full play to the structural advantages of dense connections in the routing space, and can realize the sharing of parameters of the front and back connections, reducing the model size.

[0048] Furthermore, in order to calculate the probability of selecting each path, three paths are set up, including upsampling, downsampling, and unchanged selection paths. The purpose is to ensure that the selection can be adaptive under data-driven conditions; the feature fusion module is used to unify and transform features of different scales, better mine road features at different scales, and better perform road extraction tasks.

[0049] Furthermore, in order to give full play to the structural advantages of dense connections in the routing space, the model can select the optimal network structure under data-driven conditions.

[0050] Furthermore, in order to perform a series of upsampling and restoration operations on the downsampled feature maps of the encoding operation, the feature maps of the pre-structure are restored to the structure of the original image size, thereby performing the end-to-end road extraction task.

[0051] Furthermore, in order to iterate the training of the network, the most important thing for the neural network fitting task is to compare the prediction and the true value at the end of the training, so as to backpropagate the network parameters. The loss function setting can better fit the network parameters and the iterative model prediction effect.

[0052] Furthermore, the Dice coefficient loss function Loss Cal And binary cross entropy loss function Loss Cross , calculate the constraint loss function Loss Cal In order to solve the problem of "foreground-background imbalance", the DR-Net model also uses the Dice coefficient combined with the binary cross entropy loss function as the loss function of the network part, thereby reducing the impact of uneven positive and negative samples during training.

[0053] It can be understood that the beneficial effects of the second aspect mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here.

[0054] In summary, the present invention proposes a dynamic routing neural network DR-Net, which not only alleviates the problem of limited model expression ability when a single result faces multi-scale data, but also improves the efficiency of road extraction. The DR-Net model can adjust the network structure according to changes in real-time input data, and can effectively extract multi-scale roads in reality. Finally, computational constraints are also taken into account, which can further reduce costs and improve network efficiency.

[0055] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 This is the DR-Net structure diagram;

[0057] Figure 2 Diagram of the scale-adaptive convolution process;

[0058] Figure 3 is the routing space graph;

[0059] Figure 4 Schematic diagram for skip connections and multi-path selection;

[0060] Figure 5 It is the routing process diagram of the routing unit;

[0061] Figure 6 Comparison of extraction results in different network fog scenes, where (a) is the original image, (b) is the mask image, (c) is DeepLabV3, (d) is U-Net, (e) is HRNetV2, and (f) is DR-Net-A. DETAILED DESCRIPTION

[0062] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0063] In the description of the present invention, it is to be understood that the terms “include” and “comprise” indicate the presence of the described features, wholes, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or collections thereof.

[0064] It should also be understood that the terms used in the present specification are only for the purpose of describing particular embodiments and are not intended to limit the present invention. As used in the present specification and the appended claims, the singular forms "a", "an", and "the" are intended to include the plural forms unless the context clearly indicates otherwise.

[0065] It should be further understood that the term "and / or" as used in the present specification and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items. For example, A and / or B may represent: A exists alone, A and B exist simultaneously, and B exists alone. In addition, the character " / " in the present invention generally indicates that the associated objects are in an "or" relationship.

[0066] It should be understood that although the terms "first," "second," and "third" may be used to describe preset ranges in embodiments of the present invention, these preset ranges should not be limited to these terms. These terms are merely used to distinguish one preset range from another. For example, without departing from the scope of embodiments of the present invention, the first preset range may also be referred to as the second preset range, and similarly, the second preset range may also be referred to as the first preset range.

[0067] The word "if," as used herein, may be interpreted as "at the time of" or "when" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrases "if it is determined" or "if (stated condition or event) is detected" may be interpreted as "when it is determined" or "in response to the determination" or "when detecting (stated condition or event)" or "in response to detecting (stated condition or event)," depending on the context.

[0068] The accompanying drawings illustrate various schematic diagrams of structures according to embodiments disclosed herein. These figures are not drawn to scale; for clarity, some details are exaggerated and some details may be omitted. The shapes of the various regions and layers shown in the figures, as well as their relative sizes and positional relationships, are merely exemplary and may deviate in practice due to manufacturing tolerances or technical limitations. Those skilled in the art may design regions / layers with different shapes, sizes, and relative positions as needed.

[0069] The present invention provides a road extraction method based on a dynamic routing neural network. Taking into account the narrow applicability of artificial features and the unsuitability of current mainstream deep learning network extraction methods for large-scale use, the method can generate relevant routes based on the scale distribution of road data, selectively fuse and transform features of different scales, and generate a data-adaptive network structure, thereby effectively alleviating the imbalanced distribution of roads and backgrounds and multi-scale problems. Considering that practical application scenarios of road extraction often have computational cost constraints such as model computational complexity and parameter quantity, scale-adaptive convolution is used to replace standard convolution to reduce the number of network parameters and computational complexity, thereby achieving lightweight processing of the model. In addition, in the road extraction algorithm, a switch module is designed to add computational cost constraints to the loss function. On the basis of achieving computational cost control, reasonable pruning of the model under different constraint conditions is satisfied. That is, at the expense of lower extraction accuracy, the computational overhead of the model is greatly reduced, meeting the scenario requirements of various constraint conditions in real application scenarios.

[0070] The present invention provides a road extraction method based on a dynamic routing neural network, comprising the following steps:

[0071] S1 and STEM modules perform downsampling to obtain feature maps;

[0072] See also Figure 1 , is the structure of the DR-Net network, which is mainly divided into four parts: STAM module, routing space, decoder, and loss function.

[0073] Figure 1 The STEM module in the upper left corner downsamples the original input image by 1 / 4 using two 3×3 scale-adaptive convolutions, outputting a feature map 1 / 4 the size of the original input to the routing space. Using scale-adaptive convolutions can significantly reduce the number of network parameters and computational cost, enabling lightweight processing of network inputs and significantly lowering computational requirements.

[0074] See also Figure 2 The DR-Net used in this paper uses scale-adaptive convolution in the feature processing modules of the STEM module and the routing unit. The convolution operation principle remains unchanged throughout the scale-adaptive convolution process. Compared with conventional convolution, scale-adaptive convolution uses fewer parameters and lowers computational cost while ensuring the same feature map.

[0075] Convolutional neural networks offer advantages such as simple structure and high accuracy. However, their feature extraction phase uses fixed-size convolution kernels, making them unable to adaptively extract detailed lane information at multiple scales. To address this issue, a scaling factor is introduced to adaptively expand and contract the convolution kernel. Scale-adaptive convolution kernels automatically adjust the size of their receptive field based on the size of the object.

[0076] The initialization formula of the scale regression layer is:

[0077]

[0078] Among them, ω0 is the initialization convolution kernel of the scale regression layer, b0 is the initialization bias of the scale regression layer, and a is the position in the convolution kernel.

[0079] The initial scale coefficient generated by formula (3) is almost close to 1, that is, the scale-adaptive convolution will start from the standard convolution and gradually learn the appropriate scale coefficient from the samples during the training process, and has fewer parameters and reduces the amount of calculation.

[0080] S2, routing space processing feature map, after processing, can select data-dependent routing paths according to the scale distribution of real-time input data to form a network structure;

[0081] Existing deep convolutional neural network methods are limited by fixed-size convolution kernels, resulting in significant discrepancies in the extraction of roads at different scales in images. For example, the classic neural network architectures VGG and FCN offer significant advantages. While VGG networks are highly capable of identifying narrow roads, they struggle with complex backgrounds. Conversely, FCN networks can effectively mitigate the effects of complex backgrounds, but their use of deconvolution for feature recovery can lead to a loss of edge detail on narrow roads, making them unsuitable for extracting small-scale roads.

[0082] The present invention considers combining the advantages of dynamic structure neural networks with the multi-scale nature of road data to design a scale-adaptive dynamic neural network, thereby alleviating the problems caused by multi-scale differences. Figure 1 The middle part is the routing space, which is divided into L layers and consists of several routing units. There are no more than 4 routing units in each layer, and the sampling ratio of adjacent routing units is stipulated to be 2, so the minimum scale of the feature map is 1 / 32 of the original input size. Each routing unit consists of a feature processing module and a switch module. The feature processing module is responsible for fusing and transforming input features of different scales; the switch module selects the routing path to connect to the next layer based on the different scale features of the input. There are 3 options in total, namely 2x upsampling, keeping the resolution unchanged, and 2x downsampling. In order to give full play to the performance of dynamic routing in the routing space, the switch module supports multi-path selection. In addition, cross-connection connection is also supported between routing units.

[0083] DR-Net uses routing space as the super network body, routing space such as Figure 4 As shown in Figure 2, during the inference process, the DR-Net model selects different forward propagation paths based on the scale distribution of different input data and dynamically builds the network structure in the routing space.

[0084] See also Figure 3 The routing space is composed of several routing units, with a total of L layers. The number of routing units in the first three layers is specified to be 1, 2, and 3 respectively, and the number of routing units in the remaining layers does not exceed 4. Each routing unit has three different options for scale transformation, namely upsampling, maintaining resolution, and downsampling. The sampling ratio of adjacent routing units in the same layer is 2. Therefore, there are four different scale features in the routing space, which are 1 / 4, 1 / 8, 1 / 16, and 1 / 32 from large to small. In order to give full play to the structural advantages of the dense connection of the routing space, the routing units of the same level with the same scale support skip connection (Skip Connection) during the forward propagation process; the routing units of the front and back layers support multi-path routing, that is, when the feature scale is transformed, some routing nodes can choose up to 3 scale transformation paths, such as Figure 5 shown.

[0085] See also Figure 4, which is a diagram of multi-path selection and jump connections between routing units. Multi-path selection is shown in the figure as follows:

[0086] During rescaling, Routing Unit A performs a 2x upsampling operation to connect to Routing Unit B, maintains resolution to connect to Routing Unit F, and downsamples by a factor of two to connect to Routing Unit C, selecting three different paths in total. Routing Unit E connects to Routing Units F and G through a 2x upsampling and resolution-maintaining rescaling operation, selecting two different paths in total. Furthermore, Routing Unit A directly connects to Routing Unit F while maintaining resolution, thus achieving a skip connection between Routing Units.

[0087] The routing space consists of several routing units, which are divided into a feature fusion module and a switch module. The switch module is responsible for calculating the probability of selecting each path. The feature processing module is responsible for fusing the different feature maps input from the previous layer and performing feature transformation based on the routing probabilities of different paths calculated by the switch module, outputting them to the routing units in the next layer.

[0088] See also Figure 5 , shows the routing process of a routing unit, where They represent the input features of the l-1th layer in the routing space with sizes of 2s, s, and s / 2 respectively. The input features to be processed by the routing unit corresponding to the layer l with a size of s. They represent the output features of the routing unit with the resolution level s of the lth layer in the routing space to the next layer with the size of 2s, s, and s / 2 respectively.

[0089] Expressed as:

[0090]

[0091] (1) Switch module

[0092] like Figure 6 , the input of the switch module is different input characteristics The switch module is essentially a differentiable lightweight convolution operation module to learn data-adaptive feature vectors. Then, the activation function δ is used to calculate the routing path probability of each scale transformation. The whole process can be expressed as:

[0093]

[0094]

[0095] Among them, the provisions It is the convolution operation function of the feature map Map. is the convolution parameter. G represents average pooling, σ represents ReLU activation, and N represents batch normalization. are all convolution parameters, represents the probability of each routing path, δ represents the activation function, and its value range is [0,1).

[0096] when When , it means that all the paths connecting the routing units with the characteristic scale of s in the first layer of the routing space and the scale of j in the first + 1 layer are retained. When all feature transformation paths of a routing unit are marked as closed, the subsequent calculation work of the feature processing module will be canceled, thereby reducing the calculation cost. In addition, when the calculation cost constraint is given, Before, the input features Perform 1 / 4 downsampling to reduce computational overhead.

[0097] (2) Feature processing module

[0098] like Figure 6 , the input of the feature processing module is also different input features The feature processing module first Perform feature fusion to generate a feature map of scale s Then the activation coefficients of different paths calculated by the switch module are The value size of the feature map is selectively Perform scale transformation, the process is expressed as:

[0099]

[0100]

[0101] Among them, O represents the set of all operations in the feature processing module, including two scale-adaptive convolutions of size 3×3 and bilinear interpolation operations. Represents the feature map Feature transformation from scale s to scale j, j∈{s / 2,s,2s}, This is achieved through a 1×1 convolution operation and a bilinear interpolation operation, which not only converts the feature map size to twice the original size but also halves the number of filters. This is achieved through a convolution operation of size 1×1 and stride 2, which converts the feature map size to 1 / 2 and doubles the number of filters.

[0102] Combining formulas (5), (6), (7), and (8), the overall process of the two modules of the routing unit can be summarized as formulas (9) and (10):

[0103]

[0104]

[0105] S3, the decoder performs decoding operation;

[0106] The decoder module is fixed at the end of the network. First, the 1 / 32 size feature map is upsampled by 2 times and superimposed on the 1 / 16 size feature map. The 1 / 16 size result generated by the superposition is superimposed on the 1 / 8 size feature map by 2 times upsampling. Then the 1 / 8 size result generated by the superposition is superimposed on the 1 / 4 size feature map by 2 times upsampling. Finally, 4 times upsampling is performed to ensure that the output and input size are the same, realizing an end-to-end segmentation process.

[0107] S4. The loss function used in the network is used to correct errors through continuous backpropagation during training, and finally the model parameters are obtained for road extraction.

[0108] (1) Calculate the constraint loss function

[0109] In reality, the computational complexity of network structures is often limited by application scenarios and other factors. With this in mind, DR-Net incorporates computational constraints into the loss function, improving network routing efficiency while still meeting the computational budget.

[0110] The computational operations of each routing unit in DR-Net can be divided into three parts, namely feature fusion, feature transfer and switch module. Calculate costs as follows:

[0111]

[0112] in, Indicates the calculation part where the current routing unit selects different paths to perform feature changes. Represents the computational part of the fusion module that performs feature fusion. Represents the computational part of the switch module that selects the scaling path.

[0113] Combining all routing units in the routing space, the overall computational cost of the routing space C(Space) is:

[0114]

[0115] The loss function that defines the computational constraints is defined as follows:

[0116] Loss Cal =(C(Space) / C0-u 2 ) (13)

[0117] Among them, C0 represents the given computational constraint, and u is the coefficient of the budget constraint, which ranges from 0 to 1. Under the condition that other conditions remain unchanged, the lower the u value, the greater the loss. Cal The higher the value of u, the more the network will selectively prune some routing units and scale transformation paths with low weights according to the u value during the inference process, thereby reducing the loss function Loss by reducing the sum of the calculations of all routing units C(Space). Cal .

[0118] (2) Network loss function

[0119] In addition to solving the "foreground-background imbalance" problem by enhancing the focus of target semantic learning, DR-Net also uses the Dice coefficient combined with the binary cross-entropy loss function as the loss function of the network part, thereby reducing the impact of uneven positive and negative samples during training.

[0120] (a) Dice coefficient loss function

[0121] The Dice coefficient is usually used to express the similarity between two samples. It is a measurement function of set similarity and is expressed as follows:

[0122]

[0123] Here, |X| represents the number of elements in set X, |Y| represents the number of elements in set Y, and |X∩Y represents the number of intersections between sets X and Y.

[0124] In deep learning, the Dice coefficient is often used to address the imbalance of positive and negative samples. In high-resolution remote sensing images, the road area usually only occupies a small part of the original input image. This excessive proportion of negative samples in the input data will lead to insufficient utilization of positive samples during the training process, and the direction of gradient update will deviate from the optimal descent direction. The final model results are not accurate enough. In combination with the semantic segmentation problem, Formula (17) is improved as follows:

[0125]

[0126] Among them, p i Indicates the road probability of the i-th pixel in the prediction result, g i Represents the true value of the i-th pixel in the true value image, N is the total number of pixels in the input image, and the value of smooth is 10 -7 As a smoothing process, it is mainly used to avoid the situation where the denominator is equal to 0 during the calculation process. The value range of D is between 0 and 1.

[0127] In DR-Net, the loss function of the Dice coefficient is defined as follows:

[0128]

[0129] (b) Binary cross entropy loss function

[0130] Road extraction is essentially a binary semantic segmentation problem, so the binary cross-entropy loss function is used in the loss function of the evaluation network instead of the conventional squared error loss function. Cross entropy is calculated using the softmax function, which converts the network's exponential output into a probabilistic expression. Their formulas are as follows:

[0131]

[0132]

[0133] in, represents the index output form of the i-th calculation category, the denominator represents the sum of the index output forms of all categories, k represents the total number of categories, y i is the true value label of the i-th category, p i is the probability of the i-th category, k is the total number of categories, and N is the total number of training samples.

[0134] Road extraction is a binary classification problem, and formula (19) can be simplified to a binary cross entropy loss:

[0135]

[0136]

[0137] When propagating in the last layer, the partial derivatives of the parameters w and b are obtained:

[0138]

[0139]

[0140] The partial derivatives of parameters w and b are proportional to the difference between the expected output and the actual output. That is, during backpropagation, when the difference between the expected and actual output values ​​is large, the parameter change is proportional to the difference, resulting in a larger parameter change and a faster convergence effect.

[0141] Combining formula (16) and formula (19), the loss function of the network part is:

[0142] Loss Net =Loss Dice +Loss Cross (twenty three)

[0143] Combining formula (13) and formula (23), the overall loss function of the joint calculation constraint is:

[0144] Loss = λ1·Loss Net +λ2·Loss Cal (twenty four)

[0145] Among them, λ1 is the loss function coefficient of the network part, which is usually set to 1 in experiments; λ2 is the loss function coefficient of the computational constraint part. According to the requirements of computational cost, the computational constraint is satisfied by setting different proportional coefficients. Unlike the parameter u in formula (13), λ2 is inversely proportional to the computational cost of the network, that is, the larger λ2 is, the lower the computational cost is.

[0146] In another embodiment of the present invention, a road extraction system based on a dynamic routing neural network is provided. The system can be used to implement the above-mentioned road extraction method based on a dynamic routing neural network. Specifically, the road extraction system based on a dynamic routing neural network includes a sampling module, a processing module, a decoding module, and an extraction module.

[0147] Among them, the sampling module downsamples the original input image through the DR-Net network to obtain the feature map;

[0148] The processing module uses the routing space to process the feature maps obtained by the sampling module according to the scale distribution of the real-time input data, and selects data-dependent routing paths to form a scale-adaptive dynamic neural network structure;

[0149] The decoding module decodes the dynamic neural network structure obtained by the processing module through the DR-Net network to obtain the corresponding feature map;

[0150] The extraction module uses the loss function to correct the error of the dynamic routing neural network and upsamples the feature map obtained by the decoding module to achieve end-to-end road extraction.

[0151] In another embodiment of the present invention, a terminal device is provided, which includes a processor and a memory, wherein the memory is used to store a computer program, the computer program includes program instructions, and the processor is used to execute the program instructions stored in the computer storage medium. The processor may be a central processing unit (CPU), or may be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. It is the computing core and control core of the terminal, which is suitable for implementing one or more instructions, specifically suitable for loading and executing one or more instructions to implement corresponding method processes or corresponding functions; the processor described in the embodiment of the present invention can be used for the operation of the road extraction method based on the dynamic routing neural network, including:

[0152] The original input image is downsampled through the DR-Net network to obtain a feature map. According to the scale distribution of real-time input data, the feature map is processed using the routing space, and data-dependent routing paths are selected to form a scale-adaptive dynamic neural network structure. The dynamic neural network structure is decoded through the DR-Net network to obtain the corresponding feature map. The loss function is used to correct the error of the dynamic routing neural network and upsample the feature map to achieve end-to-end road extraction.

[0153] In another embodiment of the present invention, the present invention further provides a storage medium, specifically a computer-readable storage medium (Memory), which is a memory device in a terminal device for storing programs and data. It can be understood that the computer-readable storage medium here can include both the built-in storage medium in the terminal device and, of course, the extended storage medium supported by the terminal device. The computer-readable storage medium provides a storage space, which stores the operating system of the terminal. In addition, one or more instructions suitable for being loaded and executed by the processor are also stored in the storage space. These instructions can be one or more computer programs (including program codes). It should be noted that the computer-readable storage medium here can be a high-speed RAM memory or a non-volatile memory (Non-Volatile Memory), such as at least one disk memory.

[0154] The processor may load and execute one or more instructions stored in a computer-readable storage medium to implement the corresponding steps of the road extraction method based on a dynamic routing neural network in the above embodiment. The processor may load and execute the following steps:

[0155] The original input image is downsampled through the DR-Net network to obtain a feature map. According to the scale distribution of real-time input data, the feature map is processed using the routing space, and data-dependent routing paths are selected to form a scale-adaptive dynamic neural network structure. The dynamic neural network structure is decoded through the DR-Net network to obtain the corresponding feature map. The loss function is used to correct the error of the dynamic routing neural network and upsample the feature map to achieve end-to-end road extraction.

[0156] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings herein can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the claimed invention, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0157] The above steps are used to implement the end-to-end segmentation process of the road. The network structure and concept of the present invention are highly universal and can calculate the intermediate network structure based on real-time input, while saving computing resources. Figure 6 For comparison experiments between other segmentation networks and the present invention, for greater universality, high-resolution images usually contain clouds or even more interference. The images in this experiment contain clouds for interference.

[0158] Figure 6 The results of road extraction by several networks in representative areas of fields, deserts, and cities under cloud cover are shown. It can be seen that DeepLabV3's extraction effect is generally worse than the other three networks, and its anti-interference ability to clouds and fog is the lowest; U-Net and HRNetV2 perform well in segmentation details at some scales, but there are many misjudgments; compared with the other three structures, DR-Net's greatest advantage is that it can reasonably predict road information under the cloud layer based on the known visible road positions and trends, making the overall prediction result more coherent.

[0159]

[0160] The table above summarizes the quantitative evaluation results of each network on the cloud-fog simulation test set. It can be seen that DR-Net outperforms other networks in both IoU and CloudIoU. While DR-Net-A's FLOPs are not the lowest, it is 13.6% and 17.9% higher than the lowest DeepLabV3 network in IoU and CloudIoU, respectively.

[0161] In summary, the road extraction method and system based on dynamic routing neural network of the present invention have better extraction effect in road detection, and have more advantages in high-resolution image extraction effect under conditions such as interference or cloud cover.

[0162] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0163] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant description of other embodiments.

[0164] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed in the present invention can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present invention.

[0165] In the embodiments provided by the present invention, it should be understood that the disclosed devices / terminals and methods can be implemented in other ways. For example, the device / terminal embodiments described above are merely illustrative. For example, the division of the modules or units is merely a logical functional division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be through some interface, indirect coupling or communication connection of devices or units, and can be electrical, mechanical, or other forms.

[0166] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0167] In addition, the functional units in the various embodiments of the present invention may be integrated into a single processing unit, each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0168] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention implements all or part of the process in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by a processor, it can implement the steps of the above-mentioned various method embodiments. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0169] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the steps in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0170] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0171] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 The steps for the function specified in one or more boxes.

[0172] The above content is only for explaining the technical idea of ​​the present invention and cannot be used to limit the protection scope of the present invention. Any changes made on the basis of the technical solution in accordance with the technical idea proposed by the present invention shall fall within the protection scope of the claims of the present invention.

Claims

1. A road extraction method based on dynamic routing neural network, characterized in that: The following steps are involved: S1. Downsample the original input image through the DR-Net network to obtain a feature map. Downsample the original input image by 1 / 4 through two scale-adaptive convolutions to obtain a feature map with a size of 1 / 4 of the original input. The scale regression layer initialization formula of the scale-adaptive convolution is: in, is the initialization convolution kernel of the scale regression layer, is the initialization bias of the scale regression layer, is the position in the convolution kernel, For the sample, is a normal distribution, is the standard deviation; S2. Based on the scale distribution of the real-time input data, the feature map obtained in step S1 is processed using the routing space, and a data-dependent routing path is selected to form a scale-adaptive dynamic neural network structure, specifically: The routing space is divided into L layers, consisting of several routing units. The number of routing units in the first three layers is 1, 2, and 3 respectively, and the number of routing units in each of the remaining layers does not exceed 4. The sampling ratio of adjacent routing units is 2, and the minimum feature map size is 1 / 32 of the original input size. Routing units of the same level with the same scale support skip connections during forward propagation. The routing units of the front and back layers support multi-path routing, that is, when the feature scale is transformed, the routing node selects up to three scale transformation paths; S3, decoding the dynamic neural network structure obtained in step S2 through the DR-Net network to obtain the corresponding feature map; S4. Use the loss function to correct the error of the dynamic routing neural network and upsample the feature map obtained in step S3 to achieve end-to-end road extraction.

2. The road extraction method based on dynamic routing neural network according to claim 1 is characterized in that: In step S2, each routing unit is divided into a feature fusion module and a switch module. The feature processing module is responsible for fusing and transforming input features of different scales. The switch module selects the routing path to connect to the next layer based on the input features of different scales. There are three options: 2x upsampling, maintaining the same resolution, and 2x downsampling. The switch module supports multiple path selection and supports cross-connection connection between routing units.

3. The road extraction method based on dynamic routing neural network according to claim 2 is characterized in that: The multi-path selection is as follows: When performing scale transformation, routing unit A performs 2x upsampling to connect to routing unit B, maintains resolution to connect to routing unit F, and downsamples twice to connect to routing unit C, selecting a total of three different paths. Routing unit E connects to routing units F and G through 2x upsampling and resolution-maintaining scale transformation operations, selecting a total of two different paths. Routing unit A maintains resolution and directly connects to routing unit F, thus realizing jump connections between routing units.

4. The road extraction method based on dynamic routing neural network according to claim 1, characterized in that: In step S3, the decoding operation is specifically as follows: The 1 / 32 size feature map is upsampled by 2 times and superimposed on the 1 / 16 size feature map. The 1 / 16 size result generated by the superposition is upsampled by 2 times and superimposed on the 1 / 8 size feature map. The 1 / 8 size result generated by the superposition is upsampled by 2 times and superimposed on the 1 / 4 size feature map, and finally upsampled by 4 times.

5. The road extraction method based on dynamic routing neural network according to claim 1, characterized in that: In step S4, the loss function used by the dynamic neural network structure is for: in, is the loss function coefficient of the network part, To calculate the loss function coefficient of the constraint part, is the loss function of the network part, is the loss function for computing constraints.

6. The road extraction method based on dynamic routing neural network according to claim 5, characterized in that: Loss function of the network part Including Dice coefficient loss function and the binary cross entropy loss function , as follows: in, is the total number of pixels in the input image, To represent the road probability of the i-th pixel in the prediction result, is the true value of the i-th pixel in the true image, 10-7 is used for smoothing. is the true value label of the category, is the probability that the sample is a positive example; Compute the constraint loss function for: in, is the overall computational cost of the routing space, For the given computational constraints, is the coefficient of the budget constraint.

7. A road extraction system based on dynamic routing neural network, characterized in that: include: The sampling module downsamples the original input image through the DR-Net network to obtain a feature map. The original input image is downsampled by 1 / 4 through two scale-adaptive convolutions to obtain a feature map with a size of 1 / 4 of the original input. The scale regression layer initialization formula of the scale-adaptive convolution is: in, is the initialization convolution kernel of the scale regression layer, is the initialization bias of the scale regression layer, is the position in the convolution kernel, For the sample, is a normal distribution, is the standard deviation; The processing module uses the routing space to process the feature maps obtained by the sampling module according to the scale distribution of the real-time input data, and selects data-dependent routing paths to form a scale-adaptive dynamic neural network structure. Specifically: The routing space is divided into L layers, consisting of several routing units. The number of routing units in the first three layers is 1, 2, and 3 respectively, and the number of routing units in each of the remaining layers does not exceed 4. The sampling ratio of adjacent routing units is 2, and the minimum scale of the feature map is 1 / 32 of the original input size. Routing units of the same level with the same scale support skip connections during the forward propagation process. Routing units in the previous and next layers support multi-path routing, that is, when the feature scale is transformed, the routing node selects at most 3 scale transformation paths. The decoding module decodes the dynamic neural network structure obtained by the processing module through the DR-Net network to obtain the corresponding feature map; The extraction module uses the loss function to correct the error of the dynamic routing neural network and upsamples the feature map obtained by the decoding module to achieve end-to-end road extraction.

8. The road extraction system based on dynamic routing neural network according to claim 7, characterized in that: Each routing unit is divided into a feature fusion module and a switch module. The feature processing module is responsible for fusing and transforming input features of different scales. The switch module selects the routing path to the next layer based on the input features of different scales. There are three options: 2x upsampling, maintaining the same resolution, and 2x downsampling. The switch module supports multiple path selection and supports cross-connection connection between routing units. The multi-path selection is as follows: When performing scale transformation, routing unit A performs 2x upsampling to connect to routing unit B, maintains resolution to connect to routing unit F, and downsamples twice to connect to routing unit C, selecting a total of three different paths. Routing unit E connects to routing units F and G through 2x upsampling and resolution-maintaining scale transformation operations, selecting a total of two different paths. Routing unit A maintains resolution and directly connects to routing unit F, thus realizing jump connections between routing units.

9. The road extraction system based on dynamic routing neural network according to claim 7, characterized in that: The decoding operation is as follows: The 1 / 32 size feature map is upsampled by 2 times and superimposed on the 1 / 16 size feature map. The 1 / 16 size result generated by the superposition is upsampled by 2 times and superimposed on the 1 / 8 size feature map. The 1 / 8 size result generated by the superposition is upsampled by 2 times and superimposed on the 1 / 4 size feature map, and finally upsampled by 4 times.

10. The road extraction system based on dynamic routing neural network according to claim 7, characterized in that: Loss function used by dynamic neural network architecture for: in, is the loss function coefficient of the network part, To calculate the loss function coefficient of the constraint part, is the loss function of the network part, is the loss function for computing constraints; Loss function of the network part Including Dice coefficient loss function and the binary cross entropy loss function , as follows: in, is the total number of pixels in the input image, To represent the road probability of the i-th pixel in the prediction result, is the true value of the i-th pixel in the true image, 10-7 is used for smoothing. is the true value label of the category, is the probability that the sample is a positive example; Compute the constraint loss function for: in, is the overall computational cost of the routing space, For the given computational constraints, is the coefficient of the budget constraint.

Images