Terrain category-based region segmentation method, apparatus and device, and storage medium

Abstract

The application provides a terrain category-based region segmentation method and device, equipment and a storage medium, comprising: extracting feature maps of different scales from an input image through a backbone network; inputting the feature maps of different scales into a passability perception feature modulation module respectively to obtain modulation feature maps modulated based on terrain categories in the feature maps; fusing the modulation feature maps to obtain an output feature map; decoding the output feature map to obtain a segmentation result, the segmentation result comprising marked positions corresponding to different terrain categories. The passability perception feature modulation module is embedded in a key path to extract a passability context vector of the terrain category, so that the network adaptively enhances terrain distinguishing features in forward propagation, thereby identifying corresponding terrains and improving the accuracy of path planning, even without classification of corresponding semantic categories.

Description

Technical Field

[0001] This application belongs to the field of artificial intelligence technology, specifically relating to a method, apparatus, device, and storage medium for region segmentation based on terrain categories. Background Technology

[0002] With the increasing demands for environmental perception capabilities in applications such as autonomous vehicles, outdoor robots, and geological surveys, accurately identifying passable areas in complex terrains (such as flat, rough, bumpy, restricted, and obstacle-filled terrains) has become a research hotspot in the fields of computer vision and robot navigation. Traditional methods based on handcrafted features (such as edge detection and color and texture-based segmentation) are often highly sensitive to changes in lighting, occlusion, and background clutter in real-world outdoor scenarios, making it difficult to obtain robust segmentation results.

[0003] Furthermore, while early semantic segmentation networks achieved good results in medical imaging and indoor scenes, when directly applied to outdoor terrain, they mostly focused on segmenting general semantic categories (such as "roads," "buildings," and "vegetation"). For more detailed semantic categories, such as the segmentation of small obstacles or narrow passages, there was a lack of specialized modeling, resulting in insufficient representation capabilities of the model for certain scenes and affecting the accuracy of path planning. Summary of the Invention

[0004] This application proposes a region segmentation method, apparatus, device, and storage medium based on terrain categories, which can solve the technical problem that the lack of specialized modeling for certain semantic categories leads to insufficient model representation ability for certain scenarios, affecting the accuracy of path planning.

[0005] The first aspect of this application proposes a region segmentation method based on terrain categories, including:

[0006] The backbone network extracts feature maps of different scales from the input image.

[0007] The feature maps at different scales are input into the accessibility-aware feature modulation module to obtain modulated feature maps based on the terrain category in each feature map.

[0008] The multiple modulation feature maps are fused to obtain an output feature map;

[0009] The output feature map is decoded to obtain the segmentation result, which includes the marked locations corresponding to different terrain categories.

[0010] An embodiment of the second aspect of this application provides a region segmentation device based on terrain category, comprising:

[0011] The extraction module is used to extract feature maps of different scales from the input image through the backbone network;

[0012] The modulation module is used to input the feature maps of different scales into the accessibility-aware feature modulation module to obtain a modulated feature map based on the terrain category in each feature map.

[0013] The fusion module is used to fuse multiple modulation feature maps to obtain an output feature map;

[0014] The decoding module is used to decode the output feature map to obtain the segmentation result, which includes the marker positions corresponding to different terrain categories.

[0015] An embodiment of the third aspect of this application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method described in the first aspect above.

[0016] An embodiment of the fourth aspect of this application provides a computer-readable storage medium having a computer program stored thereon, the program being executed by a processor to implement the method described in the first aspect above.

[0017] The technical solutions provided in this application embodiment have at least the following technical effects or advantages:

[0018] This application proposes a method, apparatus, device, and storage medium for region segmentation based on terrain categories, comprising: extracting feature maps of different scales from an input image through a backbone network; inputting the feature maps of different scales into a accessibility-aware feature modulation module to obtain modulated feature maps modulated based on the terrain categories in each feature map; fusing multiple modulated feature maps to obtain an output feature map; and decoding the output feature map to obtain a segmentation result, the segmentation result including the labeled positions corresponding to different terrain categories. The embodiments of this application embed a accessibility-aware feature modulation module into the critical path to extract the accessibility context vector of the terrain category, enabling the network to adaptively enhance terrain discriminative features during forward propagation. This allows for the identification of corresponding terrains even without corresponding semantic category classifications, thereby improving the accuracy of path planning.

[0019] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0020] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the scope of this application. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings:

[0021] Figure 1 A flowchart of a region segmentation method based on terrain category provided in an embodiment of this application is shown;

[0022] Figure 2 This illustration shows a schematic diagram of the structure of a mobility sensing feature modulation module according to an embodiment of this application;

[0023] Figure 3 This invention provides a schematic diagram of the structure of an ASPP module according to an embodiment of the present application.

[0024] Figure 4 This illustration shows a schematic diagram of the structure of a accessibility sensing network provided in an embodiment of this application;

[0025] Figure 5 This illustration shows a schematic diagram of a terrain-based region segmentation device according to an embodiment of this application;

[0026] Figure 6 This illustration shows a schematic diagram of the structure of an electronic device according to an embodiment of this application;

[0027] Figure 7 A schematic diagram of a storage medium provided in one embodiment of this application is shown. Detailed Implementation

[0028] Exemplary embodiments of this application will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of this application are shown in the drawings, it should be understood that this application may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided to enable a more thorough understanding of this application and to fully convey the scope of this application to those skilled in the art.

[0029] It should be noted that, unless otherwise stated, the technical or scientific terms used in this application shall have the ordinary meaning as understood by one of ordinary skill in the art to which this application pertains.

[0030] The terrain-based region segmentation method of this application can be executed by a computing device that applies cloud computing and virtualization technologies. The computing device can be a server, such as a single server, multiple servers, a server cluster, a cloud computing platform, etc. Optionally, the computing device can also be a terminal device, such as a mobile phone, tablet computer, game console, portable computer, desktop computer, advertising machine, all-in-one machine, etc. This application does not limit the type or number of computing devices.

[0031] To address the aforementioned issues, this application proposes a terrain category-based region segmentation method, apparatus, device, and storage medium, comprising: extracting feature maps of different scales from an input image through a backbone network; inputting the feature maps of different scales into a accessibility-aware feature modulation module to obtain modulated feature maps modulated based on the terrain categories in each feature map; fusing multiple modulated feature maps to obtain an output feature map; and decoding the output feature map to obtain a segmentation result, the segmentation result including the labeled positions corresponding to different terrain categories. This application's embodiment embeds a accessibility-aware feature modulation module into the critical path to extract the accessibility context vector of the terrain category, enabling the network to adaptively enhance terrain discriminative features during forward propagation. Therefore, even without corresponding semantic category classification, the network can rely on terrain discriminative features to identify the corresponding terrain, improving the accuracy of path planning.

[0032] The following describes a region segmentation method based on terrain category according to an embodiment of this application, with reference to the accompanying drawings.

[0033] See Figure 1 The method specifically includes the following steps:

[0034] S101. Extract feature maps of different scales from the input image through the backbone network.

[0035] The scale can be represented by resolution. Generally, the larger the scale, the larger the resolution, and the higher the number of feature layers. Features with larger scales include more image content, but fewer image details. Therefore, the higher the number of feature layers, the smaller the resolution, and the more image details are included.

[0036] The backbone network is the core part of a deep learning model used to extract features from images or data. Common examples include ResNet, VGG, and MobileNet.

[0037] Backbone networks employ deep residual learning or lightweight structure optimization, enabling them to effectively extract both low-level and high-level features from images. The backbone encoder's role is to extract feature maps at different scales from the input image, where the resolution of each feature map layer gradually decreases while the number of channels gradually increases. Each feature map represents a different scale, ranging from details to high-level semantic information.

[0038] In some embodiments, to reduce noise interference and improve the efficiency of region segmentation, low-level feature maps, i.e., large-scale feature maps, are typically removed. Assume the ResNet network outputs five feature maps at different scales (sampling stride of 2): C1 (input image 1 / 2 resolution), C2 (input image 1 / 4 resolution), C3 (input image 1 / 8 resolution), C4 (input image 1 / 16 resolution), and C5 (input image 1 / 32 resolution), with C5 being the highest-level feature map. Since C1 and C2 primarily capture low-level features such as edges, color, and texture, they have almost no discriminative power for high-level semantics like "terrain category" and are more prone to incorporating noise. Therefore, the low-level feature maps C1 and C2 are ultimately removed, and the remaining high-level feature maps are input into the accessibility-aware feature modulation module for subsequent processing.

[0039] S102. Input the feature maps of different scales into the accessibility-aware feature modulation module to obtain the modulated feature map based on the terrain category in each feature map.

[0040] Understandably, most image segmentation technologies in the navigation field currently use semantic segmentation networks to segment the semantic information of each part of an image, such as "roads," "buildings," and "vegetation," and then send the segmentation results to a decision model so that the decision model can determine the navigation route based on the segmentation results.

[0041] However, for more detailed terrain classification, such as the segmentation of small obstacles or narrow passages, there is a lack of specialized modeling. Semantic segmentation networks cannot segment small obstacles or narrow passages, resulting in the segmentation results possibly lacking segmentation information of small obstacles or narrow passages in the image, leading to inaccurate navigation routes determined by the decision model based on the segmentation results.

[0042] Based on this, in this embodiment of the application, feature maps of different scales are input into the accessibility perception feature modulation module to obtain modulated feature maps based on the terrain category in each feature map.

[0043] In the perception-decision link of autonomous driving or off-road robots, the accessibility perception feature modulation module is a pluggable sub-network that reweights "accessibility" according to terrain category. It first infers the terrain category of the current grid / graph node through explicit or implicit means, and outputs a terrain-conditional accessibility tensor for subsequent trajectory planning.

[0044] The accessibility-aware feature modulation module can determine the accessibility corresponding to different terrain categories of each channel based on feature maps at different scales. Then, based on the accessibility corresponding to different terrain categories of each channel, the feature maps at different scales are modulated to obtain modulated feature maps for subsequent trajectory planning.

[0045] The terrain categories can include flat, rough, bumpy, restricted, obstacle, background, etc.

[0046] S103. Fuse multiple modulation feature maps to obtain an output feature map.

[0047] It is understandable that fusing multiple modulation feature maps, i.e. fusing feature maps of different scales, can fuse different scales of information from details to high-level semantics in an image, so as to make the output result more expressive.

[0048] The fusion method can be to directly fuse the modulation feature maps, or to input each modulation feature map into the convolutional normalization activation module and then fuse them.

[0049] In some embodiments, in order to capture multi-scale contextual information within a large receptive field, the modulation feature map corresponding to the highest layer feature map can be input into the dilated spatial pyramid pooling module for convolution with different dilation rates, and the resulting dilated modulation feature map can be fused with the remaining modulation feature maps to obtain the output feature map.

[0050] S104. Decode the output feature map to obtain the segmentation result.

[0051] The segmentation results include the marked locations corresponding to different terrain categories.

[0052] Understandably, subsequent trajectory planning is more convenient, as the output feature map can be labeled, that is, the output feature map can be decoded based on the terrain category to obtain segmentation results including the labeled positions corresponding to different terrain categories.

[0053] This application proposes a method, apparatus, device, and storage medium for region segmentation based on terrain categories, comprising: extracting feature maps of different scales from an input image through a backbone network; inputting the feature maps of different scales into a accessibility-aware feature modulation module to obtain modulated feature maps modulated based on the terrain categories in each feature map; fusing multiple modulated feature maps to obtain an output feature map; and decoding the output feature map to obtain a segmentation result, the segmentation result including the labeled positions corresponding to different terrain categories. The embodiments of this application embed a accessibility-aware feature modulation module into the critical path to extract the accessibility context vector of the terrain category, enabling the network to adaptively enhance terrain discriminative features during forward propagation. This allows for the identification of corresponding terrains even without corresponding semantic category classifications, thereby improving the accuracy of path planning.

[0054] In some embodiments, feature maps of different scales are input into the accessibility-aware feature modulation module to obtain modulated feature maps based on terrain category information in each feature map. This includes: performing global average pooling on the first feature map to obtain a global context information vector, where the first feature map is any feature map at different scales; inputting the global context information vector into a two-layer fully connected network to obtain a accessibility context vector, where the accessibility context vector includes accessibility corresponding to each terrain category; determining the attention weights corresponding to each channel based on the accessibility context vector; and modulating the first feature map based on the attention weights corresponding to each channel to obtain a modulated feature map.

[0055] Among them, the global context information vector is a vector that compresses all spatial location information of the entire feature map into a fixed length vector, and the traversability context vector is the probability that the entire scene is judged by the network to belong to a certain terrain category.

[0056] For example, if the terrain categories are flat, rough, bumpy, restricted, obstacle, and background, the drivability context vector z = (0.05, 0.10, 0.05, 0.75, 0.03, 0.02) indicates that the probability of this scene being flat terrain is 5%, rough terrain is 10%, bumpy terrain is 5%, restricted terrain is 75%, obstacle terrain is 3%, and background terrain is 2%. Based on this drivability context vector, the current scene is judged to be "mainly a large restricted area", with other terrains accounting for only a small proportion, and the overall drivability is extremely low.

[0057] In some embodiments, the structure of the accessibility-aware feature modulation module is as follows: Figure 2 As shown in the embodiments of this application, the accessibility-aware feature modulation module is described in terms of the processing order of the feature maps.

[0058] like Figure 2 As shown, the accessibility-aware feature modulation module includes a global average pooling layer, a first fully connected layer, a first activation layer, a second fully connected layer, a second activation layer, a terrain-conditional weight generation layer, and a normalization layer.

[0059] First, the first feature map is input into the global average pooling layer for global average pooling processing to obtain the global context information vector. The specific process is shown in equation (1):

[0060] (1)

[0061] in, This is a global context information vector. H represents the scale of the first feature map, where H is the height and W is the width. This is the first feature map. This represents the values ​​of all channels of the feature map at position (i,j).

[0062] The global context information vector is input into a two-layer fully connected network to obtain a mobility context vector, which includes the mobility corresponding to each terrain category.

[0063] In some embodiments, a global context information vector is input into a two-layer fully connected network to obtain a traversability context vector, including:

[0064] (2)

[0065] in, This is the accessibility context vector. This is a global context information vector. The weights are the weights of the first fully connected layer. For the number of channels, The bottleneck dimension is obtained by compressing the number of channels based on a preset compression factor. For the weights of the second fully connected layer, To preset the number of terrain categories, For the first layer bias, For the second layer bias, For the first layer activation function, This is the activation function for the second layer.

[0066] The first activation function can be the ReLU activation function, and the second activation function can be the Sigmoid activation function or the Softmax activation function.

[0067] Specifically, the global context information vector is input into the first fully connected layer, "compressing" the C original channels to... There are several bottleneck channels, and each bottleneck channel is given a learnable bias starting point. It can be the ratio of the number of channels to the number of preset terrain categories, or other values, which are not limited here.

[0068] By reducing the number of parameters from C·K to C· + ·K, significantly reducing computation.

[0069] Furthermore, the output of the first fully connected layer is input into the first activation layer, and negative values ​​are filtered out by the ReLU activation function. Nonlinearity is introduced, which retains and strengthens the most useful signal for accessibility discrimination while reducing dimensionality and suppressing redundant noise.

[0070] Among these methods, filtering out negative values ​​forces the network to extract the most critical features for terrain discrimination in a smaller space, and introducing nonlinearity allows the feature maps of bottleneck channels to have stronger expressive power, avoiding the information loss caused by pure linear compression.

[0071] Secondly, the compact features output from the first activation layer are input into the second fully connected layer, which then... The intermediate features are weighted and combined into K raw scores, giving each terrain type a learnable baseline score. The K raw scores are then independently compressed into confidence levels of 0–1 using the Sigmoid function, or the K raw scores are transformed into probability distributions, with the largest probability distribution summing to 1.

[0072] Furthermore, the accessibility context vector is input into the terrain-conditional weight generation layer to obtain the attention weights corresponding to each channel.

[0073] In some embodiments, the attention weights for each channel are determined based on the accessibility context vector, including:

[0074] (3)

[0075] in, The attention weights corresponding to the channels. This is the weight matrix. This is the normalization function.

[0076] The weight matrix is ​​obtained by automatically updating the magnification factor for each terrain category in each channel through backpropagation.

[0077] To make it easier to understand, let's take an example to illustrate the process of determining the attention weight for each channel based on the accessibility context vector.

[0078] Assuming there are 6 terrain categories K, namely flat, rough, bumpy, restricted, obstacle, and background, 4 feature channels C, 1 feature map space H, 1 W, and a corresponding drivability context vector z = (0.05, 0.10, 0.05, 0.75, 0.03, 0.02), this indicates that the probability of the scene belonging to flat terrain is 5%, rough terrain is 10%, bumpy terrain is 5%, restricted terrain is 75%, obstacle terrain is 3%, and background terrain is 2%. Based on this drivability context vector, the current scene is judged to be "mainly a large restricted area", with other terrains accounting for only a small proportion, resulting in extremely low overall drivability.

[0079] The weight matrix is ​​shown in Table 1:

[0080]

[0081] Through formula (3) The original channel attention weight vector can be obtained as (0.1700, 0.1275, 0.1060, 0.0945).

[0082] Furthermore, to ensure fair comparison and collaborative updates of values ​​at different scales and magnitudes under the same "ruler," and to guarantee stable network training and rapid convergence, the original channel attention weight vectors were modified... Normalization yields the normalized channel attention weight vector (0.2952, 0.2534, 0.2341, 0.2173), indicating that channel 0 has the highest weight: the network considers it most important for "current terrain discrimination," and it will be retained and amplified by 29.25%. Channel 1 has an importance of approximately 25.34%, channel 2 has an importance of approximately 23.41%, and channel 3 has the lowest weight, retaining only 21.73% of the information.

[0083] In some embodiments, the first feature map is modulated according to the attention weights corresponding to each channel to obtain a modulated feature map, including:

[0084] (4)

[0085] in, For modulation feature maps, This is the first feature map. This represents a channel-wise element-wise multiplication operation.

[0086] Since the attention weights corresponding to each channel are the "terrain-channel mapping" results obtained by linearly combining the terrain probability traversability context vector with the weight matrix and then normalizing them, the amplification / suppression ratio of each channel has been encoded into the terrain category information when multiplying channel by channel, thus realizing the modulation of the feature map with terrain semantics.

[0087] After multiplying the first feature map element-wise with the attention weights corresponding to each channel, each pixel and each channel of the first feature map can be rescaled according to "terrain preference" to obtain a modulated feature map that modulates the first feature map based on the terrain category.

[0088] Furthermore, the feature maps at different scales are input into the corresponding accessibility-aware feature modulation modules to obtain the corresponding modulation feature maps.

[0089] In some embodiments, multiple modulation feature maps are fused to obtain an output feature map, including: inputting the modulation feature map corresponding to the smallest scale into an expanded spatial pyramid pooling module to obtain an expanded modulation feature map after expansion at different expansion rates and fusion; inputting the remaining modulation feature maps into a convolutional normalization activation module to obtain corresponding activated modulation feature maps; and performing progressive upsampling fusion processing on the expanded modulation feature map and multiple activated modulation feature maps to obtain the output feature map.

[0090] Understandably, multiple modulated feature maps simply write "terrain semantics" into the channel weights without expanding the pixel-level receptive field, which can easily lead to breaks and misjudgments during decoding. The scales of the multiple modulated feature maps are different and the feature distribution after modulation may be offset, which can lead to unstable gradients and slow convergence when directly decoded.

[0091] To expand the pixel-level receptive field, the modulation feature map corresponding to the smallest scale can be input into the expanded spatial pyramid pooling module to obtain the expanded modulation feature map after expansion at different expansion rates and fusion.

[0092] like Figure 3 As shown, the Atrous Spatial Pyramid Pooling (ASPP) module includes two 1×1 convolutional modules, three 3×3 convolutional modules with different dilation rates, and a global pooling module. The 1×1 convolutional modules, whose input is the modulated feature map, are used to maintain the receptive field of 1×1, quickly reduce / increase dimensionality, and capture local details. The 3×3 convolutional modules with different dilation rates are used to capture contextual information at different distances, with dilation rates of 6, 12, and 18, respectively. The global pooling module is used to obtain the global semantics of the entire image. Thus, the output feature maps of the above five modules are interpolated back to the original resolution through the 1×1 convolutional modules.

[0093] The ASPP module can be used to obtain local texture and distant semantics at different distances from the modulation feature map, thereby expanding the receptive field.

[0094] It should be noted that if modulation feature maps of all scales are fed into ASPP, the number of parameters and memory increases quadratically with the resolution. The highest-scale modulation feature map retains the most information about edges, textures, and small obstacles, which are key to determining whether something is "passable" or "impassable." By feeding it into ASPP's multi-scale dilated convolution, the receptive field can be expanded again without sacrificing detail, thus completing the semantic context.

[0095] To address the potential shift in the modulated feature distribution, the remaining modulated feature maps can be input into the convolutional normalization activation module to obtain the corresponding activated modulated feature maps. The convolutional normalization activation module includes a 1×1 convolutional layer, a normalization layer, and a nonlinear layer, which can pull the modulated feature maps back to zero mean / unit variance and inject nonlinearity to alleviate internal covariate drift, making it easier for the decoder to converge.

[0096] Furthermore, the expanded modulation feature map and multiple activation modulation feature maps need to be adjusted to the same size for fusion.

[0097] The progressive upsampling fusion process starts with low-resolution high-level features, gradually amplifies them in stages, and gradually merges them with high-resolution low-level features, rather than upsampling directly to the final size all at once.

[0098] Furthermore, since the expanded modulation feature map is the smallest scale and corresponds to the lowest resolution, an upsampling operation can be used to make the resolution of the expanded modulation feature map consistent with the resolution of the lowest resolution active modulation feature map among the multiple active modulation feature maps. The expanded modulation feature map is then fused with the active modulation feature map. The fused feature map is then further upsampled, and the resulting feature map is fused with the corresponding active modulation feature map again. The expanded modulation feature map and multiple active modulation feature maps are fused step by step in the above manner to obtain the output feature map.

[0099] In some embodiments, in order to reduce the computational load and GPU memory usage by more than half while preserving spatial details, thereby reducing the burden on subsequent multi-scale fusion, a dimensionality reduction operation can be performed on multiple activation modulation feature maps before fusion. This involves reducing the number of channels in multiple activation modulation feature maps to a preset number, which can be flexibly set based on the actual situation.

[0100] In some embodiments, decoding the output feature map to obtain a segmentation result includes: inputting the output feature map into a preset convolutional module to obtain a compressed feature map with the same number of channels as the preset number of terrain categories; obtaining the confidence score of different terrain categories in each pixel of the compressed feature map; normalizing the confidence score of different terrain categories in each pixel of the compressed feature map to obtain a prediction probability map; and performing a category indexing operation on each pixel in the prediction probability map to obtain a segmentation result including a terrain category index map corresponding to the prediction probability map, wherein the terrain category index map includes the marker positions corresponding to different terrain categories.

[0101] The preset convolution module is used to compress the output feature map. Since the output feature map is modulated based on the terrain category, the number of channels can be compressed to the preset number of terrain categories, so that each channel can carry the confidence score of one type of terrain.

[0102] Since each pixel includes the confidence scores of all channels, compressing the number of channels to the preset number of terrain categories yields the category confidence score for each pixel.

[0103] Furthermore, to reduce computational complexity, the class confidence score corresponding to each pixel can be normalized to obtain the class distribution probability corresponding to each pixel.

[0104] In this process, normalization functions such as Softmax can be used to normalize the confidence scores of different terrain categories for each pixel in the compressed feature map, thereby obtaining a predicted probability map.

[0105] The category index operation can be performed on each pixel in the predicted probability map, that is, the category number with the highest probability is selected for each pixel to obtain the category index corresponding to each pixel, and thus the terrain category index map is obtained.

[0106] To make it easier to understand, let's take an example to illustrate the process of decoding the output feature map to obtain the segmentation result.

[0107] Assuming there are six terrain categories: flat, rough, bumpy, no-entry, obstacle, and background, the output feature map is input into a preset convolutional module to obtain a compressed feature map with 6 channels, where each pixel corresponds to the confidence level of the six terrain categories.

[0108] The compressed feature map is activated by a preset activation function, and the confidence scores of each pixel corresponding to the six terrain categories are converted into probability distributions along the channel dimension.

[0109] Perform a category indexing operation on each pixel in the predicted probability map to obtain a segmentation result including the terrain category index map corresponding to the predicted probability map. That is, for each pixel, obtain the terrain category with the highest probability in the probability distribution corresponding to the terrain category, and determine the pixel corresponding to the terrain category. Further determine the terrain category corresponding to each pixel in the predicted probability map, thereby obtaining the terrain category index map corresponding to the predicted probability map.

[0110] Furthermore, this terrain category index map can be used as a segmentation result to determine navigation routes.

[0111] It is understood that the above modules constitute a accessibility sensing network. In order to fully describe the above-mentioned terrain category-based region segmentation, this application provides a schematic diagram of the accessibility sensing network structure, and the above-mentioned terrain category-based region segmentation is explained in conjunction with the schematic diagram.

[0112] like Figure 5 As shown, the accessibility-aware network includes a backbone network, multiple accessibility-aware feature modulation modules, an ASPP module, a convolutional normalization activation module, and a decoding module.

[0113] First, the input image is fed into the backbone network to obtain five feature maps at different scales (sampling stride of 2), namely C1 (1 / 2 resolution of input image), C2 (1 / 4 resolution of input image), C3 (1 / 8 resolution of input image), C4 (1 / 16 resolution of input image), and C5 (1 / 32 resolution of input image).

[0114] Since C1 and C2 primarily capture low-level features such as edges, colors, and textures, they have almost no discriminative power for high-level semantics like "terrain category" and are prone to introducing noise. Therefore, the low-level feature maps of C1 and C2 are ultimately removed, and the remaining high-level feature maps are input into the accessibility-aware feature modulation module for further processing.

[0115] Input C3, C4 and C5 into the accessibility-aware feature modulation module to obtain modulated feature maps based on the terrain category in each feature map.

[0116] Accessibility sensing feature modulation module, such as Figure 2 As shown, the modulation process is the same as described above, and will not be repeated here.

[0117] To expand the pixel-level receptive field, the modulation feature map corresponding to the highest-level feature map is input into the ASPP module, that is, C5 is input into the ASPP module to obtain the corresponding expanded modulation feature map. The structure of the ASPP module is as follows: Figure 3 As shown, the execution process of the ASPP module is the same as described above, and will not be repeated here.

[0118] To address the potential shift in the feature distribution after modulation, the remaining modulation feature maps can be input into the convolutional normalization activation module to obtain the corresponding activated modulation feature maps.

[0119] Since it is necessary to adjust the expanded modulation feature map and multiple active modulation feature maps to the same size for fusion, a progressive upsampling fusion process is performed on the expanded modulation feature map and multiple active modulation feature maps to obtain the output feature map.

[0120] The decoding module includes a pre-defined convolution module, a normalization module, and a class indexing module.

[0121] The output feature map is input into the decoding module. First, the preset convolution module compresses the output feature map. Since the output feature map is modulated based on the terrain category, the number of channels can be compressed to the preset number of terrain categories. This allows each channel to carry the confidence score of one type of terrain.

[0122] Furthermore, to reduce computational complexity, the compressed output features are input into the normalization module, which normalizes the class confidence score corresponding to each pixel to obtain the class distribution probability corresponding to each pixel.

[0123] Furthermore, the category indexing module can be used to perform a category indexing operation on each pixel in the predicted probability map, that is, to select the category number with the highest probability for each pixel, to obtain the category index corresponding to each pixel, and thus obtain the terrain category index map.

[0124] Furthermore, this terrain category index map can be used as a segmentation result to determine navigation routes.

[0125] It should be noted that the above-mentioned accessibility-aware network can be trained using a loss function, where the loss function can be a weighted cross-entropy loss, as shown in equation (5):

[0126]

[0127] in, The loss is a weighted cross-entropy loss, where N is the number of pixels. It is the category weight. Let i be the true label and predicted probability of pixel i in category k.

[0128] This application also provides a region segmentation apparatus based on terrain categories, which is used to execute the region segmentation method based on terrain categories provided in any of the above embodiments. Figure 5 As shown, the device includes an extraction unit 501, a modulation unit 502, a fusion unit 503, and a decoding unit 504.

[0129] Extraction unit 501 is used to extract feature maps of different scales from the input image through the backbone network;

[0130] Modulation unit 502 is used to input the feature maps of different scales into the accessibility perception feature modulation unit to obtain a modulated feature map based on the terrain category in each feature map.

[0131] Fusion unit 503 is used to fuse multiple modulation feature maps to obtain an output feature map;

[0132] The decoding unit 504 is used to decode the output feature map to obtain the segmentation result, which includes the marker positions corresponding to different terrain categories.

[0133] This application proposes a method, apparatus, device, and storage medium for region segmentation based on terrain categories, comprising: extracting feature maps of different scales from an input image through a backbone network; inputting the feature maps of different scales into a accessibility-aware feature modulation unit to obtain modulated feature maps modulated based on the terrain categories in each feature map; fusing multiple modulated feature maps to obtain an output feature map; and decoding the output feature map to obtain a segmentation result, the segmentation result including the labeled positions corresponding to different terrain categories. The embodiments of this application embed a accessibility-aware feature modulation module into the critical path to extract the accessibility context vector of the terrain category, enabling the network to adaptively enhance terrain discriminative features during forward propagation. This allows for the identification of corresponding terrains even without corresponding semantic category classifications, thereby improving the accuracy of path planning.

[0134] In some embodiments, the modulation unit 502 is specifically used for:

[0135] For the first feature map, global average pooling is performed on the first feature map to obtain a global context information vector. The first feature map is any feature map among the feature maps of different scales.

[0136] The global context information vector is input into a two-layer fully connected network to obtain a traversability context vector, which includes the traversability corresponding to each terrain category.

[0137] Based on the accessibility context vector, determine the attention weight corresponding to each channel;

[0138] Based on the attention weights corresponding to each channel, the first feature map is modulated to obtain a modulated feature map.

[0139] In some embodiments, the modulation unit 502 is further specifically used for:

[0140]

[0141] in, This is the accessibility context vector. The global context information vector, The weights are the weights of the first fully connected layer. For the number of channels, The bottleneck dimension is obtained by compressing the number of channels based on a preset compression coefficient. For the weights of the second fully connected layer, To preset the number of terrain categories, For the first layer bias, For the second layer bias, For the first layer activation function, This is the activation function for the second layer.

[0142] In some embodiments, the modulation unit 502 is further specifically used for:

[0143]

[0144] in, The attention weights corresponding to the channels. This is the weight matrix. This is the normalization function.

[0145] In some embodiments, the modulation unit 502 is further specifically used for:

[0146]

[0147] in, For modulation feature maps, This is the first feature map. This represents a channel-wise element-wise multiplication operation.

[0148] In some embodiments, the fusion unit 503 is specifically used for:

[0149] The modulation feature map corresponding to the smallest scale is input into the extended spatial pyramid pooling unit to obtain the extended modulation feature map after expansion at different expansion rates and fusion.

[0150] The remaining modulation feature maps are input into the convolutional normalized activation unit to obtain the corresponding activation modulation feature maps;

[0151] The expanded modulation feature map and the multiple activated modulation feature maps are subjected to progressive upsampling fusion processing to obtain an output feature map.

[0152] In some embodiments, the decoding unit 504 is specifically used for:

[0153] The output feature map is input into a preset convolutional unit to obtain a compressed feature map with the same number of channels as the preset number of terrain categories.

[0154] Obtain the confidence score of different terrain categories for each pixel in the compressed feature map;

[0155] The confidence scores of different terrain categories for each pixel in the compressed feature map are normalized to obtain a prediction probability map.

[0156] Perform a category indexing operation on each pixel in the predicted probability map to obtain a segmentation result including a terrain category index map corresponding to the predicted probability map, wherein the terrain category index map includes the marker positions corresponding to different terrain categories.

[0157] This application also provides an electronic device for performing the above-described terrain-based region segmentation method. Please refer to... Figure 6 It illustrates a schematic diagram of an electronic device provided by some embodiments of this application. For example... Figure 6 As shown, the electronic device 7 includes: a processor 700, a memory 701, a bus 702, and a communication interface 703. The processor 700, the communication interface 703, and the memory 701 are connected via the bus 702. The memory 701 stores a computer program that can run on the processor 700. When the processor 700 runs the computer program, it executes the terrain category-based region segmentation method provided in any of the foregoing embodiments of this application.

[0158] The memory 701 may include high-speed random access memory (RAM) or non-volatile memory, such as at least one disk storage device. Communication between the device network element and at least one other network element is achieved through at least one communication interface 703 (which can be wired or wireless), such as the Internet, wide area network, local area network, metropolitan area network, etc.

[0159] Bus 702 can be an ISA bus, PCI bus, or EISA bus, etc. Buses can be divided into address buses, data buses, control buses, etc. Memory 701 is used to store programs. After receiving execution instructions, processor 700 executes the programs. The terrain-based region segmentation method disclosed in any of the aforementioned embodiments of this application can be applied to processor 700, or implemented by processor 700.

[0160] The processor 700 may be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above method can be completed by the integrated logic circuitry in the hardware of the processor 700 or by instructions in software form. The processor 700 may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), an off-the-shelf programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor may be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software units in the decoding processor. The software units may reside in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located in memory 701. Processor 700 reads the information in memory 701 and, in conjunction with its hardware, completes the steps of the above method.

[0161] The electronic device provided in this application embodiment and the terrain category-based region segmentation method provided in this application embodiment are based on the same inventive concept and have the same beneficial effects as the methods they adopt, operate or implement.

[0162] This application also provides a computer-readable storage medium corresponding to the terrain category-based region segmentation method provided in the foregoing embodiments. Please refer to... Figure 7 The computer-readable storage medium shown is an optical disc 30, on which a computer program (i.e., a program product) is stored. When the computer program is run by a processor, it executes the terrain-based region segmentation method provided in any of the aforementioned embodiments.

[0163] It should be noted that examples of computer-readable storage media may also include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other optical and magnetic storage media, which will not be elaborated here.

[0164] The computer-readable storage medium provided in the above embodiments of this application and the terrain category-based region segmentation method provided in the embodiments of this application are based on the same inventive concept and have the same beneficial effects as the methods adopted, run or implemented by the applications stored therein.

[0165] It should be noted that:

[0166] Numerous specific details are set forth in the specification provided herein. However, it will be understood that embodiments of this application may be practiced without these specific details. In some instances, well-known structures and techniques have not been shown in detail so as not to obscure the understanding of this specification.

[0167] Furthermore, those skilled in the art will understand that although some embodiments herein include certain features included in other embodiments but not others, combinations of features from different embodiments are intended to be within the scope of this application and form different embodiments. For example, in the following claims, any of the claimed embodiments can be used in any combination.

[0168] The above are merely preferred embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A region segmentation method based on terrain categories, characterized in that, include: The backbone network extracts feature maps of different scales from the input image. The feature maps at different scales are input into the accessibility-aware feature modulation module to obtain modulated feature maps based on the terrain category in each feature map. The multiple modulation feature maps are fused to obtain an output feature map; The output feature map is decoded to obtain the segmentation result, which includes the marked locations corresponding to different terrain categories; The step of inputting the feature maps of different scales into the accessibility-aware feature modulation module to obtain modulated feature maps based on the terrain category in each feature map includes: For the first feature map, global average pooling is performed on the first feature map to obtain a global context information vector. The first feature map is any feature map among the feature maps of different scales. The global context information vector is input into a two-layer fully connected network to obtain a traversability context vector, which includes the traversability corresponding to each terrain category. Based on the accessibility context vector, determine the attention weight corresponding to each channel; Based on the attention weights corresponding to each channel, the first feature map is modulated to obtain a modulated feature map. The step of determining the attention weight corresponding to each channel based on the accessibility context vector includes: in, The attention weights corresponding to the channels. This is the weight matrix. For normalization function, This is the accessibility context vector. For the number of channels, The number of preset terrain categories.

2. The method according to claim 1, characterized in that, The step of inputting the global context information vector into a two-layer fully connected network to obtain a traversability context vector includes: in, The global context information vector, The weights are the weights of the first fully connected layer. The bottleneck dimension is obtained by compressing the number of channels based on a preset compression coefficient. For the weights of the second fully connected layer, For the first layer bias, For the second layer bias, For the first layer activation function, This is the activation function for the second layer.

3. The method according to claim 1, characterized in that, The step of modulating the first feature map according to the attention weights corresponding to each channel to obtain a modulated feature map includes: in, For modulation feature maps, This is the first feature map. This represents a channel-wise element-wise multiplication operation.

4. The method according to claim 1, characterized in that, The process of fusing multiple modulation feature maps to obtain an output feature map includes: The modulation feature map corresponding to the smallest scale is input into the extended spatial pyramid pooling module to obtain the extended modulation feature map after expansion at different expansion rates and fusion. The remaining modulation feature maps are input into the convolutional normalization activation module to obtain the corresponding activation modulation feature maps; The expanded modulation feature map and the multiple activated modulation feature maps are subjected to progressive upsampling fusion processing to obtain an output feature map.

5. The method according to claim 1, characterized in that, Decoding the output feature map to obtain the segmentation result includes: The output feature map is input into a preset convolution module to obtain a compressed feature map with the same number of channels as the preset number of terrain categories. Obtain the confidence score of different terrain categories for each pixel in the compressed feature map; The confidence scores of different terrain categories for each pixel in the compressed feature map are normalized to obtain a prediction probability map. Perform a category indexing operation on each pixel in the predicted probability map to obtain a segmentation result including a terrain category index map corresponding to the predicted probability map, wherein the terrain category index map includes the marker positions corresponding to different terrain categories.

6. A region segmentation device based on terrain category, characterized in that, include: The extraction module is used to extract feature maps of different scales from the input image through the backbone network; The modulation module is used to input the feature maps of different scales into the accessibility-aware feature modulation module to obtain a modulated feature map based on the terrain category in each feature map. The modulation module is specifically used to: perform global average pooling on the first feature map to obtain a global context information vector, wherein the first feature map is any feature map among the feature maps of different scales; The global context information vector is input into a two-layer fully connected network to obtain a traversability context vector, which includes the traversability corresponding to each terrain category. Based on the accessibility context vector, determine the attention weight corresponding to each channel; Based on the attention weights corresponding to each channel, the first feature map is modulated to obtain a modulated feature map. The step of determining the attention weight corresponding to each channel based on the accessibility context vector includes: in, The attention weights corresponding to the channels. This is the weight matrix. For normalization function, This is the accessibility context vector. For the number of channels, The number of preset terrain categories; The fusion module is used to fuse multiple modulation feature maps to obtain an output feature map; The decoding module is used to decode the output feature map to obtain the segmentation result, which includes the marker positions corresponding to different terrain categories.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, The processor executes the computer program to implement the method as described in any one of claims 1-5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, The program is executed by a processor to implement the method as described in any one of claims 1-5.

Images