Complex environment image semantic segmentation method based on fine-grained enhancement network

Abstract

The invention discloses a complex environment image semantic segmentation method based on a fine-grained enhancement network, and the method comprises the steps: obtaining a to-be-detected environment image, inputting the to-be-detected environment image into a pre-trained fine-grained enhancement network model, and obtaining a semantic segmentation result of the to-be-detected environment image. According to the method, a fine-grained enhancement network model constructed through optimization is utilized, global feature information and local feature information are subjected to deep extraction and multi-scale fusion through a backbone network layer, the feature extraction capability of the model is enhanced, and then the feature extraction capability of the model is enhanced by means of up-sampling and detail enhancement processing of a neck network layer. The detail recovery capability of the model is enhanced, the image noise is effectively reduced, the semantic segmentation result of the environment image is obtained through the detection head, and accurate semantic segmentation for the complex environment image is realized.

Description

Technical Field

[0001] The present invention relates to the fields of deep learning technology and image processing technology, and in particular to a complex environment image semantic segmentation method based on a fine-grained enhancement network. Background Art

[0002] In complex environments such as mountains and hills, autonomous unmanned systems can more effectively identify terrain features and various objects in the environment through high-precision semantic segmentation technology, thereby building a detailed and accurate environmental model. In complex environments such as mountains and hills, the types of objects often have high spatial heterogeneity and fine-grained changes, and the spatial location distribution of different types of objects is also diverse, which makes the semantic segmentation task particularly complex. Therefore, studying accurate semantic segmentation methods is of great significance to improving the intelligence level and application breadth of autonomous unmanned systems.

[0003] Semantic segmentation methods based on deep learning are usually divided into two-stage methods and one-stage methods, which have significant differences in processing procedures and application scenarios. Two-stage methods such as Mask R-CNN, DeepLab series and HRNet first generate candidate regions and then perform fine pixel-level classification on these regions. This method usually performs well in segmentation accuracy, but the model is complex and difficult to meet real-time requirements. Relatively speaking, one-stage methods such as YOLO series, SAM (Segment Anything Model) and BiSeNet directly complete target detection and segmentation tasks in a unified network. They do not rely on candidate region generation, have faster inference speed and lower computational cost, so they are more likely to meet semantic segmentation scenarios with high real-time requirements such as autonomous driving and robot vision.

[0004] In recent years, single-stage semantic segmentation methods represented by YOLO have been widely used in fields such as autonomous driving, monitoring, and industrial inspection. The ASF-YOLO model proposed by Kang et al. enhances the network's ability to extract multi-scale information by introducing SSFF and TFE modules, and improves segmentation performance by focusing on useful channels and spatial positions; Asifa et al. effectively improve the accuracy of YOLOv5 in vehicle detection and tracking by combining segmentation technology, SURF feature extraction, feature matching, and Kalman filtering; Yuan et al. proposed an adaptive multi-scale AM ​​YOLO algorithm, which fully integrates feature information of different scales and context details by introducing a multi-scale adaptive feature enhancement module MAEM and a bidirectional feature pyramid network RBiFPN; Bhavana et al.'s POT-YOLO method uses a contrast-stretched adaptive Gaussian star filter (CAGF) for image preprocessing, and combines the Sobel edge detector to achieve high-precision pit segmentation.

[0005] The above scholars have basically achieved efficient semantic segmentation in a variety of scenarios using the single-stage YOLO algorithm. However, the existing methods are still insufficient in fine-grained recognition capabilities in complex environments such as mountains and hills, resulting in room for further improvement in semantic segmentation accuracy. Summary of the invention

[0006] In view of the above-mentioned deficiencies in the prior art, the present invention provides a semantic segmentation method for complex environment images based on a fine-grained enhancement network, so as to improve the semantic segmentation accuracy for complex environments such as mountains and hills.

[0007] In order to solve the above technical problems, the present invention adopts the following technical solutions:

[0008] A complex environment image semantic segmentation method based on a fine-grained enhancement network obtains an environment image to be detected, inputs it into a pre-trained fine-grained enhancement network model, and obtains a semantic segmentation result of the environment image to be detected;

[0009] The fine-grained enhancement network model includes a backbone network layer, a neck network layer and a detection head; the backbone network layer is used to extract global features and local features of the environment image input into the feature reconstruction network and perform multi-scale fusion processing to obtain a multi-scale feature map; the neck network layer is used to upsample and perform detail enhancement processing on the multi-scale feature map to obtain an enhanced feature map; the detection head is used to obtain the semantic segmentation result of the environment image to be detected according to the enhanced feature map as the output of the fine-grained enhancement network model.

[0010] Specifically, in the fine-grained enhancement network model:

[0011] The backbone network layer includes sequentially connected convolutional layers, 4 convolutional fusion units and 1 spatial pyramid pooling module; each convolutional fusion unit includes a cascaded convolutional layer and a C2f-SSA module;

[0012] The neck network layer includes two feature dynamic upsampling units and two convolutional attention enhancement units connected in sequence; wherein the feature dynamic upsampling unit includes a Dysample-ConvBN upsampling module, a connection layer and a C2f-SSA module connected in sequence; and the convolutional attention enhancement unit includes a convolutional layer, a connection layer and a C2f-SSA module connected in sequence;

[0013] The detection head is used to obtain a semantic segmentation result of the environment image according to the output of the neck network layer;

[0014] Among them, the input of the fine-grained enhancement network model is used as the input of the backbone network layer; in the backbone network layer, the outputs of the second convolutional fusion unit and the third convolutional fusion unit are also used as the input of the connection layer in the second feature dynamic upsampling unit and the connection layer in the first feature dynamic upsampling unit in the neck network layer respectively; the output of the spatial pyramid pooling module in the backbone network layer is used as the input of the neck network layer, and is also used as the input of the connection layer in the second convolutional attention enhancement unit in the neck network layer; in the neck network layer, the output of the first feature dynamic upsampling unit is also used as the input of the connection layer in the first convolutional attention enhancement unit, and the output of the second feature dynamic upsampling unit and the outputs of the two convolutional attention enhancement units are all output to the detection head.

[0015] Specifically, the C2f-SSA module includes a cascaded first convolutional layer, a Split splitting module, multiple bottleneck modules, a connection layer, a second convolutional layer and an SSA hybrid attention module, wherein the two split outputs of the Split splitting module are respectively used as inputs of the first fast convolutional module and the connection layer, the outputs of each fast convolutional module are also used as inputs of the connection layer, and the output of the SSA hybrid attention module is used as the output of the C2f-SSA module.

[0016] Specifically, the processing process of each bottleneck module is as follows: after performing two convolution operations on the input feature map of the bottleneck module, it is superimposed with the input feature map of the bottleneck module to obtain the output feature map of the bottleneck module.

[0017] Specifically, the SSA hybrid attention module includes a cascaded SE attention module and a SA attention module;

[0018] Among them, the processing process of the SE attention module is:

[0019] First, the input feature map of the SE attention module is processed by the convolution layer, and then the convolution result is processed by global average pooling and full connection mapping in sequence. The obtained full connection mapping result is multiplied by the previous convolution result to obtain the output feature map;

[0020] The processing process of the SA attention module is:

[0021] First, the output feature map of the SE attention module is grouped into channels to obtain several channel group feature maps;

[0022] Then, for each channel group feature map, it is divided into two sub-feature maps for channel attention operation and spatial attention operation respectively; wherein, in the channel attention operation branch, the sub-feature map is processed by average pooling, full connection mapping and Sigmoid activation function in turn, and then multiplied with the sub-feature map entering the branch to obtain the channel attention feature map; in the spatial attention operation branch, the sub-feature map is processed by group normalization, full connection mapping and Sigmoid activation function in turn, and then multiplied with the sub-feature map entering the branch to obtain the spatial attention feature map; then the channel attention feature map and the spatial attention feature map are merged by element-by-element multiplication operation to obtain the attention feature sub-map of the channel group;

[0023] Finally, the attention feature sub-graphs of each channel group are rearranged and spliced ​​to obtain the attention feature graph as the output.

[0024] Specifically, the spatial pyramid pooling module includes a cascaded first convolutional layer, three two-dimensional maximum pooling layers, a connection layer and a second convolutional layer, and the outputs of the first convolutional layer and the three two-dimensional maximum pooling layers are also used as inputs of the connection layer, and the output of the second convolutional layer is used as the output of the spatial pyramid pooling module.

[0025] Specifically, the Dysample-ConvBN upsampling module includes a cascaded Dysample upsampling module and a ConvBN module;

[0026] The processing process of the Dysample upsampling module is as follows: first, a sampling set is created for the input feature map of the Dysample upsampling module through a sampling point generator, and then the input feature map is resampled using the sampling point positions in the sampling set through a grid sampling function to obtain an upsampled feature;

[0027] The ConvBN module includes a two-dimensional convolutional layer, a two-dimensional batch normalization layer and a SiLU activation function layer connected in sequence;

[0028] Use the ConvBN module to upsample the output of the Dysample upsampling module to obtain the upsampled result X g Perform local feature extraction to obtain the upsampled features output by the Dysample-ConvBN upsampling module Figure X ′.

[0029] Specifically, the fine-grained enhancement network model is trained in the following manner:

[0030] S101: preparing an environment sample image dataset, which is divided into a training image dataset and a test image dataset, wherein the environment sample images in the environment sample image dataset are all pre-labeled with semantic segmentation positions and categories;

[0031] S102: inputting the training image data set into the fine-grained enhancement network model for training, and performing parameter optimization training on the fine-grained enhancement network model with the goal of minimizing the loss function until the fine-grained enhancement network model converges, thereby obtaining a trained fine-grained enhancement network model;

[0032] S103: Test the fine-grained enhancement network model through a test image data set to confirm the semantic segmentation performance of the trained fine-grained enhancement network model; if the semantic segmentation performance meets the requirements, end the training of the fine-grained enhancement network model; otherwise, return to step S102.

[0033] Specifically, among the loss functions used to train the fine-grained enhancement network model, the SIoU loss function is selected for positioning loss, and the binary cross entropy BCE loss function is selected for classification loss.

[0034] Specifically, the SIoU loss function is calculated by the following formula:

[0035]

[0036] Among them, Loss SIoU represents SIoU loss; IoU represents the intersection-over-union loss between the semantic segmentation prediction box and the semantic segmentation target box; Δ is the distance loss between the semantic segmentation prediction box and the semantic segmentation target box, and Ω is the shape loss between the semantic segmentation prediction box and the semantic segmentation target box;

[0037] The shape loss Ω is:

[0038]

[0039] Among them, w and h are the pixel width and pixel height of the semantic segmentation prediction box respectively; w gt 、h gt are the pixel width and pixel height of the semantic segmentation target box respectively; e is a natural constant; θ is the shape attention coefficient;

[0040] The distance loss Δ is:

[0041]

[0042] in, They are the pixel x coordinate and pixel y coordinate of the center point of the semantic segmentation prediction box respectively; are the pixel x-coordinate and pixel y-coordinate of the center point of the semantic segmentation target box respectively; s w 、sh are the pixel width and pixel height of the minimum bounding rectangle of the semantic segmentation prediction box and the semantic segmentation target box respectively; Λ is the angle loss between the semantic segmentation prediction box and the semantic segmentation target box:

[0043]

[0044] Among them, σ c is the pixel distance between the center point of the semantic segmentation target box and the semantic segmentation prediction box; c h It is the pixel y coordinate difference between the semantic segmentation target box and the semantic segmentation prediction box.

[0045] Compared with the prior art, the present invention has the following technical effects:

[0046] 1. The present invention provides a complex environment image semantic segmentation method based on a fine-grained enhancement network. The method utilizes an optimized fine-grained enhancement network model to extract and fuse global feature information and local feature information at multiple scales through a backbone network layer, thereby enhancing the model's feature extraction capability. The method then uses upsampling and detail enhancement processing of a neck network layer to enhance the model's detail recovery capability and effectively reduce image noise. The semantic segmentation result of the environment image is then obtained through a detection head, thereby achieving accurate semantic segmentation of complex environment images.

[0047] 2. In the fine-grained enhanced network model of the present invention, the C2f module is combined with the SSA hybrid attention module to design the C2f-SSA module, which is introduced into the backbone network layer and the neck network layer at the same time. By arranging and calibrating the channels of the feature maps, the global feature information and local feature information are deeply extracted and multi-scale fused to capture segmentation categories of different sizes and enhance the feature extraction capability of the model.

[0048] 3. In the fine-grained enhancement network model of the present invention, a lightweight Dysample-ConvBN upsampling module is constructed in the neck network layer, which accurately controls the point sampling operation through a dynamically adjusted upsampling scale factor, and combines upsampling and convolution operations to enhance the model's detail recovery capability, effectively reduce image noise, and also save computing resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to make the purpose, technical solution and advantages of the invention more clear, the present invention will be further described in detail below with reference to the accompanying drawings, in which:

[0050] Figure 1 A schematic diagram of the architecture of the fine-grained enhancement network model used in the method of the present invention;

[0051] Figure 2 It is a schematic diagram of the structure of the C2f-SSA module;

[0052] Figure 3 This is a schematic diagram of the structure of the bottleneck module Bottleneck in the C2f-SSA module;

[0053] Figure 4 Schematic diagram of the structure of the SSA hybrid attention module;

[0054] Figure 5 It is the processing flow chart of SE attention module;

[0055] Figure 6 It is a schematic diagram of the structure of the ConvBN module;

[0056] Figure 7 It is a schematic diagram of the structure of the Dysample-ConvBN module;

[0057] Figure 8 It is a schematic diagram of the control process of the static range factor and the dynamic range factor in the Dysample upsampling module processing;

[0058] Fig. 9 It is a schematic diagram of the structure of the SPPF spatial pyramid pooling module;

[0059] Fig.10 A comparison chart of semantic segmentation results of an environment image in the MT dataset by different models in the embodiment;

[0060] Fig.11 A comparison chart of semantic segmentation results of another environment image in the MT dataset by different models in the embodiment;

[0061] Fig.12 A physical picture of an intelligent following robot using the model proposed by the present invention in the embodiment;

[0062] Fig.13 This is a graph showing the semantic segmentation results of the environment image of the intelligent follower robot in the embodiment using the model proposed in the present invention. DETAILED DESCRIPTION

[0063] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. The components of the embodiments of the present invention generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the claimed invention, but only represents selected embodiments of the present invention. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work belong to the scope of protection of the present invention.

[0064] The present invention will be further described in detail below in conjunction with the accompanying drawings.

[0065] The present invention discloses a semantic segmentation method for complex environment images based on fine-grained enhancement network. The method obtains the environment image to be detected and inputs it into a pre-trained fine-grained enhancement network model to obtain the semantic segmentation result of the environment image to be detected. The fine-grained enhancement network model used in the method of the present invention is obtained by optimizing and improving the YOLOv8 network as the basic model framework. Figure 1 As shown, it includes a backbone network layer, a neck network layer and a detection head; wherein the backbone network layer is used to extract global features and local features of the environment image input to the feature reconstruction network and perform multi-scale fusion processing to obtain a multi-scale feature map; the neck network layer is used to upsample and perform detail enhancement processing on the multi-scale feature map to obtain an enhanced feature map; the detection head is used to obtain the semantic segmentation result of the environment image to be detected according to the enhanced feature map as the output of the fine-grained enhancement network model.

[0066] The present invention is a complex environment image semantic segmentation method based on a fine-grained enhancement network. The present invention utilizes an optimized fine-grained enhancement network model to deeply extract and multi-scale fuse global feature information and local feature information through a backbone network layer to enhance the feature extraction capability of the model. Then, with the help of upsampling and detail enhancement processing of a neck network layer, the detail recovery capability of the model is enhanced and image noise is effectively reduced. Furthermore, the semantic segmentation result of the environment image is obtained through a detection head, thereby achieving accurate semantic segmentation of complex environment images.

[0067] The following is a more detailed description of the complex environment image semantic segmentation method based on the fine-grained enhancement network of the present invention and the fine-grained enhancement network model used.

[0068] 1 Fine-grained Enhanced Network Model

[0069] The existing models are not capable of fine-grained recognition in complex environments such as mountains and hills, resulting in the need for further improvement in semantic segmentation accuracy. Therefore, the present invention proposes a semantic segmentation method based on FENet, which uses a pre-trained fine-grained enhancement network model to perform semantic segmentation on environmental images. The fine-grained enhancement network model used in the method of the present invention is obtained by optimizing and improving the YOLOv8 network as the basic model framework. Figure 1 As shown, it includes a backbone network layer, a neck network layer and a detection head.

[0070] The backbone network layer includes sequentially connected convolutional layers, 4 convolutional fusion units and 1 spatial pyramid pooling module; each convolutional fusion unit includes a cascaded convolutional layer and a C2f-SSA module.

[0071] The neck network layer includes two feature dynamic upsampling units and two convolutional attention enhancement units connected in sequence; among them, the feature dynamic upsampling unit includes a Dysample-ConvBN upsampling module, a connection layer and a C2f-SSA module connected in sequence; the convolutional attention enhancement unit includes a convolutional layer, a connection layer and a C2f-SSA module connected in sequence.

[0072] The detection head is used to obtain the semantic segmentation result of the environment image based on the output of the neck network layer.

[0073] Among them, Figure 1 As shown in the figure, the input of the fine-grained enhancement network model is used as the input of the backbone network layer; in the backbone network layer, the outputs of the second convolutional fusion unit and the third convolutional fusion unit are also used as the input of the connection layer in the second feature dynamic upsampling unit and the connection layer in the first feature dynamic upsampling unit in the neck network layer, respectively; the output of the spatial pyramid pooling module in the backbone network layer is used as the input of the neck network layer, and is also used as the input of the connection layer in the second convolutional attention enhancement unit in the neck network layer; in the neck network layer, the output of the first feature dynamic upsampling unit is also used as the input of the connection layer in the first convolutional attention enhancement unit, and the output of the second feature dynamic upsampling unit and the outputs of the two convolutional attention enhancement units are all output to the detection head.

[0074] The processing process of the fine-grained enhancement network model is as follows: first, the model inputs a plateau mountain image. After being processed by the Conv module in the Backbone part, it enters the C2f-SSA module to realize the rearrangement of the number of channels and multi-level feature fusion, so as to capture segmentation categories of different sizes and improve the feature extraction ability of the model. Then, the SPPF module is used to further fuse multi-scale feature information; then, the feature map enters the Neck layer, and the Dysample-ConvBN module is used to upsample the feature map. The point sampling is controlled by the upsampling scale factor to improve the model's detail recovery ability, and then it is refined layer by layer through the Contact, C2f-SSA and Conv modules; finally, in the Head part, the SIoU loss function is used to optimize the bounding box regression to improve the semantic segmentation accuracy, and finally the segmentation categories and their corresponding confidence levels in complex environments such as mountains and hills are output.

[0075] The fine-grained enhancement network model used in the method of the present invention is mainly optimized and improved based on the YOLOv8 network: first, the C2f module is combined with the SSA hybrid attention module to design the C2f-SSA (C2f-SE-ShuffleAttention) module, which is introduced into the backbone network layer and the neck network layer at the same time. By arranging and calibrating the channels of the feature map, the global feature information and local feature information are deeply extracted and multi-scale fused to capture segmentation categories of different sizes and enhance the feature extraction ability of the model; secondly, in the neck network layer, a lightweight Dysample-ConvBN upsampling module (Dynamic Upsmaple Conv-BN), which accurately controls the point sampling operation through the dynamically adjusted upsampling scale factor, and combines upsampling and convolution operations to enhance the detail recovery ability of the model, effectively reduce image noise, and also save computing resources; not only that, in the training of the fine-grained enhancement network model, the present invention also uses the SIoU loss function to replace the original CIoU loss to further optimize the positioning regression performance of the semantic segmentation bounding box, which will be introduced later; finally, the semantic segmentation of complex environment images is more accurately achieved.

[0076] Next, the main optimization improvements in the fine-grained enhancement network model are introduced in detail.

[0077] 1.1C2f-SSA module

[0078] In order to better realize the extraction and multi-scale fusion of global and local features in environmental images and improve the feature extraction ability of the model, the present invention combines the SE (Squeeze and Excitation) attention mechanism with the SA (ShuffleAttention) attention mechanism to form an SSA (SE and Shuffle Attention) hybrid attention structure, and connects it to the tail of the C2f module to construct the C2f-SSA (C2f-SE-Shuffle Attention) module.

[0079] The overall module structure of the C2f-SSA module is as follows Figure 2 As shown in FIG, it includes a cascaded first convolution layer, a Split module, multiple bottleneck modules, a connection layer, a second convolution layer, and an SSA hybrid attention module, wherein the two split outputs of the Split module are respectively used as the input of the first fast convolution module and the connection layer, and the outputs of each fast convolution module are also used as the input of the connection layer, and the output of the SSA hybrid attention module is used as the output of the C2f-SSA module. Each bottleneck module Bottleneck performs a secondary convolution splicing process, such as Figure 3 As shown in the figure, the processing process is as follows: after performing two convolution operations on the input feature map of the bottleneck module, it is superimposed with the input feature map of the bottleneck module to obtain the output feature map of the bottleneck module.

[0080] The processing flow of the C2f-SSA module is as follows: First, the input feature map of the C2f-SSA module passes through a 1×1 first convolution layer to increase the number of channels to twice the original, thereby enhancing the feature expression ability of the model; then, the convolutional feature map is divided into two parts by the Split module, one part of which is directly passed to the connection layer Contact module, and the other part continues to be processed; the feature map that continues to be processed passes through multiple bottleneck modules Bottleneck to extract global features and deeper local feature information, and the feature map output by each bottleneck module Bottleneck enters the connection layer Contact module and is fused and spliced ​​with the convolutional feature map directly passed from the first convolution layer; the fused feature map passes through the second convolution layer to compress the number of channels to reduce the computational complexity; then, the feature map realizes multi-scale feature fusion of global features and local features through the SSA hybrid attention module, and finally generates an optimized output feature map.

[0081] The SSA hybrid attention module is composed of a cascaded SE attention module and an SA attention module, realizing the combination of SE (Squeeze and Excitation) attention mechanism and SA (Shuffle Attention) attention mechanism. Its design idea combines group convolution, spatial attention mechanism, channel attention mechanism and ShuffleNetV2, so it can increase the network's perception of key information while maintaining low computational complexity. The network structure diagram is shown below: Figure 4 shown.

[0082] The purpose of the SE (Squeeze and Excitation) attention module is to assign different weights to different positions of the image from the perspective of the channel domain through a weight matrix, so that the network can obtain more important feature information. The processing process is as follows: Figure 5 As shown in the figure: the input feature map of the SE attention module is first processed by the convolution layer, and then the convolution result is processed by global average pooling and full connection mapping in turn. The obtained full connection mapping result is multiplied by the previous convolution result to obtain the output feature map.

[0083] Specifically, the input feature of size C'×W'×H' is input to the SE attention module Figure X , it will first go through a convolution operation F tr Generate a feature map U of size C×W×H; then perform global average pooling F on the feature map U sq , generating a 1×1×C vector, so that each channel can be represented by a value; then by adjusting the fully connected layer F ex The weight W is mapped to generate an adaptive weight; finally, the generated appropriate weight is used to multiply the feature map U to assign the weight F scale , and obtain the final output features Figure X ', its size is consistent with the feature map U.

[0084] The SA (Shuffle Attention) attention module further processes the output feature map of the SE attention module, such as Figure 4 As described above, the processing process in the SA attention module is:

[0085] First, the output feature map of the SE attention module is grouped into channels to obtain several channel group feature maps;

[0086] Then, for each channel group feature map, it is divided into two sub-feature maps for channel attention operation and spatial attention operation respectively; wherein, in the channel attention operation branch, the sub-feature map is processed by average pooling, full connection mapping and Sigmoid activation function in turn, and then multiplied with the sub-feature map entering the branch to obtain the channel attention feature map; in the spatial attention operation branch, the sub-feature map is processed by group normalization, full connection mapping and Sigmoid activation function in turn, and then multiplied with the sub-feature map entering the branch to obtain the spatial attention feature map; then the channel attention feature map and the spatial attention feature map are merged by element-by-element multiplication operation to obtain the attention feature sub-map of the channel group;

[0087] Finally, the attention feature sub-graphs of each channel group are rearranged and spliced ​​to obtain the attention feature graph as the output.

[0088] Specifically, in the processing of the SSA hybrid attention module, such as Figure 4 As shown, first input features Figure X After the SE attention module is introduced, the SA attention module is entered for grouping operation, and the C channels are divided into g groups. Each channel group feature map contains C / g channels. Then, each channel group feature map is divided into two branch sub-feature maps. Each branch sub-feature map contains C / 2g channels. The two sub-feature maps are firstly processed by the average pooling layer (F gp ) and normalization layer (GN), and then each passes through a fully connected layer (F c ) mapping, and then generate weights through Sigmoid activation function operation, and then multiply them with the sub-feature maps entering their respective branches to obtain channel attention feature maps and spatial attention feature maps respectively; then, the channel attention feature maps and spatial attention feature maps output by the two branches are merged through element-by-element multiplication operation, and the features of different groups are mixed to increase the information flow across channels to obtain the attention feature sub-map of the channel group; finally, all the attention feature sub-maps of each channel group are merged, and the channel shuffle operator Channel Shuffle is used to reorder each channel group to enable information communication between different sub-features to generate the final output attention feature map S.

[0089] Through this design, the network can more effectively utilize information in different channels and enhance the expressiveness of features, which is particularly important when dealing with visual tasks with complex backgrounds and changing objects. The global and local attention mechanisms of the SSA hybrid attention module provide a wider range of contextual information, helping the model to better generalize to new or unseen data, and improving the stability and reliability of the model in practical applications. Although it adds some calculation steps, due to its design optimization (such as the relatively low computational cost of Channel Shuffle), it does not significantly increase the computational burden of the model as a whole, which enables the model to maintain good operating efficiency even in resource-constrained environments.

[0090] When performing feature extraction, the types and distribution of objects in complex environments such as mountains and hills are relatively complex, which may have a negative impact on segmentation accuracy. In order to further enhance the segmentation ability of the model in outdoor complex scenes, the present invention integrates SSA into the output end of the C2f module and designs the C2f-SSA module to help the model better focus on important areas and features in the image, thereby forming a strategy to enhance feature processing capabilities. This integration method improves network accuracy while also ensuring the efficiency of the model in mountain scene segmentation. In the design of the C2f-SSA module, the SSA hybrid attention module is applied to the output feature map, which can recalibrate and optimize the channels of the feature map at the global and local levels, thereby achieving higher-level information fusion in the deep network structure. In summary, the introduction of the C2f-SSA module not only adds powerful feature processing functions, but also optimizes the performance and efficiency of the entire model, providing strong technical support for processing highly complex visual tasks.

[0091] 1.2Dysample-ConvBN upsampling module

[0092] In order to enhance the model's detail restoration capability and realize real-time semantic segmentation of environmental images, the Conv module is combined with the Dysample upsampling structure to construct the Dysample-ConvBN upsampling module. The designed Dysample-ConvBN module is a lightweight dynamic upsampling module. It formulates upsampling from the perspective of point sampling, and accurately controls the point sampling operation through dynamically adjusted upsampling scale factors. While enhancing the model's detail restoration capability and effectively reducing image noise, it also saves computing resources and can more quickly and accurately segment different categories of object features in complex environmental images such as mountains and hills.

[0093] Since the Dysample upsampling module mainly relies on pixel-level interpolation and offset, and does not combine additional feature extraction or optimization steps when processing image details, the output image may still retain noise or artifacts. This module introduces the ConvBN module at the output stage of dynamic upsampling for further convolution normalization operations to enhance the image quality. The structure of the ConvBN module is as follows: Figure 6 As shown in Figure 1, it includes a two-dimensional convolutional layer, a two-dimensional batch normalization layer, and a SiLU activation function layer connected in sequence. The architecture of the Dysample-ConvBN upsampling module is shown in Figure 1. Figure 7 As shown, it includes a cascaded Dysample upsampling module and a ConvBN module. The ConvBN module extracts local features from the image output after upsampling the Dysample upsampling module, which helps the model's complex pattern training and learning, and can further improve the detail recovery ability and denoising effect.

[0094] like Figure 7 As shown in the figure, the main process of the Dysample-ConvBN upsampling module is as follows:

[0095] The Dysample upsampling module first performs the input feature Figure X Create a sampling set δ through the sampling point generator, and then use the sampling point positions in the sampling set to input features through the grid sampling function grid_Sample Figure X Resample to get the upsampled feature X g .

[0096] In the Dysample upsampling module, for a given upsampling scale factor s and an input feature of size C×H×W Figure X , the sampling point generator is used for input features Figure X Perform bilinear interpolation, using the input channel as C and the output channel as 2gs 2 A linear layer with a size of 2gs 2 ×H×W, control the offset O through a static range factor or a dynamic range factor, and then reshape the offset O to 2g×sH×sW through pixel transformation, add the offset O to the original sampling grid G ​​to get the sampling set δ; then, use the sampling point positions in the sampling set δ through the grid sampling function grid_Sample to sample the input feature Figure X Resample to generate an upsampled result X of size C×sH×sW g The processing expression of the Dysample upsampling module can be expressed as:

[0097] X g =grid_sample(X,δ).

[0098] The control process of the static range factor and dynamic range factor in the Dysample upsampling module is as follows: Figure 8 shown.

[0099] Using static range factors to control the offset avoids 2 The positional relationship between adjacent points makes the initial sampling position unevenly distributed, leading to the problem that unnecessary points also participate in upsampling. Figure 8 As shown in (a), the initial position is changed to bilinear initialization through DySample upsampling, and the offset is multiplied by a correction factor of 0.25, thereby avoiding the output artifacts caused by step-by-step transmission errors and satisfying the theoretical boundary conditions of overlap and non-overlap. At this time, the offset can be expressed by the following formula:

[0100] O = 0.25 linear (X).

[0101] DySample upsampling generates the dynamic range factor point by point by linearly projecting the input features, such as Figure 8 As shown in (b), by introducing a sigmoid function and a static factor of 0.5, the dynamic scope is controlled within the range of [0, 0.5], further enhancing the flexibility of the offset. At this time, the offset can be expressed by the following formula:

[0102] O=0.5sigmoid(linear1(X))·linear2(X).

[0103] The point sampling set δ is the sum of the offset O and the original sampling grid G, as shown in the following formula:

[0104] δ=O+G.

[0105] Compared to CARAFE, which generates content-aware upsampling kernels to reconstruct input features, DySample upsampling is performed by generating sampling point positions, collecting a point at each position and dividing it into s 2 sampling points, as long as s 2 If the sampling points can continue to be dynamically segmented, then the entire upsampling process only needs to collect one point. According to the kernel-based sampling standard, DySample upsampling uses a 2×2 bilinear kernel, while CARAFE upsampling uses a 5×5 bilinear kernel. The number of kernels in CARAFE upsampling is much higher than that in DySample. In addition, the kernel weights of CARAFE upsampling are obtained based on learning and require the help of feature maps of K×K channels, while the kernel weights of DySample upsampling are based on x and y as position conditions and only require feature maps of 2 channels. It can be seen that the internal structure of CARAFE upsampling is more complex and the inference speed is slower, while DySample upsampling is more efficient.

[0106] Finally, the ConvBN module is used to upsample the output of the Dysample upsampling module X g Perform local feature extraction to obtain the upsampled features output by the Dysample-ConvBN upsampling module Figure X ′.

[0107] 1.3SPPF spatial pyramid pooling module

[0108] The structure of the SPPF spatial pyramid pooling module is as follows Fig. 9 As shown, it includes a cascaded first convolution layer, three two-dimensional maximum pooling layers, a connection layer and a second convolution layer, and the outputs of the first convolution layer and the three two-dimensional maximum pooling layers are also used as inputs of the connection layer, and the output of the second convolution layer is used as the output of the spatial pyramid pooling module.

[0109] The SPPF (Spatial Pyramid Pooling Fast) module obtains multi-scale feature information through pooling operations of different scales. This multi-scale feature representation helps the model better capture the shape, size and location information of the target. Compared with the traditional SPP (Spatial Pyramid Pooling) that implements multi-scale feature extraction by applying pooling kernels of different sizes in parallel, the SPPF module reduces the amount of calculation by serially applying pooling kernels of the same size, effectively improving the running speed of the model while maintaining the feature extraction capability.

[0110] 2 Training of fine-grained enhancement network model

[0111] In the present invention, the fine-grained enhancement network model is trained in the following manner:

[0112] S101: preparing an environment sample image dataset, which is divided into a training image dataset and a test image dataset, wherein the environment sample images in the environment sample image dataset are all pre-labeled with semantic segmentation positions and categories;

[0113] S102: inputting the training image data set into the fine-grained enhancement network model for training, and performing parameter optimization training on the fine-grained enhancement network model with the goal of minimizing the loss function until the fine-grained enhancement network model converges, thereby obtaining a trained fine-grained enhancement network model;

[0114] S103: Test the fine-grained enhancement network model through a test image data set to confirm the semantic segmentation performance of the trained fine-grained enhancement network model; if the semantic segmentation performance meets the requirements, end the training of the fine-grained enhancement network model; otherwise, return to step S102.

[0115] Among the loss functions used in the training process, the SIoU loss function is selected for positioning loss, and the binary cross entropy BCE loss function is selected for classification loss, which helps to balance the convergence regression speed and training accuracy of model training.

[0116] In the semantic segmentation task, the loss function is an important parameter of deep learning regression and a measure of the accuracy of the expected results. Its main function is to help optimize the model so that it can accurately locate the target and improve the accuracy of semantic segmentation. The loss function of the YOLOv8 network model in the application of image semantic segmentation includes positioning loss and classification loss. Among them, the positioning loss is used to describe the position deviation between the semantic segmentation prediction box output by the model prediction and the semantic segmentation target box marked by the sample image, and the classification loss is used to describe the classification deviation between the semantic segmentation prediction object category output by the model prediction and the semantic segmentation target object category marked by the sample image. During the training process, the model will adjust its own parameters by minimizing the loss value to make the semantic segmentation prediction result closer to the target result marked by the sample image. In the YOLOv8 network model, the positioning loss usually selects the CIoU loss function, and the classification loss usually selects the binary cross entropy BCE loss function.

[0117] In the training of the fine-grained enhancement network model of the present invention, the classification loss continues to use the binary cross entropy BCE loss function, but the positioning loss uses the SIoU loss function instead of the CIoU loss. CIoU loss introduces the aspect ratio for regression based on the DIoU loss, but the introduced aspect ratio has certain irrationality, and it is impossible to take into account the regression direction each time, resulting in the model training to a certain extent. The problem of slow convergence speed and low efficiency. Therefore, the present invention selects the SIoU loss function instead of the CIoU loss as the loss function for semantic segmentation and positioning to avoid the above problems.

[0118] SIoU loss is a loss function used for object detection and semantic segmentation tasks, which includes three parts of loss: angle loss (Angle cost), distance loss (Distance cost), and shape loss (Shape cost). Specifically, the expression of SIoU loss function is:

[0119]

[0120] Among them, Loss SIoU represents SIoU loss; IoU represents the intersection over union loss between the semantic segmentation prediction box and the semantic segmentation target box; Δ is the distance loss between the semantic segmentation prediction box and the semantic segmentation target box, Ω is the shape loss between the semantic segmentation prediction box and the semantic segmentation target box, and the distance loss Δ is related to the angle loss Λ.

[0121] Specifically, the angle loss Λ is:

[0122]

[0123] In the formula, They are the pixel x coordinate and pixel y coordinate of the center point of the semantic segmentation prediction box respectively; are the pixel x-coordinate and pixel y-coordinate of the center point of the semantic segmentation target box respectively; σ c is the pixel distance between the center point of the semantic segmentation target box and the semantic segmentation prediction box; c h It is the pixel y coordinate difference between the semantic segmentation target box and the semantic segmentation prediction box.

[0124] The distance loss Δ is:

[0125]

[0126] In the formula, s w 、s h are the pixel width and pixel height of the minimum bounding rectangle of the semantic segmentation prediction box and the semantic segmentation target box respectively. It can be seen that when α is close to 0, the distance loss is small, and when α is equal to π / 4, the distance loss reaches the maximum.

[0127] The shape loss Ω is:

[0128]

[0129] Among them, w and h are the pixel width and pixel height of the semantic segmentation prediction box respectively; w gt 、h gt are the pixel width and pixel height of the semantic segmentation target box respectively; θ is the shape attention coefficient. The shape attention coefficient θ usually takes values ​​of 2 to 6. The smaller the value, the more attention the SIoU loss function pays to the shape loss. If the shape attention coefficient θ takes a value of 1, the SIoU loss function will pay attention to the shape loss first, thereby reducing the attention to the distance loss and angle loss. Therefore, the shape attention coefficient θ will usually prefer a value greater than 1. According to research and analysis, as the optimal value, setting the value of the shape attention coefficient θ to 4 is the best.

[0130] 3. Embodiment

[0131] In this embodiment, a data set is used to test and verify the method of the present invention and other methods to better reflect the technical advantages of the method of the present invention.

[0132] 3.1 Experimental Dataset

[0133] In order to verify the performance of the fine-grained enhanced network model proposed in the present invention as a semantic segmentation neural network model in the reference scene of semantic segmentation of complex environment images, this experiment used a self-built mountain dataset (MT). The dataset consists of 2854 images taken by a Realsense D455 camera at a test site in Chongqing from different angles, different lighting conditions and different distances. In order to enhance the diversity of the dataset, the images were further subjected to image enhancement operations such as random rotation, random cropping and random sharpening, and finally the dataset was expanded to 6204 images, which were annotated using labelme, covering six categories. According to the ratio of 8:1:1 for training set, test set and validation set, 4963 training sets, 621 test sets and 620 validation sets were obtained. The dataset contains a variety of road types and natural environments, which can be applied to the variable conditions in mountain scenes, and is helpful to evaluate the performance of the algorithm in complex environments such as mountains and hills. Through this dataset, the experiment can preliminarily evaluate the performance of the semantic segmentation neural network model proposed in the present invention in complex and diverse environments.

[0134] 3.2 Experimental Environment

[0135] The configuration table of the experimental platform is shown in Table 1. In the training of semantic segmentation neural network models, all models are trained without pre-trained weight files, the initial learning rate (lr0) is 0.01, the number of early stopping rounds (patience) is 50 rounds, the batch size (batch) is 30, the number of workers (works) is set to 8 threads, the input image size (imgsz) is (640,640), the momentum (momentum) is 0.937, the weight decay (weight_decay) is 0.0005, the number of rounds in the warm-up phase (warmup_epochs) is 3.0, the initial momentum in the warm-up phase (warmup_momentum) is 0.8, the initial bias learning rate in the warm-up phase (warmup_bias_lr) is 0.1, and the training cycle (epochs) is 300 times.

[0136] Table 1 Experimental platform configuration table

[0137]

[0138] 3.3 Results Analysis

[0139] 3.3.1 Quantitative analysis

[0140] (1) Comparative experiment

[0141] In order to quantitatively analyze the semantic segmentation performance of the fine-grained enhanced network model proposed in this invention, the same scale model is used, Mosaic data enhancement is enabled, and the experimental parameters are kept consistent. In the MT dataset, YOLOv5n-seg, YOLOv7-seg, YOLOv8n-seg, YOLOv8s-seg, and YOLOv9t-seg are used to conduct comparative experiments with the proposed model. At the same time, Precision, F1 score, mAP50, mAP50-95, Parameters, and GFLOPs are used to quantitatively evaluate all models. The experimental results are shown in Table 2.

[0142] Table 2 Comparative experimental results

[0143]

[0144] As can be seen from Table 2, the model proposed in the present invention is better than the YOLOv5n-seg and YOLOv9t-seg models. Specifically, on the basis of being lighter and more efficient, the Precision, F1 score, mAP50 and mAP50-95 results of the model proposed in the present invention are significantly better than those of the YOLOv5n-seg and YOLOv9t-seg models. Among them, the Precision increased by 4.51% and 1.65%, the F1 score increased by 1.3% and 0.7%, the mAP50 increased by 0.96% and 0.43%, and the mAP50-95 increased by 2.51% and 6.75%. This significant performance improvement shows that the model proposed in the present invention has more outstanding capabilities in feature extraction and segmentation accuracy of mountain data.

[0145] Although the parameters and GFLOPs of the original model YOLOv8n-seg are slightly smaller than those of the model proposed in this invention, its relatively simple original structure makes the segmentation effect of the model significantly inferior to that of the model proposed in this invention. Compared with YOLOv8n-seg, the precision, F1 score, mAP50 and mAP50-95 of the model proposed in this invention are improved by 1.11%, 1.04%, 2.17% and 5.26% respectively. This further highlights the superiority of the model proposed in this invention in terms of architecture design and training strategy, and proves that the model proposed in this invention can effectively improve the detection accuracy and segmentation quality while maintaining a lightweight design, thereby achieving more accurate segmentation.

[0146] In addition, the YOLOv8s-seg model has a comprehensive performance comparable to that of the model proposed in the present invention, with greatly improved Parameters and GFLOPs. Among them, the F1 and mAP50-95 of YOLOv8s-seg are slightly better than those of the model proposed in the present invention, but the Precision and mAP50 are not as good as those of the model proposed in the present invention. This shows that although the complexity of YOLOv8s-seg is higher, it has not significantly surpassed the model proposed in the present invention in specific indicators.

[0147] Compared with the model proposed in the present invention, the parameters and GFLOPs of the YOLOv7-seg model increased significantly by 34,000,000 and 130 respectively, but its F1 score, mAP50 and mAP50-95 only increased by 3.98%, 3.96% and 3.88% respectively, and the precision is still not as good as the model proposed in the present invention. In general, the model proposed in the present invention not only maintains a reasonable computational overhead and parameter amount while maintaining high-precision segmentation, but also surpasses more complex network models in multiple key performance indicators, and can be better applied to mountain segmentation scenarios.

[0148] It can be seen that the model proposed in the present invention accurately controls the point sampling operation through the upsampling scale factor dynamically adjusted by the Dysmaple-conv module, enhances the detail recovery ability of the model and effectively reduces the image noise; the multi-level fusion of global and local features is realized through the C2f-SSA module, which improves the feature extraction ability of the model.

[0149] (2) Ablation experiment

[0150] In order to further verify the effectiveness of the improvements made in the model proposed in this invention and enhance the robustness of the model, an ablation experiment is designed for verification. Taking YOLOv8n-seg as the baseline model, the number of training rounds is 300 rounds, and the Dysmple-conv module, C2f-SSA module, and SIoU loss function are superimposed step by step, and then the improved model is trained and tested. The ablation experiment results are shown in Table 3.

[0151] Table 3 Results of ablation experiments

[0152]

[0153] It can be seen from Table 3 that under the same experimental conditions, the evaluation indicators Precision, F1 score, mAP50 and mAP50-95 are improved to varying degrees with each iteration of the baseline model, which proves that the model proposed in this invention is effective.

[0154] Among them, experiment ① uses YOLOv8n-seg as the baseline model; experiment ② designs the Dysample-ConvBN module, which aims to improve the detail recovery ability of the model through the dynamic upsampling mechanism. Compared with experiment ①, the Precision, F1 score, mAP50 and mAP50-95 of experiment ② are increased by 0.11%, 0.23%, 1.05% and 1% respectively, indicating that this module improves the accuracy of the model; experiment ③ further adds the C2f-SSA module on the basis of experiment ②, by enhancing the global and local The feature extraction capability increases the model's Precision, mAP50 and mAP50-95 by 0.37%, 0.2% and 2.69% respectively. Experiment ④ is the experimental result of the model proposed in this invention. On the basis of superimposing the Dysmaple-conv module and the C2f-SSA module, the SIoU loss function is further introduced to more accurately locate the target. Compared with Experiment ①, these four evaluation indicators are improved by 1.11%, 1.04%, 2.17% and 5.26% respectively, which further proves the effectiveness of the method proposed in this invention.

[0155] Experiment ② designed the Dysample-ConvBN module, dynamically adjusted the upsampling scale factor from the perspective of point sampling, and accurately controlled the point sampling operation to enhance the model's detail recovery ability. The convolution operation at the output end further improved the clarity and detail performance of the image by learning complex features, and smoothed the input image, thereby improving the overall performance of the model. Experiment ③ superimposed the C2f-SSA module on the basis of using lightweight dynamic upsampling. Under the action of the SE attention mechanism and the Shuffle Attention mechanism, the network's perception of key information was improved, and it was able to integrate a wider range of contextual information and make full use of the correlation information between channels. This design effectively recalibrated and optimized the channels of the feature map at the global and local levels, thereby achieving higher-level information fusion in the deep network and significantly improving the accuracy of the model. Experiment ④ introduced the SIoU loss function with a reasonable aspect ratio on the basis of the previous one, which accelerated the convergence speed of the model and improved the convergence efficiency of the model. Multi-scale targets in complex environments such as mountains and hills can also be better measured by the multi-scale factor introduced by SIoU. Therefore, the proposed model fully considers the characteristics of network segmentation in complex mountainous environment scenes through the Dysample-ConvBN module, C2f-SSA module and SIoU loss function, better extracts and transmits details and semantic information in the model, and significantly improves the performance of the model in detail recovery, feature extraction and convergence efficiency. This improvement not only improves the segmentation accuracy, but also meets the application needs of actual engineering.

[0156] 3.3.2 Comparative Analysis of Semantic Segmentation Results

[0157] In order to verify the excellent semantic segmentation performance of the model proposed in this paper, YOLOv5n-seg, YOLOv8n-seg, and YOLOv9t-seg are used to compare semantic segmentation with the model proposed in this paper in the MT dataset. Two environment images in the dataset are selected, and the semantic segmentation results of each model are shown as follows: Fig.10 and Fig.11 As shown. Fig.10 and Fig.11 In the figure, sub-image (a) is the segmentation result of yolov5n-seg, sub-image (b) is the segmentation result of yolov8n-seg, sub-image (c) is the segmentation result of yolov9t-seg, and sub-image (d) is the segmentation result of the model proposed in the present invention; in sub-image (a), the colors of the mask and the detection frame are dark blue, pink, white, blue, cyan, and green, respectively corresponding to six categories: sky, vegetation, road, obstacle, sign, and pedestrian; in sub-images (b), (c), and (d), the colors of the mask and the detection frame are yellow, green, brown, red, orange, and pink, respectively corresponding to the six categories of sky, vegetation, road, obstacle, sign, and pedestrian. The number next to the category label indicates the confidence of the prediction result. Fig.10 and Fig.11 It can be seen that the model proposed in the present invention can effectively segment various categories such as sky, vegetation, roads, etc. in complex scenes. Compared with YOLOv5n-seg, YOLOv8n-seg and YOLOv9t-seg, the model proposed in the present invention shows higher confidence in the segmentation of the above six categories, thereby achieving a better segmentation effect.

[0158] Specifically, the performance of YOLOv5n-seg is slightly unstable, and the detection results of some key categories such as "sky" and "obstacles" fluctuate greatly, which shows that the robustness of the model is insufficient when processing mountain scenes, and its adaptability to different data needs to be enhanced. The performance of YOLOv8n-seg and YOLOv9t-seg also fluctuates, especially the poor segmentation effect of the "sky" category, reflecting their limitations in feature extraction and model generalization capabilities, and they are relatively weak in processing complex and changeable mountain scenes. In contrast, the detection results of the model proposed in the present invention on key categories such as "vegetation" and "road" are more stable, and the accuracy is significantly improved, indicating that the model proposed in the present invention can accurately segment and identify these important mountain scene elements. At the same time, the detection value of the "road" category has reached a high level many times, demonstrating its excellent ability in mountain road segmentation. In addition, the performance of the model proposed in the present invention in each category is more balanced, avoiding extreme errors in a certain category, and can provide reliable detection results on multiple key categories. In summary, the semantic segmentation neural network model proposed in the present invention shows higher stability and accuracy when processing mountain scenes, and can more effectively segment and identify key scene elements. The balance of its comprehensive performance enables it to provide reliable detection results in multiple key categories, reflecting its wide adaptability and superior performance in practical applications.

[0159] 4. Model Application

[0160] The above experiments show that the model proposed in the present invention meets the requirements of practical engineering applications. Therefore, the model proposed in the present invention is applied to a four-wheel drive following robot. Fig.12 As shown in the figure, NVIDIA Jetson Xavier NX is used as the experimental device, the operating system is Ubuntu 18.04, the deep learning framework is Pytorch, and the Python version used is 3.8.0.

[0161] After the intelligent follower robot is used to collect environmental images in a mountainous environment, the semantic segmentation results obtained using the model proposed by the present invention are as follows: Fig.13 shown. Fig.13 In the figure, the colors of the mask and detection box are yellow, green, brown, red, orange, and pink, corresponding to the six categories of sky, vegetation, road, obstacle, sign, and pedestrian, respectively. The number next to the category label indicates the confidence of the prediction result. Fig.13It can be seen that the model proposed in the present invention shows a good segmentation effect in the application of four-wheel drive follower robot. The model has achieved a high degree of confidence when processing pixels of different categories such as sky, vegetation, road, marker and obstacle. Specifically, the segmentation confidence of "sky" is as high as 0.98, which shows that the model can almost accurately identify the background category of sky, providing stable visual information for the robot; the recognition confidence of "vegetation" is 0.91, and this accuracy ensures that the robot can accurately judge the distribution of vegetation in the natural environment, so as to make reasonable obstacle avoidance decisions; the segmentation confidence of "road" is 0.97, and this high confidence means that the robot can clearly identify the driving path, providing a solid foundation for its stable navigation; although the confidence of "marker" and "obstacle" is slightly lower, 0.59 and 0.52 respectively, this is still within an acceptable range, and through further training and optimization, it is expected to achieve higher accuracy. Therefore, the model proposed in the present invention shows good generalization performance and can be effectively applied to the intelligent follower robot system.

[0162] 5. Overview

[0163] The present invention is a complex environment image semantic segmentation method based on a fine-grained enhancement network. The present invention utilizes an optimized fine-grained enhancement network model to deeply extract and multi-scale fuse global feature information and local feature information through a backbone network layer to enhance the feature extraction capability of the model. Then, with the help of upsampling and detail enhancement processing of a neck network layer, the detail recovery capability of the model is enhanced and image noise is effectively reduced. Furthermore, the semantic segmentation result of the environment image is obtained through a detection head, thereby achieving accurate semantic segmentation of complex environment images. In the fine-grained enhancement network model proposed by the method of the present invention, the C2f module is combined with the SSA hybrid attention module to design a C2f-SSA module, which is introduced into the backbone network layer and the neck network layer at the same time. By arranging and calibrating the channels of the feature map, the global feature information and local feature information are deeply extracted and multi-scale fused to capture segmentation categories of different sizes and enhance the feature extraction ability of the model; secondly, in the neck network layer, a lightweight Dysample-ConvBN upsampling module is constructed, which accurately controls the point sampling operation through a dynamically adjusted upsampling scale factor, and combines upsampling and convolution operations to enhance the detail recovery ability of the model, effectively reduce image noise, and also save computing resources; not only that, in the training of the fine-grained enhancement network model, the present invention also uses the SIoU loss function to replace the original CIoU loss to further optimize the positioning regression performance of the semantic segmentation bounding box; finally, the semantic segmentation of complex environment images is more accurately achieved.

[0164] The experimental results show that the accuracy, F1 score, mAP50, and mAP50-95 of the proposed model in the MT dataset are 92.7%, 87.05%, 88.66%, and 66.59%, respectively, which are 1.11%, 1.04%, 2.17%, and 5.26% higher than the baseline model YOLOv8n-seg, respectively, meeting the needs of practical engineering. In addition, the proposed model is successfully applied to an intelligent follower robot, demonstrating its effectiveness in practical applications.

[0165] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described with reference to the preferred embodiments of the present invention, it should be understood by those skilled in the art that various changes may be made in form and details without departing from the spirit and scope of the present invention as defined in the appended claims.

Claims

1. A complex environment image semantic segmentation method based on fine-grained enhancement network, characterized in that: Obtain an environment image to be detected, input it into a pre-trained fine-grained enhancement network model, and obtain a semantic segmentation result of the environment image to be detected; The fine-grained enhancement network model includes a backbone network layer, a neck network layer and a detection head; the backbone network layer is used to extract global features and local features of the environment image input into the feature reconstruction network and perform multi-scale fusion processing to obtain a multi-scale feature map; the neck network layer is used to upsample and perform detail enhancement processing on the multi-scale feature map to obtain an enhanced feature map; the detection head is used to obtain the semantic segmentation result of the environment image to be detected according to the enhanced feature map as the output of the fine-grained enhancement network model.

2. According to claim 1, the method for semantic segmentation of complex environment images based on fine-grained enhancement network is characterized in that: In the fine-grained enhancement network model: The backbone network layer includes sequentially connected convolutional layers, 4 convolutional fusion units and 1 spatial pyramid pooling module; each convolutional fusion unit includes a cascaded convolutional layer and a C2f-SSA module; The neck network layer includes two feature dynamic upsampling units and two convolutional attention enhancement units connected in sequence; wherein the feature dynamic upsampling unit includes a Dysample-ConvBN upsampling module, a connection layer and a C2f-SSA module connected in sequence; and the convolutional attention enhancement unit includes a convolutional layer, a connection layer and a C2f-SSA module connected in sequence; The detection head is used to obtain a semantic segmentation result of the environment image according to the output of the neck network layer; Among them, the input of the fine-grained enhancement network model is used as the input of the backbone network layer; in the backbone network layer, the outputs of the second convolutional fusion unit and the third convolutional fusion unit are also used as the input of the connection layer in the second feature dynamic upsampling unit and the connection layer in the first feature dynamic upsampling unit in the neck network layer respectively; the output of the spatial pyramid pooling module in the backbone network layer is used as the input of the neck network layer, and is also used as the input of the connection layer in the second convolutional attention enhancement unit in the neck network layer; in the neck network layer, the output of the first feature dynamic upsampling unit is also used as the input of the connection layer in the first convolutional attention enhancement unit, and the output of the second feature dynamic upsampling unit and the outputs of the two convolutional attention enhancement units are all output to the detection head.

3. The complex environment image semantic segmentation method based on fine-grained enhancement network according to claim 2 is characterized in that: The C2f-SSA module includes a cascaded first convolutional layer, a Split module, multiple bottleneck modules, a connection layer, a second convolutional layer and an SSA hybrid attention module, wherein the two split outputs of the Split module are respectively used as inputs of the first fast convolutional module and the connection layer, the outputs of each fast convolution module are also used as inputs of the connection layer, and the output of the SSA hybrid attention module is used as the output of the C2f-SSA module.

4. The complex environment image semantic segmentation method based on fine-grained enhancement network according to claim 3 is characterized in that: The processing process of each bottleneck module is as follows: after performing two convolution operations on the input feature map of the bottleneck module, it is superimposed with the input feature map of the bottleneck module to obtain the output feature map of the bottleneck module.

5. The complex environment image semantic segmentation method based on fine-grained enhancement network according to claim 2 is characterized in that: The SSA hybrid attention module includes a cascaded SE attention module and an SA attention module; Among them, the processing process of the SE attention module is: First, the input feature map of the SE attention module is processed by the convolution layer, and then the convolution result is processed by global average pooling and full connection mapping in sequence. The obtained full connection mapping result is multiplied by the previous convolution result to obtain the output feature map; The processing process of the SA attention module is: First, the output feature map of the SE attention module is grouped into channels to obtain several channel group feature maps; Then, for each channel group feature map, it is divided into two sub-feature maps for channel attention operation and spatial attention operation respectively; wherein, in the channel attention operation branch, the sub-feature map is processed by average pooling, full connection mapping and Sigmoid activation function in turn, and then multiplied with the sub-feature map entering the branch to obtain the channel attention feature map; in the spatial attention operation branch, the sub-feature map is processed by group normalization, full connection mapping and Sigmoid activation function in turn, and then multiplied with the sub-feature map entering the branch to obtain the spatial attention feature map; then the channel attention feature map and the spatial attention feature map are merged by element-by-element multiplication operation to obtain the attention feature sub-map of the channel group; Finally, the attention feature sub-graphs of each channel group are rearranged and spliced ​​to obtain the attention feature graph as the output.

6. The complex environment image semantic segmentation method based on fine-grained enhancement network according to claim 2 is characterized in that: The spatial pyramid pooling module includes a cascaded first convolutional layer, three two-dimensional maximum pooling layers, a connection layer and a second convolutional layer, and the outputs of the first convolutional layer and the three two-dimensional maximum pooling layers are also used as inputs of the connection layer, and the output of the second convolutional layer is used as the output of the spatial pyramid pooling module.

7. The complex environment image semantic segmentation method based on fine-grained enhancement network according to claim 2 is characterized in that: The Dysample-ConvBN upsampling module includes a cascaded Dysample upsampling module and a ConvBN module; The processing process of the Dysample upsampling module is as follows: first, a sampling set is created for the input feature map of the Dysample upsampling module through a sampling point generator, and then the input feature map is resampled using the sampling point positions in the sampling set through a grid sampling function to obtain an upsampled feature; The ConvBN module includes a two-dimensional convolutional layer, a two-dimensional batch normalization layer and a SiLU activation function layer connected in sequence; Use the ConvBN module to upsample the output of the Dysample upsampling module to obtain the upsampled result X g Local feature extraction is performed to obtain the upsampled feature map X′ as the output of the Dysample-ConvBN upsampling module.

8. The complex environment image semantic segmentation method based on fine-grained enhancement network according to claim 1 is characterized in that: The fine-grained enhancement network model is trained in the following way: S101: preparing an environment sample image dataset, which is divided into a training image dataset and a test image dataset, wherein the environment sample images in the environment sample image dataset are all pre-labeled with semantic segmentation positions and categories; S102: inputting the training image data set into the fine-grained enhancement network model for training, and performing parameter optimization training on the fine-grained enhancement network model with the goal of minimizing the loss function until the fine-grained enhancement network model converges, thereby obtaining a trained fine-grained enhancement network model; S103: Testing the fine-grained enhancement network model using a test image data set to confirm the semantic segmentation performance of the trained fine-grained enhancement network model; if the semantic segmentation performance meets the requirements, the training of the fine-grained enhancement network model is terminated; Otherwise, return to step S102.

9. The complex environment image semantic segmentation method based on fine-grained enhancement network according to claim 8 is characterized in that: Among the loss functions used to train the fine-grained enhancement network model, the SIoU loss function is selected for positioning loss, and the binary cross entropy BCE loss function is selected for classification loss.

10. The complex environment image semantic segmentation method based on fine-grained enhancement network according to claim 9, characterized in that: The SIoU loss function is calculated by the following formula: Among them, Loss SIoU represents SIoU loss; IoU represents the intersection-over-union loss between the semantic segmentation prediction box and the semantic segmentation target box; Δ is the distance loss between the semantic segmentation prediction box and the semantic segmentation target box, and Ω is the shape loss between the semantic segmentation prediction box and the semantic segmentation target box; The shape loss Ω is: Among them, w and h are the pixel width and pixel height of the semantic segmentation prediction box respectively; w gt 、h gt are the pixel width and pixel height of the semantic segmentation target box respectively; e is a natural constant; θ is the shape attention coefficient; The distance loss Δ is: in, They are the pixel x coordinate and pixel y coordinate of the center point of the semantic segmentation prediction box respectively; are the pixel x-coordinate and pixel y-coordinate of the center point of the semantic segmentation target box respectively; s w 、s h are the pixel width and pixel height of the minimum bounding rectangle of the semantic segmentation prediction box and the semantic segmentation target box respectively; Λ is the angle loss between the semantic segmentation prediction box and the semantic segmentation target box: Among them, σ c is the pixel distance between the center point of the semantic segmentation target box and the semantic segmentation prediction box; c h It is the pixel y coordinate difference between the semantic segmentation target box and the semantic segmentation prediction box.

Images