Double-branch real-time semantic segmentation method based on multi-scale semantic detail fusion

Abstract

The invention discloses a double-branch real-time semantic segmentation method based on multi-scale semantic detail fusion, and the method comprises the steps: constructing a real-time semantic segmentation model of multi-scale semantic detail fusion in a Pytorch framework, and extracting backbone network information through an LAE module; a BDF module is adopted between a high-resolution branch and a low-resolution branch to carry out information interaction at different levels, context semantic information in different branches is used, lightweight coordinate attention is utilized to improve information communication of double branches, and the learning ability of the whole network to detail information in a feature map is improved; an MLPP module is adopted at the tail end of a low-resolution branch to represent context feature information, context fusion is promoted in a local-to-global mode, features of adjacent scales are gradually clustered through deep convolution, the size of a model is reduced, the calculation speed is increased, the semantic segmentation result of an image can be accurately obtained, and the method is suitable for being applied to the field of image segmentation. Therefore, the balance between the accuracy and the speed of real-time image semantic segmentation is realized.

Description

Technical Field

[0001] The present invention relates to computer vision semantic segmentation technology, in particular to a dual-branch real-time semantic segmentation method based on multi-scale semantic detail fusion. Background Art

[0002] Lightweight semantic segmentation is a fundamental and challenging task in the field of signal processing, and is used in many real-world multimedia applications such as autonomous driving, drones, robot vision, and virtual reality. The goal of semantic segmentation is to assign a unique semantic category label to each pixel in an image. Although significant progress has been made in semantic segmentation, these high-precision semantic segmentation networks often involve a large number of models, and the model size and huge computational cost are not suitable for many real-world applications that require online estimation and real-time decision making.

[0003] The semantic segmentation algorithm based on dual-resolution network is a relatively mature semantic segmentation technology in the past few years. For example, the BiSeNet series consists of detail branches and semantic branches. The detail branch preserves spatial detail information through wide channels and small stride paths, while the semantic branch uses fast downsampling strategies and global average pooling to expand the network's receptive field and enhance feature representation. ICNet (Image Cascade Network) cuts the original image into three different sizes, uses image pyramids as input, and constructs a multi-branch image cascade network. DDRNet (Deep Dual Resolution Network) is split into two parallel branches after a convolution operation, one of which is responsible for generating high-resolution feature maps to maintain accurate spatial location information; the other branch extracts more extensive contextual semantic features through multiple downsampling. At the same time, DDRNet establishes multiple bidirectional connections between the two branches to promote interaction and fusion between features of different resolutions. In addition, a deep aggregation pyramid pooling module (DAPPM) is designed at the end of the low-resolution branch to further capture global semantic information. Although these advanced and lightweight networks have achieved good segmentation results, the above-mentioned dual-resolution architecture affects the acquisition of spatial position features when bilateral information interaction between branches, resulting in poor perception of spatial detail information by these dual-branch structures. Therefore, image pyramids or self-attention are used to encode multi-scale context, but they all face heavy calculations that are not suitable for real-time semantic segmentation. Summary of the invention

[0004] The purpose of the present invention is to provide a dual-branch real-time semantic segmentation method based on multi-scale semantic detail fusion to address the deficiencies of the prior art. This method can improve the accuracy of segmentation while ensuring the real-time performance of the model.

[0005] The technical solution for achieving the purpose of the present invention is:

[0006] A dual-branch real-time semantic segmentation method based on multi-scale semantic detail fusion involves an image input end, an image processing end and an image output end. The image input end inputs an RGB image F through an external imaging device. in , the image processing end processes F in The feature output segmentation results are sent to the image output end to display the final segmentation results, including:

[0007] 1) The completion process of the image input end includes:

[0008] 1-1) Obtain RGB image F through an external imaging device in As input;

[0009] 1-2) Image preprocessing: RGB image F in Perform random scaling, rotation, and splicing operations;

[0010] 2) The completion process of the image processing end: including:

[0011] 2-1) Image morphology reshape operation: Reshape the input F after preprocessing in Crop to 1024×1024×3 size;

[0012] 2-2) Fast downsampling feature extraction module FDFE (Fast downsampling feature extraction, referred to as FDFE): The F obtained in step 2-1) in Input the FDFE module for fast downsampling and feature extraction. The modeling process is shown in formula (1):

[0013] f s =LAE(LAE s＝2 (LAE(LAE s＝2 (LAE(LAE s＝2 (F in )))))) (1),

[0014] Among them, f s represents the feature map after fast downsampling feature extraction, LAE is a lightweight aggregation expansion module, s represents the step size, and the FDFE process is as follows: the F obtained in step 2-1) inInput six layers of LAE (Lightweight aggregation expansion, LAE for short) module, where the first layer is a LAE module with 16 convolution kernels and a step size of 2, the second layer is a LAE module with 16 convolution kernels and a step size of 1, the third layer is a LAE module with 32 convolution kernels and a step size of 2, the fourth layer is a LAE module with 32 convolution kernels and a step size of 1, the fifth layer is a LAE module with 64 convolution kernels and a step size of 2, and the sixth layer is a LAE module with 64 convolution kernels and a step size of 1. After feature extraction of the four-layer LAE module, f is obtained. s , size is 128×128×64;

[0015] Among them, the specific steps of the lightweight aggregation expansion module LAE are: when the step size is 1: first, a 3×3 depth-separable convolution is performed to obtain local feature information and increase the dimension. Secondly, a 3×3 depth-separable convolution is used to independently extract features on each channel. After that, a 1×1 point convolution is used for channel fusion and dimensionality reduction. Finally, the input and output features are residually connected to obtain the final output features; when the step size is 2: a right branch is introduced to perform the same convolution operation as the left branch, and then the output features of the left and right branches are residually connected to obtain the final output features.

[0016] 2-3) Dual branch multi-scale semantic detail fusion module DBMSDF (Dual branch multi-scale semantic detail fusion, referred to as DBMSDF): The f obtained in step 2-2) s Input DBMSDF for detail fusion and multi-scale semantic feature fusion. The modeling process is shown in formula (2), formula (3) and formula (4):

[0017] f h =LAE(LAE s＝1 (BDF(LAE s＝1 (BDF(LAE s＝1 (f s )))))) (2),

[0018] f l =LAE(LAE s＝1 (BDF(LAE s＝1 (BDF(LAE s＝1 (f s )))))) (3),

[0019] f fuse =f h +f l (4)

[0020] Among them, f h 、f l They represent the high resolution branch HSB (HSB) and the low resolution branch LSB (LSB), respectively. The feature f obtained by formula (4) fuse The fusion feature representing the final output, the DBMSDF process is as follows: f5 obtained in step 2-2) is input into the high-resolution branch HSB and the low-resolution branch LSB respectively to obtain the output feature f h and f l , the size is 128×128×128, 16×16×128, for f l Upsample to f h Same size, last f l and f h Add them together to get f fuse The size is 128×128×128. The specific operation steps are as follows: input f5 of size 128×128×64 into two layers of LAE modules with 64 convolution kernels and a step size of 1, and obtain the output feature h1 of size 128×128×64. At the same time, input a layer of LAE module with 128 convolution kernels and a step size of 2 and a layer of LAE module with 128 convolution kernels and a step size of 1, and obtain the output feature l1 of size 64×64×128. Then h1 and l1 are simultaneously input into the BDF module to obtain the output features h2 and l2 of size 128×128×64 and 64×64×128. Bilateral detail fusion BDF (Bilateral The feature size is not changed by the BDF module. h2 is input into two LAE modules with 64 convolution kernels and 1 step size, and the output feature h3 is 128×128×64. l2 is input into one LAE module with 256 convolution kernels and 2 step size, and one LAE module with 256 convolution kernels and 1 step size, and the output feature l3 is 32×32×256. h3 and l3 are input into the BDF module at the same time, and the output feature is The sizes of features h4 and l4 are 128×128×64 and 32×32×256 respectively. Then h4 is input into a layer of LAE module with 128 convolution kernels and a step size of 1, and the output feature h5 is 128×128×128 in size. l4 is input into a layer of LAE module with 512 convolution kernels and a step size of 2 and an MLPP module with 128 convolution kernels, and the output feature l5 is 16×16×128 in size. Finally, l5 is upsampled to the size of h5 and added to obtain the output feature f fuse The size is 128×128×128;

[0021] The specific steps of the bilateral detail fusion module BDF are as follows: let the output feature H of size H×W×C in the high-resolution branch in and the low-resolution branch is The output feature L in Input into the bilateral detail fusion module, where H×W represents the resolution size, C represents the number of channels, n∈{2,4}, first, let H in With L in 3×3 depth-separable convolutions are performed respectively. Secondly, after obtaining the high- and low-resolution feature maps, different weight feature maps are calculated through lightweight coordinate attention. Then, the high-resolution weight feature map is downsampled to the input feature L in Size after L in Perform residual connection, and the low-resolution weight feature map is upsampled to the input feature H in Size and H in Finally, the fused features are aggregated using 3×3 depth-separable convolution and 1×1 point convolution operations in turn. The aggregated features are residually connected with the fused features to obtain the output feature H out and L out ;

[0022] The specific steps of the multi-scale lightweight pyramid pooling module MLPP are as follows: the MLPP module first converts the input feature F into in Reduce the dimension and project to F0, where F in ∈R H×W×C , Subsequently, the generated feature map F0 undergoes an adaptive maximum pooling operation and a set of adaptive average pooling to form a multi-scale feature F1 in parallel, n∈{1,2,3,4}. The resolution of the feature F1 after maximum pooling is 1×1, and the resolutions of the features F2, F3 and F4 after average pooling are 1 / 8, 1 / 4 and 1 / 2 of the input features, respectively. In order to better aggregate these multi-scale contexts, starting from the left, F1 is upsampled to the resolution of F2, and then the enlarged feature Added to F2, the obtained features are passed through a 3×3 depth-separable convolution to produce fused features Then P2 is upsampled to F3 resolution, and the fusion process is repeated until all context features are used to obtain feature and Next, all aggregated features are upsampled to the P4 resolution for channel concatenation. Finally, a layer of 1×1 ordinary convolution is performed to aggregate information and then a hierarchical residual connection is constructed with F0.

[0023] 2-4) Using the shallow high-resolution detail features output from step 2-2) to calculate the auxiliary loss, using the output features from step 2-3) to calculate the main loss and boundary loss;

[0024] 2-5) The feature f obtained in step 2-3) fuse Input the segmentation head Seghead for segmentation to obtain the final segmentation result;

[0025] 3) Completion process: Based on the segmentation result obtained by the image processing end, the imaging device is used for real-time output, and the output result is the image covered by the mask and the average intersection-union ratio.

[0026] Compared with the prior art, the advantages of this technical solution are:

[0027] 1. The LAE module, as the backbone network of the model, has strong feature extraction capabilities and smaller parameters;

[0028] 2. Through the BDF module, different levels of information interaction are carried out between the high-resolution branch and the low-resolution branch, the contextual semantic information in different branches is used, and the lightweight coordinate attention is used to improve the information communication between the two branches, thereby improving the learning ability of the entire network for the detailed information in the feature map;

[0029] 3. The MLPP module uses a multi-scale hierarchical feature pyramid to represent contextual feature information, promotes context fusion from a local to global perspective, and uses deep convolution to gradually aggregate features of adjacent scales, reducing the size of the model and improving the calculation speed;

[0030] 4. Compared with other methods, this technical solution can reduce the floating point number and parameters of the algorithm, effectively save the algorithm running time, improve the real-time performance of the algorithm, and improve the segmentation accuracy; BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 Schematic diagram of the process of the embodiment method;

[0032] Figure 2 Schematic diagram of the structure of the fast downsampling feature extraction module FDFE in the embodiment;

[0033] Figure 3 Schematic diagram of the structure of a dual-branch multi-scale semantic detail fusion module in an embodiment;

[0034] Figure 4 It is a structural schematic diagram of a lightweight aggregation expansion module LAE in an embodiment;

[0035] Figure 5 Schematic diagram of the structure of the bilateral detail fusion module BDF in the embodiment;

[0036] Figure 6 Schematic diagram of the structure of the lightweight pyramid pooling module MLPP in the embodiment. DETAILED DESCRIPTION

[0037] The content of the present invention is further described below in conjunction with the drawings and embodiments, but the present invention is not limited thereto.

[0038] Example:

[0039] Reference Figure 1 , a dual-branch real-time semantic segmentation method based on multi-scale semantic detail fusion, involving an image input end, an image processing end and an image output end. The image input end inputs an RGB image F through an external imaging device in , the image processing end processes F in The feature output segmentation results are sent to the image output end to display the final segmentation results, including:

[0040] 1) The completion process of the image input end includes:

[0041] 1-1) Obtain RGB image F through an external imaging device in As input;

[0042] 1-2) Image preprocessing: RGB image F in Perform random scaling, rotation, and splicing operations;

[0043] 2) The completion process of the image processing end: including:

[0044] 2-1) Image morphology reshape operation: Reshape the input F after preprocessing in Crop to 1024×1024×3 size;

[0045] 2-2) Fast downsampling feature extraction module FDFE: The F obtained in step 2-1) in Input the FDFE module for fast downsampling and feature extraction. The modeling process is shown in formula (1):

[0046] f s =LAE(LAE s＝2 (LAE(LAE s＝2 (LAE(LAE s＝2 (F in )))))) (1),

[0047] Among them, f s represents the feature map after fast downsampling feature extraction, LAE is a lightweight aggregation expansion module, s represents the step size, and the FDFE process is as follows: the F obtained in step 2-1) inInput the six-layer LAE module, where the first layer is a LAE module with 16 convolution kernels and a step size of 2, the second layer is a LAE module with 16 convolution kernels and a step size of 1, the third layer is a LAE module with 32 convolution kernels and a step size of 2, the fourth layer is a LAE module with 32 convolution kernels and a step size of 1, the fifth layer is a LAE module with 64 convolution kernels and a step size of 2, and the sixth layer is a LAE module with 64 convolution kernels and a step size of 1. After feature extraction of the four-layer LAE module, f is obtained. s , size is 128×128×64;

[0048] Among them, the specific steps of the lightweight aggregation expansion module LAE are: when the step size is 1: first, a 3×3 depth-separable convolution is performed to obtain local feature information and increase the dimension. Secondly, a 3×3 depth-separable convolution is used to independently extract features on each channel. After that, a 1×1 point convolution is used for channel fusion and dimensionality reduction. Finally, the input and output features are residually connected to obtain the final output features; when the step size is 2: a right branch is introduced to perform the same convolution operation as the left branch, and then the output features of the left and right branches are residually connected to obtain the final output features.

[0049] 2-3) Dual-branch multi-scale semantic detail fusion module DBMSDF: The f obtained in step 2-2) s Input DBMSDF for detail fusion and multi-scale semantic feature fusion. The modeling process is shown in formula (2), formula (3) and formula (4):

[0050] f h =LAE(LAE s＝1 (BDF(LAE s＝1 (BDF(LAE s＝1 (f s )))))) (2),

[0051] f l =LAE(LAE s＝1 (BDF(LAE s＝1 (BDF(LAE s＝1 (f s )))))) (3),

[0052] f fuse =f h +f l (4)

[0053] Among them, f h 、f l They represent the high-resolution branch HSB and the low-resolution branch LSB respectively. The feature f obtained by formula (4) fuseThe fusion feature representing the final output, the DBMSDF process is as follows: f5 obtained in step 2-2) is input into the high-resolution branch HSB and the low-resolution branch LSB respectively to obtain the output feature f h and f l , the size is 128×128×128, 16×16×128, for f l Upsample to f h Same size, last f l and f h Add them together to get f fuse The size is 128×128×128. The specific operation steps are as follows: input f5 of size 128×128×64 into two layers of LAE modules with 64 convolution kernels and a step size of 1, and obtain the output feature h1 of size 128×128×64. At the same time, input a layer of LAE module with 128 convolution kernels and a step size of 2 and a layer of LAE module with 128 convolution kernels and a step size of 1, and obtain the output feature l1 of size 64×64×128. Then h1 and l1 are simultaneously input into the BDF module to obtain the output features h2 and l2 of size 128×128×64 and 64×64×128. The bilateral detail fusion BDF module does not change the size of the feature. Then input h2 into two layers of LAE modules with 64 convolution kernels and a step size of 1, and obtain the output feature h3 of size 12 8×128×64, l2 inputs a layer of LAE modules with 256 convolution kernels and a step size of 2 and a layer of LAE modules with 256 convolution kernels and a step size of 1, and the output feature l3 size is 32×32×256, h3 and l3 are simultaneously input into the BDF module, and the output features h4 and l4 size are 128×128×64 and 32×32×256, then h4 is input into a layer of LAE modules with 128 convolution kernels and a step size of 1, and the output feature h5 size is 128×128×128, l4 inputs a layer of LAE modules with 512 convolution kernels and a step size of 2 and an MLPP module with 128 convolution kernels, and the output feature l5 size is 16×16×128, and finally l5 is upsampled to the size of h5 and added to obtain the output feature f fuse The size is 128×128×128;

[0054] The specific steps of the bilateral detail fusion module BDF are as follows: let the output feature H of size H×W×C in the high-resolution branch in and the low-resolution branch is The output feature L in Input into the bilateral detail fusion module, where H×W represents the resolution size, C represents the number of channels, n∈{2,4}, first, let H in With L in3×3 depth-separable convolutions are performed respectively. Secondly, after obtaining the high- and low-resolution feature maps, different weight feature maps are calculated through lightweight coordinate attention. Then, the high-resolution weight feature map is downsampled to the input feature L in Size after L in Perform residual connection, and the low-resolution weight feature map is upsampled to the input feature H in Size and H in Finally, the fused features are aggregated using 3×3 depth-separable convolution and 1×1 point convolution operations in turn. The aggregated features are residually connected with the fused features to obtain the output feature H out and L out ;

[0055] The specific steps of the multi-scale lightweight pyramid pooling module MLPP are as follows: the MLPP module first converts the input feature F into in Reduce the dimension and project to F0, where F in ∈R H×W×C , Subsequently, the generated feature map F0 undergoes an adaptive maximum pooling operation and a set of adaptive average pooling to form a multi-scale feature F1 in parallel, n∈{1,2,3,4}. The resolution of the feature F1 after maximum pooling is 1×1, and the resolutions of the features F2, F3 and F4 after average pooling are 1 / 8, 1 / 4 and 1 / 2 of the input features, respectively. In order to better aggregate these multi-scale contexts, starting from the left, F1 is upsampled to the resolution of F2, and then the enlarged feature Added to F2, the obtained features are passed through a 3×3 depth-separable convolution to produce fused features Then P2 is upsampled to F3 resolution, and the fusion process is repeated until all context features are used to obtain feature and Next, all aggregated features are upsampled to the P4 resolution for channel concatenation. Finally, a layer of 1×1 ordinary convolution is performed to aggregate information and then a hierarchical residual connection is constructed with F0.

[0056] 2-4) Using the shallow high-resolution detail features output from step 2-2) to calculate the auxiliary loss, using the output features from step 2-3) to calculate the main loss and boundary loss;

[0057] 2-5) The feature f obtained in step 2-3) fuse Input the segmentation head Seghead for segmentation to obtain the final segmentation result;

[0058] 3) Completion process: Based on the segmentation result obtained by the image processing end, the imaging device is used for real-time output, and the output result is the image covered by the mask and the average intersection-union ratio.

[0059] In this example, F is obtained from the image sensor and input into the image preprocessing module to obtain the initial weight F in , and send the obtained initial weights to the fast downsampling feature extraction module FDFE to obtain the feature f s , and then send the feature to the dual-branch multi-scale semantic detail fusion module DBMSDF in the present invention to fuse the high-resolution feature with the low-resolution feature to obtain the final output feature f fuse , and sent to the SgeHead segmentation head network to obtain the final segmentation result.

[0060] like Figure 2 As shown in the figure, the fast downsampling feature extraction module FDFE takes the input F with a size of 1024×1024×3 in Input the six-layer LAE module, where the first layer is a LAE module with 16 convolution kernels and a step size of 2, the second layer is a LAE module with 16 convolution kernels and a step size of 1, the third layer is a LAE module with 32 convolution kernels and a step size of 2, the fourth layer is a LAE module with 32 convolution kernels and a step size of 1, the fifth layer is a LAE module with 64 convolution kernels and a step size of 2, and the sixth layer is a LAE module with 64 convolution kernels and a step size of 1. After feature extraction of the four-layer LAE module, f is obtained. s , size is 128×128×64;

[0061] like Figure 3 As shown in the figure, the dual-branch multi-scale semantic detail fusion module DBMSDF takes the input f of size 128×128×64 as sInput two layers of LAE modules with 64 convolution kernels and a step size of 1, and obtain the output feature h1 with a size of 128×128×64. At the same time, input a layer of LAE module with a number of convolution kernels 0 of 128 and a step size of 2 and a layer of LAE module with a number of convolution kernels 128 and a step size of 1, and obtain the output feature l1 with a size of 64×64×128. Then h1 and l1 are simultaneously input into the BDF module to obtain the output features h2 and l2 with a size of 128×128×64 and 64×64×128. The BDF module does not change the size of the feature. Then h2 is input into two layers of LAE modules with a number of convolution kernels 64 and a step size of 1, and the output feature h3 has a size of 128×128×64. l2 inputs a layer of convolution kernels with a number of 25 6. The LAE module with a step size of 2 and a LAE module with a convolution kernel number of 256 and a step size of 1 obtain the output feature l3 size of 32×32×256. h3 and l3 are simultaneously input into the BDF module to obtain the output features h4 and l4 sizes of 128×128×64 and 32×32×256. Then h4 is input into a LAE module with a convolution kernel number of 128 and a step size of 1 to obtain the output feature h5 size of 128×128×128. l4 is input into a LAE module with a convolution kernel number of 512 and a step size of 2 and an MLPP module with a convolution kernel number of 128 to obtain the output feature l5 size of 16×16×128. Finally, l5 is upsampled to the size of h5 and added to obtain the output feature f fuse The size is 128×128×128.

[0062] like Figure 4 As shown in the figure, the lightweight aggregation expansion module LAE first obtains local feature information and increases the dimension through a 3×3 depthwise separable convolution; secondly, a 3×3 depthwise separable convolution is used to extract features independently on each channel; then, a 1×1 point convolution is used for channel fusion and dimension reduction; finally, the input and output features are residually connected to obtain the final output features. When the step size is 2, the left branch structure is the same as the structure used when the step size is 1, and the right branch also performs the same operation as the left branch. Finally, the output features of the left and right branches are residually connected to obtain the final output features. The two consecutive 3×3 depthwise separable convolutions in this module have the same receptive field as the 5×5 depthwise separable convolution, but the number of parameters is smaller.

[0063] like Figure 5 As shown in the figure, the bilateral detail fusion module BDF makes the output feature H of size H×W×C in the high-resolution branch in and the low-resolution branch is The output feature L in Input into the bilateral detail fusion module, where H×W represents the resolution size, C represents the number of channels, n∈{2,4}. First, let H in With Lin 3×3 depth-separable convolutions are performed respectively. Secondly, after obtaining the high- and low-resolution feature maps, different weight feature maps are calculated through lightweight coordinate attention. Then, the high-resolution weight feature map is downsampled to the input feature L in Size after L in Perform residual connection, and the low-resolution weight feature map is upsampled to the input feature H in Size and H in Finally, the fused features are aggregated using 3×3 depthwise separable convolution and 1×1 pointwise convolution operations in turn. The aggregated features are residually connected with the fused features to obtain the output feature H out and L out .

[0064] like Figure 6 As shown in the figure, the multi-scale lightweight pyramid pooling module MLPP first converts the input feature F into in Reduce the dimension and project to F0, where F in ∈R H×W×C , Subsequently, the generated feature map F0 undergoes an adaptive maximum pooling operation and a set of adaptive average pooling to form a multi-scale feature F1 in parallel, n∈{1,2,3,4}. The resolution of the feature F1 after maximum pooling is 1×1, and the resolutions of the features F2, F3, and F4 after average pooling are 1 / 8, 1 / 4, and 1 / 2 of the input features, respectively. In order to better aggregate these multi-scale contexts, starting from the left, F1 is upsampled to the resolution of F2, and then the enlarged feature Added to F2, the obtained features are passed through a 3×3 depth-separable convolution to produce fused features Then P2 is upsampled to F3 resolution, and the above process is repeated until all context features are used to obtain feature and Next, all aggregated features are upsampled to P4 resolution for channel concatenation. Finally, after a layer of 1×1 ordinary convolution operation, information is aggregated and a hierarchical residual connection is constructed with F0.

Claims

1. A dual-branch real-time semantic segmentation method based on multi-scale semantic detail fusion, involving an image input end, an image processing end and an image output end, characterized in that: The image input end inputs the RGB image F through an external imaging device in , the image processing end processes F in The feature output segmentation results are sent to the image output end to display the final segmentation results, including: 1) The completion process of the image input end includes: 1-1) Obtain RGB image F through an external imaging device in As input; 1-2) Image preprocessing: RGB image F in Perform random scaling, rotation, and splicing operations; 2) The completion process of the image processing end: including: 2-1) Image morphology reshape operation: Reshape the input F after preprocessing in Crop to 1024×1024×3 size; 2-2) Fast downsampling feature extraction module FDFE: The F obtained in step 2-1) in Input the FDFE module for fast downsampling and feature extraction. The modeling process is shown in formula (1): f s =LAE(LAE s＝2 (LAE(LAE s＝2 (LAE(LAE s＝2 (F in )))))) (1), Among them, f s represents the feature map after fast downsampling feature extraction, LAE is a lightweight aggregation expansion module, s represents the step size, and the FDFE process is as follows: the F obtained in step 2-1) in Input the six-layer LAE module, where the first layer is a LAE module with 16 convolution kernels and a step size of 2, the second layer is a LAE module with 16 convolution kernels and a step size of 1, the third layer is a LAE module with 32 convolution kernels and a step size of 2, the fourth layer is a LAE module with 32 convolution kernels and a step size of 1, the fifth layer is a LAE module with 64 convolution kernels and a step size of 2, and the sixth layer is a LAE module with 64 convolution kernels and a step size of 1. After feature extraction of the four-layer LAE module, f is obtained. s , size is 128×128×64; Among them, the specific steps of the lightweight aggregation expansion module LAE are: when the step size is 1: first, a 3×3 depth-separable convolution is performed to obtain local feature information and increase the dimension. Secondly, a 3×3 depth-separable convolution is used to independently extract features on each channel. After that, a 1×1 point convolution is used for channel fusion and dimensionality reduction. Finally, the input and output features are residually connected to obtain the final output features; when the step size is 2: a right branch is introduced to perform the same convolution operation as the left branch, and then the output features of the left and right branches are residually connected to obtain the final output features. 2-3) Dual-branch multi-scale semantic detail fusion module DBMSDF: The f obtained in step 2-2) s Input DBMSDF for detail fusion and multi-scale semantic feature fusion. The modeling process is shown in formula (2), formula (3) and formula (4): f h =NO(NO s＝1 (BDF(NO s＝1 (BDF(NO s＝1 (f s )))))) (2), f l =NO(NO s＝1 (BDF(NO s＝1 (BDF(NO s＝1 (f s )))))) (3), f fuse =f h +f l (4), Among them, f h 、f l They represent the high-resolution branch HSB and the low-resolution branch LSB respectively. The feature f obtained by formula (4) fuse The fusion feature representing the final output, the DBMSDF process is as follows: f5 obtained in step 2-2) is input into the high-resolution branch HSB and the low-resolution branch LSB respectively to obtain the output feature f h and f l , the size is 128×128×128, 16×16×128, for f l Upsample to f h Same size, last f l and f h Add them together to get f fuse The size is 128×128×128. The specific operation steps are as follows: input f5 of size 128×128×64 into two layers of LAE modules with 64 convolution kernels and a step size of 1, and obtain the output feature h1 of size 128×128×64. At the same time, input a layer of LAE module with 128 convolution kernels and a step size of 2 and a layer of LAE module with 128 convolution kernels and a step size of 1, and obtain the output feature l1 of size 64×64×128. Then h1 and l1 are simultaneously input into the BDF module to obtain the output features h2 and l2 of size 128×128×64 and 64×64×128. The bilateral detail fusion BDF module does not change the size of the feature. Then input h2 into two layers of LAE modules with 64 convolution kernels and a step size of 1, and obtain the output feature h3 of size 12 8×128×64, l2 inputs a layer of LAE modules with 256 convolution kernels and a step size of 2 and a layer of LAE modules with 256 convolution kernels and a step size of 1, and the output feature l3 size is 32×32×256, h3 and l3 are simultaneously input into the BDF module, and the output features h4 and l4 size are 128×128×64 and 32×32×256, then h4 is input into a layer of LAE modules with 128 convolution kernels and a step size of 1, and the output feature h5 size is 128×128×128, l4 inputs a layer of LAE modules with 512 convolution kernels and a step size of 2 and an MLPP module with 128 convolution kernels, and the output feature l5 size is 16×16×128, and finally l5 is upsampled to the size of h5 and added to obtain the output feature f fuse The size is 128×128×128; The specific steps of the bilateral detail fusion module BDF are as follows: let the output feature H of size H×W×C in the high-resolution branch in and the low-resolution branch is The output feature L in Input into the bilateral detail fusion module, where H×W represents the resolution size, C represents the number of channels, n∈{2,4}, first, let H in With L in 3×3 depth-separable convolutions are performed respectively. Secondly, after obtaining the high- and low-resolution feature maps, different weight feature maps are calculated through lightweight coordinate attention. Then, the high-resolution weight feature map is downsampled to the input feature L in Size after L in Perform residual connection, and the low-resolution weight feature map is upsampled to the input feature H in Size and H in Finally, the fused features are aggregated using 3×3 depth-separable convolution and 1×1 point convolution operations in turn. The aggregated features are residually connected with the fused features to obtain the output feature H out and L out ; The specific steps of the multi-scale lightweight pyramid pooling module MLPP are as follows: the MLPP module first converts the input feature F through 1×1 convolution and batch normalization. in Reduce the dimension and project to F0, where F in ∈R H×W×C , Subsequently, the generated feature map F0 undergoes an adaptive maximum pooling operation and a set of adaptive average pooling to form a multi-scale feature F1 in parallel, n∈{1,2,3,4}. The resolution of the feature F1 after maximum pooling is 1×1, and the resolutions of the features F2, F3, and F4 after average pooling are 1 / 8, 1 / 4, and 1 / 2 of the input feature, respectively. Starting from the left, F1 is upsampled to the resolution of F2, and then the enlarged feature Added to F2, the obtained features are passed through a 3×3 depth-separable convolution to produce fused features Then P2 is upsampled to F3 resolution, and the fusion process is repeated until all context features are used to obtain feature and Next, all aggregated features are upsampled to the P4 resolution for channel concatenation. Finally, a layer of 1×1 ordinary convolution is performed to aggregate information and then a hierarchical residual connection is constructed with F0. 2-4) Using the shallow high-resolution detail features output from step 2-2) to calculate the auxiliary loss, using the output features from step 2-3) to calculate the main loss and boundary loss; 2-5) The feature f obtained in step 2-3) fuse Input the segmentation head Seghead for segmentation to obtain the final segmentation result; 3) Completion process: Based on the segmentation result obtained by the image processing end, the imaging device is used for real-time output, and the output result is the image covered by the mask and the average intersection-union ratio.

Images