Greening waste material segmentation method and device based on cascade feature fusion and hybrid coding

Abstract

The present invention discloses a method and device for segmenting greening waste materials based on cascade feature fusion and hybrid coding, including: obtaining a greening waste material scene image, and preprocessing the greening waste material scene image; inputting the preprocessed greening waste material scene image into an improved semantic segmentation network structure, and outputting a segmentation result through the improved semantic segmentation network structure. The improved semantic segmentation network structure of the present invention uses high-resolution feature maps for cascade feature fusion, uses a self-attention mechanism of efficient hybrid coding and a ring feature fusion network to perform feature differentiation and expand differences on the C5, C4, and C3 layers respectively, and finally increases the shallow representation capability through a U-shaped structure, so that high-precision segmentation operations of greening waste material scenes can be performed at the edge end with low power consumption and low computing power. The present invention can use low-resource edge computing devices in mobile or fixed scenarios to complete high-precision semantic segmentation tasks for greening waste material scenes.

Description

Technical Field

[0001] The present invention relates to the technical field of greening waste material scene image segmentation, and in particular to a greening waste material segmentation method and device based on cascade feature fusion and hybrid coding. Background Art

[0002] Currently, the mainstream high-precision methods for semantic segmentation scenarios are all based on various variants of transformers, such as large semantic segmentation models such as SAM (Segment Anything), FastSAM (Fast Segment Anything), MobileSAM (MobileSegment Anything), or HRNet networks (High-resolution networks) that use high-resolution features, and Deeplabv3+ that uses encoder-decoder structures and pooling pyramids.

[0003] For various SAM variant models, while having high segmentation capabilities, the introduction of the MFP (Masked Feature Prediction) mechanism and negative sample processing makes it more flexible in processing objects of various shapes and sizes. However, the use of a multi-layer Transformer encoder and the direct processing of the embedding of the original image result in a huge number of model parameters. Although the HRNet network, which makes full use of high-resolution features, has high accuracy, it also has the problem of large number of parameters and serious resource consumption due to the intensive use of high-resolution features. In addition, the subsequent head processing is simple, and the segmentation effect between similar targets is not good. Although codec networks such as Deeplabv3+ can reduce resource consumption, their accuracy is insufficient. This leads to a greater demand, but the edge end with limited computing resources cannot deploy a semantic segmentation network with real-time reasoning and high accuracy, especially for small targets and difficult-to-distinguish targets. The segmentation accuracy declines more seriously.

[0004] The scene of landscaping waste is a common sanitation problem. In this scene, branches fall or are abandoned on the lawn after being trimmed. Since the abandoned branches and trunks are small and highly compatible with the lawn and other environments, the various mainstream semantic segmentation models of various sizes are currently unable to segment the targets or have poor segmentation effects at the edge due to resource consumption or accuracy issues, making it impossible to arrange reasonable sanitation resources for timely processing.

[0005] In view of this, this application is hereby filed. Summary of the invention

[0006] The technical problem to be solved by the present invention is that the existing image semantic segmentation model has poor segmentation effect on the abandoned branches and trunks in the image of the greening waste material scene, because the greening waste material scene is caused by branches falling or being abandoned on the lawn after being trimmed, and the abandoned branches and trunks are relatively small and highly compatible with the lawn and other environments. The purpose of the present invention is to provide a greening waste material segmentation method and device based on cascaded feature fusion and hybrid coding. The present invention uses multi-layer cascaded feature fusion and hybrid dilated convolution HDC, deep separable convolution and efficient hybrid encoder (HCEM) and other technologies to design a high-precision, low-power consumption improved semantic segmentation network structure. The improved semantic segmentation network structure solves the problem of poor segmentation effect of greening waste material scenes at the vehicle-mounted edge end, and improves the segmentation accuracy of greening waste material scenes with small targets and high foreground and background fusion.

[0007] The present invention is achieved through the following technical solutions:

[0008] In a first aspect, the present invention provides a method for segmenting greening waste materials based on cascaded feature fusion and hybrid coding, the method comprising:

[0009] Acquire a greening waste material scene image, and preprocess the greening waste material scene image to obtain a preprocessed greening waste material scene image;

[0010] The pre-processed greening waste scene image is input into the improved semantic segmentation network structure, and the segmentation result is output through the improved semantic segmentation network structure;

[0011] Improved semantic segmentation network structure including backbone network, efficient hybrid encoding module, decoding and head module;

[0012] The backbone network adopts a multi-layer cascade feature extraction network, which is used to downsample the pre-processed greening waste scene image and then enter the multi-layer cascade feature layer for feature extraction to obtain four feature maps with different resolutions; the multi-layer cascade feature layer includes the first feature layer C2, the second feature layer C3, the third feature layer C4 and the fourth feature layer C5;

[0013] An efficient hybrid coding module is used to perform cascade feature fusion on the three feature maps of different resolutions from C3 to C5 extracted by the multi-layer cascade feature extraction network, expand the differences between targets, and obtain the fused feature map;

[0014] The decoding and head module is used to reconstruct the output position encoding of the fused feature map, perform upsampling fusion of the three-layer U-shaped structure with the output branches of the C2 to C4 layers, and output the segmentation result.

[0015] Furthermore, the multi-layer cascade feature extraction network improves the original backbone network of the HRNet network by replacing all the common convolutional blocks in the original backbone network cascade with the depthwise separable convolutional layer (DSC), which greatly reduces the number of parameters. At the same time, the low-level semantic information of the first feature layer C2 and the second feature layer C3 of the shallow feature layer is retained, that is, the high resolution of the C2 and C3 layers is retained.

[0016] In the third feature layer C4 and the fourth feature layer C5 of the deep feature layer, a hybrid dilated convolution layer (HDC) is used to extract features, expand the receptive field of the convolution and do not introduce the checkerboard effect caused by the dilated convolution.

[0017] Furthermore, the first feature map of each layer in the multi-layer cascade feature layer is downsampled using a convolution block with a stride of 2 to obtain the first feature map of the next layer; each layer is interactively connected through upsampling and downsampling, and then passes through the stage stage in sequence. Each stage stage is connected by multiple basic blocks BasicBlock. The three stages use 2 BasicBlocks, 4 BasicBlocks, and 2 BasicBlocks respectively.

[0018] The first feature layer C2 and the second feature layer C3 are processed by depthwise separable convolutional layers, the third feature layer C4 and the fourth feature layer C5 are processed by mixed dilated convolutional layers, and each layer uniformly uses the GELU activation function.

[0019] Furthermore, the efficient hybrid coding module includes an intrascale interaction network IIF and a ring network TF;

[0020] The inner scale interaction network IIF is an inner scale interaction network designed based on Transformer. It is used to apply the self-attention mechanism to the semantic features of the fourth feature layer C5, which has smaller parameters, deeper and richer parameters, and capture the differences between objects at similar positions in the image for subsequent ring network TF to perform differential expansion.

[0021] The ring network TF includes three fusion module nodes, which are used to directly input the feature maps extracted by the second feature layer C3 and the third feature layer C4, and perform feature fusion of the second feature layer C3, the third feature layer C4, and the fourth feature layer C5 to expand feature differentiation.

[0022] Furthermore, the loss function FSL(pt,s(I)) in the decoding and header modules is:

[0023] FSL(pt,s(I))=-αts(I)(1-pt)γlog(pt)

[0024] Where pt is the true label of the target in the greening waste scene image; 1-pt is the probability of the background in the greening waste scene image; αt is the category weight; γ is the adjustment factor used to balance the loss of easy-to-classify and difficult-to-classify samples; s(I) is the weight factor, and the value of s(I) depends on the size of the object in the greening waste scene image; I is the input greening waste scene image.

[0025] Furthermore, the weight factor is determined as follows:

[0026] s(I)=1-[log(mask_t) / log(mask_i)]

[0027] Where mask_t represents the foreground mask pixel value; mask_i represents the background mask pixel value.

[0028] In a second aspect, the present invention further provides a greening waste material segmentation device based on cascade feature fusion and hybrid coding, the device uses the greening waste material segmentation method based on cascade feature fusion and hybrid coding; the device comprises:

[0029] An acquisition unit, used for acquiring an image of a greening waste material scene;

[0030] A preprocessing unit, used for preprocessing the greening waste material scene image to obtain a preprocessed greening waste material scene image;

[0031] A segmentation unit, used for inputting the preprocessed greening waste scene image into the improved semantic segmentation network structure, and outputting the segmentation result through the improved semantic segmentation network structure;

[0032] Among them, the improved semantic segmentation network structure includes a backbone network, an efficient hybrid encoding module, and a decoding and head module;

[0033] The backbone network adopts a multi-layer cascade feature extraction network, which is used to downsample the pre-processed greening waste scene image and then enter the multi-layer cascade feature layer for feature extraction to obtain four feature maps with different resolutions; the multi-layer cascade feature layer includes the first feature layer C2, the second feature layer C3, the third feature layer C4 and the fourth feature layer C5;

[0034] An efficient hybrid coding module is used to perform cascade feature fusion on the three feature maps of different resolutions from C3 to C5 extracted by the multi-layer cascade feature extraction network, expand the differences between targets, and obtain the fused feature map;

[0035] The decoding and head module is used to reconstruct the output position encoding of the fused feature map, perform upsampling fusion of the three-layer U-shaped structure with the output branches of the C2 to C4 layers, and output the segmentation result.

[0036] Furthermore, the multi-layer cascade feature extraction network improves the original backbone network of the HRNet network by replacing all the common convolutional blocks in the original backbone network cascade with the depthwise separable convolutional layer (DSC), which greatly reduces the number of parameters. At the same time, the low-level semantic information of the first feature layer C2 and the second feature layer C3 of the shallow feature layer is retained, that is, the high resolution of the C2 and C3 layers is retained.

[0037] In the third feature layer C4 and the fourth feature layer C5 of the deep feature layer, a hybrid dilated convolution layer (HDC) is used to extract features, expand the receptive field of the convolution and do not introduce the checkerboard effect caused by the dilated convolution.

[0038] In a third aspect, the present invention further provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the above-mentioned greening waste segmentation method based on cascaded feature fusion and hybrid coding is implemented.

[0039] In a fourth aspect, the present invention further provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the above-mentioned greening waste segmentation method based on cascaded feature fusion and hybrid coding.

[0040] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0041] 1. The present invention is based on a method and device for segmenting abandoned greening materials based on cascaded feature fusion and hybrid coding. The present invention improves the semantic segmentation network structure UHTNet by designing a cascaded feature layer combined with an efficient hybrid encoder and a U-shaped structure; it not only utilizes the high-resolution features to retain details, but also uses an efficient hybrid coding module to enhance the distinction between different targets, while greatly reducing the number of parameters of the C2, C3 layers and the attention mechanism module, achieving high-precision segmentation of abandoned greening materials with small foreground targets and close similarity to the background at the edge. The present invention solves the segmentation accuracy and segmentation effect of abandoned greening materials scenes with small targets and high foreground and background fusion at the vehicle-mounted edge.

[0042] 2. The present invention is based on a greening waste segmentation method and device based on cascaded feature fusion and hybrid coding, and designs a new loss function for greening waste in business scenarios. By introducing a weight factor s(I) related to the object size to balance the weight influence of foreground and background on back propagation, the accuracy of the improved semantic segmentation network structure is further improved.

[0043] 3. The improved semantic segmentation network structure designed by the present invention can achieve higher segmentation accuracy by expanding the number of network layers (for example, increasing the number of modules in the stage) and designing a backbone network with stronger extraction capabilities, and achieve better results when resources are sufficient. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] The drawings described herein are used to provide a further understanding of the embodiments of the present invention, constitute a part of this application, and do not constitute a limitation of the embodiments of the present invention. In the drawings:

[0045] Figure 1 It is a flow chart of the greening waste material segmentation method based on cascade feature fusion and hybrid coding of the present invention;

[0046] Figure 2 This is a schematic diagram of the improved semantic segmentation network structure of the present invention;

[0047] Figure 3 This is a structural block diagram of the greening waste material segmentation device based on cascaded feature fusion and hybrid coding of the present invention. DETAILED DESCRIPTION

[0048] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with embodiments and drawings. The exemplary embodiments of the present invention and their description are only used to explain the present invention and are not intended to limit the present invention.

[0049] Since the greening waste scene is branches falling or being discarded on the lawn after being trimmed, and the discarded branches and trunks are relatively small and highly compatible with the lawn and other environments, the existing image semantic segmentation model has a problem of poor segmentation effect on the discarded branches and trunks in the greening waste scene image. The present invention designs a greening waste segmentation method and device based on cascaded feature fusion and hybrid coding. The present invention uses multi-layer cascaded feature fusion and hybrid dilated convolution HDC, deep separable convolution and high-efficiency hybrid encoder (HCEM) and other technologies to design a high-precision, low-power improved semantic segmentation network structure UHTNet (U-shaped network based on HD feature maps and transformer). The improved semantic segmentation network structure solves the problem of poor segmentation effect of greening waste scenes at the vehicle-mounted edge end, and improves the segmentation accuracy of greening waste scenes with small targets and high foreground and background fusion.

[0050] The key technical points of the present invention are as follows:

[0051] 1. Redesign of multi-layer cascade feature network. Analyze the problems of the current top semantic segmentation network, redesign the network structure, remove all the general convolution blocks in the original cascade, and use deep separable convolution (DSC) instead to retain the low-level semantic information of the shallow feature layers C2 and C3 while greatly reducing the number of parameters; use hybrid dilated convolution (HDC) to extract features in the deep feature layers C4 and C5, expand the receptive field under limited computing resources, reduce the loss caused by downsampling, and alleviate the chessboard effect caused by dilated convolution. The purpose of this is to increase the receptive field of the network layer and reduce the loss of image resolution caused by downsampling when computing resources are limited and the number and size of convolution kernels cannot be increased.

[0052] Second, the features extracted by the multi-layer cascade feature network are further processed using an efficient hybrid coding module (HCEM). The HCEM module consists of two parts: one is the inner scale interaction network (IIF) designed by Transformer, and the other is a ring network (TF) composed of three fusion module (Fusion) nodes. The inner scale interaction network (IIF) applies the self-attention mechanism to the semantic features of the C5 layer with smaller parameters, deeper and richer parameters, captures the differences between objects in similar positions in the image, and helps the subsequent ring network (TF) to expand the differences. The C3 and C4 layers directly enter the ring network (TF). The purpose of the entire module is not only to expand the differences between objects through the efficient hybrid coding module, but also to further alleviate the impact of the shallow feature layers C2 and C3 on the loss of information between channels due to the fusion of the C4 and C5 layers.

[0053] The design of the above efficient hybrid coding module (HCEM) extracts and fuses the three outputs of the multi-layer cascade features, and uses the intra-scale interaction (IIF) designed by Transformer for the C5 layer with smaller feature maps and richer semantic features. This can shorten the sequence length of the input encoder. At the same time, the self-attention mechanism is used to enrich the features of the abstract C5 layer to increase the differences between similar targets. The ring network (TF) further expands this differentiation, which greatly reduces the amount of calculation without damaging the performance of the model.

[0054] 3. The decoding and head modules adopt a three-layer U-shaped structure, which is connected with the C4, C3, and C2 layers upsampling in sequence through the output of the HCEM module. The purpose of using the U-shaped structure is to make full use of the high-resolution feature layer to better distinguish details such as edges and colors. The head directly uses 1x1 convolution for mask prediction.

[0055] After the decoding and head modules reconstruct the position encoding of the output of the high-efficiency hybrid coding module (HCEM), they are fused with the upsampling of the three-layer U-shaped structure with the output branches of the C4 layer, C3 layer, and C2 layer, and finally the convolution layer is used to obtain the final mask, that is, the segmentation result.

[0056] 4. Semantic segmentation: It is often more difficult for small and similar models to segment small targets. This is partly because the mask ratio of the foreground and background is too different. The loss value of the background target has a greater impact during back propagation. Therefore, a new loss function is designed based on FocalLoss to adapt to the semantic segmentation of small targets at the vehicle edge, and a weight factor based on the object size is added.

[0057] Example 1

[0058] like Figure 1 As shown, the greening waste material segmentation method based on cascade feature fusion and hybrid coding of the present invention, the hardware parts relied on by the method of the present invention include: (1) high-definition camera; parameters: 2 million pixel (1920*1080) high-definition camera, the distance between the defense detection area and the camera is less than 10 meters and greater than 1 meter, ipx6 waterproof. (2) computing platform; parameters: edge computing device, memory and video memory of not less than 8G, processor with more than 6 cores, and main frequency of not less than 2.3GHz.

[0059] The greening waste segmentation method based on cascade feature fusion and hybrid coding of the present invention can complete the high-precision semantic segmentation task of greening waste scenes using low-resource edge computing devices in mobile or fixed scenes. By using high-resolution feature maps for cascade feature fusion, using the self-attention mechanism of efficient hybrid coding and the ring feature fusion network to differentiate and expand the features of the C5, C4, and C3 layers respectively, and finally increasing the shallow representation capability through the U-shaped structure, it is possible to perform high-precision segmentation operations of greening waste scenes at the edge end with low power consumption and low computing power.

[0060] The method includes:

[0061] Acquire a greening waste material scene image, and preprocess the greening waste material scene image to obtain a preprocessed greening waste material scene image;

[0062] The pre-processed greening waste scene image is input into the improved semantic segmentation network structure, and the segmentation result is output through the improved semantic segmentation network structure;

[0063] like Figure 2 As shown, the improved semantic segmentation network structure includes a backbone network, an efficient hybrid encoding module, and a decoding and head module;

[0064] The backbone network adopts a multi-layer cascade feature extraction network, which is used to downsample the pre-processed greening waste scene image and then enter the multi-layer cascade feature layer for feature extraction to obtain four feature maps with different resolutions; the multi-layer cascade feature layer includes the first feature layer C2, the second feature layer C3, the third feature layer C4 and the fourth feature layer C5;

[0065] An efficient hybrid coding module is used to perform cascade feature fusion on the three feature maps of different resolutions from C3 to C5 extracted by the multi-layer cascade feature extraction network, expand the differences between targets, and obtain the fused feature map;

[0066] The decoding and head module is used to reconstruct the output position encoding of the fused feature map, perform upsampling fusion of the three-layer U-shaped structure with the output branches of the C2 to C4 layers, and output the segmentation result.

[0067] The present invention uses a backbone network (Backbone) to use high-resolution cascaded feature fusion with low parameters to retain shallow representation capabilities, and uses mixed diffusion convolution to increase the receptive field and alleviate the checkerboard effect. The IIF and TF modules of the efficient hybrid encoder are used for feature differentiation, which strengthens the distinction between the details of different objects while reducing the impact of channel information collection brought by the deep separable convolution. The IIF module of the self-attention mechanism only embeds the output of the C5 layer as input for semantic feature enrichment processing, so a large number of invalid features will not be added. Finally, a U-shaped structure is used in the decoding stage, combining the C2, C3, C4 with stronger representation capabilities and the HCEM output with higher feature differentiation for cross-scale feature fusion.

[0068] In this embodiment, the multi-layer cascade feature extraction network is an improvement on the original backbone network of the HRNet network. All the general convolution blocks in the original backbone network cascade are replaced with a depthwise separable convolution layer (DSC), which greatly reduces the number of parameters. At the same time, the low-level semantic information of the first feature layer C2 and the second feature layer C3 of the shallow feature layer is retained, that is, the high resolution of the C2 and C3 layers is retained.

[0069] In the third feature layer C4 and the fourth feature layer C5 of the deep feature layer, a hybrid dilated convolution layer (HDC) is used to extract features, expand the receptive field of the convolution and do not introduce the checkerboard effect caused by the dilated convolution.

[0070] In this embodiment, the efficient hybrid coding module includes an intrascale interaction network IIF and a ring network TF;

[0071] The inner scale interaction network IIF is an inner scale interaction network designed based on Transformer. It is used to apply the self-attention mechanism to the semantic features of the fourth feature layer C5, which has smaller parameters, deeper and richer parameters, and capture the differences between objects at similar positions in the image for subsequent ring network TF to perform differential expansion.

[0072] The ring network TF includes three fusion module nodes, which are used to directly input the feature maps extracted by the second feature layer C3 and the third feature layer C4, and perform feature fusion of the second feature layer C3, the third feature layer C4, and the fourth feature layer C5 to expand feature differentiation.

[0073] In this embodiment, the present invention designs a new loss function. A weight factor related to the object size is added. The present invention calls this new loss function Focal Size Loss (FSL):

[0074] Suppose an input image X, whose size is H×W, and an output feature map Y, whose size is also H×W. Each element yi in the feature map Y represents the predicted probability distribution of the corresponding pixel. We simplify the predicted probability distribution of each pixel to a two-class classification problem, namely, pixels within the object (foreground) and pixels outside the object (background).

[0075] The original Focal Loss formula is: FL(pt) = -αt(1-pt)γlog(pt)

[0076] Among them, pt is the true label of the target in the greening waste scene image; 1-pt is the probability of the background in the greening waste scene image; αt is the category weight; γ is the adjustment factor used to balance the loss of easy-to-classify and difficult-to-classify samples.

[0077] In FSL, the present invention introduces a weight factor s(I) related to the object size, where I is the input green waste scene image, and the value of s(I) depends on the object size in the green waste scene image. We use the object mask to determine the size of the object and adjust the loss accordingly.

[0078] Therefore, the new loss function FSL(pt,s(I)) in the decoding and head modules is:

[0079] FSL(pt,s(I))=-αts(I)(1-pt)γlog(pt)

[0080] s(I)=1-[log(mask_t) / log(mask_i)]

[0081] Wherein, pt is the true label of the target in the green waste scene image; 1-pt is the probability of the background in the green waste scene image; αt is the category weight; γ is the adjustment factor used to balance the loss of easy-to-classify and difficult-to-classify samples; s(I) is the weight factor, and the value of s(I) depends on the size of the object in the green waste scene image; I is the input green waste scene image; mask_t represents the foreground mask pixel value, such as 20x20; mask_i represents the background mask pixel value, such as 800x800.

[0082] The weight factor s(I) will make smaller objects account for a larger proportion in the loss function, thereby improving the segmentation accuracy of small objects.

[0083] By using the FSL loss function, we find that the training results are 0.62 percentage points higher than using only FocalLoss.

[0084] The present invention is specifically implemented as follows:

[0085] (1) The greening waste scene images are collected through sampling equipment, and the greening waste scene images are preprocessed into input images of 1024x768 size.

[0086] (2) Model structure stage: Improving the semantic segmentation network structure

[0087] Backbone network: After downsampling the preprocessed image of size 1024x768, it enters the multi-layer cascade feature layer. The multi-layer cascade feature layer includes four layers. The first feature map of each layer is downsampled by a convolution block with a step size of 2 to obtain the first feature map of the next layer; each layer is interactively connected through upsampling and downsampling, and then passes through the stage stage in sequence. Each stage stage is connected by multiple basic blocks BasicBlock. The three stages use 2 basic blocks BasicBlock, 4 basic blocks BasicBlock, and 2 basic blocks BasicBlock respectively. The first feature layer C2 and the second feature layer C3 are processed by depth-separable convolution layers, and the third feature layer C4 and the fourth feature layer C5 are processed by mixed dilated convolution layers. Each layer uses the GELU activation function.

[0088] High-efficiency hybrid coding module (HCEM): The HCEM module receives three inputs. The output of the C5 layer is embedded through the IIF module and then enters the self-attention mechanism network. The output vector is then adjusted to a matrix according to the position encoding and enters the TF module with C4 and C3 to expand the feature differentiation.

[0089] Decode and Head module (Decode&Head): It adopts a U-shaped structure, flattens the output of the HCEM module and reconstructs it into a matrix according to the position code, upsamples the features of the C4, C3, and C2 layers respectively, and then concatenates the channels (cat) and passes through the channel reduction convolution block, and finally completes the mask prediction through a convolution layer whose output channels are the number of categories.

[0090] Among them, the loss function stage is used to calculate the predicted mask and label (groundtrue) in the decoding stage to obtain the loss for back propagation; according to the actual business situation of small target segmentation at the edge, the present invention redesigns a loss function FSL(pt,s(I)), which gives different weight factors s(I) to different targets by counting the object size in the mask. Combined with the actual training effect, compared with FL, the training effect of increasing the weight factor can increase by 0.6% to 0.8%.

[0091] The improved semantic segmentation network structure designed by the present invention can achieve higher segmentation accuracy by expanding the number of network layers (for example, increasing the number of modules in the stage) and designing a Backbone with stronger extraction capabilities, and achieve better results when resources are sufficient.

[0092] Example 2

[0093] like Figure 3 As shown, the difference between this embodiment and embodiment 1 is that this embodiment provides a greening waste material segmentation device based on cascade feature fusion and hybrid coding, and the device uses the greening waste material segmentation method based on cascade feature fusion and hybrid coding in embodiment 1; the device corresponds one-to-one to the greening waste material segmentation method based on cascade feature fusion and hybrid coding in embodiment 1; the device includes:

[0094] An acquisition unit, used for acquiring an image of a greening waste material scene;

[0095] A preprocessing unit, used for preprocessing the greening waste material scene image to obtain a preprocessed greening waste material scene image;

[0096] A segmentation unit, used for inputting the preprocessed greening waste scene image into the improved semantic segmentation network structure, and outputting the segmentation result through the improved semantic segmentation network structure;

[0097] Among them, the improved semantic segmentation network structure includes a backbone network, an efficient hybrid encoding module, and a decoding and head module;

[0098] The backbone network adopts a multi-layer cascade feature extraction network, which is used to downsample the pre-processed greening waste scene image and then enter the multi-layer cascade feature layer for feature extraction to obtain four feature maps with different resolutions; the multi-layer cascade feature layer includes the first feature layer C2, the second feature layer C3, the third feature layer C4 and the fourth feature layer C5;

[0099] An efficient hybrid coding module is used to perform cascade feature fusion on the three feature maps of different resolutions from C3 to C5 extracted by the multi-layer cascade feature extraction network, expand the differences between targets, and obtain the fused feature map;

[0100] The decoding and head module is used to reconstruct the output position encoding of the fused feature map, perform upsampling fusion of the three-layer U-shaped structure with the output branches of the C2 to C4 layers, and output the segmentation result.

[0101] As a further implementation, the multi-layer cascade feature extraction network improves the original backbone network of the HRNet network, and replaces all the general convolution blocks in the original backbone network cascade with the depth-separable convolution layer (DSC), which greatly reduces the number of parameters; at the same time, the low-level semantic information of the first feature layer C2 and the second feature layer C3 of the shallow feature layer is retained, that is, the high resolution of the C2 and C3 layers is retained;

[0102] In the third feature layer C4 and the fourth feature layer C5 of the deep feature layer, a hybrid dilated convolution layer (HDC) is used to extract features, expand the receptive field of the convolution and do not introduce the checkerboard effect caused by the dilated convolution.

[0103] Among them, the execution process of each unit can be executed according to the process steps of the greening waste segmentation method based on cascaded feature fusion and hybrid coding in Example 1, and will not be repeated one by one in this embodiment.

[0104] At the same time, the present invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the above-mentioned greening waste segmentation method based on cascaded feature fusion and hybrid coding is implemented.

[0105] At the same time, the present invention also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the above-mentioned greening waste segmentation method based on cascaded feature fusion and hybrid coding.

[0106] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0107] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0108] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0109] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0110] The specific implementation methods described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation method of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A greening waste material segmentation method based on cascade feature fusion and hybrid coding, characterized in that: The method includes: Acquire a greening waste material scene image, and preprocess the greening waste material scene image to obtain a preprocessed greening waste material scene image; Inputting the preprocessed greening waste scene image into the improved semantic segmentation network structure, and outputting the segmentation result through the improved semantic segmentation network structure; The improved semantic segmentation network structure includes a backbone network, an efficient hybrid encoding module, and a decoding and head module; The backbone network adopts a multi-layer cascade feature extraction network, which is used to downsample the pre-processed greening waste scene image and then enter the multi-layer cascade feature layer for feature extraction to obtain four feature maps with different resolutions; the multi-layer cascade feature layer includes a first feature layer C2, a second feature layer C3, a third feature layer C4 and a fourth feature layer C5; An efficient hybrid coding module, used for cascading feature fusion of three feature maps of different resolutions from layers C3 to C5 extracted by the multi-layer cascade feature extraction network, expanding the differences between targets, and obtaining a fused feature map; The decoding and head module is used to reconstruct the output position encoding of the fused feature map, perform upsampling fusion of the three-layer U-shaped structure with the output branches of the C2 to C4 layers, and output the segmentation result.

2. The greening waste material segmentation method based on cascaded feature fusion and hybrid coding according to claim 1 is characterized in that: The multi-layer cascade feature extraction network is an improvement on the original backbone network of the HRNet network. All the general convolution blocks in the cascade of the original backbone network are replaced with depth-separable convolution layers to reduce the number of parameters. At the same time, the low-level semantic information of the first feature layer C2 and the second feature layer C3 of the shallow feature layer is retained, that is, the high resolution of the first feature layer C2 and the second feature layer C3 is retained. In the third feature layer C4 and the fourth feature layer C5 of the deep feature layer, a hybrid dilated convolution layer is used to extract features, expand the receptive field of the convolution and do not introduce the checkerboard effect caused by the hole convolution.

3. The greening waste material segmentation method based on cascaded feature fusion and hybrid coding according to claim 2 is characterized in that: The first feature map of each layer in the multi-layer cascade feature layer is downsampled by a convolution block with a step size of 2 to obtain the first feature map of the next layer; each layer is interactively connected through upsampling and downsampling, and then passes through the stage stage in sequence, each stage stage is connected by multiple basic blocks, and the three stage stages use 2 basic blocks, 4 basic blocks, and 2 basic blocks respectively; The first feature layer C2 and the second feature layer C3 are processed by depth-wise separable convolutional layers, the third feature layer C4 and the fourth feature layer C5 are processed by mixed dilated convolutional layers, and each layer uniformly uses the GELU activation function.

4. The greening waste material segmentation method based on cascaded feature fusion and hybrid coding according to claim 1 is characterized in that: The efficient hybrid coding module includes an inner scale interactive network and a ring network; The inner scale interaction network is an inner scale interaction network designed based on Transformer, which is used to apply the self-attention mechanism to the semantic features of the fourth feature layer C5, capture the differences between objects at similar positions in the image, and provide the subsequent ring network for differential expansion; The ring network includes three fusion module nodes, which are used to directly input the feature maps extracted by the second feature layer C3 and the third feature layer C4, and perform feature fusion of the second feature layer C3, the third feature layer C4, and the fourth feature layer C5 to expand feature differentiation.

5. The greening waste material segmentation method based on cascaded feature fusion and hybrid coding according to claim 1 is characterized in that: The loss function FSL(pt,s(I)) in the decoding and header module is: FSL(pt,s(I))=-αts(I)(1-pt)γlog(pt) Where pt is the true label of the target in the greening waste scene image; 1-pt is the probability of the background in the greening waste scene image; αt is the category weight; γ is the adjustment factor used to balance the loss of easy-to-classify and difficult-to-classify samples; s(I) is the weight factor, and the value of s(I) depends on the size of the object in the greening waste scene image; I is the input greening waste scene image.

6. The greening waste material segmentation method based on cascaded feature fusion and hybrid coding according to claim 5 is characterized in that: The weight factor is determined as follows: s(I)=1-[log(mask_t) / log(mask_i)] Where mask_t represents the foreground mask pixel value; mask_i represents the background mask pixel value.

7. A greening waste material segmentation device based on cascade feature fusion and hybrid coding, characterized in that: The device includes: An acquisition unit, used for acquiring an image of a greening waste material scene; A preprocessing unit, used for preprocessing the greening waste material scene image to obtain a preprocessed greening waste material scene image; A segmentation unit, used for inputting the preprocessed greening waste scene image into the improved semantic segmentation network structure, and outputting the segmentation result through the improved semantic segmentation network structure; The improved semantic segmentation network structure includes a backbone network, an efficient hybrid coding module, and a decoding and head module; The backbone network adopts a multi-layer cascade feature extraction network, which is used to downsample the pre-processed greening waste scene image and then enter the multi-layer cascade feature layer for feature extraction to obtain four feature maps with different resolutions; the multi-layer cascade feature layer includes a first feature layer C2, a second feature layer C3, a third feature layer C4 and a fourth feature layer C5; An efficient hybrid coding module, used for cascading feature fusion of three feature maps of different resolutions from layers C3 to C5 extracted by the multi-layer cascade feature extraction network, expanding the differences between targets, and obtaining a fused feature map; The decoding and head module is used to reconstruct the output position encoding of the fused feature map, perform upsampling fusion of the three-layer U-shaped structure with the output branches of the C2 to C4 layers, and output the segmentation result.

8. The greening waste material segmentation device based on cascade feature fusion and hybrid coding according to claim 7 is characterized in that: The multi-layer cascade feature extraction network is an improvement on the original backbone network of the HRNet network. All the general convolution blocks in the cascade of the original backbone network are replaced with depth-separable convolution layers to reduce the number of parameters. At the same time, the low-level semantic information of the first feature layer C2 and the second feature layer C3 of the shallow feature layer is retained, that is, the high resolution of the first feature layer C2 and the second feature layer C3 is retained. In the third feature layer C4 and the fourth feature layer C5 of the deep feature layer, a hybrid dilated convolution layer is used to extract features, expand the receptive field of the convolution and do not introduce the checkerboard effect caused by the hole convolution.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the greening waste segmentation method based on cascaded feature fusion and hybrid coding as described in any one of claims 1 to 6 is implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the greening waste segmentation method based on cascaded feature fusion and hybrid coding as described in any one of claims 1 to 6 is implemented.

Images