Lightweight segmentation method based on parallel multi-scale details and semantic coding

Abstract

The application discloses a new multiscale space detail and semantic background extraction method, and further constructs an efficient image semantic segmentation method suitable for resource-limited devices through a parallel multiscale detail and semantic coding network.The method mainly comprises the following steps: training a light-weight semantic segmentation neural network model; taking an original image as input, and obtaining a mask with a highlighted region of interest through the network model trained in the previous step.The method can segment the region of interest from a city scene image, and is an effective image semantic segmentation method.

Description

Technical Field

[0001] The present invention relates to image semantic segmentation technology, and in particular to a lightweight semantic segmentation method based on parallel multi-scale detail and semantic coding, belonging to the field of digital image processing. Background Art

[0002] Semantic segmentation is a key task in computer vision. Raw data (such as images) is converted into masks that highlight regions of interest. Semantic segmentation is a crucial foundation for scene understanding in computer vision, and demand for this task is growing in both academia and industry. With the advancement of deep learning technology, semantic segmentation has also gradually become part of the deep learning landscape.

[0003] The human brain processes visual information using a hierarchical organization of neurons, with low-level features such as edges and textures processed in early layers and higher-level features such as object components and spatial relationships processed in later layers. This hierarchical organization enables the brain to integrate abstract information from multiple scales and levels, resulting in efficient and powerful visual perception. The ability to leverage multi-scale information, from low-level feature extraction to high-level semantic interpretation, enables humans to quickly and accurately identify objects and scenes in a variety of contexts and environments. Similarly, in computer vision tasks such as semantic segmentation, researchers have been attempting to mimic the human visual system's ability to leverage multi-scale information in visual signals. Effective modeling of multi-scale information is crucial for nearly all computer vision tasks. In semantic segmentation, we require local details to determine object boundaries, information at the object level to identify object categories, and sometimes even information from the broader surrounding environment for robust judgment. Designing hierarchical networks to extract multi-scale information has been a persistent concern in the computer vision community. Summary of the Invention

[0004] The purpose of this invention is to rethink the extraction of multi-scale spatial details and semantic background, and to construct an efficient image semantic segmentation method suitable for resource-limited devices through parallel multi-scale detail and semantic encoding networks.

[0005] The lightweight semantic segmentation based on Parallel Multi-Scale Detail and Semantic Encoding Network (PMDSEN) proposed in this paper mainly includes the following steps:

[0006] (1) Train a lightweight semantic segmentation neural network model;

[0007] (2) The original image is used as input and the network model trained in step (1) is used to obtain a mask with a prominent region of interest. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Figure 1 This is a principle block diagram of the lightweight segmentation method based on parallel multi-scale detail and semantic encoding (Parallel Multi-Scale Detail and Semantic Encoding Network, PMDSEN) of the present invention.

[0009] Figure 2 It is the principle block diagram of the multi-scale semantic extractor (MSSE) part in the principle block diagram of the lightweight segmentation method based on parallel multi-scale detail and semantic encoding (Parallel Multi-Scale Detail and Semantic Encoding Network, PMDSEN) of the present invention.

[0010] Figure 3 This is the principle block diagram of the dynamic weighted feature fusion (DWFF) part in the principle block diagram of the lightweight segmentation method based on parallel multi-scale detail and semantic encoding (Parallel Multi-Scale Detail and Semantic Encoding Network, PMDSEN) of the present invention.

[0011] Figure 4 This is a comparison of the segmentation results of the present invention and two other methods on images from the validation set. From left to right: input image, segmentation label, method 1, method 2, and the segmentation result of the present invention.

[0012] Figure 5 This is a comparison of the segmentation results of the present invention and two other methods on images from the test set. From left to right: input image, method 1, method 2, and the segmentation result of the present invention. (Note: The test set images do not have segmentation labels) DETAILED DESCRIPTION

[0013] The present invention will be further described below in conjunction with the accompanying drawings:

[0014] Figure 1 In [1], lightweight semantic segmentation based on parallel multi-scale details and semantic encoding can be divided into the following steps:

[0015] (1) Train a lightweight semantic segmentation neural network model;

[0016] (2) Taking the original image as input, the network model trained in step (1) is used to obtain a mask with the region of interest highlighted;

[0017] Specifically, in step (1), the trained convolutional neural network model is as shown in Figure 1 The network structure mainly includes three parts: hierarchical pyramid module (HPM), multi-scale semantic extractor (MSSE), and semantic injection and spatial recovery unit (SISR). First, the input image is subjected to hierarchical feature extraction by HPM to obtain multi-scale features of the image. Then, MSSE is used to further extract detailed spatial information and macro semantic information. Finally, SISR uses dynamic weight feature fusion (DWFF) method to gradually fuse shallow and deep features, thereby decoding an image mask with the region of interest highlighted.

[0018] For any image input The output is CLASS represents the number of segmentation categories. The overall implementation of the network model is based on the following process:

[0019] {F SHALLOW ,F MIDDLE ,F DEEP}=f HPM (I) (1)

[0020] F DEEP ′=f MSSE (F DEEP ) (2)

[0021] I SEG =f SISR ({F SHALLOW ,F MIDDLE ,F DEEP ′}) (3)

[0022] where f HPM (·) represents the HPM extractor, and the output of the hierarchical multi-scale feature is represented as {F SHALLOW ,F MIDDLE ,F DEEP}; f MSSE (·) represents the MSSE extractor, and the output of the feature is represented as F DEEP ′; f SISR (·) represents the SISR unit, which gradually decodes the final segmentation mapping in combination with the input hierarchical features.

[0023] The implementation of the core part HPM is based on a Parallel Multi-scale Detail and Semantic Encoding (PMSDSE) based on the following procedure for a given input

[0024]

[0025] where Linear(·) denotes a linear layer, MSLRI(·) denotes a Multi-scale Large-range Context Information (MSLRI) extractor, LDI(·) denotes a Local Detail Information (LDI) extractor, Concat(a, b) denotes a concatenation a, b operation, and Shuffle(·) denotes a channel shuffle operation.

[0026] where the implementation of MSLRI is based on the following procedure.

[0027]

[0028]

[0029] F MSLRI = Concat(Linear(Concat(y1, y2, y3, y4)), X') (7)

[0030] where the implementation of LDI is based on the following procedure. The MSLRI unit is responsible for coarse and complex large-range relation extraction. Semantic segmentation needs more large-range surrounding information to obtain the whole object level information, understand the overall scene structure and the relationship between objects, and realize accurate object class recognition and robust segmentation. In addition, local detail information is very important for distinguishing objects and backgrounds and identifying fine-grained details such as edges, corners, and texture information. Therefore, the present application uses the LDI branch to enrich and detailed local information. Specifically, a Local Detail Extraction Layer (LDEL) is used to capture subtle details at a fine scale. This procedure can be represented as:

[0031] F LDI = LDEL(X') (8)

[0032] The implementation of the core part MSSE is based on the following procedure. For a given input feature

[0033]

[0034] F MSSE = Y + Linear(Concat(y1, y2, y3, y4)) (10)

[0035] The implementation of the core part of SISR is based on the following process. It mainly contains two modules: PMSDSE and Dynamic Weight Feature Fusion (DWFF). The features F DEEP are progressively upsampled and fused with the hierarchical features from HPM, gradually recovering the output features with spatially fine and semantically explicit, i.e., the segmentation mask with the salient region highlighted. The implementation of DWFF is based on the following process: assuming the features from HPM and MSSE are denoted as Z1and Z2,

[0036]

[0037]

[0038] The Dynamic Weight Feature Fusion (DWFF) method can effectively fuse shallow and deep features, and the specific implementation details are as follows: the DWFF strategy dynamically adjusts the weight distribution through two operations, namely fusion and selection. (a) The fusion operation combines the information of the two data streams to generate global feature descriptors. To achieve this goal, we first combine the two feature streams from the shallow Z1and deep features Z2using the element sum method, i.e., Z = Z1+ Z2, and then apply global average pooling in the spatial dimension to calculate the channel-level statistical features Then, a compressed feature representation is generated using a channel compression convolutional layer which is fed into two parallel channel nonlinear expansion layers, resulting in feature descriptors and (b) The selection operation applies a SoftMax function along v1and v2to obtain attention activations, which adaptively recalibrate the dual feature streams before their aggregation. Specifically, we use attention activations to allocate adaptive weighted contributions from the shallow Z1and deep features Z2, and the entire feature recalibration and aggregation process can be defined as: where z1and z2represent the adaptively allocated weights.

[0039] Implementation of up and down sampling layers. In the process of down-sampling the features, a considerable amount of spatial information will be lost. In order to make up for the loss of spatial details to some extent, on the one hand, we down-sample the input image to match the size of the features, and use convolution operation to fuse it, so as to introduce rich original spatial details. On the other hand, we use spatial convolution with a span of 2 to double the number of channels to extract rich detailed representations, and use maximum and average pooling to extract the most prominent features to create a more general representation that can identify interest areas and objects as well as edges and textures. In order to restore the spatial resolution, we use transpose convolution in the feature up-sampling process.

[0040] In step (2), the urban scene image is input into the network model trained in step (1) to obtain a final semantic segmentation result.

[0041] In order to better illustrate the effectiveness of the present application, comparative experiments were carried out on the semantic segmentation dataset Cityscapes. In the experiment, two typical lightweight semantic segmentation algorithms were selected as comparative methods.

[0042] The comparative lightweight semantic segmentation algorithm is:

[0043] Method 1: The method proposed by Yu et al., reference literature "Yu C, Gao C, Wang J, et al. Bisenet v2: Bilateral network with guided aggregation for real-time semantic segmentation [J]. International Journal of Computer Vision, 2021, 129: 3051-3068."

[0044] Method 2: The method proposed by Fan et al., reference literature "Fan M, Lai S, Huang J, et al. Rethinking bisenet for real-time semantic segmentation [C] / / Proceedings of the IEEE / CVF conference on computer vision and pattern recognition. 2021: 9716-9725."

[0045] The contents of the comparative experiment are as follows:

[0046] Experiment 1: The commonly used semantic segmentation dataset Cityscapes is respectively segmented by method 1 to method 2 and the present application. The semantic segmentation results are respectively as follows:Figure 2 、 Figure 3 and Figure 4 The objective evaluation parameters for the segmentation results are shown in Table 1. The objective evaluation parameter used is mIoU (Mean Intersection over Union). A higher value indicates better segmentation results. The number of parameters and computational complexity measure the complexity of the model's storage burden. Lower values ​​indicate a lower burden and greater ease of deployment.

[0047] Table 1

[0048]

[0049] Obviously, with lower parameter amount and complexity, our method (parameter amount / computational amount@mIoU: 0.92M / 10.2G@73.2%) achieves better segmentation results than recent methods, achieving a better balance between accuracy and complexity.

[0050] Compared with other methods, the present invention can simultaneously extract rich and detailed local information, as well as coarse and complex large-scale features. Therefore, the present invention achieves better segmentation results for large objects, such as cars, and detailed objects, such as traffic lights and telephone poles.

[0051] In summary, compared with the comparative method, the reconstruction results of the present invention have great advantages in both subjective and objective evaluation. Therefore, the present invention is an effective lightweight image segmentation method.

Claims

1. Parallel Multi-Scale Detail and Semantic Encoding Network (PMDSEN) is a lightweight segmentation network based on parallel multi-scale detail and semantic encoding. Train a lightweight semantic segmentation convolutional neural network model, then use the original image as input and pass it through the convolutional neural network model trained in step 1 to obtain a mask with a prominent region of interest; The convolutional neural network mainly consists of three parts: Hierarchical Pyramid Module (HPM), Multi-scale Semantic Extractor (MSSE), and Semantic Injection and Spatial Recovery (SISR). First, the HPM performs hierarchical feature extraction on the input image to obtain multi-scale image features. Then, MSSE further extracts detailed spatial information and macro-semantic information. Finally, SISR uses the Dynamic Weight Feature Fusion (DWFF) method to gradually fuse shallow and deep features to decode an image mask with a prominent region of interest. For any image input The output is H, W, and CLASS represent the height, width, and number of segmentation categories of the image, respectively. The overall network implementation is based on the following process: {F SHALLOW ,F MIDDLE ,F DEEP }=f HPM (I) (1) F DEEP ′=f MSSE (F DEEP ) (2) I SEG =f SISR ({F SHALLOW ,F MIDDLE ,F DEEP ′}) (3) Among them, f HPM (·) represents the HPM extractor, whose output hierarchical multi-scale features are represented as {F SHALLOW ,F MIDDLE ,F DEEP };f MSSE (·) represents the MSSE extractor, and its output feature is represented as F DEEP ′;f SISR (·) represents the SISR unit, which combines the input hierarchical features to gradually decode the final segmentation map; The core part of HPM is based on the following process: The core part of HPM is based on the Parallel Multi-scale Detail and Semantic Encoding (PMSDSE) unit. PMSDSE generates output features Where Linear(·) represents a linear layer, MSLRI(·) represents a multi-scale long-range context information (MSLRI) extractor, LDI(·) represents a local detail information (LDI) extractor, Concat(a,b) represents the concatenation of a and b, and Shuffle(·) represents a channel shuffle operation. The implementation of MSLRI(·) is based on the following process: F MSLRI =Concat(Linear(Concat(y1,y2,y3,y4)),X′) (7) Among them, X′ and F MSLRI They represent the input and output features of MSLRI(·), FAL(·) represents the feature aggregation layer (FAL), which can be implemented by depthwise separable convolution; The implementation of LDI(·) is based on the following process: F LDI =LDEL(X′) (8) Among them, X′ and F LDI denote the input and output features of LDI(·), LDEL(·) denotes the local detail encoding layer (LDEL), which can be implemented by depthwise separable convolution; Concat(·,·) denotes the feature concatenation operation; The core part of MSSE is implemented based on the following process: for a given input feature MSSE generates output features F MSSE =Y+Linear(Concat(y1,y2,y3,y4)) (10) Among them, PL i (·) represents the Pool-Linear (PL) operation, and ↑ represents the upsampling operation; The core part of SISR is implemented based on the following process: It mainly consists of two modules: PMSDSE and DWFF. The features from MSSE are gradually upsampled and fused with the hierarchical features from HPM, gradually recovering the output features with fine spatial resolution and clear semantics, that is, the segmentation mask with prominent region of interest. The implementation of DWFF is based on the following process: Assume that the features from HPM and MSSE are denoted as Z1 and Z2 respectively. Among them, GAP(·) represents the global average pooling (GAP) operation, Split(·) represents the feature separation operation along the channel, and Represents the weight of adaptive allocation, Z1, Z2 represent the shallow and deep dual feature flows, Z SISR Represents the final dynamic weighted fusion features.

Images