Lightweight segmentation method based on parallel multi-scale detail and semantic coding

Abstract

The invention discloses a novel method for extracting multi-scale space details and semantic backgrounds, and further constructs an efficient image semantic segmentation method suitable for resource limited equipment through a parallel multi-scale detail and semantic coding network. The method mainly comprises the following steps: training a lightweight semantic segmentation neural network model; and taking an original image as input, and obtaining a mask with a prominent region of interest through the network model trained in the previous step. According to the method, the region of interest can be segmented from the city scene image, and the method is an effective image semantic segmentation method.

Description

Technical Field

[0001] The present invention relates to image semantic segmentation technology, and in particular to a lightweight semantic segmentation method based on parallel multi-scale detail and semantic coding, and belongs to the field of digital image processing. Background Art

[0002] Semantic segmentation is an important task in computer vision, where raw data (such as images) are converted into masks with highlighted regions of interest. Semantic segmentation is an important foundation for scene understanding in computer vision, and the demand for this task is growing in both academia and industry. With the development of deep learning technology, semantic segmentation has gradually fallen into the vortex of deep learning.

[0003] When processing visual information, the human brain uses a hierarchical organization of neurons, where low-level features such as edges and textures are processed in early layers, and high-level features such as object components and spatial relationships are processed in later layers. This hierarchical organization enables the brain to integrate abstract information from multiple scales and levels, thereby achieving efficient and powerful visual perception. The ability to use multi-scale information from low-level feature extraction to high-level semantic interpretation enables humans to quickly and accurately identify objects and scenes in various backgrounds and environments. Similarly, in computer vision tasks such as semantic segmentation, researchers have been trying to mimic the ability of the human visual system to use multi-scale information in visual signals. It is well known that for almost all computer vision tasks, effective modeling of multi-scale information is crucial. In semantic segmentation tasks, we need local detail information to determine object boundaries, information at the entire object level to identify object categories, and sometimes even a larger range of surrounding environment information for robust judgment. How to design a hierarchical network to achieve multi-scale information extraction has always been a concern in the computer vision community. Summary of the invention

[0004] The purpose of this invention is to rethink the extraction of multi-scale spatial details and semantic background, and to construct an efficient image semantic segmentation method suitable for resource-limited devices through parallel multi-scale detail and semantic encoding networks.

[0005] The lightweight semantic segmentation based on parallel multi-scale detail and semantic encoding (Parallel Multi-Scale Detail and Semantic Encoding Network, PMDSEN) proposed in the present invention mainly includes the following steps:

[0006] (1) Train a lightweight semantic segmentation neural network model;

[0007] (2) Taking the original image as input, the network model trained in step (1) is used to obtain a mask with a prominent region of interest. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Figure 1 It is a principle block diagram of the lightweight segmentation method based on parallel multi-scale detail and semantic encoding (Parallel Multi-Scale Detail and Semantic Encoding Network, PMDSEN) of the present invention.

[0009] Figure 2 It is a principle block diagram of the multi-scale semantic extractor (MSSE) part in the principle block diagram of the lightweight segmentation based on parallel multi-scale detail and semantic encoding (Parallel Multi-Scale Detail and Semantic Encoding Network, PMDSEN) method of the present invention.

[0010] Figure 3 It is a principle block diagram of the dynamic weighted feature fusion (DWFF) part in the principle block diagram of the lightweight segmentation based on parallel multi-scale detail and semantic encoding (Parallel Multi-Scale Detail and Semantic Encoding Network, PMDSEN) method of the present invention.

[0011] Figure 4 This is a comparison of the segmentation results of the present invention and the other two methods on the validation set images. From left to right: input image, segmentation label, method one, method two and the segmentation result of the present invention.

[0012] Figure 5 This is a comparison of the segmentation results of the present invention and the other two methods on the test set images. From left to right: input image, method one, method two and the segmentation result of the present invention. (Note: the test set images have no segmentation labels) DETAILED DESCRIPTION

[0013] The present invention will be further described below in conjunction with the accompanying drawings:

[0014] Figure 1 In the paper, lightweight semantic segmentation based on parallel multi-scale details and semantic encoding can be divided into the following steps:

[0015] (1) Train a lightweight semantic segmentation neural network model;

[0016] (2) Taking the original image as input, the network model trained in step (1) is used to obtain a mask with a prominent region of interest;

[0017] Specifically, in step (1), the convolutional neural network model to be trained is as follows: Figure 1 As shown in the figure. The network structure mainly includes three parts: Hierarchical Pyramid Module (HPM), Multi-scale Semantic Extractor (MSSE) and Semantic Injection and Spatial Recovery (SISR). First, HPM is used to extract hierarchical features of the input image to obtain multi-scale features of the image. Then, MSSE is used to further extract detailed spatial information and macro semantic information. Finally, SISR uses the Dynamic Weight Feature Fusion (DWFF) method to gradually fuse shallow and deep features, thereby decoding an image mask with a prominent region of interest.

[0018] For any image input The output is CLASS represents the number of segmentation categories. The overall implementation of the network model is based on the following process:

[0019] {F SHALLOW ,F MIDDLE ,F DEEP} = f HPM (I) (1)

[0020] F DEEP ′=f MSSE (F DEEP ) (2)

[0021] I SEG =f SISR ({F SHALLOW ,F MIDDLE ,F DEEP ′}) (3)

[0022] Among them, f HPM (·) represents the HPM extractor, and its output hierarchical multi-scale features are represented as {F SHALLOW ,F MIDDLE ,F DEEP}; f MSSE (·) represents the MSSE extractor, and its output feature is represented as F DEEP ′; f SISR (·) denotes a SISR unit, which combines the input hierarchical features to gradually decode the final segmentation map.

[0023] The core part of HPM is implemented based on the Parallel Multi-scale Detail and Semantic Encoding (PMSDSE) unit. Based on the following process, for a given input

[0024]

[0025] Among them, Linear(·) represents the linear layer, MSLRI(·) represents the Multi-scale Large-range Context Information (MSLRI) extractor, LDI(·) represents the Local Detail Information (LDI) extractor, Concat(a,b) represents the concatenation of a and b, and Shuffle(·) represents the channel shuffle operation.

[0026] Among them, the implementation of MSLRI is based on the following process.

[0027]

[0028]

[0029] F MSLRI =Concat(Linear(Concat(y1,y2,y3,y4)),X′) (7)

[0030] Among them, the implementation of LDI is based on the following process. The MSLRI unit is responsible for coarse and complex large-scale relationship extraction. Semantic segmentation requires a larger range of surrounding environment information to obtain information at the entire object level, understand the overall scene structure and the relationship between objects, and achieve accurate object category recognition and robust segmentation. In addition, local detail information is very important for distinguishing objects from backgrounds and identifying fine-grained details, such as edges, corners, and texture information. Therefore, the present invention uses LDI branches to enrich and detail local information. Specifically, the Local Detail Extraction Layer (LDEL) is used to capture subtle details on a fine scale. This process can be expressed as:

[0031] F LDI =LDEL(X′) (8)

[0032] The implementation of the core part MSSE is based on the following process. For a given input feature

[0033]

[0034] F MSSE =Y+Linear(Concat(y1,y2,y3,y4)) (10)

[0035] The core part of SISR is implemented based on the following process. It mainly consists of two modules: PMSDSE and Dynamic Weight Feature Fusion (DWFF). The feature F from MSSE DEEP ′ is gradually upsampled and fused with the hierarchical features from HPM, gradually recovering the output features with fine spatial resolution and clear semantics, that is, the segmentation mask with prominent regions of interest. The implementation of DWFF is based on the following process: Assume that the features from HPM and MSSE are represented as Z1 and Z2 respectively,

[0036]

[0037]

[0038] The Dynamic Weight Feature Fusion (DWFF) method can effectively fuse shallow and deep features. The specific implementation details are as follows: The DWFF strategy dynamically adjusts the weight distribution through two operations, namely fusion and selection. (a) The fusion operation combines the information of the two data streams to generate a global feature descriptor. To achieve this goal, we first combine the dual feature streams from the shallow layer Z1 and the deep feature Z2 using the element-wise addition method, that is, Z = Z1 + Z2, and then in the spatial dimension Global average pooling is applied to calculate channel-level statistical features Then, a channel-squeezed convolutional layer is used to generate a compressed feature representation Send it to two parallel channel nonlinear expansion layers to obtain the feature descriptor and (b) The selection operation applies the SoftMax function along v1 and v2 to obtain the attention activation, and adaptively recalibrates the dual feature streams before aggregating them. Specifically, we use the attention activation to assign adaptive weighted contributions from the shallow layer Z1 and the deep feature Z2. The entire feature recalibration and aggregation process can be defined as: Among them, z1 and z2 represent the weights of adaptive allocation.

[0039] Implementation of upsampling and downsampling layers. In the process of downsampling features, a considerable amount of spatial information is lost. To compensate for the lost spatial details to some extent, on the one hand, we downsample the input image to match the size of the features and fuse them using convolution operations to introduce rich original spatial details. On the other hand, we use spatial convolution with a stride of 2 to double the number of channels to extract rich detailed representations, and use max and average pooling to extract the most prominent features to create a more general representation that can identify regions of interest and objects as well as edges and textures, respectively. To restore spatial resolution, we use transposed convolutions during feature upsampling.

[0040] In the step (2), the urban scene image is input into the network model trained in the step (1) to obtain the final semantic segmentation result.

[0041] In order to better illustrate the effectiveness of the present invention, a comparative experiment was conducted on the semantic segmentation dataset Cityscapes. In the experiment, two typical lightweight semantic segmentation algorithms were selected as comparison methods.

[0042] The lightweight semantic segmentation algorithms compared are:

[0043] Method 1: The method proposed by Yu et al., reference "Yu C, Gao C, Wang J, et al. Bisenet v2: Bilateral network with guided aggregation for real-time semantic segmentation [J]. International Journal of Computer Vision, 2021, 129: 3051-3068."

[0044] Method 2: The method proposed by Fan et al., reference "Fan M, Lai S, Huang J, et al. Rethinkingbisenet for real-time semantic segmentation[C] / / Proceedings of the IEEE / CVFconference on computer vision and pattern recognition.2021:9716-9725."

[0045] The contents of the comparative experiment are as follows:

[0046] Experiment 1: Semantic segmentation experiments were performed on the commonly used semantic segmentation dataset Cityscapes using methods 1, 2, and the present invention. The semantic segmentation results are shown in Figure 1. Figure 2 , Figure 3 and Figure 4 The objective evaluation parameters of the segmentation results are shown in Table 1. The objective evaluation parameter uses mIoU (Mean Intersection over Union). The higher the parameter value, the better the segmentation effect. The parameter quantity and calculation amount are used to measure the complexity of the storage burden of the model. The lower the value, the smaller the burden and the more conducive to actual deployment.

[0047] Table 1

[0048]

[0049] Obviously, with lower parameter amount and complexity, the present invention (parameter amount / computation amount @mIoU: 0.92M / 10.2G@73.2%) obtains better segmentation results than recent methods, achieving a better balance between accuracy and complexity.

[0050] Compared with other methods, the present invention can simultaneously extract rich and detailed local information, as well as coarse and complex large-scale features. Therefore, the present invention achieves better segmentation results for large objects, such as cars, and detailed objects, such as traffic lights and telephone poles.

[0051] In summary, compared with the comparison method, the reconstruction result of the present invention has great advantages in both subjective and objective evaluation. Therefore, the present invention is an effective lightweight image segmentation method.

Claims

1. A lightweight segmentation method based on Parallel Multi-Scale Detail and Semantic Encoding Network (PMDSEN), which is characterized by The following steps are involved: Step 1: Train a lightweight semantic segmentation neural network model; Step 2: Take the original image as input and pass it through the network model trained in step 1 to obtain a mask with a prominent region of interest.

2. The lightweight segmentation method based on parallel multi-scale detail and semantic coding according to claim 1 is characterized in that The convolutional neural network model described in step 1; the network model mainly includes three parts: Hierarchical Pyramid Module (HPM), Multi-scale Semantic Extractor (MSSE) and Semantic Injection and Spatial Recovery (SISR); first, the input image is subjected to hierarchical feature extraction by HPM to obtain multi-scale features of the image; then, detailed spatial information and macroscopic semantic information are further extracted by MSSE; finally, the shallow and deep features are gradually fused by SISR using the Dynamic Weight Feature Fusion (DWFF) method, thereby decoding an image mask with a prominent region of interest; For any image input The output is CLASS represents the number of segmentation categories; the overall implementation of the network model is based on the following process: {F SHALLOW ,F MIDDLE ,F DEEP }=f HPM (I) (1) F DEEP ′=f MSSE (F DEEP ) (2) I SEG =f SISR ({F SHALLOW ,F MIDDLE ,F DEEP ′}) (3) Among them, f HPM (·) represents the HPM extractor, and its output hierarchical multi-scale features are represented as {F SHALLOW ,F MIDDLE ,F DEEP }; f MSSE (·) represents the MSSE extractor, and its output feature is represented as F DEEP ′; f SISR (·) denotes a SISR unit, which combines the input hierarchical features to gradually decode the final segmentation map.

3. The lightweight segmentation method based on parallel multi-scale detail and semantic coding according to claim 1 is characterized in that The convolutional neural network model described in step 1; the core part HPM is implemented based on the following process: The core part HPM is implemented based on the parallel multi-scale detail and semantic encoding unit (Parallel Multi-scale Detail andSemantic Encoding, PMSDSE) based on the following process, for a given input Among them, Linear(·) represents the linear layer, MSLRI(·) represents the Multi-scale Large-range Context Information (MSLRI) extractor, LDI(·) represents the Local Detail Information (LDI) extractor, Concat(a,b) represents the concatenation of a and b, and Shuffle(·) represents the channel shuffle operation; Among them, the implementation of MSLRI is based on the following process: F MSLRI =Concat(Linear(Concat(y1,y2,y3,y4)),X′) (7) The implementation of LDI is based on the following process: F LDI =LDEL(X′) (8) FAL(·) and LDEL(·) represent the feature aggregation layer (FAL) and the local detail encoding layer (LDEL), respectively, which can be implemented by depthwise separable convolution; Concat(·,·) represents the feature concatenation operation.

4. The lightweight segmentation method based on parallel multi-scale detail and semantic coding according to claim 1, characterized in that The convolutional neural network model described in step 1; the implementation of the core part MSSE is based on the following process: The implementation of the core part MSSE is based on the following process: For a given input feature F MSSE =Y+Linear(Concat(y1,y2,y3,y4)) (10) Among them, PL i (·) represents the Pool-Linear (PL) operation, and ↑ represents the upsampling operation.

5. The lightweight segmentation method based on parallel multi-scale detail and semantic coding according to claim 1, characterized in that The convolutional neural network model described in step 1; the core part of SISR is implemented based on the following process: The core part of SISR is implemented based on the following process: It mainly consists of two modules: PMSDSE and DWFF, the feature F from MSSE DEEP ′ is gradually upsampled and fused with the hierarchical features from HPM, gradually recovering the output features with fine spatial resolution and clear semantics, that is, the segmentation mask with prominent regions of interest. The implementation of DWFF is based on the following process: Assume that the features from HPM and MSSE are represented as Z1 and Z2 respectively, in, and Represents the weight of adaptive allocation; Z1, Z2 represent the shallow and deep dual feature flows; Z SISR Represents the final dynamic weighted fusion features.

Images