Image semantic segmentation method, device and medium based on multi-scale strip-shaped dilated convolution

By adopting multi-scale bar-hole convolutional neural network in semantic segmentation, combining bilinear interpolation method and bilinear pooling path, the problem of poor segmentation effect of strip objects in the existing technology is solved, and a higher level is achieved The segmentation accuracy and stability reduce the impact of irrelevant information.

CN114581657BActive Publication Date: 2024-12-03XIANGTAN UNIV
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202111166760.6
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-09-30
Publication Date
2024-12-03
Estimated Expiration
2041-09-30

AI Technical Summary

Technical Problem

In the semantic segmentation of the prior art, square pooling leads to poor segmentation effect of strip-shaped objects, and single-scale strip pooling cannot effectively deal with multi-scale objects, resulting in mixed irrelevant information and polluting the segmentation effect.

Method used

Design a multi-scale bar cavity convolution neural network, through the backbone network feature extraction module and the multi-scale bar cavity convolution module, use bilinear interpolation method to restore the original image size, combining horizontal and vertical bar pooling paths Carry out information fusion to reduce the amount of calculation and increase the receptive field, and reduce irrelevant information pollution.

Benefits of technology

It improves the accuracy of semantic segmentation and model stability, prevents overfitting, enhances the segmentation ability of multi-scale objects, reduces the impact of irrelevant information, and improves the segmentation effect.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114581657B_ABST
    Figure CN114581657B_ABST
Patent Text Reader

Abstract

The present invention discloses an image semantic segmentation method, device and medium based on multi-scale strip hole convolution. The present invention forms a network model by adding a multi-scale strip hole convolution module after the backbone network feature extraction module, so as to further extract multi-scale information from the feature map output by each layer belonging to the middle layer. The labeled image data set is expanded and then the network model is trained to obtain the final semantic segmentation result. In view of the problem of poor segmentation effect caused by strip-shaped segmentation target objects and objects of different scales in the image, the present invention adopts a strip hole convolution module and extracts multi-scale information from different layers, and fuses different paths to further improve the segmentation accuracy, thereby solving the problem of serious loss of segmentation accuracy of multi-scale target segmentation objects.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the field of deep learning and machine vision, and specifically relates to an image semantic segmentation method, device and medium based on multi-scale strip-shaped dilated convolution. Background Art

[0002] The purpose of semantic segmentation is to segment objects of different categories in an image and give semantic information so that different types of objects can be distinguished in the image. It can be understood as a pixel-level classification task. Semantic segmentation can be applied to many scenarios, such as autonomous driving, clothing classification, medical image analysis, etc.

[0003] With the introduction of the fully convolutional network FCN, semantic segmentation has set off a wave of convolution and pooling operations, but the process of deconvolution and upsampling operations will cause the image to lose a lot of detail information and context information. Many articles propose to consider multi-scale information for semantic segmentation. In order to obtain global context information, PSPNet proposes a pyramid pooling module to aggregate context information based on different regions. This module contains information between different scales and different regions. In order to take into account both the environmental information and detail information of the image, the classic model U-Net network is proposed. Downsampling is used to gradually display environmental information, and the upsampling process combines information from each layer to restore detail information, thereby improving segmentation accuracy. However, it is still essentially an inheritance of the structure of the Encoder-Deconder model. The emergence of the Deeplab series brought about the method of dilated convolution. The convolution layer introduced a new parameter of "dilation rate", which expanded the receptive field of the model, enabling it to perceive feature information in a larger range. In order to solve the problem that different objects in the same image are treated equally and that some targets are not significant enough due to factors such as lighting and occlusion, DANet proposed a dual attention mechanism, which introduces global information from the space and channel respectively, and adaptively integrates the dependencies between local features and global features. Considering that the importance of each channel varies, the SENet network proposed a compression and excitation module, which assigns weights to each channel of the feature map through a two-layer fully connected structure while reducing the amount of calculation, selectively emphasizes the interdependent channel map, integrates the related features between all channels, and obtains the dependency relationship between channels. In order to improve the segmentation accuracy of strip objects and capture the anisotropic context flexibility that is widely present in real scenes, SPNet proposed a strip pooling module, which is different from the traditional square pooling window. Strip pooling can capture the long-distance relationship of isolated areas and reduce the interference information pollution of irrelevant areas.

[0004] The existing technology has some defects: square pooling leads to poor segmentation effect for a large number of strip-shaped objects in the picture. There are often many strip-shaped target segmentation objects in scene segmentation, and the square window will bring a lot of polluted information. For example, in the network model Deeplab, although multi-scale information is used, the segmentation effect of strip-shaped objects is not good because they are all square convolution pooling windows. On the other hand, the network model SPNet uses strip pooling to improve the segmentation effect of strip-shaped objects to a certain extent, but due to the single strip pooling scale, it is impossible to balance the various scales of objects in the picture. At the same time, due to the single-scale strip pooling, irrelevant information will be mixed into the pooling window, which will cause certain irrelevant information pollution to the segmentation effect. Summary of the invention

[0005] In view of the above defects or improvement needs of the prior art, the purpose of the present invention is to provide a semantic segmentation method, device and medium of a multi-scale fused strip hole convolutional neural network that solves the problem that the existing convolutional neural network suitable for semantic segmentation has poor segmentation effect on multi-scale strip objects.

[0006] In order to achieve the above technical purpose, the technical solution of the present invention is:

[0007] A method for image semantic segmentation based on a multi-scale strip-shaped hole convolutional neural network comprises the following steps:

[0008] Step 1: Build a neural network model framework;

[0009] The neural network model framework includes a backbone network feature extraction module and a multi-scale strip hole convolution module; the output feature maps of multiple different layers belonging to the middle layer of the backbone network feature extraction module are respectively input into the strip hole convolution module to further extract multi-scale information; then a one-dimensional convolution is performed to adjust the number of channels, and then the bilinear interpolation method is used to expand the size to restore the original image size, and finally the final result is obtained by fusion;

[0010] Step 2: Preprocess the dataset of labeled images to expand the number of images and unify the size, and then train the neural network model based on these images until the training is completed;

[0011] Step 3: Input the image to be recognized into the trained neural network model to obtain the recognition result.

[0012] In the described method, in the described step 1, the backbone network feature extraction module is based on the existing Backbone backbone network.

[0013] The method, in step 1, the strip-shaped hole convolution module includes:

[0014] Two parallel paths, fusion operation layer and multiplication operation;

[0015] The two parallel paths both include a strip pooling layer, a convolution layer, a strip hole convolution layer and an expand layer.

[0016] One path is a horizontal pooling path, in which the window of the strip pooling layer is 1×W, so as to transform the original input tensor of size C×H×W into C×H×1, where C is the number of channels, H is the height of the image, and W is the width of the image. The convolution layer includes a convolution kernel of size 3×1, and the strip dilated convolution layer includes a dilated convolution kernel of size 3×1 and a dilation rate of 2. The expand layer restores the size of the output tensor to be consistent with the input tensor;

[0017] The other path is the vertical pooling path, in which the window of the strip pooling layer is H×1, the size of the dilated convolution kernel in the strip dilated convolution layer is 1×3, and the rest is consistent with the horizontal pooling path;

[0018] The input tensor enters two parallel paths for processing respectively, and the output tensors of the two paths are then fused and multiplied to obtain the final output result.

[0019] In the method, the fusion operation layer in the strip-shaped hole convolution module adds the elements at the same position of the output tensors of two parallel paths, and the multiplication operation layer multiplies the fused output tensor with the elements at the same position of the original input tensor.

[0020] In the method, in step 2, the preprocessing includes rotating, scaling, cropping and flipping the image to obtain different images to expand the number of images in the data set, and then unifying the image size by scaling and cropping.

[0021] In the method, the neural network model is trained in step 4 by using an adaptive learning method to train the neural network model, and the cross entropy loss is used as the loss function; wherein the mathematical expression of the adaptive learning is:

[0022]

[0023] Where current_rate is the current learning rate, initial_rate is the initial learning rate, current_step is the current iteration number, max_step is the maximum iteration number, and power is the momentum;

[0024] The expression of the cross entropy loss is:

[0025]

[0026] Where y is the true value, that is, the label of the image. is the predicted value, and log is the logarithm.

[0027] An electronic device, comprising:

[0028] one or more processors;

[0029] a storage device for storing one or more programs,

[0030] When the one or more programs are executed by the one or more processors, the one or more processors implement the described method.

[0031] A computer readable medium stores a computer program, which implements the method when executed by a processor.

[0032] The technical effects of the present invention are:

[0033] (1) The image preprocessing operation adopted by the present invention not only improves the accuracy of the model, but also enhances the stability of the model and can prevent overfitting.

[0034] (2) To address the problem that a single-scale strip pooling window has poor segmentation effect on multi-scale strip segmentation targets, the present invention designs a strip hole convolution module to be added after multiple intermediate layers of the backbone network, which is beneficial to the segmentation of multi-scale objects and improves the accuracy of semantic segmentation.

[0035] (3) The strip-shaped atrous convolution operation in the strip-shaped atrous convolution module designed by the present invention increases the receptive field without increasing the amount of calculation, and reduces the pollution of irrelevant information through the atrous convolution operation, thereby improving the segmentation effect of the model.

[0036] (4) The advantage of using multi-path fusion is that it can integrate information from multiple aspects and highlight important information.

[0037] (5) In the strip-shaped atrous convolution module of the present invention, the result is multiplied by the input tensor after the strip-shaped atrous convolution operation layer to further highlight the key information, which is beneficial to improving the segmentation effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 This is a structural diagram of the neural network model of the present invention.

[0039] Figure 2 It is the overall flow chart of the present invention.

[0040] Figure 3 Schematic diagram of the strip-shaped hole convolution module of the present invention.

[0041] Figure 4This is a diagram of the segmentation effect of adding the strip-shaped hole convolution module described in the present invention to different layers of resnet-101.

[0042] Figure 5 (a) is the original image of the four images.

[0043] Figure 5 (b) is the label map of 4 images.

[0044] Figure 5 (c) shows the segmentation effect of 4 images based on SPNet.

[0045] Figure 5 (d) is a diagram showing the segmentation effects of 4 images under the neural network framework of the present invention. DETAILED DESCRIPTION

[0046] The present invention will be further described below in conjunction with the accompanying drawings.

[0047] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail in conjunction with the accompanying drawings and embodiments. It should be understood that the specific examples described herein are only used to explain the present invention and are not used to limit the present invention.

[0048] like Figure 1 The figure shows the structure diagram of the neural network model of this embodiment. The backbone network used in this embodiment is resnet-101. In actual application, backbone networks such as resnet-50, mobilenet, xception, etc. can also be used.

[0049] exist Figure 1 The neural network model structure diagram shown includes the operations of each layer of resnet-101, the strip hole convolution module, the one-dimensional convolution layer, and the upsampling module.

[0050] like Figure 2 The flowchart of this embodiment is shown, which mainly includes the following steps: 1) downloading the data set and preprocessing the data; 2) building the neural network model; 3) setting and adjusting the network model parameters; 4) analyzing the segmentation effect diagram according to the cross entropy loss function and the MIOU value. The specific steps of this embodiment are as follows:

[0051] S1. Prepare the data set and preprocess the images in the data set.

[0052] The data set in this embodiment is all marked pictures with labels. The preprocessing process first includes rotation, scaling, cropping and flipping, etc., and the number of pictures is expanded by these changes. Then, pictures of uniform size are obtained by scaling, cropping and other means. Since the original picture data set is greatly expanded through preprocessing, these large numbers of pictures are used to make the model contact as many exceptions as possible during training, so as to continuously correct the model. Therefore, preprocessing not only enhances the stability of the model, but also prevents the model from overfitting.

[0053] S2. Build the neural network model framework required for this embodiment. The network framework of this embodiment is based on the resnet-101 backbone network as the backbone network feature extraction module, and then add a strip hole convolution module. In this embodiment, the output feature maps of the second, third, and fourth layers of the middle layer of resnet-101 are input into the strip hole convolution module, so that multi-scale information can be extracted based on these three layers. The output tensors of these three layers through the hole convolution module are adjusted through a one-dimensional convolution to adjust the number of channels, and then the bilinear interpolation method is used to upsample the size of the expansion tensor, so that the original image size is restored, and the three tensors are fused to obtain the final result.

[0054] Among them, the strip hole convolution module in the constructed neural network model framework sends the input tensor x to two parallel pooling paths respectively. One of the horizontal pooling paths sends the input tensor to the strip pooling layer with a window of 1×W. After the strip pooling layer, the size of the input tensor x becomes C×H×1. Then there is a convolution operation with a convolution kernel size of 3×1, which is used to adjust the features of the current position and the adjacent position. The second is the strip hole convolution for re-extracting the features. The hole convolution kernel size is 3×1 and the hole rate is 2. This operation can remove some redundant information obtained after the strip pooling layer. At this time, the size of the tensor is still C×H×1, and then the output tensor size is changed to the original C×H×W through the expand operation. The other parallel vertical pooling path also uses the same operation, but the strip pooling window is H×1, so that the output tensor becomes C×1×W, and the corresponding strip hole convolution The convolution kernel size is changed to 1×3. Finally, the corresponding elements of the output results of the two parallel paths are added together to comprehensively consider the global information.

[0055] The above operation can be expressed as follows:

[0056] y4=S[f(ac(σ(sp hw (x))))]

[0057] Among them, sp hwrepresents the strip pooling operation, σ represents the 3×1 (or 1×3) one-dimensional convolution operation, and ac represents the 3×1 (or 1×3) strip hole convolution, that is, the vertical pooling path uses the same operation as the horizontal pooling path, except that σ is a 1×3 one-dimensional convolution and ac is a 1×3 strip hole convolution. f represents the expand operation. S represents the sum of the corresponding positions of each element of the two parallel paths of the submodule.

[0058] Taking into account the correlation between the corresponding position and the adjacent position, this embodiment multiplies the obtained result with the original input tensor x to obtain the output tensor z.

[0059]

[0060] The output of the strip hole convolution module is tensor z. Represents the multiplication of corresponding elements.

[0061] S3. Setting and adjusting network model parameters.

[0062] The GPU used in the present invention is two 11G 1080Ti.

[0063] In the early stage of neural network model training, the optimal point is far away from the extreme point, and the learning rate is generally set to a larger value. A larger learning rate can quickly approach the extreme point. In the middle and late stages of training, because it is close to the extreme point and the model is about to converge, the learning rate is reduced accordingly. The decay rate of the Poly learning strategy is mainly controlled by the value of power. When power = 1, the learning rate is a constant value. When power < 1, the learning rate decay rate changes from slow to fast. When power > 1, the learning rate decay rate changes from fast to slow. This embodiment uses an adaptive learning rate method to train the neural network model of the present invention. The expression of the adaptive learning rate is:

[0064]

[0065] Among them, current_rate is the current learning rate, initial_rate is the initial learning rate, current_step is the current iteration number, max_step is the maximum iteration number, power is the momentum, set to 0.9, and the initial learning rate is set to 5e-4.

[0066] This embodiment uses a simple cross entropy loss as the loss function, and its mathematical expression is:

[0067]

[0068] Where y is the true value, is the predicted value.

[0069] S4. This embodiment also evaluates the segmentation effect image using the mean pixel overlap (mIoU) ratio. The ratio of the intersection and union of the predicted result and the true value of each category is summed and averaged.

[0070]

[0071] Among them, i represents the true value, j represents the predicted value, and p ij It represents the probability of predicting i as j.

[0072] According to the cross entropy loss value and mIoU value, the segmentation effect diagram is analyzed.

[0073] According to an embodiment of the present invention, the present invention also provides an electronic device and a computer-readable medium.

[0074] Electronic equipment includes:

[0075] one or more processors;

[0076] a storage device for storing one or more programs,

[0077] When the one or more programs are executed by the one or more processors, the one or more processors implement the aforementioned method.

[0078] In specific use, users can interact with servers, which are also electronic devices, through electronic devices as terminal devices and based on the network to achieve functions such as receiving or sending messages. Terminal devices are generally various electronic devices equipped with display devices and used based on human-computer interfaces, including but not limited to smart phones, tablet computers, laptops and desktop computers. Various specific application software can be installed on the terminal devices as needed, including but not limited to web browser software, instant messaging software, social platform software, shopping software, etc.

[0079] The server is a network service end used to provide various services, such as a backend server that provides corresponding semantic segmentation services for images received from terminal devices, so as to implement semantic segmentation of received images and return the final semantic segmentation results to the terminal device.

[0080] The semantic segmentation method provided in this embodiment is generally executed by a server. In actual application, if necessary conditions are met, the terminal device can also directly execute the semantic segmentation.

[0081] Similarly, the computer-readable medium of the present invention stores a computer program thereon, and when the program is executed by a processor, the semantic segmentation method of the embodiment of the present invention is implemented.

[0082] Figure 1This is a structural diagram of the neural network model of the present invention. Figure 2 This is the overall flow chart of this embodiment. Figure 3 Schematic diagram of the strip-shaped hole convolution module of this embodiment; Figure 4 The segmentation effect diagram of adding the strip hole convolution module described in this embodiment to different layers of resnet-101; Figure 5 (a) is the original image of the four images; Figure 5 (b) is the label map of 4 pictures; Figure 5 (c) shows the segmentation effect of 4 images based on SPNet; Figure 5 (d) is a diagram showing the segmentation effects of 4 images under the neural network framework of the present invention.

[0083] It can be seen from the segmentation effect diagram that the segmentation effect diagram produced by the neural network framework used in the present invention has higher accuracy, indicating that the network structure adopted in the present invention can further improve the segmentation effect.

[0084] The image preprocessing operation adopted in this embodiment can prevent the model from overfitting and improve the adaptability of the model. In order to segment the multi-scale target objects in the image, the strip hole convolution module is added to the last three layers of resnet-101 to extract multi-scale information; the strip hole convolution module designed by the present invention can increase the receptive field without increasing the amount of calculation, and reduce the unnecessary pollution caused by the pixels being mixed with the surrounding irrelevant information; in order to solve the problem of serious loss of edge detail information due to the increase in the number of network layers, the second layer of information of resnet-101 is fused to retain the edge detail information.

[0085] The above description is only a preferred specific implementation manner of the present invention, but the scope of protection of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent substitutions or changes according to the technical scheme and concept of the present invention within the technical scope disclosed by the present invention, and they should be covered by the scope of protection of the present invention.

Claims

1. An image semantic segmentation method based on a multi-scale strip-shaped hole convolutional network, characterized in that: The following steps are involved: Step 1: Build a neural network model framework; The neural network model framework includes a backbone network feature extraction module and a multi-scale strip hole convolution module; the output feature maps of multiple different layers belonging to the middle layer of the backbone network feature extraction module are respectively input into the strip hole convolution module to further extract multi-scale information; then a one-dimensional convolution is performed to adjust the number of channels, and then the bilinear interpolation method is used to expand the size to restore the original image size, and finally the final result is obtained by fusion; Step 2: Preprocess the dataset of labeled images to expand the number of images and unify the size, and then train the neural network model based on these images until the training is completed; Step 3: Input the image to be recognized into the trained neural network model to obtain the recognition result; In the step 1, the strip-shaped hole convolution module includes: Two parallel paths, a fusion operation layer and a multiplication operation layer; The two parallel paths both include a strip pooling layer, a convolution layer, a strip hole convolution layer and an expand layer; One path is a horizontal pooling path, in which the window of the strip pooling layer is 1×W, so as to change the original input tensor of size C×H×W to C×H×1, where C is the number of channels, H is the height of the image, and W is the width of the image. The convolution layer includes a convolution kernel of size 3×1, and the strip dilated convolution layer includes a dilated convolution kernel of size 3×1 and a dilation rate of 2. The expand layer restores the size of the output tensor to be consistent with the input tensor; The other path is the vertical pooling path, in which the window of the strip pooling layer is H×1, the size of the dilated convolution kernel in the strip dilated convolution layer is 1×3, and the rest is consistent with the horizontal pooling path; The input tensor enters two parallel paths for processing at the same time, and the output tensors of the two paths are then fused and multiplied to obtain the final output result. The fusion operation layer in the strip-shaped hole convolution module adds the elements at the same position of the output tensors of the two parallel paths, and the multiplication operation layer multiplies the fused output tensor by the elements at the same position of the original input tensor.

2. The method according to claim 1, characterized in that In the step 1, the backbone network feature extraction module is based on the existing Backbone backbone network.

3. The method according to claim 1, characterized in that In the step 2, the preprocessing includes rotating, scaling, cropping and flipping the image to obtain different images to expand the number of images in the data set, and then unifying the image size by scaling and cropping.

4. The method according to claim 1, characterized in that: In step 2, the neural network model is trained by using an adaptive learning method, and the cross entropy loss is used as the loss function; wherein the mathematical expression of the adaptive learning is: Where current_rate is the current learning rate, initial_rate is the initial learning rate, current_step is the current iteration number, max_step is the maximum iteration number, and power is the momentum; The expression of the cross entropy loss is: Where y is the true value, that is, the label of the image. is the predicted value, and log is the logarithm.

5. An electronic device, characterized in that: include: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 4.

6. A computer readable medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 4 is implemented.