A bilateral semantic segmentation method based on super-resolution

Abstract

The application discloses a kind of bilateral semantic segmentation methods based on super-resolution, based on super-resolution technology and attention mechanism technology, using the semantic segmentation method of double branch, with high flexibility, it can be in low rate field without increasing additional calculation amount, improve the accuracy of image segmentation;The application exchanges the backbone network of main branch to obtain a more advanced semantic segmentation method, and simultaneously fuses the image channel and pixel association to obtain a high-accuracy segmentation result under high resolution;And the segmentation result is sent into the fusion module to guide the main branch to learn segmentation.

Description

Technical Field

[0001] This invention relates to the field of image semantic segmentation, and more particularly to a bilateral semantic segmentation method based on super-resolution. Background Technology

[0002] Over the past decade, machine learning technologies based on deep learning have garnered widespread attention. For example, the autonomous driving technology for new energy vehicles, which has emerged in recent years, is gradually being tested by the general public. The foundation enabling this technology is image segmentation, which gives machines the ability to recognize roads, pedestrians, traffic lights, and road markings. However, the entire deep learning process requires suitable algorithms and a sufficient amount of raw road images for machine learning. This is where image semantic segmentation technology comes in.

[0003] Currently, image semantic segmentation technology is mainly applied in land segmentation, autonomous driving, face segmentation, clothing classification, and precision agriculture. However, each field still faces unresolved challenges. For example, in land segmentation, large-scale publicly available datasets are needed to monitor deforestation, urbanization progress, and urban planning. In autonomous driving, for instance, perception, planning, and execution of commands are required in constantly changing environments, where safety is paramount. This task demands the highest possible accuracy. Semantic segmentation provides free-space information on roads and detects ground markings and traffic signs. However, balancing real-time performance with accuracy remains a significant challenge.

[0004] In recent years, facial recognition technology has been widely used, often requiring smaller devices to operate. This leads to the challenge of quickly and accurately segmenting low-resolution images. Current semantic segmentation techniques, while pursuing accuracy, often require high-resolution image input, resulting in a dramatic increase in computational load, making it difficult or even impossible to compute on mobile devices or low-spec terminals. Conversely, reducing image resolution input to the semantic segmentation model leads to a sharp drop in segmentation accuracy. According to experiments, taking the popular semantic segmentation technology Deeplabv3+ as an example, the accuracy drops to 512... 1024 resolution as input and 448 Using 896 resolution as input, the segmentation accuracy dropped from 70% to 63.2% compared to using 256. With 512 as input, the accuracy is only 56.5%. The drop in accuracy is quite significant. This is clearly insufficient for practical applications requiring precise recognition. Therefore, a segmentation method that can perform fast computation under minimal conditions while improving accuracy is needed. Summary of the Invention

[0005] The purpose of this section is to outline some aspects of embodiments of the present invention and to briefly describe some preferred embodiments. Simplifications or omissions may be made in this section, as well as in the abstract and title of this application, to avoid obscuring the purpose of these documents; however, such simplifications or omissions should not be construed as limiting the scope of the invention.

[0006] In view of the aforementioned existing problems, this invention is proposed. Therefore, this invention provides a super-resolution-based bilateral semantic segmentation method to address the practical problems where image semantic segmentation generates a large amount of additional data computation and is slow in speed. Furthermore, when pursuing high-resolution image input, some mobile devices and low-configuration terminal devices are unable to perform the computation, thus significantly reducing the accuracy of image semantic segmentation.

[0007] To solve the above-mentioned technical problems, the present invention provides the following technical solution:

[0008] This invention provides a bilateral semantic segmentation method based on super-resolution, comprising:

[0009] The acquired images are input into the segmentation grid of the main branch to obtain the corresponding feature maps;

[0010] The feature map is fed into the branch to obtain a new feature map;

[0011] Image channels and pixels are calculated from two sub-branches within the branch, respectively;

[0012] The image channels and pixels are correlated and fused to obtain high-accuracy segmentation results at high resolution;

[0013] The segmentation results are sent to the fusion module, and the fusion results guide the main branch to perform segmentation learning.

[0014] As a preferred embodiment of the super-resolution-based bilateral semantic segmentation method described in this invention, the image is input into the segmentation grid of the main branch to obtain the corresponding feature map, and the steps include:

[0015] The backbone network passes the input image through convolutional layers and then into pooling layers to reduce the number of parameters in the parameter matrix and subsequent convolutional layers, thereby mitigating overfitting. The image is then passed through convolutional layers three more times in the same way to obtain the corresponding feature map.

[0016] As a preferred embodiment of the super-resolution-based bilateral semantic segmentation method of the present invention, wherein: the feature map is fed into the secondary branch and reconstructed to obtain a new feature map, including: reconstructing a high-resolution image in the secondary branch, and then inputting the high-resolution image into the sub-branch to obtain a new feature map.

[0017] As a preferred embodiment of the super-resolution-based bilateral semantic segmentation method described in this invention, the step of constructing a high-resolution image from the branches includes:

[0018] The branch network uses subpixel convolution to effectively reconstruct fine-grained structural information of high-resolution input based on the feature map obtained from the backbone network, which is a single-image super-resolution module.

[0019] As a preferred embodiment of the super-resolution-based bilateral semantic segmentation method of the present invention, the method comprises: dividing the two sub-branches in the branch into an inter-channel attention module and an inter-pixel attention module.

[0020] As a preferred embodiment of the super-resolution-based bilateral semantic segmentation method of the present invention, the inter-channel attention module calculates image channels and pixels, and the steps include:

[0021] The feature map A with the size H×W×C is reshaped to obtain feature map C×N, denoted as B, and multiplied with its transpose, and then subjected to a softmax operation.

[0022] The feature map obtained after the softmax operation with size C×C is denoted as X. After transposing it, it is multiplied by B again, and then the size H×W×C is obtained by reshaping. After multiplying by a coefficient β, the feature map is obtained and denoted as D.

[0023] Add the obtained feature map D to feature map A to obtain the final result;

[0024] The initial value of coefficient β is 0, which is the optimal value obtained through deep learning and debugging.

[0025] As a preferred embodiment of the super-resolution-based bilateral semantic segmentation method of the present invention, wherein: the inter-pixel attention module calculates image channels and pixels, including:

[0026] The feature map A, with dimensions H×W×C, is passed through a convolutional layer to obtain a new feature map B, with dimensions C×H×W. Then, it is reshaped to a size of C×N, where N = H×W. This reshape is then multiplied by the transpose of B, and after a softmax operation, a new feature map S with dimensions N×N is obtained. The sum of each row in S is 1. This can be understood as the pixel weight at position j to the pixel weight at position i, that is, the sum of the weights of all pixels j to a fixed pixel i is 1;

[0027] The obtained feature map S is transposed and multiplied with the feature map B of size C×N after the reshape operation to obtain a feature map of size C×N, and then the size C×H×W is obtained by reshape operation.

[0028] Multiply the obtained feature map with specifications C×H×W by a coefficient α, and then add it to feature map A to obtain the final result;

[0029] The initial value of coefficient α is 0, which is the optimal value obtained through deep learning and debugging.

[0030] As a preferred embodiment of the super-resolution-based bilateral semantic segmentation method described in this invention, the image channel and pixel association fusion to obtain a high-accuracy segmentation result at high resolution includes: summing the two convolutional feature maps generated in the inter-pixel attention module and the inter-channel attention module, and then feeding the generated new feature map into a convolutional layer to obtain the final segmentation result.

[0031] As a preferred embodiment of the super-resolution-based bilateral semantic segmentation method of the present invention, the element-wise summation of the feature maps includes: connecting two convolutional feature maps to a convolutional layer respectively, and then performing an elementwise summation operation on the two convolutional layers.

[0032] As a preferred embodiment of the super-resolution-based bilateral semantic segmentation method described in this invention, the segmentation result is fed into a fusion module, and the fusion result guides the main branch in segmentation learning, including:

[0033] The feature map and the newly generated feature map are fused to give the low-resolution image additional image structure information.

[0034] By using the loss function of feature fusion, we can find the local optimum, match more suitable hyperparameters, guide the semantic segmentation of the main branch and the segmentation learning of the guidance module, and finally achieve the optimal semantic segmentation result.

[0035] The loss function for feature fusion is expressed as follows:

[0036] ;

[0037] ;

[0038] ;

[0039] in, Represents the probability of classification. This represents the super-resolution output image obtained by the super-resolution module, where N represents the total number of pixels in the current image. Represents pixels Given the output of the main branch segmented grid, determine the probability of classifying it as target class y. Indicates the true classification of the current pixel; is a hyperparameter that can be adjusted during the actual segmentation learning process, and is generally set to 0.1; L is the total loss function, which consists of multiple cross-entropy. Cross-entropy is a commonly used loss function for semantic segmentation. This represents the mean square error.

[0040] Compared with existing technologies, the invention offers the following advantages: a significant improvement in computational speed, enhancing semantic segmentation accuracy without increasing computational load; the use of super-resolution to reconstruct features from low-resolution feature maps, resulting in fine-grained information at high resolution, leading to clearer category classification; the cascaded attention mechanism module also improves segmentation accuracy; more importantly, the super-resolution and attention mechanism modules in this branch can be removed during the actual inference phase, requiring only higher computational load during training; thus, under the same basic network architecture, the network architecture incorporating this branch can improve segmentation accuracy by 3% to 5% with low-resolution input. Attached Figure Description

[0041] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. Wherein:

[0042] Figure 1 This is a flowchart of a bilateral semantic segmentation method based on super-resolution according to an embodiment of the present invention;

[0043] Figure 2 This is a schematic diagram of the inter-channel attention module in a bilateral semantic segmentation method based on super-resolution according to an embodiment of the present invention;

[0044] Figure 3 This is a schematic diagram of the inter-pixel attention module in a bilateral semantic segmentation method based on super-resolution according to an embodiment of the present invention;

[0045] Figure 4 This is a segmentation accuracy result diagram of the bilateral semantic segmentation method based on super-resolution according to an embodiment of the present invention;

[0046] Figure 5 This is a schematic diagram of the super-resolution module of the bilateral semantic segmentation method based on super-resolution according to an embodiment of the present invention. Detailed Implementation

[0047] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the protection scope of the present invention.

[0048] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0049] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in different places in this specification does not necessarily refer to the same embodiment, nor is it a single or selective embodiment that is mutually exclusive with other embodiments.

[0050] This invention is described in detail with reference to the schematic diagrams. When detailing the embodiments of this invention, for ease of explanation, the cross-sectional views illustrating the device structure may be partially enlarged, not adhering to the usual scale. Furthermore, the schematic diagrams are merely examples and should not be construed as limiting the scope of protection of this invention. In actual fabrication, the three-dimensional spatial dimensions of length, width, and depth should be included.

[0051] Furthermore, in the description of this invention, it should be noted that the terms "upper," "lower," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. These terms are used solely for the convenience of describing the invention and for simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the invention. In addition, the terms "first," "second," or "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0052] Unless otherwise explicitly specified and limited, the terms "installation," "connection," and "joining" in this invention should be interpreted broadly. For example, they can refer to fixed connections, detachable connections, or integral connections; similarly, they can refer to mechanical connections, electrical connections, or direct connections, or indirect connections through an intermediate medium, or internal connections between two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0053] Example 1

[0054] Reference Figure 1 This is the first embodiment of the present invention, which provides a bilateral semantic segmentation method based on super-resolution, including:

[0055] S1. Input the image into the segmentation grid of the main branch to obtain the corresponding feature map; taking VGG16 as an example, remove the last fully connected layer to obtain a feature map with a size of 14×14×512;

[0056] S2. The obtained feature map is divided into two parts. The first part is sent to the corresponding Decoder module of VGG16, which usually performs bilinear interpolation to the input image size to obtain the semantic segmentation result; the second part is sent to the super-resolution module.

[0057] S3. Through the two sub-branches under super-resolution, the inter-channel attention module and the inter-pixel attention module, the image channels and pixels are calculated respectively;

[0058] S4. The image channels and pixels are associated and fused to obtain a high-accuracy segmentation result at high resolution;

[0059] S5. The segmentation results are fed into the fusion module, and the fusion results are used to guide the semantic segmentation of the main branch and the segmentation learning of the guidance module, so as to achieve the optimal semantic segmentation result.

[0060] By using a two-branch semantic segmentation method, the computational speed of the system can be effectively improved without increasing the computational load.

[0061] Example 2

[0062] Reference Figure 2 and Figure 3 This is the second embodiment of the present invention, which provides a bilateral semantic segmentation method based on super-resolution, including: a structure diagram of the inter-channel attention module and a structure diagram of the inter-pixel attention module;

[0063] Figure 2 The image channel attention module is refined as follows:

[0064] The feature map A with the size H×W×C is reshaped to obtain feature map C×N, denoted as B, and multiplied with its transpose, and then subjected to a softmax operation.

[0065] The feature map obtained after the softmax operation with size C×C is denoted as X. After transposing it, it is multiplied by B again, and then the size H×W×C is obtained by reshaping. After multiplying by a coefficient β, the feature map is obtained and denoted as D.

[0066] Add the obtained feature map D to feature map A to obtain the final result;

[0067] Figure 3 The inter-pixel attention module is refined as follows:

[0068] The feature map A, with dimensions H×W×C, is passed through a convolutional layer to obtain a new feature map B, with dimensions C×H×W. Then, it is reshaped to a size of C×N, where N = H×W. This reshape is then multiplied by the transpose of B, and after a softmax operation, a new feature map S with dimensions N×N is obtained. The sum of each row in S is 1. This can be understood as the pixel weight at position j to the pixel weight at position i, that is, the sum of the weights of all pixels j to a fixed pixel i is 1;

[0069] The obtained feature map S is transposed and multiplied with the feature map B of size C×N after the reshape operation to obtain a feature map of size C×N, and then the size C×H×W is obtained by reshape operation.

[0070] Multiply the obtained feature map with specifications C×H×W by a coefficient α, and then add it to feature map A to obtain the final result;

[0071] The initial values ​​of coefficients β and α are 0, which are the optimal values ​​obtained through deep learning and debugging.

[0072] This invention effectively improves the accuracy of image segmentation by performing a series of operations on the feature map through inter-channel attention modules and inter-pixel attention modules.

[0073] Example 3

[0074] Reference Figure 4 This is the third embodiment of the present invention, which provides a bilateral semantic segmentation method based on super-resolution, including:

[0075] 2000 images were randomly selected from CityScapes as the dataset, and their resolutions were adjusted to 256×512, 320×640, 384×768, 448×896, and 512×1024, respectively. Then, the semantic segmentation method adopted in this invention, taking VGG16 as an example, was input to obtain the MIOU, which is the result of the average intersection-union ratio.

[0076] Example 4

[0077] Reference Figure 5 This is the fourth embodiment of the present invention, which provides a bilateral semantic segmentation method based on super-resolution, including:

[0078] Taking VGG16 as an example, a 14×14×512 feature map is obtained. To achieve a super-resolution of 448×448×3, two convolutional layers are used to obtain a 112×112×12 feature map. These feature maps are then arranged with 12 channels per pixel to obtain a 2×2×3 image. The above operation is performed on the remaining pixels, and these generated images are then stitched together to obtain the reconstructed 448×448×3 super-resolution image. The formula is as follows:

[0079] ;

[0080] in, It involves convolving an H×W×C image into an H×W×C image. A combination of C convolutional layers and sub-pixel convolutional layers It's a low-resolution image. It is a super-resolution image;

[0081] This invention utilizes a super-resolution-based approach to reconstruct features from feature maps generated at low resolution, thereby obtaining fine-grained information at high resolution and making the image clearer.

[0082] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A bilateral semantic segmentation method based on super-resolution, characterized in that, include: The acquired images are input into the segmentation grid of the main branch to obtain the corresponding feature maps; The feature map is fed into the branch to obtain a new feature map; Image channels and pixels are calculated from two sub-branches within the branch, respectively; The image channels and pixels are correlated and fused to obtain high-accuracy segmentation results at high resolution; The segmentation results are sent to the fusion module, and the fusion results are used to guide the main branch in segmentation learning. After the segmentation learning is completed, two sub-branches are deleted, and the main branch is used for inference; The image is input into the segmentation grid of the main branch to obtain the corresponding feature map. The steps include: The backbone network passes the input image through convolutional layers and then into pooling layers to reduce the number of parameters in the parameter matrix and subsequent convolutional layers, thereby mitigating model overfitting. The image is then passed through convolutional layers three more times in the same way to obtain the corresponding feature map. Image channel and pixel correlation fusion yields high-accuracy segmentation results at high resolution, including: The two convolutional feature maps generated in the inter-pixel attention module and the inter-channel attention module are summed element by element, and then the newly generated feature map is fed into a convolutional layer to obtain the final segmentation result. The segmentation results are sent to the fusion module, and the fusion results guide the main branch in segmentation learning, including: The feature map and the newly generated feature map are fused to give the low-resolution image additional image structure information. By using a loss function based on feature fusion, we can find a local optimum, match more suitable hyperparameters, guide the semantic segmentation of the main branch and the segmentation learning of the guidance module, and finally achieve the optimal semantic segmentation result.

2. The bilateral semantic segmentation method based on super-resolution as described in claim 1, characterized in that, The feature map is fed into the slave branch and reconstructed to obtain a new feature map, including: A high-resolution image is reconstructed from the branch, and then the high-resolution image is input into the sub-branch to obtain a new feature map.

3. The bilateral semantic segmentation method based on super-resolution as described in claim 2, characterized in that, The construction of a high-resolution image from the branch includes: The branch network uses subpixel convolution to effectively reconstruct fine-grained structural information of the high-resolution input based on the feature map obtained from the backbone network, which is called a single-image super-resolution module.

4. The bilateral semantic segmentation method based on super-resolution as described in claim 3, characterized in that, The branch is divided into two sub-branches: the sub-branch is divided into an inter-channel attention module and an inter-pixel attention module.

5. The bilateral semantic segmentation method based on super-resolution as described in claim 3 or 4, characterized in that, The inter-channel attention module calculates image channels and pixels, including the following steps: The feature map A with the size H×W×C is reshaped to obtain feature map C×N, denoted as B, and multiplied with its transpose, and then subjected to a softmax operation. The feature map obtained after the softmax operation with size C×C is denoted as X. After transposing it, it is multiplied by B again, and then the size H×W×C is obtained by reshaping. After multiplying by a coefficient β, the feature map is obtained and denoted as D. Add the obtained feature map D to feature map A to obtain the final result; The initial value of coefficient β is 0, which is the optimal value obtained through deep learning and debugging.

6. The bilateral semantic segmentation method based on super-resolution as described in claim 5, characterized in that, The inter-pixel attention module calculates image channels and pixels, including: The feature map A, with dimensions H×W×C, is passed through a convolutional layer to obtain a new feature map B, with dimensions C×H×W. Then, it is reshaped to a size of C×N, where N = H×W. This reshape is then multiplied by the transpose of B, and after a softmax operation, a new feature map S with dimensions N×N is obtained. The sum of each row in S is 1. This can be understood as the pixel weight at position j to the pixel weight at position i, that is, the sum of the weights of all pixels j to a fixed pixel i is 1; The obtained feature map S is transposed and multiplied with the feature map B of size C×N after the reshape operation to obtain a feature map of size C×N, and then the size C×H×W is obtained by reshape operation. Multiply the feature map with the specification C×H×W obtained in the previous step by a coefficient α, and then add it to the feature map A to obtain the final result; The initial value of coefficient α is 0, which is the optimal value obtained through deep learning and debugging.

7. The bilateral semantic segmentation method based on super-resolution as described in claim 6, characterized in that, The feature map is summed element by element, including: Connect each of the two convolutional feature maps to a convolutional layer, and then perform an elementwise summation operation on the two convolutional layers.

Images