Image super-resolution enhanced segmentation method and system based on step-by-step SMFANet and U-Net

Abstract

The invention relates to an image super-resolution enhancement segmentation method and system based on a step-by-step SMFANet and a U-Net. The method aims to solve the problems of detail loss and low calculation efficiency of a traditional model in low-resolution image processing. According to the method, a staged processing flow is adopted, firstly, image super-resolution reconstruction is achieved through a distributed SMFANet module, and then refined segmentation is conducted through optimized U-Net. According to the method, preliminary network pre-training can be carried out on different data sets, the deep learning method is combined, the segmentation accuracy can be ensured without failure when the method faces different practical situations, and meanwhile, a high-quality image after super-resolution reconstruction can be obtained.

Description

Technical Field

[0001] The present application relates to the field of computer vision, and specifically to a step-by-step image processing system that achieves efficient and high-precision segmentation of low-resolution images through the collaborative optimization of independently trained super-resolution networks (SMFANet) and segmentation networks (U-Net), and is suitable for scenarios such as medical imaging and satellite remote sensing. Background Art

[0002] Traditional semantic segmentation networks are prone to losing details and being limited by resolution when performing semantic segmentation. In the encoder-decoder structure, the feature map resolution decreases layer by layer, resulting in insufficient segmentation accuracy for small objects or fine structures (blood vessels in medical images, road edges in satellite images). Traditional methods are also sensitive to noise and low-quality inputs. Low-resolution images or noise interference (such as artifacts in medical CT images and fog and cloud occlusion in remote sensing images) can significantly degrade segmentation performance, making them lacking in robustness. Traditional high-resolution image processing requires extensive computing resources, making them difficult to deploy on mobile devices or in real-time scenarios. Traditional super-resolution networks suffer from large parameter counts and slow inference speeds. SMFANet, by employing depthwise separable convolutions and partially convolutional feedforward networks (PCFNs), reduces the model's parameter count to only 43% of that of traditional CNNs and achieves inference speeds 10 times faster than SwinIR on the NVIDIA V100. Summary of the Invention

[0003] In order to overcome the problems existing in the related art, the present application provides a step-by-step SMFANet and U-Net image super-resolution enhancement segmentation method and system. The system can super-resolve low-resolution images without increasing a large amount of computing resources, so that the super-resolved images can be input into the segmentation network for segmentation.

[0004] This application provides an image super-resolution enhancement and segmentation method and system based on step-by-step SMFANet and U-Net, comprising the following steps:

[0005] SMFANet network training: Use public datasets (such as DIV2K and Flickr2K) to pre-train the network;

[0006] U-Net network training: Use public datasets (such as VOC, MS COCO, etc.) to pre-train the network;

[0007] Super-resolution pre-training of SMFANet on segmentation datasets: Using pre-prepared datasets, SMFANet is trained on specific data.

[0008] SMFANet performs super-resolution: super-resolution is performed on images that need to be segmented;

[0009] U-Net network for semantic segmentation: Use the super-resolution image of the SMFANet network and put it into the pre-trained U-Net network for segmentation;

[0010] In one embodiment, the network architecture includes:

[0011] Feature transfer mechanism: SMFANet and U-Net complete reasoning independently without sharing parameters or gradient transfer.

[0012] In one embodiment, the training strategy optimizes:

[0013] Training is divided into stages: 1. SMFANet pre-training: train SMFANet alone to complete the super-resolution task; 2. U-Net frozen training, fix SMFANet parameters, only train the U-Net network, and use Dice and cross entropy loss functions as loss functions.

[0014] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The above and other objects, features and advantages of the present application will become more apparent through a more detailed description of exemplary embodiments of the present application in conjunction with the accompanying drawings, wherein the same reference numerals generally represent the same components in the exemplary embodiments of the present application.

[0016] Figure 1 1 is a flow chart of an image super-resolution enhancement and segmentation method and system based on step-by-step SMFANet and U-Net as shown in Example 1 of the present application;

[0017] Figure 2 These are the segmentation effect diagrams before and after super-resolution shown in the embodiments of the present application.

[0018] Figure 3 This is a comparison between SMFANet+U-Net and traditional SRCNN+U-Net shown in the embodiment of this application. DETAILED DESCRIPTION

[0019] The preferred embodiments of the present application will be described in more detail below with reference to the accompanying drawings. Although the preferred embodiments of the present application are shown in the accompanying drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments described herein. Instead, these embodiments are provided to make the present application more thorough and complete, and to fully convey the scope of the present application to those skilled in the art.

[0020] The terms used in this application are for the purpose of describing specific embodiments only and are not intended to limit this application. As used in this application and the appended claims, the singular forms "a," "an," "the," and "the" are intended to include the plural forms, unless the context clearly indicates otherwise. It should also be understood that the term "and / or" as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items.

[0021] It should be understood that although the terms "first", "second", "third", etc. may be used in this application to describe various information, this information should not be limited to these terms. These terms are only used to distinguish information of the same type from each other. For example, without departing from the scope of this application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of this application, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0022] Example 1

[0023] Existing super-resolution algorithms, such as SRCNN, suffer from poor accuracy and slow performance. We cannot simultaneously super-resolve images during segmentation, and segmentation of smaller images presents challenges. Therefore, we implement SMFANet super-resolution and then feed the super-resolved image into a U-Net network for semantic segmentation. This allows us to simultaneously generate a super-resolved image and a semantically segmented segmentation map, enabling more practical applications.

[0024] Therefore, this application proposes an image super-resolution enhancement and segmentation method and system based on step-by-step SMFANet and U-Net. By using the SMFANet network and the U-Net network respectively, the two networks are trained, and then the trained networks are applied, so that super-resolution and semantically segmented images can be obtained at the same time.

[0025] The technical solutions of the embodiments of the present application are described in detail below with reference to the accompanying drawings.

[0026] Figure 1 This is a flow chart of the image super-resolution enhancement and segmentation method and system based on the step-by-step SMFANet and U-Net shown in Example 1 of the present application.

[0027] Figure 2 These are the segmentation effect diagrams before and after super-resolution shown in the embodiments of the present application.

[0028] Figure 3 This is a comparison between SMFANet+U-Net and traditional SRCNN+U-Net shown in the embodiment of this application.

[0029] See also Figure 1 and Figure 2 .

[0030] An image super-resolution enhancement and segmentation method and system based on a step-by-step SMFANet and U-Net in an embodiment of the present application includes the following steps:

[0031] S101 uses the DIV2K dataset to pre-train the SMFANet network, so that the SMFANet network can achieve a PSNR of more than 26 and an SSIM of more than 0.73 on the Set5, Set14, B100 and other datasets on the DIV2K dataset.

[0032] S102 uses the VOC dataset to pre-train the U-Net network, so that the U-Net network can achieve an IoU average of more than 0.50 on the VOC dataset.

[0033] S103 pre-trains the SMFANet network on a super-resolution dataset. We process the VOC dataset using the bicubic algorithm and downsample it by x4 times as our pre-training dataset to train the SMFANet network.

[0034] S104 SMFANet network for super-resolution: For our trained SMFANet network, we directly apply it to our non-super-resolved images, perform super-resolution on the images, and save them.

[0035] S105 We put the super-resolution image obtained in S105 into the U-Net network for semantic segmentation.

[0036] The beneficial effects of the embodiments of the present application are as follows: Based on the image super-resolution enhancement and segmentation method based on the step-by-step SMFANet and U-Net, we train SMFANet and U-Net for super-resolution and semantic segmentation respectively by processing different data sets. With the help of these two networks, we can obtain relatively clear super-resolution images and semantic segmentation images of the super-resolution images.

[0037] The embodiments of the present application have been described above. The above description is illustrative and not exhaustive, and is not limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is selected to best explain the principles of the embodiments, their practical applications, or improvements to the technology in the market, or to enable other persons skilled in the art to understand the embodiments disclosed herein.

Claims

1. A method for image semantic segmentation, characterized in that The following steps are involved: 1.1 Super-resolution reconstruction of low-resolution input images is performed through an independent SMFANet network to generate high-resolution images. The SMFANet network uses multi-scale residual blocks and channel-spatial attention modules for feature extraction and optimizes a joint loss function consisting of edge gradient loss and semantic perception loss. 1.2 Input the high-resolution image generated in step 1.1 into an independent U-Net network for semantic segmentation, where the convolution kernel size and downsampling multiple of the U-Net network are adapted to the high-resolution input features; 1.3 During the training phase, SMFANet and U-Net independently optimize parameters in stages, where the semantic perception loss of SMFANet is calculated by the similarity of the feature maps of the pre-trained U-Net network.

2. The image semantic segmentation method according to claim 1, wherein: The semantic perception loss of SMFANet described in 2.1 is achieved in the following way: The encoder output feature map of the pre-trained U-Net is used to calculate the cosine similarity with the super-resolution image feature map; higher loss weights are applied to key areas of the segmentation task (such as lesions in medical images or buildings in satellite images). The edge gradient loss described in 2.2 uses the Sobel operator to extract image gradient information, and constrains the edge consistency between the super-resolution image and the true high-resolution image through the L1 norm.

3. The image semantic segmentation method according to claim 1, wherein: 3.1 The structural optimization of the U-Net network includes: The initial convolution kernel size of the encoder is expanded from 3×3 to 5×5 to adapt to the large-scale features of high-resolution input; the number of downsampling times is reduced to 2 to retain more spatial detail information. 3.2 During the U-Net training phase, dynamic data augmentation is applied to the super-resolved images, including random occlusion and elastic deformation operations.

4. The image semantic segmentation method according to claim 1, wherein: 4.1 The collaborative training process of SMFANet and U-Net includes: The first stage: pre-train SMFANet using a super-resolution dataset (such as DIV2K); the second stage: freeze the U-Net parameters and jointly optimize the semantic perception loss and edge gradient loss of SMFANet on the segmentation dataset; the third stage: freeze the SMFANet parameters and train the U-Net network using super-resolution images.

Images