Image super-resolution enhancement method for image segmentation

Abstract

The invention discloses an image super-resolution enhancement method for image segmentation, which combines super-resolution enhancement and image semantic segmentation by constructing a task-driven joint training framework, trains a segmentation network and a super-resolution enhancement network at the same time, and optimizes the super-resolution enhancement network through mutual collaborative optimization. The segmentation result and the super-resolution enhancement result are enhanced mutually, and image segmentation is completed. The method is different from an existing single training method, the overall performance of super-resolution enhancement and semantic segmentation tasks can be remarkably improved, the quality and semantic consistency of the enhanced image are improved by optimizing the loss functions of super-resolution enhancement and semantic segmentation at the same time and fully combining detail information and semantic information of the image, and the image quality is improved. By introducing a targeted loss function, high-frequency details and context information of the image are effectively integrated, the precision of semantic segmentation is improved while image enhancement is ensured, the explanation ability of the model is improved, and the robustness and adaptability of the model in a complex scene are also remarkably enhanced.

Description

Technical Field

[0001] The invention belongs to the technical field of computer vision and image processing, and in particular relates to an image super-resolution enhancement method oriented to image segmentation. Background Art

[0002] Medical image segmentation helps doctors analyze images more effectively by identifying and depicting different tissues or organ regions. Its application not only helps to quantitatively and qualitatively analyze tissues, but also promotes disease research, especially in cancer radiotherapy, it can accurately locate high-risk organs and protect healthy tissues. At present, automatic segmentation methods represented by deep learning methods (such as convolutional neural networks and Transformer networks) have made significant progress, but they are still difficult to be applied on a large scale in actual clinical environments due to problems such as low-resolution imaging and blurred organ boundaries. To this end, the industry has tried to solve this problem by introducing attention mechanisms, conditional random fields, and super-resolution technologies to improve segmentation accuracy and image quality. The above methods have achieved good performance in quantitative indicators based on human visual perception, but they still have deficiencies in image detail retention and visual quality based on machine visual perception, and cannot provide optimal input for subsequent computer vision applications such as image segmentation. To solve this problem, some practitioners have tried to combine low-level visual enhancement tasks with high-level visual semantic tasks to improve overall performance. However, these methods still follow the path of first independently training a low-level visual enhancement task network (such as image denoising, image super-resolution) or a high-level visual semantic task network (such as image classification, image segmentation), and then using the training results to optimize the performance of another task. Under this paradigm, low-level visual enhancement tasks and high-level visual semantic tasks are relatively separated, and the goals of the two are not unified. Summary of the invention

[0003] To solve the above technical problems, the present invention provides an image super-resolution enhancement method for image segmentation, which organically combines low-level visual enhancement tasks with high-level visual semantic tasks, and optimizes both low-level visual enhancement tasks and high-level visual semantic tasks from a holistic perspective, so that image super-resolution enhancement serves subsequent image segmentation tasks.

[0004] The technical solution adopted by the present invention is: an image super-resolution enhancement method for image segmentation, the specific steps are as follows:

[0005] S1. Build a task-driven joint training framework, including super-resolution enhancement network, segmentation network, and loss function module;

[0006] The super-resolution enhancement network adopts a convolutional neural network structure, that is, an improved network based on U-Net, and the specific structure includes: an encoder and a decoder. The segmentation network is used to perform detailed semantic segmentation operations on the high-resolution image obtained by the super-resolution enhancement network, and the specific structure includes: a spatial path module, a context path module, and a feature fusion module. The loss function module includes: enhancement loss, edge strength loss, segmentation loss, and semantic loss.

[0007] S2, based on the framework constructed in step S1, the segmentation network and the super-resolution enhancement network are trained simultaneously, and the segmentation results and the super-resolution enhancement results are enhanced mutually through mutual collaborative optimization to complete the image segmentation;

[0008] In the framework, each loss function in the loss function module is used to simultaneously enhance the semantic information of super-resolution enhancement and improve the visual quality and evaluation results of the image during the training process.

[0009] Furthermore, in step S1, the super-resolution enhancement network, segmentation network, and loss function module are specifically as follows:

[0010] (1) Super-resolution enhancement network;

[0011] In the super-resolution enhancement network, the encoder gradually extracts low-level features of the image and compresses its spatial information through convolution operations and activation functions, and the decoder restores and enhances the details and resolution of the image through upsampling and deconvolution operations. There is also a jump connection between the encoder and the decoder for transmitting low-level feature information.

[0012] The encoder includes: 1 feature encoding module, 4 attention modules, and 4 downsampling layers; the feature encoding module includes: 4 convolution layers with 3×3 convolution kernels; the attention module includes a maximum pooling and an average pooling, through which the input features are passed to a convolution, a network layer composed of nonlinear ReLU activation and convolution, and then the attention feature information is obtained through convolution and Sigmoid activation function.

[0013] In the encoder, the scaling factor is set to 0.5, and the spatial resolution of the feature map is reduced through the downsampling layer, and each convolution layer generates a feature map of 64 channels.

[0014] The decoder includes: 5 feature decoding modules and 4 sub-pixel convolution modules; the feature decoding module includes: 1 convolution layer with 1×1 convolution kernel and 4 convolution layers with 3×3 convolution kernel; the sub-pixel convolution module generates an output channel number with 4 times the number of channels of the original feature map through the convolution layer, and then reorganizes these channels using the spatial rearrangement operation: by rearranging the pixel values ​​of these additional channels pixel by pixel into the spatial dimension, a resolution of the original input feature map is formed. Figure 2 times the output feature image.

[0015] The super-resolution enhancement network incorporates a boundary processing mechanism, that is, by gradually refining the boundary information of the feature map during the decoder upsampling process, fusing the encoder features with the same resolution as the current feature map through jump connections, combining the attention module with enhanced attention to key boundary areas, and using the sub-pixel convolution module upsampling method to improve the effect of detail enhancement and ensure the clarity and accuracy of the enhancement results.

[0016] (2) Segment the network;

[0017] In the segmentation network, the spatial path module includes: 3 feature extraction layers; the feature extraction layer includes a convolution layer with a 3×3 convolution kernel, each convolution layer is followed by a ReLU activation function, and is equipped with a maximum pooling layer.

[0018] In the segmentation network, the context path module includes: 4 convolution layers with 5×5 convolution kernels and 3 downsampling layers; and each convolution layer with 5×5 convolution kernels is followed by batch normalization and ReLU activation function.

[0019] In the segmentation network, the feature fusion module combines the output features of the spatial path and the context path to form the final segmentation feature map. The feature fusion module uses element-level addition or concatenation to merge the features of the two paths. After fusion, the output features are processed by an additional convolution layer with a 1×1 convolution kernel to finally generate a pixel-level classification result.

[0020] (3) Loss function module;

[0021] In the loss function module, the enhanced loss adopts the pixel-level loss mean square error loss, and the specific expression is as follows:

[0022]

[0023] Among them, H and W represent the size of the enhanced image, which is H×W, and I sr and I hr Represent the image enhanced by the network and the original real image respectively, and (i, j) represents the pixel point of the current image.

[0024] In the loss function module, the edge strength loss definition expression is as follows:

[0025]

[0026] Among them, Δ(·) represents the canny edge strength detection function, and |·| represents the calculation of the absolute value function.

[0027] The edge strength loss and enhancement loss are combined to obtain the content loss function of the super-resolution enhancement network, which is expressed as follows:

[0028] L Con =L Re +αL Edge

[0029] Among them, α represents the proportional hyperparameter required for the edge strength loss function, and α is set to 1.2.

[0030] In the loss function module, the segmentation network outputs the main segmentation result X m and auxiliary segmentation result X a , then the segmentation loss L Seg The expression is as follows:

[0031]

[0032] Among them, L1 represents the main loss function, which is used to supervise the network output of the spatial path, L2 represents the auxiliary loss function, which is used to supervise the output of the context path module, and L Seg Represents the final segmentation loss function, which is used to train the entire segmentation network, n represents the number of all pixels in an image, and cls represents the true category label.

[0033] In the loss function module, the segmentation network has two segmentation results, so the main semantic loss L is defined Sem_main and auxiliary semantic loss L Sem_aux , the expression is as follows:

[0034]

[0035]

[0036] Among them, cls o Represents the one-hot vector conversion of the true category label cls. The semantic loss expression is as follows:

[0037] L Sem =L Sem_main +βL Sem_aux

[0038] Among them, β represents a constant used to balance the main semantic loss and auxiliary semantic loss, which is set to 0.1.

[0039] Finally, the content loss function of the super-resolution enhancement network is combined to obtain the final joint loss function L Joint , the expression is as follows:

[0040] L Joint =L Con +γL Sem

[0041] Among them, γ represents the hyperparameter of the importance of semantic loss. The adjustment formula of γ is as follows:

[0042] γ=iter-1

[0043] Among them, iter represents the iter-th iteration.

[0044] Furthermore, the step S2 is specifically as follows:

[0045] First, the reconstruction target is downsampled to obtain a low-resolution image, which is then input into the super-resolution enhancement network. This low-resolution image is extracted through multi-layer convolution operations in the encoder part to extract the underlying features of the image, and the activation function is used to enhance the nonlinear expression ability to obtain the feature map of the image.

[0046] The obtained feature map then enters the decoder, and the spatial resolution is gradually restored through a 4-layer sub-pixel convolution module. At the same time, the super-resolution enhancement network introduces jump connections at each decoding stage to pass the features of the encoder's corresponding resolution to the decoder, retaining the image edges and texture details. The attention module acts on key areas, namely edges and high-frequency areas, so that the network focuses on the restoration and enhancement of important details. After the above-mentioned hierarchical processing, the super-resolution enhancement network finally converts the low-resolution input image into a clear, detailed, high-resolution output image.

[0047] Among them, based on the loss function module, the enhancement loss and edge strength loss are calculated by enhancing the high-resolution results and the reconstruction target, which constitute the content loss in the super-resolution enhancement network.

[0048] Then, the generated high-resolution image is passed to the segmentation network, and the spatial path module and context path module are used to extract the spatial information features and content information features of the image respectively. After extracting the spatial information features and content information features, the segmentation network effectively combines these two parts of features through the feature fusion module to obtain the segmentation result.

[0049] Among them, based on the loss function module, the segmentation loss is calculated by the output of the segmentation network and the segmentation target, and this loss is used to train the segmentation network.

[0050] Finally, after the segmentation network obtains the segmentation result, it is compared with the segmentation target to calculate the semantic loss. The semantic loss is passed to the super-resolution enhancement network through back propagation, and the semantic loss is combined with the content loss of the super-resolution reconstruction network to finally obtain the joint loss, which is input into the super-resolution enhancement network for optimization training to generate high-resolution images containing more semantic information, improve image quality and detail richness, and complete image super-resolution enhancement.

[0051] Beneficial effects of the present invention: The method of the present invention combines super-resolution enhancement with image semantic segmentation by constructing a task-driven joint training framework, and simultaneously trains the segmentation network and the super-resolution enhancement network. Through mutual collaborative optimization, the segmentation result and the super-resolution enhancement result are mutually enhanced to complete image segmentation. The method of the present invention is different from the existing single training method, and can significantly improve the overall performance of super-resolution enhancement and semantic segmentation tasks. By simultaneously optimizing the loss functions of super-resolution enhancement and semantic segmentation, the detailed information and semantic information of the image are fully combined to improve the quality and semantic consistency of the enhanced image. By introducing a targeted loss function, the high-frequency details and contextual information of the image are effectively integrated to ensure that the accuracy of semantic segmentation is improved while enhancing the image, which not only improves the explanatory power of the model, but also significantly enhances its robustness and adaptability in complex scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 The present invention is a flow chart of an image super-resolution enhancement method for image segmentation.

[0053] Figure 2 4 is a structural diagram of a super-resolution enhancement network in an embodiment of the present invention.

[0054] Figure 3 4 is a structural diagram of a segmented network in an embodiment of the present invention. DETAILED DESCRIPTION

[0055] The method of the present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0056] like Figure 1 As shown, a flow chart of an image super-resolution enhancement method for image segmentation of the present invention, the specific steps are as follows:

[0057] S1. Build a task-driven joint training framework, including super-resolution enhancement network, segmentation network, and loss function module;

[0058] Among them, Figure 2As shown, the super-resolution enhancement network adopts a convolutional neural network structure, that is, an improved network based on U-Net, which is used to process low-resolution images to generate high-resolution images, improve the quality of input images, and thus provide a more accurate and detailed feature basis for subsequent segmentation tasks, thereby improving segmentation performance. The specific structure includes: encoder and decoder. Figure 3 As shown, the segmentation network is used to perform detailed semantic segmentation operations on the high-resolution image obtained by the super-resolution enhancement network, extract the features of the region of interest in the image, and process these features through multi-layer convolution, and gradually perform semantic classification and pixel-level annotation. The specific structure includes: spatial path module, context path module, feature fusion module. The loss function module includes: enhancement loss, edge strength loss, segmentation loss, and semantic loss.

[0059] S2, based on the framework constructed in step S1, the segmentation network and the super-resolution enhancement network are trained simultaneously, and the segmentation results and the super-resolution enhancement results are enhanced mutually through mutual collaborative optimization to complete the image segmentation;

[0060] In the framework, each loss function in the loss function module is used to simultaneously enhance the semantic information of super-resolution enhancement and improve the visual quality and evaluation results of the image during the training process.

[0061] In this embodiment, in step S1, the super-resolution enhancement network, the segmentation network, and the loss function module are specifically as follows:

[0062] (1) Figure 2 As shown, super-resolution enhancement network (low-level vision task);

[0063] In the super-resolution enhancement network, the encoder gradually extracts low-level features of the image and compresses its spatial information (feature extraction) through convolution operations and activation functions, and the decoder restores and enhances the details and resolution of the image (feature reconstruction) through upsampling and deconvolution operations. There is also a jump connection between the encoder and the decoder for transmitting low-level feature information, thereby retaining more details during the reconstruction process.

[0064] The encoder includes: 1 feature encoding module, 4 attention modules, and 4 downsampling layers; the feature encoding module includes: 4 convolution layers with 3×3 convolution kernels; the attention module includes a maximum pooling and an average pooling, through which the input features are passed to a convolution, a network layer composed of nonlinear ReLU activation and convolution, and then the attention feature information is obtained through convolution and Sigmoid activation function.

[0065] In the encoder, the scaling factor is set to 0.5, and the spatial resolution of the feature map is reduced through the downsampling layer to expand the receptive field, thereby improving the network's ability to perceive global information. Each convolutional layer generates a feature map of 64 channels, providing rich low-level features for subsequent processing.

[0066] Compared with the pooling layer used in the existing U-Net, the super-resolution enhancement network uses an attention module instead. The attention module can enhance the network's selective attention to important features, and help the network capture key semantic information more effectively by dynamically adjusting the weights of different areas in the feature map. Compared with the existing pooling operation, the attention module not only avoids the loss of spatial information, but also significantly improves the network's recognition ability in complex scenes, especially when processing images with rich details or containing multiple targets, better retaining subtle features in the image. This improvement enables the network to more accurately process key areas in the super-resolution enhancement process for semantic segmentation tasks.

[0067] The decoder includes: 5 feature decoding modules and 4 sub-pixel convolution modules; the feature decoding module includes: 1 convolution layer with 1×1 convolution kernel and 4 convolution layers with 3×3 convolution kernel; the sub-pixel convolution module generates an output channel number with 4 times the number of channels of the original feature map through the convolution layer, and then reorganizes these channels using the spatial rearrangement operation: by rearranging the pixel values ​​of these additional channels pixel by pixel into the spatial dimension, a resolution of the original input feature map is formed. Figure 2 times the output feature image.

[0068] In order to solve the common problems of blurred boundaries and loss of details in image super-resolution enhancement tasks, the super-resolution enhancement network incorporates a boundary processing mechanism, that is, by gradually refining the boundary information of the feature map during the decoder upsampling process, fusing the encoder features with the same resolution as the current feature map through jump connections, and combining the attention module's enhanced attention to key boundary areas, the network can more accurately enhance the image boundaries and improve the accuracy of the segmentation results. The sub-pixel convolution module upsampling method is also used to further improve the effect of detail enhancement, ensuring the clarity and accuracy of the enhancement results.

[0069] (2) Figure 3 As shown, segmentation network (high-level vision task);

[0070] In the segmentation network, the spatial path module is designed to extract high-resolution spatial features, including: 3 feature extraction layers; the feature extraction layer includes a convolution layer with a 3×3 convolution kernel, each convolution layer is followed by a ReLU activation function, and in order to enhance the expression ability of the network, the spatial path is equipped with a maximum pooling layer; the spatial path uses fewer convolution operations to maintain the details and edge information of the image, and the maximum pooling layer effectively downsamples the feature map while reducing the computational complexity, so that the final output has a higher spatial resolution, so that the segmentation network can effectively capture important features in the image.

[0071] In the segmentation network, the context path module obtains global context information through a deep convolutional neural network, which is composed of multiple convolutional layers and downsampling operations to reduce the spatial resolution of the feature map and increase the number of feature channels, including: 4 convolutional layers with 5×5 convolution kernels and 3 downsampling layers; and each convolutional layer with 5×5 convolution kernels is followed by batch normalization and ReLU activation function to further increase the number of feature channels and enhance the nonlinear expression ability of the network. After the downsampling operation of the downsampling layer, the feature map generated by the context path has a lower spatial resolution, but contains rich semantic information, providing global context support for the segmentation task, making the network more robust when processing complex scenes.

[0072] In the segmentation network, the feature fusion module combines the output features of the spatial path and the context path to form the final segmentation feature map. The feature fusion module uses element-level addition or concatenation to merge the features of the two paths, thereby combining details and semantic information. After fusion, the output features are processed by an additional convolutional layer with a 1×1 convolution kernel (reducing the feature dimension and enhancing the learning ability of the model) to further refine the feature map and finally generate pixel-level classification results.

[0073] (3) Loss function module;

[0074] In the loss function module, in order to improve the visual quality of the enhanced image, the pixel-level loss mean square error loss is used to minimize the difference in pixel values ​​between the enhanced image and the real image. This type of loss helps to restore the details and texture of the image and improve the clarity of the image. The specific expression is as follows:

[0075]

[0076] Among them, H and W represent the size of the enhanced image, which is H×W, and I sr and I hr Represent the image enhanced by the network and the original real image respectively, and (i, j) represents the pixel point of the current image.

[0077] Although the enhancement loss function has ensured that the basic visual quality of the image after enhancement meets the standard, when facing subsequent advanced visual tasks, it pursues not only the optimization of visual quality, but also focuses on the complete preservation and richness of edge details in the image. Pure enhancement loss only focuses on pixel-level error correction, ignoring the complex texture and fine structure information contained in the image, which greatly limits the detail and authenticity of image enhancement. In view of this, in order to significantly improve the image retention and reproduction capabilities at the detail level, this embodiment introduces an edge strength loss term as a supplement, aiming to drive the enhancement model through this loss term to pay more attention to the capture and enhancement of image texture information during the optimization process.

[0078] Then in the loss function module, the edge strength loss definition expression is as follows:

[0079]

[0080] Among them, Δ(·) represents the canny edge strength detection function, and |·| represents the calculation of the absolute value function. The above formula can measure the difference in semantic information of edge strength between the enhanced image and the true image, so that the enhancement network can more accurately capture important semantic boundaries when restoring the image, thereby improving its performance in complex scenes.

[0081] The edge strength loss and enhancement loss are combined to obtain the content loss function of the super-resolution enhancement network, which is expressed as follows:

[0082] L Con =L Re +αL Edge

[0083] Among them, α represents the proportional hyperparameter required for the edge strength loss function, which is set to 1.2. This parameter is specifically used to adjust the sensitivity of the edge strength detection function, which directly affects the weight and emphasis given by the enhancement network to the image edge information.

[0084] In the loss function module, the segmentation network outputs the main segmentation result X m and auxiliary segmentation result X a , then the segmentation loss L Seg The expression is as follows:

[0085]

[0086] Among them, L1 represents the main loss function, which is used to supervise the network output of the spatial path, L2 represents the auxiliary loss function, which is used to supervise the output of the context path module, and L Seg Represents the final segmentation loss function, which is used to train the entire segmentation network. n represents the number of all pixels in an image, and cls represents the true category label. The probability of the sample being in the true category is calculated. The larger the probability, the closer its negative logarithm is to 0; conversely, the smaller the probability, the closer its negative logarithm is to positive infinity.

[0087] In the loss function module, the segmentation network has two segmentation results. In order to effectively and in real time improve the semantic information of the enhanced image of the super-resolution enhancement network, the main semantic loss L is defined Sem main and auxiliary semantic loss L Sem_aux , the expression is as follows:

[0088]

[0089]

[0090] Among them, cls o Represents the one-hot vector transformation of the true category label cls. The main semantic loss and auxiliary semantic loss reflect the semantic information contained in the fused image from different angles. Finally, the semantic loss expression is as follows:

[0091] L Sem =L Sem_main +βL Sem_aux

[0092] Among them, β represents a constant used to balance the main semantic loss and auxiliary semantic loss, which is set to 0.1.

[0093] Combining the content loss function of the super-resolution enhancement network, the final joint loss function L can be obtained Joint , the expression is as follows:

[0094] L Joint =L Con +γL Sem

[0095] Among them, γ represents the hyperparameter of the importance of semantic loss. The design of the joint loss function aims to highlight a core idea: through the adaptive training strategy of integrating low-level and high-level features, as the training process deepens, the adaptability of the segmentation network to the fusion model gradually increases. In this process, the adaptability shows an increasing trend. The adjustment formula of γ is as follows:

[0096] γ=iter-1

[0097] Here, iter represents the iterth iteration. As the number of iterations increases, the segmentation network can more accurately fit the output of the super-resolution enhancement network through continuous learning and adjustment, thereby achieving better enhancement of image details and structures.

[0098] In this embodiment, step S2 is specifically as follows:

[0099] First, the reconstruction target is downsampled to obtain a low-resolution image, which is then input into the super-resolution enhancement network. This low-resolution image is extracted through the multi-layer convolution operation of the encoder part to extract the underlying features of the image, and the activation function is used to enhance the nonlinear expression ability to obtain the feature map of the image. During the encoding process, the spatial resolution of the image is gradually reduced, but the feature expression is enriched layer by layer, laying the foundation for subsequent high-resolution enhancement.

[0100] The obtained feature map then enters the decoder, and the spatial resolution is gradually restored through a 4-layer sub-pixel convolution module. At the same time, the super-resolution enhancement network introduces jump connections at each decoding stage to pass the features of the encoder's corresponding resolution to the decoder, retaining the image edges and texture details. The attention module further acts on key areas, namely edges and high-frequency areas, so that the network focuses on the restoration and enhancement of important details. After the above-mentioned hierarchical processing, the super-resolution enhancement network finally converts the low-resolution input image into a clear, detailed, high-resolution output image.

[0101] Among them, based on the loss function module, the enhancement loss and edge strength loss are calculated by the enhanced high-resolution result (high-resolution output image) and the reconstruction target, which constitute the content loss in the super-resolution enhancement network.

[0102] Then, the generated high-resolution image is passed to the segmentation network, and the spatial information features and content information features of the image are extracted using the spatial path module and the context path module, respectively.

[0103] In the spatial path, the segmentation network focuses on capturing detailed information in the image, such as edges, textures, and shapes, through three convolution operations and three pooling layers, which constitute the spatial information features of the image. These features are essential for maintaining the high resolution and fine structure of the image. The contextual path focuses on extracting the global content information features of the image. It uses deep convolutional layers and larger convolution kernels to expand the receptive field, thereby capturing broader and deeper semantic information in the image. These features help the network understand the overall content and contextual relationships of the image, providing strong support for subsequent semantic segmentation. After extracting the spatial information features and content information features, the segmentation network effectively combines these two parts of features through the feature fusion module to achieve refined segmentation and obtain the segmentation result.

[0104] Among them, based on the loss function module, the segmentation loss is calculated by the output of the segmentation network and the segmentation target (the real label marked in advance), and this loss is used to train the segmentation network.

[0105] Finally, after the segmentation network obtains the segmentation result, the result is compared with the segmentation target to calculate the semantic loss, which is passed to the super-resolution enhancement network by back propagation (the semantic information of the segmentation network is fed back to the enhancement network, prompting it to give priority to enhancing image details during the enhancement process, and providing richer and more accurate input data for subsequent segmentation. This not only improves the semantic expression ability of the image, but also significantly improves the segmentation performance, making the segmentation result more refined and in line with actual application needs), and the semantic loss is combined with the content loss of the super-resolution reconstruction network itself to finally obtain a joint loss, which is input into the super-resolution enhancement network for optimization training. In this embodiment, the Synapse data set is used for training, and the SGD method is used to optimize the loss function. The training batch operation number is set to 32, and the learning rate is set to 0.0001. After 2000 rounds of iterative training, it is considered that the neural network model parameters have been trained. After the training is completed, a high-resolution image containing more semantic information is generated, the image quality and detail richness are improved, and the image super-resolution enhancement is completed.

[0106] In summary, the method of the present invention is different from the existing single training method, and can significantly improve the overall performance of super-resolution enhancement and semantic segmentation tasks. By simultaneously optimizing the loss functions of super-resolution enhancement and semantic segmentation, the detailed information and semantic information of the image are fully combined to improve the quality and semantic consistency of the enhanced image. By introducing a targeted loss function, the high-frequency details and contextual information of the image are effectively integrated to ensure that the accuracy of semantic segmentation is improved while enhancing the image. This not only improves the explanatory power of the model, but also significantly enhances its robustness and adaptability in complex scenarios.

[0107] Those skilled in the art will appreciate that the embodiments described herein are intended to help readers understand the principles of the present invention, and should be understood that the protection scope of the present invention is not limited to such specific statements and embodiments. Those skilled in the art can make various other specific variations and combinations that do not deviate from the essence of the present invention based on the technical revelations disclosed by the present invention, and these variations and combinations are still within the protection scope of the present invention.

Claims

1. An image super-resolution enhancement method for image segmentation, the specific steps are as follows: S1. Build a task-driven joint training framework, including: Super-resolution enhancement network, segmentation network, and loss function module; The super-resolution enhancement network adopts a convolutional neural network structure, that is, an improved network based on U-Net, and the specific structure includes: an encoder and a decoder; the segmentation network is used to perform detailed semantic segmentation operations on the high-resolution image obtained by the super-resolution enhancement network, and the specific structure includes: a spatial path module, a context path module, and a feature fusion module; the loss function module includes: enhancement loss, edge strength loss, segmentation loss, and semantic loss; S2, based on the framework constructed in step S1, the segmentation network and the super-resolution enhancement network are trained simultaneously, and the segmentation results and the super-resolution enhancement results are enhanced mutually through mutual collaborative optimization to complete the image segmentation; In the framework, each loss function in the loss function module is used to simultaneously enhance the semantic information of super-resolution enhancement and improve the visual quality and evaluation results of the image during the training process.

2. The image super-resolution enhancement method for image segmentation according to claim 1, characterized in that: In step S1, the super-resolution enhancement network, segmentation network, and loss function module are specifically as follows: (1) Super-resolution enhancement network; In the super-resolution enhancement network, the encoder gradually extracts low-level features of the image and compresses its spatial information through convolution operations and activation functions, and the decoder restores and enhances the details and resolution of the image through upsampling and deconvolution operations. There is also a jump connection between the encoder and the decoder for transmitting low-level feature information. The encoder includes: 1 feature encoding module, 4 attention modules, and 4 downsampling layers; the feature encoding module includes: 4 convolution layers with 3×3 convolution kernels; the attention module includes a maximum pooling and an average pooling, through which the input features are passed to a convolution, a network layer composed of nonlinear ReLU activation and convolution, and then the attention feature information is obtained through convolution and Sigmoid activation function; In the encoder, the scaling factor is set to 0.5, and the spatial resolution of the feature map is reduced by the downsampling layer, and each convolution layer generates a feature map of 64 channels; The decoder includes: 5 feature decoding modules and 4 sub-pixel convolution modules; the feature decoding module includes: 1 convolution layer with 1×1 convolution kernel and 4 convolution layers with 3×3 convolution kernel; the sub-pixel convolution module generates an output channel number with 4 times the number of channels of the original feature map through the convolution layer, and then reorganizes these channels using a spatial rearrangement operation: by rearranging the pixel values ​​of these additional channels pixel by pixel to the spatial dimension, an output feature image with a resolution twice that of the original input feature map is formed; The super-resolution enhancement network incorporates a boundary processing mechanism, that is, by gradually refining the boundary information of the feature map during the decoder upsampling process, fusing the encoder features with the same resolution as the current feature map through a skip connection, combining the attention module to strengthen the focus on key boundary areas, and also by using the sub-pixel convolution module upsampling method, the effect of detail enhancement is improved to ensure the clarity and accuracy of the enhancement result; (2) Segmentation network; In the segmentation network, the spatial path module includes: 3 feature extraction layers; the feature extraction layer includes a convolution layer with a 3×3 convolution kernel, each convolution layer is followed by a ReLU activation function, and is equipped with a maximum pooling layer; In the segmentation network, the context path module includes: 4 convolution layers with 5×5 convolution kernels and 3 downsampling layers; and each convolution layer with 5×5 convolution kernels is followed by batch normalization and ReLU activation function; In the segmentation network, the feature fusion module combines the output features of the spatial path and the context path to form a final segmentation feature map; the feature fusion module merges the features of the two paths by element-level addition or concatenation. After fusion, the output features are processed by an additional convolution layer with a 1×1 convolution kernel to finally generate a pixel-level classification result; (3) Loss function module; In the loss function module, the enhanced loss adopts the pixel-level loss mean square error loss, and the specific expression is as follows: Among them, H and W represent the size of the enhanced image, which is H×W, and I sr and I hr Represent the image enhanced by the network and the original real image respectively, (i, j) represents the pixel point of the current image; In the loss function module, the edge strength loss definition expression is as follows: Where Δ(·) represents the canny edge strength detection function, and |·| represents the calculation of the absolute value function; The edge strength loss and enhancement loss are combined to obtain the content loss function of the super-resolution enhancement network, which is expressed as follows: L Con =L Re +αL Edge Among them, α represents the proportional hyperparameter required for the edge strength loss function, and α is set to 1.2; In the loss function module, the segmentation network outputs the main segmentation result X m and auxiliary segmentation result X a , then the segmentation loss L Seg The expression is as follows: Among them, L1 represents the main loss function, which is used to supervise the network output of the spatial path, L2 represents the auxiliary loss function, which is used to supervise the output of the context path module, and L Seg represents the final segmentation loss function, which is used to train the entire segmentation network, n represents the number of all pixels in an image, and cls represents the true category label; In the loss function module, the segmentation network has two segmentation results, so the main semantic loss L is defined Sem_main and auxiliary semantic loss L Sem_aux , the expression is as follows: Among them, cls o Represents the one-hot vector conversion of the true category label cls; the semantic loss expression is as follows: L Sem =L Sem_main +βL Sem_aux Among them, β represents a constant used to balance the main semantic loss and auxiliary semantic loss, which is set to 0.1; Finally, the content loss function of the super-resolution enhancement network is combined to obtain the final joint loss function L Joint , the expression is as follows: THE Joint =L Con +γL Sem Among them, γ represents the hyperparameter of the importance of semantic loss; the adjustment formula of γ is as follows: γ=iter-1 Among them, iter represents the iter-th iteration.

3. The image super-resolution enhancement method for image segmentation according to claim 1, characterized in that: The step S2 is specifically as follows: First, the reconstruction target is downsampled to obtain a low-resolution image, which is then input into the super-resolution enhancement network. This low-resolution image is subjected to multi-layer convolution operations in the encoder part to extract the underlying features of the image, and the activation function is used to enhance the nonlinear expression capability to obtain the feature map of the image. The obtained feature map then enters the decoder and gradually restores the spatial resolution through a 4-layer sub-pixel convolution module; At the same time, the super-resolution enhancement network introduces skip connections at each decoding stage to pass the features of the encoder's corresponding resolution to the decoder, retaining the image edge and texture details. The attention module acts on key areas, namely edges and high-frequency areas, so that the network can focus on restoring and enhancing important details. After the above-mentioned hierarchical processing, the super-resolution enhancement network finally converts the low-resolution input image into a clear, detailed, high-resolution output image. Among them, based on the loss function module, the enhancement loss and edge strength loss are calculated by the enhanced high-resolution result and the reconstruction target, which constitute the content loss in the super-resolution enhancement network; Then, the generated high-resolution image is passed to the segmentation network, and the spatial information features and content information features of the image are extracted using the spatial path module and the context path module respectively. After extracting the spatial information features and content information features, the segmentation network effectively combines these two parts of features through the feature fusion module to obtain the segmentation result; Among them, based on the loss function module, the segmentation loss is calculated by the output of the segmentation network and the segmentation target, and the loss is used to train the segmentation network; Finally, after the segmentation network obtains the segmentation result, it is compared with the segmentation target to calculate the semantic loss. The semantic loss is passed to the super-resolution enhancement network through back propagation, and the semantic loss is combined with the content loss of the super-resolution reconstruction network to finally obtain the joint loss, which is input into the super-resolution enhancement network for optimization training to generate high-resolution images containing more semantic information, improve image quality and detail richness, and complete image super-resolution enhancement.

Images