Transform-based wound image enhancement method

Abstract

The invention provides a Transform-based wound image enhancement method, which comprises the following steps of: simulating and adjusting related parameters in a camera internal ISP (Internet Service Provider) process in a mode of fully supervised training and dynamic inquiry learning, introducing a local mapping prediction image pixel-level multiplication graph and addition graph with an attention mechanism, mapping an image from a standard RGB (Red, Green, Blue) space to a camera RGB space, and carrying out image enhancement. According to the invention, image enhancement is carried out through an end-to-end Transform network, the problems of image degradation and the like caused by low illumination are solved, image enhancement is carried out on the problem of non-uniform illumination of a wound image caused by a shooting process, and the problems of image distortion and the like caused by non-uniform illumination are solved.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and in particular relates to a wound surface wound image enhancement method based on Transformer. Background Art

[0002] Medical images have become an indispensable part of modern medical diagnosis, directly affecting the accuracy of doctors' diagnosis and the timeliness of disease treatment. Wound images contain large amounts of information and are rich in details, but in the process of image acquisition, the images are interfered by various factors, so that the obtained medical images contain various noises. Since the equipment and instruments collect data at different times and locations, the illumination level in the shooting area will be different, which may cause the collected images to have different brightness, contrast and color distribution. This may cause trouble for subsequent wound analysis and directly affect the accuracy of doctors' judgment of wounds.

[0003] In order to solve the problem of uneven illumination, researchers have proposed some methods and technologies: (1) Illumination correction: By performing illumination correction on the collected images, the images are made consistent in terms of brightness, contrast and color. The illumination correction algorithm can adjust the brightness value of each pixel according to the brightness distribution of the image, thereby achieving image illumination balance; (2) High dynamic range image fusion: For scenes with large illumination differences, high dynamic range (HDR) image fusion technology can be used. This technology achieves a wider brightness range in the panoramic image and reduces illumination unevenness by fusing multiple images with different exposure levels; (3) Image enhancement method based on grayscale transformation: Grayscale transformation is one of the important means of image enhancement, which can increase the dynamic range of the image, expand the contrast, make the image clear and have obvious features. Grayscale transformation can effectively change the histogram distribution of the image and make the pixel distribution more uniform.

[0004] Although the above methods have certain advantages in solving the problem of uneven illumination, they also have some potential disadvantages: (1) The illumination correction method may lead to over-processing or distortion. When performing global illumination correction, the details of some local areas may be lost or become unrealistic. Over-correction will lead to excessive adjustment of the brightness and contrast of the image, which may affect the visibility and details of the image. (2) HDR image stitching works better when processing scenes with large illumination differences, but for scenes with discontinuous illumination changes or complex transitions, it may not be able to completely eliminate the illumination unevenness, which may cause the stitched image to have unnatural transitions or inconsistent brightness and darkness in some areas. (3) Image enhancement methods based on grayscale transformation still have many limitations for processing low-illuminance images, especially for extremely low-illuminance images, such as over-enhancement of images, color distortion, noise highlighting, insufficient detail enhancement, etc.; the ability to adjust brightness is also insufficient, such as insufficient enhancement of low-brightness areas and insufficient brightness suppression of local high-brightness areas.

[0005] Features such as brightness, color, and details in low-light images often exist in information of different scales, so it is extremely challenging to achieve high-quality low-light image enhancement. Existing image enhancement methods cannot fully utilize multi-scale features, nor can they effectively combine features of multiple scales, and cannot comprehensively improve the brightness, color, and detail quality of images. To solve the above problems, the present invention proposes a Transformer-based image enhancement network, which performs image enhancement through an end-to-end Transformer network to overcome problems such as image distortion caused by uneven illumination. Summary of the invention

[0006] In view of the deficiencies in the prior art, the present invention provides a wound image enhancement method based on Transformer, which performs image enhancement through an end-to-end Transformer network, thus overcoming problems such as image distortion caused by uneven lighting.

[0007] The present invention achieves the above technical objectives through the following technical means.

[0008] A wound image enhancement method based on Transformer includes the following processes:

[0009] Step 1: The wound surface image to be enhanced is acquired by an image acquisition device and input into an image processing device for image processing;

[0010] Step 2: The wound image undergoes a 3×3 depth convolution and then enters the local mapping branch module to calculate the local components: multiplication coefficient and addition coefficient A to adjust the impact of image illumination;

[0011] Step 3: Introduce a coordinated attention module to assist the network in calculating the local components M and A, further increase the information of the local components calculated in the local mapping branch module, and then use a 3×3 depth convolution to reduce the dimension of the channel after passing through the coordinated attention module, and then output the new local components;

[0012] Step 4: The wound surface image is input into the global mapping branch module for processing to obtain three global prediction parameters: noise coefficient, gamma parameter, and color matrix;

[0013] Step 5: Combined with the local components output from step 3, the three global prediction parameters obtained in step 4 are used to reconfigure the image parameters so that the input image can be adaptively restored to the image under the target illumination, and finally the final low-light image enhancement result is obtained.

[0014] Furthermore, in step 2, the local mapping branch module includes two independent networks, which are used to generate multiplication coefficients and addition coefficients respectively, and each independent network is composed of three pixel enhancement modules. After the input image passes through the pixel enhancement module, the input features are combined with the output features by element addition, and the original image details are maintained by skip connections;

[0015] The specific process of the pixel enhancement module processing the input image is as follows: first, the input image passes through a 3×3 depth convolution and is superimposed with the original input features, then passes through a normalization module, and then uses point-by-point convolution, 5×5 depth convolution, and point-by-point convolution in sequence to enhance local details, then passes through a normalization module again, and finally uses two point-by-point convolutions to enhance the feature representation. At the same time, a Gaussian error linear unit activation function is used between the two point-by-point convolutions to better preserve the input information.

[0016] Furthermore, before performing the matrix fusion channel, the normalization module performs scaling and biasing by learning two parameters a and b to improve the adaptability of the learnable matrix and achieve illumination normalization of the input features.

[0017] Furthermore, in step 3, the processing of the coordinated attention module includes two steps: coordinating information embedding and coordinating attention generation;

[0018] Coordinated information embedding: By decomposing the global pooling, it is converted into a one-dimensional feature encoding to preserve the position information. For a given input, a pooling layer of size H×1 or 1×W is first used to encode the channels along the horizontal and vertical coordinates respectively. By aggregating features along the horizontal and vertical directions, long-term dependencies are obtained, and position information is preserved, which further helps the network focus on the target of interest.

[0019] Generation of coordinated attention: After the above-mentioned coordinated information embedding transformation, its output is connected, and then transformed using a 1×1 convolution transformation function. The feature map is then decomposed along the spatial dimension, and then a 1×1 convolution is used to increase the dimension, and then the Sigmoid activation function is combined to obtain the final attention vector.

[0020] Furthermore, in step 4, the global mapping branch module uses the attention module of the Transformer network to extract the global information of the image to generate the required parameters, and a randomly initialized global variable is input into the global prediction branch. During the training phase, the query parameters are dynamically updated in each iteration to continuously match the target image. The network adaptively adjusts the parameters that manipulate the global information of the image, and uses the Gaussian error linear unit as the activation function of the global prediction branch network structure. The activation function uses the probability statistics of the input itself to provide random regularization while maintaining the input information.

[0021] Furthermore, the processing process of the global mapping branch module is as follows:

[0022] Before entering the global mapping branch module, the input image first passes through a lightweight encoder composed of two 3×3 deep convolutions. The lightweight encoder encodes high-dimensional features at a lower resolution and uses a higher feature representation to extract the global-level features of the image; then the generated feature map is input into the global prediction module, and the input features are superimposed with the original input features after a 3×3 deep convolution, and then the extracted features are classified through two fully connected layers to obtain the key K and value V of the image itself, which are then combined with the embeddable learnable global parameter Q introduced by the network for calculation, and new features are obtained through the fully connected layer, and then the required predicted parameters are obtained through layer normalization and two fully connected layers. At the same time, the global mapping branch module also uses the Gaussian error linear unit activation function; and before the global mapping branch module outputs the parameters, additional initialization parameters are added to assist training, maintain the stability of training and generate the required parameters, and the output parameters are then respectively passed through a point-by-point convolution to reduce the channel dimension, and finally the three parameters of global prediction are obtained: noise coefficient, gamma parameter, and color matrix.

[0023] The present invention has the following beneficial effects:

[0024] (1) The image information is richer and the effect is closer to the real experience: the local mapping and global prediction branches in the architecture of the present invention can simultaneously extract the local and global information of the image, which can further enrich the image information; secondly, unlike the problems of color distortion and overexposure caused by other algorithm enhancements, the result of the algorithm enhancement of the present invention is consistent with the visual perception of the human eye, and there is no problem of image color distortion.

[0025] (2) Lightweight network structure: In order to increase the adaptability of the network to the input image resolution, the algorithm of the present invention adopts a Transformer network structure composed of a depth-wise convolution network to implement the PEM module, which can greatly reduce the parameters in the network and further save the amount of calculation.

[0026] (3) Superiority of the attention mechanism: The coordinated attention mechanism CA is introduced to further increase the information of the local components calculated in the local mapping branch. Compared with other attention mechanisms, the CA attention mechanism not only considers the channel information, but also integrates the direction-related position information, and is more flexible and lightweight. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 It is a flowchart of the image enhancement method;

[0028] Figure 2 It is a schematic diagram of the local mapping branch module;

[0029] Figure 3 It is a schematic diagram of the structure of the pixel enhancement module;

[0030] Figure 4 It is a schematic diagram of the normalized module;

[0031] Figure 5 This is a schematic diagram of the attention coordination module;

[0032] Figure 6 Schematic diagram of the global mapping branch module. DETAILED DESCRIPTION

[0033] The present invention is further described below in conjunction with the accompanying drawings and specific embodiments, but the protection scope of the present invention is not limited thereto.

[0034] Embodiment 1:

[0035] The wound surface wound image enhancement method based on Transformer of the present invention comprises the following process:

[0036] Step 1: The wound surface image to be enhanced (hereinafter referred to as input image) is acquired by an image acquisition device and input into an image processing device for subsequent image processing;

[0037] Step 2: Reference Figures 1 to 3 ,First, the input image passes through a 3×3DWConv, and then the local mapping branch module is used to calculate the local components (multiplication coefficient M and addition coefficient A) to adjust the influence of image illumination to maintain the resolution of the input image and further preserve the details in the image;

[0038] Among them, the local mapping branch module is as follows Figure 2 As shown in the figure, it includes two independent networks, which are used to generate the multiplication coefficient M and the addition coefficient A respectively. Each independent network consists of three pixel enhancement modules (Pixel-wise EnhancementModule, PEM modules). After the input image passes through the PEM module, the input features are combined with the output features by element addition. The jump connection can better maintain the details of the original image.

[0039] The specific process of the PEM module processing the input image is as follows:

[0040] like Figure 3 As shown in the figure, the input image first passes through a 3×3DWConv and is superimposed with the original input features, then passes through a normalization module, and then uses PWConv, 5×5DWConv, and PWConv in sequence to enhance local details, then passes through a normalization module again, and finally uses two PWConvs to enhance feature representation. At the same time, a Gaussian Error Linear Unit (GELU) activation function is used between the two PWConvs to better maintain the input information; among them, PWConv is Point Wise Convolution, which means point-by-point convolution; DWConv is Depth WiseConvolution, which means deep convolution; deep convolution uses different convolution kernels to extract features from different channels, and point-by-point convolution is a normal convolution with a convolution kernel size of 1×1, that is, 1×1Conv, which performs convolution on a certain point or a certain pixel.

[0041] The normalized module structure is as follows Figure 4 As shown, the normalization module performs scaling and biasing by learning two parameters a and b before performing the matrix fusion channel, which can improve the adaptability of the learnable matrix and better realize the illumination normalization of the input features. Figure 3 middle, represents addition operation, ⊙ represents element-by-element multiplication, Represents matrix multiplication.

[0042] Step 3: If Figure 1 , 5 As shown in the figure, in order to obtain more information about the image and improve the accuracy of mapping the image from sRGB space to raw-RGB space while achieving lightweight network and saving computation, a coordinated attention module (CA) is further introduced to assist the network in calculating the local components M and A, further increase the information of the local components calculated in the local mapping branch module, and then use 3×3DWConv to reduce the dimension of the channel after the coordinated attention module, and then output the new local components M and A;

[0043] Compared with other attention mechanisms, the CA attention mechanism not only considers channel information, but also integrates direction-related position information, and is more flexible and lightweight; the CA attention mechanism is mainly divided into two steps, namely, coordinating information embedding and coordinating attention generation;

[0044] Coordinated information embedding: By decomposing the global pooling, it is converted into a one-dimensional feature encoding to retain the position information. For a given input, a pooling layer of size H×1 or 1×W is first used to encode the channels along the horizontal coordinate and the vertical coordinate respectively. By aggregating features along the horizontal and vertical directions, long-term dependencies can be effectively obtained and position information can be better preserved, which can further help the network focus on the target of interest more accurately.

[0045] Generation of coordinated attention: After the above coordinated information embedding transformation, the output is connected, and then transformed using the 1×1 convolution transformation function. The feature map is then decomposed along the spatial dimension, and then the 1×1 convolution is used to increase the dimension, and then the sigmoid activation function is combined to obtain the final attention vector.

[0046] Figure 5 In the figure, Avg Pool is Average Pooling, which means average pooling; X Avg Pool is Average Pooling on X-axis, which means average pooling along the horizontal coordinate; Y Avg Pool is Average Pooling onY-axis, which means average pooling along the vertical coordinate; Conv2d is two-dimensional convolution, which means two-dimensional convolution; BatchNorm is Batch Normalization, which means batch normalization; H is Height, which means height; W is Width, which means width; C is Channel, which means the number of channels; r is reduction, which means the set parameter; Residual is residual; Re-weight means re-weighting; Concat is Concatenate, which means series connection; Non-linear means nonlinearity.

[0047] Step 4: Figure 1 , 6As shown in the figure, the global mapping branch module is used to simulate the camera ISP process G, so as to realize the lightweight and low-parameter network. The global mapping branch module uses the attention module of the Transformer network to extract the global information of the image to generate the required parameters. A randomly initialized global variable is input into the global prediction branch. During the training phase, the query parameters are dynamically updated in each iteration to continuously match the target image. The network can adaptively adjust the parameters for manipulating the global information of the image, and can better utilize the advantage of the Transformer in obtaining global information. GELU is used as the activation function of the global prediction branch network structure. The function uses the probability statistics of the input itself to provide random regularization while maintaining the input information. The specific processing process is as follows:

[0048] The input image first passes through a lightweight encoder consisting of two 3×3DWConv before entering the global mapping branch module. The lightweight encoder encodes high-dimensional features at a lower resolution, which can save computational costs and is conducive to lightweight design. The use of higher feature representation will help extract global-level features of the image; Figure 6 As shown, the generated feature map is then input into the global prediction module (Global Prediction Module, GPM), the input features are superimposed with the original input features after a 3×3DWConv, and then the extracted features are classified through two fully connected layers (Fully Connected Layer, FC) to obtain the key K and value V of the image itself, and then combined with the embeddable global parameter Q introduced by the network for calculation, and new features are obtained through the fully connected layer, and then the required predicted parameters are obtained through layer normalization (Layer Norm, LN) and two fully connected layers; at the same time, the global mapping branch module also uses the GELU activation function; before the global mapping branch module outputs the parameters, additional initialization parameters are added to assist training, maintain the stability of training and generate the required parameters, and the output parameters are respectively passed through a 1×1Conv to reduce the channel dimension to obtain three parameters of global prediction: noise coefficient, gamma parameter, and color matrix.

[0049] Step 5: Combine the local components M and A output from step 3, and reconfigure the image parameters using the three global prediction parameters obtained in step 4 so that the input image can be adaptively restored to the image under target illumination, and finally obtain the final low-light image enhancement result.

[0050] Embodiment 2:

[0051] This embodiment provides a computer-readable storage medium, which stores program data. When the program data is executed by one or more processors, the device where the storage medium is located controls the Transformer-based wound image enhancement method described in Example 1.

[0052] The embodiments are preferred implementations of the present invention, but the present invention is not limited to the above-mentioned implementations. Any obvious improvements, substitutions or modifications that can be made by those skilled in the art without departing from the essential content of the present invention belong to the protection scope of the present invention.

Claims

1. A wound image enhancement method based on Transformer, characterized in that: The process includes the following: Step 1: The wound surface image to be enhanced is acquired by an image acquisition device and input into an image processing device for image processing; Step 2: The wound image is subjected to a 3×3 depth convolution and then input into the local mapping branch module to calculate the local components: multiplication coefficients and addition coefficients to adjust the influence of image illumination; Step 3: Continue to assist the network for local components by coordinating the attention module and The calculation of the local component calculated in the local mapping branch module is further increased. After the coordinated attention module, a 3×3 depth convolution is performed to reduce the dimension of the channel, and then the new local component is output; Step 4: The wound surface image is input into the global mapping branch module for processing to obtain three global prediction parameters: noise coefficient, gamma parameter, and color matrix; Step 5: Combined with the local components output from step 3, the three global prediction parameters obtained in step 4 are used to reconfigure the image parameters so that the input image can be adaptively restored to the image under the target illumination, and finally the final low-light image enhancement result is obtained.

2. The wound image enhancement method based on Transformer according to claim 1, characterized in that: In step 2, the local mapping branch module includes two independent networks, which are used to generate multiplication coefficients and addition coefficients respectively. Each independent network is composed of three pixel enhancement modules. After the input image passes through the pixel enhancement module, the input features are combined with the output features by element addition, and the original image details are maintained by skip connections. The specific process of the pixel enhancement module processing the input image is as follows: first, the input image passes through a 3×3 depth convolution and is superimposed with the original input features, then passes through a normalization module, and then uses point-by-point convolution, 5×5 depth convolution, and point-by-point convolution in sequence to enhance local details, then passes through a normalization module again, and finally uses two point-by-point convolutions to enhance the feature representation. At the same time, a Gaussian error linear unit activation function is used between the two point-by-point convolutions to better preserve the input information.

3. The wound image enhancement method based on Transformer according to claim 2, characterized in that: The normalization module learns two parameters before performing the matrix fusion channel. and To perform scaling and biasing to improve the adaptability of the learnable matrix and achieve illumination normalization of the input features.

4. The wound image enhancement method based on Transformer according to claim 1, characterized in that: In step 3, the processing of the coordinated attention module includes two steps: coordinating information embedding and coordinating attention generation; Coordinated information embedding: By decomposing the global pooling, it is converted into a one-dimensional feature encoding to preserve the position information. For a given input, first use a or The pooling layer encodes the channels along the horizontal and vertical coordinates respectively, and obtains long-term dependencies and saves position information by aggregating features along the horizontal and vertical directions, which further helps the network focus on the target of interest; Generation of coordinated attention: After the above-mentioned coordinated information embedding transformation, its output is connected, and then transformed using a 1×1 convolution transformation function. The feature map is then decomposed along the spatial dimension, and then a 1×1 convolution is used to increase the dimension, and then the Sigmoid activation function is combined to obtain the final attention vector.

5. The wound image enhancement method based on Transformer according to claim 1, characterized in that: In step 4, the global mapping branch module uses the attention module of the Transformer network to extract the global information of the image to generate the required parameters. A randomly initialized global variable is input into the global prediction branch. During the training phase, the query parameters are dynamically updated in each iteration to continuously match the target image. The network adaptively adjusts the parameters for manipulating the global information of the image. The Gaussian error linear unit is used as the activation function of the global prediction branch network structure. The activation function uses the probability statistics of the input itself to provide random regularization while maintaining the input information.

6. The wound image enhancement method based on Transformer according to claim 5, characterized in that: The processing process of the global mapping branch module is as follows: The input image first passes through a lightweight encoder consisting of two 3×3 depth convolutions before entering the global mapping branch module. The lightweight encoder encodes high-dimensional features at a lower resolution and uses a higher feature representation to extract global-level features of the image. The generated feature map is then input into the global prediction module. The input features are superimposed with the original input features after a 3×3 deep convolution. The extracted features are then classified through two fully connected layers to obtain the key of the image itself. Sum , and then with the global parameters introduced by the network that can be embedded in the learner The calculations are combined and new features are obtained through the fully connected layer. The required prediction parameters are obtained through layer normalization and two fully connected layers. At the same time, the global mapping branch module also uses the Gaussian error linear unit activation function. In addition, before the global mapping branch module outputs the parameters, additional initialization parameters are added to assist the training to maintain the stability of the training and generate the required parameters. The output parameters are then passed through a point-by-point convolution to reduce the channel dimension, and finally the three parameters of the global prediction are obtained: noise coefficient, gamma parameter, and color matrix.

Images