Method for establishing exposure correction model of endoscope image and image correction method

Abstract

The present invention relates to the field of endoscopic image processing, and in particular to a method for establishing an endoscopic image exposure correction model and an image correction method. An exposure correction model is established, and the exposure correction model includes a first convolution module for capturing initial feature information of an input image, multiple exposure correction modules for performing exposure correction processing, and a second convolution module for restoring the number of channels of a final feature map obtained by processing the multiple exposure correction modules. The convolution kernels of the first convolution module and the second convolution module are both 3×3. The multiple exposure correction modules are connected in sequence, and the output of the previous exposure correction module serves as the input of the next exposure correction module. The efficient extraction of global and local features is achieved through a visual state space model, which significantly reduces the amount of calculation. This structure effectively improves the image processing speed.

Description

Technical Field

[0001] The present invention relates to the field of endoscope image processing, and in particular to a method for establishing an endoscope image exposure correction model and an image correction method. Background Art

[0002] Endoscope is a detection instrument that integrates traditional optics, ergonomics, precision machinery, modern electronics, mathematics, software, etc. It can enter the stomach through the mouth or other natural channels into the body. It can be used to see lesions that cannot be shown by X-rays.

[0003] Existing technologies have not adequately addressed the adaptability of endoscopes under complex lighting conditions. For overexposure and underexposure scenarios common in endoscope applications, these methods have limited correction capabilities and cannot guarantee image stability under a variety of exposure conditions. Summary of the invention

[0004] The object of the present invention is to provide a method for establishing an endoscopic image exposure correction model and an image correction method for improving image stability.

[0005] In order to achieve the above object, the present invention adopts such technical solution:

[0006] A method for establishing an exposure correction model for an endoscope image, establishing an exposure correction model, the exposure correction model comprising a first convolution module for capturing initial feature information of an input image, multiple exposure correction modules for performing exposure correction processing, and a second convolution module for restoring the number of channels of a final feature map obtained by processing the multiple exposure correction modules, the convolution kernels of the first convolution module and the second convolution module are both 3×3, the multiple exposure correction modules are connected in sequence, and the output of the previous exposure correction module is used as the input of the next exposure correction module;

[0007] Each of the exposure correction modules includes a first normalization module that performs normalization processing on the input image, a multi-scale 2D selective scanning module that performs feature extraction in four directions on the normalized input image, a second normalization module that performs normalization processing on the feature extraction image, and a feedforward layer network module that performs local modulation on the normalized feature extraction image. A connection module that performs residual connection between the modulated feature map and the input image is connected between two adjacent exposure correction modules.

[0008] Preferably, the feedforward layer network module includes a third convolution kernel module for performing local feature extraction on the feature extraction image, an activation function, and a fourth convolution kernel module for modulating intermediate features. The convolution kernel of the third convolution kernel module is 3×3, and the convolution kernel of the fourth convolution kernel module is 1×1.

[0009] Preferably, the activation function is a GELU activation function.

[0010] Preferably, the multi-scale 2D selective scanning module includes three downsampling modules for multi-dimensional downsampling of the input image, four SS2D modules for scanning the input image and the downsampled image, three upsampling modules for upsampling the scanned images of the downsampled images, a channel merging module for merging the upsampled image and the scanned image of the input image, a 1×1 convolution module, an activation function, and a multiplication processing module for multiplying the feature map output by the activation function and the input image.

[0011] The image correction method based on the exposure correction model is obtained by any of the above-mentioned methods for establishing the endoscopic image exposure correction model. The method for image correction includes the following steps:

[0012] S1: Acquire a set of endoscopic images with normal exposure, preprocess the set of endoscopic images, and acquire an abnormal exposure training set;

[0013] S2: using the exposure correction model to perform exposure correction processing on the abnormal exposure training set, and outputting a restored feature atlas;

[0014] S3: Perform a residual connection operation on each image in the restored feature atlas and the corresponding image in the abnormal exposure training set to obtain a restored data set, and output the restored data set.

[0015] Preferably, the preprocessing in step S1 uses the LECARM model to randomly adjust the exposure range of the endoscopic image set.

[0016] Preferably, in step S1, the exposure range is (-1, 1).

[0017] Preferably, the specific correction steps of each exposure correction model in step S2 include:

[0018] S2-1: The first convolution module captures initial feature information of the input image to obtain an initial feature image;

[0019] S2-2: the first exposure correction module performs exposure correction processing on the initial feature image, and the output of the first exposure correction module is used as the input of the second exposure correction module. After multiple rounds of exposure correction processing, a final feature image is obtained;

[0020] S2-3: The second convolution module restores the number of channels of the final feature map to obtain a restored feature map.

[0021] Preferably, the exposure correction module processes the initial feature image including:

[0022] Normalize the input image;

[0023] Perform feature extraction in four directions on the normalized image to obtain a feature extraction image;

[0024] Performing normalization processing on the feature extraction image;

[0025] Local modulation is performed on the normalized feature extraction image.

[0026] Preferably, the modulated feature map is residually connected with the input image and then input into the next exposure correction module for exposure correction processing.

[0027] By adopting the aforementioned design scheme, the beneficial effects of the present invention are as follows: the present application adopts a combination of a multi-scale 2D selective scanning module and a feedforward layer network module. Unlike the large convolutional network or self-attention mechanism relied on by traditional methods, the efficient extraction of global and local features is achieved through the visual state space model, which significantly reduces the amount of calculation. This structure not only improves the image processing speed, but also makes the correction model more lightweight and easy to deploy on resource-constrained devices. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 It is a pre-processing flow chart of the present invention;

[0029] Figure 2 is an example of a preprocessed image of the present invention;

[0030] Figure 3 It is a schematic diagram of the algorithm structure of the present invention;

[0031] Figure 4 It is a scanning schematic diagram of the multi-scale 2D selective scanning module of the present invention;

[0032] Figure 5 It is a schematic diagram of the exposure correction effect of the present invention. DETAILED DESCRIPTION

[0033] In order to make the purpose, technical scheme and advantages of the present invention clearer, the present invention will be further described in detail below in conjunction with the accompanying drawings. Obviously, the described embodiments are only partial embodiments of the present invention, rather than all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0034] The method for establishing an exposure correction model for endoscope images is to establish an exposure correction model, such as Figure 3As shown, the exposure correction model includes a first convolution module for capturing initial feature information of an input image, multiple exposure correction modules for performing exposure correction processing, and a second convolution module for restoring the number of channels of a final feature map obtained by processing the multiple exposure correction modules. The convolution kernels of the first convolution module and the second convolution module are both 3×3, and the multiple exposure correction modules are connected in sequence, and the output of the previous exposure correction module serves as the input of the next exposure correction module.

[0035] In this embodiment, the first convolution module is used to capture the initial feature information of the image and increase the number of channels, which not only effectively extracts the local features in the image, but also enhances the contextual information of the image, laying the foundation for subsequent multi-scale processing.

[0036] The second convolution module is used to restore the number of channels of the processed image so that the output image is consistent with the input image format, facilitating seamless application in clinical environments.

[0037] In this embodiment, the exposure correction model includes 8 exposure correction modules. After ablation experiment testing, it is found that the use of 8 exposure correction modules has the best correction effect on images with abnormal exposure.

[0038] Each exposure correction module includes a first normalization module that performs normalization processing on the input image, a multi-scale 2D selective scanning module that performs feature extraction in four directions on the normalized input image, a second normalization module that performs normalization processing on the feature extraction image, and a feedforward layer network module that performs local modulation on the normalized feature extraction image. A connection module that performs residual connection between the modulated feature map and the input image is connected between two adjacent exposure correction modules.

[0039] In this embodiment, the first normalization module is used to standardize the input image to ensure that the input images of different channels are processed at the same scale, which is beneficial to the stability of network training; the second normalization module is used to balance the feature distribution after fusion, enhance the stability of downstream processing, and improve the quality of the final output image.

[0040] In this embodiment, the feedforward layer network module includes a third convolution kernel module for performing local feature extraction on the feature extraction image, an activation function, and a fourth convolution kernel module for modulating the intermediate features. The convolution kernel of the third convolution kernel module is 3×3, and the 3×3 convolution kernel can efficiently extract local information within a smaller receptive field. The convolution operation performs a weight-sharing linear operation on the feature extraction image through a sliding window to achieve feature mapping; the convolution kernel of the fourth convolution kernel module is 1×1, which is used to optimize feature representation and enhance correction effect; the activation function is a GELU activation function, and nonlinearity is introduced through the activation function to ensure that the network can learn complex nonlinear mapping relationships.

[0041] The multi-scale 2D selective scanning module includes three downsampling modules for multi-dimensional downsampling of the input image, four SS2D modules for scanning the input image and the downsampled image, three upsampling modules for upsampling the scanned images of the downsampled images, a channel merging module for merging the upsampled image and the scanned image of the input image, a 1×1 convolution module, an activation function, and a multiplication processing module for multiplying the feature map output by the activation function and the input image.

[0042] In this embodiment, four SS2D modules are used to perform feature extraction on the downsampled image in four directions. The four directions here refer to the horizontal direction from left to right, the horizontal direction from right to left, the vertical direction from top to bottom, and the vertical direction from bottom to top. The feature extraction in these directions ensures that the contextual information of the image is captured in different spatial dimensions, thereby enhancing the SS2D module's understanding of the spatial structure and exposure anomalies.

[0043] In summary, the multi-scale 2D selective scanning module realizes multi-scale feature extraction and long-distance dependency capture through the visual state space model, and can dynamically select key features under different exposure conditions. Its innovative scan expansion, feature pyramid and scan merging operations make the correction model more robust and adaptable when processing overexposed and underexposed images; the exposure correction model is highly optimized for endoscopic images, significantly reducing the computing resource requirements while ensuring the quality of the corrected image, and its lightweight design enables the model to achieve real-time correction on low-power devices, which is particularly suitable for resource-constrained medical equipment and real-time surgical scenarios.

[0044] This embodiment also provides a method for image correction using the above exposure correction model.

[0045] The image correction method based on the exposure correction model is obtained by the above-mentioned method for establishing the endoscope image exposure correction model. The method for image correction includes the following steps:

[0046] S1: Obtain a set of endoscopic images with normal exposure, such as Figure 1 and Figure 2 As shown, the endoscopic image set is preprocessed to obtain an abnormal exposure training set; in this embodiment, the LECARM model is used to randomly adjust the exposure range of the images in the endoscopic image set, and the exposure range is (-1, 1) to obtain under-exposed and over-exposed images.

[0047] S2: Use the exposure correction model to perform exposure correction on the abnormal exposure training set and output a restored feature atlas;

[0048] In this embodiment, the specific correction steps of the exposure correction model include:

[0049] S2-1: The first convolution module captures initial feature information of the input image to obtain an initial feature image;

[0050] S2-2: The first exposure correction module performs exposure correction processing on the initial feature image. The output of the first exposure correction module is used as the input of the second exposure correction module. After multiple rounds of exposure correction processing, the final feature map is obtained;

[0051] like Figure 4 As shown, the exposure correction module processes the initial feature image including:

[0052] The input image is normalized to standardize the input image and ensure that the features of different channels are processed at the same scale, which is beneficial to the stability of the exposure correction model.

[0053] Feature extraction is performed on the normalized image in four directions to obtain a feature extracted image. In this embodiment, the normalized image is split into four groups in the channel dimension, three of which are downsampled at different scales and input into the multi-scale 2D selective scanning module for feature extraction, and then an upsampling operation at the corresponding scale is performed. The downsampling operation enables nonlinear operations to be performed on smaller feature maps, which is used to reduce the number of operations of the exposure correction model and is suitable for real-time application scenarios. Moreover, by performing nonlinear operations at different scales and finally fusing multi-scale features, the model can be made more robust and perform better for abnormal exposure areas of different sizes and spatial distributions.

[0054] The remaining group is directly input into the multi-scale 2D selective scanning module for feature extraction, and one group of images is retained for direct feature extraction to help retain high-resolution detail information. Finally, the feature maps of the four groups are spliced ​​in the channel dimension during output to form a feature extraction image.

[0055] Normalize the feature extraction image; balance the feature distribution after fusion, enhance the stability of downstream processing, and improve the quality of the final output feature extraction image.

[0056] Performing local modulation on the normalized feature extraction image;

[0057] The modulated feature map is residually connected with the input image and then input into the next exposure correction module for exposure correction processing.

[0058] S2-3: The second convolution module restores the number of channels of the final feature map to obtain a restored feature map.

[0059] S3: Perform residual connection operation on each image in the restored feature atlas and the corresponding image in the abnormal exposure training set to obtain a restored data set, and output the restored data set, such as Figure 5 In this embodiment, a residual connection operation is performed to make the output image consistent with the input image format, which is convenient for seamless application in a clinical environment. This step ensures that the corrected image is consistent with the input requirements in terms of overall brightness, detail expression and format.

[0060] In summary, this application adopts a combination of a multi-scale 2D selective scanning module and a feedforward layer network module. Unlike the traditional methods that rely on large convolutional networks or self-attention mechanisms, it realizes efficient extraction of global and local features through the visual state space model, significantly reducing the amount of calculation. This structure not only improves the image processing speed, but also makes the correction model more lightweight and easy to deploy on resource-constrained devices.

[0061] The specific implementation methods described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation method of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for establishing an endoscope image exposure correction model, characterized in that: Establishing an exposure correction model, the exposure correction model includes a first convolution module for capturing initial feature information of an input image, multiple exposure correction modules for performing exposure correction processing, and a second convolution module for restoring the number of channels of a final feature map obtained by processing the multiple exposure correction modules, wherein the convolution kernels of the first convolution module and the second convolution module are both 3×3, the multiple exposure correction modules are connected in sequence, and the output of the previous exposure correction module serves as the input of the next exposure correction module; Each of the exposure correction modules includes a first normalization module that performs normalization processing on the input image, a multi-scale 2D selective scanning module that performs feature extraction in four directions on the normalized input image, a second normalization module that performs normalization processing on the feature extraction image, and a feedforward layer network module that performs local modulation on the normalized feature extraction image. A connection module that performs residual connection between the modulated feature map and the input image is connected between two adjacent exposure correction modules.

2. The method for establishing an endoscopic image exposure correction model according to claim 1, wherein: The feedforward layer network module includes a third convolution kernel module for performing local feature extraction on the feature extraction image, an activation function, and a fourth convolution kernel module for modulating intermediate features. The convolution kernel of the third convolution kernel module is 3×3, and the convolution kernel of the fourth convolution kernel module is 1×1.

3. The method for establishing an endoscopic image exposure correction model according to claim 2, wherein: The activation function is the GELU activation function.

4. The method for establishing an endoscopic image exposure correction model according to claim 1, wherein: The multi-scale 2D selective scanning module includes three downsampling modules for multi-dimensional downsampling of the input image, four SS2D modules for scanning the input image and the downsampled image, three upsampling modules for upsampling the scanned images of the downsampled images, a channel merging module for merging the upsampled image and the scanned image of the input image, a 1×1 convolution module, an activation function, and a multiplication processing module for multiplying the feature map output by the activation function and the input image.

5. An image correction method based on an exposure correction model, characterized in that: The exposure correction model is obtained by the method for establishing an endoscopic image exposure correction model according to any one of claims 1 to 4, and the method for performing image correction comprises the following steps: S1: Acquire a set of endoscopic images with normal exposure, preprocess the endoscopic image set, and acquire an abnormal exposure training set; S2: using the exposure correction model to perform exposure correction processing on the abnormal exposure training set, and outputting a restored feature atlas; S3: Perform a residual connection operation on each image in the restored feature atlas and the corresponding image in the abnormal exposure training set to obtain a restored data set, and output the restored data set.

6. The image correction method based on the exposure correction model according to claim 5, characterized in that: In the preprocessing of step S1, the LECARM model is used to perform random exposure range adjustment on the endoscopic image set.

7. The image correction method based on the exposure correction model according to claim 6, characterized in that: In step S1, the exposure range is (-1, 1).

8. The image correction method based on the exposure correction model according to claim 5, characterized in that: The specific correction steps of each exposure correction model in step S2 include: S2-1: The first convolution module captures initial feature information of the input image to obtain an initial feature image; S2-2: the first exposure correction module performs exposure correction processing on the initial feature image, and the output of the first exposure correction module is used as the input of the second exposure correction module. After multiple rounds of exposure correction processing, a final feature image is obtained; S2-3: The second convolution module restores the number of channels of the final feature map to obtain a restored feature map.

9. The image correction method based on the exposure correction model according to claim 8, characterized in that: The processing of the initial feature image by the exposure correction module includes: Normalize the input image; Perform feature extraction in four directions on the normalized image to obtain a feature extraction image; Performing normalization processing on the feature extraction image; Local modulation is performed on the normalized feature extraction image.

10. The image correction method based on the exposure correction model according to claim 9, characterized in that: The modulated feature map is residually connected with the input image and then input into the next exposure correction module for exposure correction processing.

Images