Multi-stage progressive super-resolution method for medical image based on optical magnification principle

Abstract

The invention relates to the technical field of medical image processing, in particular to a multi-stage progressive super-resolution method for medical images based on the optical magnification principle, which specifically comprises the following steps of: collecting the medical images based on the optical magnification principle to form a data set, and preprocessing the medical images in the data set to obtain a preprocessed data set; a multi-stage progressive multi-attention network model is constructed, the model comprises three stages, each of the first stage and the second stage comprises a shallow feature extraction module, an initial feature extraction module and an attention supervision module, and the third stage comprises a shallow feature extraction module, an initial feature extraction module and an image reconstruction module. Inputting a low-resolution image into the model for feature extraction, and finally generating an optical magnification principle high-resolution medical image; according to the method, details and refined medical images can be restored, and then clearer optical magnification principle medical images are generated.

Description

Technical Field

[0001] The present invention relates to the technical field of medical image processing, and in particular to a multi-stage progressive super-resolution method for medical images based on the optical magnification principle. Background Art

[0002] In the current field of medical imaging, medical images based on the principle of optical magnification have become an indispensable tool in clinical diagnosis and treatment, such as dermatoscopes, otoscopes, and ophthalmoscopes. These images are widely used in many fields such as dermatology, retinal diseases, and otolaryngology, helping doctors to observe lesions and diagnose diseases more accurately. It not only provides intuitive observation of disease lesions, but also reveals many potential pathological features. Doctors can further obtain rich pathological information and make more accurate judgments and treatments by analyzing the texture, structure, and color of the lesion area in the image. Specifically, actinic keratosis, basal cell carcinoma, benign keratosis-like lesions (seborrheic keratosis and lichen planus keratosis), dermatofibroma, melanoma, melanocytic nevus and vascular lesions (hemangioma, angiokeratoma), cataract, glaucoma, diabetic retinopathy, acute otitis media, chronic suppurative otitis media, ear ventilation tubes, excessive earwax, foreign bodies in the ear, and otitis externa are all common diseases in daily life. These skin, ear, and retinal diseases are of various types, both benign and malignant, which seriously affect the patient's appearance and quality of life, so accurate diagnosis plays a vital role in improving symptoms and preventing damage.

[0003] Medical imaging devices based on the principle of optical magnification, such as dermatoscopes, ophthalmoscopes, and otoscopes, play an indispensable role in the diagnosis and treatment of diseases. Generally speaking, doctors' decisions often rely on the imaging results of these medical devices. High-quality medical images can better help doctors identify the type of disease and locate the location of the disease. However, although these medical devices based on the principle of optical magnification have been widely used in clinical practice, they still face many challenges, including equipment limitations, changes in lighting conditions, and individual differences among patients. These factors often result in low resolution of the generated medical images and blurred details, which in turn causes certain troubles in the diagnosis and treatment of the patient's condition. Therefore, clear and accurate medical images based on the principle of optical magnification play a vital role in the treatment of lesions in related locations.

[0004] Super-resolution is a method for generating high-resolution images, which aims to generate high-resolution images from low-resolution images. It can effectively overcome the problem of unclear resolution of medical images. With the successful application of deep learning in super-resolution tasks, the model can generate high-resolution images from low-resolution inputs by training on a large number of data sets. At present, a series of optical magnification principle medical image super-resolution tasks mainly use neural networks as the main technical methods. These methods can directly learn end-to-end nonlinear mapping regression between low-resolution and high-resolution images, and have good representation capabilities.

[0005] There are two obvious drawbacks in the previous neural network-based super-resolution reconstruction methods of optical magnification principle medical images. First, these methods have weak ability to combine local and global features, and there is spatial and channel redundancy in the feature extraction process, which leads to unsatisfactory performance results of the model. Secondly, most methods adopt a single-stage design concept and cannot make full use of detail information at different scales, thus limiting the generation of higher-resolution medical images and failing to provide doctors with clearer medical images. Therefore, how to design a new method to overcome the drawbacks of previous methods and further utilize multi-scale information to achieve better optical magnification principle medical image reconstruction while balancing the order of feature extraction strategies is still a challenge. Therefore, the present invention provides a multi-stage progressive super-resolution method for optical magnification principle medical images to solve the above problems. Summary of the invention

[0006] In view of the shortcomings of the prior art, the present invention develops a multi-stage progressive super-resolution method for optical magnification principle medical images, the main purpose of which is to repair details and refine medical images, thereby generating clearer optical magnification principle medical images.

[0007] The technical solution of the present invention to solve the technical problem is a multi-stage progressive super-resolution method for medical images based on the optical magnification principle, which is as follows: S1. Collect medical images used for optical magnification principle to form a data set, the medical images include low-resolution images and original high-resolution images, and preprocess the low-resolution medical images in the data set to obtain a preprocessed data set, and then divide the data set into a training set, a validation set and a test set; S2. Construct a multi-stage progressive multi-attention network model, which includes three stages. The first and second stages both include a shallow feature extraction module, an initial feature extraction module, and a supervised attention module. The third stage includes a shallow feature extraction module, an initial feature extraction module, and an image reconstruction module. The low-resolution image is output to the model for feature extraction, and finally a high-resolution medical image of the optical magnification principle is generated. S3. Input the data in the training set, validation set, and test set into the multi-stage progressive multiple attention network model in sequence. Input the data in the training set into the model and train the model with the loss function. Then input the data in the validation set into the trained model and adjust the model parameters with the evaluation index. Finally, input the data in the test set into the adjusted model to test the model and output the final result.

[0008] S1 is as follows: S1.1. Constructing the dataset: Select medical images based on scaling factors and build a dataset based on medical images , medical images specifically include low-resolution medical images used for optical magnification principles and original high-resolution images. The dataset is represented as ,in, Indicates Medical images, Indicates A low-resolution medical image of 64×64 pixels used for optical magnification. Indicates The original high-resolution image used for the optical magnification principle is 64S×64S in size, where S represents the scaling factor. represents the number of medical images in the dataset, represents the index of the medical image, ; S1.2. Dataset preprocessing and partitioning: For the dataset The low-resolution medical images in the dataset are horizontally flipped and rotated for data augmentation, and then the preprocessed dataset is divided into training set, validation set and test set in proportion.

[0009] S2 is as follows: The low-resolution image goes through three stages in sequence. The low-resolution image is input to the first stage, the output of the first stage and the low-resolution image are used as the input of the second stage, and the output of the second stage and the low-resolution image are used as the output of the third stage. The third stage finally outputs a high-resolution medical image based on the optical magnification principle.

[0010] The process in the first stage of the multi-stage progressive multiple attention network model is as follows: S2.1. First, input the low-resolution image Divide into small scales, and then input the division results into the shallow feature extraction module of the first stage to obtain small scale shallow features. , , and , and then input the results of the shallow feature extraction module into the initial feature extraction module of the first stage for preliminary feature capture, and then splice the captured small-scale initial features to obtain the initial features, and finally input the initial features and the low-resolution image into the supervised attention module of the first stage to obtain the final output of the first stage; in, , Indicates height, Indicates width, Indicates the number of channels of the input low-resolution image, , , , , Indicates the number of channels of features output by the shallow feature extraction module in the first stage, (1) The small-scale division operation is as follows: , in, represents the small-scale partitioning operation, Indicates the result of small-scale division; (2) The shallow feature extraction module of the first stage consists of a convolution with a kernel size of 3 and a step size of 1. The calculation in the shallow feature extraction module of the first stage is as follows: , in, Represents the convolution operation; (3) The initial feature extraction module of the first stage is actually a variety of attention groups, which include an overall channel attention module, a multi-convolution feature fusion module and a dual attention module. The multi-convolution feature fusion module includes a channel reconstruction unit, a spatial reconstruction unit and a standard convolution group. First, the small-scale shallow features are input into the overall channel attention module to capture pixel information. Then, the output of the overall channel attention module is input into the multi-convolution feature fusion module to fuse local information and global information. Finally, the output of the multi-convolution feature fusion module is input into the dual attention module to capture context information through different windows to obtain a small-scale initial feature. ; (4) Small size initial features Stitching is then performed and the stitching result is combined with the low-resolution image They are input together into the supervised attention module of the first stage to obtain the final output of the first stage. The specific calculation is as follows: , , , , , , , in, A tensor concatenation method representing scale, and Represents two different splicing results, Represents the collection of splicing results, represents the supervised attention module, represents the intermediate output of the supervised attention module, and Represents two different outputs of the supervised attention module in the first stage.

[0011] The initial feature extraction module of the first stage is as follows: (3-1) The calculation in the overall channel attention module is as follows: , , , , in, represents the intermediate output of the overall channel attention module, Indicates Small-scale shallow features, , represents the convolution operation, express Activation function operation, represents the output of the overall channel attention module, express Operation, represents the convolution operation, represents the operation of the channel attention module, express Activation function operation, express Activation function operation, Represents a tie pooling operation; (3-2) The calculation in the multiple convolution feature fusion module is as follows: 1) Space reconstruction unit: , , , , in, represents the normalization operation, express The result of the processing, and denote the trainable scaling and translation parameters, respectively. and Respectively The mean and standard deviation of represents the constant term that maintains the stability of division, represents the normalized correlation weight, Represents two different weights obtained after threshold processing, Indicates threshold gating, the threshold is set to 0.5, express and The result obtained after the tensor cutting operation after multiplication, express and The result obtained after the tensor cutting operation after multiplication, represents a tensor cutting operation, represents a tensor concatenation operation, represents the operation of the spatial reconstruction unit, represents the output of the spatial reconstruction unit; 2) Channel reconstruction unit: , , , , , , in, Express The result obtained after the tensor cutting operation, represents grouped convolution, Express The result obtained after performing group convolution operation, Express The result of tensor concatenation operation is: represents a tensor concatenation operation, express and The result obtained after tensor splicing operation, represents the normalized exponential function operation, represents the weight of the normalized exponential function operation, represents the intermediate features of the fusion strategy process, Represents the operation of the channel reconstruction unit, represents the output of the channel reconstruction unit; 3) Standard convolution group: , , , , , , , , in, represents a space-channel operation, represents channel-space operation, represents the result of the space-channel operation, represents the result of the channel-space operation, represents the spatial-channel standard convolution result, represents the channel-space standard convolution result, express The module input, , Indicates The result of substandard convolution, represents the total number of standard convolutions, represents the index of the number of standard convolutions, , Represents and branch results, represents the difference branch result, Represents the final output features of the multiple convolutional feature fusion module; (3-3) The calculation in the dual attention module is as follows: , , , , , The internal operations of the DAB module are as follows: , , in, Represents the normalized result of the layer, represents the self-attention mechanism of deformable size window, represents a two-layer routing attention mechanism, represents the intermediate output of the dual attention module, represents the operation of a multilayer perceptron, represents the output of the dual attention module, represents the overall channel attention, represents the multiple convolution feature fusion module, represents the dual attention module, Indicates the first stage of Multiple attention group operations, represents the number of operations of various attention groups, , Represents the final feature output of the first stage continuous multi-attention group module.

[0012] The process in the second stage of the multi-stage progressive multiple attention network model is as follows: S2.2, first input low-resolution image According to the mesoscale division, the division results are then input into the shallow feature extraction module of the second stage to obtain the mesoscale shallow feature and , and then input the results of the shallow feature extraction module and the output of the supervised attention module of the first stage into the initial feature extraction module of the second stage for preliminary feature capture, and then splice the captured medium-scale initial features to obtain the initial features, and finally input the initial features and the low-resolution image into the supervised attention module of the second stage to obtain the final output of the second stage; in, , , Indicates the number of channels of features output by the shallow feature extraction module in the second stage; (1) The specific operation of mesoscale division is as follows: , in, represents the mesoscale partitioning operation, represents the result of mesoscale partitioning; (2) The shallow feature extraction module of the second stage is composed of a convolution with a convolution kernel size of 3 and a step size of 1. The calculation in the shallow feature extraction module of the second stage is as follows: , in, Represents the convolution operation; (3) The structure of the initial feature extraction module in the second stage is the same as that in the first stage. The mid-scale shallow features and the output of the supervised attention module in the first stage are respectively input into the initial feature extraction module in the second stage to obtain the output of the initial feature extraction module in the second stage. The specific calculation is as follows: , , , , in, The tensor concatenation method representing the channel, and Represents two different results of channel tensor concatenation, and Represents two different mesoscale initial features output by the initial feature extraction module of the second stage; (4) The initial mesoscale features and Stitching is then performed and the stitching result is combined with the low-resolution image They are input together into the supervised attention module of the second stage to obtain the final output of the second stage. The specific calculation is as follows: , , in, A tensor concatenation method representing scale, represents the operation of the supervised attention module, Represents the final output of the second stage.

[0013] The process in the third stage of the multi-stage progressive multiple attention network model is as follows: S2.3, first input the low-resolution image According to the large-scale division, the division results are then input into the shallow feature extraction module of the third stage to obtain the large-scale shallow feature , and then the results of the shallow feature extraction module and the output of the second stage supervised attention module are respectively input into the initial feature extraction module of the third stage for preliminary feature capture, and then the captured large-scale initial features are spliced ​​into image size to obtain initial features, and finally the initial features and low-resolution images are input into the third stage supervised attention module to obtain the final output of the third stage, i.e., the high-resolution medical image of the optical magnification principle; in, , Indicates the number of channels of features output by the shallow feature extraction module in the third stage; (1) The large-scale partitioning operation is as follows: , in, represents a large-scale partitioning operation, Represents the result of large-scale division, that is, the input low-resolution image ; (2) The shallow feature extraction module of the third stage consists of a convolution with a kernel size of 3 and a step size of 1. The calculation in the shallow feature extraction module of the third stage is as follows: , in, Represents the convolution operation; (3) The structure of the initial feature extraction module in the third stage is the same as that in the first stage. The large-scale shallow features and the output of the supervised attention module in the second stage are respectively input into the initial feature extraction module in the third stage to obtain the output of the initial feature extraction module in the third stage. The specific calculation is as follows: , , , in, The tensor concatenation method representing the channel, Represents the result of concatenating the tensors of the channels, Indicates The result of the attention group operation of the consecutive channels, Indicates Channel attention modules, represents the total number of channel attention modules, , Represents the large-scale initial features output by the initial feature extraction module in the third stage; (4) Large-scale initial features Input the third stage image reconstruction module, select the scaling factor, and Contains multi-scale information to generate high-resolution medical images using optical magnification , where the image reconstruction module uses The method is used to reorganize pixels. The specific calculation is as follows: .

[0014] S3 is as follows: The loss function is calculated as follows: , in, Represents the prediction results and the true value The loss function and prediction results That is, to generate high-resolution medical images based on the principle of optical magnification. represents the number of medical images in the training set, Indicates Prediction results of medical images, Indicates The true value of a medical image; The peak signal-to-noise ratio PSNP and structural similarity SSIM are used as evaluation indicators to verify the generated optical magnification principle high-resolution medical image. The larger the PSNR and SSIM, the better the quality of the generated optical magnification principle high-resolution medical image.

[0015] The effects provided in the content of the invention are only the effects of the embodiments, rather than all the effects of the invention. The above technical solution has the following advantages or beneficial effects: The present invention adopts a multi-stage progressive architecture design and combines a variety of attention mechanisms to reconstruct related medical images. The present invention uses a variable window attention mechanism to perform continuous feature interaction under different scale images, thereby generating higher resolution optical magnification principle medical images through input low-resolution optical magnification principle medical images under different scaling factors, thereby being more in line with clinical medical diagnostic applications; at the same time, the present invention proposes a new feature extraction method, a multiple convolution feature fusion module, which is mainly composed of a channel reconstruction unit, a spatial reconstruction unit, and a standard convolution group. While balancing the order of spatial and channel feature extraction, the model's local detail features and global structural features are combined to the greatest extent, thereby generating higher quality optical magnification principle medical images and further improving model performance. Therefore, the present invention overcomes the drawbacks of previous methods, can fully extract the deep-level features of the image, and adopt a multi-stage design concept to gradually repair details and refine the image, thereby generating clearer optical magnification principle medical images. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The accompanying drawings are used to provide further understanding of the present invention and constitute a part of the specification. They are used to explain the present invention together with the embodiments of the present invention and do not constitute a limitation of the present invention.

[0017] Figure 1 This is a structural diagram of the multi-stage progressive multiple attention model in the present invention.

[0018] Figure 2 This is a comparison of skin disease image processing results.

[0019] Figure 3 This is a comparison of eye disease image processing results.

[0020] Figure 4 This is a comparison of ear disease image processing results. DETAILED DESCRIPTION

[0021] In order to clearly illustrate the technical features of the present invention, the present invention is described in detail below through specific implementation methods and in conjunction with the accompanying drawings.

[0022] Example 1 like Figure 1 As shown, a multi-stage progressive super-resolution method for medical images based on optical magnification principle is as follows: S1. Collect medical images used for optical magnification principle to form a data set, the medical images include low-resolution images and original high-resolution images, and preprocess the low-resolution medical images in the data set to obtain a preprocessed data set, and then divide the data set into a training set, a validation set and a test set; S2. Construct a multi-stage progressive multi-attention network model, which includes three stages. The first and second stages both include a shallow feature extraction module, an initial feature extraction module, and a supervised attention module. The third stage includes a shallow feature extraction module, an initial feature extraction module, and an image reconstruction module. The low-resolution image is output to the model for feature extraction, and finally a high-resolution medical image of the optical magnification principle is generated. S3. Input the data in the training set, validation set, and test set into the multi-stage progressive multiple attention network model in sequence. Input the data in the training set into the model and train the model with the loss function. Then input the data in the validation set into the trained model and adjust the model parameters with the evaluation index. Finally, input the data in the test set into the adjusted model to test the model and output the final result.

[0023] S1 is as follows: S1.1. Constructing the dataset: Select medical images based on scaling factors and build a dataset based on medical images , medical images specifically include low-resolution medical images used for optical magnification principles and original high-resolution images. The dataset is represented as ,in, Indicates Medical images, Indicates A low-resolution medical image of 64×64 pixels used for optical magnification. Indicates The original high-resolution image used for the optical magnification principle is 64S×64S in size, where S represents the scaling factor. represents the number of medical images in the dataset, represents the index of the medical image, ; S1.2. Dataset preprocessing and partitioning: For the dataset The low-resolution medical images in the dataset are horizontally flipped and rotated for data augmentation, and then the preprocessed dataset is divided into training set, validation set and test set in proportion.

[0024] S2 is as follows: The low-resolution image goes through three stages in sequence. The low-resolution image is input to the first stage, the output of the first stage and the low-resolution image are used as the input of the second stage, and the output of the second stage and the low-resolution image are used as the output of the third stage. The third stage finally outputs a high-resolution medical image based on the optical magnification principle.

[0025] The process in the first stage of the multi-stage progressive multiple attention network model is as follows: S2.1. First, input the low-resolution image Divide into small scales, and then input the division results into the shallow feature extraction module of the first stage to obtain small scale shallow features. , , and , and then input the results of the shallow feature extraction module into the initial feature extraction module of the first stage for preliminary feature capture, and then splice the captured small-scale initial features to obtain the initial features, and finally input the initial features and the low-resolution image into the supervised attention module of the first stage to obtain the final output of the first stage; in, , Indicates height, Indicates width, Indicates the number of channels of the input low-resolution image, , , , , Indicates the number of channels of features output by the shallow feature extraction module in the first stage, (1) The small-scale division operation is as follows: , in, represents the small-scale partitioning operation, Indicates the result of small-scale division; (2) The shallow feature extraction module of the first stage consists of a convolution with a kernel size of 3 and a step size of 1. The calculation in the shallow feature extraction module of the first stage is as follows: , in, Represents the convolution operation; (3) The initial feature extraction module of the first stage is actually a variety of attention groups, which include an overall channel attention module, a multi-convolution feature fusion module and a dual attention module. The multi-convolution feature fusion module includes a channel reconstruction unit, a spatial reconstruction unit and a standard convolution group. First, the small-scale shallow features are input into the overall channel attention module to capture pixel information. Then, the output of the overall channel attention module is input into the multi-convolution feature fusion module to fuse local information and global information. Finally, the output of the multi-convolution feature fusion module is input into the dual attention module to capture context information through different windows to obtain a small-scale initial feature. ; (4) Small size initial features Stitching is then performed and the stitching result is combined with the low-resolution image They are input together into the supervised attention module of the first stage to obtain the final output of the first stage. The specific calculation is as follows: , , , , , , , in, A tensor concatenation method representing scale, and Represents two different splicing results, Represents the collection of splicing results, represents the supervised attention module, represents the intermediate output of the supervised attention module, and Represents two different outputs of the supervised attention module in the first stage.

[0026] The initial feature extraction module of the first stage is as follows: (3-1) The calculation in the overall channel attention module is as follows: , , , , in, represents the intermediate output of the overall channel attention module, Indicates Small-scale shallow features, , represents the convolution operation, express Activation function operation, represents the output of the overall channel attention module, express Operation, represents the convolution operation, represents the operation of the channel attention module, express Activation function operation, express Activation function operation, Represents a tie pooling operation; (3-2) The calculation in the multiple convolution feature fusion module is as follows: 1) Space reconstruction unit: , , , , in, represents the normalization operation, express The result of the processing, and denote the trainable scaling and translation parameters, respectively. and Respectively The mean and standard deviation of represents the constant term that maintains the stability of division, represents the normalized correlation weight, Represents two different weights obtained after threshold processing, Indicates threshold gating, the threshold is set to 0.5, express and The result obtained after the tensor cutting operation after multiplication, express and The result obtained after the tensor cutting operation after multiplication, represents a tensor cutting operation, represents a tensor concatenation operation, represents the operation of the spatial reconstruction unit, represents the output of the spatial reconstruction unit; 2) Channel reconstruction unit: , , , , , , in, Express The result obtained after the tensor cutting operation, represents grouped convolution, Express The result obtained after performing group convolution operation, Express The result of tensor concatenation operation is: represents a tensor concatenation operation, express and The result obtained after tensor splicing operation, represents the normalized exponential function operation, represents the weight of the normalized exponential function operation, represents the intermediate features of the fusion strategy process, Represents the operation of the channel reconstruction unit, represents the output of the channel reconstruction unit; 3) Standard convolution group: , , , , , , , , in, represents a space-channel operation, represents channel-space operation, represents the result of the space-channel operation, represents the result of the channel-space operation, represents the spatial-channel standard convolution result, represents the channel-space standard convolution result, express The module input, , Indicates The result of substandard convolution, represents the total number of standard convolutions, represents the index of the number of standard convolutions, , Represents and branch results, represents the difference branch result, Represents the final output features of the multiple convolutional feature fusion module; (3-3) The calculation in the dual attention module is as follows: , , , , , The internal operations of the DAB module are as follows: , , in, Represents the normalized result of the layer, represents the self-attention mechanism of deformable size window, represents a two-layer routing attention mechanism, represents the intermediate output of the dual attention module, represents the operation of a multilayer perceptron, represents the output of the dual attention module, represents the overall channel attention, represents the multiple convolution feature fusion module, represents the dual attention module, Indicates the first stage of Multiple attention group operations, represents the number of operations of various attention groups, , Represents the final feature output of the first stage continuous multi-attention group module.

[0027] The process in the second stage of the multi-stage progressive multiple attention network model is as follows: S2.2, first input low-resolution image According to the mesoscale division, the division results are then input into the shallow feature extraction module of the second stage to obtain the mesoscale shallow feature and , and then input the results of the shallow feature extraction module and the output of the supervised attention module of the first stage into the initial feature extraction module of the second stage for preliminary feature capture, and then splice the captured medium-scale initial features to obtain the initial features, and finally input the initial features and the low-resolution image into the supervised attention module of the second stage to obtain the final output of the second stage; in, , , Indicates the number of channels of features output by the shallow feature extraction module in the second stage; (1) The specific operation of mesoscale division is as follows: , in, represents the mesoscale partitioning operation, represents the result of mesoscale partitioning; (2) The shallow feature extraction module of the second stage is composed of a convolution with a convolution kernel size of 3 and a step size of 1. The calculation in the shallow feature extraction module of the second stage is as follows: , in, Represents the convolution operation; (3) The structure of the initial feature extraction module in the second stage is the same as that in the first stage. The mid-scale shallow features and the output of the supervised attention module in the first stage are respectively input into the initial feature extraction module in the second stage to obtain the output of the initial feature extraction module in the second stage. The specific calculation is as follows: , , , , in, The tensor concatenation method representing the channel, and Represents two different results of channel tensor concatenation, and Represents two different mesoscale initial features output by the initial feature extraction module of the second stage; (4) The initial mesoscale features and Stitching is then performed and the stitching result is combined with the low-resolution image They are input together into the supervised attention module of the second stage to obtain the final output of the second stage. The specific calculation is as follows: , , in, A tensor concatenation method representing scale, represents the operation of the supervised attention module, Represents the final output of the second stage.

[0028] The process in the third stage of the multi-stage progressive multiple attention network model is as follows: S2.3, first input low-resolution image According to the large-scale division, the division results are then input into the shallow feature extraction module of the third stage to obtain the large-scale shallow feature , and then the results of the shallow feature extraction module and the output of the second stage supervised attention module are respectively input into the initial feature extraction module of the third stage for preliminary feature capture, and then the captured large-scale initial features are spliced ​​into image size to obtain initial features, and finally the initial features and low-resolution images are input into the third stage supervised attention module to obtain the final output of the third stage, i.e., the high-resolution medical image of the optical magnification principle; in, , Indicates the number of channels of features output by the shallow feature extraction module in the third stage; (1) The large-scale partitioning operation is as follows: , in, represents a large-scale partitioning operation, Represents the result of large-scale division, that is, the input low-resolution image ; (2) The shallow feature extraction module of the third stage consists of a convolution with a kernel size of 3 and a step size of 1. The calculation in the shallow feature extraction module of the third stage is as follows: , in, Represents the convolution operation; (3) The structure of the initial feature extraction module in the third stage is the same as that in the first stage. The large-scale shallow features and the output of the supervised attention module in the second stage are respectively input into the initial feature extraction module in the third stage to obtain the output of the initial feature extraction module in the third stage. The specific calculation is as follows: , , , in, The tensor concatenation method representing the channel, Represents the result of concatenating the tensors of the channels, Indicates The result of the attention group operation of the consecutive channels, Indicates Channel attention modules, represents the total number of channel attention modules, , Represents the large-scale initial features output by the initial feature extraction module in the third stage; (4) Large-scale initial features Input the third stage image reconstruction module, select the scaling factor, and Contains multi-scale information to generate high-resolution medical images using optical magnification , where the image reconstruction module uses The method is used to reorganize pixels. The specific calculation is as follows: .

[0029] S3 is as follows: The loss function is calculated as follows: , in, Represents the prediction results and the true value The loss function and prediction results That is, to generate high-resolution medical images based on the principle of optical magnification. represents the number of medical images in the training set, Indicates Prediction results of medical images, Indicates The true value of a medical image; The peak signal-to-noise ratio PSNP and structural similarity SSIM are used as evaluation indicators to verify the generated optical magnification principle high-resolution medical image. The larger the PSNR and SSIM, the better the quality of the generated optical magnification principle high-resolution medical image.

[0030] Example 2 like Figures 2 to 4 As shown in the figure, they are the control group images of dermatoscope, glasses and otoscope, respectively, and the dermatoscope dataset, ophthalmoscope dataset and otoscope dataset are constructed respectively. The specific process is as follows: The dermoscopic data set contains 10,000 high-resolution dermoscopic images and 10,000 low-resolution dermoscopic images in total, and collects images of important diagnostic categories in the field of skin lesions such as actinic keratosis, basal cell carcinoma, benign keratosis-like lesions (seborrheic keratosis and lichen planus keratosis), dermatofibroma, melanoma, melanocytic nevus and vascular lesions (hemangioma, angiokeratoma). The present invention takes out 7,000 dermoscopic images from the 10,000 high-resolution dermoscopic images and 10,000 low-resolution dermoscopic images to form a training set, and then divides the remaining 3,000 high-resolution dermoscopic images and 3,000 low-resolution dermoscopic images into three equal parts to form two validation sets and one test set; like Figure 2 As shown in FIG. 1 , seven skin disease images are selected from the dermoscopic dataset, and the low-resolution dermoscopic images in the dermoscopic dataset are processed by the method of the present invention to obtain a multi-stage progressive multiple attention model output. Figure 2 It can be seen that the clarity of the image output by the model is significantly higher than that of the low-resolution dermoscopic image, and the clarity of the image output by the model can reach the level of the original high-resolution dermoscopic image.

[0031] An ophthalmoscope data set is constructed. The ophthalmoscope data set contains 4216 high-resolution dermoscopic images and 4216 low-resolution dermoscopic images in total. The ophthalmoscope images of cataract, glaucoma, diabetic retinopathy and left and right eyes under normal conditions are collected. The present invention takes out 3316 dermoscopic images from the 4216 high-resolution dermoscopic images and the 4216 low-resolution dermoscopic images to form a training set, and then divides the remaining 900-resolution dermoscopic images and 900-resolution dermoscopic images into three equal parts to form two validation sets and one test set; like Figure 3 As shown in FIG. 1 , four disease images are selected from the glasses dataset, and the low-resolution glasses images in the glasses dataset are processed by the method of the present invention to obtain a multi-stage progressive multiple attention model output. Figure 3 It can be seen that the clarity of the image output by the model is significantly higher than that of the low-resolution glasses image, and the clarity of the image output by the model can reach the level of the original high-resolution glasses image.

[0032] An otoscope dataset was constructed, which contained 956 high-resolution dermoscopic images and 956 low-resolution dermoscopic images in total. The otoscope images of acute otitis media, chronic suppurative otitis media, ear ventilation tubes, excessive earwax, foreign bodies in the ear, otitis externa, pseudotympanic membrane, tympanic membrane and normal conditions were collected. The present invention took out 656 dermoscopic images from the 956 high-resolution dermoscopic images and 956 low-resolution dermoscopic images to form a training set, and then divided the remaining 300 high-resolution dermoscopic images and 300 low-resolution dermoscopic images into three equal parts to form two validation sets and one test set.

[0033] like Figure 4 As shown, eight disease images and a group of normal images are selected from the otoscope dataset, and the low-resolution otoscope images in the otoscope dataset are processed by the method of the present invention to obtain a multi-stage progressive multiple attention model output. Figure 4 It can be seen that the clarity of the image output by the model is significantly higher than that of the low-resolution otoscope image, and the clarity of the image output by the model can reach the level of the original high-resolution otoscope image.

[0034] In summary, it can be seen that the present invention can restore image details and refine medical images, thereby generating clearer optical magnification principle medical images.

[0035] Although the above describes the specific implementation mode of the invention in conjunction with the drawings, it is not intended to limit the scope of protection of the invention. Based on the technical solution of the present invention, various modifications or variations that can be made by those skilled in the art without creative work are still within the scope of protection of the present invention.

Claims

1. A multi-stage progressive super-resolution method for medical images based on optical magnification principle, characterized in that: The following steps are involved: S1. Collect medical images used for optical magnification principle to form a data set, the medical images include low-resolution images and original high-resolution images, and preprocess the low-resolution medical images in the data set to obtain a preprocessed data set, and then divide the data set into a training set, a validation set and a test set; S2. Construct a multi-stage progressive multi-attention network model, which includes three stages. The first and second stages both include a shallow feature extraction module, an initial feature extraction module, and a supervised attention module. The third stage includes a shallow feature extraction module, an initial feature extraction module, and an image reconstruction module. The low-resolution image is output to the model for feature extraction, and finally a high-resolution medical image of the optical magnification principle is generated. S3. Input the data in the training set, validation set, and test set into the multi-stage progressive multiple attention network model in sequence. Input the data in the training set into the model and train the model with the loss function. Then input the data in the validation set into the trained model and adjust the model parameters with the evaluation index. Finally, input the data in the test set into the adjusted model to test the model and output the final result.

2. The multi-stage progressive super-resolution method for optical magnification principle medical images according to claim 1, characterized in that: S1 is as follows: S1.

1. Constructing the dataset: Select medical images based on scaling factors and build a dataset based on medical images , medical images specifically include low-resolution medical images used for optical magnification principles and original high-resolution images. The dataset is represented as ,in, Indicates Medical images, Indicates A low-resolution medical image of 64×64 pixels used for optical magnification. Indicates The original high-resolution image used for the optical magnification principle is 64S×64S in size, where S represents the scaling factor. represents the number of medical images in the dataset, represents the index of the medical image, ; S1.

2. Dataset preprocessing and partitioning: For the dataset The low-resolution medical images in the dataset are horizontally flipped and rotated for data augmentation, and then the preprocessed dataset is divided into training set, validation set and test set in proportion.

3. The multi-stage progressive super-resolution method for optical magnification principle medical images according to claim 2, characterized in that: S2 is as follows: The low-resolution image is output to the model for feature extraction: the low-resolution image goes through three stages in sequence. The low-resolution image is input to the first stage, the output of the first stage and the low-resolution image are used as the input of the second stage, and the output of the second stage and the low-resolution image are used as the output of the third stage. The third stage finally outputs a high-resolution medical image based on the optical magnification principle.

4. The multi-stage progressive super-resolution method for optical magnification principle medical images according to claim 3, characterized in that: The process in the first stage of the multi-stage progressive multiple attention network model is as follows: S2.

1. First, input the low-resolution image Divide into small scales, and then input the division results into the shallow feature extraction module of the first stage to obtain small scale shallow features. , , and , and then input the results of the shallow feature extraction module into the initial feature extraction module of the first stage for preliminary feature capture, and then splice the captured small-scale initial features to obtain the initial features, and finally input the initial features and the low-resolution image into the supervised attention module of the first stage to obtain the final output of the first stage; in, , Indicates height, Indicates width, Indicates the number of channels of the input low-resolution image, , , , , Indicates the number of channels of features output by the shallow feature extraction module in the first stage, (1) The small-scale division operation is as follows: ， in, represents a small-scale partitioning operation, Indicates the result of small-scale division; (2) The shallow feature extraction module of the first stage consists of a convolution with a kernel size of 3 and a step size of 1. The calculation in the shallow feature extraction module of the first stage is as follows: ， in, Represents the convolution operation; (3) The initial feature extraction module of the first stage is actually a variety of attention groups, which include the overall channel attention module, the multi-convolution feature fusion module and the dual attention module. The multi-convolution feature fusion module includes the channel reconstruction unit, the spatial reconstruction unit and the standard convolution group. First, the small-scale shallow features are input into the overall channel attention module to capture the pixel information. Then, the output of the overall channel attention module is input into the multi-convolution feature fusion module to fuse the local information and the global information. Finally, the output of the multi-convolution feature fusion module is input into the dual attention module to capture the context information through different windows to obtain the small-scale initial features. ; (4) Small size initial features Stitching is then performed and the stitching result is combined with the low-resolution image They are input together into the supervised attention module of the first stage to obtain the final output of the first stage. The specific calculation is as follows: ， ， ， ， ， ， ， in, A tensor concatenation method representing scale, and Represents two different splicing results, Represents the collection of splicing results, represents the supervised attention module, represents the intermediate output of the supervised attention module, and Represents two different outputs of the supervised attention module in the first stage.

5. The multi-stage progressive super-resolution method for optical magnification principle medical images according to claim 4, characterized in that: The initial feature extraction module of the first stage is as follows: (3-1) The calculation in the overall channel attention module is as follows: ， ， ， ， in, represents the intermediate output of the overall channel attention module, Indicates Small-scale shallow features, , represents the convolution operation, express Activation function operation, represents the output of the overall channel attention module, express Operation, represents the convolution operation, represents the operation of the channel attention module, express Activation function operation, express Activation function operation, Represents a tie pooling operation; (3-2) The calculation in the multiple convolution feature fusion module is as follows: 1) Space reconstruction unit: ， ， ， ， in, represents the normalization operation, express The result of the processing, and denote the trainable scaling and translation parameters, respectively. and Respectively The mean and standard deviation of represents the constant term that maintains the stability of division, represents the normalized correlation weight, Represents two different weights obtained after threshold processing, Indicates threshold gating, the threshold is set to 0.5, express and The result obtained after the tensor cutting operation after multiplication, express and The result obtained after the tensor cutting operation after multiplication, represents a tensor cutting operation, represents a tensor concatenation operation, represents the operation of the spatial reconstruction unit, represents the output of the spatial reconstruction unit; 2) Channel reconstruction unit: ， ， ， ， ， ， in, Express The result obtained after the tensor cutting operation, represents grouped convolution, Express The result obtained after performing group convolution operation is Express The result of tensor concatenation operation is: represents a tensor concatenation operation, express and The result obtained after tensor splicing operation, represents the normalized exponential function operation, represents the weight of the normalized exponential function operation, represents the intermediate features of the fusion strategy process, Represents the operation of the channel reconstruction unit, represents the output of the channel reconstruction unit; 3) Standard convolution group: ， ， ， ， ， ， ， ， in, represents a space-channel operation, represents channel-space operation, represents the result of the space-channel operation, represents the result of the channel-space operation, represents the spatial-channel standard convolution result, represents the channel-space standard convolution result, express The module input, , Indicates The result of substandard convolution, represents the total number of standard convolutions, represents the index of the number of standard convolutions, , Represents and branch results, represents the difference branch result, Represents the final output features of the multiple convolutional feature fusion module; (3-3) The calculation in the dual attention module is as follows: ， ， ， ， ， The internal operations of the DAB module are as follows: ， ， in, Represents the normalized result of the layer, represents the self-attention mechanism of deformable size window, represents a two-layer routing attention mechanism, represents the intermediate output of the dual attention module, represents the operation of a multilayer perceptron, represents the output of the dual attention module, represents the overall channel attention, represents the multiple convolution feature fusion module, represents the dual attention module, Indicates the first stage of Multiple attention group operations, represents the number of operations of various attention groups, , Represents the final feature output of the first stage continuous multi-attention group module.

6. The multi-stage progressive super-resolution method for optical magnification principle medical images according to claim 5, characterized in that: The process in the second stage of the multi-stage progressive multiple attention network model is as follows: S2.2, first input low-resolution image According to the mesoscale division, the division results are then input into the shallow feature extraction module of the second stage to obtain the mesoscale shallow feature and , and then input the results of the shallow feature extraction module and the output of the supervised attention module of the first stage into the initial feature extraction module of the second stage for preliminary feature capture, and then splice the captured mid-scale initial features to obtain the initial features, and finally input the initial features and the low-resolution image into the supervised attention module of the second stage to obtain the final output of the second stage; in, , , Indicates the number of channels of features output by the shallow feature extraction module in the second stage; (1) The specific operation of mesoscale division is as follows: ， in, represents the mesoscale partitioning operation, represents the result of mesoscale partitioning; (2) The shallow feature extraction module of the second stage is composed of a convolution with a convolution kernel size of 3 and a step size of 1. The calculation in the shallow feature extraction module of the second stage is as follows: ， in, Represents the convolution operation; (3) The structure of the initial feature extraction module in the second stage is the same as that in the first stage. The mid-scale shallow features and the output of the supervised attention module in the first stage are respectively input into the initial feature extraction module in the second stage to obtain the output of the initial feature extraction module in the second stage. The specific calculation is as follows: ， ， ， ， in, The tensor concatenation method representing the channel, and Represents two different results of channel tensor concatenation, and Represents two different mesoscale initial features output by the initial feature extraction module of the second stage; (4) The initial mesoscale features and Stitching is then performed and the stitching result is combined with the low-resolution image They are input together into the supervised attention module of the second stage to obtain the final output of the second stage. The specific calculation is as follows: ， ， in, A tensor concatenation method representing scale, represents the operation of the supervised attention module, Represents the final output of the second stage.

7. The multi-stage progressive super-resolution method for optical magnification principle medical images according to claim 6, characterized in that: The process in the third stage of the multi-stage progressive multiple attention network model is as follows: S2.3, first input low-resolution image According to the large-scale division, the division results are then input into the shallow feature extraction module of the third stage to obtain the large-scale shallow feature , and then the results of the shallow feature extraction module and the output of the second stage supervised attention module are respectively input into the initial feature extraction module of the third stage for preliminary feature capture, and then the captured large-scale initial features are spliced ​​into image size to obtain initial features, and finally the initial features and low-resolution images are input into the third stage supervised attention module to obtain the final output of the third stage, i.e., the high-resolution medical image of the optical magnification principle; in, , Indicates the number of channels of features output by the shallow feature extraction module in the third stage; (1) The large-scale partitioning operation is as follows: ， in, represents a large-scale partitioning operation, Represents the result of large-scale division, that is, the input low-resolution image ; (2) The shallow feature extraction module of the third stage consists of a convolution with a kernel size of 3 and a step size of 1. The calculation in the shallow feature extraction module of the third stage is as follows: ， in, Represents the convolution operation; (3) The structure of the initial feature extraction module in the third stage is the same as that in the first stage. The large-scale shallow features and the output of the supervised attention module in the second stage are respectively input into the initial feature extraction module in the third stage to obtain the output of the initial feature extraction module in the third stage. The specific calculation is as follows: ， ， ， in, The tensor concatenation method representing the channel, Represents the result of concatenating the tensors of the channels, Indicates The result of the attention group operation of the consecutive channels, Indicates Channel attention modules, represents the total number of channel attention modules, , Represents the large-scale initial features output by the initial feature extraction module in the third stage; (4) Large-scale initial features Input the third stage image reconstruction module, select the scaling factor, and Contains multi-scale information to generate high-resolution medical images using optical magnification , where the image reconstruction module uses The method is used to reorganize pixels. The specific calculation is as follows: 。 8. The multi-stage progressive super-resolution method for optical magnification principle medical images according to claim 7, characterized in that S3 The details are as follows: The loss function is calculated as follows: ， in, Represents the prediction results and the true value The loss function and prediction results That is, to generate high-resolution medical images based on the principle of optical magnification. represents the number of medical images in the training set, Indicates Prediction results of medical images, Indicates The true value of a medical image; The peak signal-to-noise ratio PSNP and structural similarity SSIM are used as evaluation indicators to verify the generated optical magnification principle high-resolution medical image. The larger the PSNR and SSIM, the better the quality of the generated optical magnification principle high-resolution medical image.

Images