A deep learning-based microscopic image enhancement method and system

Abstract

The application discloses a microscopic image enhancement method and system based on deep learning, and relates to the technical field of image enhancement; features obtained after sequentially inputting a microscopic image through four convolution pooling layers are all substituted into a preset channel compensation module to obtain a first compensation feature set; after a fifth compensation feature is obtained by substituting a fourth convolution feature into a bottleneck module, the fifth compensation feature is added to the first compensation feature set to obtain a second compensation feature set; each compensation feature in the first compensation feature set is traversed, adjacent compensation features of the compensation feature in the second compensation feature set are determined, and the compensation feature and the adjacent compensation features are substituted into a fusion module to obtain a fusion feature; the fifth compensation feature is substituted into a weighted attention module to obtain a first attention feature; and the fusion feature set and the first attention feature are subjected to upsampling fusion to obtain a target microscopic enhancement image. The method realizes the improvement of image quality and the improvement of microscopic image improvement efficiency under the premise of reducing the calculation resources and time consumption.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of image enhancement, and particularly relates to a microscopic image enhancement method and system based on deep learning. BACKGROUND

[0002] In the fields of biomedical research, material science analysis, etc., microscopic images are the key means to obtain microstructure information. However, due to the physical characteristics of imaging equipment, sample preparation conditions and interference of the imaging environment, the microscopic images generally have quality problems such as low contrast, blurred details and serious noise interference. These problems make it difficult to accurately identify the key structures in the images, bringing great challenges to subsequent feature extraction, target detection and analysis, and further affecting the accuracy and reliability of scientific research conclusions.

[0003] At present, the traditional microscopic image enhancement methods mainly include histogram equalization, homomorphic filtering, wavelet transform, etc. Histogram equalization can improve the contrast by adjusting the gray scale distribution of the image, but it is easy to cause the loss of local detail information; homomorphic filtering can enhance the high-frequency components of the image to a certain extent, but it is highly dependent on parameter setting and has insufficient generalization ability; wavelet transform can process the image at multiple scales, but has high computational complexity and is difficult to adaptively optimize the complex features of the microscopic image. In recent years, the image enhancement technology based on deep learning has achieved remarkable results in the field of natural images, but due to the unique texture features, imaging noise distribution and low signal-to-noise ratio characteristics of the microscopic images, the existing deep learning models cannot be directly applied to the microscopic image enhancement, which requires a large amount of time and computing resources, and also cannot significantly improve the image quality, resulting in low efficiency of microscopic image enhancement. SUMMARY

[0004] The purpose of the present application is to solve the problem that the existing deep learning model cannot be directly applied to the microscopic image enhancement, which requires a large amount of time and computing resources, and also cannot significantly improve the image quality, resulting in low efficiency of microscopic image enhancement, and to propose a microscopic image enhancement method and system based on deep learning.

[0005] In the first aspect of the present application, a microscopic image enhancement method based on deep learning is first proposed, which comprises:

[0006] An input microscopic image is obtained, and the input microscopic image is sequentially passed through four convolutional pooling layers to obtain a convolutional feature set; the convolutional feature set includes a first convolutional feature, a second convolutional feature, a third convolutional feature and a fourth convolutional feature;

[0007] The preset channel compensation module is used to obtain a first compensation feature set by inputting each convolution feature in the convolution feature set, wherein the first compensation feature set comprises a first compensation feature, a second compensation feature, a third compensation feature and a fourth compensation feature;

[0008] The fifth compensation feature is obtained by inputting the fourth convolution feature into the bottleneck module, and the second compensation feature set is obtained by adding the fifth compensation feature to the first compensation feature set;

[0009] Each compensation feature in the first compensation feature set is sequentially traversed, and adjacent compensation features of the compensation feature in the second compensation feature set are determined, the fusion module is used to obtain a fusion feature by inputting the compensation feature and the adjacent compensation feature, and a fusion feature set is obtained by sequentially obtaining all fusion features;

[0010] The first attention feature is obtained by inputting the fifth compensation feature into the weighted attention module;

[0011] The target microscopic enhancement image is obtained by performing upsampling fusion on the fusion feature set and the first attention feature.

[0012] Optionally, obtaining the first compensation feature set by inputting each convolution feature in the convolution feature set into the preset channel compensation module comprises:

[0013] R, G and B channel features are extracted from a target convolution feature to obtain an R channel feature, a G channel feature and a B channel feature, wherein the target convolution feature is any one of the convolution feature set;

[0014] Average gray values of the R channel feature, the G channel feature and the B channel feature are calculated respectively, and the average gray values are sorted from large to small to obtain a high channel, a middle channel and a low channel;

[0015] A target high channel is obtained by performing pixel compensation on the high channel according to the middle channel and the low channel;

[0016] A target middle channel and a target low channel are obtained by performing pixel compensation on the middle channel and the low channel according to the high channel respectively;

[0017] The target high channel, the target middle channel and the target low channel are fused to obtain a compensation feature, and a first compensation feature set is obtained by obtaining all compensation features.

[0018] Optionally, the fusion module is used to obtain a fusion feature by inputting the compensation feature and the adjacent compensation feature comprises:

[0019] The scale of the adjacent compensation feature is normalized to obtain a normalized feature by taking the scale of the compensation feature as a reference;

[0020] If the adjacent compensation feature is 2, the normalized feature includes a left normalized feature and a right normalized feature, the left normalized feature is substituted into the first enhancement module to obtain a left adjacent feature, the right normalized feature is substituted into the second enhancement module to obtain a right adjacent feature, and the compensation feature, the left adjacent feature and the right adjacent feature are fused to obtain a fused feature;

[0021] If the adjacent compensation feature is 1, the normalized feature includes a right normalized feature, the right normalized feature is substituted into the second enhancement module to obtain a right adjacent feature, and the compensation feature and the right adjacent feature are fused to obtain a fused feature.

[0022] Optionally, the weighted attention module is used to include:

[0023] An input feature is obtained, global average pooling, maximum pooling and atrous convolution are performed on the input feature to obtain a first pooled feature, a second pooled feature and an atrous convolution feature;

[0024] The first pooled feature, the second pooled feature and the atrous convolution feature are fused to obtain a fused pooled feature;

[0025] After reshaping convolution is performed on the fused pooled feature, a first attention matrix is obtained through a softmax function;

[0026] An identity matrix and a learnable matrix are initialized, the learnable matrix is updated through an Adam optimizer to obtain an updated learnable matrix, the identity matrix and the first attention matrix are multiplied to obtain a second attention matrix, and the second attention matrix and the updated learnable matrix are added to obtain a target attention matrix;

[0027] After the input feature and the target attention matrix are multiplied element by element, an output feature is obtained through a GELU activation function.

[0028] Optionally, the fused feature set includes a first fused feature, a second fused feature, a third fused feature and a fourth fused feature; upsampling fusion of the fused feature set and the first attention feature to obtain a target microscopic enhancement image includes:

[0029] The first attention feature and the fourth fused feature are channel spliced to obtain a first spliced feature, the first spliced feature is substituted into the weighted attention module and then upsampled to obtain a second attention feature;

[0030] The second attention feature and the third fused feature are channel spliced to obtain a second spliced feature, the second spliced feature is substituted into the weighted attention module and then upsampled to obtain a third attention feature;

[0031] The third attention feature and the second fusion feature are channel spliced to obtain a third spliced feature, and the third spliced feature is substituted into a weighted attention module to obtain a fourth attention feature after upsampling.

[0032] The fourth attention feature and the first fusion feature are channel spliced to obtain a fourth spliced feature, and the fourth spliced feature is subjected to a convolution operation after upsampling to obtain a target microscopic enhancement image.

[0033] In a second aspect of the embodiment of the present application, a microscopic image enhancement system based on deep learning is provided, comprising:

[0034] A convolution pooling module is configured to obtain an input microscopic image, and sequentially pass the input microscopic image through four convolution pooling layers to obtain a convolution feature set; the convolution feature set comprises a first convolution feature, a second convolution feature, a third convolution feature and a fourth convolution feature;

[0035] A first compensation feature set determination module is configured to substitute each convolution feature in the convolution feature set into a preset channel compensation module to obtain a first compensation feature set; the first compensation feature set comprises a first compensation feature, a second compensation feature, a third compensation feature and a fourth compensation feature;

[0036] A second compensation feature set determination module is configured to substitute the fourth convolution feature into a bottleneck module to obtain a fifth compensation feature, and add the fifth compensation feature to the first compensation feature set to obtain a second compensation feature set;

[0037] A feature fusion module is configured to sequentially traverse each compensation feature in the first compensation feature set, determine adjacent compensation features of the compensation feature in the second compensation feature set, substitute the compensation feature and the adjacent compensation features into a fusion module to obtain a fusion feature, and sequentially obtain all fusion features to obtain a fusion feature set;

[0038] An attention feature extraction module is configured to substitute the fifth compensation feature into a weighted attention module to obtain a first attention feature;

[0039] An upsampling fusion module is configured to perform upsampling fusion on the fusion feature set and the first attention feature to obtain a target microscopic enhancement image.

[0040] Optionally, the first compensation feature set determination module comprises:

[0041] A channel extraction module is configured to perform R, G and B channel extraction on a target convolution feature to obtain an R channel feature, a G channel feature and a B channel feature; the target convolution feature is any one of the convolution feature set;

[0042] The channel sorting module is configured to calculate average gray values of R, G and B channel features respectively, and sort the average gray values from large to small to obtain a high channel, a middle channel and a low channel.

[0043] The first pixel compensation module is configured to perform pixel compensation on the high channel according to the middle channel and the low channel to obtain a target high channel.

[0044] The second pixel compensation module is configured to perform pixel compensation on the middle channel and the low channel respectively according to the high channel to obtain a target middle channel and a target low channel.

[0045] The channel fusion module is configured to perform channel fusion on the target high channel, the target middle channel and the target low channel to obtain a compensation feature, and obtain a first compensation feature set by collecting all compensation features.

[0046] Optionally, the feature fusion module comprises:

[0047] The feature normalization module is configured to normalize a scale of a neighboring compensation feature with reference to a scale of the compensation feature to obtain a normalized feature.

[0048] The first feature fusion module is configured to, if the neighboring compensation feature is two, the normalized feature comprises a left normalized feature and a right normalized feature, the left normalized feature is substituted into a first enhancement module to obtain a left neighboring feature, the right normalized feature is substituted into a second enhancement module to obtain a right neighboring feature, and feature fusion is performed on the compensation feature, the left neighboring feature and the right neighboring feature to obtain a fusion feature.

[0049] The second feature fusion module is configured to, if the neighboring compensation feature is one, the normalized feature comprises a right normalized feature, the right normalized feature is substituted into a second enhancement module to obtain a right neighboring feature, and feature fusion is performed on the compensation feature and the right neighboring feature to obtain a fusion feature.

[0050] Optionally, the weighted attention module comprises:

[0051] The input feature extraction module is configured to obtain an input feature, and perform global average pooling, maximum pooling and atrous convolution on the input feature to obtain a first pooled feature, a second pooled feature and an atrous convolution feature.

[0052] The fusion pooled feature determination module is configured to fuse the first pooled feature, the second pooled feature and the atrous convolution feature to obtain a fusion pooled feature.

[0053] The first attention matrix generation module is configured to perform reshaping convolution on the fusion pooled feature, and then pass the fusion pooled feature through a softmax function to obtain a first attention matrix.

[0054] The target attention matrix generation module is configured to initialize a unit matrix and a learnable matrix, update the learnable matrix through an Adam optimizer to obtain an updated learnable matrix, multiply the unit matrix and the first attention matrix to obtain a second attention matrix, and add the second attention matrix and the updated learnable matrix to obtain a target attention matrix.

[0055] The matrix multiplication module is configured to multiply the input feature and the target attention matrix element by element, and then pass the result through a GELU activation function to obtain an output feature.

[0056] Optionally, the set of fusion features includes a first fusion feature, a second fusion feature, a third fusion feature and a fourth fusion feature; and the up-sampling fusion module includes:

[0057] The first spliced feature generation module is configured to splice the first attention feature and the fourth fusion feature in a channel to obtain a first spliced feature, and then pass the first spliced feature into a weighted attention module and perform up-sampling to obtain a second attention feature.

[0058] The second spliced feature generation module is configured to splice the second attention feature and the third fusion feature in a channel to obtain a second spliced feature, and then pass the second spliced feature into a weighted attention module and perform up-sampling to obtain a third attention feature.

[0059] The third spliced feature generation module is configured to splice the third attention feature and the second fusion feature in a channel to obtain a third spliced feature, and then pass the third spliced feature into a weighted attention module and perform up-sampling to obtain a fourth attention feature.

[0060] The fourth spliced feature generation module is configured to splice the fourth attention feature and the first fusion feature in a channel to obtain a fourth spliced feature, and then perform up-sampling on the fourth spliced feature and perform convolution operation to obtain a target microscopic enhanced image.

[0061] The present application has the following advantages:

[0062] The application provides a microscopic image enhancement method based on deep learning, acquires an input microscopic image, and sequentially passes the input microscopic image through four convolution pooling layers to obtain a convolution feature set; the convolution feature set comprises a first convolution feature, a second convolution feature, a third convolution feature and a fourth convolution feature; each convolution feature in the convolution feature set is substituted into a preset channel compensation module to obtain a first compensation feature set; the first compensation feature set comprises a first compensation feature, a second compensation feature, a third compensation feature and a fourth compensation feature; the fourth convolution feature is substituted into a bottleneck module to obtain a fifth compensation feature, and the fifth compensation feature is added to the first compensation feature set to obtain a second compensation feature set; each compensation feature in the first compensation feature set is sequentially traversed, the adjacent compensation feature of the compensation feature in the second compensation feature set is determined, the compensation feature and the adjacent compensation feature are substituted into a fusion module to obtain a fusion feature, and all fusion features are sequentially acquired to obtain a fusion feature set; the fifth compensation feature is substituted into a weighted attention module to obtain a first attention feature; the fusion feature set and the first attention feature are subjected to upsampling fusion to obtain a target microscopic enhancement image. The image features are extracted from multiple scales through the four convolution pooling layers, the channel compensation module is used to optimize the feature channel information, the bottleneck module is used to generate supplementary features, the feature fusion strategy is used to integrate the feature information, the weighted attention module is used to focus on important areas, and finally the upsampling fusion is performed, so that the image resolution and details are effectively enhanced, the key information is highlighted, the image quality is enhanced under the premise of reducing the computing resources and time consumption, and the microscopic image enhancement efficiency is improved. BRIEF DESCRIPTION OF DRAWINGS

[0063] The application will be further described below with reference to the drawings.

[0064] Figure 1 A flowchart of a microscopic image enhancement method based on deep learning provided for an embodiment of the application;

[0065] Figure 2 A model framework diagram of a microscopic image enhancement based on deep learning provided for an embodiment of the application;

[0066] Figure 3 A framework diagram of a microscopic image enhancement system based on deep learning provided for an embodiment of the application. DETAILED DESCRIPTION

[0067] The technical solutions in the embodiments of the application will be clearly and completely described below with reference to the drawings in the embodiments of the application. Obviously, the described embodiments are only part of the embodiments of the application, rather than all the embodiments of the application.

[0068] Based on the embodiments in the application, all other embodiments obtained by those skilled in the art without creative labor fall within the protection scope of the application.

[0069] The embodiment of the present application provides a microscopic image enhancement method based on deep learning. Figure 1 , Figure 1 A flow chart of a microscopic image enhancement method based on deep learning is provided in the embodiment of the present application. The method comprises the following steps:

[0070] S101, acquiring an input microscopic image, sequentially passing the input microscopic image through four convolution pooling layers to obtain a convolution feature set;

[0071] S102, substituting each convolution feature in the convolution feature set into a preset channel compensation module to obtain a first compensation feature set;

[0072] S103, substituting the fourth convolution feature into a bottleneck module to obtain a fifth compensation feature, and adding the fifth compensation feature to the first compensation feature set to obtain a second compensation feature set;

[0073] S104, sequentially traversing each compensation feature in the first compensation feature set, determining adjacent compensation features of the compensation feature in the second compensation feature set, substituting the compensation feature and the adjacent compensation feature into a fusion module to obtain a fusion feature, and sequentially obtaining all fusion features to obtain a fusion feature set;

[0074] S105, substituting the fifth compensation feature into a weighted attention module to obtain a first attention feature;

[0075] S106, performing upsampling fusion on the fusion feature set and the first attention feature to obtain a target microscopic enhancement image.

[0076] The convolution feature set comprises a first convolution feature, a second convolution feature, a third convolution feature and a fourth convolution feature; and the first compensation feature set comprises a first compensation feature, a second compensation feature, a third compensation feature and a fourth compensation feature.

[0077] Based on the microscopic image enhancement method based on deep learning provided in the embodiment of the present application, image features are extracted from multiple scales through four convolution pooling layers, feature channel information is optimized by using a channel compensation module, supplementary features are generated by means of a bottleneck module, feature information is integrated through a feature fusion strategy, important areas are focused by using a weighted attention module, and finally, upsampling fusion is performed, so that the image resolution and details are effectively enhanced, key information is highlighted, the image quality is enhanced under the premise of reducing the computing resources and time consumption, and the microscopic image enhancement efficiency is improved.

[0078] In an implementation, the input microscopic image is processed through four convolutional pooling layers in sequence, and image features can be extracted from different scales and levels. The structure of the multiple convolutional layers can capture feature information of different sizes and complexities in the image, has better extraction capability for possible small structures and details in the microscopic image, and is helpful for subsequent more comprehensive analysis and enhancement of the image.

[0079] In an implementation, each convolutional feature is substituted into a preset channel compensation module, which can adjust and compensate the channel dimension for different convolutional features, is helpful for optimizing the channel information of each convolutional feature, makes the feature representation more rich and accurate, improves the quality of the feature, and further improves the effect of the final image enhancement.

[0080] In an implementation, referring to Figure 2 , Figure 2The model framework diagram of the micro image enhancement based on deep learning provided by the embodiment of the application is as follows: the input micro image is first subjected to a convolution pooling layer (A1) to obtain first convolution features; the first convolution features are subjected to a convolution pooling layer (A2) to obtain second convolution features; the second convolution features are subjected to a convolution pooling layer (A3) to obtain third convolution features; the third convolution features are subjected to a convolution pooling layer (A4) to obtain fourth convolution features; wherein A1, A2, A3 and A4 have the same structure and are subjected to two-dimensional convolution operation (convolution kernel K=3) first, and then subjected to maximum pooling operation; the first convolution features are subjected to a channel compensation module (B1) to obtain first compensation features; the second convolution features are subjected to a channel compensation module (B2) to obtain second compensation features; the third convolution features are subjected to a channel compensation module (B3) to obtain third compensation features; the fourth convolution features are subjected to a channel compensation module (B4) to obtain fourth compensation features; wherein B1, B2, B3 and B4 have the same structure; the fourth convolution features are subjected to a bottleneck module to obtain fifth compensation features, the bottleneck module comprises three layers, the first layer is 1x1 convolution, the second layer is 3x3 convolution, and the third layer is 1x1 convolution; the first compensation features and the second compensation features are subjected to a fusion module (C1) to obtain first fusion features; the first compensation features, the second compensation features and the third compensation features are subjected to a fusion module (C2) to obtain second fusion features; the second compensation features, the third compensation features and the fourth compensation features are subjected to a fusion module (C3) to obtain third fusion features; the third compensation features, the fourth compensation features and the fifth compensation features are subjected to a fusion module (C4) to obtain fourth fusion features; the fifth compensation features are subjected to a weighted attention module (D1) to obtain first attention features; the first attention features and the fourth fusion features are subjected to channel splicing (one circle with the letter C in the figure represents channel splicing) to obtain first splicing features, and the first splicing features are subjected to an up-sampling after a weighted attention module (D2) to obtain second attention features; the second attention features and the third fusion features are subjected to channel splicing to obtain second splicing features, and the second splicing features are subjected to an up-sampling after a weighted attention module (D3) to obtain third attention features; the third attention features and the second fusion features are subjected to channel splicing to obtain third splicing features, and the third splicing features are subjected to an up-sampling after a weighted attention module (D4) to obtain fourth attention features; the fourth attention features and the first fusion features are subjected to channel splicing, and then subjected to an up-sampling and then subjected to convolution operation to obtain a target micro-enhanced image.

[0081] In an implementation, the fourth convolutional feature is substituted into the bottleneck module to obtain a fifth compensation feature, and the fifth compensation feature is added to the first compensation feature set. The bottleneck module can generally compress and transform the features to generate representative features. Adjacent compensation features are determined in sequence, and the fusion module is substituted into the adjacent compensation features to obtain a fusion feature set. Thus, the information of different compensation features and their adjacent features can be comprehensively utilized, the correlation and complementarity between the features are fully utilized, the content of the fused features can more comprehensively and accurately reflect the content of the image, and the quality of the image is further improved.

[0082] In an implementation, the fifth compensation feature is substituted into the weighted attention module to obtain a first attention feature. The weighted attention mechanism can make the model pay more attention to important regions and features in the image and suppress unimportant information. In a microscopic image, this helps to highlight the structures and details of interest and improve the contrast and clarity of the image.

[0083] In an implementation, the fusion feature set and the first attention feature are up-sampled and fused to finally obtain a target microscopic enhancement image. The up-sampling operation can restore the resolution of the image, and the fusion operation comprehensively utilizes the feature information from different sources, so that the enhanced image has both high resolution and rich detail information, meeting the needs of microscopic images for high resolution and detail display.

[0084] In an embodiment, substituting each convolutional feature in the convolutional feature set into the preset channel compensation module to obtain a first compensation feature set includes:

[0085] R, G, and B channel features are extracted from the target convolutional feature. The target convolutional feature is any one of the convolutional feature set.

[0086] The average gray values of the R, G, and B channel features are calculated respectively, and the average gray values are sorted from large to small to obtain a high channel, a middle channel, and a low channel.

[0087] The high channel is pixel compensated according to the middle channel and the low channel to obtain a target high channel.

[0088] The middle channel and the low channel are pixel compensated according to the target high channel to obtain a target middle channel and a target low channel.

[0089] The target high channel, the target middle channel, and the target low channel are channel fused to obtain a compensation feature. All compensation features are obtained to obtain a first compensation feature set.

[0090] In one implementation, the differences in the image gray scale information of different channels can be accurately identified by extracting R, G, B channel features and calculating the average gray scale value ranking, and the high, medium and low channels are divided according to the average gray scale value to provide a basis for channel compensation. The average gray scale value reflects the overall brightness of the channel, and compensation based thereon can reasonably adjust the pixel values of each channel and improve the overall visual effect and feature performance of the image.

[0091] In one implementation, the target high channel is obtained by performing pixel compensation on the high channel according to the medium channel and the low channel through the formula updating each pixel in the high channel, wherein I a is the pixel value in the high channel, is the lowest pixel value in the high channel, is the highest pixel value in the high channel, is the highest pixel value in the medium channel, is the highest pixel value in the low channel, is the updated pixel value in the high channel, and the target high channel is obtained according to all the updated pixel values. Similarly, the target medium channel and the target low channel are also updated by pixels, and the update formulas are updating each pixel in the medium channel, and updating each pixel in the low channel, wherein is the updated pixel value in the medium channel, and I is the average gray scale value of the high channel, is the average gray scale value of the medium channel, is the average gray scale value of the low channel, is the updated pixel value in the low channel, and I c is the pixel value in the low channel.

[0092] In one implementation, the high channel is compensated according to the medium channel and the low channel, which can balance the differences in gray scale between the high channel and other channels. For example, in some microscopic images, a certain channel may have a weak sample characteristic signal, and compensation can enhance the high channel features, highlight the key information, and improve the image contrast and clarity.

[0093] In one implementation, the high channel compensates for the medium and low channels, which can enable the medium and low channels to obtain more high channel feature information and improve image details. When processing cell microscopic images, different cell structures exhibit different characteristics in different channels, and compensation can make the features of each channel more complete and enable the image to contain more rich information.

[0094] In one implementation, the compensated target high, medium and low channels are fused to obtain a compensated feature, which can comprehensively integrate advantages of the channels, and the fusion process makes the image more coordinated in color, contrast, details and the like, and the generated first compensated feature set is more conducive to subsequent image analysis and processing, and provides a better feature basis.

[0095] In one embodiment, the compensated feature and the adjacent compensated feature are substituted into a fusion module to obtain a fused feature, which includes:

[0096] The scale of the adjacent compensated feature is normalized with reference to the scale of the compensated feature to obtain a normalized feature.

[0097] If the adjacent compensated feature is two, the normalized feature includes a left normalized feature and a right normalized feature, the left normalized feature is substituted into a first enhancement module to obtain a left adjacent feature, the right normalized feature is substituted into a second enhancement module to obtain a right adjacent feature, and the compensated feature, the left adjacent feature and the right adjacent feature are fused to obtain the fused feature.

[0098] If the adjacent compensated feature is one, the normalized feature includes a right normalized feature, the right normalized feature is substituted into a second enhancement module to obtain a right adjacent feature, and the compensated feature and the right adjacent feature are fused to obtain the fused feature.

[0099] In one implementation, the adjacent compensated feature is normalized with reference to the scale of the compensated feature, which can eliminate differences in scales of different features. In a microscopic image, the scales of features extracted by different convolution layers can be different, and normalization can enable the features to be compared and fused in the same scale space, avoid some features from being hidden or excessively highlighted due to scale differences, ensure that each feature participates fairly in subsequent processing, and improve the feature fusion effect.

[0100] In one implementation, for the case of two adjacent compensated features, the left and right normalized features are processed using a first and a second enhancement module respectively, the enhancement module can customize enhancement of characteristics of different scale features, and potential information in the adjacent features can be mined.

[0101] In one implementation, in the first compensated feature set, the first, second, third and fourth compensated features are sequentially arranged; therefore, when the compensated feature is the first compensated feature, the adjacent compensated feature is the second compensated feature, and is a right compensated feature, and the right normalized feature is obtained after normalization; when the compensated feature is the second compensated feature, the adjacent compensated features include the first and second compensated features, the first compensated feature is a left compensated feature, and the second compensated feature is a right compensated feature, and the adjacent compensated features corresponding to the third and fourth compensated features are determined in the same manner.

[0102] In an implementation, the first enhancement module includes an average pooling layer, a maximum pooling layer, a convolution layer (1*1 convolution), and an activation function Sigmoid. After the first enhancement module obtains the input feature, the input feature is respectively substituted into the first average pooling layer and the first maximum pooling layer to obtain an output average pooling feature and an output maximum pooling feature. The output average pooling feature and the output maximum pooling feature are spliced to obtain an output fusion pooling. The output fusion pooling sequentially passes through the convolution layer and the activation function Sigmoid to obtain a first output feature.

[0103] In an implementation, the second enhancement module includes an average pooling layer, a maximum pooling layer, and an activation function Sigmoid. After the second enhancement module obtains the input feature, the input feature is respectively substituted into the average pooling layer and the maximum pooling layer to obtain a second output average pooling feature and a second output maximum pooling feature. The second output average pooling feature and the second output maximum pooling feature are spliced to obtain an output fusion pooling. The output fusion pooling sequentially passes through the activation function Sigmoid to obtain a second output feature.

[0104] In an implementation, the feature fusion of the compensation feature, the left adjacent feature, and the right adjacent feature is that the left adjacent feature and the right adjacent feature are spliced and then added to the compensation feature to obtain the fusion feature; and the feature fusion of the compensation feature and the right adjacent feature is that the compensation feature is added to the right adjacent feature.

[0105] In an implementation, when the adjacent compensation feature is one, only the second enhancement module is used to process the right normalized feature, so as to ensure flexibility and adaptability of the processing mode. Even in the condition that the adjacent feature is single, the feature can be effectively enhanced, and existing feature information is fully utilized without missing any details that are helpful for image analysis.

[0106] In an implementation, the compensation feature is fused with the processed adjacent feature, so as to comprehensively integrate feature information from multiple aspects. Different scales of features often contain different levels of image information. After fusion, the feature expression is more comprehensive, more image details and semantic information are covered, and the image enhancement effect is improved.

[0107] In an embodiment, the weighted attention module is used to include:

[0108] The input feature is obtained, and the input feature is subjected to global average pooling, maximum pooling, and atrous convolution to obtain a first pooling feature, a second pooling feature, and an atrous convolution feature.

[0109] The first pooling feature, the second pooling feature, and the atrous convolution feature are fused to obtain a fusion pooling feature.

[0110] After the fusion pooling feature is subjected to reshaping convolution, a first attention matrix is obtained through a softmax function.

[0111] Initialize the unit matrix and the learnable matrix, update the learnable matrix through the Adam optimizer to obtain an updated learnable matrix, multiply the unit matrix and the first attention matrix to obtain a second attention matrix, and add the second attention matrix and the updated learnable matrix to obtain a target attention matrix;

[0112] After element-wise multiplication of the input feature and the target attention matrix, the output feature is obtained through the GELU activation function.

[0113] In an implementation, the input feature is subjected to global average pooling, maximum pooling and atrous convolution respectively, and features are extracted from multiple angles through different operation modes. Global average pooling can capture overall distribution information of the features, maximum pooling highlights local maximum values of the features, and atrous convolution expands the receptive field without increasing too much computation to obtain more context information. The combination of multiple pooling modes makes the extracted features more diverse and helps the model to learn more comprehensive image features.

[0114] In an implementation, the unit matrix and the learnable matrix are initialized to have the same scale as the first attention matrix

[0115] In an implementation, different pooling operations have different sensitivities to noise and local changes in the image. By fusing multiple pooling features, the model can reduce the limitations of a single pooling method, enhance the adaptability to image changes, and thus improve the robustness of the model. In microscopic images, there may be noise and local structural changes, and this multi-mode feature extraction can better handle these situations.

[0116] In an implementation, the first pooling feature, the second pooling feature and the atrous convolution feature are fused, which can integrate complementary information between different features. The features extracted by different pooling operations describe the image in different aspects, and the fused features can integrate these information to provide more accurate and complete image representation.

[0117] In an implementation, the first attention matrix is obtained by reshaping convolution of the fused pooling feature and passing it through a softmax function. Attention mechanism can help the model focus on key information in the input feature. In microscopic images, there may be some important cell structures or lesion areas, and attention mechanism can make the model pay more attention to these key parts and suppress irrelevant information interference, thereby improving the extraction and analysis ability of key information.

[0118] In an implementation, an identity matrix and a learnable matrix are initialized, and the learnable matrix is updated by an Adam optimizer, so that the model can dynamically adjust the weights of the attention matrix according to the data, adapt to the characteristics of different images and task requirements, and better play the role of the attention mechanism and improve the performance of the model compared with a fixed weight setting.

[0119] In an implementation, the input features and the target attention matrix are element-wise multiplied to obtain output features after passing through a GELU activation function. The GELU activation function can introduce a nonlinear transformation to enhance the expression ability of the features. Element-wise multiplication combined with the attention matrix enables the model to weight the input features according to the attention weights, further highlighting the key features, and the final output features have stronger nonlinear expression ability and are more suitable for subsequent task processing.

[0120] In an embodiment, the set of fusion features includes a first fusion feature, a second fusion feature, a third fusion feature, and a fourth fusion feature; and the upsampling fusion of the set of fusion features and the first attention feature to obtain the target microscopic enhanced image includes:

[0121] The first attention feature and the fourth fusion feature are channel spliced to obtain a first spliced feature, and the first spliced feature is substituted into the weighted attention module and then upsampled to obtain a second attention feature;

[0122] The second attention feature and the third fusion feature are channel spliced to obtain a second spliced feature, and the second spliced feature is substituted into the weighted attention module and then upsampled to obtain a third attention feature;

[0123] The third attention feature and the second fusion feature are channel spliced to obtain a third spliced feature, and the third spliced feature is substituted into the weighted attention module and then upsampled to obtain a fourth attention feature;

[0124] The fourth attention feature and the first fusion feature are channel spliced to obtain a fourth spliced feature, and the fourth spliced feature is upsampled and then subjected to convolution operation to obtain the target microscopic enhanced image.

[0125] In an implementation, channel splicing is performed on different attention features and fusion features to integrate feature information from different levels and different types. Different features may contain different aspects of information of an image, such as details and global structure, and channel splicing enables these information to complement each other to form richer and more comprehensive feature representation. In a microscopic image, different fusion features and attention features may capture different structures and features of cells, and channel splicing helps to integrate these information to better present the overall morphology and details of the cells.

[0126] In an implementation manner, the convolution operation is a 1*1 convolution; the channel splicing operation is performed in multiple rounds, and features of different stages are gradually fused, so that layer-by-layer refinement and enhancement of the features can be realized, information is further integrated in each round of splicing based on the previous round, so that the final feature representation is more accurate and detailed, and this multi-round fusion manner helps the model to learn more complex image patterns and improve the recognition ability of the subtle structure in the microscopic image.

[0127] Based on the same inventive concept, the embodiment of the present application also provides a deep learning-based microscopic image enhancement system. Figure 3 , Figure 3 A framework diagram of a deep learning-based microscopic image enhancement system provided by the embodiment of the present application comprises:

[0128] The convolution pooling module is configured to obtain the input microscopic image, and sequentially pass the input microscopic image through four convolution pooling layers to obtain a convolution feature set; the convolution feature set comprises a first convolution feature, a second convolution feature, a third convolution feature and a fourth convolution feature.

[0129] The first compensation feature set determination module is configured to substitute each convolution feature in the convolution feature set into a preset channel compensation module to obtain a first compensation feature set; the first compensation feature set comprises a first compensation feature, a second compensation feature, a third compensation feature and a fourth compensation feature.

[0130] The second compensation feature set determination module is configured to substitute the fourth convolution feature into a bottleneck module to obtain a fifth compensation feature, and add the fifth compensation feature to the first compensation feature set to obtain a second compensation feature set.

[0131] The feature fusion module is configured to sequentially traverse each compensation feature in the first compensation feature set, determine adjacent compensation features of the compensation feature in the second compensation feature set, substitute the compensation feature and the adjacent compensation features into a fusion module to obtain a fusion feature, and sequentially obtain all fusion features to obtain a fusion feature set.

[0132] The attention feature extraction module is configured to substitute the fifth compensation feature into a weighted attention module to obtain a first attention feature.

[0133] The up-sampling fusion module is configured to perform up-sampling fusion on the fusion feature set and the first attention feature to obtain a target microscopic enhancement image.

[0134] Based on the deep learning-based microscopic image enhancement system provided by the embodiment of the application, image features are extracted from multiple scales through four convolutional pooling layers, the channel compensation module is used to optimize the feature channel information, the bottleneck module is used to generate supplementary features, the feature fusion strategy is used to integrate feature information, the weighted attention module is used to focus on important areas, and finally the image resolution and details are effectively enhanced through upsampling fusion, key information is highlighted, the image quality is enhanced under the premise of reducing the computing resources and time consumption, and the microscopic image enhancement efficiency is improved.

[0135] In one embodiment, the first compensation feature set determination module comprises:

[0136] The channel extraction module is configured to perform R, G and B channel extraction on the target convolutional feature to obtain R channel features, G channel features and B channel features; the target convolutional feature is any one of the convolutional feature set;

[0137] The channel sorting module is configured to calculate average gray values of the R channel features, the G channel features and the B channel features respectively, and sort the average gray values from large to small to obtain high channels, middle channels and low channels;

[0138] The first pixel compensation module is configured to perform pixel compensation on the high channels based on the middle channels and the low channels to obtain target high channels.

[0139] The second pixel compensation module is configured to perform pixel compensation on the middle channels and the low channels based on the high channels to obtain target middle channels and target low channels.

[0140] The channel fusion module is configured to perform channel fusion on the target high channels, the target middle channels and the target low channels to obtain compensation features, and obtain a first compensation feature set by collecting all the compensation features.

[0141] In one embodiment, the feature fusion module comprises:

[0142] The feature normalization module is configured to normalize the scale of the adjacent compensation feature based on the scale of the compensation feature to obtain a normalized feature.

[0143] The first feature fusion module is configured to, if the adjacent compensation feature is two, the normalized feature comprises a left normalized feature and a right normalized feature, the left normalized feature is substituted into the first enhancement module to obtain a left adjacent feature, the right normalized feature is substituted into the second enhancement module to obtain a right adjacent feature, and the compensation feature, the left adjacent feature and the right adjacent feature are fused to obtain a fusion feature.

[0144] The second feature fusion module is configured to, if the adjacent compensation feature is one, include the right normalized feature, substitute the right normalized feature into the second enhancement module to obtain a right adjacent feature, and perform feature fusion on the compensation feature and the right adjacent feature to obtain a fused feature.

[0145] In one embodiment, the weighted attention module is configured to include:

[0146] The input feature extraction module is configured to acquire input features, and perform global average pooling, maximum pooling, and atrous convolution on the input features to obtain first pooled features, second pooled features, and atrous convolution features.

[0147] The fused pooled feature determination module is configured to fuse the first pooled features, the second pooled features, and the atrous convolution features to obtain fused pooled features.

[0148] The first attention matrix generation module is configured to perform reshaping convolution on the fused pooled features, and then pass the fused pooled features through a softmax function to obtain a first attention matrix.

[0149] The target attention matrix generation module is configured to initialize a unit matrix and a learnable matrix, update the learnable matrix through an Adam optimizer to obtain an updated learnable matrix, multiply the unit matrix and the first attention matrix to obtain a second attention matrix, and add the second attention matrix and the updated learnable matrix to obtain a target attention matrix.

[0150] The matrix multiplication module is configured to multiply the input features and the target attention matrix element by element, and then pass the input features and the target attention matrix through a GELU activation function to obtain output features.

[0151] In one embodiment, the fused feature set includes first fused features, second fused features, third fused features, and fourth fused features; and the up-sampling fusion module includes:

[0152] The first spliced feature generation module is configured to perform channel splicing on the first attention feature and the fourth fused feature to obtain a first spliced feature, substitute the first spliced feature into the weighted attention module, and then perform up-sampling to obtain a second attention feature.

[0153] The second spliced feature generation module is configured to perform channel splicing on the second attention feature and the third fused feature to obtain a second spliced feature, substitute the second spliced feature into the weighted attention module, and then perform up-sampling to obtain a third attention feature.

[0154] The third spliced feature generation module is configured to perform channel splicing on the third attention feature and the second fused feature to obtain a third spliced feature, substitute the third spliced feature into the weighted attention module, and then perform up-sampling to obtain a fourth attention feature.

[0155] The fourth splicing feature generation module is configured to perform channel splicing on the fourth attention feature and the first fusion feature to obtain a fourth splicing feature, and perform convolution operation on the fourth splicing feature after up-sampling to obtain the target microscopic enhancement image.

[0156] The above has described one embodiment of the present application in detail, but the content is only the preferred embodiment of the present application, and cannot be considered as used for limiting the implementation range of the present application. Any equivalent changes and improvements made according to the scope of the present application should still belong to the patent coverage range of the present application.

Claims

1. A deep learning-based method for enhancing microscopic images, characterized in that, The method includes: An input microscopic image is acquired, and the input microscopic image is sequentially passed through four convolutional pooling layers to obtain a convolutional feature set; the convolutional feature set includes a first convolutional feature, a second convolutional feature, a third convolutional feature, and a fourth convolutional feature; Substitute each convolutional feature in the convolutional feature set into the preset channel compensation module to obtain the first compensation feature set; the first compensation feature set includes a first compensation feature, a second compensation feature, a third compensation feature, and a fourth compensation feature; The fourth convolutional feature is substituted into the bottleneck module to obtain the fifth compensation feature, and the fifth compensation feature is added to the first compensation feature set to obtain the second compensation feature set; Each compensation feature in the first compensation feature set is traversed sequentially. The adjacent compensation features of the compensation feature in the second compensation feature set are determined. The compensation feature and the adjacent compensation features are substituted into the fusion module to obtain the fusion feature. All fusion features are obtained sequentially to obtain the fusion feature set. Substituting the fifth compensation feature into the weighted attention module yields the first attention feature; Upsampling and fusing the fused feature set and the first attention feature yields a target microscopic enhanced image; The weighted attention module includes: The input features are obtained, and global average pooling, max pooling, and dilated convolution are performed on the input features to obtain first pooling features, second pooling features, and dilated convolution features, respectively. The first pooling feature, the second pooling feature, and the dilated convolution feature are fused to obtain a fused pooling feature; The first attention matrix is ​​obtained by reshaping and convolving the fused pooling features and then passing them through the softmax function. Initialize the identity matrix and the learnable matrix, update the learnable matrix using the Adam optimizer to obtain the updated learnable matrix, multiply the identity matrix and the first attention matrix to obtain the second attention matrix, and add the second attention matrix and the updated learnable matrix to obtain the target attention matrix; After element-wise multiplication of the input features and the target attention matrix, the output features are obtained through the GELU activation function; The fusion feature set includes a first fusion feature, a second fusion feature, a third fusion feature, and a fourth fusion feature; upsampling and fusing the fusion feature set and the first attention feature to obtain the target microscopic enhancement image includes: The first attention feature and the fourth fusion feature are concatenated by channels to obtain the first concatenated feature. The first concatenated feature is then substituted into the weighted attention module and upsampled to obtain the second attention feature. The second attention feature and the third fusion feature are concatenated by channels to obtain the second concatenated feature. The second concatenated feature is then substituted into the weighted attention module and upsampled to obtain the third attention feature. The third attention feature and the second fusion feature are concatenated by channels to obtain the third concatenated feature. The third concatenated feature is then substituted into the weighted attention module and upsampled to obtain the fourth attention feature. The fourth attention feature and the first fusion feature are concatenated to obtain the fourth concatenated feature. The fourth concatenated feature is then upsampled and convolved to obtain the target microscopic enhanced image.

2. The microscopic image enhancement method based on deep learning according to claim 1, characterized in that, Substituting each convolutional feature in the convolutional feature set into the preset channel compensation module yields the first compensation feature set, which includes: The R, G, and B channels of the target convolutional feature are extracted to obtain R-channel features, G-channel features, and B-channel features; the target convolutional feature is any one of the convolutional feature sets; Calculate the average gray values ​​of the R channel features, G channel features, and B channel features respectively, and sort them from largest to smallest to obtain the high channel, medium channel, and low channel; The target high channel is obtained by performing pixel compensation on the high channel based on the middle channel and the low channel; Pixel compensation is performed on the middle channel and the low channel based on the high channel to obtain the target middle channel and the target low channel; Channel fusion is performed on the target high channel, the target middle channel, and the target low channel to obtain compensation features, and all compensation features are obtained to obtain the first compensation feature set.

3. The microscopic image enhancement method based on deep learning according to claim 1, characterized in that, Substituting the compensation feature and adjacent compensation features into the fusion module yields the following fused features: The scale of adjacent compensation features is normalized using the scale of this compensation feature as a reference to obtain the normalized feature. If there are two adjacent compensation features, the normalized features include left normalized features and right normalized features. The left normalized features are substituted into the first enhancement module to obtain the left adjacent features, and the right normalized features are substituted into the second enhancement module to obtain the right adjacent features. The compensation features, left adjacent features and right adjacent features are fused to obtain the fused features. If there is only one adjacent compensation feature, then the normalized feature includes the right normalized feature. The right normalized feature is substituted into the second enhancement module to obtain the right adjacent feature. The compensation feature and the right adjacent feature are then fused to obtain the fused feature.

4. A microscopic image enhancement system based on deep learning, characterized in that, The system includes: The convolutional pooling module is used to acquire the input microscopic image and sequentially pass the input microscopic image through four convolutional pooling layers to obtain a convolutional feature set; the convolutional feature set includes a first convolutional feature, a second convolutional feature, a third convolutional feature, and a fourth convolutional feature; The first compensation feature set determination module is used to substitute each convolutional feature in the convolutional feature set into the preset channel compensation module to obtain the first compensation feature set; the first compensation feature set includes a first compensation feature, a second compensation feature, a third compensation feature and a fourth compensation feature; The second compensation feature set determination module is used to substitute the fourth convolution feature into the bottleneck module to obtain the fifth compensation feature, and add the fifth compensation feature to the first compensation feature set to obtain the second compensation feature set. The feature fusion module is used to sequentially traverse each compensation feature in the first compensation feature set, determine the adjacent compensation features of the compensation feature in the second compensation feature set, substitute the compensation feature and the adjacent compensation features into the fusion module to obtain the fusion feature, and sequentially obtain all fusion features to obtain the fusion feature set. An attention feature extraction module is used to substitute the fifth compensation feature into the weighted attention module to obtain the first attention feature; An upsampling fusion module is used to upsample and fuse the fusion feature set and the first attention feature to obtain a target microscopic enhanced image; The weighted attention module includes: The input feature extraction module is used to obtain input features and perform global average pooling, max pooling and dilated convolution on the input features to obtain first pooling features, second pooling features and dilated convolution features respectively; The fusion pooling feature determination module is used to fuse the first pooling feature, the second pooling feature, and the dilated convolution feature to obtain the fusion pooling feature; The first attention matrix generation module is used to reconstruct the fused pooling features and then pass them through the softmax function to obtain the first attention matrix. The target attention matrix generation module is used to initialize the identity matrix and the learnable matrix, update the learnable matrix through the Adam optimizer to obtain the updated learnable matrix, multiply the identity matrix and the first attention matrix to obtain the second attention matrix, and add the second attention matrix and the updated learnable matrix to obtain the target attention matrix. The matrix multiplication module is used to perform element-wise multiplication of the input features and the target attention matrix, and then obtain the output features through the GELU activation function; The fusion feature set includes a first fusion feature, a second fusion feature, a third fusion feature, and a fourth fusion feature; the upsampling fusion module includes: The first splicing feature generation module is used to perform channel splicing on the first attention feature and the fourth fusion feature to obtain the first splicing feature, and then upsample the first splicing feature into the weighted attention module to obtain the second attention feature. The second splicing feature generation module is used to splice the second attention feature and the third fusion feature through channels to obtain the second splicing feature, and then upsample the second splicing feature into the weighted attention module to obtain the third attention feature. The third splicing feature generation module is used to splice the third attention feature and the second fusion feature through channels to obtain the third splicing feature, and then upsample the third splicing feature into the weighted attention module to obtain the fourth attention feature. The fourth stitching feature generation module is used to perform channel stitching on the fourth attention feature and the first fusion feature to obtain the fourth stitching feature, and then perform convolution operation on the fourth stitching feature to obtain the target microscopic enhancement image.

5. A deep learning-based microscopic image enhancement system according to claim 4, characterized in that, The first compensation feature set determination module includes: The channel extraction module is used to extract R, G, and B channels from the target convolutional features to obtain R-channel features, G-channel features, and B-channel features; the target convolutional feature is any one of the convolutional feature sets; The channel sorting module is used to calculate the average gray value of the R channel feature, G channel feature and B channel feature respectively, and sort them from largest to smallest to obtain high channel, medium channel and low channel; The first pixel compensation module is used to perform pixel compensation on the high channel based on the middle channel and the low channel to obtain the target high channel; The second pixel compensation module is used to perform pixel compensation on the middle channel and the low channel according to the high channel to obtain the target middle channel and the target low channel; The channel fusion module is used to perform channel fusion on the target high channel, the target middle channel and the target low channel to obtain compensation features, and obtain a first compensation feature set by acquiring all compensation features.

6. The microscopic image enhancement system based on deep learning according to claim 4, characterized in that, The feature fusion module includes: The feature normalization module is used to normalize the scale of adjacent compensation features with the scale of the compensation feature as a reference to obtain normalized features. The first feature fusion module is used to perform feature fusion on the compensation feature, the normalized feature including the left normalized feature and the right normalized feature if there are two adjacent compensation features, the left normalized feature is substituted into the first enhancement module to obtain the left adjacent feature, the right normalized feature is substituted into the second enhancement module to obtain the right adjacent feature, and the compensation feature, the left adjacent feature and the right adjacent feature are fused to obtain the fused feature. The second feature fusion module is used to perform feature fusion on the compensation feature and the right normalized feature if there is only one adjacent compensation feature. The right normalized feature is substituted into the second enhancement module to obtain the right adjacent feature, and the compensation feature and the right adjacent feature are fused to obtain the fused feature.

Images