An electron microscope imaging image enhancement method

Abstract

The present invention provides an electron microscope imaging image enhancement method, which belongs to the field of image enhancement. The present invention realizes image enhancement of microbial electron microscope images by constructing multiple modules that work together; the specific steps include: first, using electron microscope imaging equipment to obtain high-resolution images and generate microbial electron microscope image data sets; constructing a deep feature extraction module (DFEM) to extract multi-level features of the image; further enhancing image features through a feature processing module (FPM); using an adaptive feature enhancement module (AFE) to calculate the initial potential coding; realizing feature fusion through an adaptive attention feature fusion module (AAFF); finally, constructing a final image generation module through deconvolution and convolution operations to realize high-resolution reconstruction of the image, improve the clarity and detail performance of low-resolution microbial images, and provide support for the precise analysis and research of microbial samples.

Description

Technical Field

[0001] The invention belongs to the field of image enhancement, and in particular relates to an electron microscope imaging image enhancement method. Background Art

[0002] In microbial research, electron microscopy (EM) is a key tool that can provide high-resolution images of microbial structures; however, problems such as image noise, low contrast, and loss of detail are often encountered during EM imaging, which may be caused by improper sample preparation, instrument limitations, and unstable imaging conditions; accurate and clear images are essential for the identification and study of microorganisms, because microorganisms have complex structures and small sizes, and any loss of detail will affect the understanding of their morphology and function; data enhancement technology can better display the fine structure of microorganisms and enhance the recognition of their features by improving the clarity and contrast of images, thereby improving the accuracy and reliability of microbial identification; this is of great significance for scientific researchers to conduct microbial classification, functional analysis, and pathogen detection, so data enhancement is crucial to improving the quality of electron microscope microbial imaging and research efficiency.

[0003] At present, image enhancement technology has made significant progress in many fields; traditional methods include histogram equalization, frequency domain filtering, edge enhancement, etc. These methods are usually based on the basic statistical characteristics of images and can improve image quality to a certain extent; however, these methods have limitations in dealing with complex noise and subtle structures; in recent years, the rise of deep learning technology has brought new opportunities for image enhancement; algorithms based on convolutional neural networks (CNN), such as generative adversarial networks (GAN) and autoencoders, can generate clearer and more detailed images by learning features from a large amount of training data; these methods perform well in noise reduction, super-resolution, deblurring, etc.; despite this, deep learning models have high requirements for data volume and computing resources, and targeted optimization is also required in specific application areas; therefore, hybrid enhancement strategies that combine traditional methods with modern deep learning technologies have become a hot topic in current research, aiming to make full use of the advantages of various methods and provide better image enhancement effects. Summary of the invention

[0004] The present invention provides an electron microscope imaging image enhancement method, which aims to improve the clarity, detail expression and overall quality of electron microscope images through efficient and accurate image processing technology, so as to better support the analysis and research of microbial samples; the method effectively enhances the resolution and details of the image through the collaborative work of multiple modules, including deep feature extraction, feature processing, adaptive feature enhancement, adaptive attention feature fusion and final image generation, making scientific research and application more accurate and reliable.

[0005] The present invention aims to propose an electron microscope imaging image enhancement model and provide an electron microscope imaging image enhancement method, which includes the following steps.

[0006] S1. Preparation of microbial electron microscope image dataset: Use electron microscope imaging equipment to obtain high-resolution images of microbial samples, and annotate five types of microorganisms, namely Escherichia coli, Staphylococcus aureus, Helicobacter pylori, yeast and Bacillus subtilis. The annotated images constitute the microbial electron microscope image dataset.

[0007] S2. Construct a deep feature extraction module DFEM, including a feature extraction module and an upsampling module.

[0008] S3. Construct a feature processing module FPM, including 3×3 convolution and ReLU activation function.

[0009] S4. Construct an adaptive feature enhancement module AFE, including the calculation of the initial latent code.

[0010] S5. Construct an adaptive attention feature fusion module AAFF, including using self-attention to enhance feature fusion.

[0011] S6. Construct the final image generation module, including deconvolution, 3×3 convolution and ReLU activation function.

[0012] S7. Construct a high-resolution model of microbial electron microscope images, which consists of input, deep feature extraction module DFEM, feature processing module FPM, adaptive feature enhancement module AFE, adaptive attention feature fusion module AAFF, final image generation module, fully connected layer and output.

[0013] Preferably, in step S2, for the deep feature extraction module, a low-resolution microbial electron microscope image is input , , H, W and 3 represent the height, width and number of channels of the microbial image respectively; firstly, a 3×3 standard convolution is used to perform preliminary feature extraction to obtain , , is a 3×3 convolution; the initial feature Image features are obtained through N residual blocks , , ; The residual block includes the following three steps: first, multi-scale feature extraction, , represents 1×1 convolution, represents 3×3 convolution, represents a 5×5 convolution, represents the connection operation; then extract its channel features, , is the Sigmod activation function, is the fully connected layer, is the activation function, is the global average pooling; then extract its spatial features, , is the maximum pooling, is average pooling; then the image features obtained through N residual blocks Upsampling, we get , , , is the depthwise convolution, For the PixelShuffle operation, we get is the output of the deep feature extraction module.

[0014] Preferably, in step S2, for the deep feature extraction module, efficient feature extraction and enhancement are performed on the input image through multi-scale convolution, channel attention mechanism and spatial attention mechanism; multi-scale feature extraction is integrated in each residual block, and convolution kernels of different sizes are used to capture the multi-scale information of the image, and important features are selectively emphasized through channel attention mechanism and spatial attention mechanism; finally, the processed feature map is processed into a high-resolution image through upsampling operation; this module significantly enhances image details and quality while improving computational efficiency, achieving more efficient high-resolution reconstruction.

[0015] Preferably, in step S3, for the feature processing module, the high-resolution feature of the microorganism is obtained after being processed by the deep feature extraction module. , , , and Respectively The height, width and number of channels of the feature processing module include three convolutional layers, and each convolutional layer is followed by a ReLU activation function to perform nonlinear transformation on the output to enhance the expression ability. , , , is a 3×3 convolution, is the activation function, , is the output of the feature processing module.

[0016] Preferably, in step S3, the feature processing module is used to extract features from the input high-resolution image and represent it as a feature map. This process is implemented through convolutional neural networks. The stacking of convolutional layers and activation functions can extract and represent important features in the image. The feature map It will be used in subsequent feature aggregation and enhancement steps to generate the final high-resolution image.

[0017] Preferably, in step S4, for the adaptive feature enhancement module, the preprocessed microbial image features are obtained by the feature processing module. , , , and Respectively The height, width, and number of channels of ; calculated using bilinear interpolation middle The initial latent code of the position , correspond middle The coordinates of the feature points, , , It is a feature map middle The eigenvalue of the kth nearest neighbor point, the maximum value of k is 4, and the coordinates of the four adjacent points can be expressed as , represents the weight of the kth neighboring point, where , the initial latent code is the output of the adaptive feature enhancement module.

[0018] Preferably, in step S4, for the adaptive feature enhancement module, the initial potential code of the target position is efficiently obtained from the high-resolution feature map by bilinear interpolation , providing a basis for subsequent refinement and enhancement; bilinear interpolation uses the weighted average of the four nearest neighbor points to achieve smooth transition and detail retention in space, ensuring that the high-resolution image has good continuity and consistency in the early stage of generation; this process is simple and efficient, laying a solid foundation for the subsequent use of the attention mechanism for more complex feature aggregation and enhancement, making the final high-resolution image more delicate and realistic.

[0019] Preferably, in step S5, for the adaptive attention feature fusion module, the initial latent code of the microorganism is obtained by the adaptive feature enhancement module. , the pre-processed image features of microorganisms are obtained through the feature processing module , , , , and Respectively and The height, width and number of channels of correspond middle The coordinates of the feature points, ; First select the target location The 8 nearest neighbor eigenvalues ​​around, The coordinates of the eight adjacent points around can be expressed as , , , , , , , , Representation feature map middle The feature value of the kth nearest neighbor point at the position, the maximum value of k is 8; first calculate the initial latent code latent code With each neighboring feature The similarity between To express, , and It is a linear transformation used to map features to Query and Key space; then calculate the attention weight, the attention weight is for each neighboring feature In the resulting latent code The importance of attention weights is To express, , is the activation function, Make the weight The sum of is 1; then calculate the final potential code , is the feature representation enhanced by the attention mechanism, , It is a linear transformation used to map features to the Value space; is the output of the adaptive attention feature fusion module.

[0020] Preferably, in step S5, for the adaptive attention feature fusion module, by utilizing the self-attention mechanism, the performance in the image processing task can be significantly improved; the self-attention mechanism can accurately capture the detail information in the image, so that the high-resolution image generation is clearer and more delicate; the self-attention mechanism can calculate the correlation between each pixel and its neighboring pixels, so as to better preserve and enhance the details; secondly, the mechanism can adaptively allocate attention weights according to the similarity between features, so that the model can flexibly adjust the attention when processing different image areas, and this adjustment enables the model to more effectively cope with complex structures and changes in the image; thirdly, by aggregating neighboring features near the target position, the consistency between different areas of the image can be improved, artifacts and distortion can be reduced, thereby improving the overall image quality.

[0021] Preferably, in step S6, for the final image generation module, the input feature is , ,first Through a deconvolution layer and ReLU activation function, , is deconvolution, is the activation function; then a convolutional layer is used to process the feature map Converted to a final high-resolution image, , is a 3×3 convolution, The output of the module is the final image generation module.

[0022] Compared with the prior art, the present invention has the following technical effects:

[0023] The technical method provided by the present invention proposes deep feature extraction, feature processing, adaptive feature enhancement, adaptive attention feature fusion and high-resolution final image generation; first, the deep feature extraction module (DFEM) combines feature extraction and upsampling to effectively extract multi-level and multi-scale feature information from the input image; then, the feature processing module (FPM) further enhances the image features through 3×3 convolution and ReLU activation function, and improves the clarity and detail expression of the image; the adaptive feature enhancement module (AFE) realizes the adaptive enhancement of features by calculating the initial potential code, so that the processed features are more representative and distinguishable; the adaptive attention feature fusion module (AAFF) uses the self-attention mechanism to dynamically adjust and fuse the features, focusing on important features, and improving the feature fusion effect and the recognition performance of the model; finally, through the final image generation module of deconvolution, 3×3 convolution and ReLU activation function, the accurate reconstruction of the high-resolution image is realized; through the organic combination of each module, the technology significantly improves the quality and efficiency of image processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 The present invention provides a flow chart of an electron microscope imaging image enhancement method.

[0025] Figure 2 It is a structural diagram of the deep feature extraction module DFEM provided by the present invention.

[0026] Figure 3 It is a structural diagram of the feature processing module FPM provided by the present invention.

[0027] Figure 4 It is a high-resolution rendering of a low-resolution microbial electron microscope image provided by the present invention. DETAILED DESCRIPTION

[0028] The present invention aims to propose an electron microscope imaging image enhancement method and a high-resolution model for electron microscope imaging images of microorganisms, which includes the collaborative work of multiple key modules. First, through the deep feature extraction module (DFEM), combined with feature extraction and upsampling, multi-level and multi-scale feature information is effectively extracted from the input image. Then, through the feature processing module (FPM), 3×3 convolution and ReLU activation function are used to further enhance the image features and improve the clarity and detail performance of the image. Subsequently, by using the adaptive feature enhancement module (AFE), the adaptive enhancement of features is achieved by calculating the initial latent code, so that the processed features are more representative and discriminative. Then, through the adaptive attention feature fusion module (AAFF), the self-attention mechanism is used to dynamically adjust and fuse the features, focusing on important features, improving the feature fusion effect and the recognition performance of the model. Finally, through the final image generation module, including deconvolution, 3×3 convolution and ReLU activation function, the accurate reconstruction of high-resolution images is achieved. Through the organic combination of these modules, the model significantly improves the resolution and quality of electron microscope imaging images, making the analysis and research of microbial samples more accurate and reliable.

[0029] See also Figure 1 As shown, an electron microscope imaging image enhancement method in an embodiment of the present application has the following process.

[0030] S1. Preparation of microbial electron microscope image dataset. Electron microscope imaging equipment is used on microbial samples to obtain high-resolution images, a total of 3,000 images, and five types of microorganisms are labeled. The five types of microorganisms are Escherichia coli, Staphylococcus aureus, Helicobacter pylori, yeast and Bacillus subtilis. After the labeling is completed, Mixup is used for data enhancement to obtain a microbial electron microscope image dataset. The dataset has a total of 5,000 images, which are divided into training set and validation set according to 7:3.

[0031] S2. Construct a deep feature extraction module DFEM, including a feature extraction module and an upsampling module.

[0032] Further, in step S2, for the deep feature extraction module, the low-resolution microbial electron microscope image is input , , 640, 640 and 3 represent the height, width and number of channels of the microbial image respectively; firstly, a 3×3 standard convolution is used to perform preliminary feature extraction to obtain , , is a 3×3 convolution; the initial feature Image features are obtained through 3 residual blocks , , ; The residual block includes the following three steps: first, multi-scale feature extraction, , represents 1×1 convolution, represents 3×3 convolution, represents a 5×5 convolution, represents the connection operation; then extract its channel features, , is the Sigmod activation function, is the fully connected layer, is the activation function, is the global average pooling; then extract its spatial features, , is the maximum pooling, is average pooling; then the image features obtained after 3 residual blocks Upsampling, we get , , , is the depthwise convolution, For the PixelShuffle operation, we get is the output of the deep feature extraction module.

[0033] S3. Construct a feature processing module FPM, including 3×3 convolution and ReLU activation function.

[0034] Further, in step S3, the feature processing module obtains the high-resolution feature of the microorganism after processing by the deep feature extraction module. , , 1280, 1280 and 64 respectively represent The height, width and number of channels of the feature processing module include three convolutional layers, and each convolutional layer is followed by a ReLU activation function to perform nonlinear transformation on the output to enhance the expression ability. , , , is a 3×3 convolution, is the activation function, , is the output of the feature processing module, .

[0035] S4. Construct an adaptive feature enhancement module AFE, including the calculation of the initial latent code.

[0036] Further, in step S4, for the adaptive feature enhancement module, the preprocessed microbial image features are obtained by the feature processing module. , , 1280, 1280 and 128 represent The height, width, and number of channels of ; calculated using bilinear interpolation middle The initial latent code of the position , correspond middle The coordinates of the feature points, , , It is a feature map middle The eigenvalue of the kth nearest neighbor point, the maximum value of k is 4, and the coordinates of the four adjacent points can be expressed as , represents the weight of the kth neighboring point, where , the initial latent code is the output of the adaptive feature enhancement module, .

[0037] S5. Construct an adaptive attention feature fusion module AAFF, including using self-attention to enhance feature fusion.

[0038] Furthermore, in step S5, for the adaptive attention feature fusion module, the initial latent code of the microorganism is obtained through the adaptive feature enhancement module. , the pre-processed image features of microorganisms are obtained through the feature processing module , , , 1280, 1280 and 128 represent and The height, width and number of channels of correspond middle The coordinates of the feature points, ; First select the target location The 8 nearest neighbor eigenvalues ​​around, The coordinates of the eight adjacent points around can be expressed as , , , , , , , , Representation feature map middle The feature value of the kth nearest neighbor point at the position, the maximum value of k is 8; first calculate the initial latent code latent code With each neighboring feature The similarity between To express, , and It is a linear transformation used to map features to Query and Key space; then calculate the attention weight, the attention weight is for each neighboring feature In the resulting latent code The importance of attention weights is To express, , is the activation function, Make the weight The sum of is 1; then calculate the final potential code , is the feature representation enhanced by the attention mechanism, , It is a linear transformation used to map features to the Value space; is the output of the adaptive attention feature fusion module, .

[0039] S6. Construct the final image generation module, including deconvolution, 3×3 convolution and ReLU activation function.

[0040] Further, in step S6, for the final image generation module, the input feature is , ,first Through a deconvolution layer and ReLU activation function, , is deconvolution, is the activation function; then a convolutional layer is used to process the feature map Converted to a final high-resolution image, , is a 3×3 convolution, is the output of the final image generation module, .

[0041] S7. Construct a high-resolution model of microbial electron microscope images, which consists of input, deep feature extraction module DFEM, feature processing module FPM, adaptive feature enhancement module AFE, adaptive attention feature fusion module AAFF, final image generation module, fully connected layer and output.

[0042] Further, in step S7, for the high-resolution model of microbial electron microscope images, the height and width of the input low-resolution microbial electron microscope image are 640 and 640, and the low-resolution microbial electron microscope image is first input into the deep feature extraction module DFEM. DFEM combines feature extraction and upsampling to effectively extract multi-level and multi-scale feature information from the input image, and then inputs it into the feature processing module FPM; FPM uses 3×3 convolution and ReLU activation function to further enhance the image features, and then inputs it into the adaptive feature enhancement module AFE; the initial potential code corresponding to each feature point is obtained after processing by AFE, and then input into the adaptive attention feature fusion module AAFF to further perform feature fusion by using attention, and after the final image generation module and the fully connected layer, a high-resolution microbial electron microscope image is obtained, and the height and width of the high-resolution microbial electron microscope image are 1280 and 1280.

[0043] Furthermore, the high-resolution model of microbial electron microscope images was coded using the Pycharm application and Python language, using the Pytorch framework. The model input was trained with low-resolution microbial electron microscope images with a resolution of 640×640×3, and the model was trained from scratch for 200 epochs.

[0044] Furthermore, if Figure 4 As shown, Figure 4 The left half shows a low-resolution electron microscope image of a microorganism. Figure 4 The right half shows a high-resolution microbial electron microscope image processed by the high-resolution model of microbial electron microscope images. It can be seen that the image clarity and detail performance have been significantly improved. The processed high-resolution image not only retains the morphological characteristics of the microorganisms, but also enhances the image contrast and edge details, making the structure of the microorganism clearer.

[0045] The above are only preferred embodiments of the present invention. It should be pointed out that a person skilled in the art can make several modifications and improvements without departing from the inventive concept of the present invention, and these all fall within the protection scope of the present invention.

Claims

1. An electron microscope imaging image enhancement method, Features: The following steps are involved: S1. Preparation of microbial electron microscope image dataset: using electron microscope imaging equipment to obtain high-resolution images of microbial samples, and annotating five types of microorganisms, namely Escherichia coli, Staphylococcus, Helicobacter pylori, yeast and Bacillus subtilis. The annotated images constitute a microbial electron microscope image dataset; S2, constructing a deep feature extraction module DFEM, including a feature extraction module and an upsampling module; In step S2, for the deep feature extraction module, the low-resolution microbial electron microscope image is input , , H, W and 3 represent the height, width and number of channels of the microbial image respectively; firstly, a 3×3 standard convolution is used to perform preliminary feature extraction to obtain , , is a 3×3 convolution; the initial feature Image features are obtained through N residual blocks , , ; The residual block includes the following three steps: first, multi-scale feature extraction, , represents 1×1 convolution, represents 3×3 convolution, represents a 5×5 convolution, represents the connection operation; then extract its channel features, , is the Sigmod activation function, is the fully connected layer, is the activation function, is the global average pooling; then extract its spatial features, , is the maximum pooling, is average pooling; then the image features obtained through N residual blocks Upsampling, we get , , , is the depthwise convolution, For the PixelShuffle operation, we get is the output of the deep feature extraction module; S3, construct the feature processing module FPM, including 3×3 convolution and ReLU activation function; In step S3, for the feature processing module, the high-resolution features of the microorganisms are obtained after processing by the deep feature extraction module. , , , and Respectively The height, width and number of channels of the feature processing module include three convolutional layers, and the ReLU activation function is used after each convolutional layer. , , , is a 3×3 convolution, is the activation function, is the output of the feature processing module; S4, constructing an adaptive feature enhancement module AFE, including the calculation of the initial latent code; In step S4, for the adaptive feature enhancement module, the microbial image features are obtained through the feature processing module. , , , and Respectively The height, width, and number of channels of ; calculated using bilinear interpolation middle The initial latent code of the feature position , correspond middle The coordinates of the feature points, , , It is a feature map middle The eigenvalue of the kth nearest neighbor of , where the maximum value of k is 4. represents the weight of the kth neighboring point, the initial potential code is the output of the adaptive feature enhancement module; S5. Construct an adaptive attention feature fusion module AAFF, including using self-attention to enhance feature fusion; In step S5, for the adaptive attention feature fusion module, the initial latent code of the microorganism is obtained through the adaptive feature enhancement module. , the pre-processed image features of microorganisms are obtained through the feature processing module , , , , and Respectively and The height, width and number of channels of correspond middle The coordinates of the feature points, ; Select the target location The 8 nearest neighbor eigenvalues ​​around, Representation feature map middle The feature value of the kth nearest neighbor point at the position, the maximum value of k is 8; first calculate the initial latent code latent code With each neighboring feature The similarity between To express, , and It is a linear transformation used to map features to Query and Key space; then calculate the attention weight, the attention weight is for each neighboring feature In the resulting latent code The importance of attention weights is To express, , is the activation function, Make the weight The sum of is 1; then calculate the final potential code , is the feature representation enhanced by the attention mechanism, , It is a linear transformation used to map features to the Value space. is the output of the adaptive attention feature fusion module; S6. Construct the final image generation module, including deconvolution, 3×3 convolution and ReLU activation function; In step S6, for the final image generation module, the input features are , ,first Through a deconvolution layer and ReLU activation function, , is deconvolution, is the activation function; then a convolutional layer is used to process the feature map Converted to a final high-resolution image, , is a 3×3 convolution, The output of the module is the final image generation; S7. Construct a high-resolution model of microbial electron microscope images, which consists of input, deep feature extraction module DFEM, feature processing module FPM, adaptive feature enhancement module AFE, adaptive attention feature fusion module AAFF, final image generation module, fully connected layer and output.

Images