Pathological image reconstruction method based on space attention and multi-dimensional information

Abstract

The invention discloses a pathological image reconstruction method based on spatial attention and multi-dimensional information, and the method comprises the following steps: obtaining a data set which comprises a plurality of image pairs; establishing a super-resolution reconstruction model, fusing the spatial attention feature map and the multi-dimensional information feature map to obtain an enhanced feature map, then fusing the CFM fusion feature map and the original feature map to obtain a mixed fusion feature map, and performing up-sampling to obtain a super-resolution reconstruction image; and obtaining a low-resolution pathological image, and inputting the low-resolution pathological image into the super-resolution reconstruction model to obtain a super-resolution reconstruction image. According to the method, the spatial attention feature map and the multi-dimensional information feature map are fused, and spatial information and channel information can be comprehensively combined; fusing the CFM fusion feature map, and refining feature channels related to pathology; performing multi-stage feature fusion to obtain a mixed fusion feature map, and ensuring that basic information is fully fused into a final result; the reconstructed high-resolution pathological image is closer to a real original image, and the resolution and quality of the image are improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of image data processing, and particularly relates to a pathological image reconstruction method based on spatial attention and multi-dimensional information. Background Art

[0002] Image super-resolution technology is an advanced image processing method aimed at reconstructing high-resolution images from low-resolution images. By supplementing and restoring detailed information, super-resolution technology makes images clearer and more detailed, significantly enhancing the visual quality and information presentation ability of images. In recent years, super-resolution technology has received extensive attention and applications in many fields such as medical imaging, remote sensing images, video surveillance, and digital content creation.

[0003] In the field of digital pathology, the application of super-resolution technology is particularly important. Pathological images are often generated by scanning devices for disease diagnosis and research. However, due to hardware limitations of scanning devices and considerations of data storage and transmission costs, pathological images are usually stored at a low resolution, which may lead to the loss of key detailed information, such as the edges of cell nuclei, the boundaries of tissue structures, and the texture features of microscopic lesion areas. These information are crucial for accurate disease diagnosis. Especially in the detection of small lesions, tissue type classification, and the analysis of tumor invasion boundaries, insufficient resolution will directly affect the accuracy and reliability of diagnosis.

[0004] Traditional image super-resolution methods include algorithms based on interpolation, statistical modeling, and sparse representation, but they are insufficient in restoring complex textures and diverse features in pathological images. In recent years, the rapid development of deep learning technology has provided more powerful support for image super-resolution. Methods based on convolutional neural networks and generative adversarial networks can capture local details and global features of images, thus significantly improving super-resolution performance. However, sometimes some artifacts and noises will be introduced. For example, unnatural textures, blurred edges, or block effects may appear in the generated images, affecting the visual quality and authenticity of the images.

[0005] Therefore, the super-resolution reconstruction technology for pathological images urgently needs further development to meet the requirements of pathology research and clinical diagnosis for high-resolution images. Summary of the Invention

[0006] The technical problem to be solved by the present invention is to provide a pathological image reconstruction method based on spatial attention and multi-dimensional information in view of the deficiencies in the above-mentioned prior art. The method has a simple structure and reasonable design, and can comprehensively combine spatial information and channel information by fusing spatial attention feature maps and multi-dimensional information feature maps. Then, by fusing the CFM fusion feature maps, the feature channels related to pathology are refined. Then, a hybrid fusion feature map is obtained through multi-stage feature fusion to ensure that the basic information is fully incorporated into the final result, making the reconstructed high-resolution pathological image closer to the real original image and improving the resolution and quality of the image.

[0007] To solve the above technical problems, the technical solution adopted by the present invention is: a pathological image reconstruction method based on spatial attention and multi-dimensional information, characterized in that it includes the following steps:

[0008] Step 1: Obtain the data set W, w h represents the h-th image pair, and w h =(I LR-h , I HR-h ), where I LR-h represents the low-resolution pathological image in the h-th image pair, and I HR-h represents the high-resolution pathological image in the h-th image pair, and 0 < h < m;

[0009] Step 2: Establish a super-resolution reconstruction model;

[0010] Step 201: Obtain the feature map F LR-h of the low-resolution pathological image I shallow-h ;

[0011] Step 202: Feature learning:

[0012] Step 2021: After the feature map F shallow-h is normalized and then passes through the spatial attention feature extraction module, the spatial attention feature map I 1-up-h is obtained; after the feature map F shallow-h is normalized and passes through the multi-dimensional information feature extraction module, the multi-dimensional information feature map I 2-up-h is obtained; the spatial attention feature map I 1-up-h and the multi-dimensional information feature map I 2-up-h are fused to obtain the enhanced feature map I up-h ;

[0013] Step 2022: The feature map F shallow-h is added to the enhanced feature map I up-h to obtain the retained enhanced feature map I sha-up-h ;

[0014] Step 2023: Keep the enhanced feature map, normalize it, and then pass it through the channel fusion module CFM to output the CFM fused feature map I CFM ;

[0015] Step 2024: Keep the enhanced feature map I sha-up-h and the CFM fused feature map I CFM Add them together to obtain the hierarchical fused feature map I sha-up-CFM-h ;

[0016] Step 203: Add the hierarchical fused feature map I sha-up-CFM-h to the feature map F shallow-h to obtain the hybrid fused feature map I sha-up-CFM-sha-h ;

[0017] Step 204: Pass the hybrid fused feature map I sha-up-CFM-sha-h through the convolutional layer and the pixel shuffling layer to obtain the super-resolution reconstructed image I rebuild-h ;

[0018] Step 3: Define the loss function, train the super-resolution reconstruction model, and obtain the final super-resolution reconstruction model;

[0019] Step 4: Obtain the low-resolution pathological image to be reconstructed, input the low-resolution pathological image to be reconstructed into the final super-resolution reconstruction model, and obtain the super-resolution reconstructed image.

[0020] The above-mentioned pathological image reconstruction method based on spatial attention and multi-dimensional information is characterized in that: the steps of obtaining the spatial attention feature map I 1-up-h are as follows:

[0021] Step 2021a: Normalize the feature map F shallow-h and divide it into n parts along the channel. B i-h represents the i-th block of the h-th low-resolution pathological image, where 0 < i < n;

[0022] Step 2021b: Calculate the comprehensive feature M i-h based on the self-feature and spatial distribution feature of each block B i-h ;

[0023] Step 2021c: Obtain the attention map of the i-th block according to the comprehensive feature M i-h , multiply the attention map by the block B i-h to obtain the fused block of the i-th block;

[0024] Step 2021d: Concatenate all the fused blocks to obtain the spatial attention feature map I 1-up-h .

[0025] The above-mentioned pathological image reconstruction method based on spatial attention and multi-dimensional information is characterized in that: the method for calculating the comprehensive feature in step 2021b is: according to the formula

[0026] calculate the comprehensive feature M i-h of the block B i-h , x i-h-r represents the r-th tensor value of B i-h , μ represents the tensor mean, N represents the number of tensor values, σ 2 (B i-h ) represents the variance of B i-h along the channel direction, α and β respectively represent learnable parameters, F i-h represents the feature of B i-h .

[0027] The above-mentioned pathological image reconstruction method based on spatial attention and multi-dimensional information is characterized in that: the calculation method of F i-h is: where DSConv(·) represents the depthwise separable convolution operation, and D(·) represents the adaptive max pooling operation.

[0028] The above-mentioned pathological image reconstruction method based on spatial attention and multi-dimensional information is characterized in that: the specific method for obtaining the attention map of the i-th block according to the comprehensive feature M i-h is: according to the formula A i-h = Activation(C i-h ) to calculate the activation feature A i-h , where Activation represents passing through the activation layer, C i-h = Conv 1×1 (M i-h ), C i-h represents the comprehensive feature M i-h passing through the 1×1 convolutional layer;

[0029] Upsample the activation feature A i-h to obtain the attention map of the i-th block.

[0030] The above-mentioned pathological image reconstruction method based on spatial attention and multi-dimensional information is characterized in that: the steps for obtaining the multi-dimensional information feature map I 2-up-h are:

[0031] According to the formula get the multi-dimensional information feature map I 2-up-h , where; CAM(·) represents passing through the channel attention module, HFE(·) represents passing through the high-frequency information enhancement module; SAM(·) represents passing through the spatial attention module, I sum-hRepresents the attention-enhanced feature map.

[0032] The above-mentioned pathological image reconstruction method based on spatial attention and multi-dimensional information is characterized in that: the spatial attention feature map I 1-up-h and the multi-dimensional information feature map I 2-up-h The specific method for is: adding the spatial attention feature map I 1-up-h and the multi-dimensional information feature map I 2-up-h The result of the addition is passed through a 1×1 convolutional layer and an activation layer to obtain a multi-dimensional attention feature map. The multi-dimensional attention feature map is multiplied by the feature map F shallow-h after normalization to obtain the enhanced feature map I up-h .

[0033] The above-mentioned pathological image reconstruction method based on spatial attention and multi-dimensional information is characterized in that: step 202 can be repeated x times. The hierarchical fusion feature map output by the (x - 1)-th feature learning is used as the input of the x-th feature learning, and the output of the x-th feature learning is used as the input of step 203.

[0034] The above-mentioned pathological image reconstruction method based on spatial attention and multi-dimensional information is characterized in that: the loss function of the super-resolution reconstruction model uses pred i-h represents the pixel value of the i-th pixel point of the h-th super-resolution reconstructed image I rebuild-h , target i-h represents the pixel value of the i-th pixel point of the h-th high-resolution pathological image I HR-h , D i-h represents the high-frequency region mask of the i-th pixel point of the h-th high-resolution pathological image I HR-h , and N is the total number of pixels.

[0035] The above-mentioned pathological image reconstruction method based on spatial attention and multi-dimensional information is characterized in that: the loss function of the super-resolution reconstruction model is L = L 小 + L1, where L 1 represents the mean absolute error loss function, pred i-h represents the pixel value of the i-th pixel point of the h-th super-resolution reconstructed image I rebuild-h , target i-h represents the pixel value of the i-th pixel point of the h-th high-resolution pathological image I HR-h , D i-h represents the high-frequency region mask of the i-th pixel point of the h-th high-resolution pathological image I HR-h , and N is the total number of pixels.

[0036] The present invention has the following advantages compared with the prior art:

[0037] 1. The structure of the present invention is simple and reasonably designed, and the implementation and use operations are convenient.

[0038] 2. The present invention fuses the spatial attention feature map and the multi-dimensional information feature map to obtain the super-resolution reconstruction image, which can comprehensively combine the spatial information and the channel information. The super-resolution reconstruction image synthesizes a variety of key information, enabling the reconstructed high-resolution image to more accurately restore both in the spatial layout and the channel features, improving the clarity of the image, making the reconstructed high-resolution pathological image closer to the real original image, and helping doctors more accurately identify the lesion features.

[0039] 3. The present invention uses the spatial attention feature map to describe the spatial information of the image, calculates by combining its own features with the spatial distribution features, restores the spatial position relationship between the pathological images, emphasizes the importance of different spatial positions, and can capture the spatial distribution characteristics and mutual relationships of different regions in the pathological image.

[0040] 4. The present invention processes and suppresses the influence of irrelevant or unimportant channels through the channel attention module, enhances the high-frequency components through the high-frequency information enhancement module to restore details, and strengthens the spatial position information through the spatial attention module. By combining these three complementary features to obtain the multi-dimensional information feature map, it retains the overall semantic and structural information while enhancing the detail features, enabling the reconstructed pathological image to be closer to the original image and reducing the information loss and ambiguity during the reconstruction process.

[0041] 5. The channel fusion module CFM of the present invention makes the model more delicate in processing local details during the channel transformation by expanding and compressing the number of channels, thereby improving the resolution and clarity of the reconstructed image.

[0042] 6. The present invention obtains the hybrid fusion feature map through multi-stage feature fusion, ensuring that the basic information is fully incorporated into the final result, making the reconstructed image more complete both in terms of details and overall structure.

[0043] In summary, the present invention has a simple structure and reasonable design, fuses the spatial attention feature map and the multi-dimensional information feature map, can comprehensively combine the spatial information and the channel information; then fuses the CFM fusion feature map to delicately process the local details of the pathological image; and then obtains the hybrid fusion feature map through multi-stage feature fusion to ensure that the basic information is fully incorporated into the final result; making the reconstructed high-resolution pathological image closer to the real original image and improving the resolution and quality of the image.

[0044] The technical solution of the present invention will be further described in detail below with reference to the drawings and embodiments. Brief Description of the Drawings

[0045] Figure 1 This is the flowchart of the method of the present invention.

[0046] Figure 2 This is the flowchart of the super-resolution reconstruction model of the present invention.

[0047] Figure 3 This is the flowchart of obtaining the spatial attention feature map of the present invention.

[0048] Figure 4 This is the flowchart of obtaining the multi-dimensional information feature map of the present invention.

[0049] Figure 5 This is the flowchart of obtaining the enhanced feature map of the present invention.

[0050] Figure 6 This is the flowchart of obtaining the super-resolution reconstruction image of the present invention. Detailed implementation manners

[0051] The method of the present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments of the present invention.

[0052] It should be noted that, without conflict, the embodiments in the present application and the features in the embodiments may be combined with each other. The present invention will be described in detail below with reference to the accompanying drawings and embodiments.

[0053] It should be noted that the terms used herein are only for describing specific implementation manners and are not intended to limit the exemplary implementation manners according to the present application. As used herein, unless otherwise clearly specified in the context, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0054] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above accompanying drawings are used to distinguish similar objects and do not necessarily need to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances so that the implementation manners of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units does not necessarily need to be limited to those steps or units clearly listed, but may include other steps or units not clearly listed or inherent to these processes, methods, products, or devices.

[0055] For ease of description, spatial relative terms such as "above", "over", "on the upper surface", "upper", etc. may be used herein to describe the spatial positional relationship of one device or feature to other devices or features as shown in the figures. It should be understood that the spatial relative terms are intended to encompass different orientations in use or operation in addition to the orientation depicted in the figures. For example, if the device in the figures is inverted, the device described as "above" or "over" other devices or structures will then be positioned "below" or "beneath" the other devices or structures. Thus, the exemplary term "above" can include both the orientations of "above" and "below". The device may also be positioned in other different ways (rotated 90 degrees or in other orientations), and the corresponding explanations are made for the spatial relative descriptions used herein.

[0056] As Figure 1 shown, a method for pathological image reconstruction based on spatial attention and multi-dimensional information according to the present invention includes the following steps:

[0057] Step 1: Obtain a dataset W, w h represents the h-th image pair, and w h =(I LR-h , I HR-h ), where I LR-h represents the low-resolution pathological image in the h-th image pair, and I HR-h represents the high-resolution pathological image in the h-th image pair, 0 < h < m.

[0058] It should be noted that it is ensured that the dataset has sufficient diversity, including different types of pathological images, different lesion conditions, and different tissue sections. The images are preprocessed, such as operations like resizing and normalization, to make the images meet the requirements for input to the super-resolution reconstruction model.

[0059] The low-resolution pathological image is a tissue section image obtained by an optical microscope or other imaging techniques, and its resolution is lower than the ideal resolution required for diagnosis.

[0060] To expand the dataset, the high-resolution pathological image I HR-h is downsampled to generate the corresponding low-resolution pathological image I LR-h .

[0061] As Figures 2 to 6 shown, in this embodiment, the method for establishing a super-resolution reconstruction model in Step 2 is as follows:

[0062] Step 201: Obtain the feature map F LR-h of the low-resolution pathological image I shallow-hSpecifically, F shallow-h = Conv(I LR-h ), where Conv() represents the feature extraction module. The low-resolution pathological image I LR-h is a 3-channel RGB image. The feature extraction module uses a two-dimensional convolutional layer, Conv2d(3, dim, 3, 1, 1)). The two-dimensional convolutional layer converts the input 3-channel RGB image into a feature map F shallow-h with the dimension of dim. The size of the feature map F shallow-h is (B, dim, H, W), where H represents the height, W represents the width, dim represents the number of channels, and B represents the batch size. In a possible embodiment, dim is set to 36. The feature map F shallow-h obtained by shallow feature extraction captures local information of the image, such as basic features like texture, edges, and basic color information, providing a basic feature representation for subsequent operations.

[0063] Step 202: Feature learning:

[0064] Step 2021: The feature map F shallow-h is normalized and then passed through the spatial attention feature extraction module to obtain the spatial attention feature map I 1-up-h ; the feature map F shallow-h is normalized and then passed through the multi-dimensional information feature extraction module to obtain the multi-dimensional information feature map I 2-up-h ; the spatial attention feature map I 1-up-h and the multi-dimensional information feature map I 2-up-h are fused to obtain the enhanced feature map I up-h .

[0065] In a possible embodiment, the steps to obtain the spatial attention feature map I 1-up-h are as follows:

[0066] Step 2021a: The feature map F shallow-h is normalized and then divided into n parts along the channels. B i-h represents the i-th block of the h-th low-resolution pathological image, where 0 < i < n. In actual use, by dividing the feature map into blocks, more detailed information can be mined from different channel dimensions. For pathological images, by considering their own features in blocks, the features of different cell components in specific channels can be captured more carefully, such as the characteristics of the nucleus and cytoplasm in different channels, which helps to analyze abnormal changes in cells, such as the enlargement and deformation of the nucleus. At the same time, the limitation of single-channel information can be avoided. When reconstructing high-resolution images, the image can contain richer features, improving the quality and information content of the image, and helping to discover more potential pathological features and signs of lesions.

[0067] Step 2021b: Based on each block Bi-h Calculate the comprehensive feature M based on its own characteristics and spatial distribution characteristics i-h .

[0068] By calculating the combination of its own characteristics and spatial distribution characteristics, the position information and context information of each block in space can be taken into account. For pathological images, the spatial distribution of tissues reflects the degree and scope of lesions. Analyzing the spatial distribution characteristics of each block can accurately restore information such as the spatial position relationship between cells and the hierarchical structure of tissues. When reconstructing the image, the boundaries of tissues can be made clearer, and the transition between different tissue regions can be made more natural, so as to better present the true morphology of pathological tissues and provide more accurate image basis for pathological diagnosis.

[0069] In addition, the method of block calculation can, to a certain extent, suppress the influence of noise and artifacts. At the same time, the spatial distribution characteristics can be used to correct the distortion of the image structure caused by artifacts. When reconstructing high-resolution images, the clarity and accuracy of the images can be improved, and the interference of noise and artifacts on pathological diagnosis can be reduced.

[0070] In actual use, according to the formula calculate the comprehensive feature M of block B i-h , where x i-h represents the r-th tensor value of B i-h-r , μ represents the tensor mean, N represents the number of tensor values, σ i-h (B 2 ) represents the variance of B i-h along the channel direction, α and β respectively represent learnable parameters, and F i-h represents the feature of B i-h . i-h

[0071] It should be noted that the calculation method of F i-h is as follows: where DSConv(·) represents the depthwise separable convolution operation, and D(·) represents the adaptive max pooling operation.

[0072] Perform a depthwise separable convolution operation on the first block. In pathological images, local details such as cell morphology and tissue structure are rich. The depthwise separable convolution can accurately analyze these local features and clearly present the fine details such as the boundaries of cells and the morphology of cell nuclei.

[0073] ​Adaptive maximum pooling is first performed on other blocks. Global features such as the overall cell distribution and tissue lesion range of the pathological image can be extracted through adaptive maximum pooling operations, quickly locating the approximate location and range of the lesion area, and then performing a deep separable convolution operation. Based on the global features that have been mastered, local features such as the specific morphology of lesion cells and subtle differences between cells are further mined. Combining global information with local information allows the model to have a deeper understanding of semantic information at different levels from macro to micro in pathological images.

[0074] It should be noted that the calculation of F i-h When , the convolution kernel of the depthwise separable convolution operation is 3×3.

[0075] Step 2021c: Based on the comprehensive feature M i-h Get the attention map of the i-th block, the attention map and the block B i-h Multiply them together to get the fused block of the i-th block.

[0076] According to the comprehensive characteristics M i-h The specific method to obtain the attention map of the i-th block is:

[0077] According to formula A i-h =Activation(C i-h ) Calculate the activation feature A i-h , where Activation means passing through the activation layer, C i-h =Conv 1×1 (M i-h ), C i-h Represents the comprehensive feature M i-h After 1×1 convolution layer;

[0078] For activation feature A i-h Upsampling is performed to obtain the attention map of the i-th block.

[0079] It should be noted that the calculation of activation feature A i-h The activation function used is the GELU function.

[0080] In pathological images, only some areas often contain lesion information of diagnostic value. Attention maps can automatically focus on these key areas or small special structures. For example, in the reconstruction of tumor pathological images, it can highlight the characteristics of tumor cell aggregation areas, suppress the influence of normal tissue area characteristics on tumor characteristics, and make subsequent reconstruction more focused on the characteristics of lesions, thereby improving the correctness of tumor pathological image reconstruction and helping to improve the accuracy of subsequent lesion detection.

[0081] Step 2021d: Splice all fused blocks to obtain the spatial attention feature map I 1-up-h。

[0082] Obtain the multi-dimensional information feature map I 2-up-h The steps are as follows:

[0083] According to the formula Get the multi-dimensional information feature map I 2-up-h , where; CAM(·) represents being processed by the channel attention module, HFE(·) represents being processed by the high-frequency information enhancement module; SAM(·) represents being processed by the spatial attention module, and I sum-h represents the attention-enhanced feature map.

[0084] First of all, different channels in the pathological image contain different types of information, such as cell morphology, tissue structure, color features, etc. The channel attention module can automatically learn the importance weights of each channel, assign higher weights to the channels containing key pathological information, highlight the features of these channels, and suppress the influence of irrelevant or unimportant channels, so that the obtained result is more focused on the information valuable for reconstructing high-resolution images.

[0085] Secondly, the low-resolution pathological image I LR-h often loses a lot of high-frequency detail information, and after being processed by the high-frequency information enhancement module HFE(·), it can enhance the normalized high-frequency components, and these high-frequency components correspond to details such as cell boundaries, nuclear textures, and tissue fibers in the pathological image. When reconstructing high-resolution images, these details can be better restored, improving the clarity and readability of the images. shallow-h Furthermore, CAM(F

[0086] ) contains the features after channel attention screening and integration, emphasizing the importance and relevance between channels; HFE(CAM(F shallow-h )) highlights the details and high-frequency features. Adding the two together gives the attention-enhanced feature map I shallow-h , which not only combines the advantages of these two features, retains the overall semantic and structural information, but also enhances the detail features, providing a richer and more comprehensive feature combination for reconstructing high-resolution images, enabling the reconstructed image to more accurately present pathological features at both the macroscopic and microscopic levels. sum-h I

[0087] I sum-h After being processed by the spatial attention module, the features at different positions are weighted, so that when reconstructing high-resolution images, it can focus on the key regions in the pathological image, such as the lesion site, the abnormal cell aggregation region, etc. By highlighting the features of these key regions and suppressing the interference of irrelevant regions, it can more clearly show the important structures and features in the pathological image, providing a more accurate diagnostic basis for doctors.

[0088] After the previous steps such as channel attention and high-frequency information enhancement, I is obtained. sum-h Then, through the spatial attention module, the information in the channel dimension, high-frequency detail dimension, and spatial dimension can be deeply fused to form a highly comprehensive feature representation containing multi-dimensional information. This multi-dimensional feature representation can describe the features of pathological images more comprehensively and accurately, provide stronger feature support for reconstructing high-resolution images, thereby improving the quality and accuracy of the reconstructed images, and helping to perform pathological analysis and diagnosis more accurately.

[0089] Fused spatial attention feature map I 1-up-h and multi-dimensional information feature map I 2-up-h The specific method is as follows:

[0090] Add the spatial attention feature map I 1-up-h and the multi-dimensional information feature map I 2-up-h . The result of the addition passes through a 1×1 convolutional layer and an activation layer to obtain a multi-dimensional attention feature map. The multi-dimensional attention feature map is multiplied by the feature map F shallow-h after normalization to obtain the enhanced feature map I up-h .

[0091] The spatial attention feature map I 1-up-h mainly focuses on the spatial information of the image, emphasizes the importance of different spatial positions, and can capture the spatial distribution characteristics and interrelationships of different regions in the pathological image. The multi-dimensional information feature map I 2-up-h not only contains spatial information, but also filters and enhances the channel information through the channel attention module, and highlights the high-frequency detail information through the high-frequency information enhancement module.

[0092] Specifically, add the spatial attention feature map I 1-up-h and the multi-dimensional information feature map I 2-up-h . The result of the addition passes through a 1×1 convolutional layer and an activation layer to obtain a multi-dimensional attention feature map. The multi-dimensional attention feature map is multiplied by the feature map F shallow-h after normalization to obtain the enhanced feature map I up-h .

[0093] Fused spatial attention feature map I 1-up-h and multi-dimensional information feature map I 2-up-h, the spatial information and channel information can be comprehensively combined. The fused feature map integrates various key information, enabling the reconstructed high-resolution image to be more accurately restored in terms of both spatial layout and channel features. For example, it can clearly display the location and morphology of the lesion area, and accurately present the differences in channel features such as color and texture of different cell types, improving the clarity of the image and making the reconstructed high-resolution pathological image closer to the real original image, which helps doctors more accurately identify lesion features and make diagnoses.

[0094] Step 2022, feature map F shallow-h and enhanced feature map I up-h are added together to obtain the retained enhanced feature map I sha-up-h ;

[0095] Enhanced feature map I up-h and feature map F shallow-h are added together, which can integrate the basic information in feature map F shallow-h and the enhanced information in enhanced feature map I up-h , enabling subsequent processing to utilize both parts of information simultaneously, thus enriching the feature information; the addition makes features at different levels and of different types complement each other, reducing the noise that may exist in a single feature map.

[0096] Step 2023, the retained enhanced feature map is normalized and then passed through the channel fusion module CFM to output the CFM fused feature map I CFM ;

[0097] When reconstructing high-resolution pathological images, the channel fusion module CFM makes the model handle local details more delicately during channel transformation and pays more attention to the restoration of these details by expanding and compressing the number of channels, thereby improving the resolution and clarity of the reconstructed image.

[0098] It should be noted that the processing steps of the channel fusion module CFM are as follows:

[0099] Step 2023a, receive the retained enhanced feature map I sha-up-h ;

[0100] Step 2023b, I sha-up-h is expanded through a 3×3 convolutional layer to increase its dimension, which is defaulted to twice the original dimension, and the GELU activation function is used to enhance the non-linear expression ability.

[0101] Step 2023c, further process through a 1×1 convolutional layer, and use the GELU activation function to enhance the non-linear expression ability.

[0102] Step 2023d, compress the dimension of the feature map back to the original dim through a 1×1 convolutional layer to obtain I CFM。

[0103] Step 2024: Retain the enhanced feature map I sha-up-h Fuse with the CFM fused feature map I CFM Add them together to obtain the hierarchical fused feature map I sha-up-CFM-h 。

[0104] Retain the enhanced feature map I sha-up-h Fuse with the CFM fused feature map I CFM Adding them together can, on the one hand, retain the original information, prevent information loss, and maintain the integrity of the features; on the other hand, combining features at different levels enhances complementarity, thereby enhancing details, improving the reconstruction effect, and enhancing the image quality.

[0105] Step 203: Hierarchically fuse the feature map I sha-up-CFM-h Add it to the feature map F shallow-h Add them together to obtain the hybrid fused feature map I sha-up-CFM-sha-h ;

[0106] The feature map F shallow-h Contains the basic information initially extracted from the pathological image, covering aspects such as the basic structure of the image and cell morphology. Throughout the processing flow, although the hierarchical fused feature map I sha-up-CFM-h is obtained through feature learning, shallow-h some underlying detailed information in the feature map F sha-up-CFM-sha-h may not be fully replaced by other processing procedures. By adding them together in the last step to obtain the hybrid fused feature map I

[0107] can ensure that this basic information is fully incorporated into the final result, and multi-stage feature fusion makes the reconstructed image more complete in terms of details and overall structure. sha-up-CFM-sha-h The hybrid fused feature map I rebuild-h Passes through the convolutional layer and the pixel shuffling layer to obtain the super-resolution reconstructed image I

[0108] The main function of the pixel shuffling layer is to perform upsampling, rearranging the channel information of the input feature map into higher-resolution spatial information, that is, rearranging the channel information of the features to generate a high-resolution image. Compared with traditional interpolation methods, the pixel shuffling layer can better retain the detail and texture information of the image, thereby achieving a higher-quality reconstruction effect.

[0109] Step Three: Define the loss function, train the super-resolution reconstruction model, and obtain the final super-resolution reconstruction model.

[0110] Step 205: Define the loss function, train the super-resolution reconstruction model, and obtain the final super-resolution reconstruction model.

[0111] The specific method for training the super-resolution reconstruction model is to divide the dataset W into a training set, a validation set, and a test set.

[0112] Use the training set to iteratively train the super-resolution reconstruction model. Input the low-resolution pathological images into the super-resolution reconstruction model to obtain the generated super-resolution reconstruction image I. rebuild-h According to the generated super-resolution reconstruction image I rebuild-h and the corresponding real high-resolution pathological image I HR-h calculate the loss. According to the calculated loss, use the backpropagation algorithm to calculate the gradient of the loss with respect to the parameters of the super-resolution reconstruction model, and update the model parameters according to the gradient to gradually reduce the loss; when the preset conditions are reached, stop the training. Use the validation set to evaluate the performance of the super-resolution reconstruction model, and select the super-resolution reconstruction model with the best performance on the validation set as the final super-resolution reconstruction model; the preset conditions refer to the preset number of iterations.

[0113] The loss function of the super-resolution reconstruction model uses pred i-h to represent the pixel value of the i-th pixel of the h-th super-resolution reconstruction image I rebuild-h , target i-h to represent the pixel value of the i-th pixel of the h-th high-resolution pathological image I HR-h , D i-h to represent the high-frequency region mask of the i-th pixel of the h-th high-resolution pathological image I HR-h , and N1 is the total number of pixels.

[0114] In a possible embodiment, convert target i-h to a grayscale image, perform wavelet transform, select the high-frequency components, sum the absolute values of the high-frequency components along the last dimension to obtain an amplitude map, divide each amplitude map by the maximum value of the amplitude map, and normalize it to between 0 and 1. For each normalized data value, set a judgment rule: if each value is greater than or equal to the threshold, set its corresponding mask value to 1, and if it is less than the threshold, set its corresponding mask value to 0. Usually, the threshold is defaulted to 0.5. To make the mask match the pixel values of the pathological image, upsample the mask to the size of target i-h . The mask after the upsampling process is the final high-frequency region mask D i-h .

[0115] Through the emphasis on high-frequency information by the high-frequency region mask D i-h , the super-resolution reconstruction model more specifically learns the high-frequency information in the data, making the super-resolution reconstruction model pay more attention to the reconstruction accuracy of the detailed parts, improving the clarity and quality of the pathological images, and thus enhancing the generalization ability of the model.

[0116] For example, in a lung cancer pathological image, the high-frequency region mask D i-h emphasizes the tumor edge; in a cerebral infarction pathological image, the high-frequency region mask D i-h more clearly presents the size and internal structure of the necrotic focus, such as information on cell debris and residual blood vessels within the necrotic focus.

[0117] Step Four: Obtain the low-resolution pathological image to be reconstructed, and input the low-resolution pathological image to be reconstructed into the final super-resolution reconstruction model to obtain a super-resolution reconstructed image.

[0118] The super-resolution reconstruction model of the present application can learn how to reconstruct a high-resolution image from a low-resolution pathological image, improve the resolution and quality of the image, and provide clearer and more detailed image information for pathological diagnosis and differential diagnosis applications.

[0119] Example Two

[0120] Different from Example One, in this example, the loss function of the super-resolution reconstruction model is that the loss function of the super-resolution reconstruction model is L = L 小 +L 1 where L 1 represents the mean absolute error loss function, pred i-h represents the pixel value of the i-th pixel of the h-th super-resolution reconstructed image I rebuild-h , target i-h represents the pixel value of the i-th pixel of the h-th high-resolution pathological image I HR-h , D i-h represents the high-frequency region mask of the i-th pixel of the h-th high-resolution pathological image I HR-h , and N1 is the total number of pixels.

[0121] In a possible embodiment, target i-h is converted into a grayscale image, wavelet transform is performed, high-frequency components are selected, the absolute values of the high-frequency components are added along the last dimension to obtain an amplitude map, the amplitude map is divided by the maximum value of the amplitude map, and normalized to between 0 and 1. For each normalized data value, a judgment rule is set: if each value is greater than or equal to the threshold, the corresponding mask value is set to 1, and if it is less than the threshold, the corresponding mask value is set to 0. Usually, the threshold is defaulted to 0.5. In order to enable the mask to match the pixel values of the pathological image, the mask is upsampled to the size of target i-h . The mask after the upsampling process is the final required high-frequency region mask D i-h .

[0122] L小 Represents the wavelet high-frequency detail loss function, which emphasizes high-frequency information through the high-frequency region mask D i-h By emphasizing high-frequency information, the super-resolution reconstruction model can more specifically learn the high-frequency information in the data, making the super-resolution reconstruction model pay more attention to the reconstruction accuracy of the details, improving the clarity and quality of the pathological images, and thus enhancing the generalization ability of the model.

[0123] For example, in lung cancer pathological images, the tumor edge is emphasized through the high-frequency region mask D i-h In cerebral infarction pathological images, the size and internal structure of the necrotic focus, such as information on cell debris and vascular residues within the necrotic focus, can be presented more clearly through the high-frequency region mask D i-h

[0124] The mean absolute error loss function L 1 can accurately quantify the difference between the pathological image output by the super-resolution reconstruction module and the real image. The mean absolute error loss function L 1 calculates the average of the absolute values of the differences between the corresponding pixels of the reconstructed image and the real image, focuses on the global error, and comprehensively reflects the degree to which the reconstructed image deviates from the real image. It has a good constraining effect on the structure and trend of the super-resolution reconstruction model, enabling the super-resolution reconstruction model to learn the main structure and trend of the case images, such as the general shape of the image, the contour of the object, and the overall rhythm of the audio.

[0125] When the wavelet high-frequency detail loss function L 小 is added to the mean absolute error loss function L 1 the wavelet high-frequency detail loss function L 小 focuses on high-frequency detail features, and the mean absolute error loss function L 1 focuses on the global. After the two are combined, it can not only ensure the correctness of the overall structure but also ensure the accuracy of the details, enabling the features learned by the super-resolution reconstruction model to be more comprehensive and accurate, and making the reconstructed image closer to the original high-resolution pathological image both in terms of the whole and the details.

[0126] Embodiment III

[0127] The difference between this Embodiment I and Embodiment II is that in this embodiment, step 202 can be repeated x times. The hierarchical fusion feature map output by the (x - 1)-th feature learning is used as the input for the x-th feature learning, and the output of the x-th feature learning is used as the input for step 203.

[0128] In a possible embodiment, the feature learning of step 202 passes through a feature learning layer, and the number of layers of the feature learning layer is 8. During use, the parameters in each enhanced feature learning layer need to be adjusted to adapt to different tasks and datasets. ​

[0129] Starting from the first feature learning layer, the features are gradually refined and optimized. Subsequent feature learning layers continue to learn based on the previous one, continuously accumulating and adjusting the feature representation. Finally, the hierarchical fusion feature map output by the 8th feature learning layer is obtained. This feature map has undergone detailed feature learning at multiple levels and has fused features at different levels.

[0130] As described above, the above are only embodiments of the present invention and do not impose any limitations on the present invention. Any simple modifications, changes, and equivalent structural changes made to the above embodiments according to the technical essence of the present invention still fall within the protection scope of the technical solution of the present invention.

Claims

1. A pathological image reconstruction method based on spatial attention and multi-dimensional information, characterized in that: The following steps are involved: Step 1: Get the dataset W. w h represents the hth image pair, w h =(I LR-h , I HR-h ), I LR-h represents the low-resolution pathological image in the h-th image pair, I HR-h represents the high-resolution pathological image in the h-th image pair, 0 <h<m; Step 2: Establish a super-resolution reconstruction model; Step 201: Obtain low-resolution pathological image I LR-h The feature map F shallow-h ; Step 202: Feature learning: Step 2021: Feature map F shallow-h After normalization, it passes through the spatial attention feature extraction module to obtain the spatial attention feature map I 1-up-h ; Feature map F shallow-h After normalization, the multi-dimensional information feature extraction module is used to obtain the multi-dimensional information feature map I 2-up-h ; Fusion spatial attention feature map I 1-up-h And multi-dimensional information feature map I 2-up-h , get the enhanced feature map I up-h ; Step 2022: Feature map F shallow-h With enhanced feature map I up-h Add together to obtain the retained enhanced feature map I sha-up-h ; Step 2023: The retained enhanced feature map is normalized and then passed through the channel fusion module CFM to output the CFM fusion feature map I CFM ; Step 2024: retain enhanced feature map I sha-up-h Fusion feature map I with CFM CFM Add together to get the hierarchical fusion feature map I sha-up-CFM-h ; Step 203: Layered fusion feature map I sha-up-CFM-h With the feature map F shallow-h Add together to get the mixed fusion feature map I sha-up-CFM-sha-h ; Step 204: Mixed fusion feature map I sha-up-CFM-sha-h After the convolution layer and the pixel shuffling layer, the super-resolution reconstructed image I is obtained. rebuild-h ; Step 3: Define the loss function, train the super-resolution reconstruction model, and obtain the final super-resolution reconstruction model; Step 4: Obtain a low-resolution pathological image to be reconstructed, and input the low-resolution pathological image to be reconstructed into a final super-resolution reconstruction model to obtain a super-resolution reconstructed image.

2. A pathological image reconstruction method based on spatial attention and multi-dimensional information according to claim 1, characterized in that: Get spatial attention feature map I 1-up-h The steps are: Step 2021a: transform the feature map F shallow-h After normalization, the channel is divided into n parts, B i-h represents the i-th block of the h-th low-resolution pathological image, 0<i<n; Step 2021b: Based on each block B i-h The comprehensive feature M is calculated by its own characteristics and spatial distribution characteristics i-h ; Step 2021c: Based on the comprehensive feature M i-h Get the attention map of the i-th block, the attention map and the block B i-h Multiply them together to get the fused block of the i-th block; Step 2021d: Splice all fused blocks to obtain the spatial attention feature map I 1-up-h .

3. A pathological image reconstruction method based on spatial attention and multi-dimensional information according to claim 2, characterized in that: The method for calculating the comprehensive features in step 2021b is: according to the formula Calculate block B i-h The comprehensive characteristics of M i-h , X i-h-r Indicates B i-h The rth tensor value of , μ represents the tensor mean, N represents the number of tensor values, σ 2 (B i-h ) means B i-h The variance along the channel direction, α and β represent the learnable parameters, F i-h Indicates B i-h characteristics.

4. A pathological image reconstruction method based on spatial attention and multi-dimensional information according to claim 3, characterized in that: F i-h The calculation method is: Where DSConv(·) represents the depthwise separable convolution operation and D(·) represents the adaptive max pooling operation.

5. A pathological image reconstruction method based on spatial attention and multi-dimensional information according to claim 2, characterized in that: According to the comprehensive characteristics M i-h The specific method to obtain the attention map of the i-th block is: According to formula A i-h =Activation(C i-h ) Calculate the activation feature A i-h , where Activation means passing through the activation layer, C i-h =Conv 1×1 (M i-h ), C i-h Represents the comprehensive feature M i-h After a 1×1 convolutional layer; For activation feature A i-h Upsampling is performed to obtain the attention map of the i-th block.

6. A pathological image reconstruction method based on spatial attention and multi-dimensional information according to claim 1, characterized in that: Obtain multi-dimensional information feature map I 2-up-h The steps are: According to the formula Get multi-dimensional information feature map I 2-up-h , where CAM(·) means processed by the channel attention module, HFE(·) means processed by the high-frequency information enhancement module; SAM(·) means processed by the spatial attention module, I sum-h Represents the attention-enhanced feature map.

7. A pathological image reconstruction method based on spatial attention and multi-dimensional information according to claim 1, characterized in that: Fusion of spatial attention feature map I 1-up-h And multi-dimensional information feature map I 2-up-h The specific method is: Spatial Attention Feature Map I 1-up-h And multi-dimensional information feature map I 2-up-h To add, The result of the addition is passed through a 1×1 convolutional layer and an activation layer to obtain a multi-dimensional attention feature map. Multi-dimensional attention feature map and feature map F shallow-h After normalization and multiplication, we get the enhanced feature map I up-h .

8. A pathological image reconstruction method based on spatial attention and multi-dimensional information according to claim 1, characterized in that: Step 202 may be repeated x times, and the hierarchical fusion feature map output by the x-1th feature learning is used as the input of the xth feature learning, and the output of the xth feature learning is used as the input of step 203.

9. A pathological image reconstruction method based on spatial attention and multi-dimensional information according to claim 1, characterized in that: The loss function of the super-resolution reconstruction model is pred i-h represents the hth super-resolution reconstructed image I rebuild-h The pixel value of the i-th pixel, target i-h represents the hth high-resolution pathological image I HR-h The pixel value of the i-th pixel, D i-h Represented as the hth high-resolution pathological image I HR-h The high-frequency area mask of the i-th pixel, where N is the total number of pixels.

10. A pathological image reconstruction method based on spatial attention and multi-dimensional information according to claim 1, characterized in that: The loss function of the super-resolution reconstruction model is L = L 小 +L1, where L1 represents the mean absolute error loss function, pred i-h represents the hth super-resolution reconstructed image I rebuild-h The pixel value of the i-th pixel, target i-h represents the hth high-resolution pathological image I HR-h The pixel value of the i-th pixel, D i-h Represented as the hth high-resolution pathological image I HR-h The high-frequency area mask of the i-th pixel, where N is the total number of pixels.

Images