Pathological image analysis method and system based on deep learning

Abstract

The invention discloses a pathological image analysis method and system based on deep learning, and relates to the technical field of image processing, and the method comprises the steps: collecting a to-be-detected pathological image, and constructing a data set; constructing a U-Net network, and performing image segmentation on the data set image through the U-Net network to obtain a segmentation result; preprocessing the segmentation result to obtain an optimization result; constructing a classification model, and performing image classification on an optimization result through the classification model; and carrying out visual display on the image classification result. According to the method, each segmented image region is classified carefully through the classification model, and different cell types or pathological states are effectively identified; the combination of the U-Net and the classification model is used for processing a complex image analysis task which needs to perform accurate image segmentation and then perform detailed classification, and the high precision of the U-Net in the aspect of image segmentation and the high efficiency of the classification model in the aspect of classification tasks are utilized, so that the accuracy of segmentation and classification is greatly improved through a combined application strategy.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and more specifically to a pathological image analysis method and system based on deep learning. Background Art

[0002] Pathological image analysis is a key branch in the field of medical image processing. It can be used to interpret the complex information in medical pathological images by making full use of computer vision and image processing technology. The technological evolution can be divided into two categories: traditional methods and deep learning-based methods. Traditional methods played an important role in the early stage of pathological image analysis. This type of method focuses on using edge detection, morphological operations, texture and color analysis and other technical means to extract useful information from images. Cell boundaries can be identified through edge detection algorithms, and morphological operations help to clean up image noise or enhance structural features. Texture analysis is often used to identify lesion areas, while color analysis is particularly useful for distinguishing tissue types. These methods often rely on expert-designed feature extractors combined with machine learning algorithms such as support vector machines (SVM) or decision trees for image classification and analysis. Although these traditional methods are still effective in some specific applications, their performance is limited by the expressiveness and generalization of manual features in the face of complex pathological images.

[0003] With the development of deep learning methods, pathological image analysis technology has achieved a qualitative leap. In particular, the introduction of convolutional neural networks (CNNs) marks a major breakthrough in this field. Compared with traditional methods, deep learning technology can automatically learn complex feature representations from large amounts of data without the need for manual feature design. This advantage significantly improves the accuracy and efficiency of pathological image analysis. Deep learning technology has been successfully applied to multiple tasks such as image classification, recognition, segmentation, and target detection, greatly enriching the tools for automatic pathological image analysis. Image segmentation using CNN can accurately segment the lesion area, and the outstanding performance of deep learning models in image recognition tasks also provides strong support for the automatic diagnosis of pathological images.

[0004] In the development of deep learning methods, there are not only basic network structures for general image processing such as AlexNet, VGGNet, and ResNet, but also network structures and algorithms designed specifically for medical image characteristics (large size, high dimension, subtle features, etc.). These specialized technologies further enhance the accuracy and efficiency of pathological image analysis. However, although deep learning has achieved remarkable results in pathological image analysis, these results rely on a large amount of labeled data. The field of pathological image analysis has great development potential. With the continuous advancement of deep learning technology and the continuous emergence of new algorithms, higher requirements are expected for the accuracy, efficiency, and degree of automation of pathological image analysis. Heat maps are an indispensable visualization tool in the decision-making process of deep learning models. The key lies in improving the transparency and interpretability of the model. The current heat map generation technology still faces a series of limitations and challenges.

[0005] Therefore, how to propose a pathological image analysis method and system based on deep learning, develop a U-Net and EfficientNet joint network structure and algorithm to address the large size and high dimensionality problems unique to medical images, improve the performance of pathological image analysis, and significantly improve the heat map generation algorithm in terms of accuracy, interpretation ability, and integration with medical expertise. Improving the accuracy and efficiency of pathological image analysis and enhancing doctors' understanding of the basis for model judgment are issues that technical personnel in this field urgently need to solve. Summary of the invention

[0006] In view of this, the present invention provides a pathological image analysis method and system based on deep learning. Aiming at the large size and high dimensionality problems unique to medical images, the U-Net and EfficientNet joint network structure and algorithm are developed to improve the performance of pathological image analysis, and the heat map generation algorithm is significantly improved in terms of accuracy, interpretation ability and combination with medical expertise, thereby improving the accuracy and efficiency of pathological image analysis and enhancing doctors' understanding of the basis for model judgment. In order to achieve the above purpose, the present invention adopts the following technical solutions:

[0007] A pathological image analysis method based on deep learning, comprising:

[0008] Collect the pathological images to be tested and build a data set;

[0009] Construct a U-Net network, perform image segmentation on the dataset images through the U-Net network, and obtain the segmentation results;

[0010] Preprocess the segmentation results to obtain optimized results;

[0011] Build an EfficientNetV2 classification model and use it to classify images of the optimization results.

[0012] Visualize the image classification results.

[0013] Optionally, the U-Net network includes: a contraction path, an expansion path and a jump connection, the contraction path and the expansion path are structurally symmetrical, and the feature map of the contraction path is combined with the corresponding up-sampled feature map in the expansion path through the jump connection.

[0014] Optionally, the method further includes using a convolution operation in the contraction path to extract and learn image features; and in the expansion path, restoring the size of the image by transposed convolution to enlarge the spatial dimension of the image.

[0015] Optionally, the preprocessing of the segmentation result to obtain the optimized result includes: image formatting, normalization, image enhancement, image denoising and data enhancement processing.

[0016] Optionally, constructing the EfficientNetV2 classification model includes: replacing the MBConv block in the EfficientNet architecture with a Fused-MBConv block, the Fused-MBConv block fuses multiple convolution operations in the MBConv block, reduces the number of layers by fusing convolutional layers, and retains the network's ability to learn advanced features.

[0017] Optionally, the method further includes optimizing the EfficientNetV2 classification model: the EfficientNetV2 classification model adopts progressive learning, gradually increases the size of the image during the training process to reduce the overall training time, uses smaller images in the early stages, and gradually transitions to using larger images.

[0018] Optionally, the visual display of the image classification results includes generating a heat map based on the image classification results, introducing an attention layer in the EfficientNetV2 classification model, assigning an attention weight to each pixel or area through the attention layer, mapping the attention weight to the input image, obtaining a deep fusion heat map by weighted fusion of feature maps of different layers, and combining the prediction results of multiple models to generate a comprehensive heat map for visual display.

[0019] Optionally, generating a heat map includes: mapping numerical data in the heat map to a color space so that areas with different temperatures are presented in different colors; highlighting key features by removing non-critical signals or background noise in the heat map, enhancing the clarity and interpretability of the heat map, superimposing the processed pseudo-color image with the original image, and adjusting the intensity and manner of the superposition.

[0020] Optionally, it also includes improving the efficiency of heat map generation: reducing the number of model parameters and simplifying calculations through model pruning and quantization techniques, reducing model size and accelerating the inference process; using GPU acceleration and parallel computing frameworks to simultaneously perform data processing and model calculations on multiple computing cores; reducing network depth, using smaller convolution kernels and introducing depth-separable convolutions to design lightweight networks.

[0021] Optionally, a pathological image analysis system based on deep learning, comprising:

[0022] Acquisition module: used to collect pathological images to be tested and build data sets;

[0023] Segmentation module: used to build a U-Net network, perform image segmentation on the dataset image through the U-Net network, and obtain the segmentation result;

[0024] Optimization module: used to pre-process the segmentation results to obtain optimized results;

[0025] Classification module: used to build the EfficientNetV2 classification model and classify the images of the optimization results through the EfficientNetV2 classification model;

[0026] Visualization module: used to visualize the image classification results.

[0027] It can be seen from the above technical solutions that, compared with the prior art, the present invention discloses a pathological image analysis method and system based on deep learning, which has the following beneficial effects:

[0028] The present invention proposes a pathological image analysis method based on deep learning, including: collecting pathological images to be tested and constructing a data set; constructing a U-Net network, performing image segmentation on the data set image through the U-Net network to obtain a segmentation result; preprocessing the segmentation result to obtain an optimization result; constructing an EfficientNetV2 classification model, and performing image classification on the optimization result through the EfficientNetV2 classification model; and visually displaying the image classification result. The present invention adopts U-Net through its unique encoder-decoder structure and jump connection, which enables U-Net to perform well in extracting key areas in pathological images. By accurately segmenting the image, U-Net provides accurate input for subsequent image analysis steps; ensuring that the image area segmented by U-Net is suitable for the input requirements of EfficientNetV2, preprocessing operations such as resizing and normalization are required; using EfficientNetV2 for fast and effective image classification, and the segmented image area is then input into the EfficientNetV2 model for classification. EfficientNetV2 provides an efficient solution for image classification tasks through its optimized convolutional blocks and training methods. EfficientNetV2 performs detailed classification of each segmented image area and effectively identifies different cell types or pathological states. The combination of U-Net and EfficientNetV2 is used to handle complex image analysis tasks that require accurate image segmentation followed by detailed classification. This combination utilizes the high precision of U-Net in image segmentation and the high efficiency of EfficientNetV2 in classification tasks. The combined application strategy greatly improves the accuracy of segmentation and classification. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0030] Figure 1 A schematic flow chart of a pathological image analysis method based on deep learning provided by the present invention.

[0031] Figure 2 A structural framework diagram of a pathological image analysis system based on deep learning provided by the present invention.

[0032] Figure 3 This is a structural diagram of the U-Net model provided by the present invention.

[0033] Figure 4 This is a structural diagram of MBConv provided by the present invention.

[0034] Figure 5 This is a block structure diagram of Fused-MBConv provided by the present invention.

[0035] Figure 6 This is a flowchart of the combined application of U-Net and Effi cientNetV2 provided by the present invention.

[0036] Figure 7 This is the original slice image provided by the present invention.

[0037] Figure 8 This is a grayscale image of epithelial tissue provided by the present invention.

[0038] Fig. 9 This is a grayscale image of mesenchymal tissue provided by the present invention.

[0039] Fig.10 The present invention provides a thermal map of epithelial tissue.

[0040] Fig.11 This is a mesenchymal tissue heat map provided by the present invention.

[0041] Fig.12 A visualized heat map of the superimposed epithelial tissue provided by the present invention.

[0042] Fig.13 This is a visualized heat map of mesenchymal tissue after superposition provided by the present invention. DETAILED DESCRIPTION

[0043] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0044] The embodiment of the present invention discloses a pathological image analysis method based on deep learning, such as Figure 1 As shown, including:

[0045] Collect the pathological images to be tested and build a data set;

[0046] Construct a U-Net network, perform image segmentation on the dataset images through the U-Net network, and obtain the segmentation results;

[0047] Preprocess the segmentation results to obtain optimized results;

[0048] Build an EfficientNetV2 classification model and use it to classify images of the optimization results.

[0049] Visualize the image classification results.

[0050] Furthermore, the U-Net network includes: a contraction path, an expansion path and a jump connection. The contraction path and the expansion path are structurally symmetrical, and the feature map of the contraction path is combined with the corresponding up-sampled feature map in the expansion path through the jump connection.

[0051] Furthermore, it also includes using convolution operations in the contraction path to extract and learn image features; in the expansion path, restoring the size of the image through transposed convolution to enlarge the spatial dimension of the image.

[0052] Furthermore, the preprocessing of the segmentation result to obtain the optimized result includes: image formatting, normalization, image enhancement, image denoising and data enhancement processing.

[0053] Furthermore, the construction of the EfficientNetV2 classification model includes: replacing the MBConv block in the EfficientNet architecture with a Fused-MBConv block, the Fused-MBConv block fuses multiple convolution operations in the MBConv block, reduces the number of layers by fusing convolutional layers, and retains the network's ability to learn advanced features.

[0054] Furthermore, the method also includes optimizing the EfficientNetV2 classification model: the EfficientNetV2 classification model adopts progressive learning, gradually increases the size of the image during the training process to reduce the overall training time, uses smaller images in the early stages, and gradually transitions to using larger images.

[0055] Furthermore, the visual display of the image classification results includes generating a heat map according to the image classification results, introducing an attention layer in the EfficientNetV2 classification model, assigning an attention weight to each pixel or area through the attention layer, mapping the attention weight to the input image, obtaining a deep fusion heat map by weighted fusion of feature maps of different layers, combining the prediction results of multiple models, generating a comprehensive heat map for visual display.

[0056] Furthermore, the generation of the heat map includes: mapping the numerical data in the heat map to the color space so that areas with different temperatures are presented in different colors; highlighting key features by removing non-critical signals or background noise in the heat map, enhancing the clarity and interpretability of the heat map, superimposing the processed pseudo-color image with the original image, and adjusting the intensity and method of the superposition.

[0057] Furthermore, it also includes improving the efficiency of heat map generation: reducing the number of model parameters and simplifying calculations through model pruning and quantization technology, reducing model size and accelerating the inference process; using GPU acceleration and parallel computing frameworks to simultaneously perform data processing and model calculations on multiple computing cores; reducing network depth, using smaller convolution kernels and introducing depth-separable convolution to design lightweight networks.

[0058] In a specific embodiment, a pathological image analysis system based on deep learning, such as Figure 2 As shown, including:

[0059] Acquisition module: used to collect pathological images to be tested and build data sets;

[0060] Segmentation module: used to build a U-Net network, perform image segmentation on the dataset image through the U-Net network, and obtain the segmentation result;

[0061] Optimization module: used to pre-process the segmentation results to obtain optimized results;

[0062] Classification module: used to build the EfficientNetV2 classification model and classify the images of the optimization results through the EfficientNetV2 classification model;

[0063] Visualization module: used to visualize the image classification results.

[0064] In a specific implementation, data preprocessing is an important part of pathological image analysis, which is directly related to the efficiency and accuracy of subsequent analysis. The process includes several key steps.

[0065] (1) Image formatting

[0066] Image formatting converts images into a uniform format and size for subsequent processing, including adjusting image resolution and cropping unnecessary edges. This step ensures that images from different scanning devices can be analyzed with consistent standards.

[0067] (2) Normalization

[0068] Normalization is to scale the image data to a uniform scale, which helps the model learn and generalize faster. In this embodiment, normalization is performed to [0, 1]. The normalization formula is:

[0069]

[0070] In the formula, p min and p max Respectively represent the minimum and maximum values ​​of the image data in the entire dataset.

[0071] (3) Enhancement and noise reduction

[0072] Image enhancement can improve the contrast of the image and make the details clearer. Noise reduction uses Gaussian filtering to help remove noise from the image without losing important details. Noise reduction can be expressed by the following formula:

[0073]

[0074] Where (x, y) is the pixel location and σ is the standard deviation that controls the width of the Gaussian kernel.

[0075] (4) Data enhancement

[0076] Data augmentation helps the model learn the features of images when viewed from different angles, enhances the generalization ability of the model, and increases data diversity through operations such as rotation, flipping, and scaling.

[0077] After preprocessing is completed, subsequent processing such as edge detection and image segmentation is performed. The Sobel operator is used to calculate the gradient of the image to complete edge detection and identify the structural boundaries in the image. Image segmentation further subdivides the image into meaningful areas, making it easier to extract features for analysis.

[0078] In a specific implementation, in the field of pathological image analysis, the introduction of deep learning models marks a significant improvement in image recognition capabilities. In particular, in image segmentation and classification tasks, deep learning models have become core technologies with their powerful feature learning capabilities. In this embodiment, the combined use of U-Net and EfficientNetV2 can achieve accurate segmentation and efficient classification of pathological images, further optimizing the image processing process.

[0079] (1) Using U-Net for high-precision image segmentation

[0080] The U-Net deep learning model is used for medical image segmentation. It can obtain a high-precision model with less data training and can accurately segment objects in the image. The U-Net model architecture is as follows Figure 3 As shown. The U-Net model is a symmetrical U-shaped structure consisting of two main parts: a contraction path (encoder) and an expansion path (decoder). The model is designed for image segmentation tasks through its unique encoder-decoder structure and jump connections. This structure enables U-Net to perform well in extracting key areas in pathological images, especially in specific cell types or lesion areas. By accurately segmenting the image, U-Net provides accurate input for subsequent image analysis steps.

[0081] Furthermore, the U-Net model structure is described in detail as follows:

[0082] 1) Contraction path: It is used to capture contextual information in the image and gradually reduce the spatial dimension of the image by alternating between convolutional layers and maximum pooling layers while increasing the number of feature channels.

[0083] 2) Expansion path: used to accurately locate the edge of the target in the image, gradually restore the spatial dimension of the image by alternating upsampling layers and convolutional layers, while reducing the number of feature channels.

[0084] 3) Skip connections: A core feature of U-Net is the skip connections between its encoder and decoder. These connections combine the feature maps of the encoder directly with the corresponding upsampled feature maps in the decoder, which helps to recover the details in the image.

[0085] 4) Symmetry: The symmetrical structure of U-Net is conducive to establishing a good balance between the broad contextual information captured by the encoder and the fine local information reconstructed by the decoder.

[0086] 5) Convolution operation: In the contraction path, U-Net uses convolution operation to extract and learn image features. The convolution operation can be expressed as:

[0087] C i =σ(W*X i + b);

[0088] Among them C i is the convolution output of the i-th layer, W is the convolution kernel, X i is the input feature map, b is the bias term, and σ is the nonlinear activation function.

[0089] 6) Upsampling operation: In the expansion path, U-Net restores the size of the image by upsampling (transposed convolution). The purpose of upsampling is to gradually enlarge the spatial dimension of the image to facilitate fine edge localization.

[0090] 7) Feature Fusion: U-Net fuses the feature maps of the contraction path with the feature maps of the expansion path through skip connections, which enhances the performance of the model in image segmentation tasks. This feature fusion helps retain important contextual information and achieve higher accuracy in image segmentation.

[0091] U-Net is particularly suitable for medical image segmentation tasks through its unique structure and design. Its symmetrical structure, skip connections and feature fusion strategy work together to achieve accurate segmentation of complex objects in images.

[0092] (2) Use EfficientNetV2 for fast and effective image classification

[0093] The segmented image regions are then fed into the EfficientNetV2 model for classification. EfficientNetV2 is a new convolutional neural network (CNN) architecture that is an important improvement and extension of the original EfficientNet architecture. It achieves a balance between efficiency and accuracy by systematically optimizing the scaling of the network's depth, width, and resolution, providing an efficient solution for image classification tasks. EfficientNet can carefully classify each segmented image region and effectively identify different cell types or pathological states. The core contribution of EfficientNet lies in the compound scaling method strategy, which effectively improves the performance of the model while maintaining a relatively low computational cost. The compound scaling method is designed to evenly scale the network's depth (number of layers), width (number of channels per layer), and resolution of the input image, which can achieve better results than scaling any dimension alone. Compound scaling uses the following formula to systematically scale the depth, width, and resolution of the network:

[0094]

[0095] Where d, w, and r represent the scaling factors of depth, width, and resolution, respectively. α, β, and γ are constants determined by grid search on the original model (EfficientNet-B0) to balance the scaling of these three dimensions. φ is an adjustable parameter that controls the overall scaling of the model.

[0096] EfficientNet widely uses the inverted residual structure based on MobileNet, namely the MBConv structure. Figure 4 As shown in the figure, the MBConv structure effectively improves computational efficiency and model performance by changing the dimension of input features layer by layer. EfficientNet successfully improves the performance of the model at multiple scales through its innovative compound scaling method while controlling the consumption of computing resources. This balanced design approach makes EfficientNet an important model choice in multiple fields and applications, and its application advantages are more prominent in scenarios that have high requirements for efficiency and accuracy.

[0097] Furthermore, EfficientNetV2 is an improvement and extension based on the EfficientNet architecture. The design of EfficientNetV2 focuses on further improving the speed and efficiency of model training while maintaining or improving the accuracy of the model at various scales. The model has significant optimizations in training efficiency and is more suitable for large-scale training and real-time applications. The specific optimization measures are:

[0098] 1) Improved compound scaling method: EfficientNetV2 still uses the compound scaling principle of EfficientNet, and on this basis, combines the joint scaling of width, depth and resolution to optimize the original method, especially in terms of convolution operation, attention mechanism and training efficiency. The following are the specific details of the improved compound scaling method:

[0099] Optimizing convolution operations

[0100] Fused-MBConv: In EfficientNetV2, the implementation of convolution operations has been optimized, using the "Fused-MBConv" structure. The traditional MBConv layer includes depth convolution, standard convolution, and point convolution in calculation. EfficientNetV2 fuses these convolution operations and merges them into one layer, making the model calculation more efficient;

[0101] Optimized convolution kernel: EfficientNetV2 uses a more efficient convolution kernel strategy than EfficientNet. By adjusting the size and stride of the convolution kernel, the computational efficiency of the convolution layer is further improved. The performance improvement is particularly obvious in the case of smaller models.

[0102] Improved attention mechanism

[0103] More efficient attention mechanism: EfficientNetV2 optimizes the original Squeeze-and-Excitation (SE) module and proposes a more lightweight attention mechanism (Lite-SE). This optimization simplifies the calculation process of attention weights, effectively controls the number of model parameters and calculations, and improves training speed and inference efficiency;

[0104] Enhanced feature representation capability: The new attention mechanism focuses more on adaptively adjusting weights at different scales, allowing the network to better focus on key areas and improve the model's ability to capture details.

[0105] Improved training speed and efficiency

[0106] Hybrid convolution and attention mechanism: EfficientNetV2 reduces redundant steps in the calculation process and optimizes memory usage by introducing more efficient convolution and attention structures, making the requirements for hardware resources during training lower and shortening the training time;

[0107] Optimized regularization technology: EfficientNetV2 uses a new regularization technology to improve the stability of the training process, allowing the model to converge in fewer iterations, thereby further improving training efficiency.

[0108] Improvements to compound scaling strategies

[0109] Global scaling strategy: EfficientNetV2 retains the compound scaling principle of EfficientNet, but improves the scaling strategy. By introducing a new scaling strategy, EfficientNetV2 not only scales in width, depth, and resolution, but also takes into account the interaction between each dimension to ensure that each dimension can be better balanced during training;

[0110] Width-first scaling: In the process of model scaling, EfficientNetV2 tends to prioritize scaling the width (i.e., increasing the number of channels) rather than blindly expanding the depth or resolution. This width-first scaling strategy can effectively improve computational efficiency and ensure better performance with the same computing resources.

[0111] New network structure design

[0112] Lighter basic network: EfficientNetV2 optimizes the basic network structure, making the calculation of each network layer more efficient, thereby improving the overall performance of the model. In the design, the concepts of "sparse connection" and "dynamic calculation" are emphasized, so that the model can use computing resources more intelligently;

[0113] Application of depth-wise separable convolution: To further improve efficiency, EfficientNetV2 continues to use depth-wise separable convolution, which significantly reduces the amount of computation and parameters, making the network more efficient during reasoning.

[0114] Compared with EfficientNet, EfficientNetV2 further optimizes the calculation path based on compound scaling, improving the training speed and computing efficiency. By optimizing the structure of the convolution layer, introducing a more efficient attention mechanism, and combining a new scaling strategy, EfficientNetV2 achieves higher performance and efficiency. The improved compound scaling method enables EfficientNetV2 to achieve higher performance and faster training speed while maintaining a small amount of computation through more efficient convolution operations, simplified attention mechanisms, optimized training processes, and smarter scaling strategies. These improvements provide better support for large-scale deployment in practical applications, especially in scenarios that require efficient training and inference, and have significant advantages, improving training speed and efficiency.

[0115] 2) More efficient convolution block Fused-MBConv: EfficientNetV2 introduced the Fused-MBConv block. The structure of Fused-MBConv is shown in the figure below: Figure 5 As shown in Figure 1, the Fused-MBConv block is an improvement on the original MBConv block, which improves efficiency by fusing multiple convolution operations in the traditional MBConv block into fewer operations. By fusing convolutional layers to reduce the number of layers, the computational complexity can be reduced, thereby speeding up the training process while retaining the network's ability to learn high-level features.

[0116] 3) Optimization of the training process: EfficientNetV2 uses progressive learning. The progressive learning strategy adopted can be regarded as a method of dynamically adjusting the input resolution. By gradually increasing the size of the image during the training process, the overall training time is reduced, thereby controlling the complexity and resource consumption of the training process. This method helps the model converge faster by using smaller images in the early stages and gradually transitioning to larger images.

[0117] EfficientNetV2 successfully improves the training efficiency and performance of the model through structural optimization and training process improvements. It provides a more flexible and efficient framework based on EfficientNet to adapt to the rapidly developing needs of machine learning applications. Through the application of technologies such as Fused-MBConv blocks and progressive learning, EfficientNetV2 improves model efficiency while maintaining high accuracy.

[0118] (3) Joint application of U-Net and EfficientNetV2

[0119] In the field of pathological image analysis, the application of deep learning models has become the key to improving diagnostic accuracy and efficiency. By jointly applying U-Net and EfficientNetV2, the advantages of each can be used to handle complex image tasks. This combination can improve processing efficiency while maintaining high accuracy, and can achieve accurate segmentation and efficient classification of pathological images, further optimizing the image processing process. The flowchart of the joint application is shown below. Figure 6 shown.

[0120] 1) Using U-Net for high-precision image segmentation

[0121] U-Net performs well in extracting key areas in pathological images through its unique encoder-decoder structure and skip connections. By accurately segmenting the image, U-Net provides accurate input for subsequent image analysis steps.

[0122] 2) Ensure that the image area segmented by U-Net is suitable for the input requirements of EfficientNetV2, and preprocessing operations such as resizing and normalization are required.

[0123] 3) Use EfficientNetV2 for fast and effective image classification

[0124] The segmented image regions are then input into the EfficientNetV2 model for classification. EfficientNetV2 provides an efficient solution for image classification tasks through its optimized convolutional blocks and training methods. In this step, EfficientNetV2 carefully classifies each segmented image region and effectively identifies different cell types or pathological states.

[0125] The combination of U-Net and EfficientNetV2 is used to handle complex image analysis tasks that require accurate image segmentation followed by detailed classification. This combination takes advantage of the high precision of U-Net in image segmentation and the high efficiency of EfficientNetV2 in classification tasks. The joint application strategy greatly improves the accuracy of segmentation and classification.

[0126] In a specific embodiment, in medical image analysis, heatmaps are an important visualization tool used to highlight image regions that a model focuses on when making a diagnosis or classification decision. The accuracy and interpretability of heatmaps are critical to improving the accuracy and confidence of medical diagnoses.

[0127] (1) Ensemble Learning

[0128] Ensemble learning generates a comprehensive heat map by combining the predictions of multiple independent models. This approach aims to leverage the unique perspectives and strengths of different models to improve the accuracy and stability of the heat map.

[0129] Assume there are N models, each with M i Generate a heat map H i . Comprehensive heat map H ensemble This can be obtained through weighted average or voting mechanism:

[0130]

[0131] In the formula, w i It is model M i The weights are adjusted based on the performance or reliability of the model.

[0132] (2) Application of Attention Mechanism

[0133] The attention mechanism can make the model focus more on the areas that contribute most to the final decision when generating heat maps. This method enhances the interpretability and focus of the model by simulating the mechanism of human attention. An attention layer is introduced in the model, which assigns an attention weight to each pixel or area. The heat map is generated by mapping the attention weights to the input image:

[0134] H attention (x,y)=A(x,y)×I(x,y);

[0135] Where A(x,y) is the attention weight at position (x,y), and I(x,y) is the pixel value of the input image at that position.

[0136] (3) Deep feature fusion

[0137] Combining features from different layers of the network, we can generate heat maps that reflect multi-level cognitive processes to improve their detail and interpretability. Suppose the network has L layers, and each layer l corresponds to a feature map F l . Deep fusion heat map H fusion By weighted fusion of feature maps of different layers, we can get:

[0138]

[0139] In the formula, α l is the weight of the l-th layer feature, and Up(·) is the upsampling operation used to adjust all feature maps to the same size.

[0140] The above method provides more accurate and interpretable heat maps, thereby enhancing the practicality and reliability of pathological image analysis. Through integrated learning, attention mechanism and deep feature fusion, the heat map generation algorithm can better reveal the decision-making process and focus of the model, providing more valuable reference information for medical experts.

[0141] In a specific implementation, during the generation of the heat map, color mapping and filtering techniques are also required to improve the readability and practicality of the heat map.

[0142] (1) Color Mapping

[0143] Color mapping refers to the process of mapping numerical data in a heat map to a color space, with the goal of making areas of different temperatures (or attention) appear in different colors for easy visual identification. Let the temperature (or weight) of a point in the heat map be T, and the color mapping function be C(T), which maps T to color C. Color mapping usually involves mapping the minimum and maximum values ​​in the data range to the two ends of the color range, from blue (low temperature) to red (high temperature):

[0144]

[0145] Among them, T min and T max are the minimum and maximum values ​​in the data, respectively, and color_start and color_end are the starting and ending colors of the mapping, respectively.

[0146] (2) Filtration technology

[0147] Filtering techniques aim to enhance the clarity and interpretability of heatmaps by removing non-critical signals or background noise in the heatmap and highlighting key features.

[0148] Applying semantic segmentation filtering based on deep learning can accurately distinguish the foreground (key signal) and background (non-key signal) in the image. Let (S(x,y) be the classification result of the semantic segmentation model for the position (x,y) in the image (1 represents the key area and 0 represents the non-key area), then the filtered heat map H filtered It is expressed as:

[0149] H filtered (x,y)=H(x,y)×S(x,y);

[0150] Where H(x,y) is the temperature or weight of the original heat map at position (x,y).

[0151] (3) Overlay and Adjustment

[0152] In the heat map generation algorithm, overlay and adjustment is a key step, which is used to overlay the processed pseudo-color image with the original image and adjust the intensity and method of overlay as needed to achieve the best visual effect and information display.

[0153] First, the processed pseudo-color image is superimposed with the original image. The purpose of superposition is to retain the structure and features of the original image, and on this basis, add the visual information of pseudo-color. In this way, the generated heat map can have both the basic information of the original image and the specific features or regions that can be highlighted. Secondly, a coefficient is multiplied during superposition to adjust the intensity of the pseudo-color image. In the present embodiment, the gray value of the original image is multiplied by 0.6 and then added to the pseudo-color image. This coefficient can be adjusted according to demand to control the display intensity of the pseudo-color, so that the image is clearer or softer, so as to better show specific features. In fact, the superposition method is flexible, and different mixing modes can be used according to specific needs to obtain different visual effects. Through appropriate superposition and adjustment, the generated heat map can be made more informative and readable, thereby better showing the features and information in the image, which can help people better understand and analyze image data.

[0154] Through the above-mentioned optimized color mapping strategy, the heat map can be displayed with more intuitive and vivid color contrast, helping doctors and researchers to quickly identify the key areas in the image. At the same time, the application of advanced filtering techniques such as semantic segmentation filtering based on deep learning can effectively remove non-critical signals in the heat map, reduce visual interference, and improve the interpretability of the heat map. The application of these technologies not only enhances the visual effect of the heat map, but also improves its practical value in medical diagnosis.

[0155] In a specific embodiment, in pathological image analysis, the efficiency of the heat map generation algorithm is further improved, and in scenarios where real-time processing or processing of large-scale data sets is required, optimization measures are used to accelerate heat map generation while maintaining its accuracy and interpretability.

[0156] (1) Model compression and acceleration

[0157] Model compression includes technologies such as model pruning and quantization, which can reduce the model size and speed up the inference process by reducing the number of model parameters and simplifying calculations. Model pruning usually involves removing certain weights or neurons in the model, setting a threshold θ, and setting the absolute value of the weight less than θ to zero:

[0158]

[0159] Where w is the original weight, w ′ is the weight after pruning.

[0160] (2) Parallel processing strategy

[0161] Using GPU acceleration and parallel computing frameworks, data processing and model calculations can be performed simultaneously on multiple computing cores, significantly improving the speed of heat map generation. Parallel processing does not involve direct formula changes, but it allows multiple operations to be performed simultaneously. When processing heat maps, the image can be split into multiple blocks, each GPU core processes one block, and then the results are merged.

[0162] (3) Efficient encoder-decoder structure

[0163] By designing a more efficient network structure and using lightweight encoders and decoders, the generation of heat maps can be accelerated without significantly sacrificing accuracy. Lightweight network structures usually have fewer parameters and faster calculations. Lightweight networks are designed by reducing network depth, using smaller convolution kernels, and introducing depth-separable convolutions:

[0164] Y = D(K*X);

[0165] Where X is the input feature map, X is the convolution kernel, * represents the convolution operation, D represents the linear combination step of depth-wise separable convolution, and Y is the output feature map.

[0166] Through the above optimization measures, the efficiency of heat map generation can be significantly improved, making it more suitable for real-time or large-scale pathological image analysis applications. These methods not only speed up the calculation process, but also maintain the accuracy and interpretability of the algorithm through efficient network design, thus providing strong technical support for medical diagnosis and research.

[0167] In a specific embodiment, an experimental evaluation is performed, and the specific steps are as follows:

[0168] (1) Heatmap generation effect and performance index evaluation

[0169] When further studying the new heat map generation algorithm, this embodiment chooses the TCGA (The Cancer Genome Atlas) dataset as the basis, which is a widely recognized and richly annotated medical image library focusing on cancer-related research.

[0170] When evaluating a new heat map generation algorithm, this embodiment focuses on evaluating two aspects: one is the generation effect of the heat map and its interpretability, and the other is the performance indicators of the algorithm, including precision, recall rate and generation speed. Heat map generation effect and interpretability evaluation: Evaluate the effect of the algorithm in generating heat maps, especially its interpretability, that is, whether it can accurately identify lesion areas or diagnosis-related features. Use pathological images in the TCGA dataset as input to generate heat maps, and evaluate their correspondence with the actual lesion area and their interpretability; Performance indicator evaluation: Evaluate the performance of the new algorithm through quantitative indicators such as precision, recall rate and generation speed. Compare the heat map generated by the algorithm with the expert annotations in the TCGA dataset, and calculate indicators such as precision and recall rate; at the same time, measure the time required to generate the heat map to evaluate the efficiency of the algorithm.

[0171] (2) Demonstration of the effect of heat map generation algorithm

[0172] The heat map visualization process generated by the algorithm in this embodiment is shown as follows: Figure 7 The grayscale image of epithelial tissue is shown in Figure 8 As shown, the grayscale image of mesenchymal tissue is as follows Fig. 9 As shown in the figure, the epithelial tissue heat map is as follows Fig.10 As shown in the figure, the heat map of mesenchymal tissue is as follows Fig.11 The visualized heat map of epithelial tissue after superposition is shown in Fig.12 The superimposed mesenchymal visualization heat map is shown in Fig.13 As shown. Figure 7-Figure 13 It can be seen that after visualization, the epithelial tissue area and the mesenchymal tissue area can be directly seen, which greatly helps the doctor's diagnosis.

[0173] (3) Quantitative evaluation of algorithm performance

[0174] The generated heatmaps were visually evaluated by organizing an expert review panel. The evaluation criteria included the interpretability of the heatmaps and the accuracy of the identification of key areas. The heatmaps were evaluated to see whether they provided doctors with a clear visual explanation of the lesion areas and diagnosis-related features. The key areas identified in the heatmaps were evaluated for consistency with the actual lesion areas.

[0175] In the process of evaluating the new heat map generation algorithm, a subset of the TCGA dataset was used, containing 1000 pathological images. The number of actual lesion areas (real key areas): 1500, the number of lesion areas identified by the algorithm: 1400, of which the number of correctly identified lesion areas (true positive examples TP): 1300, the number of non-lesion areas incorrectly identified as lesion areas (false positive examples FP): 100, and the number of actual lesion areas not identified by the algorithm (false negative examples FN): 200.

[0176] Performance index calculation:

[0177] (1)Precision

[0178] Precision measures the accuracy of the algorithm in identifying key areas. Based on the above data, the calculation formula and results are as follows:

[0179]

[0180] (2) Recall

[0181] The recall rate measures the ability of the algorithm to cover all real key areas. Based on the above data, the calculation formula and results are as follows:

[0182]

[0183] (3) Running time

[0184] Operating environment: Processor: 12th Gen Intel(R) Core(TM) i5-12500 3.00 GHz, with 32.0 GB RAM, 64-bit operating system, x64-based processor. Running the algorithm on this processor, the total time to process 1,000 images is 8,260 seconds, and the average processing time per image is 8.26 seconds.

[0185] According to this set of data, the new heat map generation algorithm shows high precision (92.86%) and good recall (86.67%), while maintaining a processing speed of 8.26 seconds per image. Such performance indicators show a good balance between accuracy and efficiency of the algorithm, which is particularly suitable for processing large-scale medical image datasets. These results provide important baseline information for further optimization and application.

[0186] Through visual evaluation, we can gain an intuitive understanding of the algorithm in terms of medical image interpretability and accuracy. Through quantitative evaluation indicators, we can specifically quantify the accuracy, recall and efficiency of the algorithm, providing an objective performance evaluation. Combining qualitative feedback from experts with the results of quantitative evaluation, we can deeply analyze the specific contributions of the new algorithm in improving diagnostic accuracy, enhancing image interpretability and improving processing efficiency, thus laying the foundation for future applications of medical image analysis.

[0187] The heat map generated by the algorithm is highly interpretable and can clearly mark the lesion area and diagnosis-related features, providing doctors with an intuitive visual reference. This intuitiveness greatly promotes the accuracy and efficiency of the diagnostic process. It is particularly outstanding in terms of precision and recall, and can accurately identify key features in pathological images while covering most actual lesion areas. It also has the ability to process quickly and can quickly generate heat maps without sacrificing accuracy, making it suitable for large-scale pathological image analysis and real-time application scenarios.

[0188] This paper proposes a pathological image analysis method based on deep learning. Using the Cancer Genome Atlas (TCGA) dataset, the algorithm demonstrates its strong potential in the field of pathological image analysis, especially in identifying and annotating cancer-related lesion areas. Through a comprehensive evaluation of the heat map generated by the algorithm, including the interpretability and performance indicators of the generated effect (precision, recall rate, generation speed), it is proved that the method can not only provide highly interpretable visual output, but also achieve a high level of performance. The experimental results reveal that the algorithm achieved an accuracy of 92.86%, a recall rate of 86.67%, and a processing time of only 8.26 seconds per image, reflecting its good balance between efficiency and accuracy. The application of the heat map generation algorithm proposed in the present invention in pathological image analysis shows its excellent performance and broad application prospects. Through in-depth research and optimization, this algorithm has the potential to have a significant impact on medical diagnosis, bioinformatics, and the broader field of image analysis in the future.

[0189] The present invention accurately locates the key features in pathological images through a deep learning model and generates high-quality heat maps to visualize these features. It can provide more detailed and accurate image area positioning, so that doctors can more clearly understand the judgment basis of the model, thereby improving the accuracy and efficiency of pathological image analysis. By optimizing the internal mechanism of the deep learning model, the accuracy and resolution of the heat map are greatly improved. This not only enables the heat map to reveal the specific areas of concern to the model, but also can be accurate to the details of the cell or tissue structure level, which is crucial for the diagnosis of complex diseases. At the same time, a filtering mechanism based on medical knowledge is introduced, which can effectively distinguish which of the features concerned by the model are clinically significant and which may be just accidental noise. This mechanism not only improves the medical interpretability of the heat map, but also allows doctors to trust the diagnosis results of the model more. In addition, this innovative heat map generation algorithm also focuses on improving the visual clarity and aesthetics of the heat map. By using more intuitive color mapping and gradient technology, the different densities and distributions of disease features can be more clearly represented. Such visual optimization not only enables doctors to interpret heat maps more easily, but is also extremely helpful for understanding the heterogeneity of diseases. By optimizing the algorithm structure and calculation process, the speed of heat map generation has been greatly improved, and even large-scale pathology image data sets can be processed quickly. This is particularly important for clinical applications, as it means that doctors can obtain the required analysis results in a shorter time, thereby making diagnosis and treatment decisions more quickly. Through significant improvements in accuracy, interpretation capabilities, and integration with medical expertise, not only can the accuracy and efficiency of pathology image analysis be improved, but also doctors' understanding of the basis for model judgments can be enhanced through better visualization effects. This not only promotes the application of deep learning technology in the field of medical image analysis, but also provides new possibilities for the accuracy of disease diagnosis and treatment in the future.

[0190] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0191] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A pathological image analysis method based on deep learning, characterized in that: include: Collect the pathological images to be tested and build a data set; Construct a U-Net network, perform image segmentation on the dataset images through the U-Net network, and obtain the segmentation results; Preprocess the segmentation results to obtain optimized results; Build an EfficientNetV2 classification model and use it to classify images of the optimization results. Visualize the image classification results.

2. A pathological image analysis method based on deep learning according to claim 1, characterized in that: The U-Net network includes: a contraction path, an expansion path and a jump connection. The contraction path and the expansion path are structurally symmetrical. The feature map of the contraction path is combined with the corresponding up-sampled feature map in the expansion path through the jump connection.

3. The pathological image analysis method based on deep learning according to claim 1, characterized in that: It also includes,using convolution operations in the contraction path to extract and learn image features; In the expansion path, the image size is restored through transposed convolution and the spatial dimension of the image is enlarged.

4. The pathological image analysis method based on deep learning according to claim 1, characterized in that: The preprocessing of the segmentation results to obtain the optimized results includes: image formatting, normalization, image enhancement, image denoising and data enhancement processing.

5. The pathological image analysis method based on deep learning according to claim 1, characterized in that: The construction of the EfficientNetV2 classification model includes: replacing the MBConv block in the EfficientNet architecture with a Fused-MBConv block, the Fused-MBConv block fuses multiple convolution operations in the MBConv block, reduces the number of layers by fusing convolutional layers, and retains the ability of the network to learn advanced features.

6. A pathological image analysis method based on deep learning according to claim 5, characterized in that: It also includes optimizing the EfficientNetV2 classification model: the EfficientNetV2 classification model uses progressive learning, gradually increases the size of the image during the training process to reduce the overall training time, uses small images in the early stages, and gradually transitions to using large images.

7. The pathological image analysis method based on deep learning according to claim 1, characterized in that: The visual display of the image classification results includes generating a heat map according to the image classification results, introducing an attention layer in the EfficientNetV2 classification model, assigning an attention weight to each pixel or area through the attention layer, mapping the attention weight to the input image, obtaining a deep fusion heat map by weighted fusion of feature maps of different layers, combining the prediction results of multiple models, generating a comprehensive heat map for visual display.

8. The pathological image analysis method based on deep learning according to claim 7, characterized in that: The method of generating a heat map includes: mapping numerical data in the heat map to a color space so that areas with different temperatures are presented in different colors; highlighting key features by removing non-critical signals or background noise in the heat map, enhancing the clarity and interpretability of the heat map, superimposing the processed pseudo-color image with the original image, and adjusting the intensity and method of the superposition.

9. A pathological image analysis method based on deep learning according to claim 8, characterized in that: It also includes improving the efficiency of heat map generation: reducing the number of model parameters and simplifying calculations through model pruning and quantization techniques, reducing model size and accelerating the inference process; using GPU acceleration and parallel computing frameworks to simultaneously perform data processing and model calculations on multiple computing cores; reducing network depth, using smaller convolution kernels and introducing depth-separable convolutions to design lightweight networks.

10. A pathological image analysis system based on deep learning, characterized in that: include: Acquisition module: used to collect pathological images to be tested and build data sets; Segmentation module: used to build a U-Net network, perform image segmentation on the dataset image through the U-Net network, and obtain the segmentation result; Optimization module: used to pre-process the segmentation results to obtain optimized results; Classification module: used to build the EfficientNetV2 classification model and classify the images of the optimization results through the EfficientNetV2 classification model; Visualization module: used to visualize the image classification results.

Images