A Pathological Tissue Grading Method and System Based on Sliding Window Attention Mechanism

Abstract

This application provides a pathological tissue grading method and system based on a sliding window attention mechanism. The method includes: acquiring a pathological image to be processed; acquiring a standardized pathological image corresponding to the pathological image to be processed based on a pathological image staining standardization model; and performing grading processing on the standardized pathological image based on a pathological tissue grading model to obtain a pathological image grading result. In the pathological tissue grading model, a first convolutional module extracts local features of the standardized pathological image, a residual module extracts global features of the standardized pathological image, and a classification module determines the pathological image grading result based on a fused feature that integrates local and global features. This application embodiment utilizes a pathological tissue grading model to fuse local and global features of the pathological image, improving the model's ability to extract contextual information, effectively avoiding the problem of convolutional neural networks lacking global information, and improving the accuracy and computational efficiency of the grading model.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence image analysis technology, and in particular to a method and system for grading pathological tissues based on a sliding window attention mechanism. Background Technology

[0002] Pathological grading primarily involves analyzing the results of a patient's pathological tissue and classifying them into three grades: Grade I, Grade II, and Grade III. Pathological grading is used to comprehensively assess and guide subsequent treatment and prognosis. Generally, the higher the grade, the more malignant the cancer cells and the worse the prognosis. High grades usually require more aggressive treatment strategies, while low grades may have a better prognosis and allow for more conservative treatment. Therefore, pathological grading is a crucial basis for pathological diagnosis, assisting physicians in determining treatment plans and assessing prognosis, and is of great significance in improving patient survival rates.

[0003] With the development of digital healthcare, intelligent grading of pathological tissues using artificial intelligence technology has become a new trend. However, due to the complex tissue structure and diverse lesion features of pathological images, traditional image classification algorithms often fail to fully capture the rich feature information, resulting in limited grading accuracy and failing to meet practical applications. Summary of the Invention

[0004] In view of this, this application provides a pathological tissue grading method and system based on a sliding window attention mechanism to improve the accuracy of pathological tissue grading results.

[0005] The technical solution is as follows:

[0006] In a first aspect, embodiments of this application provide a pathological tissue grading method based on a sliding window attention mechanism, the method comprising:

[0007] Acquire the pathological images to be processed;

[0008] The pathological image to be processed is subjected to staining standardization processing based on the pathological image staining standardization model to obtain a standardized pathological image.

[0009] The standardized pathological image is graded based on a pathological tissue grading model to obtain a pathological image grading result. The pathological tissue grading model includes a first convolutional module, a residual module that introduces a sliding window multi-head attention mechanism, and a classification module. The first convolutional module is used to extract local features of the standardized pathological image, the residual module is used to extract global features of the standardized pathological image, and the classification module is used to determine the pathological image grading result based on a fused feature that integrates the local features and the global features.

[0010] In one optional implementation, the step of performing staining normalization processing on the pathological image to be processed based on the pathological image staining normalization model to obtain a normalized pathological image includes:

[0011] The pathological image to be processed is input into the pathological image staining normalization model to obtain the normalized pathological image; wherein, the pathological image staining normalization model is trained based on an improved recurrent generative adversarial network model, the improved recurrent generative adversarial network model includes a generator network and a loss function, the generator network includes a multi-head attention mechanism module and a spectral normalization module, and the loss function includes a geometric consistency loss function.

[0012] In one optional implementation, the first convolutional module includes two basic convolutional layers; wherein the basic convolutional layers include a convolutional layer, a batch normalization layer, and a LeakyReLU activation function layer.

[0013] In one optional implementation, the residual module includes three residual attention sub-modules; wherein,

[0014] The residual attention submodule includes two second convolutional submodules, an image patch embedding layer, a sliding window attention layer, a feature fusion layer, and a feature output layer;

[0015] The second convolutional submodule includes a convolutional layer, a batch normalization layer, and a Mish activation function layer. The image patch embedding layer is used to reduce the feature dimension of the feature map output by the second convolutional submodule. The feature fusion layer is used to fuse the local features output by the first convolutional module and the global features output by the sliding window attention layer to obtain the fused features.

[0016] In one optional implementation, the classification module includes: an adaptive pooling layer, a linear flattening operation layer, a fully connected layer, and a Softmax function; wherein, the adaptive pooling layer is used to obtain the sampled features corresponding to the fused features through pooling operations; the linear flattening operation layer is used to convert the sampled features into feature vectors; the fully connected layer and the Softmax function are used to map the feature vectors to the corresponding dimensions of the output category, thereby obtaining the pathological image grading result.

[0017] In one optional implementation, the training process of the pathological image staining normalization model includes:

[0018] Obtain an initial dataset, which includes pathological tissue slice images of different grades;

[0019] The initial dataset is preprocessed to obtain a preprocessed dataset. The preprocessing includes annotation and cropping.

[0020] A first training dataset is obtained based on the preprocessed dataset. The first training dataset includes source domain pathological images and target domain pathological images. The source domain pathological images are pathological images that have not undergone staining and normalization, and the target domain pathological images are images that are expected to be obtained after staining and normalization of the source domain pathological images.

[0021] An initial pathological image staining normalization model is constructed based on an improved recurrent generative adversarial network model. The improved recurrent generative adversarial network model includes a generator network and a loss function. The generator network includes a multi-head attention mechanism module and a spectral normalization module. The loss function includes a geometric consistency loss function.

[0022] The initial pathological image staining normalization model is iteratively trained using the first training dataset to obtain the pathological image staining normalization model.

[0023] In one optional implementation, the training process of the pathological tissue grading model includes:

[0024] A second training dataset is obtained based on the preprocessed dataset, the second training dataset including a training set and a test set;

[0025] An initial pathological tissue grading model is constructed, which includes a first convolutional module, a residual module that introduces a sliding window multi-head attention mechanism, and a classification module.

[0026] The initial pathological tissue grading model is iteratively trained using the second training dataset to obtain the pathological tissue grading model.

[0027] Secondly, embodiments of this application provide a pathological tissue grading system based on a sliding window attention mechanism, the system comprising: an intelligent diagnostic module;

[0028] The intelligent diagnostic module includes:

[0029] The image acquisition submodule is used to acquire pathological images to be processed;

[0030] The image processing submodule is used to perform staining standardization processing on the pathological image to be processed based on the pathological image staining standardization model to obtain a standardized pathological image.

[0031] The image grading submodule is used to grade the standardized pathological image based on the pathological tissue grading model to obtain the pathological image grading result. The pathological tissue grading model includes a first convolution module, a residual module that introduces a sliding window multi-head attention mechanism, and a classification module. The first convolution module is used to extract local features of the standardized pathological image, the residual module is used to extract global features of the standardized pathological image, and the classification module is used to determine the pathological image grading result based on a fused feature that integrates the local features and the global features.

[0032] In one optional implementation, the image processing submodule is specifically used to: input the pathological image to be processed into the pathological image staining normalization model to obtain the normalized pathological image; wherein, the pathological image staining normalization model is trained based on an improved recurrent generative adversarial network model, the improved recurrent generative adversarial network model includes a generator network and a loss function, the generator network includes a multi-head attention mechanism module and a spectral normalization module, and the loss function includes a geometric consistency loss function.

[0033] In one optional implementation, the system further includes: a model training module;

[0034] The model training module includes:

[0035] The data acquisition submodule is used to acquire an initial dataset, which includes pathological tissue slice images of different grades;

[0036] The data processing submodule is used to preprocess the initial dataset to obtain a preprocessed dataset. The preprocessing includes annotation and cropping.

[0037] The data acquisition submodule is further configured to acquire a first training dataset based on the preprocessed dataset. The first training dataset includes source domain pathological images and target domain pathological images, wherein the source domain pathological images are pathological images that have not undergone staining and standardization, and the target domain pathological images are images that are expected to be obtained after staining and standardization of the source domain pathological images.

[0038] The model building submodule is used to build an initial pathological image staining normalization model based on an improved recurrent generative adversarial network model. The improved recurrent generative adversarial network model includes a generator network and a loss function. The generator network includes a multi-head attention mechanism module and a spectral normalization module. The loss function includes a geometric consistency loss function.

[0039] The model training submodule is used to iteratively train the initial pathological image staining normalization model using the first training dataset to obtain the pathological image staining normalization model.

[0040] The above technical solution has the following beneficial effects:

[0041] This application provides a pathological tissue grading method based on a sliding window attention mechanism. When executing the method, a pathological image to be processed is acquired; the pathological image is subjected to staining standardization processing based on a pathological image staining standardization model to obtain a standardized pathological image; the standardized pathological image is then graded based on a pathological tissue grading model to obtain a pathological image grading result. The pathological tissue grading model includes a first convolutional module, a residual module incorporating a sliding window multi-head attention mechanism, and a classification module. The first convolutional module extracts local features of the standardized pathological image, the residual module extracts global features of the standardized pathological image, and the classification module determines the pathological image grading result based on a fused feature that integrates the local and global features. Therefore, this application utilizes a grading model to extract and fuse local and global features of the pathological image, improving the model's ability to extract contextual information, effectively avoiding the problem of convolutional neural networks lacking global information, and improving the accuracy and computational efficiency of the grading model.

[0042] This application also provides a system corresponding to the above method, which has the same beneficial effects as the above method. Attached Figure Description

[0043] To more clearly illustrate the technical solutions in this embodiment or the prior art, the drawings used in the description of the embodiment or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0044] Figure 1 A flowchart illustrating a pathological tissue grading method based on a sliding window attention mechanism, provided for an embodiment of this application;

[0045] Figure 2 A schematic diagram of the algorithm structure corresponding to a pathological tissue grading method based on a sliding window attention mechanism provided in an embodiment of this application;

[0046] Figure 3 A schematic diagram of a pathological tissue grading system based on a sliding window attention mechanism provided in an embodiment of this application;

[0047] Figure 4 This is a schematic diagram of the structure of an intelligent diagnostic module provided in an embodiment of this application;

[0048] Figure 5 This is a schematic diagram of the structure of a model training module provided in an embodiment of this application;

[0049] Figure 6 This is a schematic diagram illustrating the operation of a pathological tissue grading system based on a sliding window attention mechanism, provided as an embodiment of this application. Detailed Implementation

[0050] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0051] As introduced in the background section, due to the complex tissue structure and diverse lesion features of pathological images, traditional image classification algorithms often fail to fully capture the rich feature information, resulting in limited grading accuracy of the algorithm model and failing to meet practical applications.

[0052] To improve the accuracy of pathological tissue grading results, please refer to... Figure 1 The diagram illustrates a flowchart of a pathological tissue grading method based on a sliding window attention mechanism provided in an embodiment of this application. The method includes:

[0053] Step S100: Obtain the pathological image to be processed.

[0054] Specifically, the method involves acquiring images of tissues that require pathological grading. The pathological images to be processed can be different types of tissue pathological images, such as breast cancer tissue pathological images, lung cancer tissue pathological images, etc. The pathological tissue grading method provided in this application does not limit the specific tissue targeted and can achieve intelligent grading for tissue slice images of different types of cancer lesions.

[0055] The pathological image to be processed is a stained image. The initial pathological image is a tissue section image made of human tissue cells, which has no color and cannot be identified under a microscope. In order to distinguish different cell structures, the section image needs to be stained to obtain the pathological image to be processed in this application. This application does not limit the staining method. For example, HE staining, PAS staining, Masson staining, etc. can be used.

[0056] It should be noted that the pathological image staining standardization model and pathological tissue grading model used in subsequent steps require corresponding datasets during training. For example, when the model needs to grade lung cancer images, it needs to be trained using a lung cancer image dataset; if the model needs to grade breast cancer images, it needs to be trained using a breast cancer image dataset. In other words, the purpose of the grading model corresponds to the training dataset used by the grading model.

[0057] Step S200: Based on the pathological image staining standardization model, perform staining standardization processing on the pathological image to be processed to obtain a standardized pathological image.

[0058] Specifically, due to the possible differences in staining methods and operations used in the pathological images to be processed, this application embodiment also uses a pathological image staining standardization model to perform staining standardization processing on the pathological images to be processed, thereby improving the quality and realism of the generated images, ensuring the quality and consistency of the images subsequently input into the pathological tissue grading model, and avoiding the problem of poor accuracy of grading results due to the inconsistency of the original input data of the model.

[0059] In this embodiment of the application, the pathological image staining normalization model can be obtained by training an improved Cycle-Generative Adversarial Network (CycleGAN).

[0060] In one optional implementation, step S200 specifically includes: inputting the pathological image to be processed into a pathological image staining normalization model to obtain a normalized pathological image; the pathological image staining normalization model is trained based on an improved CycleGAN model, the improved CycleGAN model includes a generator network and a loss function, the generator network includes a multi-head attention mechanism module and a spectral normalization module, and the loss function includes a geometric consistency loss function.

[0061] Since convolutional neural networks in traditional CycleGAN model generators struggle to capture global image information, this application provides an improved CycleGAN model that focuses not only on local features of lesion areas but also on contextual relationships between different lesion areas, achieving comprehensive feature extraction.

[0062] Specifically, the pathological image staining standardization model is trained using an improved recurrent generative adversarial network (CycleGAN) model. In the improved CycleGAN model's generator structure, the original instance normalization is replaced with spectral normalization. The spectral normalization operation used in the discriminator is introduced into the generator, constraining the spectral norm of the weight matrix. Simultaneously, a multi-head attention mechanism is introduced into the generator, enabling it to capture not only local information in the pathological image but also contextual information about the region of interest, extracting richer features. Furthermore, a geometric consistency loss is introduced into the CycleGAN model's loss function. These improvements constrain the generator to produce pathological images with more realistic colors and more standardized staining, contributing to the accuracy of subsequent grading.

[0063] The improved CycleGAN model provided in this application is described below:

[0064] The original loss function of the CycleGAN model consists of adversarial loss and cycle consistency loss. CycleGAN has two pairs of generators and discriminators, resulting in four corresponding losses. Specifically, the discriminator loss from X to Y is the adversarial loss, calculated as shown in formula (1):

[0065]

[0066] Where G is the generator; D Y X is the discriminator; X is the source domain image, i.e., the original image; Y is the target domain image, i.e., the processed image. Represents the expected value, used to calculate the expectation of the generator and discriminator over the entire data distribution during training; x ~ p data(x) Represents the source domain image distribution; y ~ p data(y) This represents the distribution of the target domain image.

[0067] Similarly, the discriminator loss from Y to X is calculated as shown in formula (2):

[0068]

[0069] Where F is the generator; D X X is the discriminator; X is the source domain image, i.e., the original image; Y is the target domain image, i.e., the processed image. Represents the expected value, used to calculate the expectation of the generator and discriminator over the entire data distribution during training; x ~ p data(x) Represents the source domain image distribution; y ~ p data(y) This represents the distribution of the target domain image.

[0070] However, adversarial loss only guarantees that generator X matches the image set Y or generator Y matches the image set X, and cannot guarantee the integrity and consistency of the image content. In practical applications, it is desirable to change the style of the original image without changing its original content.

[0071] Therefore, a cycle consistency loss is generated to ensure the consistency of the initial content of the image during the conversion process. The calculation method of the cycle consistency loss function for the two generators is shown in formula (3):

[0072]

[0073] Here, G and F correspond to two generators, x is the source domain image (original image), and y is the target domain image (processed image). Represents the expected value, used to calculate the expectation of the generator and discriminator over the entire data distribution during training; x ~ p data(x) Represents the source domain image distribution; y ~ p data(y) Let ||||1 represent the distribution of the target domain image, and ||||1 is the L1 norm.

[0074] The overall loss of the CycleGAN model is the sum of the two discriminators and the two generators, as shown in equation (4):

[0075]

[0076] Where G and F correspond to two generators, and D X and D Y These correspond to two discriminators, with λ being the weight ratio of the adversarial loss function to the recurrent loss function.

[0077] First, the improved CycleGAN model provided in this application introduces a multi-head attention mechanism in the generator to capture the contextual relationships of lesion regions in images. The multi-head self-attention mechanism is a further refinement of the self-attention layer. By introducing multiple attention heads, it expands the attention mechanism to improve the model's attention allocation to diverse features. Specifically, the input sequence data is first divided into multiple heads, and then multiple independent linear transformations are used to map the query matrix, key matrix, and value matrix, transforming the input data into their respective representation spaces. Then, self-attention calculation is performed on each head, and these different attention results are concatenated to obtain more comprehensive information. Finally, a linear transformation is used again to transform the concatenated result, producing the final output.

[0078] The calculation method for the multi-head attention mechanism is shown in formula (5):

[0079] MultiHead(Q,K,V)=Concat(head1,...,head h WO (5)

[0080] Where Q, K, and V are the query vector, key vector, and value vector, respectively, and h is the number of heads. i For the output of the i-th head, W O This is the output transformation matrix.

[0081] The output head of each attention head i The calculation method is shown in formula (6):

[0082] head i =Attention(QW i Q ,KW i K VW i V (6)

[0083] Among them, Wi Q W i K W i V Let be the query transformation matrix, key transformation matrix, and value transformation matrix of the i-th attention head, respectively, and Attention be the attention calculation function.

[0084] This paper introduces a multi-headed self-attention mechanism into the generator of the CycleGAN model. Multi-headed self-attention is a technique used in Natural Language Processing (NLP) tasks. It allows the model to focus on information in different representation subspaces, which helps to capture contextual information from different parts of the input data, thereby improving the model's representational ability and its ability to handle complex semantic relationships. The embodiments of this application introduce a multi-headed self-attention mechanism into the generator of the CycleGAN model to extract richer features and improve the performance of the generator.

[0085] Secondly, to prevent mode collapse and improve model stability, the normalization module in the generator of the CycleGAN model was replaced with spectral normalization instead of instance normalization.

[0086] Spectral normalization is a normalization technique for the discriminator in Generative Adversarial Networks (GANs), primarily aimed at improving its performance and stability. By normalizing the discriminator's weight matrix, each element is kept within a preset threshold, limiting the range of variation in the weight matrix. This normalization method helps reduce the instability of the discriminator during training, improving its performance and stability.

[0087] The main idea behind spectral normalization is to normalize the spectrum of each pixel value to the same distribution. Specifically, it converts each pixel value into a complex number and then normalizes its magnitude to 1. This approach results in a more uniform spectral distribution across different images, making the model more likely to converge.

[0088] Finally, a geometric consistency loss is introduced into the loss function of the CycleGAN model to ensure that the pathological images generated by the generator meet the geometric consistency requirements.

[0089] Geometric consistency loss helps suppress the misalignment effect where the generated image does not perfectly match the original image geometrically, making the generated image more accurately match the geometry of the original image. The goal of geometric consistency loss is to preserve the geometry and structure of the image, ensuring that the generated image is as geometrically consistent as possible with the original image. By comparing the gradient information of the original and generated images, the differences in their geometric structure can be measured, thereby guiding the generator to learn a more accurate geometry.

[0090] Suppose x represents an image in domain X, y represents an image in domain Y, and G... X G represents a generator from domain X to domain Y. Y Let G be the generator that maps a region from domain Y to domain X. Geometric consistency loss can be defined as the loss of the original image in domain X after passing through G. Y The geometric structure difference between the generated image and the original image. Specifically, the formula for geometric consistency loss is shown in equation (7):

[0091]

[0092] Where N represents the number of samples, This represents the gradient information of the image x in the domain X. This indicates that the generator G has passed through it. Y The generated image G Y The gradient information of (x), where 1 is the L1 norm.

[0093] It should be noted that the geometric consistency loss can be directly added to the original loss function of the CycleGAN model, i.e., the above formula (4).

[0094] This application introduces a geometric consistency loss into the CycleGAN model and combines it with the model's original loss. The geometric consistency loss measures the differences in the geometric structure of images by comparing their gradient information. Gradient information reflects local changes in the image; therefore, comparing the gradient information of two images helps measure their geometric differences. By minimizing the geometric consistency loss, it helps to solve the problems of geometric distortion and artifacts, making the generated images more closely match the target domain, improving the quality and realism of the generated images, and thus improving the performance of the generative model.

[0095] Step S300: The standardized pathological image is graded based on the pathological tissue grading model to obtain the pathological image grading result; wherein, the pathological tissue grading model includes a first convolution module, a residual module that introduces a sliding window multi-head attention mechanism, and a classification module. The first convolution module is used to extract local features of the standardized pathological image, the residual module is used to extract global features of the standardized pathological image, and the classification module is used to determine the pathological image grading result based on the fused features that integrate local and global features.

[0096] Specifically, the standardized pathological images obtained after staining and standardization are input into the pre-trained pathological tissue grading model to obtain the pathological image grading results output by the model. The pathological tissue grading model consists of multiple modules, including: a first convolutional module for extracting local features of the standardized pathological images, a residual module for extracting global features of the standardized pathological images, and a classification module for determining the pathological image grading results based on fused features that integrate local and global features. Among them, a sliding window multi-head attention mechanism is introduced in the residual module.

[0097] The following is combined with Figure 2 The diagram illustrates the algorithm structure of a pathological tissue grading method based on a sliding window attention mechanism. The pathological image to be processed is standardized by a pathological image staining standardization model. Then, the standardized pathological image is input into the pathological tissue grading model to obtain the pathological image grading result. The pathological tissue grading model will be explained in detail below.

[0098] (i) The first convolutional module consists of two basic convolutional layers. Each basic convolutional layer comprises a 3×3 convolutional layer for performing two-dimensional convolution operations, a batch normalization layer for normalizing the output of the convolutional layer and accelerating model training and enhancing model stability, and a LeakyReLU activation function layer with a negative slope parameter. The calculation process is shown in Equation (8):

[0099] x i+1 =LeakyReLU(BatchNorm(Conv(x)) i ))) (8)

[0100] Where, x i For input, x i+1 The output is given, and x satisfies i ∈R C×W×H R represents the real number field, x i ∈R C×W×H x represents i It is a real number value, where C, W, and H are the number of channels, height, and weight, respectively.

[0101] The input image undergoes initial feature extraction and dimensionality reduction operations through two basic convolutional layers. These layers process the input image with a series of convolutions, batch normalization, and activation functions to extract local features, which subsequently help the model better understand the content and structure of the input image. Through these two basic convolutional layers, the model can extract preliminary feature representations from the original input image and pass them to subsequent residual blocks for deeper feature extraction and representation learning. This initial feature extraction and dimensionality reduction helps the model learn image representations more effectively and reduces the number of parameters the model needs to learn, thereby improving the model's computational efficiency and generalization ability.

[0102] (ii) Input the local features initially extracted by the two convolutional layers in the previous step into the residual module to continue extracting global features. In the residual module, two second convolutional sub-modules are used first. The second convolutional sub-module includes a convolutional layer, a batch normalization layer, and a Mish activation function layer. The purpose of using two second convolutional sub-modules is to introduce nonlinear transformations and increase the expressive power of the model.

[0103] The first convolutional layer receives the input signal and applies convolution operations, padding, and the activation function Mish to extract features. This step introduces a nonlinear transformation, allowing the model to learn nonlinear relationships and complex feature representations. The second convolutional layer further processes the output of the first convolutional layer. Similarly, it applies convolution operations, padding, and the activation function Mish to further enhance the nonlinearity of the feature representation. Through these two convolutional layers, the residual attention class can stack and combine feature representations layer by layer, thereby increasing the depth and complexity of the model. This deep nonlinear transformation can better capture fine-grained features and high-level abstract representations in the input signal, and the calculation process is shown in Equation (9).

[0104]

[0105] Where x is the input, and the input and output of each stage satisfy x,x1,x2∈R. C×W×H R represents the real number field, x i ∈R C×W×H x represents i It is a real number value, where C, W, and H are the number of channels, height, and weight, respectively.

[0106] The second convolutional submodule in this application differs from the first convolutional module. Specifically, the Mish activation function replaces the ReLU activation function in the first convolutional module. This design takes into account the neuron death phenomenon associated with ReLU. Neuron death refers to the phenomenon where, during training, the activation values ​​of some neurons remain constant at zero, ceasing to update. This can cause some neurons in the network to become ineffective, unable to respond efficiently to input data, thus affecting the model's expressive power and performance. The reason is that when a neuron's output is less than or equal to zero, the derivative of the ReLU activation function is zero. This means that for inputs less than zero, the gradient during backpropagation is also zero, causing weight updates to stagnate. If the outputs of some neurons remain negative, the weights of these neurons may never be updated, resulting in neuron death.

[0107] Understandably, the activation function used in the second convolutional submodule of this application is the Mish activation function. The Mish activation function has characteristics different from other activation functions: First, the Mish activation function is unbounded at the upper limit but bounded at the lower limit. The unbounded nature avoids gradient saturation because it ensures the function value can increase infinitely, allowing the training process to maintain a relatively fast speed. The bounded nature achieves regularization, restricting the function's value to a finite range, thus limiting the model's degrees of freedom and reducing the risk of overfitting. Second, the Mish activation function is non-monotonic, which helps keep the value within a small negative range, thereby stabilizing the model's gradient flow and improving gradient propagation. Finally, regarding smoothness, the Mish activation function has a smoother slope near zero, making it easier to handle gradient vanishing and gradient exploding problems during optimization.

[0108] (III) After two convolutional layers, a patch embedding layer further transforms the feature maps extracted by convolution into a series of patch embedding representations. Since the feature maps typically have a large number of channels after two convolutional layers (for example, if the input pathological image is a 3-channel RGB image, after convolution operations in the first and second convolutional sub-modules, the resulting feature map has 128 channels), the patch embedding layer maps each patch to a lower-dimensional embedding space through convolution operations. This helps reduce the dimensionality of the features, lowers computational complexity, and improves model efficiency. Convolutional layers can extract local features of an image, but the feature map remains a global representation. By dividing the feature map into a series of patches and embedding each patch, the feature information of local regions can be better captured. Each patch embedding vector contains the feature information of the region where the patch is located, thus providing a perception of the local structure of the image. Then, after converting the feature map into patch embedding representations, the image can be viewed as a sequence, where each patch can be considered a sequence element. This transformation allows for the application of subsequent window self-attention models, enabling more flexible and efficient image processing. By introducing the concept of serialization, the advantages of sequence models can be leveraged to handle long-distance dependencies when processing images.

[0109] It should be noted that, in Figure 2 The image patch embedding layer is not shown. The image patch embedding layer can be set between the second convolutional submodule and the sliding window attention module, or it can be set in the sliding window attention module.

[0110] (iv) Following the image patch embedding layer, the residual module introduces a sliding window attention layer to further enhance the model's ability to perceive global contextual information. The sliding window attention layer introduces a sliding window multi-head attention mechanism (SWA) to handle the relationships between different positions of the input features. It uses projections of the query vector (Q), key vector (K), and value vector (V) to calculate attention weights and applies these weights to the value matrix to obtain a weighted representation. The input to the sliding window attention layer is a four-dimensional tensor. The layer first segments the input features into non-overlapping windows. Then, self-attention is calculated for each window to obtain representations of different positions within the window. The attention calculation uses multiple attention heads, each with its own query, key, and value projections. When calculating attention weights, the query and key matrices can be selectively scaled, and the attention weights can be regularized by applying dropout. Finally, by applying the attention weights to the value vector and merging the weighted value vectors, the output of the window multi-head self-attention module is obtained. The output shape is (B, H, W, C), which is the same as the shape of the input features, where B is the batch size, C is the number of channels, H is the height, and W is the width.

[0111] Overall, the sliding window attention layer models the relationships between different locations of the input features, extracting location information and long-distance cross-location interaction information, thereby helping to improve the model's performance in image classification tasks. The calculation process is shown in formula (10):

[0112] x1=SW-MSA(x) (10)

[0113] Where x is the input and x1 is the output, both satisfying x,x1∈R C×W×H R represents the real number field, and C, W, and H represent the number of channels, height, and weight, respectively.

[0114] (v) The processed features (output of the sliding window attention layer) are added to the residual connections (output of the first convolutional module) through the feature fusion layer and output through the feature output layer. The function of the feature fusion layer is to introduce residual connections, which allows information to propagate directly across layers in the network. The residual connections aim to solve the gradient vanishing and gradient explosion problems in the training process of deep neural networks. By directly adding the input signal to the output after a series of layers, the information of the original input can be preserved, making it easier for the network to learn the residual part, thereby improving the performance of the model and the training effect.

[0115] (vi) The classification module includes: an adaptive pooling layer, a linear flattening operation layer, a fully connected layer, and a Softmax function; among which...

[0116] The adaptive pooling layer is used to obtain the sampled features corresponding to the fused features through pooling operations. Specifically, the adaptive pooling layer reduces the size of the residual module output feature map to 1×1, performs global feature extraction and dimensional compression to obtain semantic information in the global scope, and at the same time reduces the number of parameters and computation. The adaptive pooling layer is a pooling operation that can automatically adjust the size of the pooling window according to the size of the input feature map, so that the size of the output feature map is fixed to the specified size. In this case, reducing the size of the feature map to 1×1 means performing pooling operations on the entire feature map, thereby capturing features in the global scope; the linear flattening operation layer is used to convert the sampled features into feature vectors, flatten the feature map into a one-dimensional vector, remove the spatial structure of the feature map, and convert it into the input for classification through the fully connected layer. The fully connected layer and the normalization exponent Softmax function are used to map the feature vector to the corresponding dimension of the output category. The fully connected layer maps the flattened feature vector to the dimension of the output category. The output category is the number of categories classified by the model, that is, the pathological image grading result is obtained. The calculation process of the adaptive pooling layer and the linear flattening operation layer is shown in formula (11):

[0117] y=Linear(AdaptiveMaxPool(x)) (11)

[0118] Where x is the input, satisfying x∈R C×W×H R represents the real number field, C, W, and H are the number of channels, height, and weight, respectively, and y is a one-dimensional array used for classification, satisfying y∈R class×1×1 , where class is the number of classes in the dataset used for model training.

[0119] In practical applications, the above model was trained on a private dataset of breast cancer pathology histology, and the final binary classification accuracy of the model on this dataset was 97.60%.

[0120] In summary, this application provides a pathological image staining normalization algorithm based on the CycleGAN model. By introducing a multi-head attention mechanism into the CycleGAN generator, it addresses the issues of cyclic consistency generative adversarial networks neglecting contextual information of lesion regions and the consistency of image geometry, as well as training instability, when performing pathological image staining normalization. Spectral normalization is employed to improve model stability, and geometric consistency loss is introduced to make the generated images more closely resemble real pathological samples. Furthermore, it utilizes a pathological tissue grading model to extract local and global features of pathological images and fuses them, enhancing the model's ability to extract contextual information. This effectively avoids the problem of convolutional neural networks lacking global information, improving the accuracy and computational efficiency of the grading model.

[0121] In an optional implementation, the pathological image staining normalization model provided in this application embodiment is obtained by pre-training an improved recurrent generative adversarial network model. The training process of the pathological image staining normalization model may include the following steps:

[0122] Step S201: Obtain the initial dataset, which includes images of pathological tissue sections at different grades.

[0123] Specifically, the initial dataset can come from private data collected from partner hospitals or downloaded from publicly available pathology datasets. The initial dataset includes pathological tissue slide images of different grades.

[0124] For different types of pathological tissue grading, the type of pathological tissue slide images in the initial dataset must correspond to the type of pathological tissue grading that the model to be built is targeting. For example, for the pathological histological grading of breast cancer, the initial dataset may include pathological tissue slide images of grade I breast cancer and grade II breast cancer. For the pathological histological grading of lung cancer, the initial dataset may include pathological tissue slide images of grade I lung cancer and grade II lung cancer.

[0125] For example, publicly available pathological image datasets for breast cancer include BreakHis, short for Breast Cancer Histoathological Image Classification; publicly available pathological image datasets for lung cancer include LC25000, short for Lung and Colon, which includes 25,000 histopathological images. It should be noted that this application does not limit the source of the initial dataset in its embodiments.

[0126] Step S202: Preprocess the initial dataset to obtain the preprocessed dataset. The preprocessing includes labeling and cropping.

[0127] Specifically, since the initial dataset is generally a full-scanned image of a pathological section with a very high pixel count of the image, in order to better adapt to the pathological image staining normalization model provided in this application, this application preprocesses the initial dataset. The preprocessing includes annotation and cropping processes. Among them, the annotation process includes: using the digital slide assistant system supporting the dedicated scanner in the pathology department to export the areas marked by a pathologist in the slide, in the jpeg format, and the image is exported under a 40x objective lens to ensure a relatively high pixel count of the image to facilitate subsequent cropping of the image; the cropping process includes: first setting the addresses of the original image folder and the saved image folder, and presetting the cropping size of the image to be 224 pixels in both height and width; then traversing the image file list, opening the image file and obtaining the original size of the image; then calculating the magnification factor, mainly calculating the magnification factor between the original image and the target size; then using a nested loop function to perform cropping according to the magnification factor, where the outer loop controls the rows and the inner loop controls the columns, and calculating the coordinates of the upper left corner and the lower right corner of the cropping area to complete the cropping; finally, naming and saving the cropped images in sequence. It should be noted that the cropping process can be implemented through code without manual operation.

[0128] The cropping process also includes: randomly selecting cropped images of different grades and adding them to the preprocessed dataset, which can avoid the problem of the decrease in the accuracy of the classification model caused by class imbalance.

[0129] Step S203: Obtain a first training dataset based on the preprocessed dataset. The first training dataset includes source-domain pathological images and target-domain pathological images. Among them, the source-domain pathological images are pathological images that have not undergone staining normalization processing, and the target-domain pathological images are the images expected to be obtained after the source-domain pathological images are subjected to staining normalization processing.

[0130] Specifically, obtain a first training dataset based on the preprocessed dataset obtained in the foregoing steps. The first training dataset is a dataset used to train the pathological image staining normalization model. The first training dataset includes source-domain pathological images and target-domain pathological images. The source-domain pathological images are images that need to undergo staining normalization processing, and the target-domain pathological images are the standardized images that the source-domain images are expected to be converted into.

[0131] The source-domain pathological images can be directly obtained based on the preprocessed dataset. The acquisition of the target-domain pathological images can be obtained through manual processing, or can be obtained through processing by the staining normalization model combined with manual fine-tuning, which is not limited herein.

[0132] Step S204: Construct an initial pathological image staining normalization model based on an improved recurrent generative adversarial network model. The improved recurrent generative adversarial network model includes a generator network and a loss function. The generator network includes a multi-head attention mechanism module and a spectral normalization module. The loss function includes a geometric consistency loss function.

[0133] Specifically, an initial pathological image staining normalization model is constructed based on an improved recurrent generative adversarial network model. The improved recurrent generative adversarial network model is introduced in the previous discussion and will not be elaborated here.

[0134] Step S205: Iteratively train the initial pathological image staining normalization model using the first training dataset to obtain the pathological image staining normalization model.

[0135] Specifically, an adversarial training approach is used to train the staining standardization model of the initial pathological images. In each training iteration, the training process is divided into two steps: the conversion from the source domain pathological image to the target domain pathological image and the conversion from the target domain pathological image to the source domain pathological image. The above steps are repeated to iteratively train the generator and discriminator of the model until the predetermined number of training rounds is reached or the loss converges.

[0136] In one optional implementation, the pathological tissue grading model provided in this application embodiment is obtained through pre-training. The training process of the pathological tissue grading model may include the following steps:

[0137] Step S301: Obtain the initial dataset, which includes images of pathological tissue sections at different grades.

[0138] This step is the same as step S201 mentioned above, and will not be repeated here.

[0139] Step S302: Obtain a second training dataset based on the preprocessed dataset. The second training dataset includes a training set and a test set.

[0140] Specifically, the preprocessed dataset can be divided into a training set and a test set in a 7:3 ratio.

[0141] Step S303: Construct an initial pathological tissue grading model, which includes a first convolutional module, a residual module that introduces a sliding window multi-head attention mechanism, and a classification module.

[0142] Step S304: Iteratively train the initial pathological tissue grading model using the second training dataset to obtain the pathological tissue grading model.

[0143] Specifically, an initial pathological tissue grading model is constructed. The algorithm structure for the initial pathological tissue grading model can be found in the appendix to this application. Figure 2The explanation is omitted here.

[0144] It should be noted that the model training step can be performed by a cloud-based computer. The local machine can be configured with the model to be trained, model parameters, and dataset address according to training requirements. The model parameters that need to be set include the number of classification categories (set according to the actual categories), the number of training epochs (for example, 300), the batch size (for example, 16 or 32), and the learning rate (for example, 1e-6, i.e., 1×10⁻⁶). -6 ).

[0145] In practical applications, to compare the model performance of the pathological tissue grading method based on the sliding window attention mechanism provided in the embodiments of this application, the accuracy of pathological image classification was compared between pure convolutional neural networks, pure Transformer models, and models that combine residuals with other attention mechanisms. The experimental results show that, compared with other models, the pathological tissue grading method based on the sliding window attention mechanism provided in the embodiments of this application can achieve higher results.

[0146] Corresponding to the methods described above, this application also provides a pathological tissue grading system based on a sliding window attention mechanism. This system aims to assist in the diagnosis of pathological tissue grading through human-computer interaction. The system is primarily implemented using a browser / server (B / S) architecture, employing the Django framework, and consists of a front-end and a back-end. The front-end utilizes HTML, CSS, and JS languages ​​to implement the interactive interface, while the back-end uses Python to implement functions such as training and calling deep learning models and using databases.

[0147] See Figure 3 The diagram shown illustrates the system's structure, which includes a front-end and a back-end. The front-end includes a doctor login module, a patient information management module, an image reading module, an intelligent diagnosis module, a diagnostic report module, and a model training module. The back-end includes an administrator login module, a doctor information management module, and a pathology image upload module.

[0148] Further reference Figure 4 The diagram shows a structural schematic of an intelligent diagnostic module. The intelligent diagnostic module 400 includes:

[0149] Image acquisition submodule 401 is used to acquire pathological images to be processed;

[0150] Image processing submodule 402 is used to perform staining standardization processing on the pathological image to be processed based on the pathological image staining standardization model to obtain a standardized pathological image.

[0151] The image grading submodule 403 is used to grade standardized pathological images based on a pathological tissue grading model to obtain pathological image grading results. The pathological tissue grading model includes a first convolution module, a residual module that introduces a sliding window multi-head attention mechanism, and a classification module. The first convolution module is used to extract local features of the standardized pathological image, the residual module is used to extract global features of the standardized pathological image, and the classification module is used to determine the pathological image grading results based on the fused features that integrate local and global features.

[0152] In one optional implementation, the image processing submodule 402 is specifically used to: input the pathological image to be processed into a pathological image staining normalization model to obtain a normalized pathological image; wherein, the pathological image staining normalization model is trained based on an improved recurrent generative adversarial network model, the improved recurrent generative adversarial network model including a generator network and a loss function, the generator network including a multi-head attention mechanism module and a spectral normalization module, and the loss function including a geometric consistency loss function.

[0153] In one alternative implementation, the first convolutional module in the pathological tissue grading model includes two basic convolutional layers; wherein the basic convolutional layer includes a convolutional layer, a batch normalization layer, and a LeakyReLU activation function layer.

[0154] In one optional implementation, the residual module in the pathological tissue grading model includes three residual attention sub-modules. The residual attention sub-module includes two second convolutional sub-modules, an image patch embedding layer, a sliding window attention layer, a feature fusion layer, and a feature output layer. The second convolutional sub-module includes a convolutional layer, a batch normalization layer, and a Mish activation function layer. The image patch embedding layer is used to reduce the feature dimension of the feature map output by the second convolutional sub-module. The feature fusion layer is used to fuse the local features output by the first convolutional module and the global features output by the sliding window attention layer to obtain fused features.

[0155] In one optional implementation, the classification module in the pathological tissue grading model includes: an adaptive pooling layer, a linear flattening operation layer, a fully connected layer, and a Softmax function; wherein, the adaptive pooling layer is used to obtain the sampled features corresponding to the fused features through pooling operations; the linear flattening operation layer is used to transform the sampled features into feature vectors; and the fully connected layer and the Softmax function are used to map the feature vectors to the corresponding dimensions of the output categories, thereby obtaining the pathological image grading results.

[0156] In one alternative implementation, refer to Figure 5 The diagram shows a structural schematic of a model training module 500, which includes:

[0157] The data acquisition submodule 501 is used to acquire the initial dataset, which includes pathological tissue slice images of different grades.

[0158] The data processing submodule 502 is used to preprocess the initial dataset to obtain the preprocessed dataset. The preprocessing includes labeling and cropping.

[0159] The data acquisition submodule 501 is also used to acquire a first training dataset based on the preprocessed dataset. The first training dataset includes source domain pathological images and target domain pathological images, wherein the source domain pathological images are pathological images that have not undergone staining and standardization, and the target domain pathological images are images that are expected to be obtained after staining and standardization of the source domain pathological images.

[0160] The model building submodule 503 is used to build an initial pathological image staining normalization model based on an improved recurrent generative adversarial network model. The improved recurrent generative adversarial network model includes a generator network and a loss function. The generator network includes a multi-head attention mechanism module and a spectral normalization module. The loss function includes a geometric consistency loss function.

[0161] The model training submodule 504 is used to iteratively train the initial pathological image staining normalization model using the first training dataset to obtain the pathological image staining normalization model.

[0162] In an optional implementation, the data acquisition submodule 501 is further configured to acquire a second training dataset based on the preprocessed dataset, the second training dataset including a training set and a test set;

[0163] The model building submodule 503 is also used to build an initial pathological tissue grading model, which includes a first convolution module, a residual module that introduces a sliding window multi-head attention mechanism, and a classification module.

[0164] The model training submodule 504 is also used to iteratively train the initial pathological tissue grading model using a second training dataset to obtain the pathological tissue grading model.

[0165] The following is combined with Figure 3 The following describes the various modules in the pathological tissue grading system based on the sliding window attention mechanism provided in the embodiments of this application:

[0166] Doctor login module and administrator login module: Whether you are a doctor or an administrator, you must first enter your username and password to access the system's front-end or back-end. All three—username, password, and verification code—must be correct; otherwise, you will be denied access and given an error message. To ensure system security, administrator usernames and passwords are preset in the database, while doctor usernames and passwords are added by the administrator, who also authorizes the corresponding permissions.

[0167] The specific operation process of the doctor login module and the administrator login module is as follows: After the administrator or doctor clicks the system icon on the user terminal, they enter the system login interface, then enter their account, password and real-time verification code, and click "Login". The system will trigger the backend dologin() method for the system login function. First, the verification code is checked, and then the login user's account and password are traversed in the database to determine whether they match the information in the database. If the verification is successful, the user can log in to the system. If the match fails, the user will be redirected to the login page and given an appropriate prompt.

[0168] The patient information management module is used for registering basic patient information and auxiliary diagnostic information, as well as uploading pathological images. It serves as the main interface for subsequent functions and includes patient information browsing, modification, and addition units. On the patient information management interface, doctors can browse information on previously treated patients, including basic details such as account number, name, gender, date of birth, contact number, and home address. Doctors can also modify patient information and add new information for newly treated patients.

[0169] The specific operation process of the patient information management module is as follows: After logging into the system, the doctor sends a command to the service layer to retrieve patient information. Upon receiving the command, the service layer executes the `index()` operation, traversing the database. If the patient information is found, it is listed separately for the doctor to proceed with further diagnosis; if not found, a corresponding prompt is given. The doctor then clicks the "Add" button to send add information to the service layer. After responding, the service layer executes the `insert()` operation to add the patient information to the database and display it on the interface.

[0170] Image viewing module: Used for viewing pathology images. Doctors click the "View" button on the viewing interface to view pathology images. The left side of the page displays the pathology image to be viewed, along with commonly used viewing buttons such as zoom in, zoom out, rotate, adjust brightness, and return to home position. The mouse wheel can be used to zoom in and out, and the left mouse button can be used to drag the image. The right side of the page displays the patient's pathology report, with a table showing the patient's basic information. In the "Visual Findings" section, doctors manually enter their diagnostic comments for this pathology image and click submit to save it to the database, which is then displayed on the page.

[0171] The image viewing module operates as follows: After the doctor clicks the "View" button in the image bar, they send a command to the service layer to view the image. Upon receiving the command, the service layer immediately responds by executing the `opensee()` function, retrieving the pathological image from the database, and displaying it on the front end. All image viewing operations are handled independently by the front-end code, ensuring a smooth viewing experience. After the doctor enters their diagnostic opinion on the front-end interface and confirms its accuracy, they click the "Submit" button, sending a command to the service layer. The service layer then executes the `dodiag()` function, saving the diagnostic opinion in the database for later use in generating a diagnostic report.

[0172] Intelligent Diagnosis Module: This module is used for intelligent diagnosis of patient pathological images. It achieves this function by calling a pre-trained deep learning model, as discussed above. After writing the manual review and diagnosis, the doctor returns to the pathological image management page and clicks the "AI Diagnosis" button on the page to perform an intelligent diagnosis of the pathological image. Once completed, the doctor will be redirected to the AI ​​diagnosis interface, which includes the pathological image, the intelligent diagnosis result, and the manual diagnosis result stored in the database, assisting the doctor in making a comprehensive evaluation of the diagnosis of the pathological image.

[0173] The specific operation process of the intelligent diagnosis module is as follows: when the doctor clicks the "AI Diagnosis" button, the front end sends a command to the service layer. The service layer receives the pathological image name and retrieves the pathological image from the system. Then, the service layer responds by executing the aidetect() function to call the pathological tissue grading model to perform intelligent grading diagnosis of the pathological image. Finally, the diagnosis results are saved to the database and displayed on the front end interface.

[0174] Diagnostic Report Module: After completing manual and intelligent diagnoses, doctors can click the "Diagnostic Report" button on the pathology image management page to jump to the diagnostic report page. The diagnostic report page uses the actual pathology diagnostic report as a template, which includes basic patient information, basic pathology image information, manual and intelligent diagnostic results, etc. The doctor will give a final diagnostic opinion after comprehensive evaluation. Afterwards, clicking the "Export PDF" button will download the pathology diagnostic report for easy printing and retention later.

[0175] The diagnostic report module operates as follows: The method for generating diagnostic reports is implemented using front-end code. First, a diagnostic report table is created on the front-end, and the content from the database is displayed accordingly. Then, the diagnostic report portion of the front-end interface is captured, generated into a PDF format, and named. Specifically, after the doctor clicks "Diagnostic Report," the server executes the `report()` function, accesses the database, executes the `get()` method to retrieve the data, and displays the data in the corresponding location on the front-end interface. After the doctor clicks the "Export PDF" button, the front-end executes the `exportPDF()` function to generate the diagnostic report PDF file and saves it locally.

[0176] Model Training Module: In addition to meeting doctors' diagnostic needs, this system also includes a model training module for clinical research. After entering the clinical research page, doctors select the model to be trained on the left and input training parameters, such as classification category, number of training epochs, and learning rate, then click the "Start Training" button. The system will then begin training the model; the training time depends on the hardware configuration and parameter settings. After training is complete, the right side of the page will display the model's accuracy, sensitivity, and specificity, along with accuracy and loss curves. Doctors can then click the "View Log" button to review the training log, understanding the specific process of each training epoch for reference in clinical research.

[0177] The specific operation process of the model training module is as follows: The doctor clicks "Clinical Research" in the navigation bar to enter the model training page, selects the corresponding model and sets the parameters, and then clicks the "Start Training" button. The front end sends an instruction to the service layer, and the service layer responds by executing the train() function, receiving the parameters and calling the corresponding model for training. After the training is completed, the training data is returned and sent to the front end for display. At the same time, the model weights and training logs are saved in the database for subsequent use in intelligent diagnosis.

[0178] Doctor Information Management Module: This module is based on the administrator's login backend. The administrator can operate this module to manage doctor information, including adding, modifying and deleting doctor information, and modifying and adjusting the relationship between doctors and departments.

[0179] Pathology Image Upload Module: In this system, the administrator is primarily responsible for uploading pathology images. In practical applications, most unprocessed pathology images are WSI images, typically stored in TIFF format. A single pathology image is quite large, occupying approximately 1 to 2 GB of storage space, making them unviewable directly on the front-end interface and requiring image format conversion. Simultaneously, to save doctors' diagnostic time and simplify their diagnostic procedures, the administrator performs format conversion and uploading of pathology images, storing them in the database and displaying them on the doctor's interface for convenient image review.

[0180] The specific operation of the pathology image upload module is as follows: Before uploading a pathology image, the administrator pre-determines the image format. If it is a WSI image, they click the "WSI Format Conversion" button, enter parameters on the left side of the page, including the WSI image folder address, image index, and target storage folder address, and then click the "Start Conversion" button. The front-end sends a command, and the service layer receives the parameters from the front-end and executes the `dowsijpg()` function, calling the format conversion model to perform the format conversion. After the conversion is complete, the administrator uploads the converted pathology image to the database. Specifically, the administrator clicks the "Upload Pathology Image" button, selects the pathology image on the upload page, and clicks the "Submit" button. The front-end sends a command, and the service layer responds by executing the `doupdate()` function, connecting to the database, uploading the pathology image address to the database for storage, and simultaneously returning the patient's pathology image to the doctor's system for subsequent diagnostic operations. If the pathology image is in a general format, it is uploaded directly.

[0181] It should be noted that the steps and related technical features of each module in the pathological tissue grading system based on the sliding window attention mechanism provided in this application correspond to the method provided in the application. The description of the device part can be found in the embodiments of the aforementioned method part, and will not be repeated here.

[0182] In summary, this application provides a pathological tissue grading system based on a sliding window attention mechanism. This system enables a combined manual and intelligent diagnostic process for pathological images. First, by registering and uploading basic patient information, including pathological image information, subsequent image viewing is performed, followed by a preliminary manual diagnosis. Then, the intelligent diagnosis module calls a trained pathological tissue grading model to fully extract local and global information from the pathological images, improving diagnostic accuracy and reducing misdiagnosis and missed diagnosis. Simultaneously, the system includes functions to assist medical research, namely model training. Model training generates the model's corresponding weight file and the final training results. Furthermore, this system is implemented based on a B / S architecture, overcoming the limitations of C / S architecture systems on local hardware. System functions can be accessed through a browser, and the system can be hosted in the cloud, enabling medical resource sharing and empowering grassroots hospitals with limited resources. This not only reduces the burden on doctors but also improves the accuracy of their pathological diagnoses, reducing the rates of misdiagnosis and missed diagnosis.

[0183] See below. Figure 6 The diagram illustrates the operation of a pathological tissue grading system based on a sliding window attention mechanism. After the administrator logs into the system, the backend grants login permissions to doctors. Doctors then log into the system frontend to perform subsequent operations. First, they register patient information. After the administrator uploads the corresponding pathological images of the patient, the doctors perform manual image review and diagnosis, as well as intelligent diagnosis using the pathological tissue grading model. The diagnostic results are then combined to form a diagnostic report, which is downloaded, printed, and signed by the reviewing doctor. This improves the accuracy of doctors' pathological diagnoses and reduces the rates of misdiagnosis and missed diagnosis.

[0184] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0185] Those skilled in the art will understand that the flowchart shown is merely an example in which the embodiments of this application can be implemented, and the scope of application of the embodiments of this application is not limited by any aspect of the flowchart.

[0186] In the several embodiments provided in this application, it should be understood that the disclosed methods, apparatuses, and devices can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual couplings or direct couplings or communication connections may be through some communication interfaces; indirect couplings or communication connections between devices or units may be electrical, mechanical, or other forms.

[0187] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs. Furthermore, the functional units in the various embodiments of this application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0188] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0189] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A pathological tissue grading method based on a sliding window attention mechanism, characterized in that, The method includes: Acquire the pathological images to be processed; The pathological image to be processed is subjected to staining standardization processing based on the pathological image staining standardization model to obtain a standardized pathological image. The standardized pathological images are graded based on a pathological tissue grading model to obtain the pathological image grading results. The pathological tissue grading model includes a first convolutional module, a residual module incorporating a sliding window multi-head attention mechanism, and a classification module. The first convolutional module extracts local features of the standardized pathological images, the residual module extracts global features of the standardized pathological images, and the classification module determines the pathological image grading results based on a fused feature that integrates the local and global features. The residual module includes three residual attention sub-modules: two second convolutional sub-modules, an image patch embedding layer, a sliding window attention layer, a feature fusion layer, and a feature output layer.

2. The method according to claim 1, characterized in that, The staining standardization process based on the pathological image staining standardization model is used to perform staining standardization on the pathological image to be processed, resulting in a standardized pathological image, including: The pathological image to be processed is input into the pathological image staining normalization model to obtain the normalized pathological image; wherein, the pathological image staining normalization model is trained based on an improved recurrent generative adversarial network model, the improved recurrent generative adversarial network model includes a generator network and a loss function, the generator network includes a multi-head attention mechanism module and a spectral normalization module, and the loss function includes a geometric consistency loss function.

3. The method according to claim 1, characterized in that, The first convolutional module includes two basic convolutional layers; wherein, the basic convolutional layer includes a convolutional layer, a batch normalization layer, and a LeakyReLU activation function layer.

4. The method according to claim 3, characterized in that, The second convolutional submodule includes a convolutional layer, a batch normalization layer, and a Mish activation function layer. The image patch embedding layer is used to reduce the feature dimension of the feature map output by the second convolutional submodule. The feature fusion layer is used to fuse the local features output by the first convolutional module and the global features output by the sliding window attention layer to obtain the fused features.

5. The method according to claim 4, characterized in that, The classification module includes: an adaptive pooling layer, a linear flattening operation layer, a fully connected layer, and a Softmax function; wherein, the adaptive pooling layer is used to obtain the sampled features corresponding to the fused features through pooling operations; the linear flattening operation layer is used to convert the sampled features into feature vectors; the fully connected layer and the Softmax function are used to map the feature vectors to the corresponding dimensions of the output categories, thereby obtaining the pathological image grading results.

6. The method according to any one of claims 2 to 5, characterized in that, The training process of the pathological image staining standardization model includes: Obtain an initial dataset, which includes pathological tissue slice images of different grades; The initial dataset is preprocessed to obtain a preprocessed dataset. The preprocessing includes annotation and cropping. A first training dataset is obtained based on the preprocessed dataset. The first training dataset includes source domain pathological images and target domain pathological images. The source domain pathological images are pathological images that have not undergone staining and normalization, and the target domain pathological images are images that are expected to be obtained after staining and normalization of the source domain pathological images. An initial pathological image staining normalization model is constructed based on an improved recurrent generative adversarial network model. The improved recurrent generative adversarial network model includes a generator network and a loss function. The generator network includes a multi-head attention mechanism module and a spectral normalization module. The loss function includes a geometric consistency loss function. The initial pathological image staining normalization model is iteratively trained using the first training dataset to obtain the pathological image staining normalization model.

7. The method according to claim 6, characterized in that, The training process of the pathological tissue grading model includes: A second training dataset is obtained based on the preprocessed dataset, the second training dataset including a training set and a test set; An initial pathological tissue grading model is constructed, which includes a first convolutional module, a residual module that introduces a sliding window multi-head attention mechanism, and a classification module. The initial pathological tissue grading model is iteratively trained using the second training dataset to obtain the pathological tissue grading model.

8. A pathological tissue grading system based on a sliding window attention mechanism, characterized in that, The system includes: an intelligent diagnostic module; The intelligent diagnostic module includes: The image acquisition submodule is used to acquire pathological images to be processed; The image processing submodule is used to perform staining standardization processing on the pathological image to be processed based on the pathological image staining standardization model to obtain a standardized pathological image. The image grading submodule is used to grade the standardized pathological image based on the pathological tissue grading model to obtain the pathological image grading result. The pathological tissue grading model includes a first convolutional module, a residual module incorporating a sliding window multi-head attention mechanism, and a classification module. The first convolutional module extracts local features of the standardized pathological image, the residual module extracts global features of the standardized pathological image, and the classification module determines the pathological image grading result based on a fused feature that integrates the local and global features. The residual module includes three residual attention submodules: two second convolutional submodules, an image patch embedding layer, a sliding window attention layer, a feature fusion layer, and a feature output layer.

9. The system according to claim 8, characterized in that, The image processing submodule is specifically used for: inputting the pathological image to be processed into the pathological image staining normalization model to obtain the normalized pathological image; wherein, the pathological image staining normalization model is trained based on an improved recurrent generative adversarial network model, the improved recurrent generative adversarial network model includes a generator network and a loss function, the generator network includes a multi-head attention mechanism module and a spectral normalization module, and the loss function includes a geometric consistency loss function.

10. The system according to claim 9, characterized in that, The system also includes: a model training module; The model training module includes: The data acquisition submodule is used to acquire an initial dataset, which includes pathological tissue slice images of different grades; The data processing submodule is used to preprocess the initial dataset to obtain a preprocessed dataset. The preprocessing includes annotation and cropping. The data acquisition submodule is further configured to acquire a first training dataset based on the preprocessed dataset. The first training dataset includes source domain pathological images and target domain pathological images, wherein the source domain pathological images are pathological images that have not undergone staining and standardization, and the target domain pathological images are images that are expected to be obtained after staining and standardization of the source domain pathological images. The model building submodule is used to build an initial pathological image staining normalization model based on an improved recurrent generative adversarial network model. The improved recurrent generative adversarial network model includes a generator network and a loss function. The generator network includes a multi-head attention mechanism module and a spectral normalization module. The loss function includes a geometric consistency loss function. The model training submodule is used to iteratively train the initial pathological image staining normalization model using the first training dataset to obtain the pathological image staining normalization model.

Images