A pathological slice feature fusion method based on background-guided attention mechanism
By constructing a background vector-guided attention mechanism and fusing features from pathological image slices, the problem of existing methods failing to effectively utilize environmental information in pathological images is solved, achieving more efficient and accurate feature fusion and classification, and improving the performance of computer-aided diagnostic systems.
Patent Information
- Application Number
- CN202111398932.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-11-19
- Publication Date
- 2024-11-19
- Estimated Expiration
- 2041-11-19
AI Technical Summary
Existing methods for fusing features from pathological image sections fail to effectively utilize the environmental information of the sections, resulting in inaccurate and incomplete feature extraction. Furthermore, existing attention mechanisms ignore the overall environment of the pathological image, affecting diagnostic accuracy.
By constructing a background vector and using a cyclical step to guide the calculation of attention weights for the slices, the features of the pathological image are fused to reflect the global environmental information of the pathological image. A pre-trained deep neural network is used to extract slice features and generate an initial background vector through linear transformation. The background vector is cyclically updated to calculate the attention weights of the slices, and finally the fused features are obtained by weighted summation.
It improves the accuracy and quality of feature fusion, enhances the precision of subsequent classification tasks, reduces computation time and memory requirements, and significantly improves the performance of computer-aided diagnostic systems.
Smart Images

Figure CN114118258B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of automatic diagnosis of tissue pathology images, and in particular to a pathology slice feature fusion method based on a background-guided attention mechanism, which is an improvement on existing machine learning models for medical image analysis tasks. Background Art
[0002] In the diagnosis of many different types of cancer, the results of pathological image analysis are the gold standard and can be of great help in early diagnosis. In developed countries, the application of pathological image analysis technology has greatly reduced the mortality rate of breast cancer. However, it takes a lot of time to analyze pathological images through manual reading, and the diagnosis results are greatly affected by the subjective judgment of doctors. Therefore, the development of a reliable computer-aided diagnosis system is of great significance to alleviate the pressure on medical resources.
[0003] Traditional automatic analysis methods generally include two steps, feature engineering and classification. In the feature engineering stage, these methods require pathologists to manually design features, such as texture features, statistical features, and structural features of the image, and then extract these features from the image through image processing technology. Then, classic machine learning models, such as support vector machines, are used to classify the extracted features. The disadvantage of manually designed features is that the selected features are highly dependent on the knowledge and clinical experience of pathologists. Since these models cannot automatically adjust features according to data, the extracted features may not fully reflect the nature of the image due to insufficient existing knowledge, which ultimately affects the accuracy of classification.
[0004] With the development of deep learning, deep learning models based on convolutional neural networks have achieved great success in the field of computer vision. Since they can automatically learn features from data, deep learning-based models just make up for the shortcomings of manually designed features. The special thing about pathological images is that they are extremely large in size. Researchers need to divide a whole pathological image into many smaller slices and extract features. The labels of these slices are unknown. Therefore, after extracting the features of the slices through the deep learning model, how to fuse these features to obtain the global features that reflect the entire pathological image has become a key issue in building a computer-aided diagnosis system.
[0005] In the existing technology, researchers have proposed some solutions to this problem, including treating slice features as a sequence and fusing them using a recurrent neural network; using regularized pooling; splicing slice features together and treating them as a compressed image, etc.
[0006] The inventors of this application found that the prior art has at least the following technical problems during the implementation of the present invention:
[0007] The above methods all have some unsolvable problems. Since a pathological image contains a large number of slices, the method based on recurrent neural network faces the problem of gradient vanishing. The method based on regular pooling lacks learnable parameters, so it cannot distinguish which slices are critical to diagnosis. The feature dimension obtained by the method of compressing images is too high, making subsequent classification and other tasks difficult to carry out. The attention pooling method can solve the above problems and has become the most advanced method at present. However, the existing methods based on attention mechanism still have problems. The most important problem is that the existing attention mechanism only focuses on the characteristics of the slice itself, but ignores the environment in which the pathological image is located, which leads to the extracted features being inaccurate. Summary of the invention
[0008] The present invention solves the problem that the slice feature fusion method based on attention in the prior art cannot utilize the environmental information of the slice, resulting in inaccurate and incomplete feature extraction. A background vector reflecting the overall environmental information in the pathological image is constructed through a loop step, and then the attention weight of the slice is calculated under the guidance of the background vector. The weight is used to weightedly sum and fuse the features of the slice to obtain the feature representation of the entire pathological image, that is, the fused feature, which can improve the accuracy of feature fusion.
[0009] In order to solve the above technical problems, the present invention provides a pathological slice feature fusion method based on a background-guided attention mechanism, comprising:
[0010] S1: A pre-trained deep neural network is used to extract features of pathological image slices, and a linear transformation is used to convert the extracted features of pathological image slices into d-dimensional features;
[0011] S2: randomly generating an initial background vector of d dimensions according to a d-dimensional standard normal distribution, where the initial background vector is used to represent an estimate of the global background information of the pathological image at the initial moment;
[0012] S3: Starting from the initial background vector generated in step S2, a background vector of the pathological image is obtained through a loop module, and the background vector is used to represent the global background information of the pathological image;
[0013] S4: Using the background vector obtained in step S3, calculate the attention weight of each slice feature, and the attention weight is used to reflect the importance of each slice when the feature is fused;
[0014] S5: Perform weighted summation of the slice features and the corresponding attention weights to obtain the fused pathological image features.
[0015] In one embodiment, the linear transformation used in step S1 is:
[0016]
[0017] in, represents the feature of the i-th slice in the pathological image, W e and b e are respectively the parameter matrix and bias matrix of the above linear transformation. The linear transformation is implemented through a linear layer of a neural network, and the slice feature dimension obtained after the transformation is d-dimensional.
[0018] In one embodiment, the density function of the normal distribution in step S2 is:
[0019]
[0020] Among them, Σ is the covariance matrix of the distribution, which is equal to a unit matrix, μ is the mean vector of the distribution, all elements are 0, and when generating the initial background vector q0, the probability that q0 is equal to any d-dimensional vector x is f(x).
[0021] In one embodiment, step S3 includes:
[0022] S3.1: According to the initial background vector q0, the attention weight of the slice at step 0 is calculated, where the attention weight of the i-th slice at step 0 is calculated as:
[0023]
[0024] h i is the feature vector of the i-th slice obtained after the linear transformation in step S1;
[0025] S3.2: Calculate the weighted sum of all slice feature vectors to get the output vector o0 at step 0:
[0026]
[0027] N is the total number of slices in this pathology image;
[0028] S3.3: Use two linear transformations to calculate the background vector q1 for the next step. The matrices of the two linear transformations are W q , b q and W o , b o , W q and b q are the parameter matrix and bias matrix for linear transformation of background vector q0, W o and b o are the parameter matrix and bias matrix for linear transformation of the output vector o0, respectively, and are calculated as follows:
[0029] q1=sigmoid(W qq0+b q +W o o0+b o )
[0030] The linear transformation is completed by the linear layer of the neural network. The sigmoid in the formula is a nonlinear activation function, which is defined as:
[0031]
[0032] And so on, at each step t, according to the background vector q t Calculate the attention weight of the tth step with the slice feature The weighted sum is taken as the output vector o of the tth step t , and then use the output vector o of step t t And the linear transformation updates the background vector:
[0033] q t+1 =sigmoid(W q q t +b q +W o o t +b o )
[0034] q t+1 is the updated background vector obtained after step t;
[0035] S3.4: After executing S3.1 to S3.3T times in a loop, a background vector q is obtained to reflect the environmental information of the slice in the pathological image. T .
[0036] In one embodiment, the calculation method of the attention weight of the slice in step S4 is:
[0037]
[0038] In the formula, q T represents the background vector, h i is the feature vector of the i-th slice obtained after the linear transformation in step S1, a i represents the attention weight of the i-th slice.
[0039] In one embodiment, the calculation method of the fused pathological image features in step S5 is:
[0040]
[0041] In the formula, a i represents the attention weight of the i-th slice, h iis the feature vector of the i-th slice obtained after the linear transformation in step S1, and s represents the fused pathological image features.
[0042] The above one or more technical solutions in the embodiments of the present application have at least one or more of the following technical effects:
[0043] 1. The present invention proposes a method for constructing a background vector of a pathological image, which can better reflect the global environmental information of the pathological image in which the slice is located. When the background vector is classified using a machine model such as a support vector machine, the background vector shows a high correlation with the category of the pathological image itself.
[0044] 2. Based on the constructed background vector, the present invention uses a method for calculating the attention weight of pathological image slice features guided by the background vector. In this method, the relative weight of a slice is not only related to the slice itself, but also inseparable from the overall environment of the pathological image in which it is located. This method improves the accuracy of the fused features, effectively improves the quality of the fused pathological image features, and significantly improves the accuracy of subsequent classification tasks.
[0045] 3. The number of slices contained in the pathological image is large. Compared with the previous attention-based method, the feature fusion method proposed in the present invention requires shorter calculation time and occupies less memory. BRIEF DESCRIPTION OF THE DRAWINGS
[0046] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0047] Figure 1 A workflow diagram of a pathological slice feature fusion method based on a background-guided attention mechanism provided in an embodiment of the present invention (taking breast tissue pathology images as an example);
[0048] Figure 2 It is a partial area of a tissue pathology whole slide image used in an embodiment of the present invention. DETAILED DESCRIPTION
[0049] The present invention discloses a pathological slice feature fusion method based on a background-guided attention mechanism, the steps of the method include: S1: using a pre-trained deep neural network to extract the features of the pathological image slice, and using a linear transformation to convert it into a d-dimensional feature; S2: randomly generating a d-dimensional initial background vector according to the d-dimensional normal distribution; S3: starting from the initial background vector generated in step 2, through a loop module, obtaining the background vector of the pathological image; S4: using the background vector obtained in step 3, calculating the attention weight of each slice feature; S5: weighting and summing the features according to their attention weights to obtain the fused pathological image features. The present invention uses an attention mechanism based on background guidance to construct a background vector that fully reflects the environmental information of the slice in the pathological image, and can calculate the weight of the slice in a shorter time using a smaller memory, obtain a higher quality fusion feature, help develop a better computer-aided diagnosis system, and alleviate the problem of tight medical resources.
[0050] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0051] The embodiment of the present invention provides a pathological slice feature fusion method based on a background-guided attention mechanism, comprising:
[0052] S1: A pre-trained deep neural network is used to extract features of pathological image slices, and a linear transformation is used to convert the extracted features of pathological image slices into d-dimensional features;
[0053] S2: randomly generating an initial background vector of d dimensions according to a d-dimensional standard normal distribution, where the initial background vector is used to represent an estimate of the global background information of the pathological image at the initial moment;
[0054] S3: Starting from the initial background vector generated in step S2, a background vector of the pathological image is obtained through a loop module, and the background vector is used to represent the global background information of the pathological image;
[0055] S4: Using the background vector obtained in step S3, calculate the attention weight of each slice feature, and the attention weight is used to reflect the importance of each slice when the feature is fused;
[0056] S5: Perform weighted summation of the slice features and the corresponding attention weights to obtain the fused pathological image features.
[0057] During the specific implementation process, the relevant linear layers are trained according to the extracted pathological image slice features to obtain a slice feature fusion model based on the background guided attention mechanism, and the slice features are fused using the slice feature fusion model.
[0058] In S1, the pre-trained deep neural network can be a residual 50 network (ResNet50). The initial background vector in step S2 represents the guess of the global background information of the pathological image at the initial moment, and step S3 obtains the background vector of the pathological image, which represents the global background information. The weight representing the importance of each slice is then calculated during feature fusion, and finally the extracted slice features are weighted and summed according to their attention weights to obtain the fused pathological image features.
[0059] Please see attached Figure 1 , which is a flowchart of the pathology image slice fusion method based on background guided attention mechanism, taking breast tissue pathology image as an example.
[0060] In one embodiment, the linear transformation used in step S1 is:
[0061]
[0062] in, represents the feature of the i-th slice in the pathological image, W e and b e are the parameter matrix and bias matrix of the above linear transformation respectively. The linear transformation is realized by a linear layer of a neural network. The slice feature dimension obtained after the transformation is d-dimensional, h i is the slice feature obtained after linear transformation.
[0063] In one embodiment, the density function of the normal distribution in step S2 is:
[0064]
[0065] Among them, Σ is the covariance matrix of the distribution, which is equal to a unit matrix, μ is the mean vector of the distribution, all elements are 0, and when generating the initial background vector q0, the probability that q0 is equal to any d-dimensional vector x is f(x).
[0066] In one embodiment, step S3 includes:
[0067] S3.1: According to the initial background vector q0, the attention weight of the slice at step 0 is calculated, where the attention weight of the i-th slice at step 0 is calculated as:
[0068]
[0069] h iis the feature vector of the i-th slice obtained after the linear transformation in step S1;
[0070] S3.2: Calculate the weighted sum of all slice feature vectors to get the output vector o0 at step 0:
[0071]
[0072] N is the total number of slices in this pathology image;
[0073] S3.3: Use two linear transformations to calculate the background vector q1 for the next step. The matrices of the two linear transformations are W q , b q and W o , b o , W q and b q are the parameter matrix and bias matrix for linear transformation of background vector q0, W o and b o are the parameter matrix and bias matrix for linear transformation of the output vector o0, respectively, and are calculated as follows:
[0074] q1=sigmoid(W q q0+b q +W o o0+b o )
[0075] The linear transformation is completed by the linear layer of the neural network. The sigmoid in the formula is a nonlinear activation function, which is defined as:
[0076]
[0077] And so on, at each step t, according to the background vector q t Calculate the attention weight of the tth step with the slice feature The weighted sum is taken as the output vector o of the tth step t , and then use the output vector o of step t t And the linear transformation updates the background vector:
[0078] q t+1 =sigmoid(W q q t +b q +W o o t +b o )
[0079] q t+1 is the updated background vector obtained after step t;
[0080] S3.4: After executing S3.1 to S3.3T times in a loop, a background vector q is obtained to reflect the environmental information of the slice in the pathological image. T .
[0081] Step S3.3 is a cyclic execution process, which is: at each time step t, the background vector q at that moment is used t and the slice feature h i The process of calculating the attention weights, performing weighted summation, and then using linear transformation to calculate the background vector for the next moment.
[0082] Steps S3.1 to S3.3 are repeated for T steps to obtain a final background vector reflecting the environment information of the slice in the pathological image. The repeated T steps are two steps in this embodiment.
[0083] In one embodiment, the calculation method of the attention weight of the slice in step S4 is:
[0084]
[0085] In the formula, q T represents the background vector, h i is the feature vector of the i-th slice obtained after the linear transformation in step S1, a i represents the attention weight of the i-th slice.
[0086] In one embodiment, the calculation method of the fused pathological image features in step S5 is:
[0087]
[0088] In the formula, a i represents the attention weight of the i-th slice, h i is the feature vector of the i-th slice obtained after the linear transformation in step S1, and s represents the fused pathological image features.
[0089] Compared with the prior art, the beneficial effects of the present invention are:
[0090] 1. The present invention proposes a method for constructing a background vector of a pathological image, which can better reflect the global environmental information of the pathological image in which the slice is located. When the background vector is classified using a machine model such as a support vector machine, the background vector shows a high correlation with the category of the pathological image itself.
[0091] 2. Based on the constructed background vector, the present invention uses a method for calculating the attention weight of pathological image slice features guided by the background vector. In this method, the relative weight of a slice is not only related to the slice itself, but also inseparable from the overall environment of the pathological image in which it is located. This method effectively improves the quality of the fused pathological image features, and significantly improves the accuracy of subsequent classification tasks.
[0092] 3. The number of slices contained in the pathological image is large. Compared with the previous attention-based method, the feature fusion method proposed in the present invention requires shorter calculation time and occupies less memory.
[0093] The specific experimental data below further illustrate the beneficial effects and advancement of the method of the present invention. Figure 2 , is a partial area of the tissue pathology whole slide image used in the experiment of the present invention.
[0094] On the CAMELYON16 dataset used in the specific implementation, the classification index achieved by the best method before was only 0.822. The present invention improves on the original method based on the attention mechanism, uses the overall environmental information of the pathological image to calculate the attention weight, and then fuses the features of the slices, effectively improving the diagnostic accuracy of the original method, reaching 0.883 on the CAMELYON16 dataset used in the experiment. The classification index of the experimental support vector machine classification of the background vector constructed according to the method of the present invention reached 0.79, while it was only 0.17 when the average value of the slice features was used as the background vector. In the CAMELYON16 dataset, each pathological image contains about 100,000 slices. At this time, the method proposed by the present invention takes 74% less time and occupies 36% less space than the previous method based on the attention mechanism.
[0095] Finally, it should be noted that the above is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art can still modify the technical solutions described in the aforementioned embodiments or replace some of the technical features therein by equivalents. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.
Claims
1. A pathological slice feature fusion method based on background-guided attention mechanism, characterized in that: include: S1: A pre-trained deep neural network is used to extract features of pathological image slices, and a linear transformation is used to convert the extracted features of pathological image slices into d-dimensional features; S2: randomly generating an initial background vector of d dimensions according to a d-dimensional standard normal distribution, where the initial background vector is used to represent an estimate of the global background information of the pathological image at the initial moment; S3: Starting from the initial background vector generated in step S2, a background vector of the pathological image is obtained through a loop module, and the background vector is used to represent the global background information of the pathological image; S4: Using the background vector obtained in step S3, calculate the attention weight of each slice feature, and the attention weight is used to reflect the importance of each slice when the feature is fused; S5: Perform weighted summation of slice features and corresponding attention weights to obtain fused pathological image features; Wherein, step S3 comprises: S3.1: According to the initial background vector q0, the attention weight of the slice at step 0 is calculated, where the attention weight of the i-th slice at step 0 is calculated as: h i is the feature vector of the i-th slice obtained after the linear transformation in step S1; S3.2: Calculate the weighted sum of all slice feature vectors to get the output vector o0 at step 0: N is the total number of slices in this pathology image; S3.3: Use two linear transformations to calculate the background vector q1 for the next step. The matrices of the two linear transformations are W q , b q and W o , b o , W q and b q are the parameter matrix and bias matrix for linear transformation of background vector q0, W o and b o are the parameter matrix and bias matrix for linear transformation of the output vector o0, respectively, and are calculated as follows: q1=sigmoid(W q q0+b q +W o o0+b o ) The linear transformation is completed by the linear layer of the neural network. The sigmoid in the formula is a nonlinear activation function, which is defined as: And so on, at each step t, according to the background vector q t Calculate the attention weight of the tth step with the slice feature The weighted sum is taken as the output vector o of the tth step t , and then use the output vector o of step t t And the linear transformation updates the background vector: q t+1 =sigmoid(W q q t +b q +W o o t +b o ) q t+1 is the updated background vector obtained after step t; S3.4: After executing S3.1 to S3.3T times in a loop, a background vector q is obtained to reflect the environmental information of the slice in the pathological image. T .
2. The pathological section feature fusion method according to claim 1, characterized in that: The linear transformation used in step S1 is: in, represents the feature of the i-th slice in the pathological image, W e and b e are respectively the parameter matrix and bias matrix of the above linear transformation. The linear transformation is implemented through a linear layer of a neural network, and the slice feature dimension obtained after the transformation is d-dimensional.
3. The pathological section feature fusion method according to claim 1, characterized in that: The density function of the normal distribution in step S2 is: Among them, Σ is the covariance matrix of the distribution, which is equal to a unit matrix, μ is the mean vector of the distribution, all elements are 0, and when generating the initial background vector q0, the probability that q0 is equal to any d-dimensional vector x is f(x).
4. The pathological section feature fusion method according to claim 1, characterized in that: The calculation method of the attention weight of the slice in step S4 is: In the formula, q T represents the background vector, h i is the feature vector of the i-th slice obtained after the linear transformation in step S1, a i represents the attention weight of the i-th slice.
5. The pathological section feature fusion method according to claim 1, characterized in that: The calculation method of the fused pathological image features in step S5 is: In the formula, a i represents the attention weight of the i-th slice, h i is the feature vector of the i-th slice obtained after the linear transformation in step S1, and s represents the fused pathological image features.
