Cervical tissue pathological image classification method based on hybrid CNN-ViT architecture

Abstract

The invention discloses a cervical tissue pathology image classification method based on a hybrid CNN-ViT architecture, and relates to the technical field of computer vision and deep learning. According to the method, firstly, on the basis of combining CNN and ViT architectures, a linear angle cross covariance self-attention module (LACCSA) and a novel activation function ExpReLU are provided aiming at the defects of an existing method in feature extraction, classification performance and model adaptability; the LACCSA module is combined with cross covariance attention and linear angle similarity to effectively capture a dynamic relationship between global features, subtle differences between classes can be distinguished, common features in the classes can be extracted, and the precision of complex pathological image classification is remarkably improved; the ExpReLU activation function reduces the influence of data distribution offset on the model performance by enhancing the nonlinear expression ability, and optimizes the calculation efficiency at the same time; in combination with the local feature extraction capability of the CNN and the global feature capture capability of the ViT, the method realizes good balance in detail expression and global relationship modeling.

Description

Technical Field

[0001] The present invention relates to the field of computer vision and deep learning technology, and in particular to a cervical tissue pathology image classification method based on a hybrid CNN-ViT architecture. Background Art

[0002] Cervical cancer is an important disease that affects women's health worldwide. Its early diagnosis and accurate classification are crucial for the treatment and recovery of patients. Traditional cervical pathology image classification methods rely on manual annotation by experts. This method is not only time-consuming and labor-intensive, but also due to the complexity of pathology images and the differences between samples, the classification results are easily affected by subjective factors and have low accuracy. Therefore, the development of automated and intelligent cervical pathology image classification methods has become a hot topic in current research.

[0003] In recent years, deep learning technology has made significant progress in the field of medical image processing. Convolutional neural network (CNN) can effectively extract local features and is widely used in medical image classification.

[0004] However, due to the local receptive field limitation of CNN, it is not good at processing global relationships and capturing complex feature interactions. To solve this problem, researchers proposed a method based on visual transformer (ViT), which can capture the global features of the image and show good classification performance. However, ViT has high computational complexity and large resource requirements, and its feature extraction is not effective enough when processing complex medical images (such as cervical pathology images).

[0005] Some existing studies have attempted to combine CNN with ViT to balance the capabilities of local feature extraction and global relationship modeling.

[0006] For example, the literature Burmeister CA, Khan SF, G,et al.Cervical cancer therapies:Current challenges and future perspectives[J].Tumour Virus Research,2022,13:200238. proposed a multimodal deep learning framework that combined CNN with the attention mechanism for the classification task of pathological images and achieved certain results.

[0007] However, the current method still has the following shortcomings:

[0008] (1) The feature extraction stage fails to fully capture the dynamic relationship between features, resulting in a decrease in classification accuracy;

[0009] (2) In complex pathological images (such as cervical tissue pathological images), the feature similarity between categories is high, and the existing methods are not able to distinguish subtle differences;

[0010] (3) Due to the influence of data distribution deviation, the performance consistency of the model on different data sets is poor. Therefore, a new solution to the above problem is needed. Summary of the invention

[0011] The purpose of the present invention is to provide a cervical tissue pathology image classification method based on a hybrid CNN-ViT architecture, aiming to improve the accuracy and efficiency of complex pathology image classification tasks and solve the technical problems raised in the background technology.

[0012] To achieve the above object, the present invention provides the following technical solution: a cervical tissue pathology image classification method based on a hybrid CNN-ViT architecture, comprising at least the following steps:

[0013] S1: Inputting the cervical tissue pathology image into a convolutional neural network module, first extracting local detail features of the image, the convolutional neural network module is a CNN module;

[0014] S2: Input the features proposed in S1 into the visual transformer module for further processing, the visual transformer module is the ViT module, the ViT module captures the dynamic relationship between global features through the linear angular cross-covariance self-attention module, the linear angular cross-covariance self-attention module is the LACCSA module;

[0015] S3: The LACCSA module combines the cross-covariance matrix and the linear angle similarity function to effectively extract the common features of images of the same category and distinguish the subtle differences between categories;

[0016] S4: A new activation function ExpReLU is used in the ViT module to enhance the nonlinear expression ability of the model and reduce the impact of data distribution offset on model performance, thereby maintaining robustness and accuracy on image data from different sources;

[0017] S5: Experiments are conducted on the LDCH and CAISHI datasets to verify the high efficiency of this method in multi-category and binary classification tasks.

[0018] Furthermore, the design of the LACCSA module includes at least the following steps:

[0019] S200: By calculating the cross covariance matrix C = K' TQ', where K' and Q' are the key and query matrices after L2 normalization, respectively, and T represents the transpose operation of the matrix. The L2 normalization operation ensures the consistency of feature dimensions and avoids the imbalance problem caused by feature scale differences;

[0020] S201: Calculate the attention weight matrix based on the cross covariance matrix Among them, τ is the temperature coefficient, which is used to adjust the smoothness of the Softmax function and ensure the stability of the weight distribution;

[0021] S202: Generate the final output feature using the attention weight matrix: O = V·A XC The output feature O realizes dynamic capture of global features through weighted summation, which significantly improves the model's ability to understand complex pathological images. V is the value matrix.

[0022] Furthermore, the LACCSA module in S3 is combined with a linear angle similarity function to capture the angle relationship between global features;

[0023] The step of combining the linear angle similarity function with the LACCSA module comprises at least the following steps:

[0024] The angle is defined as:

[0025]

[0026] Among them, ||.|| represents the Euclidean distance, <,> represents the inner product, and x i and x j Represents the i-th and j-th vectors in the query and key vectors. The output range of θ is [0,π]. Such angles can be used as a way to measure similarity.

[0027] In this design, the angle kernel is defined as the similarity measurement function between the query Q and the key value K:

[0028]

[0029] Sim(Q i ,K j )The output range is [0,1], when Q i and K j tend to be consistent, θ approaches 0, and the similarity approaches 1; on the contrary, if Q i and K j With opposing characteristics, θ approaches π, and the similarity approaches 0;

[0030] One property of the angular kernel is that replacing the similarity measure with the angular kernel provides an efficient way to implicitly map the input data into a high-dimensional feature space;

[0031] In the high-dimensional feature space, the distance / angle is calculated based on the rich feature structure, which is set as the implicit mapping caused by this kernel. The norm of the mapped input data is:

[0032]

[0033] That is, all data in the input space are mapped to a sphere with a radius of 1 in the feature space, and the distance between two input features is:

[0034]

[0035] The Euclidean distance is positively correlated with the spectral angle, and the distance range is [0,2], taking into account the square complexity of the input token length;

[0036] To design linear attention, recall the trigonometric identities and the infinite series expansion of the arccos function, and reconstruct the similarity function as:

[0037]

[0038] is equivalent to:

[0039]

[0040] in: represents the normalized linear kernel function (Q i ,K j ) / (||Q i ||,||K j ||), which is equivalent to if Q i and K j The inner product when is a unit vector;

[0041] The first two linear angle terms are directly used as similarity measures in linear attention, while the infinite series of high-order terms introduces higher complexity. Therefore, a parameterized deep neural network (DNN) module is used to approximate the high-order residual terms.

[0042] Then the linear angular attention module can be expressed as:

[0043]

[0044] in, is a linear term with complexity O(N), M DW V is the matrix form of DWConv, so the overall complexity of linear angular attention is linear in the input token length.

[0045] Furthermore, the expression of the activation function ExpReLU is:

[0046] ExpReLU(x)=s·(ReLU(x)) 2 +e x +b

[0047] Where: s∈R and b∈R are scalars of scale and bias, respectively, shared by all channels, and can be set to a constant or learnable to obtain different ExpReLU variants, which have much less computational cost than GELU but perform better in performance;

[0048] ReLU is an activation function that is widely used in neural networks. It is defined as ReLU(x)=max(0,x);

[0049] e is a natural constant, approximately equal to 2.71828.

[0050] Compared with the prior art, the present invention has the following beneficial effects:

[0051] Firstly, based on the combination of CNN and ViT architectures, the present invention proposes a linear angle cross-covariance self-attention module (LACCSA) and a new activation function ExpReLU to address the shortcomings of existing methods in feature extraction, classification performance and model adaptability, as well as the problem of distribution offset in data sets. The LACCSA module combines cross-covariance attention and linear angle similarity to effectively capture the dynamic relationship between global features, distinguish subtle differences between categories, and extract common features within a class, significantly improving the accuracy of complex pathological image classification. The ExpReLU activation function reduces the impact of data distribution offset on model performance by enhancing nonlinear expression capabilities, while optimizing computational efficiency. Combining the local feature extraction capability of CNN and the global feature capture capability of ViT, the present invention achieves a good balance between detail expression and global relationship modeling. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for describing the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0053] Figure 1 This is a general network model diagram of a new cervical tissue pathology image classification method disclosed in the present invention;

[0054] Figure 2 It is the LACCSA diagram of the present invention;

[0055] Figure 3 This is a schematic diagram of the visualization results;

[0056] Figure 4 It is a flow chart. DETAILED DESCRIPTION

[0057] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments.

[0058] See also Figure 4 , a cervical tissue pathology image classification method based on a hybrid CNN-ViT architecture, comprising at least the following steps:

[0059] S1: Input the cervical tissue pathology image into the convolutional neural network module, first extract the local detail features of the image, the convolutional neural network module is the CNN module;

[0060] S2: The features proposed in S1 are input into the visual transformer module for further processing. The visual transformer module is the ViT module. The ViT module captures the dynamic relationship between global features through the linear angular cross-covariance self-attention module, the linear angular cross-covariance self-attention module is the LACCSA module.

[0061] S3: The LACCSA module combines the cross-covariance matrix and the linear angle similarity function to effectively extract the common features of images of the same category and distinguish the subtle differences between categories;

[0062] S4: A new activation function ExpReLU is used in the ViT module to enhance the nonlinear expression ability of the model and reduce the impact of data distribution offset on model performance, thereby maintaining robustness and accuracy on image data from different sources;

[0063] S5: Experiments are conducted on the LDCH and CAISHI datasets to verify the high efficiency of this method in multi-category and binary classification tasks.

[0064] Combining the advantages of CNN and ViT, and introducing the linear angle cross covariance self-attention module (LACCSA) and the new activation function ExpReLU. Through the LACCSA module, the model can capture the dynamic relationship between global features and effectively distinguish the subtle differences between categories; through the ExpReLU activation function, the model has higher robustness to data distribution shift, thereby improving classification performance and adaptability.

[0065] The design of the LACCSA module includes at least the following steps:

[0066] S200: By calculating the cross covariance matrix C = K' TQ', where K' and Q' are the key and query matrices after L2 normalization, respectively, and T represents the transpose operation of the matrix. The L2 normalization operation ensures the consistency of feature dimensions and avoids the imbalance problem caused by feature scale differences;

[0067] S201: Calculate the attention weight matrix based on the cross covariance matrix Among them, τ is the temperature coefficient, which is used to adjust the smoothness of the Softmax function and ensure the stability of the weight distribution;

[0068] S202: Generate the final output feature using the attention weight matrix: O = V·A XC The output feature O realizes dynamic capture of global features through weighted summation, which significantly improves the model's ability to understand complex pathological images. V is the value matrix.

[0069] The LACCSA module in S3 is combined with the linear angle similarity function to capture the angle relationship between global features;

[0070] The steps of the LACCSA module combined with the linear angle similarity function include at least the following steps:

[0071] The angle is defined as:

[0072]

[0073] Among them, ||.|| represents the Euclidean distance, <,> represents the inner product, and x i and x j Represents the i-th and j-th vectors in the query and key vectors. The output range of θ is [0,π]. Such angles can be used as a way to measure similarity.

[0074] In this design, the angle kernel is defined as the similarity measurement function between the query Q and the key value K:

[0075]

[0076] Sim(Q i ,K j )The output range is [0,1], when Q i and K j tend to be consistent, θ approaches 0, and the similarity approaches 1; on the contrary, if Q i and K j With opposing characteristics, θ approaches π, and the similarity approaches 0;

[0077] One property of the angular kernel is that replacing the similarity measure with the angular kernel provides an efficient way to implicitly map the input data into a high-dimensional feature space;

[0078] In the high-dimensional feature space, the distance / angle is calculated based on the rich feature structure, which is set as the implicit mapping caused by this kernel. The norm of the mapped input data is:

[0079]

[0080] That is, all data in the input space are mapped to a sphere with a radius of 1 in the feature space, and the distance between two input features is:

[0081]

[0082] The Euclidean distance is positively correlated with the spectral angle, and the distance range is [0,2], taking into account the square complexity of the input token length;

[0083] To design linear attention, recall the trigonometric identities and the infinite series expansion of the arccos function, and reconstruct the similarity function as:

[0084]

[0085] is equivalent to:

[0086]

[0087] in: represents the normalized linear kernel function (Q i ,K j ) / (||Q i ||,||K j ||), which is equivalent to if Q i and K j The inner product when is a unit vector;

[0088] The first two linear angle terms are directly used as similarity measures in linear attention, while the infinite series of high-order terms introduces higher complexity. Therefore, a parameterized deep neural network (DNN) module is used to approximate the high-order residual terms.

[0089] Then the linear angular attention module can be expressed as:

[0090]

[0091] in, is a linear term with complexity O(N), M DW V is the matrix form of DWConv, so the overall complexity of linear angular attention is linear in the input token length.

[0092] The expression of the activation function ExpReLU is:

[0093] ExpReLU(x)=s·(ReLU(x)) 2 +e x +b

[0094] Where: s∈R and b∈R are scalars of scale and bias, respectively, shared by all channels, and can be set to a constant or learnable to obtain different ExpReLU variants, which have much less computational cost than GELU but perform better in performance;

[0095] ReLU is an activation function that is widely used in neural networks. It is defined as ReLU(x)=max(0,x);

[0096] e is a natural constant, approximately equal to 2.71828.

[0097] See also Figure 1 , a schematic diagram of the overall network architecture of the new cervical tissue pathology image classification method described in the present invention. In the encoding stage, a linear angle cross-covariance self-attention module (LACCSA) was constructed to capture the dynamic relationship between global features by combining cross-covariance attention and linear angle similarity. In the activation function design, the ExpReLU activation function was introduced to enhance the nonlinear expression ability of the model and reduce the impact of data distribution offset on model performance. In addition, by combining the hybrid architecture of CNN and ViT, the local feature extraction and global semantic modeling are effectively balanced, which improves the performance and efficiency of the classification task. The specific implementation steps are as follows:

[0098] The technical route is briefly described as follows:

[0099] The present invention is based on a hybrid architecture of CNN and ViT, and adopts an overall design process of convolution -> transformer encoder -> classifier. Three consecutive convolution modules (including convolution and pooling operations) are used in the encoding stage to extract local features; these features are sent to the ViT module to capture global dynamic relationships through LACCSA. In the ViT module, the linear angular similarity function and the cross-covariance matrix are combined to enhance the intra-class features and distinguish the differences between classes. Finally, the robustness of the feature expression is enhanced by the ExpReLU activation function to generate the final classification result.

[0100] See also Figure 2 , the LACCSA module design is briefly described as follows:

[0101] The LACCSA module is an important part of the present invention, which is used to capture the interaction and dynamic correlation between different features in the image. The module consists of two branches: the covariance attention branch and the linear angle attention branch. The covariance attention branch provides a method with high computational efficiency and focuses on common features within the class by modeling the interaction between feature channels instead of operating between markers. This branch can effectively extract features that are more important to the classification task while significantly reducing the computational complexity. The linear angle attention branch is based on the angular similarity between feature vectors and evaluates the relationship between features through the angle kernel function, so as to accurately distinguish the subtle differences between categories, which is particularly suitable for distinguishing cervical lesion categories that are visually very close. By combining these two attention branches, the LACCSA module can capture the dynamic interaction of features globally, while distinguishing subtle changes in detail features through angular relationships. This design is particularly important for handling complex cervical tissue pathology image classification tasks.

[0102] The linear angle cross covariance self-attention module (LACCSA) proposed in the present invention combines the covariance attention branch and the linear angle attention branch, aiming to capture the global statistical relationship and angle similarity between features at the same time. The input features are first generated by linear projection to generate query (Q), key (K) and value (V), and then enter the two branches for processing respectively. In the covariance attention branch, the cross covariance matrix of the query and the key is calculated by matrix multiplication, and temperature adjustment and Softmax normalization are performed on it to generate the attention weight matrix. The weight is used to perform weighted summation on the value (V) to obtain the branch output feature. In the linear angle attention branch, the similarity of the features is calculated based on the angle relationship between the query and the key, combined with the angle kernel function. Subsequently, multi-scale features are generated using deep convolution (DWConv), and the expressiveness of the features is enhanced by global pooling operations. The outputs of the two branches are fused by element-by-element addition, and the results are normalized by the Sigmoid activation function to generate the final feature output. Through this dual-branch design, the LACCSA module effectively captures global dynamic relationships and angle information, and is particularly suitable for distinguishing intra-class detail differences and inter-class similarities in complex pathological images.

[0103] The ExpReLU activation function is briefly described as follows:

[0104] ExpReLU is a new type of activation function that aims to address the shortcomings of existing activation functions when dealing with complex data distributions. The traditional ReLU activation function is simple and efficient, but its linear nature limits the expressiveness of the model. Although GELU has superior performance, it is accompanied by a high computational cost. On this basis, SquaredReLU enhances nonlinearity through square operations, but its performance in complex tasks is still insufficient. To this end, ExpReLU is further optimized on the basis of SquaredReLU, and the performance degradation caused by data distribution offset is reduced through normalization processing, while adding the exponential term e x , significantly enhancing the nonlinear expression ability. Unlike other activation functions, ExpReLU introduces learnable scale and bias parameters, which enables it to be adaptively adjusted according to specific tasks and data, thereby improving the robustness and adaptability of the model. In the task of cervical pathological tissue image classification, ExpReLU not only reduces the computational cost, but also surpasses traditional activation functions in performance, especially showing higher stability and superiority when dealing with different data distributions.

[0105] Figure 3 Some representative visualization results are listed in the figure. By comparing the heat maps of CINI, CINII and CINIII, the changes in the feature attention area of ​​the model before and after the application of the LACCSA module are shown (3(a) is the original cervical pathological tissue image, 3(b) is the feature importance area when LACCSA is not used, and 3(c) is the feature importance area after LACCSA is used). The visualization results show that although there are similarities between CINII and CINIII, the LACCSA module guides the model to focus more on the subtle differences between the two, which helps the accuracy of classification decisions.

[0106] In summary:

[0107] Aiming at the limitations of existing deep learning models in the classification of cervical tissue pathology images, the present invention proposes a new hybrid model architecture CervicalFormer. In the encoding process, the method effectively solves the problem of insufficient feature representation ability through module design. In the encoding stage, the present invention designs a linear angle cross covariance self-attention module (LACCSA), which combines the cross covariance attention mechanism and the linear angle attention mechanism to effectively capture the dynamic relationship between global features while maintaining computational efficiency. This module is particularly suitable for extracting global features from complex pathological images and can significantly improve the ability to distinguish between categories. In addition, the present invention proposes a new activation function ExpReLU, which further reduces the impact of data distribution offset on model performance by enhancing nonlinear expression ability, so that the model maintains high robustness on pathological image data from different sources. Experimental results show that the classification performance of this method on datasets such as LDCH and CAISHI is significantly better than that of existing models, while the number of parameters and computational cost are greatly reduced, showing its potential application value in the field of cervical tissue pathology image classification.

[0108] It will be apparent to those skilled in the art that the invention is not limited to the details of the exemplary embodiments described above and that the invention can be implemented in other specific forms without departing from the spirit or essential features of the invention. Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description, and it is intended that all variations falling within the meaning and scope of the equivalent elements of the claims be included in the invention. Any reference numeral in a claim should not be considered as limiting the claim to which it relates.

Claims

1. A cervical tissue pathology image classification method based on a hybrid CNN-ViT architecture, characterized by: At least the following steps are included: S1: Inputting the cervical tissue pathology image into a convolutional neural network module, first extracting local detail features of the image, the convolutional neural network module is a CNN module; S2: Input the features proposed in S1 into the visual transformer module for further processing, the visual transformer module is the ViT module, the ViT module captures the dynamic relationship between global features through the linear angular cross-covariance self-attention module, the linear angular cross-covariance self-attention module is the LACCSA module; S3: The LACCSA module combines the cross-covariance matrix and the linear angle similarity function to effectively extract the common features of images of the same category and distinguish the subtle differences between categories; S4: A new activation function ExpReLU is used in the ViT module to enhance the nonlinear expression ability of the model and reduce the impact of data distribution offset on model performance, thereby maintaining robustness and accuracy on image data from different sources; S5: Experiments are conducted on the LDCH and CAISHI datasets to verify the high efficiency of this method in multi-category and binary classification tasks.

2. According to claim 1, a cervical tissue pathology image classification method based on a hybrid CNN-ViT architecture is characterized by: The design of the LACCSA module includes at least the following steps: S200: By calculating the cross covariance matrix C = K' T Q', where K' and Q' are the key and query matrices after L2 normalization, respectively, and T represents the transpose operation of the matrix. The L2 normalization operation ensures the consistency of feature dimensions and avoids the imbalance problem caused by feature scale differences; S201: Calculate the attention weight matrix based on the cross covariance matrix Among them, τ is the temperature coefficient, which is used to adjust the smoothness of the Softmax function and ensure the stability of the weight distribution; S202: Generate the final output feature using the attention weight matrix: O = V·A XC , the output feature O realizes the dynamic capture of global features through weighted summation, which significantly improves the model's ability to understand complex pathological images. V is the value matrix.

3. The method for classifying cervical tissue pathology images based on a hybrid CNN-ViT architecture according to claim 2, characterized in that: The LACCSA module in S3 is combined with a linear angle similarity function to capture the angle relationship between global features; The step of combining the linear angle similarity function with the LACCSA module comprises at least the following steps: The angle is defined as: Among them, ||.|| represents the Euclidean distance, <,> represents the inner product, and x i and x j Represents the i-th and j-th vectors in the query and key vectors. The output range of θ is [0,π]. Such angles can be used as a way to measure similarity. In this design, the angle kernel is defined as the similarity measurement function between the query Q and the key value K: Sim(Q i ,K j )The output range is [0,1], when Q i and K j tend to be consistent, θ approaches 0, and the similarity approaches 1; on the contrary, if Q i and K j With opposing characteristics, θ approaches π, and the similarity approaches 0; One property of the angular kernel is that replacing the similarity measure with the angular kernel provides an efficient way to implicitly map the input data into a high-dimensional feature space; In the high-dimensional feature space, the distance / angle is calculated based on the rich feature structure, which is set as the implicit mapping caused by this kernel. The norm of the mapped input data is: That is, all data in the input space are mapped to a sphere with a radius of 1 in the feature space, and the distance between two input features is: The Euclidean distance is positively correlated with the spectral angle, and the distance range is [0,2], taking into account the square complexity of the input token length; To design linear attention, recall the trigonometric identities and the infinite series expansion of the arccos function, and reconstruct the similarity function as: is equivalent to: in: represents the normalized linear kernel function (Q i ,K j ) / (||Q i ||,||K j ||), which is equivalent to if Q i and K j The inner product when is a unit vector; The first two linear angle terms are directly used as similarity measures in linear attention, while the infinite series of high-order terms introduces higher complexity. Therefore, a parameterized deep neural network (DNN) module is used to approximate the high-order residual terms. Then the linear angular attention module can be expressed as: in, is a linear term with complexity O(N), M DW V is the matrix form of DWConv, so the overall complexity of linear angular attention is linear in the input token length.

4. The method for classifying cervical tissue pathology images based on a hybrid CNN-ViT architecture according to claim 1, characterized in that: The expression of the activation function ExpReLU is: ExpReLU(x)=s·(ReLU(x)) 2 +e x +b Where: s∈R and b∈R are scalars of scale and bias, respectively, shared by all channels, and can be set to a constant or learnable to obtain different ExpReLU variants, which have much less computational cost than GELU but perform better in performance; ReLU is an activation function; e is a natural constant.

Images