Design and Training Method of Convolutional Vision Transformer for Automatic Detection of Cervical Lesion Cells

Abstract

The present invention discloses a design and training method for a convolutional vision transformer for automatic detection of cervical lesion cells. First, the present invention constructs a convolutional vision transformer model: the model includes a backbone network and a classification mapping network; in the backbone network, multiple convolutional embedding modules and convolutional attention modules are alternately cascaded; the classification mapping network is connected to the output end of the backbone network; secondly, convolutional embedding operation is performed: the feature map is input into the convolutional layer, and local features are extracted through convolutional operation; then, convolutional attention operation is performed: queries, keys, and values are calculated; the obtained queries, keys, and values are respectively flattened into sequences according to the pixel arrangement order; the multi-head self-attention mechanism is used to integrate global information of the queries, keys, and values. The present invention combines the inductive bias advantages of convolutional neural networks and the long-range dependence learning ability of vision transformers, and integrates the advantages of both to improve the accuracy, efficiency, and interpretability of cervical cell image recognition.

Description

Technical Field

[0001] The present invention relates to the field of biomedical cell image processing, and particularly to a method for designing and training a convolutional vision transformer neural network model for realizing automatic detection of cervical lesion cells. Background Art

[0002] Cervical cancer is one of the malignant tumors with relatively high incidence and mortality rates among women globally. Early screening and diagnosis are of great significance for improving the survival rate of patients. ThinPrep cytology test (TCT) is a widely used cervical cancer screening method in clinical practice at present. By analyzing microscopic images of cervical cells, abnormal cells can be detected and auxiliary diagnosis can be made. However, the traditional TCT image diagnosis relies on the experience and subjective judgment of pathologists, and has problems such as low diagnosis efficiency, high labor intensity, and poor consistency.

[0003] In recent years, with the development of machine learning, especially deep learning (i.e., deep neural network) technology, data-driven solutions have shown excellent performance in image classification and feature extraction. The design of the convolutional neural network (CNN) architecture effectively utilizes visual prior knowledge / inductive bias, such as the high correlation between adjacent pixels within the local region of an image, etc. These prior knowledge enable CNN to have excellent local feature extraction capabilities and can effectively capture fine-grained details, which is crucial for accurately representing and analyzing visual information. In addition, the hierarchical structure of the convolutional kernels in CNN is also one of the key factors. These convolutional kernels are designed to learn visual patterns at different levels of complexity, starting from simple low-level edges and textures and gradually building higher-level semantic patterns. This hierarchical learning process enables CNN to understand visual data in a more comprehensive and systematic way, highlighting its uniqueness. However, it is difficult for CNN to capture long-range dependencies. Therefore, the usual approach is to stack convolutional layers to expand the receptive field, that is, the range of information distribution obtained, so as to include pixels that are far apart into an overall consideration. However, this multi-level convolution process will bring disadvantages such as low computational efficiency, difficult optimization and modeling, and inconvenient information transmission.

[0004] The emergence of the transformer architecture enables AI systems to fully utilize the attention mechanism to efficiently learn long-range dependencies. AI models based on the transformer architecture conform well to the scaling law, that is, by continuously increasing the data volume, model parameters, and computing resources, the performance of the model will be better. Thus, the transformer has become the mainstream framework for current large AI models. Although the transformer architecture was initially used for text data, due to its powerful performance, the Vision Transformer (ViT) applicable to images has also been proposed and its performance in various computer vision tasks is increasingly recognized. However, when dealing with small-scale datasets, there is still a certain gap between the vision transformer architecture and the convolutional neural network (CNN) architecture. An important reason for this gap is that CNN contains effective prior knowledge / inductive biases and the hierarchical structure of convolutional kernels. Relatively speaking, there is still room for improvement in existing vision transformer models and methods in small-scale data application scenarios, and there are some obvious deficiencies.

[0005] 1. Require a large amount of data and have slow training convergence

[0006] Although the Vision Transformer (ViT) has achieved excellent results in large-scale image recognition benchmark tests such as ImageNet, it performs poorly when trained from scratch on small-scale datasets. This is because it lacks prior knowledge like the convolutional neural network (CNN). Due to having this prior knowledge, CNN can quickly learn some basic features on small datasets, thus accelerating the training convergence speed. Without the guidance of this prior knowledge, ViT needs to learn various features and patterns from scratch on small datasets, resulting in a long training process, slow convergence speed, and being prone to overfitting. In the actual cervical cell image recognition task, it is often difficult and costly to obtain a large amount of labeled data, which makes the training efficiency and accuracy a serious problem when ViT processes tasks with limited data volume such as cervical cell images, greatly limiting its rapid application and promotion in clinical practice.

[0007] 2. Do not utilize prior knowledge and have low learning efficiency

[0008] A series of useful prior knowledge is built into the design of CNNs, such as translation invariance, local receptive field (the concept that two nearby pixels in an image are more relevant than two distant pixels), shared filters across images, and feature hierarchy (i.e., high-level larger patterns are composed of many low-level smaller patterns). These inductive biases enable CNNs to learn image features quickly and efficiently. In contrast, ViT processes the input image by splitting it into image patches, losing the important inductive bias information in CNNs, resulting in the model not knowing which pixels are close and which are far apart, and having to learn this information solely from the sparse signals of the classification task. When processing cervical cell images, there are inherent spatial relationships and correlations among the features such as the morphology and position of the cells, and these relationships are crucial for accurately identifying abnormal cells. Due to ViT's lack of utilization of these prior knowledge, its learning efficiency is low, making it difficult to fully extract the effective features in cervical cell images, thereby affecting the accuracy and reliability of diagnosis.

[0009] 3. Other deficiencies

[0010] Limited ability to process spatial information: Although Transformers can capture long-range dependencies through the self-attention mechanism, they are not as efficient as CNNs in processing local spatial information. In cervical cell images, local texture, shape and other detailed features of the cells are very crucial for judging whether the cells are abnormal. CNNs can well focus on local regions and extract rich local features through the sliding operation of the convolutional kernel on the image. While processing these local spatial information, ViT, due to its global attention mechanism characteristics, may disperse the attention to local details, resulting in inaccurate extraction of local features of the cells, thus affecting the final diagnosis result.

[0011] Poor model interpretability: The interpretability of deep learning models is particularly important in the medical field. Doctors need to understand the decision-making basis of the model in order to better combine clinical experience for diagnosis. Due to the locality and hierarchical feature representation of its convolutional operations, CNNs can relatively easily explain the features and decision-making processes learned by the model using methods such as Grad-CAM. However, the self-attention mechanism of ViT is a global and complex computational method, making the decision-making process of the model difficult to understand and interpret. In the diagnosis of cervical cell images, it is difficult for doctors to know which features the model is based on to make judgments, which increases the uncertainty and risk of clinical applications and limits the wide application of ViT in this field.

[0012] High computational resource requirements: ViT usually requires a large amount of computational resources and memory for training and inference, especially when dealing with high-resolution images. This poses a huge challenge for some medical institutions with limited resources. In an actual clinical environment, it may not be possible to equip high-performance computing devices to support the operation of the ViT model, thus limiting its application in actual diagnostic work. Summary of the Invention

[0013] The present invention aims to overcome the deficiencies of existing vision transformer models in cervical cell image recognition, and proposes a design and training method for a convolutional vision transformer neural network model for automatic detection of cervical lesion cells, which is applicable to efficiently processing and analyzing liquid-based cervical cell images and completing the tasks of automatic detection and classification of lesion cells.

[0014] The present invention combines the inductive bias advantages of convolutional neural networks (CNNs) and the long-range dependence learning ability of vision transformers (ViTs), integrates the advantages of both to improve the accuracy, efficiency, and interpretability of cervical cell image recognition, and at the same time reduces the dependence on large-scale data and powerful computing resources.

[0015] The present invention includes the following steps:

[0016] Construct a convolutional vision transformer model:

[0017] Provide a backbone network and a classification mapping network;

[0018] In the backbone network, alternately cascade multiple convolutional embedding modules and convolutional attention modules;

[0019] Connect a classification mapping network to the output end of the backbone network for mapping the feature vector to a classification result;

[0020] Perform convolutional embedding operations:

[0021] Input the cervical cell image or the feature map of the previous layer into the convolutional layer, extract local features through convolutional operations, and output the feature map;

[0022] Process the vector at each pixel position into a token;

[0023] Perform convolutional attention operations:

[0024] Normalize the feature map;

[0025] Input the normalized feature map into multiple convolutional layers to calculate queries, keys, and values;

[0026] Flatten the obtained queries, keys, and values into sequences respectively according to the pixel arrangement order;

[0027] Utilize the multi - head self - attention mechanism to integrate global information for queries, keys, and values;

[0028] Add the output of the multi - head self - attention mechanism to the input feature map through residual connection to obtain the output feature map;

[0029] After normalizing the output feature map, input it into a multi - layer perceptron for linear projection.

[0030] Perform the classification mapping operation:

[0031] Input the feature vector output by the backbone network into the classification mapping network;

[0032] Map the feature vector to the classification result through at least one fully - connected layer;

[0033] Train the convolutional vision transformer model:

[0034] Use the labeled cervical cell image dataset to perform supervised training on the convolutional vision transformer model;

[0035] During the training process, calculate the cross - entropy loss between the model prediction result and the true label, as well as the gradients of the model parameters;

[0036] Use the optimizer to update the model parameters according to the calculated gradients.

[0037] The beneficial effects of the present invention:

[0038] Innovation in model structure design - integrating the advantages of convolutional neural networks and vision transformers: Combine the inductive bias advantages of convolutional neural networks (CNNs) (such as translational invariance, local receptive fields, weight sharing, etc.) with the long - range dependence learning ability of vision transformers (ViTs) to improve the accuracy, efficiency, and interpretability of cervical cell image recognition, while reducing the dependence on large - scale data and powerful computing resources. The backbone network is composed of multiple convolutional embeddings and convolutional attention cascaded alternately. For classification tasks, a single - layer MLP is connected behind the backbone network as the classification mapping network. For regression tasks, a regression mapping network can be connected. The present invention is mainly applied to cervical cell classification and uses the classification mapping network.

[0039] Innovation in convolutional embedding operation: In the convolutional embedding stage, input the image or feature map into the depth - separable convolutional layer, output the feature map, and treat the vector at each pixel position as a token. Note that different from the standard vision transformer, the convolutional kernel size and stride value are not fixed here, that is, multi - scale convolutional kernels with different receptive field sizes and different stride values can be used in different stages to adapt to input images or feature maps of different resolutions, capture the appropriate range of neighborhood information, and enhance the ability to capture local details.

[0040] Innovation of Convolutional Attention Mechanism: After layer normalization of the feature map, it is input into three parallel and independent convolutional layers to calculate Query, Key, and Value. The Q - K - V convolutional projection is different from the way of using a linear layer in the standard vision transformer. This convolutional projection operation can expand the local receptive field, incorporate neighborhood information, and utilize convolutional prior knowledge and inductive bias to quickly extract the morphological features of cervical cell images, providing a good foundation for multi - head self - attention analysis. Then, it is flattened into a sequence in pixel order to perform multi - head self - attention operations to capture complex relationships and overall structural information. Finally, it is added to the input feature map to complete the residual operation, integrating the advantages of convolutional networks and vision transformers.

[0041] Innovation of Model Training Strategy

[0042] Supervised Training Strategy: Directly perform supervised training on the labeled cervical cell image dataset. When the data volume is comparable to that of common convolutional network models, and detailed design and optimization are carried out on aspects such as data preparation, training process, hyperparameter adjustment, and monitoring. For example, dividing the dataset, adopting mini - batch gradient descent, monitoring the loss curve, etc., to ensure that the model learns efficiently and accurately.

[0043] Data Augmentation Techniques: During the training process, various data augmentation techniques are adopted, such as random rotation, flipping, scaling, cropping, color and brightness jittering, adding Gaussian blur and noise, etc., to expand the scale of the training dataset and improve the robustness and generalization ability of the model.

[0044] Adaptive Learning Rate Adjustment: The adaptive learning rate algorithms AdamW and cosine annealing learning rate scheduler are adopted to automatically adjust the learning rate according to the update of model parameters and the change trend of the loss function. A large learning rate is used at the beginning of training to accelerate convergence, and it is gradually decreased to improve stability and accuracy, avoiding problems such as unstable training or slow convergence.

[0045] The model provided by the method of the present invention shows higher accuracy, sensitivity, specificity, and precision in the recognition and classification tasks of cervical cell images. Especially when the training data volume is small, the advantage is more obvious compared to the standard vision transformer ViT. Brief Description of the Drawings

[0046] Figure 1 is a schematic diagram of the design of a convolutional vision transformer neural network model for automatically detecting cervical lesion cells in this application;

[0047] Figure 2 is a flow chart of supervised training in an embodiment of this application;

[0048] Figure 3 is a schematic diagram of the structure principle of the comparative model, the standard vision transformer ViT model;

[0049] Figure 4 shows the comparison of the test accuracy between the embodiment of the present application and the standard Vision Transformer (ViT) under different numbers of samples.

[0050] Figure 5 is the confusion matrix of the experimental results of the embodiment of the present application in the case of 1000 samples per class.

[0051] Figure 6 is the confusion matrix of the experimental results of the standard Vision Transformer model in the case of 1000 samples per class.

[0052] Figure 7 is the confusion matrix of the experimental results of the embodiment of the present application in the case of 20000 samples per class.

[0053] Figure 8 is the confusion matrix of the experimental results of the standard Vision Transformer model in the case of 20000 samples per class.

[0054] Figure 9 is the result graph of visualizing and analyzing the embodiment of the present application using the Grad-CAM heatmap. Detailed Embodiments

[0055] Exemplary embodiments will be described in detail below with reference to the accompanying drawings, which are shown in the drawings. When the following description refers to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present application. On the contrary, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0056] The terms used in the present application are for the purpose of describing specific embodiments only and are not intended to limit the present application. The singular forms "a", "the", and "said" used in the present application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term "and / or" as used herein refers to and includes any or all possible combinations of one or more of the associated listed items.

[0057] It should be understood that although the terms first, second, third, etc. may be used in the present application to describe various information, such information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "when" or "while" or "in response to determining".

[0058] The embodiment of the present application provides a method for designing and training a convolutional Vision Transformer for automatic detection of cervical lesion cells, including the following steps:

[0059] Construct a convolutional vision transformer model:

[0060] Provide a backbone network and a classification mapping network;

[0061] In the backbone network, alternately cascade multiple convolutional embedding modules and convolutional attention modules;

[0062] Connect the classification mapping network to the output end of the backbone network to map the feature vector to the classification result;

[0063] Perform convolutional embedding operation:

[0064] The convolutional embedding operation inputs an image or a set of feature maps with the shape of [c_in, h_in, w_in] into a channel-wise / depth-wise separable convolution (DepthwiseSeparableConv) layer, and the output is a set of feature maps with the shape of [c, h, w]. In this process, the vector at each pixel position is treated as a token.

[0065] Furthermore, in this process, the stride value is not fixed, which is different from the standard vision transformer. This design enables the model to utilize the local feature extraction ability similar to that of convolutional neural networks, thereby enhancing the model's ability to capture local details.

[0066] Perform convolutional attention operation: including the following two steps:

[0067] 1. Q - K - V convolutional projection and multi-head self-attention: Integrate the advantages of convolutional networks and vision transformers

[0068] First, the feature maps are processed by layer normalization and then input into three parallel and independent convolutional layers to calculate the query (Query), key (Key), and value (Value). This convolutional projection operation is different from the way of using linear layers to calculate Q, K, and V in the standard vision transformer. Because this Q - K - V calculation based on convolutional projection can expand the local receptive field, enabling the model to incorporate useful neighborhood information, and at the same time effectively utilize the prior knowledge and inductive bias contained in convolutional operations, such as translational invariance. Therefore, it can quickly extract the morphological features of cervical cell images, such as local information like the edges and textures of cells, thereby improving the quality of Q, K, and V, and providing a good basis for subsequent multi-head self-attention analysis.

[0069] Next, the obtained Q, K, and V are flattened into sequences respectively according to the pixel arrangement order, and then, like a standard vision transformer, perform inner-product based multi-head self-attention operation. Utilize its powerful long-distance relationship learning ability to globally integrate and analyze the feature map, so as to capture the complex relationships and overall structural information between different structures of cells and between cells.

[0070] Finally, add the output of the attention operation to the input feature map to complete the residual operation.

[0071] Through this fusion design, the advantages of convolutional neural networks in local feature extraction are retained, and the strengths of vision transformers in global information processing are also brought into play.

[0072] 2. Linear Projection

[0073] The output result of multi-head self-attention is input to a multi-layer perceptron for linear projection after layer normalization. This step introduces additional non-linear transformations, which can further enhance the modeling ability.

[0074] Ultimately, the backbone network of the convolutional vision transformer is composed of multiple convolutional embeddings and convolutional attentions cascaded alternately. If it is a classification task, a classification mapping network (i.e., classification head) is connected after the backbone network. If it is a regression task, a regression mapping network (i.e., regression head) is connected after the backbone network. Here, in this embodiment, the main application scenario is TCT cervical cell classification, so a classification mapping network, a single-layer MLP, is adopted.

[0075] Model Training:

[0076] Supervised Training Strategy: The convolutional vision transformer model proposed by the present invention can be directly trained in a supervised manner on a labeled cervical cell image dataset, and the required amount of data is comparable to that of common convolutional network models such as residual networks when the model scales are the same. During the entire supervised training process, a series of strategies are adopted to ensure that the model can efficiently and accurately learn the features and classification information in cervical cell images. Specifically:

[0077] In the data preparation stage, the labeled cervical cell image dataset is divided into a training set, a validation set, and a test set according to the ratio of 80%, 10%, and 10%. The validation set is used to monitor the performance of the model during training to prevent overfitting, and the test set is used to finally evaluate the generalization ability of the model on unseen data.

[0078] During the training process, the Mini-Batch Gradient Descent algorithm is adopted. That is, a small batch of samples (about 128 samples) are randomly selected from the training set each time and input into the convolutional vision transformer model. After a series of operations such as convolutional embedding and convolutional attention, the forward propagation is finally completed to obtain the output result of the model. Then, the loss function is calculated based on the output result of the model and the true label.

[0079] For classification tasks, the Cross-Entropy Loss function is used to calculate the difference between the class probability distribution predicted by the model and the true label. After calculating the loss function, the gradients of the loss function with respect to each parameter of the model are calculated through the Backpropagation algorithm. These gradients indicate the direction and degree of the influence of the model parameters on the loss function in the current state.

[0080] Next, the AdamW optimizer is used to update the parameters of the model according to the calculated gradients. The AdamW optimizer combines the adaptive learning rate feature of the Adam optimizer and the regularization method of Weight Decay, and can automatically adjust the learning rate of each parameter during the training process and prevent the model from overfitting.

[0081] During the training process, the performance of the model is regularly evaluated on the validation set, and indicators such as ValidationLoss and Accuracy are recorded. The changes in the training loss and validation loss are monitored in real time, and the loss curve is plotted to more comprehensively understand the learning progress and performance of the model.

[0082] By observing the trend of the loss curve, problems that may occur during the model training process, such as underfitting, overfitting, vanishing gradients, exploding gradients, etc., can be discovered in a timely manner, and corresponding measures can be taken for adjustment. Finally, the model with the best performance on the validation set is selected as the final model according to indicators such as accuracy.

[0083] Through the supervised training strategy, the convolutional vision transformer model proposed by the present invention can be efficiently trained on the labeled cervical cell image dataset, learn rich feature representations, and thus achieve accurate classification of cervical cell images. When the model scale is the same as that of common convolutional network models such as residual networks, it shows comparable or even better performance.

[0084] Furthermore, to alleviate the problem of limited cervical cell image data volume, various data augmentation techniques are adopted during the training process. For example, operations such as random rotation, flipping, scaling, and cropping are performed on the original images to generate diverse training samples. In addition, methods such as color brightness jittering, adding Gaussian blur and noise are also used to increase data diversity. Through data augmentation, not only can the scale of the training dataset be expanded, but also the robustness and generalization ability of the model can be improved, enabling the model to better handle various complex image situations in actual clinical applications.

[0085] Furthermore, the model training process adopts the adaptive learning rate algorithm AdamW and the cosine annealing learning rate scheduler (CosineAnnealingLR) to automatically adjust the learning rate according to the update situation of the model parameters and the change trend of the loss function, so that the learning rate decreases like a quarter of a cycle of cosine. That is, a larger learning rate is used at the beginning of training to accelerate the convergence speed of the model, and the learning rate is gradually decreased to improve the stability and accuracy of the model. This adaptive learning rate adjustment strategy can effectively improve the training efficiency and performance of the model, and avoid problems such as unstable training or slow convergence caused by improper learning rate setting.

[0086] This application also provides model evaluation and optimization:

[0087] Comprehensive multi-index evaluation: A comprehensive multi-index evaluation system is established to comprehensively evaluate the performance of the model, including indicators such as accuracy, sensitivity, specificity, precision, recall rate, f1 score, and confusion matrix, to evaluate the classification effect of the model on cervical cell images from different perspectives. Through the comprehensive analysis of multiple indicators, the advantages and disadvantages of the model's performance can be understood more comprehensively and accurately, providing a strong basis for model optimization.

[0088] Visual analysis and model improvement: According to the model evaluation results, visualization analysis methods such as the Grad-CAM heatmap for locating key diagnostic regions are used to optimize and improve the model targeted. If it is found that the model has deficiencies in the recognition of certain categories, the recognition ability of the model for this category can be improved by increasing the training data of relevant categories, adjusting the model structure or optimizing the training parameters. At the same time, combined with the clinical experience and feedback of medical experts, the decision-making process of the model is analyzed and explained in depth to further optimize the performance and practicality of the model, making it better meet the needs of clinical diagnosis. Embodiment

[0089] This embodiment provides a method for designing and training a convolutional vision transformer neural network model for automatic detection of cervical lesion cells. Specifically, a cervical cell image to be detected is obtained and input into the convolutional vision transformer neural network model designed in this application to obtain a classification result. Among them, the cervical cell image classification model is designed and trained through the following method, as Figure 1 and Figure 2 shown, and specifically includes the following steps:

[0090] Data collection and preprocessing: Collect a labeled cervical cell image dataset {(x, y)}, and perform standardization processing, such as cleaning, normalization, and enhancement. In a preferred embodiment, the cervical cell image is a single-cell / cluster-level local image obtained by performing image cleaning operations on the whole-slide microscope digital image of a cervical cell smear, removing the slide background noise, and then performing cell segmentation and size adjustment. The size of this image is generally 128×128.

[0091] Construct a convolutional vision transformer model: Create a convolutional vision transformer model M(x) = h(f(x)), where x is the input cervical cell image; f( ) is the backbone network for extracting the feature vector f(x); h( ) is the classification mapping network for mapping the feature vector to the classification result y = h(f(x)). The pre-trained model includes a backbone network and a classification mapping network connected in sequence. Figure 1 is an exemplary structural diagram of a convolutional vision transformer model. In a preferred embodiment, the backbone network consists of multiple stages, and each stage contains several convolutional embedding and convolutional attention operations. The classification mapping network consists of a single-layer perceptron.

[0092] Supervised training: Use the labeled cervical cell image dataset {(x, y)} to perform supervised training on the convolutional vision transformer model, and the training optimization objective is to minimize the cross-entropy loss. Specifically: Input a batch of cervical cell images into the model to obtain classification prediction results . Compare the classification prediction results with the labels to calculate the cross-entropy loss, and then use the backpropagation algorithm to calculate the gradient of the model parameters under the current loss. Use the AdamW optimizer to update the model parameters according to the calculated gradient.

[0093] In a preferred embodiment, send unlabeled cervical cell images to a pathologist for annotation, and classify them into lesion types such as Normal, ASCUS, ASC-H, LSIL, HSIL, AGC, GIL, etc., to construct a labeled cervical cell image dataset {(image, label)}, that is, {(x, y)}.

[0094] In a preferred embodiment, the backbone network f( ) of the convolutional vision transformer includes 7 cascaded [Convolutional Embedding-Convolutional Attention] blocks. The convolutional kernel size of the first block is 7x7, and the convolutional kernel size of each subsequent block is 3x3. The convolutional embedding dimension of the first block, that is, the number of channels of the output feature map, is 64; the convolutional embedding dimensions of the second and third blocks are 192; the convolutional embedding dimensions of the 4th to 7th blocks are 384, and the dimension of the output feature vector of the corresponding backbone network is 384.

[0095] The structure of the linear projection (2-layer MLP) in each convolutional attention operation is Linear(embed_dim,hidden_dim)→GELU→Dropout→Linear(hidden_dim, embed_dim)→Dropout, where embed_dim is the convolutional embedding dimension of the current block, hidden_dim is the hidden dimension, which is set to 2 - 4 times the convolutional embedding dimension; the dropout probability of the Dropout layer is set to 0.1.

[0096] In a preferred embodiment, the classification mapping network h( ) of the convolutional vision transformer is a two-layer perceptron MLP, that is, two fully connected layers. The input of this network is the output feature vector f(x) of the backbone network, and the output is the scores h( f(x)) of N classes.

[0097] Its structure is AdaptiveAvgPool2d→Flatten→LayerNorm→Linear(hidden_dim,C), including a fully connected layer with parameters (hidden_dim ×num_classes) = (384×C), where C is the number of classes. Before the fully connected layer are a two-dimensional adaptive average pooling layer AdaptiveAvgPool2d, a flattening operation Flatten, and a layer normalization operation LayerNorm.

[0098] In a preferred embodiment, a labeled dataset is used to perform supervised training on the model, and the optimization objective is to minimize the cross-entropy loss. The mathematical form of the cross-entropy loss function is:

[0099]

[0100] Among them, is the predicted probability of the true label corresponding to the current sample, which is calculated by passing the scores (logits) z output by the model through the softmax function. z[c] represents the score of the c-th class, and z[j] represents the score of the j-th class.

[0101] The training parameters are set as follows: the initial learning rate lr = 3e-4, the weight decay weight_decay = 1e-4, and the maximum number of iterations max_epochs = 200.

[0102] In this embodiment, a test set is used to evaluate the performance of the model, and metrics such as accuracy, sensitivity, specificity, precision, recall, f1-score, and confusion matrix are calculated.

[0103] Index Calculation formula Medical significance Accuracy (TP + TN) / (TP + TN + FP + FN) Overall classification accuracy Sensitivity TP / (TP + FN) Sensitivity, probability of detecting diseased cells Specificity TN / (TN + FP) Specificity, probability of misjudging normal cells Precision p = TP / (TP + FP) Precision rate, prediction accuracy of this category Recall r = TP / (TP + FN) Recall rate, equivalent to sensitivity F1-score p * r / (p + r) Comprehensive classification performance Confusion matrix [[TN, FP] [FN, TP]] Analyze misclassification patterns of each category

[0104] Among them, TP represents True Positive, TN represents True Negative, FN represents False Negative, and FP represents False Positive.

[0105] As shown in Figure 4, when the number of training examples for each category is 1000, the accuracy of this embodiment reaches 0.7303. As the number of examples gradually increases, the performance of the model steadily improves, indicating that the model has good generalization ability and scalability.

[0106] Visual analysis is performed on the prediction results, and the misclassification patterns of each category are analyzed using a confusion matrix, such as Figure 5 、 Figure 6 、 Figure 7 and Figure 8 shown. The Grad-CAM heatmap is used to locate the key diagnostic regions, such as Figure 9 shown. Figure 9 In the first column of

[0107] is the input image, the second column is the saliency map obtained by the Grad-CAM method, and the third column is the saliency map obtained by the Grad-CAM++ method. The model parameters or training strategies are adjusted according to the evaluation results to further optimize the model performance. In the classification and detection task of liquid-based cervical cells, the model and method proposed in this embodiment have obvious advantages, which helps to improve the accuracy of detection and diagnosis.

[0108] As shown in Table 1 and Figures 5 and 6, the test accuracies of this embodiment and the standard Vision Transformer (ViT) model (see Figure 3)Performance evaluation in the case of 1000 samples per class. Through comparison, it can be found that the accuracy rate of 0.7303 in this embodiment is significantly higher than the accuracy rate of 0.6019 of the comparative model. However, more importantly, when considering the precision, recall rate, and F1 score of a single class, this embodiment has significant advantages. For example, taking the positive class as an example, the precision, recall rate (i.e., sensitivity), and F1 score of this embodiment are 0.5338, 0.8773, and 0.6637 respectively. While the precision, recall rate, and F1 score of the comparative model ViT are 0.4077, 0.6895, and 0.5124 respectively. Each index of this embodiment, especially the sensitivity (negatively correlated with the missed diagnosis rate), is much better than the comparative model.

[0109] Table 1. Performance evaluation of this embodiment and the standard vision transformer model in the case of 1000 samples per class

[0110] Model and method Performance evaluation This embodiment (ConViT) Accuracy = 0.7303; '0_neg' as the target category: Precision: 0.9257; Recall: 0.6663; F1-score: 0.7749; '1_pos' as the target category: Precision: 0.5338, Recall: 0.8773; F1-score: 0.6637; Confusion matrix = [[True negative, False positive] [False negative, True positive]] = [[1695 849] [136 972]] Vision Transformer (ViT) Accuracy = 0.6019; '0_neg' as the target category: Precision: 0.8065; Recall: 0.5637; F1-score: 0.6636; '1_pos' as the target category: Precision: 0.4077; Recall: 0.6895; F1-score: 0.5124; Confusion matrix = [[True negative, False positive] [False negative, True positive]] = [[1434 1110] [344 764]]

[0111] As shown in Table 2 and Figures 7 and 8, it is the performance evaluation of this embodiment and the standard vision transformer model in the case of 20,000 samples per class. Through comparison, it can be found that although the performance of the standard vision transformer has been improved due to the increase in the amount of data, the accuracy rate of 0.8445 in this embodiment is still higher than the accuracy rate of 0.757 of the comparative model. In addition, taking the positive class as an example, the precision, recall rate (i.e., sensitivity), and F1 score of this embodiment are 0.7922, 0.9340, and 0.8573 respectively. While the precision, recall rate, and F1 score of the comparative model ViT are 0.6976, 0.9090, and 0.7894 respectively. Each index of this embodiment is still better than the comparative model.

[0112] Table 2. Performance evaluation of this embodiment and the standard vision transformer model in the case of 20,000 samples per class

[0113] Model and method Performance evaluation This embodiment (ConViT) Accuracy: 0.8445; '0_neg' as the target category: Precision: 0.9196; Recall: 0.7550; F1-score: 0.8292; '1_pos' as the target category: Precision: 0.7922; Recall: 0.9340; F1-score: 0.8573; Confusion matrix {'0_neg': 0, '1_pos': 1}: [[755 245] [ 66 934]] Vision Transformer (ViT) Accuracy: 0.7575; with '0_neg' as the target class: Precision: 0.8694; Recall: 0.6060; F1-score: 0.7142; with '1_pos' as the target class: Precision: 0.6976; Recall: 0.9090; F1-score: 0.7894; Confusion Matrix {'0_neg': 0, '1_pos': 1}: [[606 394] [ 91 909]]

[0114] Grad-CAM heatmap visualization analysis helps to locate key diagnostic regions: Koilocyte is a superficial squamous cell formed by the "basal cells" of squamous cells after being damaged by human papillomavirus (HPV), which accelerates the maturation of damaged cells and is an important indicator for cytological diagnosis of low-grade lesions. When diagnosing whether cervical cells are koilocytes in this embodiment, the key judgment region, that is, the cytoplasmic hollowing region, can be highlighted, giving users an intuitive explanation to help understand the behavior of the model.

[0115] In summary, the present invention combines the inductive bias advantages of convolutional neural networks (CNNs), such as translational invariance, local receptive fields, weight sharing, etc., with the long-range dependence learning ability of vision transformers (ViTs). This combination effectively improves the classification performance of the model while reducing the need for labeled data. In practical applications, it is often difficult to obtain a large amount of labeled data, and the method of the present invention can maintain good performance with relatively limited data volume, providing a more reliable and efficient solution for complex tasks such as TCT cell classification.

[0116] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it; although reference has been made to the foregoing embodiments

[0117] the present invention has been described in detail, and those of ordinary skill in the art should understand that: it is still possible to modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements on some of the technical features; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for designing and training a convolutional vision transformer for automatic detection of cervical lesion cells, characterized in that, It includes the following steps: Construct a convolutional vision transformer model: Provide a backbone network and a classification mapping network; In the backbone network, alternately cascade multiple convolutional embedding modules and convolutional attention modules; Connect the classification mapping network at the output end of the backbone network to map the feature vector to the classification result; Perform convolutional embedding operation: Input the cervical cell image or the feature map of the previous layer into the convolutional layer, extract local features through convolutional operation, and output the feature map; Process the vector at each pixel position into a token; Perform convolutional attention operation: Normalize the feature map; Input the normalized feature map into multiple convolutional layers to calculate queries, keys, and values; Flatten the obtained queries, keys, and values into sequences respectively according to the pixel arrangement order; Use the multi-head self-attention mechanism to integrate global information for the queries, keys, and values; Add the output of the multi-head self-attention mechanism to the input feature map through residual connection to obtain the output feature map; After normalizing the output feature map, input it into a multi-layer perceptron for linear projection.

2. The convolutional vision transformer design and training method for automatic detection of cervical lesion cells according to claim 1, wherein, In the convolutional embedding operation, the convolutional layer adopts depthwise separable convolution to reduce the computational complexity and the number of parameters.

3. The convolutional vision transformer design and training method for automatic detection of cervical lesion cells according to claim 1 or 2, characterized in that, In the convolutional embedding operation, the convolutional kernel size and the stride value of the convolutional layer are both variable, that is, multi-scale convolutional kernels with different receptive fields and stride values of different sizes are adopted to adapt to input images or feature maps of different resolutions.

4. The convolutional vision transformer design and training method for automatic detection of cervical lesion cells according to claim 1, characterized in that, In the convolutional attention operation: Input the feature map obtained by convolutional embedding into three parallel and independent convolutional layers after layer normalization to calculate queries, keys, and values; Flatten the calculated queries, keys, and values into sequences according to the pixel arrangement order; Perform multi-head self-attention operation based on inner product to capture long-range dependencies in the feature map; Add the output of the self-attention operation to the input feature map through residual connection to enhance the feature representation.

5. The convolutional vision transformer design and training method for automatic detection of cervical lesion cells according to claim 1 or 4, characterized in that In the convolutional attention operation, the convolutional layer adopts channel separable convolution operation; a convolutional kernel with a fixed scale size of 3×3 and a stride of 1 are adopted, and the padding method is set to equal-length padding, and the number of output channels is equal to the number of input channels. These settings ensure that the calculated queries, keys, and values have the same shape as the input.

6. The convolutional vision transformer design and training method for automatic detection of cervical lesion cells according to claim 5, characterized in that, In the convolutional attention operation, the number of heads of the multi-head self-attention mechanism is configurable to adapt to task requirements of different complexities.

7. The convolutional vision transformer design and training method for automatic detection of cervical lesion cells according to claim 5, characterized in that, The convolutional attention operation further includes: After the multi-head self-attention operation, apply layer normalization to stabilize the training process; Input the output after layer normalization into a multi-layer perceptron for further feature transformation.

8. The convolutional vision transformer design and training method for automatic detection of cervical lesion cells according to claim 1, characterized in that, It also includes: Train the convolutional vision transformer model: Perform supervised training on the convolutional vision transformer model using a labeled cervical cell image dataset; During the training process, calculate the cross-entropy loss between the model prediction result and the true label, as well as the gradient of the model parameters; Use an optimizer to update the model parameters according to the calculated gradient; During the training process, adopt data augmentation techniques to randomly rotate, flip, scale, crop, perform color jitter, brightness jitter, Gaussian blur, and add noise to the input images to increase data diversity.

9. The convolutional vision transformer design and training method for automatic detection of cervical lesion cells according to claim 8, characterized in that, It also includes: Adopt an adaptive learning rate adjustment strategy to automatically adjust the learning rate according to the training progress; Use a cosine annealing learning rate scheduler to gradually decrease the learning rate during training to improve the stability and accuracy of the model.

10. The convolutional vision transformer design and training method for automatic detection of cervical lesion cells according to claim 1, characterized in that, The model also includes: After training is completed, use Grad-CAM heatmaps to visualize the key diagnostic regions of the model to help understand the model's decision-making process.

Images