Multi-view fusion cervical cancer precancerous lesion image classification method based on self-supervised learning

Abstract

The invention relates to a multi-view fusion cervical cancer precancerous lesion image classification method based on self-supervised learning, and the method comprises the steps: collecting video screenshot data of a prostate endoscopic resection operation, and carrying out the preprocessing, and obtaining an optimized model weight; a second visual Transform model and a third visual Transform model which are optimized are obtained, and the second visual Transform model and the third visual Transform model which are optimized are obtained; and carrying out weighted fusion on the classification result of the colposcope acetic acid picture of the precancerous lesion and the classification result of the colposcope iodine picture of the precancerous lesion to obtain a final classification result. According to the method, a self-supervised learning method is introduced, and a large amount of label-free medical picture data is utilized, so that the generalization ability and classification accuracy of a ViT model, namely a visual Transform model, in cervical cancer precancerous lesion classification are improved; according to the method, efficient auxiliary diagnosis of CIN grading is achieved, especially in areas with limited medical resources, an efficient and low-cost solution is provided for automatic screening, doctors are helped to improve the diagnosis accuracy, the missed diagnosis rate is reduced, and important technical support is provided for early cervical cancer prevention.

Description

Technical Field

[0001] The present invention relates to the technical field of deep learning and medical image analysis, and in particular to a multi-view fusion cervical precancerous lesion image classification method based on self-supervised learning. Background Art

[0002] Cervical cancer is one of the major threats to women's health worldwide, and its early screening and accurate grading are crucial to reducing the incidence rate. However, traditional deep learning methods based on supervised learning face significant challenges in medical image processing: the acquisition of labeled data requires rich clinical experience, and is time-consuming and labor-intensive, especially in developing countries and areas with limited medical resources. This problem is particularly prominent. In addition, the reliance of supervised learning methods on large-scale, high-quality labeled data makes it difficult to effectively apply them when data is insufficient. This contradiction is particularly evident in the Transformer model.

[0003] The visual Transformer better captures the global and local features in the image through the self-attention mechanism and the multi-head attention mechanism. In the study of cervical precancerous lesion grading, Transformer has gradually been used to replace the traditional CNN architecture and has shown stronger performance. However, due to the high requirements of ViT on the scale and quality of training data, when directly transferred to the grading of cervical precancerous lesions, it will face the problems of overfitting due to insufficient data and the mismatch between the attention mode of the pre-trained model and the specific medical task. Summary of the invention

[0004] In order to solve the problems of data dependence and classification accuracy in the grading of cervical precancerous lesions, the purpose of the present invention is to provide a multi-view fusion cervical precancerous lesion image classification method based on self-supervised learning, which improves the generalization ability and classification accuracy in the grading of cervical precancerous lesions and optimizes the global and local feature extraction capabilities of the model.

[0005] To achieve the above object, the present invention adopts the following technical solution: a multi-view fusion cervical precancerous lesion image classification method based on self-supervised learning, the method comprising the following steps in order:

[0006] (1) Collecting video screenshot data of endoscopic prostate resection surgery and preprocessing it to obtain unlabeled data, inputting the unlabeled data into a self-supervised training framework for self-supervised training, optimizing feature extraction, obtaining optimized features, and optimizing the attention weights of the first visual Transformer model by optimizing the features to obtain optimized model weights;

[0007] (2) Collecting and preprocessing colposcopic images of cervical precancerous lesions of patients, obtaining annotated colposcopic acetic acid images and colposcopic iodine images of cervical precancerous lesions, inputting the optimized model weights, annotated colposcopic acetic acid images and colposcopic iodine images of cervical precancerous lesions into the second visual Transformer model and the third visual Transformer model for fine-tuning training, respectively, and using the optimized model weights as the initial weights for fine-tuning the second visual Transformer model and the third visual Transformer model, respectively, to obtain the optimized second visual Transformer model and the third visual Transformer model;

[0008] (3) The colposcopic images of cervical precancer lesions to be classified are preprocessed, and the preprocessed colposcopic images of cervical precancer lesions are input into the optimized second visual Transformer model and the third visual Transformer model for hierarchical prediction, respectively, to obtain the classification results of colposcopic acetic acid images of cervical precancer lesions and the classification results of colposcopic iodine images of cervical precancer lesions, and the classification results of colposcopic acetic acid images of cervical precancer lesions and the classification results of colposcopic iodine images of cervical precancer lesions are weightedly fused to obtain the final classification result.

[0009] In step (1), the video screenshot data of the prostate endoscopic resection surgery is collected and preprocessed to obtain unlabeled data, which specifically refers to: collecting video screenshot data from the prostate endoscopic resection surgery, removing abnormal images after screening, and obtaining screened images, wherein the abnormal images include blurred images and overexposed images; uniformly resizing the screened images to a size of 224*224, removing areas unrelated to the endoscope during the operation, and performing data enhancement processing such as rotation, flipping, and Gaussian blurring to obtain unlabeled data.

[0010] In step (1), the inputting of unlabeled data into the self-supervised training framework for self-supervised training and optimizing feature extraction to obtain optimized features specifically refers to:

[0011] (1a) Building teacher and student networks:

[0012] Self-supervised learning is achieved by building a teacher network and a student network, which have the same architecture and process different views from the same image;

[0013] (1b) Image view generation: The same image is randomly cropped, scaled, brightness adjusted, and color jittered to generate two different views, which are input into the teacher network and the student network respectively;

[0014] Teacher network update mechanism: The parameters of the teacher network are updated through exponential moving average. The specific formula is:

[0015] θ t ←ξθ t +(1-ξ)θ s

[0016] Among them, θ t is the teacher network parameter, θ s is the student network parameter, ξ is the momentum parameter;

[0017] Student network back propagation: The student network updates the student network weight parameters through the gradient back propagation algorithm;

[0018] (1c) Design loss function:

[0019] The loss function of the response feature optimizes the model parameters by calculating the distribution difference between the output of the teacher network and the student network;

[0020] Response features: Each response feature is normalized by softmax to obtain the probability distribution σ(z i ):

[0021]

[0022] Among them, z i Represents the eigenvalue of the i-th dimension of the response feature; z j Represents all feature vectors traversed during the softmax calculation process, a total of M feature vectors;

[0023] Self-supervised loss: loss function based on response features Defined as:

[0024]

[0025] Among them, τ t and τ s denote the temperature parameters of the teacher network and the student network respectively, represents the centering operation of the teacher network response characteristics; f t represents the softmax operation on the teacher network, f s Represents the operation on the student network parameters; z t represents the teacher network feature vector, z s represents the student network feature vector;

[0026] The class token vector is denoted as w, the multi-layer perceptron projection operation is denoted as f, and f(w) is denoted as the multi-layer perceptron projection operation on the class token feature; the global attention information brought by the class token feature is retained, and the result of vector concatenation is used to guide the learning of the student network to obtain better optimization; Φ s(w) The vector concatenation operation is expressed as:

[0027] Φ s (w) = concat[f(w),w]

[0028] Finally, the loss function in the self-supervised training process is expressed as:

[0029]

[0030] In the formula, F t represents the calculation of the eigenvector of the teacher network, F s represents the calculation of the eigenvector of the student network, Φ t (w) and Φ s (w) represents the splicing operation of the teacher network and student network parameters during the calculation process;

[0031] pass The cross entropy loss method is used to learn the distribution characteristics of the two different networks, the teacher network and the student network.

[0032] In step (1), the specific steps of the self-supervised training are as follows:

[0033] (5a) Weight initialization: Use the pre-trained Imagenet ViT-1k weights as the initialization weights; ImagenetViT-1k refers to the 1k-size visual Transformer trained on the Imagenet dataset;

[0034] (5b) Parameter optimization: Set the weight decay to 0.05, the batch size to 32, the learning rate to start from 5e-5 using the cosine scheduler, the training period to 10 epochs, and use the early stopping strategy and mixed precision training mechanism.

[0035] In step (2), the steps of collecting and preprocessing the colposcopic images of the patient's cervical precancerous lesions to obtain annotated colposcopic acetic acid images and iodine images of the cervical precancerous lesions specifically include the following steps in order:

[0036] (2a) Data collection: A total of 3294 colposcopic images of 966 patients were collected to form a data set. The colposcopic images included colposcopic acetic acid images and colposcopic iodine images of the same patient. The collected images were annotated into four categories according to the pathological cervical intraepithelial neoplasia grading: Normal, CIN1, CIN2, and CIN3.

[0037] (2b) Data partitioning: The dataset is divided into a training set and a test set in a ratio of 4:1, where the training set includes 2635 images and the test set includes 659 images;

[0038] (2c) Data preprocessing: The dataset is enhanced by random rotation, cropping, scaling, color transformation, and brightness adjustment. The image size is uniformly adjusted to 224 × 224 pixels and standardized.

[0039] In step (2), the specific steps of the fine-tuning training are as follows:

[0040] (6a) Initialization: Use the optimized model weights to initialize the second visual Transformer model and the third visual Transformer model;

[0041] (6b) Optimizer and parameter setting: Use the AdamW optimizer, set the weight decay to 0.05, the batch size to 64, the learning rate gradually decreased from 5e-3 to 5e-4, the training cycle to 200 rounds, and use a 20-round warm-up mechanism to accelerate the convergence of the second visual Transformer model and the third visual Transformer model.

[0042] In step (3), the weighted fusion of the classification results of the colposcopic acetic acid images of cervical precancerous lesions and the classification results of the colposcopic iodine images of cervical precancerous lesions specifically refers to: using a multi-layer perceptron as a feature fusion device, directly splicing the category mark features of the classification results of the second visual Transformer model for the acetic acid images and the category mark features of the classification results of the third visual Transformer model for the iodine images as inputs, and outputting the final classification results after weighted fusion by the multi-layer perceptron. The calculation formula of the multi-layer perceptron is:

[0043] y=σ(W*concat[x1,x2]+b)

[0044] Among them, x1 is the category mark feature vector generated by the acetic acid picture after the second visual Transformer model, x2 is the category mark feature vector generated by the iodine picture after the third visual Transformer model, W is the weight matrix, b is the bias vector, σ is the ReLU activation function, and y is the final classification result vector.

[0045] It can be seen from the above technical scheme that the beneficial effects of the present invention are: First, the present invention reduces the cost of medical annotation and improves the model performance: the present invention introduces a self-supervised learning method and utilizes a large amount of unlabeled medical image data to improve the generalization ability and classification accuracy of the ViT model, i.e., the visual Transformer model, in the grading of cervical precancerous lesions; the self-supervised method avoids the dependence on large-scale annotated data, significantly reduces the clinical expertise and time cost required for data annotation, and at the same time improves the model's recognition ability for high-risk lesions such as CIN2+; second, the present invention uses an attention mechanism to improve performance in specific fields: the present invention uses a self-supervised training stage to adjust the attention-related weights of the visual Transformer to optimize the global and local features of the model Extraction capability; optimize the cervical precancerous lesion image classification task through self-supervision, and further utilize the characteristic expression of lesions in different views through multi-view fusion. Experiments show that the present invention effectively solves the overfitting problem of the visual Transformer model on small data sets, making the model more suitable for fine-grained tasks in the field of medical imaging; thirdly, the present invention improves the auxiliary diagnosis capability of cervical cancer screening in low-resource areas: by utilizing endoscopic and colposcopy image data, the present invention realizes efficient auxiliary diagnosis of CIN grading, especially in areas with limited medical resources. The present invention provides an efficient and low-cost solution for automated screening, helps doctors improve the diagnostic accuracy and reduce the missed diagnosis rate, and provides important technical support for the prevention of early cervical cancer. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 is a flow chart of the method of the present invention;

[0047] Figure 2 is a flow chart of the method of self-supervised training and fine-tuning training in the present invention;

[0048] Figure 3 It is a schematic diagram of the structure of the visual Transformer model in the present invention;

[0049] Figure 4 It is a schematic diagram of multi-view feature fusion in the present invention. DETAILED DESCRIPTION

[0050] like Figure 1 As shown, a multi-view fusion cervical precancerous lesion image classification method based on self-supervised learning includes the following steps in sequence:

[0051] (1) Collecting video screenshot data of endoscopic prostate resection surgery and preprocessing it to obtain unlabeled data, inputting the unlabeled data into a self-supervised training framework for self-supervised training, optimizing feature extraction, obtaining optimized features, and optimizing the attention weights of the first visual Transformer model by optimizing the features to obtain optimized model weights;

[0052] (2) Collecting and preprocessing colposcopic images of cervical precancerous lesions of patients, obtaining annotated colposcopic acetic acid images and colposcopic iodine images of cervical precancerous lesions, inputting the optimized model weights, annotated colposcopic acetic acid images and colposcopic iodine images of cervical precancerous lesions into the second visual Transformer model and the third visual Transformer model for fine-tuning training, respectively, and using the optimized model weights as the initial weights for fine-tuning the second visual Transformer model and the third visual Transformer model, respectively, to obtain the optimized second visual Transformer model and the third visual Transformer model;

[0053] (3) The colposcopic images of cervical precancer lesions to be classified are preprocessed, and the preprocessed colposcopic images of cervical precancer lesions are input into the optimized second visual Transformer model and the third visual Transformer model for hierarchical prediction, respectively, to obtain the classification results of colposcopic acetic acid images of cervical precancer lesions and the classification results of colposcopic iodine images of cervical precancer lesions, and the classification results of colposcopic acetic acid images of cervical precancer lesions and the classification results of colposcopic iodine images of cervical precancer lesions are weightedly fused to obtain the final classification result.

[0054] In step (1), the video screenshot data of the prostate endoscopic resection surgery is collected and preprocessed to obtain unlabeled data, which specifically refers to: collecting video screenshot data from the prostate endoscopic resection surgery, removing abnormal images after screening, and obtaining screened images, wherein the abnormal images include blurred images and overexposed images; uniformly resizing the screened images to a size of 224*224, removing areas unrelated to the endoscope during the operation, and performing data enhancement processing such as rotation, flipping, and Gaussian blurring to obtain unlabeled data.

[0055] like Figure 2 As shown, in step (1), the inputting of unlabeled data into the self-supervised training framework for self-supervised training and optimizing feature extraction to obtain optimized features specifically refers to:

[0056] (1a) Building teacher and student networks:

[0057] Self-supervised learning is achieved by building a teacher network and a student network, which have the same architecture and process different views from the same image;

[0058] (1b) Image view generation: The same image is randomly cropped, scaled, brightness adjusted, and color jittered to generate two different views, which are input into the teacher network and the student network respectively;

[0059] Teacher network update mechanism: The parameters of the teacher network are updated through exponential moving average. The specific formula is:

[0060] θ t ←ξθ t +(1-ξ)θ s

[0061] Among them, θ t is the teacher network parameter, θ s is the student network parameter, ξ is the momentum parameter;

[0062] Student network back propagation: The student network updates the student network weight parameters through the gradient back propagation algorithm;

[0063] (1c) Design loss function:

[0064] The loss function of the response feature optimizes the model parameters by calculating the distribution difference between the output of the teacher network and the student network;

[0065] Response features: Each response feature is normalized by softmax to obtain the probability distribution σ(z i ):

[0066]

[0067] Among them, z i Represents the eigenvalue of the i-th dimension of the response feature; z j Represents all feature vectors traversed during the softmax calculation process, a total of M feature vectors;

[0068] Self-supervised loss: loss function based on response features Defined as:

[0069]

[0070] Among them, τ t and τ s denote the temperature parameters of the teacher network and the student network respectively, represents the centering operation of the teacher network response characteristics; f t represents the softmax operation on the teacher network, f sRepresents the operation on the student network parameters; z t represents the teacher network feature vector, z s represents the student network feature vector;

[0071] The class token vector is denoted as w, the multi-layer perceptron projection operation is denoted as f, and f(w) is denoted as the multi-layer perceptron projection operation on the class token feature; the global attention information brought by the class token feature is retained, and the result of vector concatenation is used to guide the learning of the student network to obtain better optimization; Φ s (w) The vector concatenation operation is expressed as:

[0072] Φ s (w) = concat[f(w),w]

[0073] Finally, the loss function in the self-supervised training process is expressed as:

[0074]

[0075] In the formula, F t represents the calculation of the eigenvector of the teacher network, F s represents the calculation of the eigenvector of the student network, Φ t (w) and Φ s (w) represents the splicing operation of the teacher network and student network parameters during the calculation process;

[0076] pass The cross entropy loss method is used to learn the distribution characteristics of the two different networks, the teacher network and the student network.

[0077] Specifically, log(F s ) calculates the logarithm of the probability distribution of the student network output. When the calculated F s When it is close to 1, log(F s ) is close to 0, indicating that the student network has a higher prediction probability for this category and is more consistent with the teacher network, and the loss value is small at this time; on the contrary, when F s When it is close to 0, log(F s ) approaches negative infinity, the loss value will increase, which will prompt the model to adjust the parameters of the student network so that its output is closer to the output of the teacher network. Multiply by -F t The importance of different categories is weighted according to the probability distribution of the teacher network, further emphasizing the guiding role of the teacher network in the learning process.

[0078] like Figure 2 As shown, in step (1), the specific steps of the self-supervised training are as follows:

[0079] (5a) Weight initialization: Use the pre-trained Imagenet ViT-1k weights as the initialization weights; ImagenetViT-1k refers to the 1k-size visual Transformer trained on the Imagenet dataset;

[0080] (5b) Parameter optimization: Set the weight decay to 0.05, the batch size to 32, the learning rate to start from 5e-5 using the cosine scheduler, the training period to 10 epochs, and use the early stopping strategy and mixed precision training mechanism.

[0081] In step (2), the steps of collecting and preprocessing the colposcopic images of the patient's cervical precancerous lesions to obtain annotated colposcopic acetic acid images and iodine images of the cervical precancerous lesions specifically include the following steps in order:

[0082] (2a) Data collection: A total of 3294 colposcopic images of 966 patients were collected to form a data set. The colposcopic images included colposcopic acetic acid images and colposcopic iodine images of the same patient. The collected images were annotated into four categories according to the pathological cervical intraepithelial neoplasia grading: Normal, CIN1, CIN2, and CIN3.

[0083] (2b) Data partitioning: The dataset is divided into a training set and a test set in a ratio of 4:1, where the training set includes 2635 images and the test set includes 659 images;

[0084] (2c) Data preprocessing: The dataset is enhanced by random rotation, cropping, scaling, color transformation, and brightness adjustment. The image size is uniformly adjusted to 224 × 224 pixels and standardized.

[0085] like Figure 2 As shown, in step (2), the specific steps of the fine-tuning training are as follows:

[0086] (6a) Initialization: Use the optimized model weights to initialize the second visual Transformer model and the third visual Transformer model;

[0087] (6b) Optimizer and parameter setting: Use the AdamW optimizer, set the weight decay to 0.05, the batch size to 64, the learning rate gradually decreased from 5e-3 to 5e-4, the training cycle to 200 rounds, and use a 20-round warm-up mechanism to accelerate the convergence of the second visual Transformer model and the third visual Transformer model.

[0088] like Figure 3As shown in the figure, the structures of the first visual Transformer model, the second visual Transformer model, and the third visual Transformer model are the same. The input colposcopic cervical precancerous lesion image is divided into multiple patches, converted into vector form by linear projection, and the position embedding code is added in this process to complete the position information of the patch in the image. Then, an additional category mark feature is added to the beginning of the input patch vector sequence, and then the sequence enters the Transformer encoder composed of multiple identical encoder layers stacked together. Each Transformer encoder layer contains normalization operations, multi-head attention mechanisms, and multi-layer perceptrons, and there are normalization operations before and after the multi-head attention mechanisms and multi-layer perceptrons. After being processed by the Transformer encoder, the category mark features are extracted, and the final classification results are obtained through multi-layer perceptron head processing.

[0089] like Figure 4 As shown, in step (3), the weighted fusion of the classification results of the colposcopic acetic acid images of cervical precancerous lesions and the classification results of the colposcopic iodine images of cervical precancerous lesions specifically refers to: using a multi-layer perceptron as a feature fusion device, directly splicing the category mark features of the classification results of the second visual Transformer model for the acetic acid images and the category mark features of the classification results of the third visual Transformer model for the iodine images as inputs, and outputting the final classification results after weighted fusion by the multi-layer perceptron. The calculation formula of the multi-layer perceptron is:

[0090] y=σ(W*concat[x1,x2]+b)

[0091] Among them, x1 is the category mark feature vector generated by the acetic acid picture after the second visual Transformer model, x2 is the category mark feature vector generated by the iodine picture after the third visual Transformer model, W is the weight matrix, b is the bias vector, σ is the ReLU activation function, and y is the final classification result vector.

[0092] In summary, the present invention reduces the cost of medical annotation and improves model performance: the present invention introduces a self-supervised learning method and utilizes a large amount of unlabeled medical image data to improve the generalization ability and classification accuracy of the ViT model, i.e., the visual Transformer model, in the grading of cervical precancerous lesions; the self-supervised approach avoids the reliance on large-scale annotated data, significantly reduces the clinical expertise and time cost required for data annotation, and at the same time improves the model's ability to recognize high-risk lesions such as CIN2+; secondly, the present invention uses an attention mechanism to improve performance in specific areas: the present invention uses a self-supervised training phase to adjust the attention-related weights of the visual Transformer, and optimizes the model's global and local feature extraction capabilities; through self-supervision The task of classifying cervical precancerous lesions images is optimized in a supervised manner, and the feature expressions of lesions in different views are further utilized through multi-view fusion. Experiments show that the present invention effectively solves the overfitting problem of the visual Transformer model on small data sets, making the model more suitable for fine-grained tasks in the field of medical imaging; thirdly, the present invention improves the auxiliary diagnosis capability of cervical cancer screening in low-resource areas: by utilizing endoscopic and colposcopy image data, the present invention realizes efficient auxiliary diagnosis of CIN grading, especially in areas with limited medical resources. The present invention provides an efficient and low-cost solution for automated screening, helps doctors improve the diagnostic accuracy and reduce the missed diagnosis rate, and provides important technical support for the prevention of early cervical cancer.

Claims

1. A multi-view fusion cervical precancerous lesion image classification method based on self-supervised learning, characterized by: The method comprises the following steps in order: (1) Collecting video screenshot data of endoscopic prostate resection surgery and preprocessing it to obtain unlabeled data, inputting the unlabeled data into a self-supervised training framework for self-supervised training, optimizing feature extraction, obtaining optimized features, and optimizing the attention weights of the first visual Transformer model by optimizing the features to obtain optimized model weights; (2) Collecting and preprocessing colposcopic images of cervical precancerous lesions of patients, obtaining annotated colposcopic acetic acid images and colposcopic iodine images of cervical precancerous lesions, inputting the optimized model weights, annotated colposcopic acetic acid images and colposcopic iodine images of cervical precancerous lesions into the second visual Transformer model and the third visual Transformer model for fine-tuning training, respectively, and using the optimized model weights as the initial weights for fine-tuning the second visual Transformer model and the third visual Transformer model, respectively, to obtain the optimized second visual Transformer model and the third visual Transformer model; (3) The colposcopic images of cervical precancer lesions to be classified are preprocessed, and the preprocessed colposcopic images of cervical precancer lesions are input into the optimized second visual Transformer model and the third visual Transformer model for hierarchical prediction, respectively, to obtain the classification results of colposcopic acetic acid images of cervical precancer lesions and the classification results of colposcopic iodine images of cervical precancer lesions, and the classification results of colposcopic acetic acid images of cervical precancer lesions and the classification results of colposcopic iodine images of cervical precancer lesions are weightedly fused to obtain the final classification result.

2. The multi-view fusion cervical precancerous lesion image classification method based on self-supervised learning according to claim 1 is characterized by: In step (1), the video screenshot data of the prostate endoscopic resection surgery is collected and preprocessed to obtain unlabeled data, which specifically refers to: collecting video screenshot data from the prostate endoscopic resection surgery, removing abnormal images after screening, and obtaining screened images, wherein the abnormal images include blurred images and overexposed images; uniformly resizing the screened images to a size of 224*224, removing areas unrelated to the endoscope during the operation, and performing data enhancement processing such as rotation, flipping, and Gaussian blurring to obtain unlabeled data.

3. The multi-view fusion cervical precancerous lesion image classification method based on self-supervised learning according to claim 1 is characterized by: In step (1), the inputting of unlabeled data into the self-supervised training framework for self-supervised training and optimizing feature extraction to obtain optimized features specifically refers to: (1a) Building teacher and student networks: Self-supervised learning is achieved by building a teacher network and a student network, which have the same architecture and process different views from the same image; (1b) Image view generation: The same image is randomly cropped, scaled, brightness adjusted, and color jittered to generate two different views, which are input into the teacher network and the student network respectively; Teacher network update mechanism: The parameters of the teacher network are updated through exponential moving average. The specific formula is: i t ←ξθ t +(1-ξ)θ s Among them, θ t is the teacher network parameter, θ s is the student network parameter, ξ is the momentum parameter; Student network back propagation: The student network updates the student network weight parameters through the gradient back propagation algorithm; (1c) Design loss function: The loss function of the response feature optimizes the model parameters by calculating the distribution difference between the output of the teacher network and the student network; Response features: Each response feature is normalized by softmax to obtain the probability distribution σ(z i ): Among them, z i Represents the eigenvalue of the i-th dimension of the response feature; z j Represents all feature vectors traversed during the softmax calculation process, a total of M feature vectors; Self-supervised loss: loss function based on response features Defined as: Among them, τ t and τ s denote the temperature parameters of the teacher network and the student network respectively, represents the centering operation of the teacher network response characteristics; f t represents the softmax operation on the teacher network, f s Represents the operation on the student network parameters; z t represents the teacher network feature vector, z s represents the student network feature vector; The class token vector is denoted as w, the multi-layer perceptron projection operation is denoted as f, and f(w) is denoted as the multi-layer perceptron projection operation on the class token feature; the global attention information brought by the class token feature is retained, and the result of vector concatenation is used to guide the learning of the student network to obtain better optimization; Φ s (w) The vector concatenation operation is expressed as: Φ s (w)=concat[f(w),w] Finally, the loss function in the self-supervised training process is expressed as: In the formula, F t represents the calculation of the eigenvector of the teacher network, F s represents the calculation of the eigenvector of the student network, Φ t (w) and Φ s (w) represents the splicing operation of the teacher network and student network parameters during the calculation process; pass The cross entropy loss method is used to learn the distribution characteristics of the two different networks, the teacher network and the student network.

4. The multi-view fusion cervical precancerous lesion image classification method based on self-supervised learning according to claim 1 is characterized by: In step (1), the specific steps of the self-supervised training are as follows: (5a) Weight initialization: Use the pre-trained Imagenet ViT-1k weights as the initialization weights; Imagenet ViT-1k refers to training a 1k-size visual Transformer on the Imagenet dataset; (5b) Parameter optimization: Set the weight decay to 0.05, the batch size to 32, the learning rate to start from 5e-5 using the cosine scheduler, the training period to 10 epochs, and use the early stopping strategy and mixed precision training mechanism.

5. The multi-view fusion cervical precancerous lesion image classification method based on self-supervised learning according to claim 1 is characterized by: In step (2), the steps of collecting and preprocessing the colposcopic images of the patient's cervical precancerous lesions to obtain annotated colposcopic acetic acid images and iodine images of the cervical precancerous lesions specifically include the following steps in order: (2a) Data collection: A total of 3294 colposcopic images of 966 patients were collected to form a data set. The colposcopic images included colposcopic acetic acid images and colposcopic iodine images of the same patient. The collected images were annotated into four categories according to the pathological cervical intraepithelial neoplasia grading: Normal, CIN1, CIN2, and CIN3. (2b) Data partitioning: The dataset is divided into a training set and a test set in a ratio of 4:1, where the training set includes 2635 images and the test set includes 659 images; (2c) Data preprocessing: The dataset is enhanced by random rotation, cropping, scaling, color transformation, and brightness adjustment. The image size is uniformly adjusted to 224 × 224 pixels and standardized.

6. The multi-view fusion cervical precancerous lesion image classification method based on self-supervised learning according to claim 1, characterized in that: In step (2), the specific steps of the fine-tuning training are as follows: (6a) Initialization: Use the optimized model weights to initialize the second visual Transformer model and the third visual Transformer model; (6b) Optimizer and parameter setting: Use the AdamW optimizer, set the weight decay to 0.05, the batch size to 64, the learning rate gradually decreased from 5e-3 to 5e-4, the training cycle to 200 rounds, and use a 20-round warm-up mechanism to accelerate the convergence of the second visual Transformer model and the third visual Transformer model.

7. The multi-view fusion cervical precancerous lesion image classification method based on self-supervised learning according to claim 1, characterized in that: In step (3), the weighted fusion of the classification results of the colposcopic acetic acid images of cervical precancerous lesions and the classification results of the colposcopic iodine images of cervical precancerous lesions specifically refers to: using a multi-layer perceptron as a feature fusion device, directly splicing the category mark features of the classification results of the second visual Transformer model for the acetic acid images and the category mark features of the classification results of the third visual Transformer model for the iodine images as inputs, and outputting the final classification results after weighted fusion by the multi-layer perceptron. The calculation formula of the multi-layer perceptron is: y=σ(W*concat[x1,x2]+b) Among them, x1 is the category mark feature vector generated by the acetic acid picture after the second visual Transformer model, x2 is the category mark feature vector generated by the iodine picture after the third visual Transformer model, W is the weight matrix, b is the bias vector, σ is the ReLU activation function, and y is the final classification result vector.

Images