A multi-view fusion cervical precancerous lesion image classification method based on self-supervised learning

Abstract

The present application relates to a kind of multi-view fusion cervical precancerous lesion image classification methods based on self-supervised learning, comprising: the video screenshot data of prostate endoscope resection surgery is collected, and pretreatment is carried out, and the optimized model weight is obtained;Second visual Transformer model, third visual Transformer model are obtained after optimization;Cervical precancerous lesion colposcope acetic acid picture classification result and cervical precancerous lesion colposcope iodine picture classification result are weighted fusion, and final classification result is obtained.The present application introduces self-supervised learning method, utilizes a large number of unlabeled medical picture data, improves the generalization ability and classification accuracy of ViT model, i.e., visual Transformer model in cervical precancerous lesion grading;Realize the efficient auxiliary diagnosis of CIN grading, especially in the area of limited medical resources, provide efficient, low-cost solution for automated screening, help doctors improve diagnostic accuracy, and reduce the rate of missed diagnosis, provide important technical support for early cervical cancer prevention.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of deep learning and medical image analysis, and particularly relates to a multi-view fusion cervical precancerous lesion image classification method based on self-supervised learning. BACKGROUND

[0002] Cervical cancer is one of the major threats to women's health worldwide, and its early screening and accurate grading are crucial to reducing the incidence. However, traditional deep learning methods based on supervised learning face significant challenges in medical image processing: the acquisition of labeled data requires rich clinical experience and is time-consuming and labor-intensive, especially in developing countries and areas with limited medical resources. In addition, the dependence of supervised learning methods on large amounts of high-quality labeled data makes it difficult to effectively apply them in data-deficient situations, and this contradiction is particularly pronounced in Transformer models.

[0003] Visual Transformer better captures global and local features in images through self-attention mechanisms and multi-head attention mechanisms. In the study of cervical precancerous lesion grading, Transformer has gradually been used to replace traditional CNN architecture, showing stronger performance. However, due to the high requirements of ViT for the size and quality of training data, direct migration to cervical precancerous lesion grading will face the problems of overfitting due to insufficient data and mismatch between the attention patterns of pre-trained models and specific medical tasks. SUMMARY

[0004] To solve the data dependence and classification accuracy problems in cervical precancerous lesion grading, the present application aims to provide a multi-view fusion cervical precancerous lesion image classification method based on self-supervised learning that improves the generalization ability and classification accuracy in cervical precancerous lesion grading and optimizes the global and local feature extraction ability of the model.

[0005] To achieve the above-mentioned purpose, the present application adopts the following technical solution: a multi-view fusion cervical precancerous lesion image classification method based on self-supervised learning, which comprises the following steps in sequence:

[0006] (1) Collect video screenshot data of prostate endoscopic resection surgery and perform preprocessing to obtain unlabeled data, input the unlabeled data into a self-supervised training framework for self-supervised training, optimize feature extraction, obtain optimized features, and optimize the attention weights of the first visual Transformer model through the optimized features to obtain optimized model weights;

[0007] (2) Collect the colposcopic images of the patient's cervical precancerous lesions and preprocess them to obtain labeled colposcopic acetic acid pictures and colposcopic iodine pictures of cervical precancerous lesions. Input the optimized model weights, labeled colposcopic acetic acid pictures, and colposcopic iodine pictures of cervical precancerous lesions into the second visual Transformer model and the third visual Transformer model for fine-tuning training. Use the optimized model weights as the initial weights for the fine-tuning training of the second visual Transformer model and the third visual Transformer model, and obtain the optimized second visual Transformer model and the third visual Transformer model, respectively.

[0008] (3) Preprocess the colposcopic images of the cervical precancerous lesions to be classified, and input the preprocessed colposcopic images of the cervical precancerous lesions into the optimized second visual Transformer model and the third visual Transformer model for hierarchical prediction to obtain the classification results of the colposcopic acetic acid pictures and the colposcopic iodine pictures of the cervical precancerous lesions. Weighted fusion of the classification results of the colposcopic acetic acid pictures and the colposcopic iodine pictures of the cervical precancerous lesions is performed to obtain the final classification result.

[0009] In step (1), the video screenshot data of the prostate endoscopic resection surgery is collected and preprocessed to obtain unannotated data, which specifically refers to: collecting video screenshot data from prostate endoscopic resection surgery, removing abnormal images after screening to obtain screened images, the abnormal images including blurred images and overexposed images; uniformly adjusting the size of the screened images to 224*224, removing the intraoperative and endoscope unrelated areas, and performing data enhancement processing such as rotation, flipping, and Gaussian blur to obtain unannotated data.

[0010] In step (1), the unannotated data is input into the self-supervised training framework for self-supervised training to optimize the feature extraction and obtain the optimized features, which specifically refers to:

[0011] (1a) Construct a teacher network and a student network:

[0012] Self-supervised learning is achieved by constructing a teacher network and a student network, both of which have the same architecture and process different views from the same image.

[0013] (1b) Image view generation: the same image is transformed into two different views after random cropping, scaling, brightness adjustment, and color jitter, and is input into the teacher network and the student network, respectively.

[0014] Teacher network update mechanism: the parameters of the teacher network are updated by exponential moving average, and the specific formula is:

[0015] θ t ←ξθ t +(1-ξ)θ s

[0016] where θ t is the teacher network parameter, θ s is the student network parameter, and ξ is the momentum parameter;

[0017] Student network backpropagation: the student network updates the student network weight parameters through the gradient backpropagation algorithm;

[0018] (1c) Design the loss function:

[0019] The loss function of the response feature optimizes the model parameters by calculating the distribution difference between the outputs of the teacher network and the student network;

[0020] Response feature: normalize each response feature through softmax to obtain the probability distribution σ(z i ) of each dimension:

[0021]

[0022] where z i represents the i-th feature value of the response feature; z j represents all feature vectors traversed in the softmax calculation process, a total of M feature vectors;

[0023] Self-supervised loss: the loss function based on the response feature is defined as:

[0024]

[0025] where τ t and τ s represent the temperature parameters of the teacher network and the student network, respectively, represents the centering operation of the teacher network response feature; f t represents the softmax operation on the teacher network, f s represents the operation on the student network parameters; z t represents the teacher network feature vector, and z s represents the student network feature vector;

[0026] The class token vector is denoted as w, the multi-layer perceptron projection operation is denoted as f, and f(w) is denoted as the multi-layer perceptron projection operation on the class token feature. The global attention information brought by the class token feature is retained, and the result after vector splicing is used to guide the learning of the student network to obtain better optimization; Φ s(w) Vector concatenation operation is denoted as:

[0027] Φ s (w) = concat[f(w), w]

[0028] Finally, the loss function in the self-supervised training process is denoted as:

[0029]

[0030] where F t denotes the calculation of the feature vector for the teacher network, F s denotes the calculation of the feature vector for the student network, Φ t (w) and Φ s (w) respectively denote the concatenation operation of the parameters of the teacher network and the student network in the calculation process.

[0031] The distribution characteristics obtained by the two different attempt networks of the teacher network and the student network are learned in the form of cross-entropy loss.

[0032] In step (1), the specific steps of the self-supervised training are as follows:

[0033] (5a) Weight initialization: use pre-trained Imagenet ViT-1k weights as initialization weights; Imagenet ViT-1k refers to training a 1k size visual Transformer on an Imagenet dataset;

[0034] (5b) Parameter optimization: set the weight decay to 0.05, the batch size to 32, and the learning rate to start from 5e-5 using a cosine scheduler, and train for 10 epochs, while using an early stopping strategy and a mixed precision training mechanism.

[0035] In step (2), the cervical precancerous lesion colposcopy images of the patient are collected and preprocessed to obtain labeled cervical precancerous lesion colposcopy acetic acid pictures and iodine pictures, which include the following steps in order:

[0036] (2a) Data collection: collect 3294 colposcopy images of 966 patients to form a dataset, wherein the colposcopy images include colposcopy acetic acid pictures and colposcopy iodine pictures of the same patient; the collected images are labeled as Normal, CIN1, CIN2 and CIN3 according to the pathological cervical intraepithelial neoplasia grading;

[0037] (2b) Data division: divide the dataset into a training set and a test set in a ratio of 4:1, wherein the training set includes 2635 images and the test set includes 659 images;

[0038] ​(2c) Data preprocessing: The data set is enhanced, including random rotation, cropping, scaling, color transformation and brightness adjustment; the size of the image is uniformly adjusted to 224x224 pixels, and standardized processing is performed.

[0039] In step (2), the specific steps of the fine-tuning training are as follows:

[0040] (6a) Initialization: The second visual Transformer model and the third visual Transformer model are initialized using the optimized model weight.

[0041] (6b) Optimizer and parameter setting: The AdamW optimizer is used, the weight decay is set to 0.05, the batch size is 64, the learning rate is gradually reduced from 5e-3 to 5e-4, the training period is 200 rounds, and the 20-round warm-up mechanism is adopted to accelerate the convergence of the second visual Transformer model and the third visual Transformer model.

[0042] In step (3), the weighted fusion of the cervical precancerous lesion colposcopy acetic acid picture classification result and the cervical precancerous lesion colposcopy iodine picture classification result is specifically referring to: using a multi-layer perception as a feature fusioner, the class label features of the classification result of the second visual Transformer model for the acetic acid picture and the class label features of the classification result of the third visual Transformer model for the iodine picture are directly spliced as input, and the final classification result is output after weighted fusion by the multi-layer perception, and the calculation formula of the multi-layer perception is:

[0043] y = σ(W*concat[x1, x2] + b)

[0044] Wherein, x1 is the class label feature vector generated by the acetic acid picture through the second visual Transformer model, x2 is the class label feature vector generated by the iodine picture through the third visual Transformer model, W is the weight matrix, b is the bias vector, σ is the ReLU activation function, and y is the final classification result vector.

[0045] From the above technical solution, the beneficial effects of the present application are: first, the present application reduces the medical annotation cost and improves the model performance: the present application introduces a self-supervised learning method, uses a large amount of unlabeled medical picture data, improves the generalization ability and classification accuracy of the ViT model, i.e. visual Transformer model, in the classification of precancerous lesions of cervical cancer; the self-supervised method avoids the dependence on large-scale labeled data, significantly reduces the clinical professional knowledge and time cost required for data labeling, and at the same time improves the recognition ability of the model for high-risk lesions such as CIN2+; second, the present application uses an attention mechanism to improve the performance in a specific field: the present application adjusts the attention-related weights of the visual Transformer in the self-supervised training stage, optimizes the global and local feature extraction ability of the model; the precancerous lesion image classification task is optimized by the self-supervised method, and the feature expression of the lesion under different views is further utilized by the multi-view fusion method, experiments show that the present application effectively solves the overfitting problem of the visual Transformer model on small data sets, and makes the model more suitable for fine-grained tasks in the medical image field; third, the present application improves the auxiliary diagnosis ability of cervical cancer screening in low-resource areas: by using endoscopic and colposcopic image data, the present application realizes efficient auxiliary diagnosis of CIN classification, especially in areas with limited medical resources, the present application provides an efficient and low-cost solution for automatic screening, helps doctors improve the diagnosis accuracy, and reduces the missed diagnosis rate, and provides important technical support for early cervical cancer prevention. BRIEF DESCRIPTION OF DRAWINGS

[0046] Figure 1 is the method flowchart of the present application;

[0047] Figure 2 is the method flowchart of self-supervised training and fine-tuning in the present application;

[0048] Figure 3 is the structural diagram of the visual Transformer model in the present application;

[0049] Figure 4 is the schematic diagram of multi-view feature fusion in the present application. DETAILED DESCRIPTION

[0050] As shown in Figure 1 , a multi-view fusion precancerous lesion image classification method based on self-supervised learning, the method comprises the following steps in sequence:

[0051] (1) Collect video screenshots of prostate endoscopy surgery and preprocess them to obtain unlabeled data. Input the unlabeled data into the self-supervised training framework for self-supervised training, optimize feature extraction, obtain optimized features, optimize the attention weights of the first vision Transformer model through optimized features, and obtain the optimized model weights.

[0052] (2) Collect colposcopy images of patients with precancerous cervical lesions and preprocess them to obtain labeled colposcopy images of precancerous cervical lesions with acetic acid and iodine. Input the optimized model weights, labeled colposcopy images of precancerous cervical lesions with acetic acid and iodine into the second vision Transformer model and the third vision Transformer model respectively for fine-tuning training. Use the optimized model weights as the initial weights when training and fine-tuning the second vision Transformer model and the third vision Transformer model to obtain the optimized second vision Transformer model and the third vision Transformer model respectively.

[0053] (3) Preprocess the colposcopy images of cervical precancerous lesions to be classified. Input the preprocessed colposcopy images of cervical precancerous lesions into the optimized second vision Transformer model and third vision Transformer model respectively for hierarchical prediction to obtain the classification results of colposcopy acetic acid images and colposcopy iodine images of cervical precancerous lesions. Perform weighted fusion of the classification results of colposcopy acetic acid images and colposcopy iodine images of cervical precancerous lesions to obtain the final classification result.

[0054] In step (1), the acquisition of video screenshots from the prostate endoscopy resection surgery and the preprocessing to obtain unlabeled data specifically refers to: collecting video screenshots from the prostate endoscopy resection surgery, removing abnormal images after screening to obtain screened images, the abnormal images including blurred images and overexposed images; uniformly adjusting the size of the screened images to 224*224, removing areas unrelated to the endoscope during the operation, and performing data enhancement processing such as rotation, flipping, and Gaussian blur to obtain unlabeled data.

[0055] like Figure 2 As shown, in step (1), the step of inputting unlabeled data into the self-supervised training framework for self-supervised training, optimizing feature extraction, and obtaining optimized features specifically refers to:

[0056] (1a) Constructing teacher and student networks:

[0057] Self-supervised learning is achieved by constructing a teacher network and a student network, both of which have the same architecture and process different views from the same image.

[0058] (1b) Image view generation: Two different views are generated from the same image after random cropping, scaling, brightness adjustment, and color jitter transformation, and are input into the teacher network and the student network, respectively;

[0059] Teacher network update mechanism: The parameters of the teacher network are updated by exponential moving average, and the specific formula is:

[0060] θ t ←ξθ t +(1-ξ)θ s

[0061] Where θ t is the teacher network parameter, θ s is the student network parameter, and ξ is the momentum parameter.

[0062] Student network back propagation: The student network updates the student network weight parameters through the gradient back propagation algorithm.

[0063] (1c) Design loss function:

[0064] The loss function of the response feature is optimized by calculating the distribution difference between the outputs of the teacher network and the student network.

[0065] Response feature: The softmax is used to normalize each response feature to obtain the probability distribution σ(z i ) of each dimension:

[0066]

[0067] Where z i represents the i-th feature value of the response feature; z j represents all feature vectors traversed in the softmax calculation process, a total of M feature vectors.

[0068] Self-supervised loss: The loss function based on the response feature is defined as:

[0069]

[0070] Where τ t and τ s represent the temperature parameters of the teacher network and the student network, respectively, represents the center operation of the teacher network response feature; f t represents the softmax operation on the teacher network, and f s ​This indicates operations on student network parameters; z t Let z represent the feature vector of the teacher network. s Represents the student network feature vector;

[0071] Let w denote the class token vector, f denote the multilayer perceptron projection operation, and f(w) denote the multilayer perceptron projection operation on the class token features. The global attention information from the class token features is preserved, and the concatenated vectors are used to guide the learning of the student network to achieve better optimization. s (w) Vector concatenation is represented as:

[0072] Φ s (w) = concat[f(w),w]

[0073] Finally, the loss function during self-supervised training is expressed as:

[0074]

[0075] In the formula, F t F represents the calculation of the eigenvectors of the teacher network. s Φ represents the calculation of the eigenvectors of the student network. t (w) and Φ s (w) represent the splicing operation of the teacher network and student network parameters during the calculation process, respectively;

[0076] pass The method of cross-entropy loss is used to learn the distribution characteristics obtained by two different networks, the teacher network and the student network.

[0077] Specifically, log(F) s The logarithm of the student network output probability distribution was calculated, and when the calculated F... s When it approaches 1, log(F) s A value close to 0 indicates that the student network predicts a high probability for that category and has good consistency with the teacher network, resulting in a small loss value; conversely, when F... s When it approaches 0, log(F) s As the value of approaches negative infinity, the loss value increases, prompting the model to adjust the parameters of the student network to make its output closer to the teacher network's output. Multiplying by -F t The importance of different categories is then weighted according to the probability distribution of the teacher network, further emphasizing the guiding role of the teacher network in the learning process.

[0078] like Figure 2 As shown, in step (1), the specific steps of the self-supervised training are as follows:

[0079] (5a) Weight initialization: using pre-trained Imagenet ViT-1k weights as initialization weights; Imagenet ViT-1k refers to training a visual Transformer of 1k size on the Imagenet dataset;

[0080] (5b) Parameter optimization: setting the weight decay to 0.05, the batch size to 32, and the learning rate to start from 5e-5 using a cosine scheduler, training for 10 epochs, while using an early stopping strategy and a mixed precision training mechanism.

[0081] In step (2), the cervical precancerous lesion colposcopy images of the patient are collected and preprocessed to obtain labeled cervical precancerous lesion colposcopy acetic acid pictures and iodine pictures, which specifically include the following steps in order:

[0082] (2a) Data collection: collecting 3294 colposcopy images of 966 patients to form a dataset, the colposcopy images including colposcopy acetic acid pictures and colposcopy iodine pictures of the same patient; labeling the collected images according to the pathological cervical intraepithelial neoplasia grade into four categories of Normal, CIN1, CIN2 and CIN3;

[0083] (2b) Data division: dividing the dataset into a training set and a test set in a ratio of 4:1, wherein the training set includes 2635 images and the test set includes 659 images;

[0084] (2c) Data preprocessing: performing enhancement processing on the dataset, including random rotation, cropping, scaling, color transformation and brightness adjustment; uniformly adjusting the size of the images to 224x224 pixels and performing standardization processing.

[0085] As shown in step (2), the specific steps of the fine-tuning training are as follows: Figure 2

[0086] (6a) Initialization: using the optimized model weights to initialize the second visual Transformer model and the third visual Transformer model;

[0087] (6b) Optimizer and parameter setting: using the AdamW optimizer, setting the weight decay to 0.05, the batch size to 64, and the learning rate to gradually decrease from 5e-3 to 5e-4, training for 200 rounds, and using a 20-round warm-up mechanism to accelerate the convergence of the second visual Transformer model and the third visual Transformer model.

[0088] As shown in step (2), the specific steps of the fine-tuning training are as follows: Figure 3 ​As shown, the structures of the first visual Transformer model, the second visual Transformer model, and the third visual Transformer model are all the same. The input colposcope precancerous lesion image is divided into multiple patches, converted into a vector form through linear projection, and position embedding coding is added in this process to complete the position information of the patch in the image. Then, an additional class label feature is added at the beginning of the input patch vector sequence, and then the sequence enters the Transformer encoder stacked by multiple identical encoder layers. Each Transformer encoder layer contains normalization operation, multi-head attention mechanism, and multi-layer perception, and normalization operation is added before and after the multi-head attention mechanism and the multi-layer perception. After processing by the Transformer encoder, the class label feature is extracted and processed by the multi-layer perception head to obtain the final classification result.

[0089] As shown in FIG. 3, in step (3), the classification results of the colposcope precancerous lesion acetic acid picture and the colposcope precancerous lesion iodine picture are fused by weighting, and the specific process is as follows. Figure 4 As shown in FIG. 3, in step (3), the classification results of the colposcope precancerous lesion acetic acid picture and the colposcope precancerous lesion iodine picture are fused by weighting, and the specific process is as follows.

[0090] y = σ (W * concat [x1, x2] + b)

[0091] Wherein, x1 is the class label feature vector generated by the second visual Transformer model for the acetic acid picture, x2 is the class label feature vector generated by the third visual Transformer model for the iodine picture, W is the weight matrix, b is the bias vector, σ is the ReLU activation function, and y is the final classification result vector.

[0092] In summary, the present application reduces the cost of medical annotation and improves the model performance: by introducing a self-supervised learning method, the present application uses a large amount of unlabeled medical picture data to improve the generalization ability and classification accuracy of the ViT model, i.e., the visual Transformer model, in the classification of cervical precancerous lesions; the self-supervised method avoids the dependence on large-scale labeled data, significantly reduces the clinical professional knowledge and time cost required for data labeling, and at the same time improves the recognition ability of the model for high-risk lesions such as CIN2+; secondly, the present application uses an attention mechanism to improve the performance in a specific field: the present application adjusts the attention-related weights of the visual Transformer in the self-supervised training stage, optimizes the global and local feature extraction ability of the model; the classification task of cervical precancerous lesion images is optimized through the self-supervised method, and the feature expression of the lesion under different views is further utilized through the multi-view fusion method, experiments show that the present application effectively solves the overfitting problem of the visual Transformer model on small data sets, making the model more suitable for fine-grained tasks in the medical image field; thirdly, the present application improves the auxiliary diagnosis ability of cervical cancer screening in low-resource areas: by using endoscopic and colposcopic image data, the present application realizes efficient auxiliary diagnosis for CIN classification, especially in areas with limited medical resources, the present application provides an efficient and low-cost solution for automated screening, helps doctors improve diagnostic accuracy and reduce the rate of missed diagnosis, and provides important technical support for early cervical cancer prevention.

Claims

1. A multi-view fusion cervical precancerous lesion image classification method based on self-supervised learning, characterized in that: The method comprises the following steps in sequence: (1) Collect video screenshot data of a prostate endoscopic resection operation and pre-process the same to obtain unannotated data, input the unannotated data into a self-supervised training framework to perform self-supervised training, optimize feature extraction, obtain optimized features, optimize the attention weight of a first visual Transformer model through the optimized features, and obtain optimized model weights; (2) Collect a patient's cervical precancerous lesion colposcopy image and pre-process the same to obtain annotated cervical precancerous lesion colposcopy acetic acid pictures and colposcopy iodine pictures, input the optimized model weights, the annotated cervical precancerous lesion colposcopy acetic acid pictures and the colposcopy iodine pictures into a second visual Transformer model and a third visual Transformer model respectively to perform fine-tuning training, take the optimized model weights as the initial weights of the second visual Transformer model and the third visual Transformer model during the fine-tuning training, and obtain the second visual Transformer model and the third visual Transformer model respectively; (3) Pre-process a cervical precancerous lesion colposcopy image to be classified, input the pre-processed cervical precancerous lesion colposcopy image into the second visual Transformer model and the third visual Transformer model respectively to perform hierarchical prediction, obtain cervical precancerous lesion colposcopy acetic acid picture classification results and cervical precancerous lesion colposcopy iodine picture classification results, and perform weighted fusion on the cervical precancerous lesion colposcopy acetic acid picture classification results and the cervical precancerous lesion colposcopy iodine picture classification results to obtain final classification results; In step (1), the inputting of the unannotated data into the self-supervised training framework to perform self-supervised training and optimize feature extraction to obtain optimized features is specifically: (1a) Construct a teacher network and a student network: Self-supervised learning is achieved by constructing a teacher network and a student network, and the two networks have the same architecture and process different views from the same image; (1b) Image view generation: two different views are generated after a same image is randomly cropped, scaled, brightness adjusted and color jittered, and are input into the teacher network and the student network respectively; Teacher network updating mechanism: the parameters of the teacher network are updated by exponential average movement, and the specific formula is: θ t ←ξθ t +(1-ξ)θ s where θ t are the teacher network parameters, θ s are the student network parameters, and ξ is a momentum parameter. Student network back propagation: the student network updates the student network weight parameters through a gradient back propagation algorithm; (1c) Design a loss function: The loss function of the response feature optimizes the model parameters by calculating the distribution difference between the outputs of the teacher network and the student network; Response features: Each response feature is normalized by softmax to get the probability distribution σ(z i ): where z i represents the i-th dimensional eigenvalue of the response feature; z j represents all the feature vectors traversed in the calculation of the softmax process, a total of M feature vectors; Self-supervised loss: loss function based on response features is defined as: where τ t and τ s denote the temperature parameters for the teacher and student networks, respectively, denotes a centering operation of the teacher network response features; f t denotes a softmax operation on the teacher network, f s denotes an operation on the student network parameters; z t denotes the teacher network feature vector, z s denotes the student network feature vector; The class token vector is denoted as w, the multi-layer perception projection operation is denoted as f, f(w) is denoted as the multi-layer perception projection operation on the class token feature; the global attention information brought by the class token feature is reserved, and the result after the vector splicing is used to guide the learning of the student network to obtain better optimization; Φ s The vector splicing operation is denoted as: Φ s (w) = concat[f(w), w] Finally, the loss function during the self-supervised training process is represented as: where F t denotes the computation of the feature vector for the teacher network, F s denotes the computation of the feature vector for the student network, Φ t (w) and Φ s (w) denote the concatenation operation of the parameters of the teacher and student networks, respectively, during the computation By The teacher network and the student network learn the distribution features obtained by two different view networks in the manner of cross-entropy loss.

2. The self-supervised learning based multi-view fusion cervical precancerous lesion image classification method according to claim 1, characterized in that: In step (1), the video screenshot data of the prostate endoscopic resection surgery is collected and preprocessed to obtain unannotated data, which specifically refers to collecting video screenshot data from prostate endoscopic resection surgery, removing abnormal images after screening to obtain screened images, the abnormal images including blurred images and overexposed images; the screened images are uniformly resized to 224*224, the intraoperative and endoscope unrelated areas are removed, and data enhancement processing such as rotation, flipping and Gaussian blur is performed to obtain unannotated data.

3. The self-supervised learning based multi-view fusion cervical precancerous lesion image classification method according to claim 1, characterized in that: In step (1), the specific steps of self-supervised training are as follows: (5a) Weight initialization: use the pre-trained Imagenet ViT-1k weight as the initialization weight; Imagenet ViT-1k refers to training a visual Transformer of 1k size on the Imagenet dataset; (5b) Parameter optimization: set the weight decay to 0.05, the batch size to 32, the learning rate to start from 5e-5 using the cosine scheduler, the training period to 10 epochs, and use the early stopping strategy and mixed precision training mechanism.

4. The self-supervised learning based multi-view fusion cervical precancerous lesion image classification method according to claim 1, characterized in that: In step (2), the cervical precancerous lesion colposcopy images of the patient are collected and preprocessed to obtain labeled cervical precancerous lesion colposcopy acetic acid pictures and iodine pictures, which specifically include the following steps in order: (2a) Data collection: collect 3294 colposcopy images of 966 patients to form a dataset, the colposcopy images including colposcopy acetic acid pictures and colposcopy iodine pictures of the same patient; the collected images are labeled as Normal, CIN1, CIN2 and CIN3 according to the pathological cervical intraepithelial neoplasia grading; (2b) Data division: divide the dataset into a training set and a test set in a ratio of 4:1, wherein the training set includes 2635 images and the test set includes 659 images; (2c) Data preprocessing: perform enhancement processing on the dataset, including random rotation, cropping, scaling, color transformation and brightness adjustment; uniformly adjust the size of the image to 224x224 pixels and perform standardization processing.

5. The self-supervised learning based multi-view fusion cervical precancerous lesion image classification method according to claim 1, characterized in that: In step (2), the specific steps of fine-tuning training are as follows: (6a) Initialization: use the optimized model weight to initialize the second visual Transformer model and the third visual Transformer model; (6b) Optimizer and parameter setting: use the AdamW optimizer, set the weight decay to 0.05, the batch size to 64, the learning rate to gradually decrease from 5e-3 to 5e-4, the training period to 200 rounds, and use a 20-round warm-up mechanism to accelerate the convergence of the second visual Transformer model and the third visual Transformer model.

6. The self-supervised learning based multi-view fusion cervical precancerous lesion image classification method according to claim 1, characterized in that: In step (3), the weighted fusion of the cervical precancerous lesion colposcope acetic acid picture classification result and the cervical precancerous lesion colposcope iodine picture classification result is specifically: adopting a multi-layer perception machine as a feature fusioner, directly splicing the class mark features of the classification results of the second visual Transformer model for the acetic acid picture and the class mark features of the classification results of the third visual Transformer model for the iodine picture as inputs, outputting a final classification result after weighted fusion by the multi-layer perception machine, and the calculation formula of the multi-layer perception machine is: y = sigma (W * concat [x1, x2] + b) Wherein, x1 is the class mark feature vector of the acetic acid picture generated by the second visual Transformer model, x2 is the class mark feature vector of the iodine picture generated by the third visual Transformer model, W is a weight matrix, b is a bias vector, sigma is a ReLU activation function, and y is a final classification result vector.

Images