Colposcope lesion detection model training method and system based on double decoding networks

Abstract

The invention discloses a colposcope lesion detection model training method and system based on a double-decoding network. The model training method comprises the following steps: acquiring an initial model and training data, wherein the initial model comprises an encoder, a decoder and a discriminator; inputting any piece of labeled data into a model to obtain a labeled prediction result, and calculating a supervised loss value based on a corresponding truth value label; semantic information under different scales is obtained in a feature engineering network dominated by an encoder-decoder; pixel-level prediction is carried out on the semantic information, and partial model parameters are updated according to the first cross entropy loss; freezing the feature engineering network weight, and updating partial model parameters according to second cross entropy loss by using deep semantic residual information; and iteratively updating parameters of the model based on the supervised loss value until convergence. Through the model training method provided by the invention, a pixel-based model with high discrimination capability on normal and abnormal images is realized.

Description

Technical Field

[0001] The present invention belongs to the technical field of machine learning, and in particular relates to a model training method for establishing semantic segmentation and classification of normal / lesioned colposcopic lesion detection based on a deep decoding network. Background Art

[0002] Semantic segmentation aims to assign a label to each pixel of a given image and is a fundamental task in the field of computer vision. Recently, it has been widely adopted in many robotic systems such as self-driving cars, robots, and medical fields.

[0003] Cervical cancer is one of the most common malignant tumors in women worldwide. According to statistics, there are 530,000 new cases and about 270,000 deaths each year worldwide. Since the 1950s, the "three-step cervical cancer screening" program has been widely used in clinical practice, allowing cervical cancer and precancerous lesions to be detected and treated early. Colposcopy is crucial in the early diagnosis of cervical cancer and precancerous lesions. At present, colposcopy mostly relies on a comprehensive colposcopy image description system, but due to the uneven level of colposcopy practitioners, the accuracy of colposcopy diagnosis varies greatly, which in turn affects the standardized diagnosis and treatment of cervical lesions, resulting in overtreatment or undertreatment.

[0004] The colposcopy image description system defines and scientifically classifies the image features of the boundaries, contours, morphology and other parts or tissues of the cervical epithelium and blood vessels, covering the standardized analysis of various signs such as acetowhite epithelium, mosaic, punctate blood vessels, and atypical blood vessels. It uses deep learning to perform colposcopy image segmentation and image feature recognition, and analyzes the image features of different lesion degrees to obtain standardized information on the development status of specific lesions. It has the potential to help beginners and improve the diagnostic quality of colposcopy and biopsy in the detection of cervical precancerous / cancerous lesions. Summary of the invention

[0005] In view of the shortcomings of the prior art, the present invention provides a model training method for establishing semantic segmentation and classification of normal / lesioned colposcopic lesion detection based on a deep decoding network. By mining the deep semantic features in the colposcopic images, an artificial intelligence model is implemented to detect and locate cervical lesions.

[0006] In order to achieve the aforementioned object of the invention, the present invention adopts the following scheme:

[0007] One aspect of the present invention provides a model training method for establishing semantic segmentation and classification of normal and pathological images based on a deep decoding network, comprising:

[0008] Step S1, obtaining an initial model and training data, wherein the initial model includes an encoder, a decoder and a discriminator; the encoder is used to encode retrospectively collected colposcopy image data into shallow features and deep features; the decoder is used to extract features from the training data and locate the lesion area; the discriminator is used for lesion detection; the training data includes labeled data and corresponding true value labels;

[0009] Step S2: input any of the labeled data into the model to obtain labeled prediction results, and calculate the supervised loss value based on the corresponding true value label; obtain semantic information at different scales in the feature engineering network dominated by the encoder-decoder;

[0010] Step S3, performing pixel-level prediction on the semantic information, and updating feature engineering weight parameters according to the first cross entropy loss;

[0011] Step S4, freezing the feature engineering network weights, using deep semantic residual information, and updating the decision network model parameters according to the second cross entropy loss;

[0012] Step S5: iteratively update the parameters of the model based on the supervised loss value until convergence.

[0013] Another aspect of the present invention provides a training system for establishing a normal and pathological semantic segmentation and classification model based on a deep decoding network, comprising:

[0014] A model and data acquisition module, the model and data acquisition module is used to acquire an initial model and training data, the initial model includes an encoder, a decoder and a discriminator; the encoder is used to encode retrospectively collected colposcopy image data into shallow features and deep features; the decoder is used to extract features from the training data and locate the lesion area; the discriminator is used for lesion detection; the training data includes labeled data and corresponding true value labels;

[0015] A semantic information acquisition module, which is used to input any of the labeled data into the model to obtain labeled prediction results, and calculate the supervised loss value based on the corresponding true value label; obtain semantic information at different scales in a feature engineering network dominated by an encoder-decoder;

[0016] A first model parameter updating module, wherein the first model parameter updating module is used to perform pixel-level prediction on the semantic information and update feature engineering weight parameters according to a first cross entropy loss;

[0017] A second model parameter updating module, wherein the second model parameter updating module is used to freeze the feature engineering network weights, and update the decision network model parameters according to the second cross entropy loss using the deep semantic residual information;

[0018] An iterative update module is used to iteratively update the parameters of the model based on the supervised loss value until convergence.

[0019] Another aspect of the present invention provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the steps in the model training method as described above are executed.

[0020] Another aspect of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program is executed in a computer, it causes the computer to execute the steps in the model training method as described above.

[0021] Compared with the prior art, the present invention has at least the following advantages: the model training method provided by the present invention combines deep decoding features and residual information to realize a pixel-based model with a high ability to distinguish normal and abnormal images. The model uses pooling features at different resolutions to capture rich contextual information, and captures clear object boundaries by gradually restoring spatial information. At the same time, it uses deep decoding features and uses residual layers to increase considerable depth to improve classification accuracy. When there is less experimental data, our model uses deep learning to predict lesions and areas, which is far higher than the level of colposcopy specialists with 5-10 years of work experience who have been trained in IFCPC terminology (sensitivity of about 65%-90%), indicating that the semantic segmentation technology of deep learning has good pixel segmentation accuracy in locating cervical lesion areas, can identify normal and cervical lesions, has advantages in segmenting areas, is instructive for assisting physicians in judging the condition, and also shows satisfactory accuracy in guiding the biopsy site. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0023] Figure 1 It is a flowchart of a model training method of a deep model provided by a typical implementation case of the present invention;

[0024] Figure 2It is a schematic diagram of the system structure and process of a model training method for a deep model provided in a typical implementation case of the present invention. DETAILED DESCRIPTION

[0025] In order to make the purpose, technical scheme and advantages of the present invention clearer, the specific embodiments of the present invention are described in detail below in conjunction with the accompanying drawings. Examples of these preferred embodiments are illustrated in the accompanying drawings. The embodiments of the present invention shown in the accompanying drawings and described according to the accompanying drawings are merely exemplary, and the present invention is not limited to these embodiments.

[0026] One aspect of the present invention provides a method for establishing a normal and pathological semantic segmentation and classification model training method based on a deep decoding network, comprising:

[0027] Step S1, obtaining an initial model and training data, wherein the initial model includes an encoder, a decoder and a discriminator; the encoder is used to encode retrospectively collected colposcopy image data into shallow features and deep features; the decoder is used to extract features from the training data and locate the lesion area; the discriminator is used for lesion detection; the training data includes labeled data and corresponding true value labels;

[0028] Step S2: input any of the labeled data into the model to obtain labeled prediction results, and calculate the supervised loss value based on the corresponding true value label; obtain semantic information at different scales in the feature engineering network dominated by the encoder-decoder;

[0029] Step S3, performing pixel-level prediction on the semantic information, and updating feature engineering weight parameters according to the first cross entropy loss;

[0030] Step S4, freezing the feature engineering network weights, using deep semantic residual information, and updating the decision network model parameters according to the second cross entropy loss;

[0031] Step S5: iteratively update the parameters of the model based on the supervised loss value until convergence.

[0032] It should be noted that the model training method is a model training method from full supervision to weak supervision. Professional physicians perform pixel-level segmentation and image-level classification on the collected colposcopy images of patients, use deep learning for semantic segmentation, and perform partial gradient update of feature engineering. The weights of the feature engineering network are frozen, and the deep decoding information features are retained. At the same time, the residual feature network is combined to perform weak supervision learning on the colposcopy images to detect whether there are lesions. The lesion detection information of the colposcopy images is predicted through deep decoding features and global information. For colposcopy lesion areas of different sizes, a multi-scale feature pyramid combined with self-attention is proposed The training module focuses on the lesions at different scales, solves the problem of insufficient semantic information, and obtains feature information of lesion areas of different sizes; a variety of hole rates are used to obtain a larger receptive field and contextual information, allowing the feature engineering model to focus on and fully learn the lesion information; the proposed multi-scale fusion module realizes information exchange across multi-scale representations, fuses the deep decoding features of colposcopy images from different scales, and fuses the lesion features of different scales to produce a more accurate, more discriminative and more robust lesion feature representation, thereby improving the performance of lesion segmentation detection and improving the performance of lesion detection in the decision classification network;

[0033] Specifically, the step S2 specifically includes: inputting any of the labeled data into a pre-trained feature extractor, the backbone features undergo 5 different operations of the feature pyramid and 1 global pooling operation to obtain 5 different outputs; convolution extracts local features, global pooling extracts global features, different void convolution rates obtain multi-scale features, and high-level features are obtained through a multi-scale self-attention mechanism and multi-scale feature fusion; low-level features are dimensionally adjusted through a 1x1 convolution, high-level features are upsampled 4 times, and low-level features are fused with high-level features as semantic information.

[0034] Specifically, the calculation formula of the dilated convolution is:

[0035]

[0036] Among them, x is the two-dimensional input feature, y is the mapping output, for each position i on the output feature map y and a convolution filter w, the calculation formula of the dilated convolution is applied, k is the convolution kernel size, and r is the dilation rate, which determines the step size of sampling the input signal.

[0037] Specifically, the multi-scale self-attention mechanism divides the feature maps under different scales of dilated convolution rates into patches with different receptive fields, and then uses a fully connected layer to embed the flattened vectors into the query embedding Q i In; a similar operation is used to obtain the key embedding K i The sum value is embedded in V i; where i represents different void convolution rates, and d is the dimension of K; the calculation formula of the attention matrix is ​​as follows:

[0038]

[0039] A multi-scale fusion block is used to realize information exchange across multi-scale representations. The fused output feature map is the sum of the transformed representations of 5 feature maps with different void convolution rates; feature mapping Fuse with other feature maps as follows:

[0040]

[0041] Among them, f * () represents the self-attention transformation at different scales, F (*) Represents a multi-scale feature map.

[0042] Specifically, the labeled prediction result includes the labeled deep result output by the decoder; the first cross entropy loss function is:

[0043]

[0044] Among them, M represents the number of categories, y c is a one-hot encoding vector, whose elements can take only two values: 0 and 1. If the category is the same as the sample category, the value is 1, otherwise it is 0. c Represents the probability that the predicted sample belongs to category c. When M=2, the first cross entropy loss function becomes a binary cross entropy loss function.

[0045] Another aspect of the present invention provides a system for training a normal and pathological semantic segmentation and classification model based on a deep decoding network, the system comprising:

[0046] A model and data acquisition module, the model and data acquisition module is used to acquire an initial model and training data, the initial model includes an encoder, a decoder and a discriminator; the encoder is used to encode retrospectively collected colposcopy image data into shallow features and deep features; the decoder is used to extract features from the training data and locate the lesion area; the discriminator is used for lesion detection; the training data includes labeled data and corresponding true value labels;

[0047] A semantic information acquisition module, which is used to input any of the labeled data into the model to obtain labeled prediction results, and calculate the supervised loss value based on the corresponding true value label; obtain semantic information at different scales in a feature engineering network dominated by an encoder-decoder;

[0048] A first model parameter updating module, wherein the first model parameter updating module is used to perform pixel-level prediction on the semantic information and update feature engineering weight parameters according to a first cross entropy loss;

[0049] A second model parameter updating module, wherein the second model parameter updating module is used to freeze the feature engineering network weights, and update the decision network model parameters according to the second cross entropy loss using the deep semantic residual information;

[0050] An iterative update module is used to iteratively update the parameters of the model based on the supervised loss value until convergence.

[0051] It should be noted that the model training system is a model training system from full supervision to weak supervision. Professional physicians perform pixel-level segmentation and image-level classification on the collected colposcopy images of patients, use deep learning for semantic segmentation, and perform partial gradient update of feature engineering. The weights of the feature engineering network are frozen, and the deep decoding information features are retained. At the same time, the residual feature network is combined to perform weak supervision learning on the colposcopy images to detect whether there are lesions. The lesion detection information of the colposcopy images is predicted through deep decoding features and global information. For colposcopy lesion areas of different sizes, a multi-scale feature pyramid combined with self-attention is proposed The training module focuses on the lesions at different scales, solves the problem of insufficient semantic information, and obtains feature information of lesion areas of different sizes; a variety of hole rates are used to obtain a larger receptive field and contextual information, allowing the feature engineering model to focus on and fully learn the lesion information; the proposed multi-scale fusion module realizes information exchange across multi-scale representations, fuses the deep decoding features of colposcopic images from different scales, and fuses the lesion features of different scales to produce a more accurate, more discriminative and more robust lesion feature representation, thereby improving the performance of lesion segmentation detection and improving the performance of lesion detection in the decision classification network.

[0052] Specifically, the semantic information acquisition module is specifically used to: input any of the labeled data into the pre-trained feature extractor respectively, and the backbone features undergo 5 different operations of the feature pyramid and 1 global pooling operation to obtain 5 different outputs; convolution extracts local features, global pooling extracts global features, and multi-scale features are obtained by different void convolution rates. High-level features are obtained through a multi-scale self-attention mechanism and multi-scale feature fusion; low-level features are dimensionally adjusted through a 1x1 convolution, and high-level features are upsampled 4 times, and low-level features are fused with high-level features as semantic information.

[0053] Specifically, the multi-scale self-attention mechanism divides the feature maps under different scales of dilated convolution rates into patches with different receptive fields, and then uses a fully connected layer to embed the flattened vectors into the query embedding Q iIn; a similar operation is used to obtain the key embedding K i The sum value is embedded in V i ; where i represents different void convolution rates, and d is the dimension of K; the calculation formula of the attention matrix is ​​as follows:

[0054]

[0055] A multi-scale fusion block is used to realize information exchange across multi-scale representations. The fused output feature map is the sum of the transformed representations of 5 feature maps with different void convolution rates; feature mapping Fuse with other feature maps as follows:

[0056]

[0057] Among them, f * () represents the self-attention transformation at different scales, F (*) Represents a multi-scale feature map.

[0058] Specifically, the present invention provides a method for establishing a normal and pathological semantic segmentation and classification model training method based on a deep decoding network, which includes:

[0059] An initial model and training data are provided, wherein the initial model includes an encoder, a decoder and a discriminator, wherein the encoder is used to encode input data into shallow features and deep features, the decoder is used to extract features from the training data, the decoder is used to locate the lesion area, and the discriminator is used to detect cervical lesions.

[0060] The training data was collected retrospectively, and the colposcopic images and basic information files recorded by the colposcopic instruments in the eastern campus of Li Huili Hospital, Ningbo Medical Center from January 2018 to December 2021 were collected, including the patient's age, TCT, and HPV results. The colposcopy examination used the routine 3% acetic acid test and 5% Lugol's solution iodine staining procedure. Multiple-point biopsies were performed on abnormal areas of the colposcopic image. Random biopsies were performed for those with no abnormalities in the colposcopic image, and endocervical curettage was performed in the type III transformation zone. The biopsy tissue was sent to the pathology department for examination, and the pathologist made a diagnosis after reading the film. The pathological diagnosis results were divided into normal or inflammatory cervix, LSIL, HSIL, and carcinoma (including microinvasive carcinoma and invasive carcinoma) according to the 2012 edition of the terminology for squamous epithelial lesions of the lower genital tract and anus. At the same time, those with no pathological diagnosis results and incomplete data were excluded.

[0061] Includes labeled data and its corresponding true value labels;

[0062] Input the labeled data into the initial model to obtain a labeled prediction result, and calculate a supervised loss value based on the corresponding true value label;

[0063] The parameters of the initial model are updated based on the supervised loss value until an end condition is met.

[0064] Specifically, the present invention also provides a normal and pathological semantic segmentation and classification model training system based on a deep decoding network, which includes:

[0065] Provide an initial model and training data. The initial model includes an encoder, a decoder and a discriminator. The encoder is used to encode the input data into shallow features and deep features. The decoder is used to extract features from the training data. The decoder is used to locate the lesion area. The discriminator is used to detect cervical lesions. Image preprocessing uses data enhancement to diversify the experimental data, simulate the real situation, and retain the data imbalance problem between samples. The real lesion annotation is retained, and the annotated area is converted into a binary mask to adapt to the semantic segmentation task of the image. The overfitting problem of the model is tested by multiple multi-fold cross-validation methods, and the data is divided into a training set and a validation set according to 8:2. Based on the feature extractor of transfer learning, the Resnet101 pre-trained model trained on the open source natural image ImageNet dataset is selected. The last few layers of the network are fine-tuned with the goal of semantic segmentation of lesion area detection (LSIL, HSIL, cancer) to improve the generalization ability of the segmentation network. In the decision stage, the segmentation head is responsible for the target output lesion area, and the discriminant layer is responsible for combining the decoupled features and deep residual information to output the predicted label.

[0066] Specifically, the present invention also provides a normal and pathological semantic segmentation and classification model established by the above-mentioned deep decoding network.

[0067] Specifically, the present invention also provides an electronic device, which includes a memory and a processor, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the steps in the above-mentioned deep learning model training method are executed or the above-mentioned deep learning model is run.

[0068] Specifically, the present invention also provides a readable storage medium, which stores a computer program. When the computer program is executed by the processor, it executes the steps in the above-mentioned deep learning model training method, or the storage medium stores the above-mentioned semantic segmentation model.

[0069] The technical solution and working principle of the present invention will be further explained and illustrated in conjunction with the accompanying drawings and specific implementation examples.

[0070] See also Figure 1 and Figure 2 A method for training a normal and pathological semantic segmentation and classification model based on a deep decoding network includes the following steps:

[0071] An initial model and training data are provided, wherein the initial model includes an encoder, a decoder and a discriminator, wherein the encoder is used to encode input data into shallow features and deep features, the decoder is used to extract features from the training data, the decoder is used to locate the lesion area, and the discriminator is used to detect cervical lesions.

[0072] The training data was collected retrospectively, and the colposcopic images and basic information files recorded by the colposcopic instruments in the eastern campus of Li Huili Hospital, Ningbo Medical Center from January 2018 to December 2021 were collected, including the patient's age, TCT, and HPV results. The colposcopy examination used the routine 3% acetic acid test and 5% Lugol's solution iodine staining procedure. Multiple-point biopsies were performed on abnormal areas of the colposcopic image. Random biopsies were performed for those with no abnormalities in the colposcopic image, and endocervical curettage was performed in the type III transformation zone. The biopsy tissue was sent to the pathology department for examination, and the pathologist made a diagnosis after reading the film. The pathological diagnosis results were divided into normal or inflammatory cervix, LSIL, HSIL, and carcinoma (including microinvasive carcinoma and invasive carcinoma) according to the 2012 edition of the terminology for squamous epithelial lesions of the lower genital tract and anus. At the same time, those with no pathological diagnosis results and incomplete data were excluded.

[0073] Includes labeled data and its corresponding true value labels;

[0074] Input the labeled data into the initial model to obtain a labeled prediction result, and calculate a supervised loss value based on the corresponding true value label;

[0075] The parameters of the initial model are updated based on the supervised loss value until an end condition is met.

[0076] As some typical application examples of the above technical solution, the execution steps of the above method include, for example:

[0077] 1) Obtain basic models and training sets of encoders, decoders, and discriminators, wherein the training sets include labeled data and their corresponding true value labels.

[0078] 2) Input any of the labeled data into the pre-trained feature extractor, and the backbone features are subjected to 5 different operations of the feature pyramid and 1 global pooling operation to obtain 5 different outputs. Convolution extracts local features, global pooling extracts global features, and multi-scale features are obtained by different void convolution rates. High-level features are obtained through multi-scale self-attention mechanism and multi-scale feature fusion. The low-level features are dimensionalized by 1x1 convolution, and the high-level features are upsampled 4 times. The low-level features are fused with the high-level features as semantic information.

[0079] 3) Perform pixel-level prediction of semantic information and update some model parameters based on the cross entropy loss.

[0080] 4) Freeze the feature engineering network weights, use deep semantic residual information, and update some model parameters based on the cross entropy loss.

[0081] 5) Iteratively update the parameters of the model based on the supervised loss value.

[0082] Based on the above technical solution, as some typical application examples, the above model training method can be implemented using the following steps:

[0083] S1. Create and initialize the basic models of encoder, decoder and discriminator.

[0084] S2. Experimental data selected 800 normal images and 1370 lesion images. The experiment used five-fold cross validation to take the average value as the final result. Any of the labeled data was input into the pre-trained feature extractor. The backbone features were subjected to 5 different operations of the feature pyramid and 1 global pooling operation to obtain 5 different outputs. Convolution extracted local features, global pooling extracted global features, and high-level features were obtained through a multi-scale self-attention fusion module. The low-level features were dimensionalized by 1x1 convolution, and the high-level features were upsampled 4 times. The low-level features were fused with the high-level features as semantic information.

[0085] S3. Make pixel-level predictions for semantic information and update some model parameters based on the cross entropy loss.

[0086] S4. Freeze the feature engineering network weights, use deep semantic residual information, and update some model parameters based on cross entropy loss.

[0087] S5. Use gradient descent to update model parameters until convergence.

[0088] In the specific implementation, the region detection and lesion type identification use random number generation to arbitrarily extract data from different lesion images containing valid labels for algorithm training.

[0089] In more detail, in some embodiments, the calculation formula of the dilated convolution is:

[0090]

[0091] Define x as the two-dimensional input feature and y as the mapped output, and apply the formula for each position i on the output feature map y and a convolution filter w.

[0092] To detect local inconsistencies in the concatenated feature maps, we introduce a multi-scale self-attention mechanism. The multi-scale self-attention mechanism divides the feature maps under different scales of dilated convolution rates into patches with different receptive fields, and then we use a fully connected layer to embed the flattened vectors into the query embedding Q iWe use similar operations to get the key embedding K i The sum value is embedded in V i . Where i represents different void convolution rates. Note that the calculation process of the matrix is ​​as follows:

[0093]

[0094] In order to achieve information exchange across multi-scale representations, we use a multi-scale fusion block. The fused output feature map is the sum of the transformed representations of the five feature maps with different dilated convolution rates. Feature Map It is fused with other feature maps as follows, where f * () represents the self-attention transformation at different scales, F (* Represents a multi-scale feature map:

[0095]

[0096] In some embodiments, the calculation formula for the predicted label is:

[0097]

[0098] in, represents the predicted label; argmax represents the conversion from the predicted probability vector to the class label.

[0099] In some embodiments, it may specifically include:

[0100] The labeled prediction results include the labeled deep results output by the decoder; Focal loss is introduced as a pixel-level loss, and an adjustable parameter is introduced to alleviate the problem of class imbalance. It will make the difficult-to-classify samples, that is, samples with low prediction confidence, so that the model pays more attention to these difficult-to-classify samples. This mechanism helps to improve the classification effect of boundary samples, that is, samples near the boundary of positive and negative samples. The formula of the loss function is:

[0101]

[0102] Among them, p is the detection result, y is the true label, and the following transformation is made for the convenience of representation:

[0103]

[0104] Smooth L1 Loss is introduced. Compared with the traditional mean squared error loss function, Smooth L1 Loss can provide a more stable gradient when there are outliers or abnormal samples, and has a smaller gradient when the prediction error is small, thereby reducing the sensitivity to abnormal samples. The formula is:

[0105] Assume x is the numerical difference between the predicted box and the true value,

[0106]

[0107] The labeled prediction result includes the labeled deep result output by the decoder; a cross entropy loss function is used as the image level loss, and the loss function is designed as

[0108]

[0109] Among them, M represents the number of categories, y c is a one-hot encoding vector, whose elements can take only two values: 0 and 1. If the category is the same as the sample category, the value is 1, otherwise it is 0. c Represents the probability that the predicted sample belongs to category c. When M=2, the first cross entropy loss function becomes a binary cross entropy loss function.

[0110] The above contents are exemplary descriptions of various technical means in the implementation scheme of the present invention. In specific applications, a complete model training can be based on the above technical solutions and perform the actual complete steps:

[0111] (1) Image preprocessing: Data enhancement is performed by resizing, rotating, flipping, etc. to diversify the experimental data. In order to simulate the real situation, the data imbalance problem between samples is retained. The real lesion annotation is retained, and the annotated area is converted into a binary mask to adapt to the semantic segmentation task of the image.

[0112] (2) Five fold cross-validation methods were used to test the overfitting problem of the model, and the data was divided into a training set and a validation set at a ratio of 8:2. Based on the feature extractor of transfer learning, the Resnet101 pre-trained model trained on the open source natural image ImageNet dataset was selected, and the last few layers of the network were fine-tuned with the goal of semantic segmentation of lesion area detection (LSIL, HSIL, cancer). Experiments were conducted based on the DeepLabv3+ model to perform preliminary semantic segmentation on the image data.

[0113] (3) Fine-tune the segmentation network by grafting to improve the generalization ability of the segmentation network. The target output of the segmentation network is the lesion area (parameters: Batch = 4, Epoch = 20, Resnet101 backbone network parameter training). Use the trained parameters to predict the lesion area of ​​the validation set colposcopy images, obtain the algorithm prediction result map, compare it with the lesion area marked by the doctor, and perform gradient update.

[0114] (4) Freeze the weights of the segmentation network and train the discriminator. By only fine-tuning the discriminator, the network avoids the overfitting problem of a large number of weights in the segmentation network.

[0115] (5) Use gradient descent to update the model parameters until the model converges, thereby obtaining a model that can be used for lesion detection.

[0116] Specifically, step (1) is a common initialization step with a basic encoder, decoder and discriminator. In each iteration, the network parameters are updated using gradient descent. Step (2) is mainly for preprocessing the data. In step (3), the supervised loss value of the encoder-decoder is calculated using the cross entropy function. In step (4), the deep decoding features are used to increase the depth using the residual layer to improve the classification accuracy.

[0117] The second aspect of the embodiments of the present invention also provides a deep learning model obtained by training the normal and pathological semantic segmentation and classification model training method based on the deep decoding network provided in any of the above-mentioned embodiments.

[0118] The third aspect of an embodiment of the present invention also provides an electronic device, which includes a memory and a processor, wherein a computer program is stored in the memory, and when the computer program is run by the processor, the deep learning model provided in any of the above-mentioned embodiments is executed.

[0119] The fourth aspect of an embodiment of the present invention also provides a readable storage medium, which stores a computer program. When the computer program is executed by the processor, it executes the above-mentioned deep learning model provided by any of the above-mentioned implementation modes.

[0120] The key technical means of the above technical solution are: collecting data sets, using various cervical epithelial and vascular signs in the international colposcopy standardized terminology to perform pixel-level semantic annotation on images, using a pre-trained network based on transfer learning as a feature extractor, and building a two-stage decision network model that integrates image segmentation and image classification methods based on multi-scale deep decoding features. Through interdisciplinary cooperation, the deep learning network architectures of various classic semantic segmentation models were comprehensively compared, and the models were optimized in several aspects such as data preprocessing, data enhancement, backbone network selection and training, model hyperparameter optimization, and algorithm improvement to build a model.

[0121] It should be understood that although this specification is described according to implementation methods, not every implementation method contains only one independent technical solution. This narrative method of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions of each embodiment can also be appropriately combined to form other implementation methods that can be understood by those skilled in the art.

Claims

1. A model training method for establishing normal / lesion semantic segmentation and classification based on a deep decoding network, characterized in that: include: Step S1, obtaining an initial model and training data, wherein the initial model includes an encoder, a decoder and a discriminator; The encoder is used to encode the retrospectively collected colposcopy image data into shallow features and deep features; The decoder is used to extract features from the training data and locate the lesion area; the discriminator is used for lesion detection; the training data includes labeled data and corresponding true value labels; Step S2: input any of the labeled data into the model to obtain labeled prediction results, and calculate the supervised loss value based on the corresponding true value label; obtain semantic information at different scales in the feature engineering network dominated by the encoder-decoder; Step S3, performing pixel-level prediction on the semantic information, and updating feature engineering weight parameters according to the first cross entropy loss; Step S4, freezing the feature engineering network weights, using deep semantic residual information, and updating the decision network model parameters according to the second cross entropy loss; Step S5: iteratively update the parameters of the model based on the supervised loss value until convergence.

2. The model training method according to claim 1, characterized in that: The step S2 specifically includes: inputting any of the labeled data into a pre-trained feature extractor respectively, subjecting the backbone features to 5 different operations of the feature pyramid and 1 global pooling operation to obtain 5 different outputs; extracting local features by convolution, extracting global features by global pooling, obtaining multi-scale features by different void convolution rates, and obtaining high-level features by multi-scale self-attention mechanism and multi-scale feature fusion; adjusting the dimension of low-level features by 1x1 convolution, and performing 4-fold upsampling of high-level features, and fusing low-level features with high-level features as semantic information.

3. The model training method according to claim 2, characterized in that: The calculation formula of the dilated convolution is: Among them, x is the two-dimensional input feature, y is the mapping output, for each position i on the output feature map y and a convolution filter w, the calculation formula of the dilated convolution is applied, k is the convolution kernel size, and r is the dilation rate, which determines the step size of sampling the input signal.

4. The model training method according to claim 2, characterized in that: The multi-scale self-attention mechanism divides the feature maps under different scales of dilated convolution rates into patches with different receptive fields, and then uses a fully connected layer to embed the flattened vectors into the query embedding Q i In; a similar operation is used to obtain the key embedding K i The sum value is embedded in V i ; where i represents different void convolution rates, and d is the dimension of K; the calculation formula of the attention matrix is ​​as follows: A multi-scale fusion block is used to realize information exchange across multi-scale representations. The fused output feature map is the sum of the transformed representations of 5 feature maps with different void convolution rates; feature mapping Fuse with other feature maps as follows: Among them, f * () represents the self-attention transformation at different scales, F (*) Represents a multi-scale feature map.

5. The model training method according to claim 1, characterized in that: The labeled prediction result includes the labeled deep result output by the decoder; the first cross entropy loss function is: Among them, M represents the number of categories, y c is a one-hot encoding vector, whose elements can take only two values: 0 and 1. If the category is the same as the sample category, the value is 1, otherwise it is 0. c Represents the probability that the predicted sample belongs to category c. When M=2, the first cross entropy loss function becomes a binary cross entropy loss function.

6. A model training system for establishing normal / lesion semantic segmentation and classification based on a deep decoding network, characterized in that: The system comprises: A model and data acquisition module, the model and data acquisition module is used to acquire an initial model and training data, the initial model includes an encoder, a decoder and a discriminator; the encoder is used to encode retrospectively collected colposcopy image data into shallow features and deep features; the decoder is used to extract features from the training data and locate the lesion area; the discriminator is used for lesion detection; the training data includes labeled data and corresponding true value labels; A semantic information acquisition module, which is used to input any of the labeled data into the model to obtain labeled prediction results, and calculate the supervised loss value based on the corresponding true value label; obtain semantic information at different scales in a feature engineering network dominated by an encoder-decoder; A first model parameter updating module, wherein the first model parameter updating module is used to perform pixel-level prediction on the semantic information and update feature engineering weight parameters according to a first cross entropy loss; A second model parameter updating module, wherein the second model parameter updating module is used to freeze the feature engineering network weights, and update the decision network model parameters according to the second cross entropy loss using the deep semantic residual information; An iterative update module is used to iteratively update the parameters of the model based on the supervised loss value until convergence.

7. The model training system according to claim 6, characterized in that: The semantic information acquisition module is specifically used to: input any of the labeled data into the pre-trained feature extractor, and the backbone features are subjected to 5 different operations of the feature pyramid and 1 global pooling operation to obtain 5 different outputs; convolution extracts local features, global pooling extracts global features, different void convolution rates obtain multi-scale features, and high-level features are obtained through multi-scale self-attention mechanism and multi-scale feature fusion; The low-level features are dimensionally adjusted through 1x1 convolution, and the high-level features are upsampled 4 times. The low-level features are fused with the high-level features as semantic information.

8. The model training system according to claim 7, characterized in that: The multi-scale self-attention mechanism divides the feature maps under different scales of dilated convolution rates into patches with different receptive fields, and then uses a fully connected layer to embed the flattened vectors into the query embedding Q i In; a similar operation is used to obtain the key embedding K i The sum value is embedded in V i ; where i represents different void convolution rates, and d is the dimension of K; the calculation formula of the attention matrix is ​​as follows: A multi-scale fusion block is used to realize information exchange across multi-scale representations. The fused output feature map is the sum of the transformed representations of 5 feature maps with different void convolution rates; feature mapping Fuse with other feature maps as follows: Among them, f * () represents the self-attention transformation at different scales, F (*) Represents a multi-scale feature map.

9. An electronic device, characterized in that: It includes a memory and a processor, wherein the memory stores a computer program, and when the computer program is run by the processor, the steps of the model training method described in any one of Claims 15 are executed.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed in a computer, the computer is caused to execute the steps in the model training method described in any one of claims 1 to 5.

Images