Cervical image classification method and system based on wavelet transform multi-modal fusion

Abstract

The invention discloses a cervical image classification method and system based on wavelet transform multi-modal fusion, and the method comprises the steps: 1) constructing a multi-branch feature coding network, and respectively processing a cervical OCT image, a colposcope normal saline image, an iodine coating image and an acetic acid white image; 2) introducing wavelet transform to perform frequency domain decomposition on the multi-modal image, obtaining high-frequency and low-frequency components and designing a cross-frequency interaction mechanism to more effectively capture the structure and detail information of the image, and extracting each modal feature through a ResNet-18 backbone network; 3) designing a feature fusion module, performing channel stacking on the three colposcope features, compressing the three colposcope features, and cascading the three colposcope features with the OCT features to form fusion features; (4) feature dimension reduction is achieved through double full connection layers, a weighted voting multi-mode decision classification strategy is adopted, a five-fold cross validation strategy is adopted, and a weighted cross entropy loss function is combined to solve the problem of data imbalance, and (5) finally Softmax classification is carried out.

Description

TECHNICAL FIELD

[0001] The application provides a cervical image classification method based on wavelet transform multi-modal fusion, and belongs to the field of medical image analysis and computer-aided diagnosis. BACKGROUND

[0002] The procedure of clinical examination of cervical lesions generally follows the "three-step" procedure. First, the thinprep cytologic test (TCT) and human papillomavirus (HPV) test are performed during the initial screening. TCT can effectively improve the detection rate of abnormal cervical cells, but cannot determine the cause of the abnormal cells found; although the HPV test can determine the high-risk HPV types that cause cervical cancer, it cannot locate the cervical lesions caused thereby. Then, the colposcope is used for examination and lesion positioning during the initial diagnosis, but its effect is affected by many factors such as the skill and experience of the colposcope doctor, the size of the tissue sample, and the particularity of the colposcope image. Finally, the biopsy is taken and sent for pathological examination for diagnosis, but it is time-consuming and still has the possibility of missed diagnosis. Therefore, there is an urgent need for a non-invasive, efficient and intelligent cervical cancer screening and diagnosis technology.

[0003] The colposcope is a group of optical instruments that magnify and image the cervical tissue through an illumination light source, and the doctor makes a careful visual assessment of the lesions in the cervical region. During the examination, the doctor will first apply physiological saline, 3-5% dilute acetic acid and Lugol's iodine solution, observe and record the color change of the cervical epithelial cells, and collect images of the cervix. After applying 3-5% acetic acid, the epithelial cells in the precancerous lesion or early cancerous state will turn white, which is called aceto-whitening reaction, which usually lasts for 2 to 4 minutes. The detection using Lugol's iodine solution is based on the fact that normal squamous epithelial cells will turn dark brown or brown after applying iodine solution due to the presence of sugar. In contrast, the precancerous lesion area of cervical cancer lacks glycogen and does not absorb iodine, showing a strong yellow color. If suspicious lesions are found, the doctor will take samples from the suspected lesion area for pathological biopsy, which is referred to as biopsy. The biopsy result is often used as the "gold standard" for diagnosis in clinical practice, but the cost of biopsy is relatively high, requiring advanced equipment and professional pathologists to support, so colposcopy can help screen high-risk groups for further pathological examination and alleviate the shortage of medical resources. Cervical precancerous lesions, i.e. cervical intraepithelial neoplasia (CIN), cover three levels, with mild neoplasia being CIN1, moderate neoplasia being CIN2, and severe neoplasia being CIN3.

[0004] Optical coherence tomography (OCT) is a new biomedical imaging technology that uses near-infrared light to obtain micron-level cross-sectional images of biological tissues and can display cell features of a 2mm deep tissue sample in real time. At present, retrospective studies on ex vivo cervical tissues have shown that the use of OCT can effectively identify the morphological features of cervical tissues, including squamous epithelium, basement membrane, cyst, cervical stroma, gland, low-grade squamous intraepithelial lesions (LSIL), high-grade squamous intraepithelial lesions (HSIL), and cervical cancer (mainly squamous cell carcinoma), which makes it possible for OCT to be used as an important auxiliary tool for colposcopic "biopsy" to screen and diagnose cervical cancer.

[0005] At the same time, deep convolutional neural networks (CNNs) have achieved results not inferior to human experts in image (such as computed tomography, nuclear magnetic resonance, ultrasound, etc.) detection, segmentation, and other tasks of cancer or rare diseases. In the fields of ophthalmology, respiratory medicine, and orthopedics, computer-aided diagnosis methods based on the above technologies can help reduce the heavy and repetitive work of doctors, reduce human errors, and thus improve work efficiency. However, this deep learning-based medical image classification method is often considered a "black box" operation, which makes it difficult to provide doctors with corresponding diagnostic evidence (or medical evidence) such as pathological tissue morphological features or imaging texture features, thus limiting its application in clinical practice.

[0006] Specifically, some researchers have tried to use CNN to build a cervical tissue image classification model for the task of cervical lesion screening and diagnosis, and have achieved good classification results on the validation set. However, the above method still has the following problems: the existing CNN model has certain limitations in feature extraction, and the capture of subtle pathological changes and high-frequency details is not ideal. Existing fusion strategies such as shallow feature concatenation or pooling often fail to fully integrate the complementary and heterogeneous information contained in multiple imaging modalities. SUMMARY

[0007] The present application aims at the following problems existing in the prior art cervical tissue image classification method based on CNN: only relying on a single imaging mode, unable to cooperatively use the surface information of colposcopy and the cross-sectional microstructure of OCT; the traditional CNN is easy to ignore the high-frequency detail features which are crucial for early lesion diagnosis in the feature extraction process; the existing fusion strategy is difficult to fully fuse the cross-modal data correlation, and the present application provides a cervical image classification method and system based on wavelet transform multi-modal fusion. The wavelet transform can effectively capture the detail features of different scales and directions of the image including high-frequency edge information, which is complementary to the features of CNN, and various modal features are fully fused by weighted voting of the features of different modalities for classification.

[0008] To solve the above technical problems, the present application provides a cervical image classification method based on wavelet transform multi-modal fusion, comprising: S1: The obtained cervical tissue OCT image and colposcopy image are divided into a training set and a test set, and preprocessed, the cervical colposcopy image is detected and the region of interest (ROI) is cropped and preprocessed, the cervical colposcopy image and the OCT image of the same person are paired, and only appear in the training set or the test set; S2: A cervical image classification model based on multi-modal feature fusion is constructed, including a plurality of feature encoding branches, each feature encoding branch processes different types of colposcopy images and OCT images, including a wavelet transform module and a backbone network; wherein each feature encoding branch processes different types of cervical images, the wavelet transform module is used for frequency domain decomposition and feature enhancement of the image, and the backbone network extracts a cervical feature map; then the colposcopy image features and the OCT image features of different modalities are fused through a feature fusion module to obtain fused features; finally, a plurality of features are weighted and fused, and a prediction result is output through a Softmax layer; S3: Five-fold cross-validation is used, and a weighted cross-entropy loss function is set according to the negative-positive ratio in the data, the input data type and the model are adjusted, and a trained multi-modal cervical image classification model is obtained; S4: The trained multi-modal cervical image classification model is used to classify and predict the cervical images in the test set, and a classification result is obtained; In one embodiment, S1 specifically comprises: S1.1: The colposcopy image is subjected to data cleaning, low-quality images such as blurred images and focus errors of the colposcopy image caused by hand shaking of the examiner are screened, and it is ensured that complete three types of cervical colposcopy image data are included; S1.2: Train a colposcope cervical os detection model using the yolov5 network to locate the cervical os in the saline, iodine, and acetic white colposcope images, and crop the cervical os region to 512x512 pixels; S1.3: Match the cervical OCT image with the colposcope image and divide the data set; In an embodiment, S2 specifically comprises: S2.1: Use multiple branches to process the input cervical OCT image, cervical saline colposcope image, cervical iodine colposcope image, and cervical acetic white colposcope image, respectively, to obtain feature maps of different modalities; S2.2: In each branch, first apply wavelet transform to frequency domain decomposition of the branch input image to obtain high-frequency and low-frequency components, use a cross-frequency interaction mechanism to fuse the high-frequency and low-frequency components, and then use ResNet-18 for further feature extraction; S2.3: After the three colposcope branches, perform convolution fusion on the extracted three colposcope features to obtain three colposcope fusion features, and concatenate the three colposcope fusion features with the OCT features obtained from the OCT branch to obtain the fusion features of colposcope and OCT; S2.4: Add two fully connected layers after the above network structure; S2.5: Weighted voting of colposcope features, OCT features, and fusion features, dynamic allocation of decision weights, dynamic adjustment of the weights of single modality and multi-modality features (a single modality refers to the OCT image feature extracted by the OCT image feature extraction branch. Three colposcope image features (saline colposcope feature, acetic white colposcope feature, and iodine colposcope feature) are extracted by three colposcope image feature extraction branches. The colposcope fusion feature obtained after fusing the three colposcope features is another "single modality feature". Fusing the three colposcope fusion features with the OCT features is the fusion feature, which is the multi-modality feature), thereby improving the accuracy of classification, and finally setting a Softmax layer to output the prediction result.

[0009] In an embodiment, the multi-branch feature encoding has 4 branches, corresponding to the cervical OCT image, cervical saline colposcope image, cervical iodine colposcope image, and cervical acetic white colposcope image, respectively. Each branch first applies wavelet transform to extract frequency domain features, and then uses ResNet-18 to further extract features, and S2.2 specifically comprises: S2.2.1: Use Haar wavelet basis function to perform two-dimensional discrete wavelet transform on the input image of the branch to obtain high-frequency and low-frequency components; S2.2.2: Feature fusion of high-frequency and low-frequency components using cross-frequency interaction mechanism. Specifically, a channel spatial attention mechanism is used to highlight key information within a specific frequency band. Channel attention is used for low-frequency components, and spatial attention is used for high-frequency components. Subsequently, the low-frequency components and high-frequency components are concatenated.

[0010] S2.2.3: ResNet-18 is used to further extract features after concatenation, resulting in a 512-dimensional feature vector.

[0011] In an embodiment, the feature fusion module includes cascading, convolution fusion, two fully connected layers, and an activation layer. S2.3 specifically includes: S2.3.1: Stack the features of the three colposcopes along the channel number to obtain a preliminary fusion of 3x512-dimensional features.

[0012] S2.3.2: Perform further feature fusion on the 3 colposcope preliminary fusion features through a 1x1 convolution kernel, then perform dimension compression to eliminate redundant dimensions, and output a fused 512-dimensional colposcope feature vector; S2.3.3: Cascade the fused 512-dimensional colposcope feature vector with the 512-dimensional OCT feature vector extracted by the OCT branch to obtain the final colposcope and OCT fusion features, with a size of 1024 dimensions; In an embodiment, S3 specifically includes: S3.1: Normalize the scaled multi-modal input data; S3.2: Load ImageNet pre-trained weights to initialize ResNet-18; S3.3: Use a weighted cross-entropy loss function with a weight coefficient proportional to the sample ratio (e.g., if the positive and negative ratio in the data sample is 1:4, then the corresponding positive cross-entropy should be multiplied by a coefficient of 4 / 5, and the negative cross-entropy should be multiplied by 1 / 5 to balance the loss function's attention to different classes); We combine the loss function associated with the fused feature representation with the loss functions corresponding to the two single-modal networks on both sides, all using cross-entropy loss functions for loss calculation. The loss functions corresponding to the two single-modal networks on both sides help to train the single-modal feature extraction modules on both sides, thereby enhancing the network's ability to process single-modal information. The fused feature loss and the two independent single-modal feature losses are multiplied by 1-γ and γ respectively to obtain three corresponding loss values, where γ is a hyperparameter. Then the total loss function is calculated based on the linear combination of these losses. where is the colposcope and OCT fusion loss, is the three colposcope fusion losses, is the OCT loss, Total loss.

[0013] S3.4: Fine-tune all network parameters until the model converges.

[0014] S3.5: Using five-fold cross-validation, obtain the trained cervical image classification model, save the relevant parameter values.

[0015] In an embodiment, S4 specifically comprises: S4.1: Adjust the cervical image in the test set to the pixel size suitable for the backbone network; S4.2: Normalize the resized image; S4.3: Load the parameters of the trained cervical image classification model for prediction.

[0016] In an embodiment, the five-fold cross-validation strategy is used in S1 to ensure that the negative and positive sample ratios in each fold are consistent with the original data set, and the multi-modal images of the same subject only appear in the same fold.

[0017] In an embodiment, the backbone network in S2 adopts the standard ResNet-18 architecture, and the network configuration strictly follows the original design.

[0018] In an embodiment, the two fully connected layers added after the feature fusion module in S2 have dimensions of 256 and 2, respectively, to reduce the fusion features to 256 and 2.

[0019] In an embodiment, the loss function in S3 is a weighted cross-entropy loss function.

[0020] In an embodiment, when training the model in S3, the model initialization in S3 uses the ImageNet pre-trained ResNet-18 weights provided by the PyTorch framework, and fine-tunes on this basis, for example, using gradient descent to fine-tune all parameters.

[0021] In an embodiment, the programming language for implementing the classification model in S3 is Python, and the software tool used is PyTorch.

[0022] Based on the same inventive concept, another aspect of the present application provides a cervical image classification based on a wavelet transform multi-modal feature fusion convolutional neural network, comprising: The data set division module is configured to divide the obtained cervical OCT images, cervical colposcopy physiological saline images, cervical colposcopy iodine application images and cervical colposcopy acetic acid white images into a training set and a test set, wherein the cervical OCT images and the cervical colposcopy images of the same person need to be paired and only exist in the training set or the test set, and the proportion of negative samples to positive samples in the training set and the test set is consistent with the overall distribution. The classification model construction module is configured to construct a cervical image classification model based on multi-modal feature fusion, which includes multiple feature encoding branches, each feature encoding branch processes different types of colposcopy images and OCT images, and includes a wavelet transform module and a backbone network; the wavelet transform module is configured to perform frequency domain decomposition and feature enhancement on the input images, and the backbone network is further configured to extract feature maps; then the feature fusion module is used to fuse the colposcopy image features and the OCT image features of different modalities to obtain fused features; finally, multiple features are weighted and fused, and a prediction result is output through a Softmax layer. The training module adopts a weighted cross-entropy loss function, the weight is consistent with the sample ratio of positive samples to negative samples, the size of the OCT images and the colposcopy images in the training set is adjusted and normalized, and then the cervical image classification model is input for training to obtain a trained cervical image classification model. The test module is configured to use the trained cervical multi-modal image classification model to classify and predict the cervical multi-modal images in the test set, and use a multi-modal decision classification mechanism to integrate multiple modal information for final decision to obtain a classification result.

[0023] The cervical image classification method based on the wavelet transform multi-modal feature fusion convolutional neural network provided by the application introduces a multi-modal feature fusion mechanism into a convolutional neural network, which can better extract features of multi-modal cervical images, mainly including a multi-branch feature encoding and a feature fusion module. The multi-branch feature encoding is configured to perform feature encoding on multi-modal input data, so that the classification model can fully extract semantic position information of different cervical image information and deep features; the multi-branch encoding mechanism and the feature fusion mechanism improve the overall classification effect of the classification model, and solve the technical problems of poor classification effect and weak interpretability of existing methods. BRIEF DESCRIPTION OF DRAWINGS

[0024] In order to more clearly illustrate the technical solutions in the embodiments of the application or the prior art, the drawings needed in the embodiments or the prior art description will be briefly introduced below. Obviously, the drawings in the following description are some embodiments of the application, and those skilled in the art can also obtain other drawings according to these drawings without creative labor.

[0025] Figure 1A flowchart for realizing a cervical image classification method based on wavelet transform multi-modal fusion is provided. Figure 2 A cervical image classification model framework based on wavelet transform multi-modal feature fusion convolutional neural network in the embodiment of the application is provided. Figure 3 A multi-branch feature extraction structure in the embodiment of the application is provided. Figure 4 A feature fusion module structure in the embodiment of the application is provided. Figure 5 A training flowchart of the cervical image classification model based on wavelet transform multi-modal fusion in the embodiment of the application is provided. Figure 6 A test flowchart of the cervical image classification model based on wavelet transform multi-modal fusion in the embodiment of the application is provided. Figure 7 A structure block diagram of the cervical image classification system based on wavelet transform multi-modal fusion in the embodiment of the application is provided. DETAILED DESCRIPTION

[0026] The present application inventors find through a large amount of research and practice that: the existing basic model mainly uses single cervical OCT image data or single cervical colposcope data for classification, and due to the less features that can be extracted by the model from the single data, the classification effect is difficult to meet the requirements of clinicians.

[0027] Therefore, in view of the above problems, the application introduces a multi-branch feature coding mechanism and a multi-modal feature fusion mechanism to optimize the basic model, wherein the multi-branch feature coding mainly codes different cervical image extraction features into feature vectors, and the feature fusion module mainly fuses features of different modalities into final fusion features.

[0028] The overall inventive concept of the application is as follows: 1) Introducing a multi-branch feature coding mechanism into a convolutional neural network for better extracting cervical OCT image, cervical colposcope normal saline image, cervical colposcope iodine painting image, and cervical colposcope acetic acid white image features; 2) Introducing a wavelet transform mechanism to perform frequency domain decomposition on the image to enhance feature expression; 3) Introducing a feature fusion mechanism, first performing convolution fusion on the three colposcope images, and then performing splicing on the OCT image to obtain cervical fusion features; 4) The fusion features are reduced to 256 dimensions through a first full connection layer, and then reduced to 2 dimensions through a layer of full connection layer, and finally classified using a Softmax function; 5) Using five-fold cross-validation to optimize the model, setting a weighted cross-entropy loss function combined with data class distribution to improve classification robustness, and using a multi-modal decision mechanism for final classification; In order to make the purposes, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are some but not all of the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0029] Embodiment one The present embodiment provides a cervical image classification method based on wavelet transform multi-modal fusion, please refer to Figure 1 , which comprises the following steps: S1: The obtained cervical tissue OCT image and colposcope image are divided into a training set and a test set, and are preprocessed, the cervical colposcope image is detected and the region of interest (ROI) is cropped and preprocessed, the cervical colposcope image and the OCT image of the same person are paired, and only appear in the training set or the test set; Specifically, all the OCT images and all the colposcope images in the same group of cervical images only exist in the training set or the test set, that is, the OCT image, the physiological saline colposcope image, the iodine colposcope image and the acetic acid white colposcope image of the same object are either only used as the training set or only used as the test set. In the specific implementation process, the original OCT image is in tag image file format (TIFF) format, which conforms to the digital imaging and communications in medicine (DICOM) specification, and is converted into portable network graphics (PNG) format, and the colposcope image is in portable network graphics format. The method for dividing the training set and the test set in S1 is k-fold cross validation.

[0030] In the specific implementation process, the data set used in the present embodiment contains 118200 OCT images (PNG format, 120 for each patient) and 2955 colposcope images (PNG format, one physiological saline, one iodine and one acetic acid white for each patient) of 985 patients with cervical tissue collected from multiple hospitals, including low risk (negative 663) and high risk (positive 322).

[0031] To verify the effectiveness of the method of the application, the data set is divided according to the following method: in order to compare with the single modal based classification model, the data set is divided into 5 parts, and 4 of them are selected as the training set in turn, and the remaining 1 part is selected as the test set, and five-fold cross-validation is carried out. In order to make the test effect more convincing, it is ensured that the training set and the test set contain completely independent samples, that is, the OCT image and the colposcopy image belonging to the same patient cannot exist in the training set and the test set at the same time.

[0032] S2: Construct a cervical image classification model based on multi-modal feature fusion, including multiple feature encoding branches, each feature encoding branch processes different types of cervical images and OCT images, including a wavelet transform module and a backbone network, wherein the wavelet transform module is used for frequency domain decomposition and feature enhancement of the input image, and the backbone network further extracts feature maps; then the colposcopy image features and the OCT image features of different modalities are fused through a feature fusion module to obtain fused features; finally, multiple features are weighted and fused, and a prediction result is output through a Softmax layer; Specifically, S2 is the framework of building a classification model, mainly including a backbone network, a wavelet transform module and a feature fusion module.

[0033] As shown in the framework structure diagram of the cervical image classification model constructed in S2. Figure 2 In general, the application combines the classical CNN model with wavelet transform and multi-modal feature fusion mechanism, then adds two fully connected layers, and sets the output dimension of the classification model to 2 for the binary classification task of cervical tissue images, that is, low risk (including inflammation, no epithelial lesions and cysts) and high risk (including HSIL).

[0034] S3: The OCT images and colposcopy images in the training set are normalized and input into the cervical image classification model for training to obtain a trained multi-modal cervical image classification model. The loss function uses weighted cross entropy, the OCT images and colposcopy images in the training set are normalized and input into the cervical image classification model for training to obtain a trained cervical image classification model.

[0035] Specifically, S3 is to train the built model by using the training set and the loss function.

[0036] Further, in S3, the feature encoding modules of each branch are added with a wavelet transform module and a feature fusion module, and then a classifier is added. The main classifier includes a fully connected layer for reducing the fusion features to 256 dimensions, and a fully connected layer for reducing the fusion features to 2 dimensions. A weighted cross-entropy loss is used, and the weight is the ratio of negative and positive samples in the data set.

[0037] Further, in S3, the parameters in the ResNet-18 model pre-trained by ImageNet are loaded for initialization, and then fine-tuning is performed, for example, gradient descent is used to fine-tune all parameters.

[0038] S4: Using the trained cervical image classification model to classify and predict the cervical images in the test set, and using a multi-modal decision classification mechanism to make a final decision to obtain a classification result.

[0039] Specifically, S4 is to use the test set to test the trained model to obtain a prediction result. As an implementation, the classification in S4 uses a Softmax function.

[0040] The application provides a cervical image classification method based on wavelet transform multi-modal fusion. Based on a convolutional neural network architecture, multi-modal cervical features are extracted and fused, so that the details and semantic information of different modalities and the weights of different modal features can be better utilized, and accurate classification of cervical images can be realized.

[0041] In an implementation, the backbone network in S2 is ResNet-18.

[0042] In an implementation, the loss function in S3 is a weighted cross-entropy loss function, and the weight of the loss function is the ratio of negative and positive samples in the data set. The programming language for implementing the classification model in S3 is Python, and the software tool used is PyTorch.

[0043] Further, the size of the cervical image in S3 and S4 is adjusted to 512 pixels x 521 pixels, and then the pixels of the image are normalized as the input of the classification model.

[0044] In an implementation, S2 specifically includes: S2.1: Using a convolutional neural network as the backbone network of the classification model; S2.2: wavelet transform is added before the backbone network to extract the frequency domain features, each modality feature is frequency decomposed and enhanced, a multi-modal feature coding module is added after the backbone network, which is used to encode each branch feature map using a learnable residual encoding layer (Encoding Layer) to obtain the coded features of each modality; S2.3: a feature fusion module is added after the feature coding, three feature vectors of the cervical colposcope branch are first stacked and then convolution fused to obtain the colposcope feature, and the colposcope feature and the OCT feature are cascaded to obtain the fusion feature; S2.4: two fully connected layers are added after the above network structure; S2.5: a Softmax layer is set to output the prediction result.

[0045] Specifically, S2.1, since ResNet was proposed, its excellent performance in image classification has been recognized by the industry, therefore, the embodiment of the application uses ResNet-18 as the backbone network of the classification model to extract cervical image features, and combines wavelet transform and multi-modal feature fusion mechanism to improve the classification effect of the classification model.

[0046] S2.2, the classification layer in ResNet-18 is deleted as the backbone network, and a multi-modal feature coding mechanism and a feature fusion mechanism are added after it to capture more rich cervical image features. The specific design of the multi-modal feature coding module is shown in the attached Figure 3 In the multi-modal feature coding module, the texture primitive coding features of each modality feature are extracted, and a 512-dimensional feature vector is generated for each modality.

[0047] S2.3, a wavelet transform module and a feature fusion module are added in the backbone network, and the specific design is shown in the attached Figure 4 In the feature fusion module, the cervical colposcope normal saline image feature vector, the cervical colposcope iodine image feature vector, and the cervical colposcope acetic acid white image feature vector are stacked along the channel number, then convolution fusion is performed and 512 dimensions are compressed and restored to obtain the colposcope feature. The colposcope feature and the OCT feature are cascaded to obtain the cervical fusion feature. The colposcope feature, the OCT feature, and the fusion feature are weighted and voted.

[0048] S2.4, two fully connected layers are added after the above network structure, and batch normalization (batch normalization) is used after each layer.

[0049] In one embodiment, the multi-branch feature coding module has four branches, respectively corresponding to the cervical OCT image branch, the cervical colposcope normal saline image branch, the cervical colposcope iodine image branch, and the cervical colposcope acetic acid white image branch, and S2.2 specifically includes: S2.2.1: 1x1 convolution operation is performed on the original feature maps of each scale extracted by the backbone network to extract features; S2.2.2: Each branch feature map is encoded by a learnable residual coding layer to extract features of each modality, and sequentially passes through 4 stages (layer1-layer4) of residual blocks: 2 basic residual blocks (channel number 64), 2 down-sampling residual blocks (channel number 128), 2 down-sampling residual blocks (channel number 256), and 2 down-sampling residual blocks (channel number 512) to obtain each branch feature vector with a channel number of 512.

[0050] In the specific implementation process, the size of the original cervical OCT image is 256x256x120, and the size of the colposcope image is 256x256x1, which is the length, width and channel number respectively. The image is encoded by a learnable residual network, 2 basic residual blocks (channel number 64), 2 down-sampling residual blocks (channel number 128), 2 down-sampling residual blocks (channel number 256), and 2 down-sampling residual blocks (channel number 512). After encoding by the learnable residual coding layer in S2.2.2, 4 512-dimensional feature vectors are obtained.

[0051] In one embodiment, the feature fusion module includes convolution fusion, cascading, two fully connected layers, and an activation layer. S2.3 specifically includes: S2.3.1: The feature vectors of the cervical colposcope saline image, the cervical colposcope iodine image, and the cervical colposcope acetic acid white image are stacked along the channel number to obtain a preliminary fused 3x512-dimensional feature map; S2.3.2: Cross-column feature fusion is performed by a 1x1 convolution kernel to compress multiple column features into single column representation, and dimension compression operation is performed on the fused features to eliminate redundant dimensions, and a fused 512-dimensional colposcope feature vector is output; S2.3.3: The fused 512-dimensional colposcope feature vector and the 512-dimensional OCT feature vector are cascaded to obtain the final colposcope and OCT fusion feature with a size of 1024 dimensions; In one embodiment, S3 specifically includes: S3.1: The OCT images and colposcope images in the training set are adjusted to a pixel size suitable for the backbone network; S3.2: The size-adjusted images are normalized; S3.3: The cervical image classification model is trained using the normalized images, wherein the initialization uses the ResNet-18 model parameters loaded from ImageNet pre-training, the weighted cross-entropy loss function is used, the weights are all set to the sample ratio of positive and negative samples, and the parameters of the classification model are updated; We combine the loss function associated with the fused feature representation with the loss functions corresponding to the two unimodal networks, both using the cross-entropy loss function for loss calculation. The loss functions corresponding to the two unimodal networks help train the unimodal feature extraction modules on both sides, thereby enhancing the network's ability to process unimodal information. The fused feature loss and the two independent unimodal feature losses are multiplied by 1-γ and γ, respectively, to obtain three corresponding loss values, where γ is a hyperparameter. Then, the total loss function is calculated based on a linear combination of these losses. ,in The loss due to the fusion of colposcopy and OCT. Loss due to fusion of three colposcopy techniques. For OCT loss, This represents the total loss.

[0052] S3.4: Obtain the trained cervical image classification model and save the relevant parameter values.

[0053] Specifically, such as Figure 5 The diagram shows the flowchart of the training process. Since the size of the acquired cervical images is not the standard input size, the size of the images in the training set is resized before being input into the classification model for training. The specific implementation process is as follows: First, the original cervical OCT images are resized to 256 pixels × 256 pixels, which is acceptable to the model. Second, the pixel values ​​of the images are normalized by subtracting the mean and dividing by 255; the same process is applied to the colposcopy images. Then, the classification model is trained using these images (initialization uses parameters from a ResNet-18 model pre-trained on ImageNet), the objective function is optimized, and the parameters of the classification model are updated. Finally, the relevant parameter values ​​are saved after training.

[0054] In one implementation, S4 specifically includes: S4.1: Adjust the cervical images in the test set to a pixel size that is compatible with the backbone network; S4.2: Normalize the resized image; S4.3: Load the parameters of the trained cervical image classification model for prediction; Specifically, such as Figure 6 The diagram shows the flowchart of the testing process: The size of the cervical images in the test set is adjusted, input into the classification model, and the prediction results of the cervical images are obtained. Specifically, the process is as follows: First, the original cervical OCT image to be tested is adjusted to 256 pixels × 256 pixels; then, the pixel values ​​of the image are subtracted from the mean and divided by 255 for normalization; the same applies to colposcopy images; finally, the classification model built based on the relevant parameters stored in S3 is input, and the softmax function is used to obtain the corresponding prediction results (classification labels).

[0055] The beneficial effects of the present application are: on the one hand, loading ResNet-18 official pre-training weights for fine-tuning can utilize some commonly used feature extractors to reduce the training cost of the classification model; on this basis, fine-tuning can also more targetedly extract unique features in cervical images. On the other hand, the introduction of wavelet transform and multi-modal feature fusion mechanism enables the classification model to better extract detailed information, semantic position information of different modalities, and the weight of learning different modal features, and through a multi-modal decision mechanism, the information is integrated to make more accurate classification judgments, thereby improving the overall classification effect of the model.

[0056] Embodiment two Based on the same inventive concept, the present embodiment provides a cervical image classification system based on wavelet transform multi-modal fusion, please refer to Figure 7 , which comprises: A data set division module 201 is used for dividing the obtained cervical tissue OCT images into a training set and a test set, wherein the cervical tissue OCT images and the cervical colposcopy images are divided into different groups according to the objects, the OCT images and the colposcopy images in each group belong to the same object, each group of OCT images has a corresponding colposcopy image, and the OCT image and colposcopy image in the same group only exist in the training set or the test set; A classification model construction module 202 is used for constructing a cervical image classification model based on multi-modal feature fusion, which comprises a plurality of feature encoding branches, each feature encoding branch processes different types of colposcopy images and OCT images, and comprises a wavelet transform module and a backbone network, wherein the wavelet transform module is used for frequency domain decomposition and feature enhancement of the input image, and the backbone network further extracts feature maps; then the feature fusion module is used to fuse the colposcopy image features and the OCT image features of different modalities to obtain fused features; finally, a plurality of features are weighted and fused, and a Softmax layer is used to output a prediction result; wherein the feature fusion module fuses the cervical OCT image features, the cervical colposcopy normal saline image features, the cervical colposcopy iodine painting image features, the cervical colposcopy acetic acid white image features, the HPV and TCT embedded features to obtain the fused features; A training module 203 adopts a weighted cross-entropy loss function, the weight is the sample positive-negative ratio, adjusts the size of the OCT images and the colposcopy images in the training set, and inputs the OCT images into the OCT image classification model for training to obtain a trained cervical image classification model; A test module 204 is used for classifying and predicting the cervical multi-modal images in the test set by using the trained cervical multi-modal image classification model, and using a multi-modal decision classification mechanism to integrate the multi-modal information for final decision to obtain a classification result.

[0057] Since the system introduced in the embodiment two of the present application is the system used for implementing the cervical image automatic classification method based on the wavelet transform multi-modal feature fusion convolutional neural network in the embodiment one of the present application, the specific structure and deformation of the system can be understood by those skilled in the art based on the method introduced in the embodiment one of the present application, and thus will not be described here again. The system used by the method in the embodiment one of the present application belongs to the scope of the present application.

[0058] In the specific implementation process, the data set used by the embodiment of the present application contains 118200 OCT images (in PNG format, 120 images for each patient) and 2955 colposcopy images (in PNG format, one image for each of physiological saline, iodine and acetic acid for each patient) of 985 patients with cervical tissues collected from multiple hospitals, including low-risk (negative 663) and high-risk (positive 322), and the relevant statistical information is shown in Table 1.

[0059] Table 1 Information of cervical image data set used in the embodiment

[0060] In order to verify the effectiveness of the method of the present application, the data set is divided according to the following method: in order to compare with the classification model based on a single mode, the data set is divided into 5 parts, and 4 parts are selected in turn as the training set, and the remaining 1 part is selected as the test set, and five-fold cross-validation is performed. In order to make the test result more convincing, it is ensured that the training set and the test set contain completely independent samples, that is, the OCT images and the colposcopy images belonging to the same patient cannot exist in the training set and the test set at the same time.

[0061] In order to prove the effectiveness of the present application, using the data set shown in Table 1, the embodiment of the present application and the two commonly used single OCT data and three colposcopy data are compared by five-fold cross-validation, and the classification results are shown in Table 2. From Table 2, it can be seen that the method of the present application has better effect on the classification accuracy and the sensitivity index. The calculation formulas of the accuracy, the specificity and the sensitivity are as follows: Accuracy = (True Positive + True Negative) / (True Positive + False Positive + True Negative + False Negative) Sensitivity = True Positive / (True Positive + False Negative) Specificity = True Negative / (True Negative + False Positive) Table 2 Comparison of classification performance of the method of the present application and the benchmark method

[0062] While the preferred embodiments of the application have been described, additional variations and modifications can be made to these embodiments by those skilled in the art once they have the benefit of the present disclosure without departing from the spirit and scope of the application. Accordingly, it is intended that the appended claims include all such modifications and variations as fall within the scope of the present application.

[0063] It is apparent that those skilled in the art can make various changes and modifications to the embodiments of the application without departing from the spirit and scope of the application. Thus, it is intended that the present application include all such modifications and alterations insofar as they come within the scope of the appended claims and their equivalents.

Claims

1. A cervical image classification method based on wavelet transform multi-modal fusion, characterized in that, The method comprises the following steps: S1: divide the obtained cervical tissue OCT image and colposcope image into a training set and a test set, and perform preprocessing, detect and crop the region of interest of the cervical colposcope image, and pair the cervical colposcope image and the OCT image of the same person, and only appear in the training set or the test set; S2: construct a cervical image classification model based on multi-modal feature fusion, comprising multiple feature encoding branches, each feature encoding branch processes different types of colposcope images and OCT images, comprising a wavelet transform module and a backbone network, wherein the wavelet transform module is used for frequency domain decomposition and feature enhancement of the input image, and the backbone network further extracts feature maps; then the features of the colposcope image and the OCT image of different modalities are fused through a feature fusion module to obtain fused features; finally, multiple features are weighted and fused, and a Softmax layer is used to output the prediction result; S3: set a weighted cross-entropy loss function according to the negative-positive ratio in the data, adjust the input data type and the model, and obtain a trained multi-modal cervical image classification model; S4: use the trained multi-modal cervical image classification model to classify and predict the cervical images in the test set to obtain the classification result.

2. The wavelet transform based multi-modal fusion based cervical image classification method as claimed in claim 1, wherein: S1 specifically comprises: S1.1: data cleaning is performed on the colposcope image, low-quality images are screened out, and it is ensured that complete three types of cervical colposcope image data are included; S1.2: a cervical colposcope cervical orifice detection model is trained by using a yolov5 network to locate the cervical orifice in the saline, iodine painting and acetic acid white colposcope images, and the cervical orifice region is cropped to a certain pixel; S1.3: match the cervical OCT image with the colposcope image and divide the data set.

3. The wavelet transform based multi-modal fusion based cervical image classification method as claimed in claim 1, wherein: S2 specifically comprises: S2.1: a plurality of branches are used to process the input cervical OCT image, cervical saline colposcope image, cervical iodine painting colposcope image and cervical acetic acid white colposcope image respectively to obtain feature maps of different modalities; S2.2: in each branch, first, wavelet transform is applied to frequency domain decomposition of the input image of the branch to obtain high-frequency and low-frequency components, and a cross-frequency interaction mechanism is used for feature fusion of the high-frequency and low-frequency components, and then ResNet-18 is used for further feature extraction; S2.3: after the three colposcope branches, the three colposcope features are stacked and convolved through a feature fusion module to obtain three colposcope fusion features, and the three colposcope fusion features are spliced with the OCT features obtained by the OCT branch to obtain the fusion features of the colposcope and the OCT; S2.4: two fully connected layers are added after the above network structure; S2.5: the colposcope features, the OCT features and the fusion features are weighted and voted, the decision weight is dynamically allocated, the weight of the single-modal feature, i.e. the colposcope feature, the OCT feature and the multi-modal feature, i.e. the fusion feature, is dynamically adjusted based on the modal confidence, so as to improve the accuracy of classification, and finally a Softmax layer is set to output the prediction result.

4. The wavelet transform based multi-modal fusion based cervical image classification method as claimed in claim 1, wherein: In S2.2, the Haar wavelet basis function is used for two-dimensional discrete wavelet transform of the input image of the branch to obtain high-frequency and low-frequency components.

5. The wavelet transform based multi-modal fusion based cervical image classification method as claimed in claim 1, wherein: The cross-frequency interaction mechanism is used for feature fusion of high-frequency and low-frequency components, including: using a channel spatial attention mechanism to highlight key information in a specific frequency band, using a channel attention mechanism for low-frequency components, and using a spatial attention mechanism for high-frequency components, and then performing a concatenation operation on the low-frequency components and the high-frequency components.

6. The wavelet transform based multi-modal fusion based cervical image classification method as claimed in claim 1, wherein: The feature fusion module of S2.3 specifically includes: S2.3.1: stack the features of the three colposcopes along the channel number to obtain a preliminary fused 3x512-dimensional feature; S2.3.2: perform 1x1 convolution kernel convolution on the three colposcope preliminary fusion features for further feature fusion, and then perform dimension compression to eliminate redundant dimensions, and output a fused 512-dimensional colposcope feature vector; S2.3.3: concatenate the fused 512-dimensional colposcope feature vector with the 512-dimensional OCT feature vector extracted by the OCT branch to obtain the final colposcope and OCT fusion feature, which is 1024-dimensional.

7. The wavelet transform based multi-modal fusion based cervical image classification method as claimed in claim 1, wherein: The total loss function is calculated in step S3: wherein is the fusion loss for colposcopy and OCT, is the three colposcopy fusion losses, is the OCT loss, is the total loss, wherein , , All loss calculations are performed using a cross-entropy loss function.

8. The wavelet transform based multi-modal fusion based cervical image classification method as claimed in claim 3, wherein: The two fully connected layers added after the feature fusion module reduce the dimensions of the fusion features to 256 and 2, respectively.

9. The wavelet transform based multi-modal fusion based cervical image classification method as claimed in claim 3, wherein: The initialization of the cervical image classification model uses the ImageNet pre-trained ResNet-18 weights provided by the PyTorch framework, and then uses gradient descent to fine-tune all parameters.

10. A cervical image classification system based on wavelet transform multi-modal fusion, characterized in that: a data set division module is used to divide the obtained cervical tissue OCT images and colposcope images into a training set and a test set, and to preprocess them, detect and crop the cervical colposcope images, and pair the cervical colposcope images and OCT images of the same person, which only appear in the training set or the test set; a classification model construction module is used to construct a cervical image classification model based on multi-modal feature fusion, including multiple feature encoding branches, each feature encoding branch processing different types of colposcope images and OCT images, including a wavelet transform module and a backbone network, wherein the wavelet transform module is used for frequency domain decomposition and feature enhancement of the input image, and the backbone network further extracts feature maps; then the colposcope image features and OCT image features of different modalities are fused through a feature fusion module to obtain fused features; finally, multiple features are weighted and fused, and a Softmax layer is used to output a prediction result; a training module is used to set a weighted cross-entropy loss function according to the negative-positive ratio in the data, adjust the input data type and the model, and obtain a trained multi-modal cervical image classification model; a test module is used to classify and predict the cervical images in the test set using the trained multi-modal cervical image classification model to obtain a classification result.

Images