Cervical image classification method and system based on wavelet transform multi-modal fusion

Abstract

This invention discloses a cervical image classification method and system based on wavelet transform multimodal fusion, comprising: 1) constructing a multi-branch feature encoding network to process cervical OCT images, colposcopy saline images, iodine smear images, and acetic acid whitening images respectively; 2) introducing wavelet transform to perform frequency domain decomposition on the multimodal images, obtaining high-frequency and low-frequency components, and designing a cross-frequency interaction mechanism to more effectively capture the structural and detailed information of the images, and extracting the features of each modality through a ResNet-18 backbone network; 3) designing a feature fusion module to stack and compress the three colposcopy features, and then concatenate them with the OCT features to form fused features; 4) achieving feature dimensionality reduction through a double fully connected layer, adopting a weighted voting multimodal decision classification strategy, and employing a five-fold cross-validation strategy, combined with a weighted cross-entropy loss function to solve the data imbalance problem; 5) finally classifying using Softmax. This invention can achieve accurate classification of cervical images.

Description

Technical Field

[0001] This invention provides a cervical image classification method based on wavelet transform multimodal fusion, which belongs to the fields of medical image analysis and computer-aided diagnosis. Background Technology

[0002] Clinically, the examination of cervical lesions generally follows a three-step process. First, during initial screening, a thin-layer cytologic test (TCT) and a human papillomavirus (HPV) test are performed. TCT effectively improves the detection rate of abnormal cervical cells, but it cannot determine the cause of the abnormal cells. While the HPV test can identify high-risk HPV types that lead to cervical cancer, it cannot locate the resulting cervical lesions. Next, during the initial consultation, colposcopy is used to examine and locate the lesions, but its effectiveness is affected by many factors, including the skill and experience of the colpologist, the size of the tissue specimen, and the specific characteristics of the colposcopy image. Finally, a biopsy is taken for pathological examination for definitive diagnosis, but this is time-consuming and still carries the possibility of missed diagnoses. Therefore, there is an urgent need for a non-invasive, efficient, and intelligent cervical cancer screening and diagnostic technology.

[0003] A colposcope is a set of optical instruments that magnifies and images the cervical tissue through a light source, allowing doctors to carefully visually assess lesions in the cervical region. During the examination, the doctor will apply saline, 3-5% dilute acetic acid, and Lugol's iodine solution in sequence, observing and recording changes in the color of the cervical epithelial cells and acquiring images of the cervix. After applying 3-5% acetic acid, epithelial cells in precancerous or early-stage cancerous states will turn white; this is called the acetic acid whitening reaction, which usually lasts for 2 to 4 minutes. The use of Lugol's iodine solution is based on the fact that normal squamous epithelial cells, due to their sugar content, will appear dark brown or brown after applying iodine. Conversely, precancerous cervical lesions, lacking glycogen, do not absorb iodine and will appear a dark mustard yellow. If a suspicious lesion is found, the doctor will take a sample from the suspected lesion area for pathological biopsy. In clinical practice, biopsy results are often considered the "gold standard" for diagnosis. However, biopsies are expensive and require advanced equipment and professional pathologists. Therefore, colposcopy can help screen high-risk individuals for further pathological examination, alleviating the strain on medical resources. Cervical intraepithelial neoplasia (CIN) is a precancerous lesion of the cervix, encompassing three levels: mild CIN1, moderate CIN2, and severe CIN3.

[0004] Optical coherence tomography (OCT) is an emerging biomedical imaging technique that uses near-infrared light to obtain micron-level cross-sectional images of biological tissues and can display cellular characteristics of tissue samples up to 2 millimeters deep in real time. Currently, retrospective studies on ex vivo cervical tissue have shown that OCT can effectively identify morphological features of cervical tissue, including squamous epithelium, basement membrane, cysts, cervical stroma, glands, low-grade squamous intraepithelial lesions (LSIL), high-grade squamous intraepithelial lesions (HSIL), and cervical cancer (primarily squamous cell carcinoma). This makes OCT an important auxiliary tool for colposcopy-guided biopsies in screening and diagnosing cervical cancer.

[0005] Meanwhile, deep convolutional neural networks (CNNs) have achieved results comparable to or better than those of human experts in tasks such as image detection and segmentation for cancer or rare diseases (e.g., computed tomography, MRI, ultrasound). In medical fields such as ophthalmology, respiratory medicine, and orthopedics, computer-aided diagnostic methods based on these technologies help reduce doctors' heavy repetitive work, decrease human error, and thus improve work efficiency. However, this deep learning-based medical image classification method is often considered a "black box" operation, making it difficult to provide doctors with corresponding diagnostic criteria (or medical evidence), such as pathological tissue morphological characteristics or imaging texture features, thus limiting its clinical application.

[0006] Specifically, some researchers have attempted to use CNNs to build cervical tissue image classification models for cervical lesion screening and diagnosis, achieving good classification results on the validation set. However, these methods still have the following problems: existing CNN models have certain limitations in feature extraction, and are not ideal for capturing subtle pathological changes and high-frequency details. Existing fusion strategies such as shallow feature stitching or pooling often fail to fully integrate the complementary and heterogeneous information contained in multiple imaging modalities. Summary of the Invention

[0007] This invention addresses the following problems in existing CNN-based cervical tissue image classification methods: reliance on a single imaging modality, failing to collaboratively utilize the surface information from colposcopy and the cross-sectional microstructure from OCT; traditional CNNs tend to overlook high-frequency details crucial for early lesion diagnosis during feature extraction; and existing fusion strategies struggle to fully integrate cross-modal data correlations. We provide a cervical image classification method and system based on wavelet transform multimodal fusion. Wavelet transform effectively captures detailed features at different scales and directions, including high-frequency edge information, which complements the features of CNNs. Classification is achieved by weighted voting of features from different modalities to fully integrate various modal features.

[0008] To address the aforementioned technical problems, this invention provides, in one aspect, a cervical image classification method based on wavelet transform multimodal fusion, comprising:

[0009] S1: The acquired cervical tissue OCT images and colposcopy images are divided into training set and test set, and preprocessed. The cervical colposcopy images are detected and the Region of Interest (ROI) is cropped. The cervical colposcopy images and OCT images of the same person are paired and appear only in the training set or test set.

[0010] S2: Construct a cervical image classification model based on multimodal feature fusion, comprising multiple feature coding branches. Each feature coding branch processes different types of colposcopy and OCT images, including a wavelet transform module and a backbone network. Each feature coding branch processes different types of cervical images; the wavelet transform module performs frequency domain decomposition and feature enhancement on the images; and the backbone network extracts cervical feature maps. Then, the feature fusion module fuses the features of colposcopy images and OCT images from different modalities to obtain fused features. Finally, the multiple features are weighted and fused, and the prediction result is output through a Softmax layer.

[0011] S3: Five-fold cross-validation is used, and a weighted cross-entropy loss function is set according to the ratio of negative to positive in the data. The input data type and model are adjusted to obtain a well-trained multimodal cervical image classification model.

[0012] S4: Use the trained multimodal cervical image classification model to classify and predict the cervical images in the test set, and obtain the classification results;

[0013] In one implementation, S1 specifically includes:

[0014] S1.1: Perform data cleaning on colposcopy images, filter out low-quality images such as blurry or out-of-focus images caused by the examiner's shaky hands, and ensure that all images contain complete data of the three types of cervical colposcopy examination images.

[0015] S1.2: Use the YOLOv5 network to train a cervical os detection model for colposcopy to locate the cervix in saline, iodine smear, and acetic acid white colposcopy images, and crop the cervical os region to 512×512 pixels.

[0016] S1.3: Match cervical OCT images with colposcopy images and divide the dataset;

[0017] In one implementation, S2 specifically includes:

[0018] S2.1: Process the input cervical OCT image, cervical colposcopy saline image, cervical colposcopy iodine smear image, and cervical colposcopy acetic acid whitening image separately using multiple branches to obtain feature maps of different modalities;

[0019] S2.2: In each branch, wavelet transform is first applied to decompose the input image of the branch in the frequency domain to obtain high-frequency and low-frequency components. The cross-frequency interaction mechanism is used to fuse the features of the high-frequency and low-frequency components. Then, ResNet-18 is used for further feature extraction.

[0020] S2.3: After the three colposcopy branches, the extracted three colposcopy features are stacked and convolved to obtain the three colposcopy fusion features. The three colposcopy fusion features are then concatenated with the OCT features obtained from the OCT branch to obtain the fusion features of colposcopy and OCT.

[0021] S2.4: Add two fully connected layers after the above network structure;

[0022] S2.5: Weighted voting is applied to colposcopy features, OCT features, and fusion features. Decision weights are dynamically allocated, and the weights of single-modal and multi-modal features (a single modality refers to the OCT image features extracted by the OCT image feature extraction branch; the three colposcopy image feature extraction branches extract three colposcopy features (saline colposcopy features, acetic acid colposcopy features, and iodine-coated colposcopy features). The three colposcopy features are fused to obtain the colposcopy fusion feature, which refers to another "single-modal feature." The fusion feature of the three colposcopy features is then fused with the OCT features, which is the multi-modal feature) to improve classification accuracy. Finally, a Softmax layer is set to output the prediction results.

[0023] In one implementation, the multi-branch feature encoding has four branches, corresponding to the cervical OCT image, the cervical colposcopy saline image, the cervical colposcopy iodine smear image, and the cervical colposcopy acetic acid whitening image, respectively. Each branch first applies wavelet transform to extract frequency domain features, and then uses ResNet-18 to further extract features. S2.2 specifically includes:

[0024] S2.2.1: The high-frequency and low-frequency components are obtained by performing a two-dimensional discrete wavelet transform on the input image of this branch using the Haar wavelet basis function;

[0025] S2.2.2: Feature fusion of high-frequency and low-frequency components is performed using a cross-frequency interaction mechanism. Specifically, a channel spatial attention mechanism is used to highlight key information within a specific frequency band. The channel attention mechanism is applied to the low-frequency components, and the spatial attention mechanism is applied to the high-frequency components. Then, the low-frequency and high-frequency components are spliced ​​together.

[0026] S2.2.3: After concatenation, ResNet-18 is used to further extract features, resulting in a feature vector with 512 channels.

[0027] In one implementation, the feature fusion module includes cascaded convolutional fusion, two fully connected layers, and an activation layer. S2.3 specifically includes:

[0028] S2.3.1: Stack the features of the three colposcopes along the channel number to obtain a preliminary fused 3×512-dimensional feature.

[0029] S2.3.2: The three colposcopy features are initially fused and then further fused by convolution with a 1×1 kernel. Dimensional compression is then performed to eliminate redundant dimensions, and the fused 512-dimensional colposcopy feature vector is output.

[0030] S2.3.3: The fused 512-dimensional colposcopy feature vector is concatenated with the 512-dimensional OCT feature vector extracted from the OCT branch to obtain the final fused colposcopy and OCT feature vector, which has a size of 1024 dimensions.

[0031] In one implementation, S3 specifically includes:

[0032] S3.1: Scale and normalize the multimodal input data respectively;

[0033] S3.2: Load ImageNet pre-trained weights to initialize ResNet-18;

[0034] S3.3: Use the weighted cross-entropy loss function, with the weight coefficients being the ratio of positive to negative samples (for example, if the ratio of positive to negative samples in the data sample is 1:4, then when calculating the weighted cross-entropy loss, the corresponding positive cross-entropy should be multiplied by a coefficient of 4 / 5, and the negative cross-entropy by 1 / 5, in order to balance the attention of the loss function to different classes).

[0035] We combine the loss function associated with the fused feature representation with the loss functions corresponding to the two unimodal networks, both using the cross-entropy loss function for loss calculation. The loss functions corresponding to the two unimodal networks help train the unimodal feature extraction modules on both sides, thereby enhancing the network's ability to process unimodal information. The fused feature loss and the two independent unimodal feature losses are multiplied by 1-γ and γ, respectively, to obtain three corresponding loss values, where γ is a hyperparameter. Then, the total loss function is calculated based on a linear combination of these losses. ,in The loss due to the fusion of colposcopy and OCT. Loss due to fusion of three colposcopy techniques. For OCT loss, This represents the total loss.

[0036] S3.4: Fine-tune all network parameters until the model converges.

[0037] S3.5: Use five-fold cross-validation to obtain a trained cervical image classification model and save the relevant parameter values.

[0038] In one implementation, S4 specifically includes:

[0039] S4.1: Adjust the cervical images in the test set to a pixel size that is compatible with the backbone network;

[0040] S4.2: Normalize the resized image;

[0041] S4.3: Load the parameters of the trained cervical image classification model for prediction.

[0042] In one implementation, a five-fold cross-validation strategy is used in S1 to ensure that the proportion of negative and positive samples in each fold remains consistent with the original dataset, and that the multimodal images of the same subject appear only in the same fold.

[0043] In one implementation, the backbone network in S2 adopts the standard ResNet-18 architecture, and its network configuration strictly follows the original design.

[0044] In one implementation, the dimensions of the two fully connected layers added after the feature fusion module in S2 reduce the dimensionality of the fused features to 256 and 2, respectively.

[0045] In one implementation, the loss function in S3 is the weighted cross-entropy loss function.

[0046] In one implementation, during model training in S3, the model initialization in S3 uses ImageNet pre-trained ResNet-18 weights provided by the PyTorch framework, and then fine-tunes the model based on these weights, for example, by using gradient descent to fine-tune all parameters.

[0047] In one implementation, the programming language for implementing the classification model in S3 is Python, and the software tool used is PyTorch.

[0048] Based on the same inventive concept, another aspect of the present invention provides a cervical image classification method based on wavelet transform multimodal feature fusion convolutional neural network, including:

[0049] The dataset partitioning module is used to divide the acquired cervical OCT images, cervical colposcopy saline images, cervical colposcopy iodine smear images, and cervical colposcopy acetic acid white images into training sets and test sets. Among them, the cervical OCT images and cervical colposcopy images of the same person must be paired and exist only in the training set or the test set. The ratio of negative samples to positive samples in the training set and the test set is consistent with the overall distribution.

[0050] The classification model construction module is used to build a cervical image classification model based on multimodal feature fusion, which includes multiple feature coding branches. Each feature coding branch processes different types of colposcopy images and OCT images, including a wavelet transform module and a backbone network. The wavelet transform module is used to perform frequency domain decomposition and feature enhancement on the input image, and the backbone network further extracts feature maps. Then, the feature fusion module fuses the features of colposcopy images and OCT images from different modalities to obtain fused features. Finally, the multiple features are weighted and fused, and the prediction result is output through a Softmax layer.

[0051] The training module uses a weighted cross-entropy loss function, with the weights consistent with the positive-negative ratio of the samples. The sizes of the OCT images and colposcopy images in the training set are adjusted and normalized before being input into the cervical image classification model for training, resulting in a well-trained cervical image classification model.

[0052] The testing module is used to classify and predict cervical multimodal images in the test set using a trained cervical multimodal image classification model, and to integrate multiple modal information through a multimodal decision classification mechanism to make a final decision and obtain the classification result.

[0053] The cervical image classification method based on wavelet transform multimodal feature fusion convolutional neural network provided by this invention introduces a multimodal feature fusion mechanism into the convolutional neural network, which can better extract features of multimodal cervical images. It mainly includes a multi-branch feature encoding module and a feature fusion module. The multi-branch feature encoding is used to encode features of the multimodal input data, enabling the classification model to fully extract semantic location information of different cervical images and deep features. Based on the multi-branch encoding mechanism and the feature fusion mechanism, the overall classification performance of the classification model is improved, solving the technical problems of poor classification performance and weak interpretability in existing methods. Attached Figure Description

[0054] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0055] Figure 1 This is a schematic diagram illustrating the implementation process of a cervical image classification method based on wavelet transform multimodal fusion.

[0056] Figure 2 This is a schematic diagram of the cervical image classification model framework based on wavelet transform multimodal feature fusion convolutional neural network in an embodiment of the present invention;

[0057] Figure 3 This is a schematic diagram of the multi-branch feature extraction structure in an embodiment of the present invention;

[0058] Figure 4 This is a schematic diagram of the feature fusion module structure in an embodiment of the present invention;

[0059] Figure 5 This is a flowchart illustrating the training process of the cervical image classification model based on wavelet transform multimodal fusion in an embodiment of the present invention.

[0060] Figure 6 This is a test flowchart of the cervical image classification model based on wavelet transform multimodal fusion in an embodiment of the present invention;

[0061] Figure 7 This is a structural block diagram of the cervical image classification system based on wavelet transform multimodal fusion in an embodiment of the present invention. Detailed Implementation

[0062] Through extensive research and practice, the inventors of this application have discovered that existing basic models mainly rely on single cervical OCT image data or single cervical colposcopy data for classification. Because the models can extract fewer features from single data, the classification results are difficult to meet the requirements of clinicians.

[0063] Therefore, to address the above problems, this invention introduces a multi-branch feature encoding mechanism and a multi-modal feature fusion mechanism to optimize the basic model. The multi-branch feature encoding mainly extracts features from different cervical images and encodes them into feature vector representations. The feature fusion module mainly fuses the features of different modalities into the final fused features.

[0064] The overall inventive concept of this invention is as follows:

[0065] 1) A multi-branch feature encoding mechanism is introduced into the convolutional neural network to better extract features from cervical OCT images, cervical colposcopy saline images, cervical colposcopy iodine smear images, and cervical colposcopy acetic acid whitening images; 2) A wavelet transform mechanism is introduced to decompose the images in the frequency domain to enhance feature representation; 3) A feature fusion mechanism is introduced, firstly performing convolutional fusion on the three colposcopy images, and then stitching them with the OCT image to obtain cervical fusion features; 4) The fused features are reduced to 256 dimensions after passing through the first fully connected layer, and then reduced to 2 dimensions after passing through another fully connected layer, and finally classified using the Softmax function; 5) A five-fold cross-validation model is adopted to optimize the model, and a weighted cross-entropy loss function is set according to the data category distribution to improve classification robustness, and a multimodal decision mechanism is used for final classification;

[0066] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0067] Example 1

[0068] This embodiment provides a cervical image classification method based on wavelet transform multimodal fusion. Please refer to [link to relevant documentation]. Figure 1 The method includes:

[0069] S1: The acquired cervical tissue OCT images and colposcopy images are divided into training and testing sets and preprocessed. The cervical colposcopy images are detected and the Region of Interest (ROI) is cropped. The cervical colposcopy images and OCT images of the same person are paired and appear only in the training or testing set.

[0070] Specifically, all OCT images and all colposcopy images in the same set of cervical images exist only in the training set or the test set. This means that OCT images, saline colposcopy images, iodine-coated colposcopy images, and acetic acid-coated colposcopy images of the same object are either used only in the training set or only in the test set. In the specific implementation process, the original OCT images are in tag image file format (TIFF) format, conforming to the digital imaging and communications in medicine (DICOM) standard, and are converted to portable network graphics (PNG) format. Colposcopy images are also in portable network graphics format. The method for dividing the training set and the test set in S1 is k-fold cross-validation.

[0071] In the specific implementation process, the dataset used in this embodiment of the invention includes 118,200 OCT images (PNG format, 120 images per patient) and 2,955 colposcopy images (PNG format, one each of saline, iodine smear, and acetic acid whitening per patient) of 985 patients with cervical tissue collected from multiple hospitals, including low-risk (663 negative) and high-risk (322 positive).

[0072] To verify the effectiveness of the method of this invention, the dataset was divided as follows: For comparison with a single-modality classification model, the dataset was divided into five parts, with four parts selected as the training set and the remaining part as the test set for five-fold cross-validation. To make the test results more convincing, it was ensured that the samples in the training and test sets were completely independent during the data partitioning process; that is, OCT images and colposcopy images belonging to the same patient could not coexist in both the training and test sets.

[0073] S2: Construct a cervical image classification model based on multimodal feature fusion, comprising multiple feature coding branches. Each feature coding branch processes different types of cervical images and OCT images, including a wavelet transform module and a backbone network. The wavelet transform module is used to perform frequency domain decomposition and feature enhancement on the input image, and the backbone network further extracts feature maps. Then, the feature fusion module fuses the features of colposcopy images and OCT images from different modalities to obtain fused features. Finally, the multiple features are weighted and fused, and the prediction result is output through a Softmax layer.

[0074] Specifically, S2 is the framework for building classification models, which mainly includes a backbone network, a wavelet transform module, and a feature fusion module.

[0075] like Figure 2 The diagram shows the framework structure of the cervical image classification model in S2. In summary, this invention combines the classic CNN model with wavelet transform and multimodal feature fusion mechanisms, then adds two fully connected layers, and sets the output dimension of the classification model to 2 for binary classification of cervical tissue images, i.e., low risk (including inflammation, aepithelial lesions, and cysts) and high risk (including HSIL).

[0076] S3: Normalized OCT and colposcopy images from the training set are input into the cervical image classification model for training, resulting in a trained multimodal cervical image classification model. The loss function used is weighted cross-entropy.

[0077] Specifically, S3 uses a training set and a loss function to train the built model.

[0078] Furthermore, during training in S3, wavelet transform and feature fusion modules are added after the feature encoding modules of each branch, and then a classifier is added. The main classifier includes a fully connected layer that reduces the dimensionality of the fused features to 256 dimensions and a fully connected layer that reduces the dimensionality of the fused features to 2 dimensions. Weighted cross-entropy loss is used, with the weights being the proportion of negative and positive samples in the dataset.

[0079] Furthermore, during training in S3, the parameters of a ResNet-18 model pre-trained on ImageNet are loaded for initialization, and then fine-tuning is performed on this basis, for example, by using gradient descent to fine-tune all parameters.

[0080] S4: Use the trained cervical image classification model to classify and predict the cervical images in the test set, and use a multimodal decision classification mechanism to make the final decision and obtain the classification result.

[0081] Specifically, S4 uses a test set to perform prediction tests on the trained model and obtain the prediction results. As one implementation method, the classification in S4 uses the Softmax function.

[0082] This invention provides a cervical image classification method based on wavelet transform multimodal fusion. Based on a convolutional neural network architecture, it extracts and fuses multimodal cervical features, thereby better utilizing the detailed and semantic information of different modalities and learning the weights of different modal features to achieve accurate classification of cervical images.

[0083] In one implementation, the backbone network in S2 is ResNet-18.

[0084] In one implementation, the loss function in S3 is a weighted cross-entropy loss function, and the weight of the loss function is the ratio of negative samples to positive samples in the dataset. The programming language for implementing the classification model in S3 is Python, and the software tool used is PyTorch.

[0085] Furthermore, the size of the cervical images in S3 and S4 was adjusted to 512 pixels × 521 pixels, and then the pixels of the images were normalized before being used as input to the classification model.

[0086] In one implementation, S2 specifically includes:

[0087] S2.1: Using a convolutional neural network as the backbone network of the classification model;

[0088] S2.2: Add wavelet transform to extract frequency domain features before the backbone network, perform frequency decomposition and enhancement on each modality feature, and add a multimodal feature encoding module after the backbone network to encode the feature maps of each branch using a learnable residual coding layer (EncodingLayer) to obtain the features encoded by each modality.

[0089] S2.3: After feature encoding, a feature fusion module is added. The three feature vectors of the cervical colposcopy branch are first superimposed and then convolved to obtain colposcopy features. The colposcopy features are concatenated with OCT features to obtain fused features.

[0090] S2.4: Add two fully connected layers after the above network structure;

[0091] S2.5: Set up a Softmax layer to output the prediction results.

[0092] Specifically

[0093] S2.1 Since ResNet was proposed, its excellent performance in image classification has been recognized by the industry. Therefore, in this embodiment of the invention, ResNet-18 is used as the backbone network of the classification model to extract cervical image features, and wavelet transform and multimodal feature fusion mechanism are combined to improve the classification effect of the classification model.

[0094] S2.2, the classification layer in ResNet-18 was removed as the backbone network, and a multimodal feature encoding mechanism and a feature fusion mechanism were added after it to capture richer cervical image features. The specific design of the multimodal feature encoding module is shown in the appendix. Figure 3 In the multimodal feature encoding module, the features encoded by the texture primitives of each modality are extracted, and a 512-dimensional feature vector is generated for each modality.

[0095] S2.3, a wavelet transform module and a feature fusion module are added to the backbone network. See the appendix for their specific design. Figure 4 In the feature fusion module, the feature vectors of the cervical colposcopy saline image, the cervical colposcopy iodine smear image, and the cervical colposcopy acetic acid whitening image are stacked along the channel number, then convolved and fused, and compressed to restore 512 dimensions to obtain the colposcopy features. The colposcopy features are concatenated with OCT features to obtain the cervical fusion features, and the colposcopy features, OCT features, and fusion features are then weighted and fused using voting.

[0096] S2.4, add two fully connected layers after the above network structure, and use batch normalization after each layer.

[0097] In one implementation, the multi-branch feature encoding module has four branches, corresponding to the cervical OCT image branch, the cervical colposcopy saline image branch, the cervical colposcopy iodine smear image branch, and the cervical colposcopy acetic acid whitening image branch, respectively. S2.2 specifically includes:

[0098] S2.2.1: Use convolutional layers to perform 1×1 convolution operations on the original feature maps of each scale extracted from the backbone network to extract features;

[0099] S2.2.2: The feature maps of each branch are encoded by a learnable residual coding layer to extract the features of each modality. The feature vectors of each branch with 512 channels are obtained by passing them through four stages (layer1-layer4): 2 basic residual blocks (64 channels), 2 downsampled residual blocks (128 channels), 2 downsampled residual blocks (256 channels), and 2 downsampled residual blocks (512 channels).

[0100] In the specific implementation process, the original cervical OCT image size is 256×256×120, and the colposcopy image size is 256×256×1, representing the length, width, and number of channels, respectively. The images are feature-encoded using a learnable residual network, resulting in 2 basic residual blocks (64 channels), 2 downsampled residual blocks (128 channels), 2 downsampled residual blocks (256 channels), and 2 downsampled residual blocks (512 channels). After encoding by the learnable residual coding layer in S2.2.2, four 512-dimensional feature vectors are obtained.

[0101] In one implementation, the feature fusion module includes convolutional fusion, cascading, two fully connected layers, and an activation layer. S2.3 specifically includes:

[0102] S2.3.1: Stack the feature vectors of the saline image, the iodine smear image, and the acetic acid whitening image of the cervical colposcopy along the channel number to obtain a preliminary fused 3×512-dimensional feature map;

[0103] S2.3.2: Cross-column feature fusion is performed using a 1×1 convolution kernel, compressing multi-column features into a single-column representation. The fused features are then subjected to dimensionality compression to eliminate redundant dimensions, and the fused 512-dimensional colposcopy feature vector is output.

[0104] S2.3.3: Concatenate the fused 512-dimensional colposcopy feature vector with the 512-dimensional OCT feature vector to obtain the final fused colposcopy and OCT feature vector, which has a size of 1024 dimensions;

[0105] In one implementation, S3 specifically includes:

[0106] S3.1: Adjust the OCT images and colposcopy images in the training set to a pixel size that is compatible with the backbone network;

[0107] S3.2: Normalize the resized image;

[0108] S3.3: Train a cervical image classification model using the normalized images. Initialization involves loading the parameters of a ResNet-18 model pre-trained on ImageNet, using a weighted cross-entropy loss function, with all weights set to the ratio of positive to negative samples, and updating the parameters of the classification model.

[0109] We combine the loss function associated with the fused feature representation with the loss functions corresponding to the two unimodal networks, both using the cross-entropy loss function for loss calculation. The loss functions corresponding to the two unimodal networks help train the unimodal feature extraction modules on both sides, thereby enhancing the network's ability to process unimodal information. The fused feature loss and the two independent unimodal feature losses are multiplied by 1-γ and γ, respectively, to obtain three corresponding loss values, where γ is a hyperparameter. Then, the total loss function is calculated based on a linear combination of these losses. ,in The loss due to the fusion of colposcopy and OCT. Loss due to fusion of three colposcopy techniques. For OCT loss, This represents the total loss.

[0110] S3.4: Obtain the trained cervical image classification model and save the relevant parameter values.

[0111] Specifically, such as Figure 5The diagram shows the flowchart of the training process. Since the size of the acquired cervical images is not the standard input size, the size of the images in the training set is resized before being input into the classification model for training. The specific implementation process is as follows: First, the original cervical OCT images are resized to 256 pixels × 256 pixels, which is acceptable to the model. Second, the pixel values ​​of the images are normalized by subtracting the mean and dividing by 255; the same process is applied to the colposcopy images. Then, the classification model is trained using these images (initialization uses parameters from a ResNet-18 model pre-trained on ImageNet), the objective function is optimized, and the parameters of the classification model are updated. Finally, the relevant parameter values ​​are saved after training.

[0112] In one implementation, S4 specifically includes:

[0113] S4.1: Adjust the cervical images in the test set to a pixel size that is compatible with the backbone network;

[0114] S4.2: Normalize the resized image;

[0115] S4.3: Load the parameters of the trained cervical image classification model for prediction;

[0116] Specifically, such as Figure 6 The diagram shows the flowchart of the testing process: The size of the cervical images in the test set is adjusted, input into the classification model, and the prediction results of the cervical images are obtained. Specifically, the process is as follows: First, the original cervical OCT image to be tested is adjusted to 256 pixels × 256 pixels; then, the pixel values ​​of the image are subtracted from the mean and divided by 255 for normalization; the same applies to colposcopy images; finally, the classification model built based on the relevant parameters stored in S3 is input, and the softmax function is used to obtain the corresponding prediction results (classification labels).

[0117] The beneficial effects of this invention are as follows: On the one hand, by loading the official pre-trained weights of ResNet-18 for fine-tuning, some commonly used feature extractors can be utilized, reducing the training cost of the classification model; fine-tuning on this basis can also more effectively extract unique features from cervical images. On the other hand, the introduction of wavelet transform and multimodal feature fusion mechanisms enables the classification model to better extract detailed information, semantic location information, and learn the weights of different modal features, and integrates information through a multimodal decision mechanism to make more accurate classification judgments, thereby improving the overall classification performance of the model.

[0118] Example 2

[0119] Based on the same inventive concept, this embodiment provides a cervical image classification system based on wavelet transform multimodal fusion. Please refer to [link to relevant documentation]. Figure 7 The system includes:

[0120] The dataset partitioning module 201 is used to divide the acquired cervical tissue OCT images into training set and test set. The cervical tissue OCT images and cervical colposcopy images are divided into different groups according to their respective objects. Each group of OCT images and colposcopy images belong to the same object. Each group of OCT images has a corresponding colposcopy image. The colposcopy image of the same group of OCT images only exists in the training set or the test set.

[0121] The classification model construction module 202 is used to construct a cervical image classification model based on multimodal feature fusion, which includes multiple feature coding branches. Each feature coding branch processes different types of colposcopy images and OCT images, including a wavelet transform module and a backbone network. The wavelet transform module is used to perform frequency domain decomposition and feature enhancement on the input image, and the backbone network further extracts feature maps. Then, the feature fusion module fuses the colposcopy image features and OCT image features of different modalities to obtain fused features. Finally, multiple features are weighted and fused, and the prediction result is output through a Softmax layer. The feature fusion module fuses cervical OCT image features, cervical colposcopy saline image features, cervical colposcopy iodine smear image features, cervical colposcopy acetic acid whitening image features, HPV, and TCT embedding features to obtain fused features.

[0122] Training module 203 uses a weighted cross-entropy loss function with weights equal to the ratio of positive to negative samples. After adjusting the size of the OCT images and colposcopy images in the training set, the images are input into the OCT image classification model for training, resulting in a well-trained cervical image classification model.

[0123] Test module 204 is used to classify and predict cervical multimodal images in the test set using a trained cervical multimodal image classification model, and to integrate multiple modal information using a multimodal decision classification mechanism to make a final decision and obtain the classification result.

[0124] Since the system described in Embodiment 2 of this invention is the system used to implement the automatic cervical image classification method based on wavelet transform multimodal feature fusion convolutional neural network in Embodiment 1 of this invention, those skilled in the art can understand the specific structure and variations of this system based on the method described in Embodiment 1 of this invention, and therefore will not be repeated here. All systems used in the method of Embodiment 1 of this invention fall within the scope of protection of this invention.

[0125] In the specific implementation process, the dataset used in this embodiment of the invention includes 118,200 OCT images (PNG format, 120 images per patient) and 2,955 colposcopy images (PNG format, one each of saline, iodine smear, and acetic acid whitening per patient) of 985 patients with cervical tissue collected from multiple hospitals. These images include low-risk (663 negative) and high-risk (322 positive) images. The relevant statistical information is shown in Table 1.

[0126] Table 1. Cervical image dataset information used in the embodiments

[0127]

[0128] To verify the effectiveness of the method of this invention, the dataset was divided as follows: For comparison with a single-modality classification model, the dataset was divided into five parts, with four parts selected as the training set and the remaining part as the test set for five-fold cross-validation. To make the test results more convincing, it was ensured that the samples in the training and test sets were completely independent during the data partitioning process; that is, OCT images and colposcopy images belonging to the same patient could not coexist in both the training and test sets.

[0129] To demonstrate the effectiveness of this invention, using the dataset shown in Table 1, the embodiments of this invention were compared with two commonly used single OCT datasets and three colposcopy datasets using a five-fold cross-validation method. The classification results are shown in Table 2. As can be seen from Table 2, the method of this invention performs better in terms of classification accuracy and sensitivity. The formulas for calculating accuracy, specificity, and sensitivity are as follows:

[0130] Accuracy = (True positive + True negative) / (True positive + False positive + True negative + False negative)

[0131] Sensitivity = True positive / (True positive + False negative)

[0132] Specificity = True negative / (True negative + False positive)

[0133] Table 2. Comparison of classification performance between the method of this invention and the benchmark method.

[0134]

[0135] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the invention.

[0136] Obviously, those skilled in the art can make various modifications and variations to the embodiments of the present invention without departing from the spirit and scope of the embodiments of the present invention. Thus, if these modifications and variations to the embodiments of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention also intends to include these modifications and variations.

Claims

1. A cervical image classification method based on wavelet transform multi-modal fusion, characterized in that, Includes the following steps: S1: The acquired cervical tissue OCT images and colposcopy images are divided into training sets and test sets, and preprocessed. The cervical colposcopy images are detected and the regions of interest are cropped. The cervical colposcopy images and OCT images of the same person are paired and appear only in the training set or the test set. S2: Construct a cervical image classification model based on multimodal feature fusion, comprising multiple feature coding branches. Each feature coding branch processes different types of colposcopy and OCT images, including a wavelet transform module and a backbone network. The wavelet transform module is used to perform frequency domain decomposition and feature enhancement on the input image, and the backbone network further extracts feature maps. Then, the feature fusion module fuses the features of colposcopy images and OCT images from different modalities to obtain fused features. Finally, the multiple features are weighted and fused, and the prediction result is output through a Softmax layer. S2 specifically includes: S2.1: Process the input cervical OCT image, cervical colposcopy saline image, cervical colposcopy iodine smear image, and cervical colposcopy acetic acid whitening image separately using multiple branches to obtain feature maps of different modalities; S2.2: In each branch, wavelet transform is first applied to decompose the input image of the branch in the frequency domain to obtain high-frequency and low-frequency components. The cross-frequency interaction mechanism is used to fuse the features of the high-frequency and low-frequency components. Then, ResNet-18 is used for further feature extraction. Feature fusion of high-frequency and low-frequency components using a cross-frequency interaction mechanism includes: using a channel spatial attention mechanism to highlight key information within a specific frequency band, applying a channel attention mechanism to low-frequency components and a spatial attention mechanism to high-frequency components, and then performing a splicing operation on the low-frequency and high-frequency components. S2.3: After the three colposcopy branches, the extracted three colposcopy features are stacked and convolved to obtain the three colposcopy fusion features through the feature fusion module. The three colposcopy fusion features are then concatenated with the OCT features obtained from the OCT branch to obtain the fusion features of colposcopy and OCT. The feature fusion module of S2.3 specifically includes: S2.3.1: Stack the features of the three colposcopes along the channel number to obtain a preliminary fused 3×512-dimensional feature; S2.3.2: The three colposcopy features are initially fused and then further fused by convolution with a 1×1 kernel. Dimensional compression is then performed to eliminate redundant dimensions, and the fused 512-dimensional colposcopy feature vector is output. S2.3.3: The fused 512-dimensional colposcopy feature vector is concatenated with the 512-dimensional OCT feature vector extracted from the OCT branch to obtain the final fused colposcopy and OCT feature vector, which has a size of 1024 dimensions. S2.4: Add two fully connected layers after the above network structure; S2.5: Weighted voting is applied to colposcopy features, OCT features, and fusion features. By dynamically allocating decision weights, the weights of single-modal features (colposcopy features), OCT features, and multimodal features (fusion features) are dynamically adjusted based on modal confidence, thereby improving classification accuracy. Finally, a Softmax layer is set to output the prediction results. S3: Set the weighted cross-entropy loss function according to the ratio of negative to positive in the data, adjust the input data type and model, and obtain a trained multimodal cervical image classification model; S4: Use the trained multimodal cervical image classification model to classify and predict cervical images in the test set, and obtain the classification results.

2. The cervical image classification method based on wavelet transform multimodal fusion as described in claim 1, characterized in that: S1 specifically includes: S1.1: Perform data cleaning on colposcopy images, filter out low-quality images, and ensure that all images contain complete data from the three types of cervical colposcopy examinations; S1.2: Use the YOLOv5 network to train a cervical os detection model for colposcopy to locate the cervix in saline, iodine smear, and acetic acid white colposcopy images, and crop the cervical os region to a certain number of pixels; S1.3: Match cervical OCT images with colposcopy images and divide the dataset.

3. The cervical image classification method based on wavelet transform multimodal fusion as described in claim 1, characterized in that: In S2.2, the Haar wavelet basis function performs a two-dimensional discrete wavelet transform on the input image of this branch to obtain high-frequency and low-frequency components.

4. The cervical image classification method based on wavelet transform multimodal fusion as described in claim 1, characterized in that: In step S3, the total loss function is calculated: ,in The loss due to the fusion of colposcopy and OCT. Loss due to fusion of three colposcopy techniques. For OCT loss, The total loss, of which , , The cross-entropy loss function is used for loss calculation in all cases.

5. The cervical image classification method based on wavelet transform multimodal fusion as described in claim 1, characterized in that: The two fully connected layers added after the feature fusion module reduce the dimensionality of the fused features to 256 and 2, respectively.

6. The cervical image classification method based on wavelet transform multimodal fusion as described in claim 1, characterized in that: The cervical image classification model was initialized using ImageNet pre-trained ResNet-18 weights provided by the PyTorch framework, and gradient descent was used to fine-tune all parameters.

7. A cervical image classification system based on wavelet transform multimodal fusion, used to implement the cervical image classification method based on wavelet transform multimodal fusion as described in any one of claims 1-6, characterized in that, include: The dataset partitioning module is used to divide the acquired cervical tissue OCT images and colposcopy images into training sets and test sets, and to perform preprocessing, detection and region of interest cropping on the cervical colposcopy images. The cervical colposcopy images and OCT images of the same person are paired and appear only in the training set or the test set. The classification model construction module is used to build a cervical image classification model based on multimodal feature fusion, which includes multiple feature coding branches. Each feature coding branch processes different types of colposcopy images and OCT images. It includes a wavelet transform module and a backbone network. The wavelet transform module is used to perform frequency domain decomposition and feature enhancement on the input image, and the backbone network further extracts feature maps. Then, the feature fusion module fuses the features of colposcopy images and OCT images from different modalities to obtain fused features. Finally, the multiple features are weighted and fused, and the prediction result is output through a Softmax layer. The training module is used to set the weighted cross-entropy loss function based on the ratio of negative to positive data, adjust the input data type and model, and obtain a trained multimodal cervical image classification model. The testing module is used to classify and predict cervical images in the test set using a trained multimodal cervical image classification model, and obtain the classification results.

Images