Colposcope image processing algorithm based on multi-modal feature fusion

Abstract

The invention discloses a colposcope image processing algorithm based on multi-modal feature fusion. The algorithm comprises the following steps: data acquisition: acquiring a colposcope image acquired by medical equipment; labeling the collected colposcope image, wherein the collected colposcope image comprises a marker of a lesion area and an image category; data preprocessing: carrying out image preprocessing on the colposcope image, carrying out preprocessing on text data, and aligning multi-modal data; dividing the preprocessed data into a training set and a test set; establishing and training a neural network model, wherein the model is used for performing feature extraction and feature fusion to obtain a colposcope image after multi-modal feature fusion; the feature fusion comprises the step of fusing the extracted image features, text features, physiological signal features and laboratory examination result features. Richer and more comprehensive input is provided for the model, so that the diagnosis performance is remarkably improved.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a colposcopy image processing algorithm based on multimodal feature fusion. Background Technology

[0002] Colposcopy is a crucial method for screening and diagnosing cervical diseases. Traditional colposcopy image analysis relies heavily on the doctor's experience and visual judgment, resulting in high subjectivity and poor diagnostic consistency. In recent years, with the development of artificial intelligence technology, image-based automatic diagnostic algorithms have been gradually applied to colposcopy image analysis; however, existing technologies still have the following problems: Limitations of single-modal data: Most existing algorithms rely solely on colposcopy images themselves, lacking integration of other relevant modal data (such as pathological data, clinical history, etc.), resulting in incomplete diagnostic information.

[0003] Insufficient feature extraction: Traditional methods often fail to fully utilize the multi-scale and multi-texture information of images during feature extraction, resulting in limited feature representation capabilities.

[0004] Lack of interpretability: Most existing deep learning models are "black box" models, which are difficult to provide intuitive diagnostic evidence and affect doctors' trust in the diagnostic results.

[0005] Based on this, this application proposes a colposcopy image processing algorithm based on multimodal feature fusion. Summary of the Invention

[0006] This invention provides a colposcopy image processing algorithm based on multimodal feature fusion, which effectively integrates multimodal data and optimizes feature extraction and model interpretability.

[0007] According to one aspect of this disclosure, a colposcopy image processing algorithm based on multimodal feature fusion is provided, the algorithm comprising: S1, Data acquisition includes: acquiring colposcopy images collected by medical equipment; annotating the acquired colposcopy images, including marking of lesion areas and image categories; the image categories include: normal and lesion; acquiring case information corresponding to the patients in the colposcopy images; S2, Data preprocessing: Perform image preprocessing on the colposcopy images, preprocess the text data, and align the multimodal data; divide the preprocessed data into training and test sets; S3, Establish and train a neural network model, which is used for feature extraction and feature fusion to obtain a colposcopy image after multimodal feature fusion; Feature extraction includes: image feature extraction, text feature extraction, physiological signal feature extraction, and laboratory test result feature extraction; feature fusion includes: fusing the extracted image features, text features, physiological signal features, and laboratory test result features.

[0008] In one possible implementation, the data acquisition further includes: Obtain the patient's physiological signals and laboratory test results. Physiological signals include heart rate and / or blood pressure, and laboratory test results include blood indicators and cytological test results.

[0009] In one possible implementation, image preprocessing of the colposcopy image includes: Image filtering: Selecting qualified colposcopy images from the acquired images; Image normalization: Adjusting the size of an image to a uniform size and normalizing the image so that the pixel values ​​are within a certain range; Noise removal: Removes noise from an image through smoothing filtering; Region segmentation: The cervical region is detected using an object detection model, and the cervical region is segmented from the image using an improved segmentation algorithm.

[0010] In one possible implementation, the text data is preprocessed, including: Text cleaning: Remove irrelevant characters and stop words from medical record text, retaining only useful information; Text vectorization: Word embedding converts text data into numerical vectors using word embedding methods; Align multimodal data to ensure that timestamps or identifiers are consistent between image and text data.

[0011] In one possible implementation, image feature extraction includes: Low-level image feature extraction: Extract low-level image features from the preprocessed image and fuse multiple low-level features into a local descriptor; High-level semantic feature extraction: High-level semantic features of images are extracted using deep learning models to obtain high-dimensional feature vectors.

[0012] In one possible implementation, text feature extraction includes: using a pre-trained language model to extract semantic features from the vectorized text data to generate a text feature vector; After numerical preprocessing and standardization of physiological signals and laboratory test results, statistical features are extracted, including: Physiological signal feature extraction: Extracting time-domain or frequency-domain features of physiological signals; Laboratory test result feature extraction: converting laboratory test results into numerical features.

[0013] In one possible implementation, feature fusion includes fusing extracted image features, text features, physiological signal features, and laboratory test result features, including: Feature alignment is performed on different types of features to unify the dimensions. Then, intermodal attention is fused on different types of features, and the weighted output is used to obtain the feature fusion result.

[0014] In one possible implementation, the result of feature fusion is output to an output layer, where the output layer configures the model parameters and training parameters. The model parameters include: the learning rate of the Adam optimizer, the L2 regularization coefficient, and the loss function; the training parameters include: dynamically adjusting the decay rate of the exponential moving average (EMA), setting the regularization parameter, and the patience early stopping parameter. The output layer outputs classification and segmentation results. The classification results include the lesion probability distribution and confidence score; the segmentation results include: lesion region mask and boundary accuracy.

[0015] In one possible implementation, model training includes: training the model using a training set and continuously adjusting parameters to optimize model performance during training; employing transfer learning to pre-train the model on a large dataset and then fine-tuning it on the training set dataset; setting parameters during training, including learning rate, optimizer, and loss function; the loss function is a loss function suitable for multimodal data, used to combine image classification loss and text relevance loss; Model validation includes: validating and evaluating the trained model using a test set, with evaluation metrics including accuracy, recall, F1 score, and AUC; Multimodal evaluation metrics: Comparative experiments were conducted to verify the effectiveness of multimodal fusion in evaluating the model's performance on multimodal data fusion.

[0016] Compared with the prior art, the beneficial effects of the present invention are: The colposcopy image processing algorithm based on multimodal feature fusion disclosed in this invention innovatively achieves deep fusion and collaborative processing of multimodal data. Most existing technologies rely solely on single-modal data from colposcopy images, lacking the integration of multi-source information such as pathological data and clinical history. This results in one-sided diagnostic information, failing to comprehensively reflect lesion characteristics and limiting diagnostic accuracy and reliability. This invention, by constructing a multimodal data fusion framework, organically combines the visual features of colposcopy images with quantitative indicators of pathological data and background information from clinical history, providing the model with richer and more comprehensive input, thereby significantly improving diagnostic performance. Simultaneously, this invention introduces deep learning technology and attention mechanisms during feature extraction and fusion, further optimizing feature representation and model decision-making capabilities. This overcomes the shortcomings of existing technologies in areas such as insufficient feature extraction and poor model interpretability, providing stronger technical support for the accurate diagnosis of cervical diseases.

[0017] The technical solution of this application has the following advantages: 1. High diagnostic accuracy: Through multimodal data fusion, the model can more comprehensively capture lesion characteristics, significantly improving diagnostic accuracy. 2. Strong model interpretability: Utilizing technologies such as LIME to provide interpretation of diagnostic results enhances doctors' trust in AI diagnostic results. 3. High data utilization efficiency: Through deep feature extraction and dimensionality reduction techniques, limited multimodal data is fully utilized, improving the model's generalization ability. 4. Significant clinical application value: This invention can effectively assist doctors in the early screening and diagnosis of cervical diseases, and has important clinical application value. Attached Figure Description

[0018] Figure 1 A flowchart of an image processing algorithm according to an embodiment of the present disclosure is shown.

[0019] Figure 2 A feature fusion processing flowchart of an embodiment of the present disclosure is shown.

[0020] Figure 3 A flowchart of the output module processing according to an embodiment of the present disclosure is shown.

[0021] Figure 4 A flowchart illustrating the image processing logic of an embodiment of the present disclosure is shown. Detailed Implementation

[0022] Various exemplary embodiments, features, and aspects of this disclosure will now be described in detail with reference to the accompanying drawings. The same reference numerals in the drawings denote elements that have the same or similar functions. Although various aspects of the embodiments are shown in the drawings, they are not necessarily drawn to scale unless specifically indicated otherwise.

[0023] The term “exemplary” as used herein means “serving as an example, embodiment, or illustration.” Any embodiment illustrated herein as “exemplary” is not necessarily to be construed as superior to or better than other embodiments.

[0024] Furthermore, to better illustrate this disclosure, numerous specific details are set forth in the following detailed description. Those skilled in the art will understand that this disclosure can be practiced without certain specific details. In some instances, methods, means, components, and circuits well known to those skilled in the art have not been described in detail in order to highlight the main points of this disclosure.

[0025] A colposcopy image processing algorithm based on multimodal feature fusion, the algorithm comprising: S1, Data acquisition includes: acquiring colposcopy images collected by medical equipment; annotating the acquired colposcopy images, including marking of lesion areas and image categories; the image categories include: normal and lesion; acquiring case information corresponding to the patients in the colposcopy images; Equipment preparation: Use professional colposcope equipment to ensure that the resolution, light source intensity and other parameters of the equipment meet the requirements of medical diagnosis.

[0026] Image Acquisition: In a clinical setting, colposcopes are used by professional medical staff to acquire colposcopy images. During acquisition, attention must be paid to image clarity, uniform lighting, and the shooting angle.

[0027] Data annotation: The collected images are annotated, including marking the lesion areas and the image category (such as normal, lesion, etc.), so that they can be used for subsequent model training.

[0028] Multimodal data acquisition: Text data acquisition: Collect medical record information related to colposcopy images, including text data such as the patient's age, symptoms, and medical history.

[0029] Other modal data acquisition: If other modal data (such as physiological signals, laboratory test results, etc.) are involved, they should also be acquired and labeled simultaneously.

[0030] S2, Data preprocessing: Perform image preprocessing on the colposcopy images, preprocess the text data, and align the multimodal data; divide the preprocessed data into training and test sets; Image preprocessing includes: Image filtering: Selecting eligible colposcopy images from the acquired images and removing irrelevant images. Image normalization: Adjusting the image size to a uniform level (e.g., 256×256 or 224×224) and normalizing the image to ensure pixel values ​​are within a certain range. Noise removal: Removing noise from the image using methods such as smoothing filtering to improve image quality. Region segmentation: Detecting the cervical region using a target detection model (e.g., YOLOv5) and segmenting the cervical region from the image using an improved segmentation algorithm (e.g., a U-Net-based segmentation network).

[0031] Text data preprocessing includes: text cleaning: removing irrelevant characters, stop words, etc. from medical record text, retaining only useful information. Text vectorization: converting text data into numerical vectors, which can be done using methods such as word embedding (e.g., Word2Vec, BERT).

[0032] Multimodal data alignment includes: Data alignment: ensuring that the timestamps or identifiers are consistent between image data and text data (and other modal data) so that they can be fused in subsequent steps.

[0033] S3, Establish and train a neural network model, which is used for feature extraction and feature fusion to obtain a colposcopy image after multimodal feature fusion; Feature extraction includes: image feature extraction, text feature extraction, physiological signal feature extraction, and laboratory test result feature extraction; feature fusion includes: fusing the extracted image features, text features, physiological signal features, and laboratory test result features.

[0034] In one possible implementation, the data acquisition further includes: Obtain the patient's physiological signals and laboratory test results. Physiological signals include heart rate and / or blood pressure, and laboratory test results include blood indicators and cytological test results.

[0035] In one possible implementation, image preprocessing of the colposcopy image includes: Image filtering: Selecting qualified colposcopy images from the acquired images; Image normalization: Adjusting the size of an image to a uniform size and normalizing the image so that the pixel values ​​are within a certain range; Noise removal: Removes noise from an image through smoothing filtering; Region segmentation: The cervical region is detected using an object detection model, and the cervical region is segmented from the image using an improved segmentation algorithm.

[0036] In one possible implementation, the text data is preprocessed, including: Text cleaning: Remove irrelevant characters and stop words from medical record text, retaining only useful information; Text vectorization: Word embedding converts text data into numerical vectors using word embedding methods; Align multimodal data to ensure that timestamps or identifiers are consistent between image and text data.

[0037] In one possible implementation, image feature extraction includes: Low-level image feature extraction: Extract low-level image features from the preprocessed image, such as SIFT, SURF, LBP features, etc., and fuse multiple low-level features into a local descriptor; High-level semantic feature extraction: High-level semantic features of images are extracted using deep learning models (such as convolutional neural networks CNN) to obtain high-dimensional feature vectors.

[0038] In one possible implementation, text feature extraction includes: using a pre-trained language model (such as BERT, RoBERTa) to extract semantic features from the vectorized text data, generating a text feature vector; After numerical preprocessing and standardization of physiological signals and laboratory test results, statistical features are extracted, including: Physiological signal feature extraction: Extracting the time-domain or frequency-domain features of physiological signals (such as heart rate, blood pressure, etc.); Laboratory test result feature extraction converts laboratory test results (such as blood indicators, cytology test results, etc.) into numerical features.

[0039] In one possible implementation, feature fusion includes fusing extracted image features, text features, physiological signal features, and laboratory test result features, including: Feature alignment is performed on different types of features to unify the dimensions. Then, intermodal attention is fused on different types of features, and the weighted output is used to obtain the feature fusion result.

[0040] Feature fusion method selection: Choose an appropriate feature fusion method based on specific needs, such as simple feature concatenation, weighted summation, or more complex fusion strategies (such as attention mechanisms, Transformer, etc.).

[0041] Feature fusion generation: This involves fusing extracted image features, text features, and other modal features to generate fused features. For example, a multimodal Transformer architecture can be used to implement feature fusion.

[0042] In one possible implementation, the result of feature fusion is output to an output layer, where the output layer configures the model parameters and training parameters. The model parameters include: the learning rate of the Adam optimizer, the L2 regularization coefficient, and the loss function; the training parameters include: dynamically adjusting the decay rate of the exponential moving average (EMA), setting the regularization parameter, and the patience early stopping parameter. The output layer outputs classification and segmentation results. The classification results include the lesion probability distribution and confidence score; the segmentation results include: lesion region mask and boundary accuracy.

[0043] In one possible implementation, model training includes: training the model using a training set and continuously adjusting parameters to optimize model performance during training; employing transfer learning to pre-train the model on a large dataset and then fine-tuning it on the training set dataset; setting parameters during training, including learning rate, optimizer, and loss function; the loss function is a loss function suitable for multimodal data, used to combine image classification loss and text relevance loss; Model validation includes: validating and evaluating the trained model using a test set, with evaluation metrics including accuracy, recall, F1 score, and AUC; Multimodal evaluation metrics: Comparative experiments were conducted to verify the effectiveness of multimodal fusion in evaluating the model's performance on multimodal data fusion.

[0044] A baseline model is a benchmark model used to compare and evaluate the performance of a new model. It is usually selected as a classic algorithm or basic method recognized in the field as a reference. Examples include traditional machine learning algorithms or simple neural network structures.

[0045] Attention-based models are deep learning models that incorporate attention mechanisms and are primarily used to address information overload problems by dynamically allocating computational resources to improve task processing efficiency.

[0046] Multimodal fusion is not a single model, but a technical approach that integrates information from different modalities (such as text, images, and speech) to enhance a model's ability to perceive, reason, and make decisions in complex environments through joint modeling.

[0047] Core concept: Multimodal fusion is achieved through the following methods: Data layer fusion: Directly merge raw data from different modalities.

[0048] Feature layer fusion: After extracting features from each modality, joint analysis is performed.

[0049] Decision-making integration: Integrating decision outcomes from different modalities.

[0050] Table 1 Comparative experimental results parameters of different models ; For example, dataset partitioning: the preprocessed multimodal dataset is divided into training and testing sets, usually in a ratio of 8:2 or 7:3.

[0051] Model selection and construction: Model architecture design: Choose a suitable multimodal model architecture, such as multimodal Transformer, multimodal CNN, etc. For example, ViT (Vision Transformer) can be used as the image feature extraction module, BERT as the text feature extraction module, and then the two can be fused together through a fusion module (such as cross-attention mechanism).

[0052] Model building: The model is designed and built based on the characteristics of multimodal feature fusion to ensure that the model can process image, text and other modal data simultaneously.

[0053] Training parameter settings: Configure parameters for the training process, such as learning rate, optimizer, and loss function. For example, you can use the Adam optimizer, cross-entropy loss function, and cosine annealing for the learning rate.

[0054] Multimodal loss function design: Design a loss function suitable for multimodal data, for example, combining image classification loss and text relevance loss. Here, the Dice loss function is used, and its formula is as follows: ; In the formula, y represents the set of pixels for the ground truth label. : The set of pixels in the segmentation results predicted by the model, N: the total number of pixels.

[0055] Model training: The model is trained using a training set, and parameters are continuously adjusted during training to optimize model performance. Transfer learning can be used, where the model is pre-trained on a large dataset and then fine-tuned on a custom dataset.

[0056] Multimodal data training strategy: During training, images, text and other modal data are input simultaneously to ensure that the model can learn the correlation between multimodal data.

[0057] Model Validation and Evaluation: Model Validation: The trained model is validated and evaluated using a test set. Evaluation metrics include accuracy, recall, F1 score, and AUC. Multimodal Evaluation Metrics: In addition to traditional evaluation metrics, the model's effectiveness in multimodal data fusion can also be evaluated. For example, comparative experiments can be used to verify the contribution of multimodal fusion to performance improvement.

[0058] Model saving: Save the trained model for later use in practical applications.

[0059] Model deployment: Deploying the model into the actual medical system to assist in diagnosis.

[0060] The various embodiments of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or technical improvements to the embodiments in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A colposcopy image processing algorithm based on multi-modal feature fusion, characterized in that, The algorithm includes: S1, Data acquisition includes: acquiring colposcopy images collected by medical equipment; annotating the acquired colposcopy images, including marking of lesion areas and image categories; the image categories include: normal and lesion; acquiring case information corresponding to the patients in the colposcopy images; S2, Data preprocessing: Perform image preprocessing on the colposcopy images, preprocess the text data, and align the multimodal data; divide the preprocessed data into training and test sets; S3, Establish and train a neural network model, which is used for feature extraction and feature fusion to obtain a colposcopy image after multimodal feature fusion; Feature extraction includes: image feature extraction, text feature extraction, physiological signal feature extraction, and laboratory test result feature extraction; feature fusion includes: fusing the extracted image features, text features, physiological signal features, and laboratory test result features.

2. The colposcopy image processing algorithm based on multi-modal feature fusion according to claim 1, characterized in that, The data acquisition also includes: Obtain the patient's physiological signals and laboratory test results. Physiological signals include heart rate and / or blood pressure, and laboratory test results include blood indicators and cytological test results.

3. The colposcopy image processing algorithm based on multi-modal feature fusion of claim 1, wherein, Image preprocessing of the colposcopy images includes: Image filtering: Selecting qualified colposcopy images from the acquired images; Image normalization: Adjusting the size of an image to a uniform size and normalizing the image so that the pixel values ​​are within a certain range; Noise removal: Removes noise from an image through smoothing filtering; Region segmentation: The cervical region is detected using an object detection model, and the cervical region is segmented from the image using an improved segmentation algorithm.

4. The colposcopy image processing algorithm based on multi-modal feature fusion of claim 1, wherein, Preprocessing of text data includes: Text cleaning: Remove irrelevant characters and stop words from medical record text, retaining only useful information; Text vectorization: Word embedding converts text data into numerical vectors using word embedding methods; Align multimodal data to ensure that timestamps or identifiers are consistent between image and text data.

5. The colposcopy image processing algorithm based on multi-modal feature fusion according to claim 3, characterized in that, Image feature extraction, including: Low-level image feature extraction: Extract low-level image features from the preprocessed image and fuse multiple low-level features into a local descriptor; High-level semantic feature extraction: High-level semantic features of images are extracted using deep learning models to obtain high-dimensional feature vectors.

6. The colposcopy image processing algorithm based on multi-modal feature fusion according to claim 4, characterized in that, Text feature extraction includes: using a pre-trained language model to extract semantic features from the vectorized text data, and generating a text feature vector; After numerical preprocessing and standardization of physiological signals and laboratory test results, statistical features are extracted, including: Physiological signal feature extraction: Extracting time-domain or frequency-domain features of physiological signals; Laboratory test result feature extraction: converting laboratory test results into numerical features.

7. The colposcopy image processing algorithm based on multi-modal feature fusion of claim 1, wherein, Feature fusion includes fusing extracted image features, text features, physiological signal features, and laboratory test result features, including: Feature alignment is performed on different types of features to unify the dimensions. Then, intermodal attention is fused on different types of features, and the weighted output is used to obtain the feature fusion result.

8. The colposcopy image processing algorithm based on multi-modal feature fusion of claim 1, wherein, The result of feature fusion is output to the output layer, where the model parameters and training parameters are configured. The model parameters include: the learning rate of the Adam optimizer, the L2 regularization coefficient, and the loss function. The training parameters include: dynamically adjusting the decay rate of the exponential moving average (EMA), setting the regularization parameter, and the patience parameter. The output layer outputs classification and segmentation results. The classification results include the lesion probability distribution and confidence score; the segmentation results include: lesion region mask and boundary accuracy.

9. The colposcopy image processing algorithm based on multi-modal feature fusion of claim 1, wherein, Model training includes: training the model using a training set and continuously adjusting parameters to optimize model performance during training; employing transfer learning to pre-train the model on a large dataset and then fine-tuning it on the training set dataset; setting parameters during training, including learning rate, optimizer, and loss function; the loss function is suitable for multimodal data and is used to combine image classification loss and text relevance loss. Model validation includes: validating and evaluating the trained model using a test set, with evaluation metrics including accuracy, recall, F1 score, and AUC; Multimodal evaluation metrics: Comparative experiments were conducted to verify the effectiveness of multimodal fusion in evaluating the model's performance on multimodal data fusion.

Images