Intelligent cervical cancer screening method based on multi-modal large language model

Abstract

The invention relates to an intelligent cervical cancer screening method based on a multi-modal large language model, which comprises the following steps: data collection and processing: collecting and processing colposcope images and cervical cancer related medical reports, and pre-training a model for extracting key image and text features and providing input for the multi-modal model to enhance the accuracy and efficiency of cervical cancer screening; setting a multi-modal fusion model; fusing the image features and the text features; model training and tuning: taking the colposcope image and the corresponding text data as input, carrying out joint training on the multi-modal model, and finding an optimal configuration; and model evaluation and verification: carrying out performance evaluation on the model, analyzing the performance of the model on different categories and carrying out clinical verification. According to the method, colposcope images can be efficiently processed and analyzed, cervical cancer screening and diagnosis can be processed, a multi-modal large language model of a medical report can be generated, prognosis prediction, disease progress prediction and risk assessment can be carried out, and doctors can be helped to make more effective diagnosis and treatment plans.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to an intelligent screening method for cervical cancer based on a multimodal large language model. Background Art

[0002] The incidence of cervical cancer has always been at a high level, and early screening and diagnosis are of great significance to improving the survival rate of patients. At present, cervical cancer screening usually adopts a three-step method, namely cervical cytology (Pap smear), high-risk human papillomavirus (HPV) detection and colposcopy. However, colposcopy and biopsy have a large workload in outpatient clinics and there is a possibility of missed diagnosis. Summary of the invention

[0003] The purpose of the present invention is to overcome the problems existing in the prior art and provide an intelligent screening method for cervical cancer based on a multimodal large language model, which analyzes big data from multiple medical centers, trains a deep learning model, and verifies it on internal and external validation sets. When encountering difficult cases, the model can quickly identify suspicious lesion sites and recommend biopsy confirmation. At the same time, the multimodal model can also process geo gene databases, mine disease-related targeted genes, and enrich them on cells, and can also perform Mendelian randomization analysis on the causality of different diseases. In addition, the model is able to process medical data in visual modes (such as colposcopy images) and text modes (such as medical reports and free-text clinical notes), so it can be applied to clinical tasks involving visual and text data.

[0004] In order to achieve the above technical objectives and the above technical effects, the present invention is implemented through the following technical solutions: An intelligent screening method for cervical cancer based on a multimodal large language model, the model comprising: Input data: colposcopy images and medical reports; Downstream tasks: disease reporting, diagnosis, and prognosis; Multi-center dataset: data from different medical centers to improve the generalization ability of the model; Multilingual model: including English, Chinese and Spanish to enhance the applicability and universality of the model; Data collection and processing: colposcopy images and cervical cancer-related medical reports are processed separately. The pre-trained model is used to extract key image and text features to provide input for the multimodal model to enhance the accuracy and efficiency of cervical cancer screening. Set up a multimodal fusion model: fuse image features with text features; Model training and tuning: Colposcopy images and corresponding text data are used as input to jointly train the multimodal model and find the optimal configuration; Model evaluation and validation: Evaluate the performance of the model, analyze the performance of the model in different categories and conduct clinical validation; Deployment and application: Integrate the trained multimodal AI model into the corresponding clinical diagnosis system and provide an interface for uploading, viewing, displaying analysis results and providing feedback; Continuous optimization and updating: Regularly update the model, incorporate new data and medical research results, continuously improve the performance of the model, collect feedback from doctors and patients, and continuously optimize the system interface and user experience.

[0005] Furthermore, the data collection and processing includes colposcopy image processing and cervical cancer related medical text processing, wherein: Colposcopic image processing: Step S1.1: Data collection: Collect a large number of colposcopy images, including normal and abnormal images, from several regular medical institutions to ensure diversity, representativeness and rigor; Step S1.2: Data annotation: Ask professional doctors to annotate the images, mark the lesion areas, and classify each image; Step S1.3: image encoder selection and training; Step S1.31: Selecting an image encoder: Selecting a suitable image encoder so as to more accurately extract colposcopy image features; Step S1.32: Pre-trained model: Use the relevant medical image pre-trained model for transfer learning, and use the pre-trained model to enhance the feature extraction ability of the model; Step S1.33: Feature extraction: Use an image encoder to deeply process the colposcopy image and convert it into a high-dimensional feature vector that the model can effectively analyze; Cervical cancer related medical text processing: Step S2.1: Collect medical report data: obtain a large number of cervical cancer related medical reports; Step S2.2: Data cleaning: remove sensitive information from the report, unify the format, and deal with typos and incomplete sentences; Step S2.3: Data annotation: annotate the key contents in the report for the training model; Step S2.4: model selection and training; Step S2.41: Select a pre-trained model: Select a pre-trained model that has been pre-trained on a large amount of general text and is suitable for processing text, so as to have good language understanding capabilities; Step S2.42: Fine-tune the model: Fine-tune the model on the collected medical report dataset so that the model can understand the professional terms and sentence structure in the medical field; Step S2.5: Understand the report: Parse and understand the content of the medical report and identify key information; Step S2.6: Generate report: Generate a report that meets medical standards based on the input medical records and examination results to ensure the completeness and accuracy of the information; Step S2.7: Entity recognition: Identify key entities in the report, including disease names, symptoms, drugs, and numerical results; Step S2.8: Relationship extraction: Identify the relationships between entities, including the relationship between symptoms and diseases, and the relationship between drugs and treatment plans.

[0006] Furthermore, the multimodal fusion model includes: Multimodal model architecture: Design a multimodal model architecture to fuse image features with text features; Fusion strategy: Use a variety of fusion strategies to fuse image and text features at the model input layer or fuse features at the model output layer.

[0007] Furthermore, the model training and tuning includes: Joint training: The colposcopy images and the corresponding text data are used as input to jointly train the multimodal model, and the loss function comprehensively considers the prediction accuracy of the image and text; Transfer learning: Use the parameters of the pre-trained model and fine-tune it to adapt to cervical cancer screening and diagnosis tasks; Model tuning: Adjust the model’s hyperparameters and find the optimal configuration through cross-validation and hyperparameter search.

[0008] Furthermore, the model evaluation and verification includes: Performance evaluation: AUC, precision, recall, and F1-score indicators are used to evaluate the performance of the model; Confusion Matrix: Use the confusion matrix to analyze the model's performance on different categories to identify potential biases and misclassifications. Clinical validation: Validation in a clinical setting ensures that the model’s predictions are consistent with the physician’s diagnosis.

[0009] Furthermore, the deployment and application also include: User interface: Set up an easy-to-use interface so that doctors can easily upload colposcopy images and view analysis results and model interpretations; Real-time feedback: Provides a real-time feedback mechanism to facilitate doctors to verify and correct the model's prediction results, further improving the accuracy of the model.

[0010] Furthermore, the process of generating a report in step S2.6 of the cervical cancer-related medical text processing is as follows: Data integration: integrating different types of medical data into a unified database; Natural Language Generation: Generate natural language reports using NLG technology, and fine-tune the pre-trained GPT model to enable it to generate reports that meet medical standards; Template generation: Create some fixed templates and automatically fill in the content according to different diseases and examination results to improve the efficiency and consistency of report generation; Audit mechanism: Add manual audit steps to ensure that automatically generated reports are accurate and professional.

[0011] Furthermore, after the disease report is generated, the diagnosis results and diagnostic suggestions are generated: Impact analysis: Use convolutional neural networks to analyze cervical cell microscopic images, identify lesion areas, and provide diagnostic results; Text analysis: Use natural language processing technology to analyze text data from medical records and laboratory reports to extract symptoms and diagnosis information; Multimodal fusion: Combine image data and text data, use multimodal neural network for comprehensive analysis, and improve diagnostic accuracy; Decision support: Provide doctors with auxiliary diagnostic suggestions to support clinical decision-making.

[0012] Furthermore, after the disease report is generated, prognosis prediction is performed: Time series analysis: Use time series models to analyze patients’ long-term data and predict disease progression; Risk assessment: Risk assessment is performed using machine learning models based on the patient’s clinical characteristics and treatment options; Personalized prediction: Provide personalized prognosis prediction based on the individual characteristics of the patient to help doctors develop more effective treatment plans; Interpretability: Ensure that the prediction model has good interpretability so that doctors can understand the prediction results and the basis.

[0013] The beneficial effects of the present invention are: The present invention can efficiently process and analyze colposcopy images, providing strong support for the screening and diagnosis of cervical cancer, and a multimodal large-scale language model that can process cervical cancer screening and diagnosis and generate medical reports, providing strong support for clinical applications, and can perform prognosis prediction, predict disease progression, perform risk assessment, and help doctors develop more effective diagnosis and treatment plans. DETAILED DESCRIPTION

[0014] The present invention will be described in detail below with reference to embodiments.

[0015] An intelligent screening method for cervical cancer based on a multimodal large language model, the model comprising: Input data: colposcopy images and medical reports; Downstream tasks: disease reporting, diagnosis, and prognosis; Multi-center dataset: data from different medical centers to improve the generalization ability of the model; Multilingual model: including English, Chinese and Spanish to enhance the applicability and universality of the model; Data collection and processing: colposcopy images and cervical cancer-related medical reports are processed separately. The pre-trained model is used to extract key image and text features to provide input for the multimodal model to enhance the accuracy and efficiency of cervical cancer screening. Set up a multimodal fusion model: fuse image features with text features; Model training and tuning: Colposcopy images and corresponding text data are used as input to jointly train the multimodal model and find the optimal configuration; Model evaluation and validation: Evaluate the performance of the model, analyze the performance of the model in different categories and conduct clinical validation; Deployment and application: Integrate the trained multimodal AI model into the corresponding clinical diagnosis system and provide an interface for uploading, viewing, displaying analysis results and providing feedback; Continuous optimization and updating: Regularly update the model, incorporate new data and medical research results, continuously improve the performance of the model, collect feedback from doctors and patients, and continuously optimize the system interface and user experience.

[0016] The data collection and processing include colposcopy image processing and cervical cancer related medical text processing, wherein: Colposcopic image processing: Step S1.1: Data collection: Collect a large number of colposcopy images, including normal and abnormal (such as cervical cancer, precancerous lesions) images, from several regular medical institutions to ensure diversity, representativeness, and rigor; Step S1.2: Data annotation: Ask professional doctors to annotate the images, mark the lesion areas, and classify each image, such as normal, precancerous lesions, and cervical cancer; Step S1.3: image encoder selection and training; Step S1.31: Select image encoder: Select an image encoder, such as ResNet, Inception, EfficientNet, or Visual Transformer (ViT), to extract features of the colposcopy image; Step S1.32: Pre-trained model: Use the relevant medical image pre-trained model for transfer learning, and use the pre-trained model to enhance the feature extraction ability of the model; Step S1.33: Feature extraction: Use an image encoder to deeply process the colposcopy image and convert it into a high-dimensional feature vector that the model can effectively analyze; Cervical cancer related medical text processing: Step S2.1: Collect medical report data: Obtain a large number of cervical cancer-related medical reports, including medical records, examination reports, laboratory results, etc.; Step S2.2: Data cleaning: remove sensitive information from the report, unify the format, and deal with typos and incomplete sentences; Step S2.3: Data annotation: annotate the key contents of the report for the training model, such as diagnosis results, symptom descriptions, treatment recommendations, etc. Step S2.4: model selection and training; Step S2.41: Select a pre-trained model: Select a pre-trained model that has been pre-trained on a large amount of general text and is suitable for processing text, such as BERT, GPT-3, so as to have good language understanding capabilities; Step S2.42: Fine-tune the model: Fine-tune the model on the collected medical report dataset so that the model can understand the professional terms and sentence structure in the medical field; Step S2.5: Understand the report: Parse and understand the content of the medical report and identify key information, such as patient information, diagnosis results, symptom description, treatment plan, etc.; Step S2.6: Generate report: Generate a report that meets medical standards based on the input medical records and examination results to ensure the completeness and accuracy of the information; Step S2.7: Entity recognition: Identify key entities in the report, including disease names, symptoms, drugs, and numerical results; Step S2.8: Relationship extraction: Identify the relationships between entities, including the relationship between symptoms and diseases, and the relationship between drugs and treatment plans.

[0017] The multimodal fusion model includes: Multimodal model architecture: Design a multimodal model architecture to fuse image features with text features. For example, a Transformer model can be used to process text features, while image features can be fused into the Transformer model as input. Fusion strategy: Use a variety of fusion strategies to fuse image and text features at the model input layer or fuse features at the model output layer.

[0018] The model training and tuning include: Joint training: The colposcopy images and corresponding text data (such as doctors’ diagnosis reports, symptom descriptions, etc.) are used as input to jointly train the multimodal model. The loss function should comprehensively consider the prediction accuracy of the image and text.

[0019] Transfer learning: Use the parameters of the pre-trained model and fine-tune it to adapt to cervical cancer screening and diagnosis tasks.

[0020] Model tuning: Adjust the model’s hyperparameters (such as learning rate, batch size, etc.) and find the optimal configuration through cross-validation and hyperparameter search.

[0021] The model evaluation and validation includes: Performance evaluation: AUC, precision, recall, and F1-score indicators are used to evaluate the performance of the model; Confusion Matrix: Use the confusion matrix to analyze the model's performance on different categories to identify potential biases and misclassifications. Clinical validation: Validation in a clinical setting ensures that the model’s predictions are consistent with the physician’s diagnosis.

[0022] The deployment and application also include: User interface: Set up an easy-to-use interface so that doctors can easily upload colposcopy images and view analysis results and model interpretations; Real-time feedback: Provides a real-time feedback mechanism to facilitate doctors to verify and correct the model's prediction results, further improving the accuracy of the model.

[0023] The process of generating a report in step S2.6 of the cervical cancer related medical text processing is as follows: Data integration: integrating different types of medical data into a unified database; Natural Language Generation: Generate natural language reports using NLG technology, and fine-tune the pre-trained GPT model to enable it to generate reports that meet medical standards; Template generation: Create some fixed templates and automatically fill in the content according to different diseases and examination results to improve the efficiency and consistency of report generation; Audit mechanism: Add manual audit steps to ensure that automatically generated reports are accurate and professional.

[0024] After the disease report is generated, continue to generate diagnostic results and diagnostic recommendations: Impact analysis: Use convolutional neural networks to analyze cervical cell microscopic images, identify lesion areas, and provide diagnostic results; Text analysis: Use natural language processing technology to analyze text data from medical records and laboratory reports to extract symptoms and diagnosis information; Multimodal fusion: Combine image data and text data, use multimodal neural network for comprehensive analysis, and improve diagnostic accuracy; Decision support: Provide doctors with auxiliary diagnostic suggestions to support clinical decision-making.

[0025] After the disease report is generated, prognosis prediction is performed: Time series analysis: Use time series models (such as LSTM, Transformer) to analyze patients’ long-term data and predict disease progression; Risk assessment: Risk assessment is performed using machine learning models (such as random forests and XGBoost) based on the patient’s clinical characteristics and treatment options; Personalized prediction: Provide personalized prognosis prediction based on the patient's individual characteristics to help doctors develop more effective treatment plans.

[0026] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. An intelligent screening method for cervical cancer based on a multimodal large language model, characterized in that: The model includes: Input data: colposcopy images and medical reports; Downstream tasks: disease reporting, diagnosis, and prognosis; Multi-center dataset: data from multiple different medical centers to improve the generalization ability of the model; Multilingual model: including English, Chinese and Spanish to enhance the versatility and universality of the model; Data collection and processing: colposcopy images and cervical cancer-related medical reports are processed separately. The pre-trained model is used to extract key image and text features to provide input for the multimodal model to enhance the accuracy and efficiency of cervical cancer screening. Set up a multimodal fusion model: fuse image features with text features; Model training and tuning: Colposcopy images and corresponding text data are used as input to jointly train the multimodal model and find the optimal configuration; Model evaluation and validation: Evaluate the performance of the model, analyze the performance of the model in different categories and conduct clinical validation; Deployment and application: Integrate the trained multimodal AI model into the corresponding clinical diagnosis system and provide an interface for uploading, viewing, displaying analysis results and providing feedback; Continuous optimization and updating: Regularly update the model, incorporate new data and medical research results, continuously improve the performance of the model, collect feedback from doctors and patients, and continuously optimize the system interface and user experience.

2. The intelligent screening method for cervical cancer based on a multimodal large language model according to claim 1, characterized in that: The data collection and processing include colposcopy image processing and cervical cancer related medical text processing, wherein: Colposcopic image processing: Step S1.1: Data collection: Collect a large number of colposcopy images, including normal and abnormal images, from several regular medical institutions to ensure the diversity, representativeness and rigor of the data; Step S1.2: Data annotation: Ask professional doctors to annotate the images, mark the lesion areas, and classify each image; Step S1.3: image encoder selection and training; Step S1.31: Selecting an image encoder: Selecting a suitable image encoder so as to more accurately extract colposcopy image features; Step S1.32: Pre-trained model: Use the relevant medical image pre-trained model for transfer learning, and use the pre-trained model to enhance the feature extraction ability of the model; Step S1.33: Feature extraction: Use an image encoder to deeply process the colposcopy image and convert it into a high-dimensional feature vector that the model can effectively analyze; Cervical cancer related medical text processing: Step S2.1: Collect medical report data: obtain a large number of cervical cancer related medical reports; Step S2.2: Data cleaning: remove sensitive information from the report, unify the format, and deal with typos and incomplete sentences; Step S2.3: Data annotation: annotate the key contents in the report for the training model; Step S2.4: model selection and training; Step S2.41: Select a pre-trained model: Select a pre-trained model that has been pre-trained on a large amount of general text and is suitable for processing text, so as to have good language understanding capabilities; Step S2.42: Fine-tune the model: Fine-tune the model on the collected medical report dataset so that the model can understand the professional terms and sentence structure in the medical field; Step S2.5: Understand the report: Parse and understand the content of the medical report and identify key information; Step S2.6: Generate report: Generate a report that meets medical standards based on the input medical records and examination results to ensure the completeness and accuracy of the information; Step S2.7: Entity recognition: Identify key entities in the report, including disease names, symptoms, drugs, and numerical results; Step S2.8: Relationship extraction: Identify the relationships between entities, including the relationship between symptoms and diseases, and the relationship between drugs and treatment plans.

3. The intelligent screening method for cervical cancer based on a multimodal large language model according to claim 1, characterized in that: The multimodal fusion model includes: Multimodal model architecture: Design a multimodal model architecture to fuse image features with text features; Fusion strategy: Use a variety of fusion strategies to fuse image and text features at the model input layer or fuse features at the model output layer.

4. The intelligent screening method for cervical cancer based on a multimodal large language model according to claim 1, characterized in that: The model training and tuning include: Joint training: The colposcopy images and the corresponding text data are used as input to jointly train the multimodal model, and the loss function comprehensively considers the prediction accuracy of the image and text; Transfer learning: Use the parameters of the pre-trained model and fine-tune it to adapt to cervical cancer screening and diagnosis tasks; Model tuning: Adjust the model’s hyperparameters and find the optimal configuration through cross-validation and hyperparameter search.

5. The intelligent screening method for cervical cancer based on a multimodal large language model according to claim 1, characterized in that: The model evaluation and validation includes: Performance evaluation: AUC, precision, recall, and F1-score indicators are used to evaluate the performance of the model; Confusion Matrix: Use the confusion matrix to analyze the model's performance on different categories to identify potential biases and misclassifications. Clinical validation: Validation in a clinical setting ensures that the model’s predictions are consistent with the physician’s diagnosis.

6. The intelligent screening method for cervical cancer based on a multimodal large language model according to claim 1, characterized in that: The deployment and application also include: User interface: Set up an easy-to-use interface so that doctors can easily upload colposcopy images and view analysis results and model interpretations; Real-time feedback: Provides a real-time feedback mechanism to facilitate doctors to verify and correct the model's prediction results, further improving the accuracy of the model.

7. The intelligent screening method for cervical cancer based on a multimodal large language model according to claim 2, characterized in that: The process of generating a report in step S2.6 of the cervical cancer related medical text processing is as follows: Data integration: integrating different types of medical data into a unified database; Natural Language Generation: Generate natural language reports using NLG technology, and fine-tune the pre-trained GPT model to enable it to generate reports that meet medical standards; Template generation: Create some fixed templates and automatically fill in the content according to different diseases and examination results to improve the efficiency and consistency of report generation; Audit mechanism: Add manual audit steps to ensure that automatically generated reports are accurate and professional.

8. The intelligent screening method for cervical cancer based on a multimodal large language model according to claim 7, characterized in that: After the disease report is generated, continue to generate diagnostic results and diagnostic recommendations: Impact analysis: Use convolutional neural networks to analyze cervical cell microscopic images, identify lesion areas, and provide diagnostic results; Text analysis: Use natural language processing technology to analyze text data from medical records and laboratory reports to extract symptoms and diagnosis information; Multimodal fusion: Combine image data and text data, use multimodal neural network for comprehensive analysis, and improve diagnostic accuracy; Decision support: Provide doctors with auxiliary diagnostic suggestions to support clinical decision-making.

9. The intelligent screening method for cervical cancer based on a multimodal large language model according to claim 7 or 8, characterized in that: After the disease report is generated, prognosis prediction is performed: Time series analysis: Use time series models to analyze patients’ long-term data and predict disease progression; Risk assessment: Risk assessment is performed using machine learning models based on the patient’s clinical characteristics and treatment options; Personalized prediction: Provide personalized prognosis prediction based on the patient's individual characteristics to help doctors develop more effective treatment plans.