AI-driven electronic health record integrated obstetric handheld ultrasound report generation method

Abstract

The invention discloses an AI-driven electronic health record integrated obstetrical handheld ultrasound report generation method. The method comprises the following steps: S1, pregnant woman information synchronization; s2, AI parameter adjustment is carried out; s3, real-time image acquisition and analysis; s4, providing real-time diagnosis suggestions; s5, updating a diagnosis result and a pregnant woman record; s6, doctor operation and feedback; and S7, continuously learning and optimizing. According to the method, the overall diagnosis quality and efficiency of obstetrical ultrasonic examination are improved by integrating multi-modal data; a personalized antenatal care report is generated in combination with an ultrasonic examination image and multi-modal health information, so that the accuracy and flexibility of medical report generation are improved; fine adjustment is carried out on the Flamingo large model through the Lora technology, and the application speciality of the large language model in the medical field is improved.

Description

Technical Field

[0001] The present invention belongs to the field of obstetric technology, and specifically relates to an AI-driven electronic health record-integrated obstetric handheld ultrasound report generation method. Background Art

[0002] Ultrasound is a commonly used auxiliary examination method in obstetrics. It can be used to examine the anatomical structure of the fetus, placenta, uterus and surrounding accessories, thereby assisting doctors in making clinical decisions, maximizing diagnostic accuracy and improving the medical experience. In recent years, obstetric handheld ultrasound equipment has been an important innovation in the field of modern medical technology. Its portability and flexibility have significantly improved the convenience of obstetric examinations, allowing doctors to quickly carry out high-quality ultrasound examinations in various medical scenarios. In addition, obstetric handheld ultrasound equipment can not only accurately examine the growth and development of the fetus, monitor the fetal heart rate, and assess the amount of amniotic fluid, but also make detailed observations of the position of the placenta, the morphology of the uterus, and the anatomical structure of the surrounding accessories, providing doctors with rich imaging information. However, human health is the result of the cooperation of multiple systems and organs, and is in dynamic change. The previous health status and illness of pregnant women may affect themselves and the fetus, so it is difficult to immediately generate accurate diagnosis and treatment plans based on the results of a single ultrasound examination. Therefore, combining multidisciplinary and multi-time point electronic health records is very beneficial and necessary for obstetric ultrasound doctors to make the best clinical decisions.

[0003] Electronic health records (EHR) have become an important part of the modern medical system, helping doctors quickly access a pregnant woman's medical history, test results, and treatment plans, improving the efficiency and quality of medical services. However, the huge amount of data and complex data content in various formats in the EHR system also bring new challenges, especially how to effectively integrate and analyze past data and current visit data and make personalized optimal diagnosis immediately is an important issue that needs to be solved.

[0004] In order to overcome the above problems, artificial intelligence (AI), especially deep learning technology, has gradually been applied to medical image analysis and electronic health record processing in recent years. In particular, the application of multimodal AI models makes it possible to combine text and image data for analysis. Such models can extract useful information from multi-source data, assist in the rapid output of diagnostic reports, and provide more comprehensive support for clinical decision-making, thereby improving the timeliness and accuracy of diagnosis. However, the existing artificial intelligence technology has the following shortcomings in the field of ultrasound report output:

[0005] 1. Insufficient data standardization and interoperability. The data formats and standards generated by different medical systems and equipment are not unified, which makes data integration and analysis difficult. Insufficient interoperability and other issues limit the cross-system use and analysis of data.

[0006] 2. Insufficient integration of multimodal data. Existing AI models can only process a single data source, such as EHR text or a single medical imaging data. They lack effective integration and unified analysis of multimodal medical data (such as text, images, experimental data, etc.) and cannot generate comprehensive inspection reports.

[0007] 3. The professionalism of personalized reports generated using large language models needs to be enhanced: the existing approach ignores the key information of B-ultrasound images, making the final report information incomplete; and the large language model is not fine-tuned, making the final report lack of professionalism in this specific field. Summary of the invention

[0008] In order to solve the above problems, the present invention proposes an AI-driven electronic health record integrated obstetric handheld ultrasound report generation method.

[0009] The AI-driven electronic health record integrated obstetric handheld ultrasound report generation method of the present invention includes the following steps: S1. synchronization of pregnant woman information; S2. AI parameter adjustment; S3. real-time image acquisition and analysis; S4. providing real-time diagnosis suggestions; S5. updating of diagnosis results and pregnant woman records; S6. doctor operation and feedback; S7. continuous learning and optimization.

[0010] The AI ​​parameter adjustment in S2 refers to fine-tuning the Flamingo model using the Lora method.

[0011] The AI ​​algorithm in the AI ​​parameter adjustment in S2 includes the following steps: S201. Data preparation; S202. Model selection and construction; S203. Model fine-tuning using Lora technology; S204. Model training and optimization.

[0012] The S201 data preparation includes data collection, data cleaning and preprocessing, and data labeling.

[0013] The data collection includes collecting electronic health record data related to pregnant women through EHR and collecting ultrasound image data of pregnant women.

[0014] S202 model selection and construction refers to selecting and downloading the pre-trained Flamingo large model, and adjusting the model configuration parameters according to the specific application scenario.

[0015] The S203 uses the Lora technology to fine-tune the model, which refers to inserting a low-rank matrix into each layer of the Flamingo model, initializing the low-rank matrix and constraining the low-rank matrix.

[0016] The weight matrix in the Flamingo model is W, and after Lora fine-tuning, it is W'=W+ΔW

[0017] Among them, ΔW is a low-rank matrix, and ΔW=AB, A and B are two low-rank matrices, the dimension of A is d×r, and the dimension of B is r×d, where r is a rank value less than d.

[0018] The low-rank matrix is ​​inserted into each layer of the Flamingo model, and the low-rank matrix includes a low-rank matrix A and a low-rank matrix B, and

[0019] W q '=W q +A q B q

[0020] W k '=W k +A k B k

[0021] W v '=W v +A v B v

[0022] Where W q , W k and W v are the weight matrices for query, key, and value, respectively, q , B q , A k , B k , A v , B v is the corresponding low-rank matrix.

[0023] The low-rank matrices A and B are randomly generated by a Gaussian distribution with a mean of 0 and a variance of 0.01.

[0024] The beneficial effects of the present invention are:

[0025] 1) Integrate multimodal data to improve the overall diagnostic quality and efficiency of ultrasound examinations: Provide personalized ultrasound plans based on the health status and examination needs of different pregnant women, unify data annotation formats to meet the needs of different clinical scenarios, and improve the accuracy and effectiveness of diagnosis and treatment. In addition, the fine-tuned large language model can use multimodal EHR data to assist doctors in making timely, comprehensive, and accurate clinical decisions, reduce the occurrence of misdiagnosis, and quickly and accurately provide diagnostic recommendations for subsequent patients, improving clinical diagnosis and treatment efficiency.

[0026] 2) Improve the accuracy and flexibility of generating personalized medical reports: Generate detailed and personalized medical reports based on the current examination results and historical data of pregnant women, improve the accuracy and personalization of the reports, and effectively reduce the incidence of misdiagnosis and missed diagnosis. At the same time, the system combines the historical health data of pregnant women and provides personalized interpretation suggestions based on the health status of each pregnant woman. While enhancing the scientificity and pertinence of clinical decision-making, it also greatly reduces the time of clinical decision-making and helps each pregnant woman save time for medical treatment.

[0027] 3) Improve the professionalism of large language models in the medical field: This invention intends to fine-tune the Flamingo large model through Lora technology. Compared with the previous patented technology of directly applying ChatGPT, it will no longer be limited to a single fixed operating mode, but will be able to flexibly set corresponding parameters based on the specific conditions of different pregnant women. It can better adapt to complex and changeable clinical scenarios and significantly improve the comprehensive performance of the model in generating obstetric ultrasound examination reports. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 It is the path of the AI-driven electronic health record integrated obstetric handheld ultrasound report generation method of the present invention.

[0029] Figure 2 It is a flow chart of fine-tuning the Flamingo model using the LoRA method of the present invention. DETAILED DESCRIPTION

[0030] Embodiments of the present invention are described in detail below, and examples of the embodiments are shown in the accompanying drawings. The embodiments described below with reference to the accompanying drawings are exemplary and intended to be used to explain the present invention, but should not be understood as limiting the present invention.

[0031] Example 1

[0032] The AI-driven electronic health record integrated obstetric handheld ultrasound report generation method of the present invention is as follows: Figure 1 As shown, the following steps are included

[0033] S1. Synchronization of maternal information: In a typical clinical setting, an obstetrician holds an ultrasound device, activates it and connects it to the hospital or clinic's EHR system, ensuring that the latest medical records of the pregnant woman are automatically synchronized.

[0034] S2.AI parameter adjustment: Based on multimodal EHR data information, an examination plan suitable for the pregnant woman is recommended, and the AI ​​algorithm is used to predict and automatically adjust the scanning parameters of the obstetric handheld ultrasound device.

[0035] S3. Real-time image acquisition and analysis: Through high-performance processing units and integrated high-precision sensors and data interfaces, the system can acquire ultrasound images and process related data in real time.

[0036] S4. Provide real-time diagnostic recommendations: During the report generation process, the system will also be updated in combination with the clinical gold standard to ensure the accuracy and authority of the report. By comparing and calibrating with the clinical gold standard, the system can provide more accurate diagnostic recommendations and results. The user interface will display diagnostic recommendations based on the latest clinical gold standard to help doctors make timely, accurate and optimal clinical decisions.

[0037] S5. Update of diagnosis results and maternal records: Update the diagnosis results to the pregnant woman’s EHR to ensure the integrity of the medical records. Through such closed-loop feedback and continuous optimization, the system can continuously improve its diagnostic capabilities and user experience.

[0038] S6. Doctor operation and feedback: To ensure the stability and reliability of the system, comprehensive functional testing is required in an actual clinical environment, feedback from doctors and pregnant women is collected, and the user interface and AI functions are continuously optimized. At the same time, all data is processed and stored through a secure data processing and storage module to ensure data privacy and security.

[0039] S7. Continuous learning and optimization: The system will continuously optimize AI models and algorithms based on doctors’ feedback and the changing health status of pregnant women, thereby improving the accuracy and efficiency of diagnosis.

[0040] In summary, the present invention provides a comprehensive and efficient solution by combining AI and EHR, which can significantly improve the diagnostic accuracy and efficiency of obstetric handheld ultrasound equipment, provide personalized diagnostic support and optimization solutions, and has broad application prospects and practical value.

[0041] Example 2: Demonstration of AI algorithm in large model fine-tuning.

[0042] In terms of large model fine-tuning, the present invention intends to use the Lora method to fine-tune the Flamingo model (see Appendix Figure 2 ),like Figure 2 As shown in the figure, the specific implementation steps at the AI ​​algorithm level are described in detail as follows:

[0043] S201. Data preparation

[0044] 1. Data Collection

[0045] EHR: Collect electronic health record data related to pregnant women, including past medical history, laboratory results, diagnostic information, etc., which can be obtained from hospital information systems or other medical data platforms.

[0046] Ultrasound image data: Collect ultrasound image data of pregnant women, including imaging information of different organs or parts. These data can be extracted from the hospital's imaging system or the mother's previous examination reports.

[0047] 2. Data cleaning and preprocessing

[0048] Data cleaning: remove noise and outliers from the data, fill in missing data, and ensure data quality.

[0049] Data standardization: Standardize data from different sources into a unified format and unit to ensure consistency in data format.

[0050] Image preprocessing: Perform denoising, normalization, enhancement and other preprocessing on ultrasound images to improve the quality and analyzability of the images.

[0051] Text processing: Perform text cleaning and standardization on EHR data.

[0052] 3. Data Annotation

[0053] Data annotation: Annotate the target variables and labels required for training models. For example, annotate lesion areas in ultrasound images and annotate relevant diagnosis and treatment information in EHR.

[0054] S202. Model selection and construction

[0055] 1. Choose the Flamingo large model

[0056] Flamingo is a large multimodal model that can handle a variety of data types such as text and images, and is suitable for complex data analysis tasks in the medical field.

[0057] Model architecture: Flamingo's architecture includes multiple Transformer layers and a self-attention mechanism for processing text and image data, which can effectively capture the correlation between different modalities.

[0058] 2. Model Initialization

[0059] Pre-trained model download: Download the pre-trained Flamingo model from the open source platform to ensure that the model has been pre-trained on a large-scale dataset and has good generalization ability.

[0060] Model configuration adjustment: Adjust the model configuration parameters according to the specific application scenario, such as the number of model layers, number of hidden units, number of attention heads, etc., to ensure that the complexity of the model adapts to the needs of specific tasks.

[0061] S203. Apply LoRA technology to fine-tune the model

[0062] 1. The core idea of ​​LoRA: LoRA (Low-Rank Adaptation) inserts low-rank matrices into the pre-trained model, so that the model only needs to adjust these low-rank matrices during fine-tuning instead of adjusting the weights of the entire model, which can greatly reduce the parameters and computing resources required for fine-tuning.

[0063] 2. Build the LoRA module

[0064] Low-rank matrix insertion: Low-rank matrices are inserted in each layer of the Flamingo model. These matrices work together with the main parameter matrix of the model to reduce the amount of parameters required for fine-tuning.

[0065] Initialize the low-rank matrix: Use random initialization or other prior knowledge to initialize the low-rank matrix to ensure that it has a good initial state.

[0066] Low-rank matrix constraint: Set the rank of the low-rank matrix. Usually a smaller rank is chosen to reduce the computational burden while maintaining the expressiveness of the model.

[0067] 3. Fine-tune your strategy

[0068] Freeze main parameters: During fine-tuning, the main parameters of the Flamingo model are fixed and only the inserted low-rank matrix parameters are adjusted. This ensures that the knowledge learned by the model during pre-training is not destroyed.

[0069] S204. Model training and optimization

[0070] 1. Data input and batch processing

[0071] Multimodal data input: EHR text data and ultrasound image data are taken as input and processed in batches. Make sure that each batch of data contains both text and image information so that the model can learn the features of both types of data at the same time.

[0072] 2. Loss function design

[0073] Personalized report loss: For the generated personalized report, BLEU score and ROUGE score are used to measure the similarity between the generated text and the reference text.

[0074] 3. Optimization algorithm selection

[0075] Adam optimizer: Use the Adam optimizer to adjust model parameters. The Adam optimizer combines momentum and adaptive learning rate, and has good convergence in practical applications.

[0076] Example 3: LoRA module design

[0077] 1. Low-rank matrix decomposition: The core idea of ​​LoRA is to convert the weight update problem of the original model into a low-rank matrix learning problem, thereby reducing the number of parameters and computational overhead. Specifically, assuming that a weight matrix in the pre-trained model is W, in the LoRA framework, we express its update as:

[0078] W'=W+ΔW

[0079] Among them, ΔW is a low-rank matrix, which can be expressed as ΔW=AB. A and B are two low-rank matrices. Usually, the dimension of A is d×r, and the dimension of B is r×d, where r is a rank value less than d, which is much smaller than the dimension d of the original weight matrix, thereby significantly reducing the number of parameters.

[0080] 2. Module insertion: LoRA modules are inserted into each layer of the Flamingo model. These modules contain low-rank matrices A and B.

[0081] W q '=W q +A q B q

[0082] W k '=W k +A k B k

[0083] W v '=W v +A v B v

[0084] Where W q , W k and W v are the weight matrices of query, key, and value, respectively. q , B q , Ak , B k , A v , B v is the corresponding low-rank matrix.

[0085] 3. Low-rank matrix initialization: For low-rank matrices A and B, they are randomly generated using a Gaussian distribution with a mean of 0 and a variance of 0.01.

[0086] 4. Rank value selection: Select the rank value r as 40.

[0087] 5. BLEU score:

[0088] The BLEU (Bilingual Evaluation Understudy) score is a standard used to measure the similarity between machine translation generated text and reference text. It mainly focuses on the matching of n-grams.

[0089] Calculation formula

[0090]

[0091] Among them, p n is the precision of n-gram, w n is the weight, usually set to an even distribution. BP (Brevity Penalty) is the length penalty phase.

[0092] Loss function:

[0093]

[0094] 6.ROUGE score:

[0095] ROUGE (Recall-Oriented Understudy for Gisting Evaluation) score is mainly used to evaluate the quality of text summarization. It includes ROUGE-N (recall based on n-gram), ROUGE-L (recall based on the longest common subsequence), etc.

[0096]

[0097] Among them, Count Gen and Count Ref are the counts of n-grams in the generated text and the reference text, respectively.

[0098] Loss function:

[0099]

[0100] Finally, in order to format the output of the prenatal examination report, we designed the ChatGPT large language prompt as follows:

[0101]

[0102] Although the above embodiments have been shown and described, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations on the present invention. Changes, modifications, substitutions and variations of the above embodiments by those of ordinary skill in the art are all within the scope of protection of the present invention.

Claims

1. An AI-driven electronic health record integrated obstetric handheld ultrasound report generation method, characterized in that: The following steps are involved: S1. Synchronization of pregnant women’s information; S2.AI parameter adjustment; S3. Real-time image acquisition and analysis; S4. Provide real-time diagnostic suggestions; S5. Update of diagnosis results and maternity records; S6. Doctor operation and feedback; S7.Continuous learning and optimization.

2. The AI-driven electronic health record integrated obstetric handheld ultrasound report generation method according to claim 1 is characterized in that: The AI ​​parameter adjustment in S2 refers to fine-tuning the Flamingo model using the Lora method.

3. The AI-driven electronic health record integrated obstetric handheld ultrasound report generation method according to claim 2 is characterized in that: The AI ​​algorithm in the AI ​​parameter adjustment in S2 includes the following steps: S201. Data preparation; S202. Model selection and construction; S203. Use Lora technology to fine-tune the model; S204. Model training and optimization.

4. The AI-driven electronic health record integrated obstetric handheld ultrasound report generation method according to claim 3 is characterized in that: The S201 data preparation includes data collection, data cleaning and preprocessing, and data labeling.

5. The AI-driven electronic health record integrated obstetric handheld ultrasound report generation method according to claim 4, characterized in that: The data collection includes collecting electronic health record data related to pregnant women through EHR and collecting ultrasound image data of pregnant women.

6. The AI-driven electronic health record integrated obstetric handheld ultrasound report generation method according to claim 3 is characterized in that: S202 model selection and construction refers to selecting and downloading the pre-trained Flamingo large model, and adjusting the model configuration parameters according to the specific application scenario.

7. The AI-driven electronic health record integrated obstetric handheld ultrasound report generation method according to claim 3 is characterized in that: The S203 uses the Lora technology to fine-tune the model, which refers to inserting a low-rank matrix into each layer of the Flamingo model, initializing the low-rank matrix and constraining the low-rank matrix.

8. The AI-driven electronic health record integrated obstetric handheld ultrasound report generation method according to claim 7, characterized in that: The weight matrix in the Flamingo model is W, and after Lora fine-tuning, it is W'=W+ΔW Among them, ΔW is a low-rank matrix, and ΔW=AB, A and B are two low-rank matrices, the dimension of A is d×r, and the dimension of B is r×d, where r is a rank value less than d.

9. The AI-driven electronic health record integrated obstetric handheld ultrasound report generation method according to claim 8, characterized in that: The low-rank matrix is ​​inserted into each layer of the Flamingo model, and the low-rank matrix includes a low-rank matrix A and a low-rank matrix B, and W q '=W q +A q B q W k '=W k +A k B k W v '=W v +A v B v Where W q , W k and W v are the weight matrices for query, key, and value, respectively, q , B q , A k , B k , A v , B v is the corresponding low-rank matrix.

10. The AI-driven electronic health record integrated obstetric handheld ultrasound report generation method according to claim 9, characterized in that: The low-rank matrices A and B are randomly generated by a Gaussian distribution with a mean of 0 and a variance of 0.01.

Images