End-to-end medical record information extraction method based on multi-modal large model

Abstract

The application discloses an end-to-end medical record information extraction method based on a multi-modal large model, comprising the following steps: data integration and preprocessing, multi-modal data integration: integrating data of different sources and types, outpatient medical records, inpatient medical records, CT examination medical records and laboratory test results to enhance the information richness of the model; basic architecture: adopting the basic architecture of PaLM-E and using a Vi T pre-training model as a visual encoder to realize joint representation learning of images and texts; model combination: realizing model combination of different scales, PaLM 8B+Vi T 4B, PaLM 62B+Vi T 22B and the like to adapt to different computing and performance requirements; unified model architecture: using a unified model architecture and parameters to execute multiple tasks through multi-mode input; instructions and examples: the application can simultaneously process and analyze multiple types of data, such as data combined by texts and images, and provide more comprehensive medical record understanding.

Description

Technical Field

[0001] This invention relates to the field of medical record information processing technology, and more specifically, to an end-to-end medical record information extraction method based on a multimodal large model. Background Technology

[0002] For image-based medical record data, existing medical record information extraction technologies first use Optical Character Recognition (OCR) to convert the content of the medical record image into text, and then use Natural Language Processing (NLP) to extract structured information from the text. This approach requires training multiple model algorithms to complete the information extraction, necessitating the labeling of a large amount of medical record image and text data, resulting in significant manual labeling costs. Furthermore, the provided model service requires at least both OCR and NLP stages to complete, leading to high model service maintenance costs. Summary of the Invention

[0003] (a) Technical problems to be solved

[0004] To address the problems existing in the prior art, this invention provides an end-to-end medical record information extraction method based on a multimodal large model, which solves the problems of high cost of manual data annotation and high maintenance cost of multiple model services in the process of medical record information extraction, while also solving the problems of inaccurate medical record extraction and insufficient semantic understanding.

[0005] (II) Technical Solution

[0006] To achieve the above objectives, the present invention provides the following technical solution: an end-to-end medical record information extraction method based on a multimodal large model, comprising the following steps:

[0007] Step 1: Data Integration and Preprocessing

[0008] Multimodal data integration: Integrating data from different sources and types, such as outpatient medical records, inpatient medical records, CT scan records, and laboratory test results, to enhance the information richness of the model;

[0009] Step Two: Model Architecture Design

[0010] Basic architecture: It adopts the basic architecture of PaLM-E and uses the ViT pre-trained model as a visual encoder to achieve joint representation learning of images and text;

[0011] Model Combinations: Enables model combinations of different scales, such as PaLM 8B+Vi T 4B and PaLM 62B+Vi T22B, to adapt to different computing and performance requirements;

[0012] Step 3: End-to-end pre-training

[0013] Unified model architecture: Use a unified model architecture and parameters to execute multiple tasks through multi-modal input;

[0014] Instructions and Examples: Provide the model with task-specific instructions and one-time examples to guide the model in learning how to perform a specific task;

[0015] Step 4: Fine-tuning and optimization

[0016] End-to-end fine-tuning: In multimodal tasks, image and text labels are interleaved to form a multimodal context input to the model, enabling end-to-end fine-tuning;

[0017] Step 5: Performance evaluation.

[0018] The present invention is further configured such that the performance evaluation in step five evaluates the model performance on different tasks and datasets.

[0019] The present invention is further configured such that the multiple tasks in step three are entity extraction, question answering, and report generation.

[0020] (III) Beneficial Effects

[0021] Compared with existing technologies, this invention provides an end-to-end medical record information extraction method based on a multimodal large model, which has the following beneficial effects:

[0022] This invention can simultaneously process and analyze multiple types of data, such as combined text and image data, providing a more comprehensive understanding of medical records. The model architecture and training methods support deployments of varying scales, allowing for scaling of model capacity and service as needed. The end-to-end training method allows the model to learn directly from raw data to the representation of the final task, reducing reliance on intermediate stages and adjustments. The model can continuously learn from new data, constantly adapting to emerging data problems. Training on large-scale and diverse data enables better generalization capabilities, achieving good training results even with low sample sizes, reducing reliance on professional data labelers, and lowering long-term operating costs. It is adaptable to different medical scenarios and tasks. Detailed Implementation

[0023] It should be noted that, unless otherwise specified, all technical and scientific terms used in this application have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains.

[0024] The end-to-end medical record information extraction method based on a multimodal large model includes the following steps:

[0025] Step 1: Data Integration and Preprocessing

[0026] Multimodal data integration: Integrating data from different sources and types, such as outpatient medical records, inpatient medical records, CT scan records, and laboratory test results, to enhance the information richness of the model.

[0027] Step Two: Model Architecture Design

[0028] Basic architecture: It adopts the basic architecture of PaLM-E and uses the ViT pre-trained model as a visual encoder to achieve joint representation learning of images and text.

[0029] Model Combinations: Enables model combinations of different scales, such as PaLM 8B+Vi T 4B and PaLM 62B+Vi T22B, to adapt to different computing and performance requirements.

[0030] Step 3: End-to-end pre-training

[0031] Unified model architecture: Using a unified model architecture and parameters, multiple tasks can be performed through multimodal inputs, such as entity extraction, question answering, and report generation.

[0032] Instructions and Examples: Provide the model with task-specific instructions and one-time examples to guide the model in learning how to perform a specific task.

[0033] Step 4: Fine-tuning and optimization

[0034] End-to-end fine-tuning: In multimodal tasks, image and text labels are interleaved to form a multimodal context input to the model, enabling end-to-end fine-tuning.

[0035] Step 5:

[0036] Performance evaluation: Evaluate model performance on different tasks and datasets, such as the performance of Med-PaLM on 14 test tasks, and comparison with professional doctors.

[0037] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. An end-to-end medical record information extraction method based on a multimodal large model, characterized by: Includes the following steps: Step 1: Data Integration and Preprocessing Multimodal data integration: Integrating data from different sources and types, such as outpatient medical records, inpatient medical records, CT scan records, and laboratory test results, to enhance the information richness of the model; Step 2: Model Architecture Design Basic architecture: It adopts the basic architecture of PaLM-E and uses the ViT pre-trained model as a visual encoder to achieve joint representation learning of images and text; Model Combinations: Enable model combinations of different scales, such as PaLM 8B+ViT 4B and PaLM 62B+ViT22B, to adapt to different computing and performance requirements; Step 3: End-to-end pre-training Unified model architecture: Use a unified model architecture and parameters to execute multiple tasks through multi-modal input; Instructions and Examples: Provide the model with task-specific instructions and one-time examples to guide the model in learning how to perform a specific task; Step 4: Fine-tuning and optimization End-to-end fine-tuning: In multimodal tasks, image and text labels are interleaved to form a multimodal context input to the model, enabling end-to-end fine-tuning; Step 5: Performance evaluation.

2. The end-to-end medical record information extraction method based on a multimodal large model according to claim 1, characterized in that: The performance evaluation in step five assesses the model's performance on different tasks and datasets.

3. The end-to-end medical record information extraction method based on a multimodal large model according to claim 1, characterized in that: The tasks in step three include entity extraction, question answering, and report generation.