Medical image aided decision-making method and system, electronic equipment and storage medium

Abstract

The embodiment of the invention provides a medical image aided decision-making method and system, electronic equipment and a storage medium, and belongs to the field of medical treatment. The method comprises the following steps: collecting multi-modal medical data, wherein the multi-modal medical data comprises a medical image and corresponding text data; constructing a multi-modal large language model, and performing pre-training to obtain a target multi-modal large language model; extracting image and text feature vectors corresponding to the medical image and the text data by using the target multi-modal large language model, and fusing the image and text feature vectors to obtain a fused feature vector; and according to different clinical tasks, adopting models corresponding to the clinical tasks, and combining with the fusion feature vector to obtain an auxiliary decision scheme. By fusing detailed clinical information in medical images and text data, the multi-modal large language model can more comprehensively understand the health condition of a patient, so that more accurate and comprehensive diagnosis suggestions are provided, the model configuration can be flexibly adjusted for different clinical tasks, and the diagnosis efficiency is improved. And a targeted auxiliary decision-making scheme is generated in combination with the fusion feature vector.

Description

Technical Field

[0001] The present invention relates to the field of medical technology, and in particular to a medical image-assisted decision-making method, system, electronic equipment and storage medium. Background Art

[0002] Medical imaging CDSS (Clinical Decision Support System) uses computer technology to analyze and interpret medical images to assist doctors in diagnosis and treatment. Conventional solutions are mainly based on single-modality data, such as X-rays, CT, MRI images or electronic medical record texts, and use deep learning and machine learning algorithms to perform image classification, lesion detection and diagnosis prediction. These systems usually require a large amount of labeled data for training, but often lack interpretability and personalization, and have limited interactive experience and real-time update capabilities. In addition, data privacy protection is also an important technical challenge.

[0003] At present, conventional medical imaging CDSS solutions mainly rely on single-modality data (images or text), require a large amount of labeled data, use black-box models, lack interpretability and personalized suggestions, have poor interactivity, are difficult to update in real time, and have insufficient privacy protection. These limitations affect diagnostic accuracy, credibility, and data security. Summary of the invention

[0004] The purpose of the embodiments of the present invention is to provide a medical image-assisted decision-making method, system, electronic device and storage medium, which are used to fully or at least partially solve the technical problems existing in the above-mentioned prior art.

[0005] In order to achieve the above object, an embodiment of the present invention provides a medical image-assisted decision-making method, comprising:

[0006] Collecting multimodal medical data, wherein the multimodal medical data includes medical images and corresponding text data;

[0007] Constructing a multimodal large language model, and pre-training the multimodal large language model using historical medical images and text data to obtain a target multimodal large language model;

[0008] Using the target multimodal large language model, respectively extracting image feature vectors and text feature vectors corresponding to the medical image and text data, and fusing the image feature vectors and text feature vectors to obtain a fused feature vector;

[0009] According to different clinical tasks, corresponding models are adopted and combined with fusion feature vectors to obtain auxiliary decision-making solutions.

[0010] Optionally, the image data is subjected to denoising, contrast enhancement and normalization processing to obtain processed image data;

[0011] The text data is segmented, stop words are removed, and standardized to obtain processed text data.

[0012] Optionally, constructing a multimodal large language model, and pre-training the multimodal large language model using historical medical images and text data to obtain a target multimodal large language model, including:

[0013] Based on the architecture of the GPT-4 model, a multimodal input layer is designed and a visual transformer is integrated to enable the multimodal large language model to learn the association between medical images and text. For image data, a pre-trained visual transformer is used to extract image feature vectors, and for text data, the Tokenizer of the GPT-4 model is used to process and obtain text feature vectors.

[0014] Construct a feature fusion mechanism to fuse image feature vectors and text feature vectors;

[0015] Design a cross entropy loss function to evaluate the performance of multimodal large language models on pre-training tasks;

[0016] Design output layers according to different clinical tasks;

[0017] After fusing the historical medical image diagnosis vectors and the text data feature vectors, the multimodal large language model is iteratively trained. In each iteration, the parameters of the multimodal large language model are updated to minimize the loss function and obtain the target multimodal large language model.

[0018] Optionally, using the target multimodal large language model to respectively extract image feature vectors and text feature vectors corresponding to the medical image and text data, and fusing the image feature vectors and text feature vectors to obtain a fused feature vector, including:

[0019] The query vector, key vector and value vector in the feature fusion mechanism are used to adjust the weights of the image feature vector and the text feature vector;

[0020] The image feature vector and the text feature vector are weighted and summed according to the weights to obtain the fused feature representation.

[0021] Optionally, according to different clinical tasks, corresponding models are adopted and combined with fusion feature vectors to obtain auxiliary decision-making solutions, including:

[0022] For diagnosis prediction and classification tasks, the classifier combines the image feature vector and the text feature vector to output the prediction result, and recommends auxiliary decision-making solutions based on the prediction result;

[0023] For diagnostic prediction regression tasks, the lesion probability is predicted through the regression model, and auxiliary decision-making plans are recommended based on the predicted lesion probability;

[0024] For disease detection tasks, the YOLO detection model is used to detect lesion areas in medical images, mark the lesion areas, and recommend auxiliary decision-making plans for the lesion areas.

[0025] On the other hand, the present invention also provides a medical image-assisted decision-making system, comprising:

[0026] A collection unit, used for collecting multimodal medical data, wherein the multimodal medical data includes medical images and corresponding text data;

[0027] A model building unit, used to build a multimodal large language model, and pre-train the multimodal large language model using historical medical images and text data to obtain a target multimodal large language model;

[0028] A feature fusion unit, used to extract image feature vectors and text feature vectors corresponding to medical images and text data respectively using a target multimodal large language model, and fuse the image feature vectors and text feature vectors to obtain a fused feature vector;

[0029] The decision-making unit is used to adopt the corresponding model according to different clinical tasks, combine the fusion feature vector, and obtain the auxiliary decision-making plan.

[0030] Optionally, the model building unit is specifically used for:

[0031] Based on the architecture of the GPT-4 model, a multimodal input layer is designed and a visual transformer is integrated to enable the multimodal large language model to learn the association between medical images and text. For image data, a pre-trained visual transformer is used to extract image feature vectors, and for text data, the Tokenizer of the GPT-4 model is used to process and obtain text feature vectors.

[0032] Construct a feature fusion mechanism to fuse image feature vectors and text feature vectors;

[0033] Design a cross entropy loss function to evaluate the performance of multimodal large language models on pre-training tasks;

[0034] Design output layers according to different clinical tasks;

[0035] After fusing the historical medical image diagnosis vectors and the text data feature vectors, the multimodal large language model is iteratively trained. In each iteration, the parameters of the multimodal large language model are updated to minimize the loss function and obtain the target multimodal large language model.

[0036] Optionally, the feature fusion unit is specifically used for:

[0037] The query vector, key vector and value vector in the feature fusion mechanism are used to adjust the weights of the image feature vector and the text feature vector;

[0038] The image feature vector and the text feature vector are weighted and summed according to the weights to obtain the fused feature representation.

[0039] On the other hand, the present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above-mentioned medical image-assisted decision-making method when executing the program.

[0040] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the medical image-assisted decision-making method described above are implemented.

[0041] Through the above technical solutions, by integrating detailed clinical information in medical images and text data, the multimodal large language model can more comprehensively understand the patient's health status, thereby providing more accurate and comprehensive diagnostic recommendations, and can flexibly adjust the model configuration for different clinical tasks, and generate targeted auxiliary decision-making solutions in combination with fusion feature vectors, which helps doctors make more scientific and reasonable decisions in complex cases, and generate personalized auxiliary decision-making solutions to meet the unique needs of different patients. Comprehensive analysis of multimodal information helps to discover subtle abnormalities or associations that may be overlooked by single-modal data, thereby reducing the risk of misdiagnosis and missed diagnosis.

[0042] Other features and advantages of the embodiments of the present invention will be described in detail in the subsequent detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] The accompanying drawings are used to provide a further understanding of the embodiments of the present invention and constitute a part of the specification. Together with the following specific implementations, they are used to explain the embodiments of the present invention, but do not constitute a limitation on the embodiments of the present invention. In the accompanying drawings:

[0044] Figure 1 is an implementation flow chart of a medical image-assisted decision-making method provided by an embodiment of the present invention;

[0045] Figure 2 It is a structural diagram of a medical image auxiliary decision-making system provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0046] The specific implementation of the embodiment of the present invention is described in detail below in conjunction with the accompanying drawings. It should be understood that the specific implementation described here is only used to illustrate and explain the embodiment of the present invention, and is not used to limit the embodiment of the present invention.

[0047] See also Figure 1 FIG. 1 is a flowchart of an implementation of a medical image-assisted decision-making method provided by an embodiment of the present invention, comprising the following execution steps:

[0048] Step 100: Collect multimodal medical data, where the multimodal medical data includes medical images and corresponding text data.

[0049] Preferably, the medical imaging data includes but is not limited to X-rays, MRI and CT scans, and the corresponding text data includes but is not limited to medical records, laboratory results and doctor's reports.

[0050] Specifically, when executing step 100, the image data is also subjected to denoising, contrast enhancement and normalization processing to obtain processed image data; and the text data is subjected to word segmentation, stop word removal and normalization processing to obtain processed text data.

[0051] Step 101: construct a multimodal large language model, and use historical medical images and text data to pre-train the multimodal large language model to obtain a target multimodal large language model.

[0052] Specifically, when executing step 101, the following steps may be specifically performed:

[0053] S1010: Based on the architecture of the GPT-4 model, a multimodal input layer is designed and a visual Transformer is integrated to enable the multimodal large language model to learn the relationship between medical images and text. For image data, a pre-trained visual Transformer is used to extract image feature vectors. For text data, the Tokenizer of the GPT-4 model is used to process it to obtain text feature vectors.

[0054] S1011: Construct a feature fusion mechanism to fuse image feature vectors and text feature vectors.

[0055] S1012: Design a cross-entropy loss function to evaluate the performance of large multimodal language models on pre-training tasks.

[0056] Specifically, the cross entropy loss function formula is as follows:

[0057]

[0058] In the formula, C represents the total number of categories, i represents the i-th category, and y i represents the sample belonging to the i-th class, is the probability distribution predicted by the multimodal large language model.

[0059] S1013: Design output layers according to different clinical tasks.

[0060] S1014: After fusing the historical medical image diagnosis vector and the text data feature vector, the multimodal large language model is iteratively trained. Each iteration updates the multimodal large language model parameters to minimize the loss function and obtain the target multimodal large language model.

[0061] Step 102: Utilize the target multimodal large language model to extract image feature vectors and text feature vectors corresponding to the medical image and text data, respectively, and fuse the image feature vectors and text feature vectors to obtain a fused feature vector.

[0062] Specifically, when executing step 102, the following steps may be specifically performed:

[0063] S1020: Using the query vector, key vector, and value vector in the feature fusion mechanism to adjust the weights of the image feature vector and the text feature vector.

[0064] S1021: Perform weighted summation on the image feature vector and the text feature vector according to the weights to obtain a fused feature representation.

[0065] Step 103: According to different clinical tasks, the corresponding models are adopted and combined with the fusion feature vectors to obtain auxiliary decision-making solutions.

[0066] Specifically, when executing step 103, the following steps may be specifically performed:

[0067] S1030: For the diagnosis prediction classification task, the classifier combines the image feature vector and the text feature vector to output the prediction result, and recommends an auxiliary decision-making plan based on the prediction result.

[0068] Preferably, a multi-layer perceptron or a Softmax classifier can be used for the diagnosis prediction classification task.

[0069] S1031: For the diagnostic prediction regression task, the lesion probability is predicted through the regression model, and an auxiliary decision-making plan is recommended based on the predicted lesion probability.

[0070] It should be understood that the regression model can be an existing regression model.

[0071] S1032: For the disease detection task, use the YOLO detection model to detect the lesion area in the medical image, mark the lesion area, and recommend auxiliary decision-making solutions for the lesion area.

[0072] Specifically, the process of using the YOLO detection model to detect lesion areas in medical images includes: Grid division: YOLO divides the input image into multiple grids, each grid is responsible for detecting objects whose center falls in it. Bounding box prediction: Each grid predicts a certain number of bounding boxes, as well as the coordinates, size, confidence, and category probability of each bounding box. Confidence: Indicates the probability of an object existing in the bounding box and the product of the intersection and union ratio between the bounding box and the real object. Category prediction: YOLOv3 and later versions use logistic regression to predict category probabilities and support multi-label detection. Post-processing: Remove duplicate bounding boxes through the non-maximum suppression (NMS) algorithm and select the final target box.

[0073] Through the above technical solutions, by integrating detailed clinical information in medical images and text data, the multimodal large language model can more comprehensively understand the patient's health status, thereby providing more accurate and comprehensive diagnostic recommendations, and can flexibly adjust the model configuration for different clinical tasks, and generate targeted auxiliary decision-making solutions in combination with fusion feature vectors, which helps doctors make more scientific and reasonable decisions in complex cases, and generate personalized auxiliary decision-making solutions to meet the unique needs of different patients. Comprehensive analysis of multimodal information helps to discover subtle abnormalities or associations that may be overlooked by single-modal data, thereby reducing the risk of misdiagnosis and missed diagnosis.

[0074] In some embodiments, see Figure 2 FIG. 1 is a schematic diagram of a medical image decision-making assistance system provided by an embodiment of the present invention, comprising:

[0075] The acquisition unit 200 is used to acquire multimodal medical data, wherein the multimodal medical data includes medical images and corresponding text data;

[0076] A model building unit 201 is used to build a multimodal large language model and pre-train the multimodal large language model using historical medical images and text data to obtain a target multimodal large language model;

[0077] A feature fusion unit 202 is used to extract image feature vectors and text feature vectors corresponding to the medical image and text data respectively using the target multimodal large language model, and fuse the image feature vectors and text feature vectors to obtain a fused feature vector;

[0078] The decision unit 203 is used to adopt the corresponding model according to different clinical tasks, combine the fusion feature vector, and obtain the auxiliary decision plan.

[0079] In some implementations, the model building unit 201 is specifically configured to:

[0080] Based on the architecture of the GPT-4 model, a multimodal input layer is designed and a visual transformer is integrated to enable the multimodal large language model to learn the association between medical images and text. For image data, a pre-trained visual transformer is used to extract image feature vectors, and for text data, the Tokenizer of the GPT-4 model is used to process and obtain text feature vectors.

[0081] Construct a feature fusion mechanism to fuse image feature vectors and text feature vectors;

[0082] Design a cross entropy loss function to evaluate the performance of multimodal large language models on pre-training tasks;

[0083] Design output layers according to different clinical tasks;

[0084] After fusing the historical medical image diagnosis vectors and the text data feature vectors, the multimodal large language model is iteratively trained. In each iteration, the parameters of the multimodal large language model are updated to minimize the loss function and obtain the target multimodal large language model.

[0085] In some implementations, the feature fusion unit 202 is specifically configured to:

[0086] The query vector, key vector and value vector in the feature fusion mechanism are used to adjust the weights of the image feature vector and the text feature vector;

[0087] The image feature vector and the text feature vector are weighted and summed according to the weights to obtain the fused feature representation.

[0088] On the other hand, the present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the medical image-assisted decision-making method described in any one of the above-mentioned embodiments when executing the program.

[0089] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the medical image-assisted decision-making method described in any of the above embodiments are implemented.

[0090] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0091] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0092] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0093] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0094] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0095] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0096] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0097] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0098] The above are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included within the scope of the claims of the present application.

Claims

1. A medical image-assisted decision-making method, characterized in that: include: Collecting multimodal medical data, wherein the multimodal medical data includes medical images and corresponding text data; Constructing a multimodal large language model, and pre-training the multimodal large language model using historical medical images and text data to obtain a target multimodal large language model; Using the target multimodal large language model, respectively extracting image feature vectors and text feature vectors corresponding to the medical image and text data, and fusing the image feature vectors and text feature vectors to obtain a fused feature vector; According to different clinical tasks, corresponding models are adopted and combined with fusion feature vectors to obtain auxiliary decision-making solutions.

2. The medical image-assisted decision-making method according to claim 1, characterized in that: Performing de-noising, contrast enhancement and normalization processing on the medical images to obtain processed image data; The text data is segmented, stop words are removed, and standardized to obtain processed text data.

3. The medical image-assisted decision-making method according to claim 1, characterized in that: Constructing a multimodal large language model, and pre-training the multimodal large language model using historical medical images and text data to obtain a target multimodal large language model, including: Based on the architecture of the GPT-4 model, a multimodal input layer is designed and a visual transformer is integrated to enable the multimodal large language model to learn the association between medical images and text. For image data, a pre-trained visual transformer is used to extract image feature vectors, and for text data, the Tokenizer of the GPT-4 model is used to process and obtain text feature vectors. Construct a feature fusion mechanism to fuse image feature vectors and text feature vectors; Design a cross entropy loss function to evaluate the performance of multimodal large language models on pre-training tasks; Design output layers according to different clinical tasks; After fusing the historical medical image diagnosis vectors and the text data feature vectors, the multimodal large language model is iteratively trained. In each iteration, the parameters of the multimodal large language model are updated to minimize the loss function and obtain the target multimodal large language model.

4. The medical image-assisted decision-making method according to claim 3, characterized in that: The target multimodal large language model is used to extract image feature vectors and text feature vectors corresponding to the medical image and text data respectively, and the image feature vectors and text feature vectors are fused to obtain a fused feature vector, including: The query vector, key vector and value vector in the feature fusion mechanism are used to adjust the weights of the image feature vector and the text feature vector; The image feature vector and the text feature vector are weighted and summed according to the weights to obtain the fused feature representation.

5. The medical image-assisted decision-making method according to claim 1, characterized in that: According to different clinical tasks, corresponding models are adopted and combined with fusion feature vectors to obtain auxiliary decision-making solutions, including: For diagnosis prediction and classification tasks, the classifier combines the image feature vector and the text feature vector to output the prediction result, and recommends auxiliary decision-making solutions based on the prediction result; For diagnostic prediction regression tasks, the lesion probability is predicted through the regression model, and auxiliary decision-making plans are recommended based on the predicted lesion probability; For disease detection tasks, the YOLO detection model is used to detect lesion areas in medical images, mark the lesion areas, and recommend auxiliary decision-making plans for the lesion areas.

6. A medical image decision-making assistance system, characterized in that: include: A collection unit, used for collecting multimodal medical data, wherein the multimodal medical data includes medical images and corresponding text data; A model building unit, used to build a multimodal large language model, and pre-train the multimodal large language model using historical medical images and text data to obtain a target multimodal large language model; A feature fusion unit, used to extract image feature vectors and text feature vectors corresponding to medical images and text data respectively using a target multimodal large language model, and fuse the image feature vectors and text feature vectors to obtain a fused feature vector; The decision-making unit is used to adopt the corresponding model according to different clinical tasks, combine the fusion feature vector, and obtain the auxiliary decision-making plan.

7. The medical image-assisted decision-making system according to claim 6, characterized in that: The model building unit is specifically used to: Based on the architecture of the GPT-4 model, a multimodal input layer is designed and a visual transformer is integrated to enable the multimodal large language model to learn the association between medical images and text. For image data, a pre-trained visual transformer is used to extract image feature vectors, and for text data, the Tokenizer of the GPT-4 model is used to process and obtain text feature vectors. Construct a feature fusion mechanism to fuse image feature vectors and text feature vectors; Design a cross entropy loss function to evaluate the performance of multimodal large language models on pre-training tasks; Design output layers according to different clinical tasks; After fusing the historical medical image diagnosis vectors and the text data feature vectors, the multimodal large language model is iteratively trained. In each iteration, the parameters of the multimodal large language model are updated to minimize the loss function and obtain the target multimodal large language model.

8. The medical image-assisted decision-making system according to claim 6, characterized in that: The feature fusion unit is specifically used for: The query vector, key vector and value vector in the feature fusion mechanism are used to adjust the weights of the image feature vector and the text feature vector; The image feature vector and the text feature vector are weighted and summed according to the weights to obtain the fused feature representation.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of the medical image-assisted decision-making method as described in any one of claims 1 to 5 are implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the medical image-assisted decision-making method as described in any one of claims 1 to 5 are implemented.

Images