High-quality medical multi-modal data synthesis method, device, equipment and medium

Abstract

The invention discloses a high-quality medical multi-modal data synthesis method and device, equipment and a medium. By integrating medical data from different modalities, more comprehensive and deep medical information collection is realized, and a rich data basis is provided for subsequent data synthesis. According to the method, disease-related data is extracted from a high-quality image and image description content thereof, so that more disease diagnosis selection questions can be generated subsequently based on the disease-related data through a data synthesis method to enrich training data of a large medical model, thereby making up for the defects of the model in clinical medicine and disease diagnosis, and improving the accuracy of disease diagnosis. And the performance of the medical large model is obviously improved. Moreover, the synthesis data improves the diversity of the medical data set, so that the large medical model can obtain different types of training data, the diversity of the training data is greatly improved, and the possibility is provided for improving the performance of the large medical model.

Description

Technical Field

[0001] The present application relates to the fields of artificial intelligence and data processing, and in particular to a high-quality medical multimodal data synthesis method, device, equipment and medium. Background Art

[0002] In the medical field, multimodal data (including images, laboratory results, clinical texts, etc.) plays a vital role in artificial intelligence (AI) in intelligent clinical decision support, disease monitoring and analysis. This type of data can provide comprehensive and rich patient information, which helps AI systems to more accurately understand the condition, predict the development of the disease, and develop personalized treatment plans. However, in practical applications, the acquisition of medical multimodal data faces many challenges.

[0003] On the one hand, due to the sensitivity of medical data, especially information involving patient privacy, its acquisition and use must strictly comply with relevant regulations, which has led many medical institutions to be cautious when sharing data, making open source medical data resources relatively scarce. Even if open source data is available, its coverage is often limited and it is difficult to meet the diversity and scale requirements required for AI model training.

[0004] On the other hand, existing medical data mainly exists in the form of a single modality, such as plain text or pure image data. However, medical data is diverse, covering multiple aspects such as medical imaging, genomes, clinical texts, etc. In order to make full use of these multi-modal data, they need to be synthesized into multimodal datasets. In this process, the connection and complementarity between data of different modalities must be taken into account to ensure that the synthesized data is closely related and unambiguous. However, the quality of medical multimodal data is currently uneven, with various noises and inconsistencies, which poses a great challenge to data synthesis.

[0005] Therefore, how to overcome the limitations of existing technologies and synthesize high-quality medical multimodal datasets has become an important issue that needs to be urgently addressed in the current field of medical AI. Summary of the invention

[0006] The present application provides a high-quality medical multimodal data synthesis method, apparatus, equipment and medium, which are used to solve the problems that the current open source medical data resources are relatively scarce, the quality of medical multimodal data is uneven, and there are various noises and inconsistencies, which are not conducive to the training of medical AI models.

[0007] In a first aspect, the present application provides a high-quality medical multimodal data synthesis method, the method comprising:

[0008] Acquire multimodal medical data;

[0009] extracting all images and their context from the medical data;

[0010] For all the images, the image description content related to the image is extracted from the context of the image through a pre-trained large language model based on a preset refined content text prompt; the matching degree between the image and the image description content is determined through a pre-trained visual language large model based on a preset image-text matching text prompt, and when it is determined that the matching degree meets the pre-configured retention condition, the image and the image description content are retained; if the target disease related to the image description content is obtained through the large language model based on a preset disease diagnosis text prompt, a preset number of similar diseases similar to the target disease and belonging to the same department are generated through the large language model based on a preset disease extension text prompt; the image description content is combined with the target disease and each of the similar diseases to form a disease diagnosis multiple-choice question, so that the disease diagnosis multiple-choice question and its corresponding diagnosis answer are used as sample data for training the medical large model; wherein the diagnosis answer corresponding to the disease diagnosis multiple-choice question is the target disease.

[0011] In a second aspect, the present application also provides a high-quality medical multimodal data synthesis device, the device comprising:

[0012] An acquisition unit, used for acquiring multimodal medical data;

[0013] an extraction unit, configured to extract all images and their contexts from the medical data;

[0014] A processing unit is used to extract image description content related to the image from the context of the image through a pre-trained large language model and based on a preset refined content text prompt for all the images; determine the degree of match between the image and the image description content through a pre-trained visual language large model and based on a preset text prompt for image-text matching, and retain the image and the image description content when it is determined that the degree of match meets a pre-configured retention condition; if a target disease related to the image description content is obtained through the large language model based on a preset disease diagnosis text prompt, then generate a preset number of similar diseases similar to the target disease and belonging to the same department through the large language model based on a preset disease extension text prompt; combine the image description content with the target disease and each of the similar diseases to form a disease diagnosis multiple-choice question, so that the disease diagnosis multiple-choice question and its corresponding diagnosis answer are used as sample data for training the medical large model; wherein the diagnosis answer corresponding to the disease diagnosis multiple-choice question is the target disease.

[0015] In a third aspect, the present application provides a computer device, comprising a processor, wherein the processor is used to implement the steps of the high-quality medical multimodal data synthesis method as described above when executing a computer program stored in a memory.

[0016] In a fourth aspect, the present application provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the high-quality medical multimodal data synthesis method as described above.

[0017] The beneficial effects of this application are as follows:

[0018] 1. By integrating medical data from different modalities, more comprehensive and in-depth medical information collection can be achieved, providing a rich data foundation for subsequent data synthesis.

[0019] 2. Extract all images and their contexts from a large amount of medical data to eliminate a large amount of redundant information in the medical data.

[0020] 3. Use a pre-trained large language model to extract the image description content of the image from the context of the image to remove the noise and redundancy in the context, that is, filter out redundant data in the context and remove unnecessary parts.

[0021] 4. Use the pre-trained visual language model to evaluate the match between each image and its image description based on the preset text prompts for image-text matching. If the match meets the pre-configured retention conditions, the image and its description are retained to ensure a high degree of consistency between the image and its description, thereby improving the overall quality of subsequent data synthesis.

[0022] 5. Extract disease-related data from high-quality images and their image descriptions, so that more disease diagnosis multiple-choice questions can be generated based on the disease-related data through data synthesis in the future, so as to enrich the training data of the medical big model, thereby making up for the shortcomings of the medical big model in clinical medicine and disease diagnosis, and significantly improving the performance of the model. In addition, synthetic data improves the diversity of medical data sets, enabling the medical big model to obtain different types of training data, greatly increasing the diversity of training data, and thus providing the possibility of improving the performance of the medical big model. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0024] Figure 1 A schematic diagram of a process for synthesizing high-quality medical multimodal data provided in an embodiment of the present application;

[0025] Figure 2 A schematic diagram of the structure of a high-quality medical multimodal data synthesis device provided in an embodiment of the present application;

[0026] Figure 3 It is a structural schematic diagram of a computer device provided in an optional embodiment of the present application. DETAILED DESCRIPTION

[0027] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application.

[0028] In order to effectively extract, generate and combine medical data for the training of large medical models and improve their intelligent diagnostic capabilities, the present application provides a method, apparatus, device and medium.

[0029] Embodiment 1:

[0030] This application provides a high-quality medical multimodal data synthesis method. Figure 1 A schematic diagram of a process for synthesizing high-quality medical multimodal data provided in an embodiment of the present application, the process comprising:

[0031] S101: Acquire multimodal medical data.

[0032] The high-quality medical multimodal data synthesis method provided in this application is applied to a computer device, which can be an intelligent device, such as a mobile terminal, a computer, etc., or a server, such as an application server, a business server, etc.

[0033] In order to ensure the diversity of subsequent data synthesis, in this application, a large amount of multimodal medical data can be obtained. Among them, the multimodality includes the following: images and text. These medical data can include medical images (such as X-rays, CT, MRI images, etc.), clinical text records, laboratory test results, etc. The data source can be medical equipment, electronic health record systems, hospital databases, etc. For example, a large amount of medical data is collected from PubMed Central (National Library of Medicine, referred to as PMC) using crawler technology.

[0034] In a possible implementation, the large amount of medical data may include corpora in multiple languages ​​to increase data diversity. For example, the large amount of medical data includes corpora such as papers and books in both English and Chinese.

[0035] S102: Extracting all images and their contexts from the medical data.

[0036] After acquiring a large amount of medical data, all images and their contexts can be extracted from the medical data to eliminate a large amount of redundant information in the medical data. Exemplarily, all included medical images and their associated contexts can be identified by writing regular expressions or using appropriate data extraction tools. The context can include patient medical records, doctor notes, laboratory test results, etc.

[0037] In a possible implementation, a label corresponding to each image may be inserted into the context corresponding to the image, so that the image and the context of the image can be uniquely located through the image label, thereby facilitating subsequent processing.

[0038] S103: For all the images, extract the image description content related to the image from the context of the image through the pre-trained large language model based on the preset refined content text prompt; determine the matching degree between the image and the image description content through the pre-trained visual language large model based on the preset image-text matching text prompt, and retain the image and the image description content when it is determined that the matching degree meets the pre-configured retention condition; if the target disease related to the image description content is obtained through the large language model based on the preset disease diagnosis text prompt, then generate a preset number of similar diseases similar to the target disease and belonging to the same department through the large language model based on the preset disease extension text prompt; combine the image description content with the target disease and each of the similar diseases to form a disease diagnosis multiple-choice question, so as to use the disease diagnosis multiple-choice question and its corresponding diagnosis answer as sample data for training the medical large model; wherein the diagnosis answer corresponding to the disease diagnosis multiple-choice question is the target disease.

[0039] For ease of illustration, for each extracted image, the following operations are performed:

[0040] S103a: Using a pre-trained large language model, extract the image description content of the image from the context of the image to remove noise and redundancy in the context, that is, filter out redundant data in the context and remove unnecessary parts.

[0041] In this process, the large language model can be guided to extract valuable information from the context of the image through the preset extraction content text prompt. For example, the preset extraction content text prompt is: "You will play the role of a professional data processing tool. Please extract the image label relevant image description (Caption) and delete unnecessary information".

[0042] In a possible implementation, the trained large language model can be obtained by fine-tuning the instructions of the basic large language model, so that the trained large language model can extract image description content related to the image from the input text content according to the preset text prompt for the extracted content. The basic large language model is pre-trained in advance by predicting the next token, and may include an encoding layer and a decoding layer. The encoding layer is responsible for converting the input text data into higher-dimensional feature data that can better reflect the semantic information in the text. The decoding layer is responsible for decoding the feature data output by the encoding layer into a readable text output.

[0043] Exemplarily, during the training process, a large number of text-image pairs (including images and their corresponding contexts) can be used as training data, and the training data corresponds to the expected image description content. These training data are used to fine-tune the instructions of the basic large language model so that it can better understand and perform the task of refining the image description content. Specifically, the input training data and the preset refined content text prompt are first input into the basic large language model. Then, the basic large language model uses the encoding layer to extract features from the input text data (including text content and refined content text prompts), and uses the decoding layer to decode the feature data into a readable text output. Finally, the error between the text output of the basic large language model and the expected image description content is calculated, and the parameters of the basic large language model are fine-tuned using the back propagation algorithm.

[0044] It should be noted that the basic large language model can be pre-trained by predicting the next token.

[0045] Through multiple iterations of training, the basic large language model gradually learned how to extract image description content related to the image from the input text content according to the preset extraction content text prompts. The realization of this process not only improves the accuracy and robustness of the basic large language model, but also provides strong support for the subsequent medical multimodal data synthesis.

[0046] S103b: Use the pre-trained visual language model to evaluate the matching degree between each image and its image description based on the preset image-text matching text prompt. If the matching degree meets the pre-configured retention conditions, the image and its description are retained to ensure a high degree of consistency between the image and its description, so as to improve the overall quality of subsequent data synthesis.

[0047] In this process, in order to accurately screen out high-quality images and their corresponding image descriptions, a pre-trained visual language model is pre-selected. The visual language model should have the ability to process image and text data and establish an effective association between the two. In addition, a text-image matching text prompt needs to be pre-designed to guide the visual language model to determine the degree of match between the image and the image description of the image, that is, to determine the degree of association between the image and the image description. For example, the preset text-image matching text prompt is: "You will play the role of a professional data processing tool. For the given image and Caption, you need to judge the degree of match. If the match is low, return False, otherwise return True."

[0048] In a possible implementation, the retention condition is that the degree of match is greater than a preset degree of match threshold, that is, if the degree of match between the image and the image description content of the image is greater than the preset degree of match threshold, it means that the image and the image description content are of high quality, then it is determined that the degree of match meets the pre-configured retention condition, and the image and the image description content are retained. If the degree of match between the image and the image description content of the image is not greater than the preset degree of match threshold, it means that the image and the image description content are of low quality, then it is determined that the degree of match does not meet the pre-configured retention condition, and the image and the image description content are removed.

[0049] In a possible implementation, the trained visual language model can be obtained by fine-tuning the instructions of the basic visual language model, so that the trained visual language model can match the image with the image description content according to the preset image-text matching text prompt, and can eliminate the image with low matching degree and its corresponding image description content. Among them, the basic visual language model may include an encoding layer and a decoding layer. The encoding layer is responsible for converting the input data into higher-dimensional feature data, which can better reflect the semantic information in the input data. The decoding layer is responsible for decoding the feature data output by the encoding layer into a readable text output.

[0050] Exemplarily, during the training process, a large number of description content text image pairs (including images and their corresponding image description content) can be used as training data, and the training data corresponds to the expected matching degree. Use these training data to fine-tune the instructions of the basic visual language large model so that it can better understand and perform the task of refining the image description content. Specifically, first, the input training data and the preset image-text matching text prompts are input into the basic visual language large model together. Then, the basic visual language large model uses the encoding layer to extract features from the input text data (including training data and image-text matching text prompts), and uses the decoding layer to decode the feature data to obtain the actual matching degree. Finally, the error between the actual matching value output by the basic visual language large model and the expected matching value is calculated, and the parameters of the basic visual language large model are fine-tuned using the back propagation algorithm.

[0051] Through multiple iterations of training, the basic visual language model gradually learned how to match images with image descriptions based on the preset text-image matching prompts, and to remove images with low matching degrees and their corresponding image descriptions. The realization of this process not only improves the accuracy and robustness of the basic visual language model, but also provides strong support for subsequent medical multimodal data synthesis.

[0052] It should be noted that in addition to fine-tuning instructions, the visual language model also has to perform a pre-task: alignment. This process is to enable the visual language model to understand the meaning of a given picture, so as to prepare for subsequent data synthesis.

[0053] S103c: If a target disease related to the image description content is obtained through the large language model based on the preset disease diagnosis text prompt, then a preset number of approximate diseases similar to the target disease and belonging to the same department are generated through the large language model based on the preset disease extension text prompt.

[0054] With the continuous advancement of medical technology, artificial intelligence is increasingly used in the medical field. However, data related to disease diagnosis is relatively scarce, which limits the accuracy and reliability of medical big models in diagnosis. In order to solve this problem, disease-related data can be extracted from high-quality images and their image descriptions, so that more disease diagnosis multiple-choice questions can be generated based on the disease-related data through data synthesis methods to enrich the training data of medical big models.

[0055] In order to obtain disease-related data, in this application, it is first necessary to identify whether the image description content is related to the disease. Exemplarily, a pre-trained large language model can be used to identify whether the image description content is related to the disease, and the target disease corresponding to the image description content when it is related to the disease, based on the preset disease diagnosis text prompt. The implementation of this step depends on the understanding and application of the large language model to the professional knowledge in the medical field. For example, the preset disease diagnosis text prompt may be set as: "You will play the role of a professional radiologist. Please determine whether the Caption is related to the disease. If so, return True and the related disease, otherwise return False". Through such disease diagnosis text prompts, the large language model can accurately identify whether the image description content is related to the disease, and when it is related to the disease, the specific target disease it corresponds to.

[0056] Once it is determined that the image description content is related to the disease, that is, the image description content and the image constitute disease-related data, the next step is to generate similar diseases that are similar to the target disease and belong to the same department. To achieve this goal, the fine-tuned large language model can be used again, but this time based on a preset disease extension text prompt. For example, the preset disease extension text prompt may be set as: "Please generate three names similar to this disease, and they must belong to the same department." With such a prompt, the model can generate a preset number of similar diseases that are similar to the target disease and belong to the same department based on the relevant knowledge of the target disease.

[0057] It should be noted that the large language model has been fine-tuned based on the similar method in the above embodiment before executing the disease diagnosis task and the extended disease approximation task. This fine-tuning process ensures that the large language model can accurately understand and execute the preset text prompts, thereby generating high-quality and qualified disease diagnosis multiple-choice questions and approximate diseases.

[0058] S103d: Combine the image description content with the target disease and each of the similar diseases to form disease diagnosis multiple-choice questions, and use the disease diagnosis multiple-choice questions and their corresponding diagnostic answers as sample data for training the medical big model; wherein the diagnostic answer corresponding to the disease diagnosis multiple-choice questions is the target disease.

[0059] After determining the target disease and similar diseases, the next step is to combine this information with the image description content to construct a disease diagnosis multiple-choice question. The disease diagnosis multiple-choice question includes the target disease and all selected similar diseases as options to test the model's ability to identify diseases.

[0060] For each constructed disease diagnosis multiple-choice question, its corresponding diagnostic answer, i.e., the target disease, needs to be clearly labeled. This step is key to ensuring the accuracy and validity of the training sample data. During the labeling process, the answer to each multiple-choice question should be strictly checked to avoid training bias caused by incorrect labeling.

[0061] After completing the construction of all disease diagnosis multiple-choice questions and the annotation of answers, the sample data are integrated to form a complete training data set. During the integration process, the sample data needs to be verified to ensure the integrity, consistency and accuracy of the data. The verification can include checking the data format, reviewing the correctness of the answers, and verifying the logical consistency between the sample data.

[0062] The above-mentioned synthetic disease diagnosis multiple-choice questions and their corresponding diagnosis answers provide high-quality sample data for the training of the medical big model. This not only helps to improve the accuracy and generalization ability of the model in disease diagnosis, but also lays a solid foundation for promoting the development of medical intelligence.

[0063] In one example, for all the images, if the large language model determines, based on the disease diagnosis text prompt, that the image description content is not related to the disease, the method further includes:

[0064] According to the image and the image description content, the aligned visual question and answer and the instruction visual question and answer for each preset scene are constructed.

[0065] After introducing a large language model to judge the relevance of image description content to the disease, if it is determined that the description content of an image is not related to the disease, it means that the image may not contain any medical features or information directly related to the specific disease, that is, non-disease diagnosis data. Therefore, it may no longer be appropriate to continue using it for disease diagnosis-related training. At this point, you can consider using the image and its corresponding image description content for other types of training, such as aligned visual question answering (VQA) and instruction VQA, to make full use of image resources.

[0066] For Aligned VQA, which is an image-based question answering task, it requires the AI ​​model to answer questions based on the image content. When constructing VQA, first, a series of preset scenarios are determined according to the application scenarios and training requirements. These scenarios can cover various aspects of daily medical environments, such as 1. Doctors and doctors; 2. Doctors and patients; 3. AI models assist doctors; 4. AI models assist patients; 5. Doctors and patients' families; 6. Evaluators and AI models; 7. Interns and doctors; 8. Medical teachers and students; 9. Senior doctors and interns; 10. Standard questions and answers, etc. For each preset scenario, a series of questions related to the image content are designed. These questions can guide the AI ​​model to focus on the key information in the image and test its understanding and reasoning ability. Generate corresponding answers for each question based on the image description content and preset scenarios.

[0067] For instruction VQA, instruction VQA is a form of question and answer that gives specific instructions or tasks in a preset scenario based on the image and its image description content, and requires the respondent to complete the task based on the image and instructions. When constructing instruction VQA, first, a series of preset scenarios are determined according to the application scenarios and training needs. These scenarios can cover various aspects of daily medical environments, such as 1. Doctors and doctors; 2. Doctors and patients; 3. AI models assist doctors; 4. AI models assist patients; 5. Doctors and patients' families; 6. Evaluators and AI models; 7. Interns and doctors; 8. Medical teachers and students; 9. Senior doctors and interns; 10. Standard questions and answers, etc. For each preset scenario, a series of instructions are designed. For each instruction, a series of questions related to the results of the instruction execution are generated. These questions should be able to test the AI ​​model's understanding and execution capabilities of the instructions. After the AI ​​model executes the instructions and answers the questions, the correct answers and necessary feedback are provided. This helps the AI ​​model correct errors and improve performance.

[0068] The beneficial effects of this application are as follows:

[0069] 1. By integrating medical data from different modalities, more comprehensive and in-depth medical information collection can be achieved, providing a rich data foundation for subsequent data synthesis.

[0070] 2. Extract all images and their contexts from a large amount of medical data to eliminate a large amount of redundant information in the medical data.

[0071] 3. Use a pre-trained large language model to extract the image description content of the image from the context of the image to remove the noise and redundancy in the context, that is, filter out redundant data in the context and remove unnecessary parts.

[0072] 4. Use the pre-trained visual language model to evaluate the match between each image and its image description based on the preset text prompts for image-text matching. If the match meets the pre-configured retention conditions, the image and its description are retained to ensure a high degree of consistency between the image and its description, thereby improving the overall quality of subsequent data synthesis.

[0073] 5. Extract disease-related data from high-quality images and their image descriptions, so that more disease diagnosis multiple-choice questions can be generated based on the disease-related data through data synthesis in the future, so as to enrich the training data of the large medical model, thereby making up for the model's shortcomings in clinical medicine and disease diagnosis, and significantly improving the performance of the model. In addition, synthetic data improves the diversity of medical data sets, enabling the model to obtain different types of training data, greatly increasing the diversity of training data, and thus providing the possibility of improving model performance.

[0074] Embodiment 2:

[0075] Based on the same inventive concept, the present application also provides a high-quality medical multimodal data synthesis device, Figure 2 A schematic diagram of the structure of a high-quality medical multimodal data synthesis device provided in an embodiment of the present application, the device comprising:

[0076] An acquisition unit 21, used for acquiring multimodal medical data;

[0077] An extraction unit 22, configured to extract all images and their contexts from the medical data;

[0078] The processing unit 23 is used to extract image description content related to the image from the context of the image through a pre-trained large language model and based on a preset refined content text prompt for all the images; determine the matching degree between the image and the image description content through a pre-trained visual language large model and based on a preset image-text matching text prompt, and retain the image and the image description content when it is determined that the matching degree meets the pre-configured retention condition; if a target disease related to the image description content is obtained through the large language model based on a preset disease diagnosis text prompt, then generate a preset number of similar diseases similar to the target disease and belonging to the same department through the large language model based on a preset disease extension text prompt; combine the image description content with the target disease and each of the similar diseases to form a disease diagnosis multiple-choice question, so as to use the disease diagnosis multiple-choice question and its corresponding diagnosis answer as sample data for training the medical large model; wherein the diagnosis answer corresponding to the disease diagnosis multiple-choice question is the target disease.

[0079] The microphone array detection device in this embodiment is presented in the form of a functional module, where the module refers to an application specific integrated circuit (ASIC), a processor and memory that executes one or more software or fixed programs, and / or other devices that can provide the above functions.

[0080] The further functional description of each of the above modules and units is the same as that of the above corresponding embodiments and will not be repeated here.

[0081] Embodiment 3:

[0082] See also Figure 3 , Figure 3 is a schematic diagram of the structure of a computer device provided by an optional embodiment of the present application, such as Figure 3 As shown, the computer device includes: one or more processors 10, a memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. Various components are connected to each other using different buses for communication, and can be installed on a common mainboard or installed in other ways as needed. The processor can process the instructions executed in the computer device, including instructions stored in or on the memory to display the graphical information of the GUI on an external input / output device (such as, a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple computer devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Figure 3 A processor 10 is taken as an example.

[0083] The processor 10 may be a central processing unit, a network processor or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be a dedicated integrated circuit, a programmable logic device or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic or any combination thereof.

[0084] The memory 20 stores instructions executable by at least one processor 10, so that at least one processor 10 executes the method shown in the above embodiment.

[0085] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created by the use of a computer device based on the presentation of a small program landing page, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may optionally include a memory remotely arranged relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0086] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid state drive; the memory 20 may also include a combination of the above types of memory.

[0087] The computer device also includes an input device 30 and an output device 40. The processor 10, the memory 20, the input device 30 and the output device 40 may be connected via a bus or other means. Figure 3 The example of connecting through bus is taken in the following.

[0088] The input device 30 can receive input digital or character information, and generate key signal input related to the user settings and function control of the computer device, such as a touch screen, a keypad, a mouse, a track pad, a touch pad, an indicator bar, one or more mouse buttons, a trackball, a joystick, etc. The output device 40 may include a display device, an auxiliary lighting device (e.g., an LED) and a tactile feedback device (e.g., a vibration motor), etc. The above-mentioned display device includes but is not limited to a liquid crystal display, a light emitting diode, a display and a plasma display. In some optional embodiments, the display device can be a touch screen.

[0089] Embodiment 4:

[0090] On the basis of the above embodiments, an embodiment of the present application further provides a computer-readable storage medium, in which a computer program executable by a processor is stored. When the program runs on the processor, the processor implements the following steps when executing:

[0091] Acquire multimodal medical data;

[0092] extracting all images and their context from the medical data;

[0093] For all the images, the image description content related to the image is extracted from the context of the image through a pre-trained large language model based on a preset refined content text prompt; the matching degree between the image and the image description content is determined through a pre-trained visual language large model based on a preset image-text matching text prompt, and when it is determined that the matching degree meets the pre-configured retention condition, the image and the image description content are retained; if the target disease related to the image description content is obtained through the large language model based on a preset disease diagnosis text prompt, a preset number of similar diseases similar to the target disease and belonging to the same department are generated through the large language model based on a preset disease extension text prompt; the image description content is combined with the target disease and each of the similar diseases to form a disease diagnosis multiple-choice question, so that the disease diagnosis multiple-choice question and its corresponding diagnosis answer are used as sample data for training the medical large model; wherein the diagnosis answer corresponding to the disease diagnosis multiple-choice question is the target disease.

[0094] Since the principle of solving the problem by the above-mentioned computer-readable storage medium is similar to the high-quality medical multimodal data synthesis method, the implementation of the above-mentioned computer-readable storage medium can refer to the embodiment of the method, and the repeated parts will not be repeated.

[0095] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A high-quality medical multimodal data synthesis method, characterized in that: The method comprises: Acquire multimodal medical data; extracting all images and their context from the medical data; For all the images, the image description content related to the image is extracted from the context of the image through a pre-trained large language model based on a preset refined content text prompt; the matching degree between the image and the image description content is determined through a pre-trained visual language large model based on a preset image-text matching text prompt, and when it is determined that the matching degree meets the pre-configured retention condition, the image and the image description content are retained; if the target disease related to the image description content is obtained through the large language model based on a preset disease diagnosis text prompt, a preset number of similar diseases similar to the target disease and belonging to the same department are generated through the large language model based on a preset disease extension text prompt; the image description content is combined with the target disease and each of the similar diseases to form a disease diagnosis multiple-choice question, so that the disease diagnosis multiple-choice question and its corresponding diagnosis answer are used as sample data for training the medical large model; wherein the diagnosis answer corresponding to the disease diagnosis multiple-choice question is the target disease.

2. The method according to claim 1, characterized in that For all the images, if the large language model determines, based on the disease diagnosis text prompt, that the image description content is not related to the disease, the method further includes: According to the image and the image description content, the aligned visual question and answer and the instruction visual question and answer for each preset scene are constructed.

3. The method according to claim 1, characterized in that The extracting all images and their contexts from the medical data comprises: All images and their contexts are extracted from the medical data through regular expressions, and corresponding image tags are inserted into corresponding positions in the contexts; wherein any image tag is used to uniquely locate the image and the context of the image.

4. The method according to claim 1, characterized in that The medical data includes multiple languages.

5. The method according to claim 1, characterized in that If the large model includes a large language model and a visual language large model, the large model is trained in the following manner: Fine-tune the instructions of the basic large model so that the trained large model can understand and execute the corresponding task according to the preset text prompt; wherein the basic large model is pre-trained in advance by predicting the next token.

6. The method according to claim 1, characterized in that The retention condition is that the matching degree is greater than a preset matching degree threshold.

7. A high-quality medical multimodal data synthesis device, characterized in that: The device comprises: An acquisition unit, used for acquiring multimodal medical data; an extraction unit, configured to extract all images and their contexts from the medical data; A processing unit is used to extract image description content related to the image from the context of the image through a pre-trained large language model and based on a preset refined content text prompt for all the images; determine the degree of match between the image and the image description content through a pre-trained visual language large model and based on a preset text prompt for image-text matching, and retain the image and the image description content when it is determined that the degree of match meets a pre-configured retention condition; if a target disease related to the image description content is obtained through the large language model based on a preset disease diagnosis text prompt, then generate a preset number of similar diseases similar to the target disease and belonging to the same department through the large language model based on a preset disease extension text prompt; combine the image description content with the target disease and each of the similar diseases to form a disease diagnosis multiple-choice question, so that the disease diagnosis multiple-choice question and its corresponding diagnosis answer are used as sample data for training the medical large model; wherein the diagnosis answer corresponding to the disease diagnosis multiple-choice question is the target disease.

8. A computer device, characterized in that: The computer device includes a processor, and the processor is used to implement the steps of the high-quality medical multimodal data synthesis method as described in any one of claims 1 to 6 when executing a computer program stored in a memory.

9. A computer-readable storage medium, characterized in that: It stores a computer program, which, when executed by a processor, implements the steps of synthesizing high-quality medical multimodal data as described in any one of claims 1 to 6 above.

Images