Method and device for generating question and answer data set based on orthopedic clinical medical record and medium

Abstract

The present application relates to a kind of method, device and medium based on orthopaedic clinical history generation question and answer data set, the method includes the following steps: obtaining multiple orthopaedic clinical history and the medical image corresponding to each orthopaedic clinical history, constructs to be labeled data set;Extract the medical entity of each orthopaedic clinical history;Respectively using the medical entity of each orthopaedic clinical history and the medical image corresponding to each orthopaedic clinical history replaces medical entity placeholder and image placeholder in visual question and answer pair template, generates multiple medical visual question and answer pair;Based on the medical visual question and answer pair based on orthopaedic clinical history generation visual question and visual answer pair orthopaedic clinical history corresponding medical image is labeled, generates question and answer data set.The present application is based on orthopaedic clinical history and realizes the automatic labeling of medical image, reduces the cost of manual annotation, and improves the quality of question and answer data set.

Description

Technical Field

[0001] The present invention relates to the field of medical visual question answering technology, and in particular to a method, device and medium for generating a question answering dataset based on orthopedic clinical medical records. Background Art

[0002] Medical visual question answering is a challenging problem that takes medical images and clinical questions about the images as input and outputs the correct answer in natural language. Unlike general visual question answering, medical visual question answering is a more challenging problem because existing medical visual question answering systems usually need to be trained on large-scale annotated datasets. However, well-annotated datasets for training medical visual question answering systems are very scarce because obtaining high-quality annotations from medical experts is very expensive. There is a need to build large-scale and high-quality medical visual question answering datasets. Existing works can be divided into two categories. The first group constructs question answering datasets by extracting images and related text from online medical textbooks. However, these data often have low data quality due to the lack of manual annotations. The second group uses manual annotations to build more reliable question answering datasets, however, their data size is too small and their annotation costs are high. Summary of the Invention

[0003] In view of this, the present invention provides a method, device and medium for generating a question-and-answer dataset based on orthopedic clinical medical records, so as to reduce the cost of manual annotation and improve the quality of the question-and-answer dataset.

[0004] To achieve the above object, the present invention provides the following solutions:

[0005] A method for generating a question-answering dataset based on orthopedic clinical medical records, the method comprising the following steps:

[0006] Acquire multiple orthopedic clinical medical records and medical images corresponding to each of the orthopedic clinical medical records to construct a dataset to be labeled; the dataset to be labeled includes multiple data pairs to be labeled, each of the data pairs to be labeled includes an orthopedic clinical medical record and a medical image corresponding to the orthopedic clinical medical record;

[0007] Extracting medical entities from each of the orthopedic clinical records;

[0008] The medical entity placeholder and the image placeholder in the visual question-answer pair template are replaced with the medical entity of each orthopedic clinical medical record and the medical image corresponding to each orthopedic clinical medical record, respectively, to generate multiple medical visual question-answer pairs; each medical visual question-answer pair includes a medical image corresponding to the orthopedic clinical medical record and a visual question and a visual answer generated based on the orthopedic clinical medical record; the visual question and the visual answer are respectively questions and answers about the medical image;

[0009] Based on the visual questions and visual answers generated based on the orthopedic clinical medical records in the medical visual question and answer pairs, the medical images corresponding to the orthopedic clinical medical records are annotated to generate a question and answer dataset; the question and answer dataset is used to train the medical question and answer system.

[0010] Optionally, extracting the medical entity of each orthopedic clinical case record specifically includes:

[0011] The BERT-BiLSTM-CRF algorithm is used to extract the medical entities of each orthopedic clinical medical record; wherein the BERT-BiLSTM-CRF algorithm is an algorithm that combines BERT (Bidirectional Encoder Representation from Transformers, a bidirectional encoder table based on Transformer), BiLSTM (Bi-directional Long Short-Term Memory) and CRF (Conditional Random Field).

[0012] Optionally, the medical entities include: body part names, disease names, symptom names, treatment names, medical examination names and drug names.

[0013] Optionally, the method further includes replacing the medical entity placeholder and the image placeholder in the visual question-answer pair template with the medical entity of each orthopedic clinical medical record and the medical image corresponding to each orthopedic clinical medical record to generate multiple medical visual question-answer pairs.

[0014] Obtain clinical diagnosis and treatment records related to medical images;

[0015] The KG-MDMF (Knowledge Graph based Medical Document Mining Framework) algorithm and the IMAS++ (An Intelligent Medical Analysis System Enhanced with Deep Graph Neural Networks) clustering function were used to cluster visual questions and visual answers in clinical medical records to determine the question type and answer type.

[0016] The BERT-BiLSTM-CRF algorithm is used to extract the medical entities of all visual questions in each question type, and all the medical entities of all visual questions in each question type are replaced with medical entity placeholders;

[0017] Generate a question template corresponding to each question type based on the visual questions after all medical entities in each question type are replaced;

[0018] The BERT-BiLSTM-CRF algorithm is used to extract the medical entities of all visual answers in each answer type, and all medical entities in the visual answers of each answer type are replaced with medical entity placeholders;

[0019] Generate an answer template corresponding to each answer type based on the visual answers after replacing all medical entities in each answer type;

[0020] A visual question-answer pair template is constructed using an image placeholder, the question template, and the answer template.

[0021] Optionally, the KG-MDMF algorithm and the clustering function of the IMAS++ system are used to cluster visual issues in clinical diagnosis and treatment records to determine the problem type, including:

[0022] All visual problems in clinical diagnosis and treatment records are divided into pre-classification dataset and secondary classification dataset;

[0023] The clustering function in the IMAS++ system is used to classify the visual problems in the pre-classified dataset and determine the problem type of each visual problem in the pre-classified dataset;

[0024] The problem type of each visual problem in the pre-classified dataset is used as a label for each visual problem to construct a training sample set;

[0025] Using the training sample set to train the KG-MDMF model to obtain a trained KG-MDMF model;

[0026] The trained KG-MDMF model is used to determine the question types of visual questions in the secondary classification dataset.

[0027] Optionally, the method further comprises replacing the medical entity placeholder and the image placeholder in the visual question-answer pair template with the medical entity of each orthopedic clinical medical record and the medical image corresponding to each orthopedic clinical medical record to generate multiple medical visual question-answer pairs, and then further comprising:

[0028] Based on the orthopedic medical knowledge graph and the KG-MDMF algorithm, the correlation between the visual question and the visual answer in each medical visual question-answer pair is calculated;

[0029] Delete medical visual question-answer pairs whose correlation between visual questions and visual answers is less than a preset threshold.

[0030] A system for generating a question-answering dataset based on orthopedic clinical medical records, the system being applied to the above-mentioned method, comprising:

[0031] A module for acquiring a dataset to be labeled is used to acquire a plurality of orthopedic clinical medical records and a medical image corresponding to each of the orthopedic clinical medical records to construct a dataset to be labeled; the dataset to be labeled includes a plurality of data pairs to be labeled, each of which includes an orthopedic clinical medical record and a medical image corresponding to the orthopedic clinical medical record;

[0032] A medical entity extraction module, configured to extract medical entities from each of the orthopedic clinical medical records;

[0033] A medical visual question-answer pair construction module is configured to replace the medical entity placeholder and the image placeholder in the visual question-answer pair template with the medical entity of each orthopedic clinical medical record and the medical image corresponding to each orthopedic clinical medical record, respectively, to generate multiple medical visual question-answer pairs; each medical visual question-answer pair includes a medical image corresponding to the orthopedic clinical medical record and a visual question and a visual answer generated based on the orthopedic clinical medical record; the visual question and the visual answer are respectively questions and answers about the medical image;

[0034] A labeling module is used to label the medical images corresponding to the orthopedic clinical medical records based on the visual questions and visual answers generated based on the orthopedic clinical medical records in the medical visual question and answer pairs, and generate a question and answer dataset; the question and answer dataset is used to train the medical question and answer system.

[0035] An electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above method when executing the computer program.

[0036] A computer-readable storage medium stores a computer program, which implements the above method when executed.

[0037] According to the specific embodiments provided by the present invention, the present invention discloses the following technical effects:

[0038] The present invention discloses a method, device, and medium for generating a question-and-answer dataset based on orthopedic clinical medical records. The method includes the following steps: obtaining multiple orthopedic clinical medical records and medical images corresponding to each of the orthopedic clinical medical records to construct a dataset to be annotated; extracting medical entities from each of the orthopedic clinical medical records; replacing medical entity placeholders and image placeholders in a visual question-and-answer pair template with the medical entities from each of the orthopedic clinical medical records and the medical images corresponding to each of the orthopedic clinical medical records to generate multiple medical visual question-and-answer pairs; annotating the medical images corresponding to the orthopedic clinical medical records based on the visual questions and visual answers generated based on the orthopedic clinical medical records in the medical visual question-and-answer pairs to generate a question-and-answer dataset. The present invention realizes automatic annotation of medical images based on orthopedic clinical medical records, reduces the cost of manual annotation, and improves the quality of the question-and-answer dataset. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technical personnel in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0040] Figure 1 A flowchart of a method for generating a question-answering dataset based on orthopedic clinical medical records provided by an embodiment of the present invention;

[0041] Figure 2 An example diagram of a visual question-answering pair provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0042] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments derived by persons of ordinary skill in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0043] The purpose of the present invention is to provide a method, device and medium for generating a question-answering dataset based on orthopedic clinical medical records, so as to reduce the cost of manual annotation and improve the quality of the question-answering dataset.

[0044] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0045] Example 1

[0046] Example 1 of the present invention provides a method for generating a question-answering dataset based on orthopedic clinical medical records, such as Figure 1 As shown, the method includes the following steps:

[0047] Step 101: Acquire multiple orthopedic clinical medical records and medical images corresponding to each of the orthopedic clinical medical records to construct a dataset to be labeled; the dataset to be labeled includes multiple data pairs to be labeled, and the data pairs to be labeled include orthopedic clinical medical records and medical images corresponding to the orthopedic clinical medical records.

[0048] Step 102: extract the medical entity of each orthopedic clinical case record.

[0049] Extracting medical entities refers to the process of identifying specific types of names or symbols within orthopedic clinical medical records. This embodiment of the present invention defines six types of medical entities: body part names, disease names, symptom names, treatment names, medical examination names, and medication names. It is necessary to extract the names of body parts, diseases, symptoms, treatments, medical examination names, and medications from orthopedic clinical medical records. To extract medical entities, this embodiment of the present invention utilizes the existing medical named entity recognition algorithm BERT-BiLSTM-CRF, where BERT is pre-trained.

[0050] Step 103: Replace the medical entity placeholder and the image placeholder in the visual question-answer pair template with the medical entity of each orthopedic clinical medical record and the medical image corresponding to each orthopedic clinical medical record, generating a plurality of medical visual question-answer pairs; each medical visual question-answer pair includes a medical image corresponding to the orthopedic clinical medical record and a visual question and a visual answer generated based on the orthopedic clinical medical record; the visual question and the visual answer are questions and answers about the medical image, respectively;

[0051] The visual question-answer pair template in step 103 is obtained in the following manner: obtain clinical diagnosis and treatment records about medical images; cluster the visual questions and visual answers in the clinical diagnosis and treatment records respectively using the KG-MDMF algorithm and the clustering function of the IMAS++ system to determine the question type and the answer type; use the BERT-BiLSTM-CRF algorithm to extract the medical entities of all visual questions in each question type, and replace all the medical entities of all visual questions in each question type with medical entity placeholders; based on the visual questions after all the medical entities in each question type are replaced, generate a question template corresponding to each question type; use the BERT-BiLSTM-CRF algorithm to extract the medical entities of all visual answers in each answer type, and replace all the medical entities of all visual answers in each answer type with medical entity placeholders; based on the visual answers after all the medical entities in each answer type are replaced, generate an answer template corresponding to each answer type; use the image placeholder, the question template and the answer template to construct a visual question-answer pair template. Specifically as follows:

[0052] First, we sampled orthopedic medical images and their associated clinical records to generate clinical context. We then manually collected frequently asked questions (FAQs) from hospital clinical records to capture real clinical needs. This embodiment of the present invention used the KG-MDMF algorithm and the clustering capabilities of the IMAS++ system to perform text clustering on the collected FAQs. This clustering process categorized the FAQs into six categories: anomaly, state presence, morphology, organ system, plane, and other attributes.

[0053] The KG-MDMF algorithm and the clustering function of the IMAS++ system are used to cluster visual problems in clinical diagnosis and treatment records and determine the problem type, specifically including: dividing all visual problems in clinical diagnosis and treatment records into a pre-classification data set and a secondary classification data set; using the clustering function in the IMAS++ system to classify the visual problems in the pre-classification data set and determine the problem type of each visual problem in the pre-classification data set; using the problem type of each visual problem in the pre-classification data set as a label for each of the visual problems to construct a training sample set; using the training sample set to train the KG-MDMF model to obtain the trained KG-MDMF model; and using the trained KG-MDMF model to determine the problem type of the visual problems in the secondary classification data set.

[0054] For example, the collected FAQs are randomly divided into two parts in a ratio of 3:7. 30% of the FAQs are clustered using the clustering function in the IMAS++ system. After clustering, these 30% of FAQs are assigned their own question type labels. These 30% of FAQs and their corresponding cluster labels are then used as training data for the KG-MDMF algorithm. The trained KG-MDMF algorithm is then used to classify the remaining 70% of FAQs by question type. Each of these 70% of FAQs is assigned its own question label. In this way, all FAQs are labeled with their own question type.

[0055] To generate the templates, we similarly use the existing BERT-BiLSTM-CRF medical entity recognition algorithm to extract medical entities from common questions and their corresponding answers. These entities are then replaced with universal placeholders, automatically normalizing the questions and answers to generate the final visual question-answer pair templates. Three types of placeholders are defined: image placeholders, medical entity placeholders, and annotation placeholders. Table 1 shows all the medical entity and annotation placeholders used in this embodiment of the present invention.

[0056] Table 1 Placeholder list

[0057]

[0058] Diseases, symptoms, treatments, tests, drugs, and body parts are placeholders for medical entities; organ systems, planes, and shapes are placeholders for annotations.

[0059] A total of 13 visual question-answer pair templates are generated in this embodiment of the present invention, as shown in Table 2. The visual question-answer pair templates in this embodiment of the present invention generally consist of an image placeholder, a medical entity placeholder for the organ in the image, a question with a medical entity placeholder, an answer with a medical entity placeholder, a question type with an annotation placeholder, and an answer type with an annotation placeholder, as shown in Table 2.

[0060] Table 2 List of visual question answering templates

[0061]

[0062]

[0063] Step 103 uses the medical entity of each orthopedic clinical medical record and the medical image corresponding to each orthopedic clinical medical record to replace the medical entity placeholder and the image placeholder in the visual question-answer pair template to generate multiple medical visual question-answer pairs, which then includes: calculating the correlation between the visual question and the visual answer in each medical visual question-answer pair based on the orthopedic medical knowledge graph and the KG-MDMF algorithm; deleting the medical visual question-answer pairs whose correlation between the visual question and the visual answer is less than a preset threshold. Specifically:

[0064] The generated medical visual question-answer pairs are automatically screened based on the orthopedic medical knowledge graph and KG-MDMF algorithm. The medical entities contained in the generated medical visual question-answer pairs are extracted, and the correlation between the visual questions and the visual answers is calculated based on the knowledge graph and KG-MDMF algorithm. A threshold (preset threshold) is set, and only the medical visual question-answer pairs with a correlation higher than the threshold are retained. The exemplary embodiment of the present invention finally obtained 19,020 pairs of medical visual question-answer pairs. According to Figure 2 In the example, the generated visual question-answer pair consists of a medical image, an organ in the medical image, a question, an answer, a question type, and an answer type. Figure 2 As shown, the question: What abnormalities are seen in the posterior ankle is generated by replacing the entity placeholder of [body part] with the extracted medical entity of “posterior ankle” in the question template of “What abnormalities are seen in [body part]?”

[0065] Step 104 , annotating the medical images corresponding to the orthopedic clinical medical records based on the visual questions and visual answers generated based on the orthopedic clinical medical records in the medical visual question-answering pairs to generate a question-answering dataset; the question-answering dataset is used to train the medical question-answering system.

[0066] Example 2

[0067] Embodiment 2 of the present invention provides a system for generating a question-and-answer dataset based on orthopedic clinical medical records. The system is applied to the method of embodiment 1 and includes:

[0068] The module for acquiring a dataset to be annotated is used to acquire multiple orthopedic clinical medical records and the medical images corresponding to each of the records to construct a dataset to be annotated. The dataset to be annotated includes multiple pairs of data to be annotated, each of which includes an orthopedic clinical medical record and the medical images corresponding to the records.

[0069] The medical entity extraction module is used to extract the medical entities of each orthopedic clinical case record.

[0070] A medical visual question-answer pair construction module is used to replace the medical entity placeholders and image placeholders in the visual question-answer pair template with the medical entities of each orthopedic clinical medical record and the medical images corresponding to each orthopedic clinical medical record to generate multiple medical visual question-answer pairs; each medical visual question-answer pair includes a medical image corresponding to the orthopedic clinical medical record and a visual question and visual answer generated based on the orthopedic clinical medical record; the visual questions and visual answers are questions and answers about the medical image, respectively.

[0071] A labeling module is used to label the medical images corresponding to the orthopedic clinical medical records based on the visual questions and visual answers generated based on the orthopedic clinical medical records in the medical visual question and answer pairs, and generate a question and answer dataset; the question and answer dataset is used to train the medical question and answer system.

[0072] The specific implementation method of the function of each module in Example 2 of the present invention is the same as the specific implementation method of each step in Example 1, and will not be repeated here.

[0073] Example 3

[0074] Embodiment 3 of the present invention provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the method of embodiment 1 is implemented when the processor executes the computer program.

[0075] In addition, when the computer program in the above-mentioned memory is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory, a random access memory, a magnetic disk or an optical disk.

[0076] Example 4

[0077] Embodiment 4 of the present invention provides a computer-readable storage medium having a computer program stored thereon, which implements the method of embodiment 1 when the computer program is executed.

[0078] An embodiment of the present invention provides a method, device, and medium for generating a question-and-answer dataset based on orthopedic clinical medical records. The method includes the following steps: obtaining multiple orthopedic clinical medical records and medical images corresponding to each of the orthopedic clinical medical records to construct a dataset to be annotated; extracting the medical entities of each of the orthopedic clinical medical records; replacing the medical entity placeholders and image placeholders in a visual question-and-answer pair template with the medical entities of each of the orthopedic clinical medical records and the medical images corresponding to each of the orthopedic clinical medical records to generate multiple medical visual question-and-answer pairs; annotating the medical images corresponding to the orthopedic clinical medical records based on the visual questions and visual answers generated based on the orthopedic clinical medical records in the medical visual question-and-answer pairs to generate a question-and-answer dataset. The present invention realizes automatic annotation of medical images based on orthopedic clinical medical records, reduces the cost of manual annotation, and improves the quality of the question-and-answer dataset.

[0079] The various embodiments in this specification are described in a progressive manner. Each embodiment focuses on the differences from other embodiments. Similar or identical parts between the various embodiments can be referred to in conjunction with each other. For the devices disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple. For relevant parts, refer to the description of the methods.

[0080] This document uses specific examples to illustrate the principles and implementation methods of the present invention. The above examples are only intended to help understand the method and core concept of the present invention. At the same time, those skilled in the art will appreciate that the specific implementation methods and application scopes may vary based on the concept of the present invention. In summary, the contents of this specification should not be construed as limiting the present invention.

Claims

1. A method for generating a question-answering dataset based on orthopedic clinical medical records, characterized in that: The method comprises the following steps: Acquire multiple orthopedic clinical medical records and medical images corresponding to each of the orthopedic clinical medical records to construct a dataset to be labeled; the dataset to be labeled includes multiple data pairs to be labeled, each of the data pairs to be labeled includes an orthopedic clinical medical record and a medical image corresponding to the orthopedic clinical medical record; Extracting medical entities from each of the orthopedic clinical records; the medical entities include: body part name, disease name, symptom name, treatment name, medical examination name, and drug name; Obtain clinical diagnosis and treatment records related to medical images; The KG-MDMF algorithm and the clustering function of the IMAS++ system were used to cluster the visual questions and visual answers in the clinical diagnosis and treatment records to determine the question type and answer type. The BERT-BiLSTM-CRF algorithm is used to extract the medical entities of all visual questions in each question type, and all the medical entities of all visual questions in each question type are replaced with medical entity placeholders; Generate a question template corresponding to each question type based on the visual questions after all medical entities in each question type are replaced; The BERT-BiLSTM-CRF algorithm is used to extract the medical entities of all visual answers in each answer type, and all medical entities in the visual answers of each answer type are replaced with medical entity placeholders; Generate an answer template corresponding to each answer type based on the visual answers after replacing all medical entities in each answer type; Constructing a visual question-answer pair template using an image placeholder, the question template, and the answer template; The medical entity placeholder and the image placeholder in the visual question-answer pair template are replaced with the medical entity of each orthopedic clinical medical record and the medical image corresponding to each orthopedic clinical medical record, respectively, to generate multiple medical visual question-answer pairs; each medical visual question-answer pair includes a medical image corresponding to the orthopedic clinical medical record and a visual question and a visual answer generated based on the orthopedic clinical medical record; the visual question and the visual answer are respectively questions and answers about the medical image; Based on the visual questions and visual answers generated based on the orthopedic clinical medical records in the medical visual question and answer pairs, the medical images corresponding to the orthopedic clinical medical records are annotated to generate a question and answer dataset; the question and answer dataset is used to train the medical question and answer system.

2. The method for generating a question-answering dataset based on orthopedic clinical medical records according to claim 1, characterized in that: Extract the medical entities of each orthopedic clinical case record, including: The BERT-BiLSTM-CRF algorithm was used to extract the medical entities of each orthopedic clinical record.

3. The method for generating a question-answering dataset based on orthopedic clinical medical records according to claim 1, characterized in that: The KG-MDMF algorithm and the clustering function of the IMAS++ system are used to cluster visual problems in clinical diagnosis and treatment records and determine the problem types, including: All visual problems in clinical diagnosis and treatment records are divided into pre-classification dataset and secondary classification dataset; The clustering function in the IMAS++ system is used to classify the visual problems in the pre-classified dataset and determine the problem type of each visual problem in the pre-classified dataset; The problem type of each visual problem in the pre-classified dataset is used as a label for each visual problem to construct a training sample set; Using the training sample set to train the KG-MDMF model to obtain a trained KG-MDMF model; The trained KG-MDMF model is used to determine the question types of visual questions in the secondary classification dataset.

4. The method for generating a question-answering dataset based on orthopedic clinical medical records according to claim 1, characterized in that: The method further comprises replacing the medical entity placeholder and the image placeholder in the visual question-answer pair template with the medical entity of each orthopedic clinical medical record and the medical image corresponding to each orthopedic clinical medical record to generate a plurality of medical visual question-answer pairs. Based on the orthopedic medical knowledge graph and the KG-MDMF algorithm, the correlation between the visual question and the visual answer in each medical visual question-answer pair is calculated; Delete medical visual question-answer pairs whose correlation between visual questions and visual answers is less than a preset threshold.

5. A system for generating a question-answering dataset based on orthopedic clinical medical records, characterized in that: The system is applied to the method according to any one of claims 1 to 4, and the system includes: A module for acquiring a dataset to be labeled is used to acquire a plurality of orthopedic clinical medical records and a medical image corresponding to each of the orthopedic clinical medical records to construct a dataset to be labeled; the dataset to be labeled includes a plurality of data pairs to be labeled, each of which includes an orthopedic clinical medical record and a medical image corresponding to the orthopedic clinical medical record; A medical entity extraction module, configured to extract medical entities from each of the orthopedic clinical medical records; A medical visual question-answer pair construction module is configured to replace the medical entity placeholder and the image placeholder in the visual question-answer pair template with the medical entity of each orthopedic clinical medical record and the medical image corresponding to each orthopedic clinical medical record, respectively, to generate multiple medical visual question-answer pairs; each medical visual question-answer pair includes a medical image corresponding to the orthopedic clinical medical record and a visual question and a visual answer generated based on the orthopedic clinical medical record; the visual question and the visual answer are respectively questions and answers about the medical image; A labeling module is used to label the medical images corresponding to the orthopedic clinical medical records based on the visual questions and visual answers generated based on the orthopedic clinical medical records in the medical visual question and answer pairs, and generate a question and answer dataset; the question and answer dataset is used to train the medical question and answer system.

6. An electronic device, characterized in that: The method comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method according to any one of claims 1 to 4 when executing the computer program.

7. A computer-readable storage medium, characterized in that A computer program is stored thereon, and when the computer program is executed, the method according to any one of claims 1 to 4 is implemented.

Images