Report generation method, system and terminal device

Abstract

The present application is applicable to the field of image processing technology, and provides a report generation method, system and terminal device, which extracts the knowledge theme of the report based on the report text data in the training data, and then inputs the original image into the visual extractor to perform end-to-end classification training in combination with the pre-obtained knowledge theme features to obtain a visual feature vector aligned with the knowledge theme features; the visual feature vector is input into the report generation module to perform report word prediction and generate a report, and by establishing an association between the image features and the knowledge theme, the image features and text features in the image description process can be effectively combined, and a report is generated based on this, so that the generated report can accurately describe the feature information in the medical image, alleviate the current phenomenon of mismatch between image and text features, and solve the current problem of low report accuracy and reliability when image description is applied to report generation of medical images.

Description

Technical Field

[0001] The present application belongs to the field of image processing technology, and in particular relates to a report generation method, system and terminal device. Background Art

[0002] With the continuous development of artificial intelligence technology, it has become possible for machines to understand and describe images. Image description is the process of converting image information into text. Image description lies at the intersection of computer vision and natural language processing. Common applications of image description include news image caption generation and image retrieval caption generation.

[0003] Current image captioning applications typically rely on fixed templates. They identify objects, attributes, and relationships from photos, infer these objects, attributes, and relationships (prepositions) using conditional random fields, and then use words from each object to populate the fixed template. Finally, they predict conjunctions and mark the end of the sentence to create a complete description. While this approach can describe the content of the image, due to the fixed template, it suffers from problems such as monotonous sentence forms and limited content.

[0004] Based on this, current image description applications have also proposed an end-to-end image description method based on a codec structure. This method does not require a fixed text template. First, the encoder determines the features in the image to obtain the feature vector, and then the decoder gradually generates the description words corresponding to each feature vector, generating the description text word by word.

[0005] However, medical images have extremely high similarity, that is, the similarity between normal images and patient images is very high, and the difference is only in the small lesion part. In addition, medical image reports usually need to accurately describe the details of the patient's condition. Therefore, medical text has the characteristics of diversity, that is, the description of medical images has both highly similar image features and highly discrete text features. As a result, it is difficult to accurately describe image details when image description is used in medical image report generation, resulting in low accuracy and reliability of the generated reports. Summary of the Invention

[0006] In view of this, embodiments of the present application provide a report generation method, system, and terminal device to solve the problem of low report accuracy and reliability when current image description applications are used to generate reports on medical images.

[0007] In a first aspect, an embodiment of the present application provides a report generation method, comprising:

[0008] Inputting the original image into the trained visual extractor for feature extraction to obtain a visual feature vector of the original image, wherein the visual feature vector corresponds to a knowledge topic extracted from report text data of the training data;

[0009] The visual feature vector is input into the trained report generation module to predict report words and generate a report.

[0010] Optionally, the report generation method further includes:

[0011] Extracting knowledge topics based on report text data of training data;

[0012] Based on the historical medical images of the training data and the knowledge topic classification results corresponding to the historical medical images, an untrained visual extractor is trained to obtain a trained visual extractor;

[0013] An untrained report generation module is trained based on historical medical reports and visual image feature vectors extracted by a trained visual extractor to obtain a trained report generation module.

[0014] Optionally, the visual extractor includes a visual encoder and a TF encoder, and the original image is input into the trained visual extractor for visual feature extraction to obtain a visual feature vector of the original image, including:

[0015] Extracting a visual feature vector corresponding to a knowledge topic from the original image by the visual extractor;

[0016] The weight of the visual feature vector and the corresponding keyword vector in the knowledge topic are determined by the TF encoder.

[0017] Optionally, the TF encoder includes a multi-head attention mechanism network and a forward propagation network.

[0018] Optionally, inputting the visual feature vector into a trained report generation module to perform report word prediction and generate a report includes:

[0019] determining, by a report generation module, a report word corresponding to the visual feature vector;

[0020] Perform word embedding on the report words to get the report.

[0021] Optionally, extracting knowledge topics based on report text data in the training data includes:

[0022] Embed the report text data into words to obtain word vectors;

[0023] Clustering is performed based on word vectors to obtain different knowledge topics.

[0024] Optionally, the original image includes an original image pair.

[0025] In a second aspect, an embodiment of the present application provides a report generation system, including a knowledge distillation module, a visual extractor, and a report generation module;

[0026] The knowledge distillation module is used to extract the knowledge topics of the report based on the report text data in the training data;

[0027] The visual extractor is configured to input an original image into the trained visual extractor for feature extraction, thereby obtaining a visual feature vector of the original image, wherein the visual feature vector corresponds to a knowledge topic extracted from report text data of the training data;

[0028] The report generation module is used to input the visual feature vector into the trained report generation module to perform report word prediction and generate a report.

[0029] In a third aspect, an embodiment of the present application provides a terminal device, comprising a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the method described in the first aspect or any optional method of the first aspect is implemented.

[0030] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described in the first aspect or any optional method of the first aspect is implemented.

[0031] In a fifth aspect, an embodiment of the present application provides a computer program product, which, when executed on a terminal device, enables the terminal device to execute the method described in the first aspect or any optional method of the first aspect.

[0032] The report generation method, system, terminal device, terminal device, computer-readable storage medium, and computer program product provided by the embodiments of the present application have the following beneficial effects:

[0033] A report generation method provided in an embodiment of the present application establishes an association between image features and knowledge topics, so that the image features and text features in the image description process can be effectively combined, and the visual feature vector combined with the knowledge topic is identified. Based on this, a report is generated, so that the generated report can accurately describe the feature information in the medical image, alleviate the current mismatch between image and text features, and solve the current problem of low report accuracy and reliability when image description is applied to report generation of medical images. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the embodiments or descriptions of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0035] Figure 1 is a schematic flow chart of a report generation method provided in an embodiment of the present application;

[0036] Figure 2 is a schematic flow chart of another report generation method provided in an embodiment of the present application;

[0037] Figure 3 This is a schematic diagram of the structure of a report generation system provided by an embodiment of the present application;

[0038] Figure 4 This is a schematic structural diagram of a terminal device provided in another embodiment of the present application;

[0039] Figure 5 It is a structural diagram of a computer-readable storage medium provided in an embodiment of the present application. DETAILED DESCRIPTION

[0040] In the following description, specific details such as specific system structures and techniques are provided for purposes of illustration rather than limitation to facilitate a thorough understanding of the embodiments of the present application. However, it will be apparent to those skilled in the art that the present application may be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, systems, circuits, and methods are omitted to avoid obscuring the description of the present application with unnecessary detail.

[0041] It should be understood that the term "and / or" used in this specification and the appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations. In addition, in the description of this specification and the appended claims, the terms "first," "second," "third," etc. are used only to distinguish descriptions and are not to be understood as indicating or implying relative importance.

[0042] It should also be understood that references to "one embodiment" or "some embodiments" in the present specification mean that one or more embodiments of the present application include a particular feature, structure, or characteristic described in conjunction with that embodiment. Thus, phrases such as "in one embodiment," "in some embodiments," "in other embodiments," and "in other embodiments" appearing in various places in this specification do not necessarily refer to the same embodiment, but rather mean "one or more but not all embodiments," unless otherwise specifically emphasized. The terms "including," "comprising," "having," and variations thereof all mean "including but not limited to," unless otherwise specifically emphasized.

[0043] Medical imaging refers to the technology and processing process of obtaining images of internal tissues of the human body or a part of the human body in a non-invasive manner for medical treatment or medical research. Medical images can be obtained through medical imaging systems.

[0044] Using artificial intelligence technology to describe medical images and generate medical reports can reduce dependence on radiologists and make medical image detection more widely popularized.

[0045] However, the difference between patient images and normal images is often only in the lesion part, that is, there are often only very subtle differences between patient images and normal images. In addition, medical reports need to describe the patient's condition very accurately. Medical reports are often relatively long and different doctors have different styles of writing medical reports. This leads to poor results when existing image description methods are directly applied to the description of medical images, that is, they cannot accurately describe the patient's condition.

[0046] Based on this, an embodiment of the present application provides a report generation method by combining knowledge topics with image features.

[0047] The report generation method, system, and terminal device provided in the embodiments of the present application are described in detail below:

[0048] See also Figure 1 , Figure 1 This is a schematic flow chart of a report generation method provided in an embodiment of the present application.

[0049] like Figure 1 As shown, the above report generation method may include S11 to S12, which are described in detail as follows:

[0050] S11: Inputting the original image into the trained visual extractor for feature extraction to obtain a visual feature vector of the original image.

[0051] In the embodiment of the present application, the original image may be a medical image acquired by a medical imaging system. The original image may include only one image or two or more patient images, i.e., medical images acquired by the medical imaging system and captured from different angles of the same patient. For example, the original image may be an X-ray of the patient's chest in the anteroposterior or lateral view.

[0052] In this embodiment of the present application, the aforementioned knowledge topics refer to knowledge topic categories extracted from report text data within a large amount of training data. Different knowledge topic categories contain a large number of keyword vectors used to describe the topic category. Each keyword vector can correspond to a visual feature, and different keyword vectors can also correspond to the same visual feature. In other words, a visual feature can be described using certain keyword vectors within a certain topic category.

[0053] In an embodiment of the present application, by inputting the original image into a trained visual extractor for processing, a visual feature vector (visual feature) corresponding to the knowledge topic can be obtained. The visual extractor is obtained after end-to-end classification training based on the pre-obtained topic features, so the visual feature vector corresponding to the knowledge topic features can be extracted.

[0054] It should be noted that multiple visual feature vectors can be extracted from the original image, and each visual feature vector has its corresponding knowledge topic classification result (i.e., knowledge topic feature). The knowledge topic is used as the label of the image in the feature extraction process to improve the matching degree between the image and the text description.

[0055] It should be noted that the knowledge topic classification results corresponding to the above-mentioned visual feature vectors refer to the keyword vectors in the knowledge topic corresponding to each visual feature vector.

[0056] In one embodiment of the present application, the above-mentioned visual extractor may include a visual encoder and a TF (Transformer) encoder.

[0057] Exemplarily, the above-mentioned visual encoder can be constructed using a ResNET-101 network (of course, other types of neural networks can also be used, which is only an example here).

[0058] In a specific application, the input original image may be an original image pair, the original image pair including a first original image and a second original image. The first original image may be a medical image captured at a first preset orientation, and the second original image may be a medical image captured at a second preset orientation. It should be noted that the first preset orientation and the second preset orientation may be set based on detection requirements, for example, setting the first preset orientation to a frontal view and the second preset orientation to a left side view, etc., and this application does not impose any limitations on this.

[0059] By extracting the visual feature vectors from the original image pair, specifically, the visual feature vectors of each image in the original image pair can be extracted separately, and then the visual feature vectors are superimposed to obtain the visual feature vectors of the original image pair. Two linear layers can be added to reduce the feature dimension before performing the classification task.

[0060] It should be noted that the above classification task is the process of determining which knowledge topic the feature vector of the image corresponds to.

[0061] In an embodiment of the present application, the above-mentioned TF encoder adopts a multi-head attention mechanism network and a forward propagation network, takes the features extracted by the visual encoder as the input of the TF encoder, and then uses the multi-head attention mechanism to determine the results of each head. The multiple heads are then spliced together to obtain the weight corresponding to each visual feature vector. This process is a many-to-many relationship, so multiple keyword vectors corresponding to the visual feature vector can be obtained, so that the visual extractor can pay attention to the visual feature vectors that are useful for generating reports and give more weight to the image feature vector.

[0062] In an embodiment of the present application, the number of heads of the above-mentioned multi-head attention mechanism can be 8 heads. For the results of each head, the zoom click attention mechanism can be used to obtain the splicing results of each head. The splicing results are subjected to residual connection and layer-based normalization operations to obtain normalized results, and then the final results are obtained based on the forward propagation network.

[0063] Here, the trained visual extractor is able to extract visual feature vectors that are aligned with the knowledge topics in the input original image pair.

[0064] Based on this, the above S11 may include the following steps:

[0065] Extracting a visual feature vector corresponding to a knowledge topic from the original image by the visual extractor;

[0066] The weight of the visual feature vector and the corresponding keyword vector in the knowledge topic are determined by the TF encoder.

[0067] S12: Input the visual feature vector into the trained report generation module to perform report word prediction and generate a report.

[0068] In an embodiment of the present application, each visual feature vector is input into the trained report generation module for processing, and the report words of each visual feature vector and each visual feature vector can be predicted. If the corresponding report words of all visual feature vectors are predicted, a report can be generated word by word.

[0069] In an embodiment of the present application, the above-mentioned report generation module can also be implemented using a multi-head attention mechanism and a forward propagation network. After passing through the visual extractor, the visual feature vector of the original image corresponding to the knowledge topic is obtained, and then the corresponding report words are determined by the report generation module for word embedding to obtain a complete report.

[0070] In one embodiment of the present application, the above-mentioned multi-head attention mechanism can also adopt an 8-head attention mechanism.

[0071] The above-mentioned forward propagation network can refer to the architecture and principles of the existing forward propagation network. To avoid repetition, this application will not elaborate on them here.

[0072] Based on this, the above S12 may include the following steps:

[0073] determining, by a report generation module, a report word corresponding to the visual feature vector;

[0074] Perform word embedding on the report words to get the report.

[0075] From the above, it can be seen that by associating image features with knowledge topics, the image features and text features in the image description process can be effectively combined, the visual feature vector combined with the knowledge topic can be identified, and a report can be generated based on this, so that the generated report can accurately describe the feature information in the medical image, alleviate the current mismatch between image and text features, and solve the current problem of low report accuracy and reliability when image description is applied to report generation of medical images.

[0076] See also Figure 2 , Figure 2 FIG. 1 shows a flow chart of a report generation method according to another embodiment of the present invention. Figure 2 As shown, different from the previous embodiment, the report generation method provided in the embodiment of the present application further includes the following steps:

[0077] S13: Extract knowledge topics based on training data.

[0078] In an embodiment of the present application, the above-mentioned training data may include report text data and historical medical images.

[0079] The report text data includes a large number of historical medical reports (report texts given by doctors), and the historical medical images include a large number of historical medical images corresponding to the historical medical reports.

[0080] In this embodiment of the present application, in order to make the generated report closer to the report given by the doctor, the text features in the historical medical report are obtained through knowledge extraction. Specifically, the report text data can be first word-embedded to obtain word vectors, and then clustered based on the word vectors to obtain different knowledge topics.

[0081] In the embodiment of the present application, BERT can be used for word embedding, and the clustering algorithm used during clustering adopts the classic UMAP algorithm.

[0082] By processing a large number of historical medical reports, we can obtain multiple knowledge topics, and each knowledge topic is different from each other. The keyword vectors within the knowledge topic have a high similarity.

[0083] S14: Based on the historical medical images and the knowledge topic classification results corresponding to the historical medical images, the untrained visual extractor is trained to obtain a trained visual extractor.

[0084] In an embodiment of the present application, the knowledge topic classification results corresponding to the historical medical image are used as the label of the historical medical image. An untrained visual extractor is trained using a large amount of training data (i.e., the historical medical image and the knowledge topic classification results corresponding to the historical medical image) to achieve convergence of the loss function. The parameters of the visual extractor at this point are determined, and the training process of the visual extractor is completed to obtain a trained visual extractor for use in subsequent applications. The trained visual extractor can then extract visual feature vectors that are aligned with the knowledge topic features.

[0085] S15: Based on the historical medical reports and the visual feature vectors extracted by the trained visual extractor, an untrained report generation module is trained to obtain a trained report generation module.

[0086] In an embodiment of the present application, the visual image feature vector extracted by the trained visual extractor is used as the true value and key value of the historical medical image, and then the predicted report words are obtained and compared with the report words corresponding to the historical medical report. In this way, the untrained report generation module is trained so that the trained report generation module can output a report with little difference from the historical medical report (that is, the loss function of the report generation module converges at this time), the parameters of the report generation module at this time are determined, the training process of the report generation module is completed, and the trained report generation module is obtained for use in subsequent applications.

[0087] From this, it can be seen that a report generation method provided by an embodiment of the present application can extract knowledge topics through report text data in training data, and can alleviate the phenomenon of mismatch between image and text features through self-learning of images and reports, so that the generated report has high readability and can accurately describe image details, so that the generated report has high accuracy and reliability.

[0088] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0089] Based on the report generation method provided in the above embodiment, the embodiment of the present invention further provides an embodiment of a report generation system for implementing the above method embodiment.

[0090] See also Figure 3 , Figure 3 This is a structural diagram of a report generation system provided in an embodiment of the present application. In the embodiment of the present application, the report generation system includes various units for executing Figure 1 and Figure 2 Each step in the corresponding embodiment. Please refer to Figure 1 as well as Figure 1 For the sake of convenience, only the parts related to this embodiment are shown. Figure 3 As shown, the report generation system 30 includes: a knowledge distillation module 31, a visual extractor 32 and a report generation module 33. Among them:

[0091] The knowledge distillation module 31 is used to extract the knowledge topics of the report based on the report text data in the training data.

[0092] The visual extractor 32 is used to train an untrained visual extractor based on the historical medical images and the knowledge topic classification results corresponding to the historical medical images to obtain a trained visual extractor.

[0093] The visual extractor 32 is further configured to input the original image into the trained visual extractor to extract the visual feature vector corresponding to the knowledge topic, thereby obtaining the visual feature vector of the original image.

[0094] The report generation module 33 is used to train an untrained report generation module based on historical medical reports and visual feature vectors of historical medical images extracted by a trained visual extractor to obtain a trained report generation module.

[0095] The report generation module 33 is further configured to input the visual feature vector into the trained report generation module to perform report word prediction and generate a report.

[0096] The visual extractor 32 may include a visual encoder and a TF encoder.

[0097] The visual extractor extracts a visual feature vector corresponding to a knowledge topic from the original image; and the TF encoder determines a weight of the visual feature vector and a keyword vector corresponding to the knowledge topic.

[0098] It should be noted that the report generation system provided in the embodiment of the present application can also establish an association between image features and knowledge topics, so that the image features and text features in the image description process can be effectively combined, and a report can be generated based on this, so that the generated report can accurately describe the feature information in the medical image, alleviate the current mismatch between image and text features, and solve the current problem of low report accuracy and reliability when image description is applied to report generation of medical images.

[0099] Figure 4 This is a schematic diagram of the structure of a terminal device provided by another embodiment of the present application. Figure 4 As shown, the terminal device 4 provided in this embodiment includes: a processor 40, a memory 41, and a computer program 42 stored in the memory 41 and executable on the processor 40, such as a program for cooperative control of a multi-agent system. When the processor 40 executes the computer program 42, the steps in the above-mentioned report generation method embodiments are implemented, such as Figure 1 Alternatively, when the processor 40 executes the computer program 42, the functions of the modules / units in the above-mentioned terminal device embodiments are realized, for example, Figure 3 Functions of units 31 to 33 are shown.

[0100] Exemplarily, the computer program 42 may be divided into one or more modules / units, which are stored in the memory 41 and executed by the processor 40 to complete the present application. The one or more modules / units may be a series of computer program instruction segments capable of completing specific functions, which are used to describe the execution process of the computer program 42 in the terminal device 4. For example, the computer program 42 may be divided into a first acquisition unit and a first processing unit. For the specific functions of each unit, please refer to Figure 3 The relevant descriptions in the corresponding embodiments are not repeated here.

[0101] The terminal device may include but is not limited to a processor 40 and a memory 41. Those skilled in the art will understand that Figure 4It is only an example of the terminal device 4 and does not constitute a limitation on the terminal device 4. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the terminal device may also include input and output devices, network access devices, buses, etc.

[0102] The processor 40 may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor.

[0103] The memory 41 may be an internal storage unit of the terminal device 4, such as a hard disk or memory of the terminal device 4. The memory 41 may also be an external storage device of the terminal device 4, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the terminal device 4. Furthermore, the memory 41 may include both an internal storage unit of the terminal device 4 and an external storage device. The memory 41 is used to store the computer program and other programs and data required by the terminal device. The memory 41 may also be used to temporarily store data that has been output or is about to be output.

[0104] The present application also provides a computer-readable storage medium. Figure 5 , Figure 5 is a schematic diagram of the structure of a computer-readable storage medium provided in an embodiment of the present application, such as Figure 5 As shown, the computer readable storage medium 5 stores a computer program 51 , which can implement the above-mentioned report generating method when executed by a processor.

[0105] An embodiment of the present application provides a computer program product. When the computer program product is run on a terminal device, the terminal device can implement the above-mentioned report generation method when executing the computer program product.

[0106] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the terminal device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0107] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant description of other embodiments.

[0108] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0109] The above-described embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present application, and should all be included in the scope of protection of the present application.

Claims

1. A report generation method, characterized in that: include: Inputting the original image into a trained visual extractor for feature extraction to obtain a visual feature vector of the original image, wherein the visual feature vector corresponds to a knowledge topic, and the knowledge topic is extracted based on report text data of the training data; the original image is an original image pair of a first original image and a second original image taken from different angles of the same patient; the visual feature vector of the original image is obtained by superimposing the visual feature vectors of the first original image and the second original image respectively extracted, and reducing the feature dimension of the superimposed visual feature vector based on two linear layers, and the visual extractor includes a visual encoder and a TF encoder; Inputting the visual feature vector into a trained report generation module to perform report word prediction and generate a report; The step of inputting the original image into the trained visual extractor to extract visual features and obtain a visual feature vector of the original image includes: Extracting a visual feature vector corresponding to a knowledge topic from the original image by the visual extractor; The classification of different knowledge topics includes multiple keyword vectors for describing the topic classification; The weight of the visual feature vector and the corresponding keyword vector in the knowledge topic are determined by the TF encoder, and the weight and the corresponding keyword are determined as the visual feature vector of the original image.

2. The report generation method according to claim 1, characterized in that: Also includes: Extracting knowledge topics based on report text data of training data; Based on the historical medical images of the training data and the knowledge topic classification results corresponding to the images, the untrained visual extractor is trained to obtain a trained visual extractor; An untrained report generation module is trained based on historical medical reports and visual image feature vectors extracted by a trained visual extractor to obtain a trained report generation module.

3. The report generation method according to claim 1, characterized in that: The TF encoder includes a multi-head attention mechanism network and a forward propagation network.

4. The report generation method according to claim 1, characterized in that: The step of inputting the visual feature vector into a trained report generation module to perform report word prediction and generate a report includes: determining, by a report generation module, a report word corresponding to the visual feature vector; Perform word embedding on the report words to get the report.

5. The report generation method according to claim 2, characterized in that: The step of extracting knowledge topics based on report text data in the training data includes: Embed the report text data into words to obtain word vectors; Clustering is performed based on word vectors to obtain different knowledge topics.

6. A report generation system, characterized in that: Includes knowledge distillation module, visual extractor and report generation module; The knowledge distillation module is used to extract the knowledge topics of the report based on the report text data in the training data; The visual extractor is used to input the original image into the trained visual extractor for feature extraction to obtain a visual feature vector of the original image, wherein the visual feature vector corresponds to a knowledge topic, and the knowledge topic is extracted based on report text data of the training data; the original image is an original image pair of a first original image and a second original image taken from different angles of the same patient; the visual feature vector of the original image is obtained by superimposing the visual feature vectors of the first original image and the second original image respectively extracted, and reducing the feature dimension of the superimposed visual feature vector based on two linear layers, and the visual extractor includes a visual encoder and a TF encoder; The report generation module is used to input the visual feature vector into the trained report generation module to perform report word prediction and generate a report; The visual extractor is also used to: The visual extractor extracts a visual feature vector corresponding to a knowledge topic from the original image; different categories of the knowledge topic contain multiple keyword vectors for describing the topic category; the TF encoder determines the weight of the visual feature vector and the corresponding keyword vector in the knowledge topic, and determines the weight and the corresponding keyword as the visual feature vector of the original image.

7. A terminal device, characterized in that: The terminal device includes a processor, a memory, and a computer program stored in the memory and executable on the processor, and the processor implements the method according to any one of claims 1 to 5 when executing the computer program.

8. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 5 is implemented.

Images