Medical image diagnosis report generation method, device, equipment, medium and product

Abstract

The invention discloses a medical image diagnosis report generation method and device, equipment, a medium and a product. According to the method, image features of a radiology image are extracted, a medical knowledge graph is encoded as priori knowledge to help identify abnormal areas in the image, and correlation among some disease expressions is considered through modeling of disease keywords. Integrating the extracted image features and the medical knowledge graph features by using a graph attention mechanism to obtain visual fusion features; and inputting the visual fusion features into a pre-trained target prediction model to finally obtain predicted continuous semantic space vectors, and further obtaining a generated radiology medical report through mapping of discrete words and the continuous semantic space vectors, so that medical image features and text disease keyword representations can be accurately aligned. And the accuracy of diagnosis report generation is further improved.

Description

Technical Field

[0001] The present application belongs to the field of artificial intelligence technology, and in particular, relates to a method, device, equipment, medium and product for generating a medical imaging diagnosis report. Background Art

[0002] With the advancement of hospital informatization and the popularization of electronic medical record systems, the scale of medical data continues to expand and show a trend of complexity. In order to improve the efficiency of medical services and reduce the workload of doctors, the development of technologies that can automatically generate high-quality medical reports has received widespread attention. As an important application of natural language processing, radiology report generation technology can effectively interpret radiological medical images and extract concise and accurate text reports for reference by medical staff, which plays an important role in improving the efficiency of medical workflows.

[0003] The task of generating radiology reports faces two unique challenges. First, the visual data is biased, and the number of normal images without pathological symptoms in the dataset far exceeds the number of abnormal images. Therefore, this unbalanced distribution of visual data affects the model's attention, making it unable to accurately capture the characteristics of rare and diverse abnormal regions. Second, the text data is biased. In the report, radiologists tend to describe all items in the image, resulting in the entire report mostly describing normal areas. In addition, there are many similar sentences used to describe the same normal area. Due to this unbalanced text distribution, using such a dataset for training will result in the generated sentences mainly describing normal areas, which will not allow the model to accurately describe key abnormalities.

[0004] In order to address the unique challenges of radiology report generation methods, recent research has focused on cross-modal alignment and filtering of visual features. However, related technologies treat every word of natural image annotation as equally important when processing radiology image reports, ignoring the importance of disease keywords, which may affect the clinical relevance of the report and ignore the interrelationships of disease manifestations in the image. Summary of the invention

[0005] The embodiments of the present application provide a method, apparatus, device, medium and product for generating a medical imaging diagnostic report, which are used to at least solve the problem in the related art of ignoring the importance of disease keywords, which affects the clinical relevance of the report and ignoring the relationship between disease manifestations in images.

[0006] In a first aspect, an embodiment of the present application provides a method for generating a medical imaging diagnosis report, comprising:

[0007] A radiological image encoder trained in advance is used to extract radiological images of a preset part to obtain radiological image features;

[0008] Encoding a pre-constructed medical knowledge graph using a pre-trained medical graph encoder to obtain medical knowledge graph features, wherein the medical knowledge graph includes a number of disease keywords matching a preset part;

[0009] The radiological image features and the medical knowledge graph features are fused using a graph attention mechanism to obtain visual fusion features;

[0010] Inputting the visual fusion feature into a pre-trained target prediction model to obtain a corresponding target continuous semantic space vector;

[0011] Using a preset mapping function, mapping the target continuous semantic space vector into a target discrete word;

[0012] A medical imaging diagnosis report is generated according to the target discrete words.

[0013] In a second aspect, an embodiment of the present application provides a medical imaging diagnostic report generating device, the device comprising:

[0014] An extraction module, used to extract radiological images of preset parts using a pre-trained radiological image encoder to obtain radiological image features;

[0015] An encoding module, used to encode a pre-constructed medical knowledge graph using a pre-trained medical graph encoder to obtain medical knowledge graph features, wherein the medical knowledge graph includes a number of disease keywords matching a preset part;

[0016] A fusion module, used for fusing the radiological image features with the medical knowledge graph features using a graph attention mechanism to obtain visual fusion features;

[0017] An input module, used for inputting the visual fusion feature into a pre-trained target prediction model to obtain a corresponding target continuous semantic space vector;

[0018] A mapping module, used to map the target continuous semantic space vector into a target discrete word using a preset mapping function;

[0019] A generating module is used to generate a medical imaging diagnosis report according to the target discrete words.

[0020] In a third aspect, an embodiment of the present application provides an electronic device comprising: a processor and a memory storing computer program instructions; when the processor executes the computer program instructions, the steps of the medical imaging diagnostic report generation method as described in any one of the embodiments of the first aspect are implemented.

[0021] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having computer program instructions stored thereon. When the computer program instructions are executed by a processor, the steps of the medical imaging diagnostic report generating method as described in any one of the embodiments of the first aspect are implemented.

[0022] In a fifth aspect, an embodiment of the present application provides a computer program product, which is stored in a storage medium and is executed by at least one processor to implement the steps of the medical imaging diagnostic report generation method provided in the first aspect of the embodiment of the present application.

[0023] The medical imaging diagnostic report generation method, apparatus, device, medium and product of the embodiments of the present application extract image features of radiological images, encode medical knowledge graphs as prior knowledge to help identify abnormal areas in images, and consider the correlation between some disease manifestations through modeling of disease keywords, and then use the graph attention mechanism to integrate the extracted image features and medical knowledge graph features to obtain visual fusion features; the visual fusion features are input into a pre-trained target prediction model to finally obtain a predicted continuous semantic space vector, and then the generated radiological medical report is obtained by mapping discrete words and continuous semantic space vectors, which can accurately align medical image features and text disease keyword representations, further improving the accuracy of diagnostic report generation. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the technical solution of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0025] Figure 1 It is a flowchart of a method for generating a medical imaging diagnosis report provided in an embodiment of the present application;

[0026] Figure 2 It is a flowchart of a method for training a target prediction model provided in an embodiment of the present application;

[0027] Figure 3 It is a flowchart of another method for generating a medical imaging diagnosis report provided in an embodiment of the present application;

[0028] Figure 4 It is a flowchart of another method for generating a medical imaging diagnosis report provided in an embodiment of the present application;

[0029] Figure 5 It is a structural schematic diagram of a medical imaging diagnosis report generating device provided in an embodiment of the present application;

[0030] Figure 6 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application.

[0031] Reference numerals:

[0032] Medical imaging diagnosis report generating device 500, extraction module 501, encoding module 502, fusion module 503, input module 504, mapping module 505, generating module 506,

[0033] Electronic device 600 , processor 601 , memory 602 , communication interface 603 , bus 610 . DETAILED DESCRIPTION

[0034] The features and exemplary embodiments of various aspects of the present application will be described in detail below. In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only intended to explain the present application, rather than to limit the present application. For those skilled in the art, the present application can be implemented without the need for some of these specific details. The following description of the embodiments is only to provide a better understanding of the present application by illustrating the examples of the present application.

[0035] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the statement "include..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0036] In recent years, with the rapid development of machine learning, deep learning and other technologies in the field of artificial intelligence and the continuous improvement of computer computing power, the application of natural language processing technology in the medical field has become increasingly in-depth. For example, intelligent diagnosis technology is used to assist doctors in disease analysis, medical image recognition technology is used to improve the accuracy of disease screening, and medical dialogue systems are used to improve patients' consultation experience. At the same time, with the advancement of hospital informatization and the popularization of electronic medical record systems, the scale of medical data continues to expand and shows a trend of complexity. In order to improve the efficiency of medical services and reduce the workload of doctors, the development of technologies that can automatically generate high-quality medical reports has received widespread attention. As an important application of natural language processing, radiology report generation technology can effectively interpret radiological medical images and extract concise and accurate text reports for reference by medical staff, which plays an important role in improving the efficiency of medical workflows.

[0037] The radiology medical report generation task uses the patient's medical images, examination results and other medical data as input, and aims to automatically or semi-automatically generate accurate and comprehensive medical documents such as diagnosis reports and treatment recommendations through intelligent algorithms. This task has received widespread attention because it has important application value in the medical field. For example, when dealing with complex cases, automatically generated medical reports can serve as a reference for doctors, helping to comprehensively assess the patient's condition and avoid the omission of important information. In addition, as an important part of medical information processing, medical report generation has many applications such as information extraction, text generation, and auxiliary diagnosis. The accuracy and comprehensiveness of report generation have a direct impact on the effectiveness of subsequent upper-level tasks such as clinical decision support systems, intelligent question and answer, and personalized treatment plan recommendations.

[0038] The task of radiology report generation is to use a designed algorithm or a trained model to enable the computer to automatically process the input radiology images and output the generated medical reports. The early work of radiology report generation mainly continued the image overview technology and achieved good generation results, such as using CNN for medical image feature extraction and RNN for report generation.

[0039] With the development of the medical field, researchers have noticed that compared with the image overview task, the radiology report generation task faces two unique challenges. First, there is a bias in the visual data. The number of normal images without pathological symptoms in the dataset far exceeds the number of abnormal images. Therefore, this unbalanced distribution of visual data will affect the model's attention, making it unable to accurately capture the characteristics of rare and diverse abnormal areas. Second, there is a bias in the text data. In the report, radiologists tend to describe all the items in the image, resulting in most of the entire report being a description of normal areas. In addition, there are many similar sentences used to describe the same normal area. Due to this unbalanced text distribution, using such a dataset for training will result in the generated sentences mainly describing normal areas, which will not enable the model to accurately describe key abnormalities.

[0040] In order to address the unique challenges of radiology report generation methods, recent research has focused on cross-modal alignment and filtering of visual features. However, related technologies mainly rely on autoregressive frameworks to generate text sequentially from tokens, which has the problem of slow inference speed and may also lead to the accumulation of sequential errors. In addition, related technologies treat every word annotated in natural images as equally important when processing radiology image reports, ignoring the importance of disease keywords, which may affect the clinical relevance of the report and ignore the relationship between disease manifestations in the image.

[0041] In order to solve the problems of related technologies, the embodiments of the present application provide a method, device, equipment, medium and product for generating a medical imaging diagnostic report.

[0042] The following is a detailed description of the method for generating a medical imaging diagnostic report provided by the embodiment of the present application through specific embodiments and their application scenarios in conjunction with the accompanying drawings.

[0043] Figure 1 FIG. 1 is a flow chart showing a method for generating a medical imaging diagnosis report according to an embodiment of the present application. Figure 1 As shown, the medical imaging diagnosis report generation method may specifically include the following steps:

[0044] S101, extracting a radiological image of a preset part using a pre-trained radiological image encoder to obtain radiological image features;

[0045] S102, encoding a pre-constructed medical knowledge graph using a pre-trained medical graph encoder to obtain medical knowledge graph features, wherein the medical knowledge graph includes a number of disease keywords matching a preset part;

[0046] S103, using a graph attention mechanism to fuse the radiological image features and the medical knowledge graph features to obtain visual fusion features;

[0047] S104, inputting the visual fusion feature into a pre-trained target prediction model to obtain a corresponding target continuous semantic space vector;

[0048] S105, using a preset mapping function, mapping the target continuous semantic space vector to a target discrete word;

[0049] S106: Generate a medical imaging diagnosis report according to the target discrete words.

[0050] Therefore, by extracting the image features of the radiological image, encoding the medical knowledge graph as prior knowledge to help identify abnormal areas in the image, and considering the correlation between some disease manifestations through the modeling of disease keywords, the graph attention mechanism is then used to integrate the extracted image features and medical knowledge graph features to obtain the visual fusion features; the visual fusion features are input into the pre-trained target prediction model to finally obtain the predicted continuous semantic space vector, and then the generated radiological medical report is obtained by mapping discrete words and continuous semantic space vectors, which can accurately align medical image features and text disease keyword representations, further improving the accuracy of diagnostic report generation.

[0051] The specific implementation methods of the above steps are introduced below.

[0052] It should be noted that the radiological images and medical knowledge graphs in the embodiments of the present application both correspond to preset parts of the human body. For example, the radiological image may be a chest X-ray, and the medical knowledge graph may be a chest knowledge graph that has been constructed.

[0053] In some embodiments, in S101, for a given radiological image, a pre-trained MedCLIP medical visual encoder is used to preliminarily extract local grid image features. MedCLIP (Contrastive Learning from Unpaired Medical Images and Text) is a pre-trained medical multimodal model.

[0054] In practice, the radiological image is divided into P×P blocks, which are then flattened into a sequence of length S, where Then, use L Img Transformer Encoder blocks are used to extract features from the image, where one encoding layer can be expressed as:

[0055]

[0056] in, is the lth layer of the input; FFN and LN represent the feedforward network and layer normalization operation, respectively, and MHA is the multi-head attention mechanism. In this way, the image feature output by the radiology image encoder is defined as h I , that is, the extracted radiological image feature h I .

[0057] In some embodiments, in S102, the medical knowledge graph is used to integrate medical prior knowledge, and the features of the knowledge graph are encoded using the Transformer Encoder module to obtain the medical knowledge graph features h G , which helps identify abnormal areas in images.

[0058] Specifically, the medical knowledge graph G med It includes several entity nodes and a root node pointing to the global feature and an adjacency matrix, wherein the adjacency matrix represents the edge, and the entity nodes correspond to the disease keywords one by one. It should be understood that nodes associated with the same organ or tissue are connected to the root node. For example, the chest knowledge graph contains 27 entities and a root node pointing to the global feature.

[0059] Specifically, define the adjacency matrix A used to represent the edge adj ={e ij}, where each node is a disease keyword. When the source node n i Connect to target node n j When setting ij is 1.

[0060] Optionally, a pre-trained BERT model can be used to initialize each entity and represent them as Then, use L grp Transformer Encoder modules are used to encode the features of the knowledge graph. That is:

[0061]

[0062] Among them, BERT (Bidirectional Encoder Representation from Transformers) is a pre-training model.

[0063] It should be noted that the present embodiment uses a relational self-attention (RSA) module to integrate the structure graph information. The adjacency matrix is ​​represented by Aadj Represents, which acts as a visibility mask in the standard self-attention mechanism.

[0064] Furthermore, in some embodiments, in S103, the medical knowledge graph feature h is integrated using graph attention. G and radiological image features h I , and use L crs A criss-cross attention module.

[0065] Specifically, a coding layer can be expressed as:

[0066]

[0067] Among them, in each cross attention head, the query comes from the radiological image features Here Key and Value come from the learned medical knowledge graph features h G .

[0068] It should be noted that before S104, a target prediction model is constructed and trained to utilize the target prediction model to output a predicted continuous semantic space vector, and then the generated medical imaging diagnosis report is obtained by mapping discrete words and continuous semantic space vectors.

[0069] Specifically, the training process of the target prediction model provided in the embodiment of the present application can refer to Figure 2 .like Figure 2 As shown, a target prediction model training method provided in an embodiment of the present application may include the following steps: S201 to S206.

[0070] S201. Acquire a plurality of radiological image samples of a preset part and a text report of each radiological image sample.

[0071] S202, for each radiology image sample and its corresponding text report, generate a corresponding pseudo label using the CheXpert model;

[0072] S203, extracting the radiological image sample by using a radiological image encoder to obtain sample image features;

[0073] S204, using a medical graph encoder to encode the medical knowledge graph sample corresponding to the radiological image sample to obtain a sample knowledge graph feature;

[0074] S205, using a graph attention mechanism to fuse the sample image features and the sample knowledge graph features to obtain a sample visual fusion feature;

[0075] S206. Using the sample visual fusion feature as the input of the prediction model and the text report as the output of the prediction model, the prediction model is trained, the target loss function is optimized using the Adam optimizer, and the weights of each layer of the prediction model are updated through a back propagation algorithm until a preset training stop condition is met, thereby obtaining a trained target prediction model, wherein the target loss function includes a text diffusion loss function and an auxiliary lesion classification loss function, and the auxiliary lesion classification loss function is constructed based on the pseudo-label.

[0076] In some embodiments, in S202, according to each radiology image sample and its corresponding text report, CheXpert is used to generate a corresponding pseudo label y for each image-text pair. CheXpert (A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparison) is a pre-trained model for automatically labeling medical image categories.

[0077] Specifically, each pseudo-label uses 14 categories to diagnose symptoms, namely:

[0078]

[0079] Among them, y i ∈{0,1} represents the prediction result of the i-th category, and a value of 1 indicates that the category exists. LC =14 is the number of categories, f lab Stands for Automated Radiology Report Marker.

[0080] In some embodiments, before S206 , a target loss function is constructed.

[0081] In the specific implementation, the text report corresponding to each radiological image sample is preprocessed to obtain several candidate discrete words; the word embedding function v i =g φ (w i ) maps the candidate discrete words into candidate continuous semantic space vectors z0={v1,v2,...,v N}, where v i ∈R d , φ represents the parameter of the embedding function; the candidate continuous semantic space vector is gradually noised to obtain random noise z t , that is, after t steps of forward diffusion process, z is obtained t ; The random noise z t Fusion features with the sample vision Input into the BART decoder, jointly optimize the text diffusion model loss and word embedding function parameters to obtain the text diffusion loss function

[0082] Among them, BART (Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension) is a pre-training model.

[0083] Optionally, the text diffusion loss function can be obtained according to the following formula:

[0084]

[0085] Among them, z θ It is a BART decoder. I is the extracted radiological image feature, where Represents mapping the vector v back to the word w, represents the mean estimate of the reverse diffusion process. The first part constrains the diffusion model, and the goal is to use reverse diffusion to convert the noised z t Restore to the initial semantic vector z0. The second part constrains the final diffusion vector z T The correlation between the distribution of and the initial vector z0 makes the diffusion process meaningful. The decoded output of the constrained diffusion model is In the first step (decoding from z1), it can be combined with the word embedding function g φ The fourth part is the loss of the language generation model, which is used to maximize the generation probability from z0 to the word sequence w.

[0086] In specific implementation, the sample visual fusion feature Perform mean pooling operation to obtain the visual fusion features after mean pooling Visual fusion features after mean pooling Perform linear classification to obtain the corresponding multi-label binary classification probability value According to the multi-label binary classification probability value and pseudo label y, and the auxiliary lesion classification loss function is calculated using the binary cross entropy function

[0087] Right now:

[0088]

[0089] Among them, Sigmoid represents the activation function, which is used for the output of hidden layer neurons.

[0090]

[0091] In specific implementation, according to the preset weight value, the text diffusion loss function and auxiliary lesion classification loss function According to the corresponding preset weight values, the target loss function is obtained. For example, the text diffusion loss function and auxiliary lesion classification loss function The weights in the overall loss function are all set to 0.5.

[0092] Right now:

[0093] Therefore, in S206, the back propagation algorithm (BP) is used to calculate the gradient, and the stochastic gradient descent method is used to optimize the target loss function to update the weights of each layer of the model.

[0094] Optionally, the stochastic gradient descent method used in this embodiment is, for example, but not limited to, the Adam (A method for Stochastic Optimization) algorithm, which is not specifically limited in this embodiment.

[0095] Optionally, the preset training stop conditions include: the learning rate of the Adam optimizer is set to 0.00003, the training batch size is set to 16, and the number of training iterations is set to 50. The maximum length of the radiology medical report is 100.

[0096] The above is a specific implementation method of the training to obtain the target prediction model provided in the embodiment of the present application. Medical reports are generated by introducing mapping learning between discrete words and continuous semantic space vectors in the text diffusion model; by designing a graph-guided image feature extractor guided by the radiology knowledge graph, key abnormalities in the image are flexibly identified; an auxiliary lesion classification loss module is designed, using pseudo labels as supervision signals to accurately align medical image features and text disease keyword representations, further improving the accuracy of the target prediction model for diagnostic report generation.

[0097] Back to Figure 1The medical imaging diagnosis report generation method shown. In some embodiments, in S104, the DDIM strategy is used to accelerate sampling. The random noise and visual fusion features are input into the decoder of BART to obtain the output of the previous time step, and the noise is gradually reduced to obtain, and the embedding function is used to select the most likely word at each position to generate a corresponding report. In this way, using the DDIM accelerated sampling strategy, the noise reduction from to can be completed with fewer steps. Then, the mapping from the continuous vector space to the discrete words is completed with the learned mapping function to obtain the medical imaging diagnosis report, that is, S105 and S106 are executed.

[0098] In another embodiment, the specific implementation method of generating a medical imaging diagnosis report can be referred to Figure 3 .

[0099] like Figure 3 As shown, another method for generating a medical imaging diagnostic report provided by an embodiment of the present application may include the following steps: first, for a radiological image, use a pre-trained MedClip medical visual encoder to extract medical image features; use a pre-trained BERT model to encode a chest knowledge graph as prior knowledge to help identify abnormal areas in the image; use a graph attention mechanism to integrate the extracted image features and medical knowledge graph features to obtain visual fusion features.

[0100] Furthermore, during the training process, the CheXpert model was used to mark pseudo labels for each image-report pair; the obtained visual fusion features were subjected to linear multi-classification to construct auxiliary lesion classification loss; in the forward diffusion, the continuous semantic vector obtained by mapping the obtained visual fusion features with the medical report was input into the BART decoder to predict the initialization vector for constructing the diffusion process loss function; the loss functions were merged and the parameters were optimized in the back propagation.

[0101] During the non-training process, the DDIM accelerated sampling strategy is used to reduce the number of diffusion steps. In each step of the reverse diffusion process, the denoised output and the visual fusion features are input into the BART decoder to obtain the output of the previous time step, and finally the predicted continuous semantic vector is obtained. Then, the generated radiology medical report is obtained by mapping discrete words and continuous semantic space vectors.

[0102] In addition, reference Figure 4 , the present application embodiment provides another method for generating a medical imaging diagnosis report. Figure 4As shown in the figure, for a given radiological image, the pre-trained MedClip medical visual encoder is used to initially extract local grid image features; the chest knowledge graph is used to integrate medical prior knowledge, and the features of the knowledge graph are encoded using the Transformer Encoder module; the graph attention mechanism is used to integrate the extracted medical graph features and radiological image features to obtain visual fusion features. Then, in the training case, the medical report is input as the training data set, and pseudo labels are generated based on the training set to generate auxiliary lesion classification losses, and then the final loss function is constructed, and the loss function is minimized during the training process through forward word mapping and forward diffusion. In the non-training case, the trained target prediction model is used for reverse diffusion and reverse word mapping to finally generate a medical report.

[0103] It should be noted that the above describes some embodiments of the present application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the above embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0104] Based on the same technical concept, corresponding to any of the above-mentioned embodiment methods, the present application also provides a medical imaging diagnosis report generating device 500.

[0105] like Figure 5 As shown, the medical imaging diagnosis report generating device 500 may include:

[0106] An extraction module 501 is used to extract a radiological image of a preset part using a pre-trained radiological image encoder to obtain radiological image features;

[0107] An encoding module 502 is used to encode a pre-constructed medical knowledge graph using a pre-trained medical graph encoder to obtain medical knowledge graph features, wherein the medical knowledge graph includes a number of disease keywords that match a preset part;

[0108] A fusion module 503 is used to fuse the radiological image features and the medical knowledge graph features using a graph attention mechanism to obtain a visual fusion feature;

[0109] An input module 504 is used to input the visual fusion feature into a pre-trained target prediction model to obtain a corresponding target continuous semantic space vector;

[0110] A mapping module 505 is used to map the target continuous semantic space vector into a target discrete word using a preset mapping function;

[0111] The generating module 506 is used to generate a medical imaging diagnosis report according to the target discrete words.

[0112] Optionally, the medical knowledge graph includes several entity nodes and a root node pointing to a global feature and an adjacency matrix, wherein the adjacency matrix represents edges, and the entity nodes correspond one-to-one to the disease keywords.

[0113] In some embodiments, the medical imaging diagnostic report generating device 500 further includes a training module ( Figure 5 Specifically, the training module includes the following units:

[0114] An acquisition unit, used for acquiring a plurality of radiological image samples of a preset part and a text report of each radiological image sample;

[0115] A labeling unit, used to generate corresponding pseudo labels using the CheXpert model for each radiology image sample and its corresponding text report;

[0116] An extraction unit, configured to extract the radiological image sample using a radiological image encoder to obtain sample image features;

[0117] An encoding unit, configured to encode the medical knowledge graph sample corresponding to the radiological image sample using a medical graph encoder to obtain a sample knowledge graph feature;

[0118] A fusion unit, used to fuse the sample image features and the sample knowledge graph features using a graph attention mechanism to obtain a sample visual fusion feature;

[0119] An updating unit is used to train the prediction model using the sample visual fusion feature as the input of the prediction model and the text report as the output of the prediction model, optimize the target loss function using the Adam optimizer, and update the weights of each layer of the prediction model through a back propagation algorithm until a preset training stop condition is met, thereby obtaining a trained target prediction model, wherein the target loss function includes a text diffusion loss function and an auxiliary lesion classification loss function, and the auxiliary lesion classification loss function is constructed based on the pseudo-label.

[0120] Optionally, the target loss function is obtained by adding the text diffusion loss function and the auxiliary lesion classification loss function according to their respective corresponding preset weight values ​​based on preset weight values.

[0121] In some optional embodiments, the training module also includes a first function construction unit, which is used to preprocess the text report corresponding to each radiological image sample to obtain a number of alternative discrete words; use a word embedding function to map the alternative discrete words into alternative continuous semantic space vectors; gradually add noise to the alternative continuous semantic space vectors to obtain random noise; input the random noise and the sample visual fusion features into the BART decoder, and jointly optimize the text diffusion model loss and the word embedding function parameters to obtain a text diffusion loss function.

[0122] In some optional embodiments, the training module also includes a second function construction unit, which is used to perform a mean pooling operation on the sample visual fusion features to obtain mean pooled visual fusion features; perform linear classification on the mean pooled visual fusion features to obtain corresponding multi-label binary classification probability values; and use a binary classification cross entropy function to calculate an auxiliary lesion classification loss function based on the multi-label binary classification probability values ​​and pseudo labels.

[0123] It should be noted that, for the convenience of description, the above device is described in various modules according to their functions. Of course, when implementing this application, the functions of each module can be implemented in the same or multiple software and / or hardware.

[0124] The device of the above embodiment is used to implement the corresponding medical imaging diagnosis report generation method in any of the above embodiments, and has the beneficial effects of the corresponding method embodiment, which will not be repeated here.

[0125] Based on the same technical concept, corresponding to any of the above-mentioned embodiment methods, the present application also provides an electronic device.

[0126] Figure 6 A more specific schematic diagram of the hardware structure of an electronic device provided by this embodiment is shown.

[0127] The electronic device 600 may include a processor 601 and a memory 602 storing computer program instructions.

[0128] Specifically, the processor 601 may include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiments of the present application.

[0129] The memory 602 may include a large capacity memory for data or instructions. By way of example and not limitation, the memory 602 may include a hard disk drive (HDD), a floppy disk drive, a flash memory, an optical disk, a magneto-optical disk, a magnetic tape, or a universal serial bus (USB) drive or a combination of two or more of these. In appropriate cases, the memory 602 may include a removable or non-removable (or fixed) medium. In appropriate cases, the memory 602 may be inside or outside the integrated gateway disaster recovery device. In a specific embodiment, the memory 602 is a non-volatile solid-state memory.

[0130] In certain embodiments, the memory may include a read-only memory (ROM), a random access memory (RAM), a magnetic disk storage medium device, an optical storage medium device, a flash memory device, an electrical, optical or other physical / tangible memory storage device. Thus, typically, the memory includes one or more tangible (non-transitory) computer-readable storage media (e.g., a memory device) encoded with software including computer-executable instructions, and when the software is executed (e.g., by one or more processors), it is operable to perform the operations described with reference to the method according to one aspect of the present application.

[0131] The processor 601 implements any one of the medical imaging diagnosis report generation methods in the above embodiments by reading and executing computer program instructions stored in the memory 602 .

[0132] In some examples, the electronic device 600 may further include a communication interface 603 and a bus 610. Figure 6 As shown, the processor 601, the memory 602, and the communication interface 603 are connected via a bus 610 and communicate with each other.

[0133] The communication interface 603 is mainly used to implement communication between various modules, devices, units and / or equipment in the embodiments of the present application.

[0134] Bus 610 includes hardware, software or both, and the parts of online data flow billing equipment are coupled to each other. For example, but not limitation, bus 610 may include accelerated graphics port (AGP) or other graphics bus, enhanced industrial standard architecture (EISA) bus, front-end bus (FSB), hypertransport (HT) interconnection, industrial standard architecture (ISA) bus, infinite bandwidth interconnection, low pin count (LPC) bus, memory bus, micro channel architecture (MCA) bus, peripheral component interconnection (PCI) bus, PCI-Express (PCI-X) bus, serial advanced technology attachment (SATA) bus, video electronics standard association local (VLB) bus or other suitable bus or two or more of these combinations. In appropriate cases, bus 610 may include one or more buses. Although the present application embodiment describes and shows a specific bus, the present application considers any suitable bus or interconnection.

[0135] Exemplarily, the electronic device 600 may be a mobile phone, a tablet computer, a laptop computer, a PDA, an in-vehicle electronic device, an ultra-mobile personal computer (UMPC), a netbook, or a personal digital assistant (PDA).

[0136] Based on the same technical concept, corresponding to any of the above-mentioned embodiments, the present application also provides a non-transitory computer-readable storage medium. The computer-readable storage medium stores computer program instructions; when the computer program instructions are executed by the processor, any of the medical imaging diagnostic report generation methods in the above-mentioned embodiments is implemented. Examples of computer-readable storage media include non-transitory computer-readable storage media, such as portable disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), portable compact disk read-only memories (CD-ROM), optical storage devices, magnetic storage devices, etc.

[0137] Based on the same technical concept, corresponding to any of the above-mentioned embodiments, the present application also provides a computer program product, which includes computer program instructions. In some embodiments, the computer program instructions can be executed by one or more processors of a computer so that the computer and / or the processor execute the medical imaging diagnostic report generation method. Corresponding to the execution subject corresponding to each step in each embodiment of the medical imaging diagnostic report generation method, the processor that executes the corresponding step may belong to the corresponding execution subject.

[0138] It should be clear that the present application is not limited to the specific configuration and processing described above and shown in the figures. For the sake of simplicity, a detailed description of the known method is omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present application is not limited to the specific steps described and shown, and those skilled in the art can make various changes, modifications and additions, or change the order between the steps after understanding the spirit of the present application.

[0139] The functional blocks shown in the above-described block diagram can be implemented as hardware, software, firmware or a combination thereof. When implemented in hardware, it can be, for example, an electronic circuit, an application specific integrated circuit (ASIC), appropriate firmware, a plug-in, a function card, etc. When implemented in software, the elements of the present application are programs or code segments that are used to perform the required tasks. The program or code segment can be stored in a machine-readable medium, or transmitted on a transmission medium or a communication link by a data signal carried in a carrier wave. "Machine-readable medium" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, optical fiber media, radio frequency (RF) links, etc. The code segment can be downloaded via a computer network such as the Internet, an intranet, etc.

[0140] It should also be noted that the exemplary embodiments mentioned in this application describe some methods or systems based on a series of steps or devices. However, this application is not limited to the order of the above steps, that is, the steps can be performed in the order mentioned in the embodiment, or in a different order from the embodiment, or several steps can be performed simultaneously.

[0141] The above reference is according to the method of the embodiment of the present application, the flow chart of the device (system) and the computer program product and / or the block diagram described various aspects of the present application.It should be understood that each square box in the flow chart and / or the block diagram and the combination of each square box in the flow chart and / or the block diagram can be realized by computer program instructions.These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer or other programmable data processing device to produce a machine so that these instructions executed by the processor of the computer or other programmable data processing device enable the realization of the function / action specified in one or more square boxes of the flow chart and / or the block diagram.Such a processor can be but is not limited to a general-purpose processor, a special-purpose processor, a special application processor or a field programmable logic circuit.It can also be understood that each square box in the block diagram and / or the flow chart and the combination of the square boxes in the block diagram and / or the flow chart can also be realized by the dedicated hardware that performs the specified function or action, or can be realized by the combination of dedicated hardware and computer instructions.

[0142] The above is only a specific implementation of the present application. Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working processes of the systems, modules and units described above can refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here. It should be understood that the protection scope of the present application is not limited to this. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in this application, and these modifications or replacements should be included in the protection scope of this application.

Claims

1. A method for generating a medical imaging diagnosis report, characterized in that: include: A radiological image encoder trained in advance is used to extract radiological images of a preset part to obtain radiological image features; Encoding a pre-constructed medical knowledge graph using a pre-trained medical graph encoder to obtain medical knowledge graph features, wherein the medical knowledge graph includes a number of disease keywords matching a preset part; The radiological image features and the medical knowledge graph features are fused using a graph attention mechanism to obtain visual fusion features; Inputting the visual fusion feature into a pre-trained target prediction model to obtain a corresponding target continuous semantic space vector; Using a preset mapping function, mapping the target continuous semantic space vector into a target discrete word; A medical imaging diagnosis report is generated according to the target discrete words.

2. The method according to claim 1, characterized in that Before inputting the visual fusion features into a pre-trained prediction model, the method further includes: Acquire a number of radiological image samples of a preset part and a text report of each radiological image sample; For each radiology image sample and its corresponding text report, the CheXpert model is used to generate corresponding pseudo labels; Extracting the radiological image sample using a radiological image encoder to obtain sample image features; Using a medical graph encoder to encode the medical knowledge graph sample corresponding to the radiological image sample to obtain a sample knowledge graph feature; The sample image features and the sample knowledge graph features are fused using a graph attention mechanism to obtain a sample visual fusion feature; The sample visual fusion feature is used as the input of the prediction model, and the text report is used as the output of the prediction model. The prediction model is trained, the target loss function is optimized using the Adam optimizer, and the weights of each layer of the prediction model are updated through the back propagation algorithm until the preset training stop condition is met, so as to obtain the trained target prediction model, wherein the target loss function includes a text diffusion loss function and an auxiliary lesion classification loss function, and the auxiliary lesion classification loss function is constructed according to the pseudo-label.

3. The method according to claim 1, characterized in that The medical knowledge graph includes a plurality of entity nodes, a root node pointing to a global feature, and an adjacency matrix, wherein the adjacency matrix represents edges, and the entity nodes correspond one-to-one to the disease keywords.

4. The method according to claim 2, characterized in that: The target loss function is obtained by adding the text diffusion loss function and the auxiliary lesion classification loss function according to the preset weight values ​​corresponding to each other.

5. The method according to claim 2 or 4, characterized in that: After fusing the sample image features and the sample knowledge graph features using the graph attention mechanism to obtain the sample visual fusion features, and before taking the sample visual fusion features as the input of a prediction model and taking the text report as the output of the prediction model and training the prediction model, the method further includes: Preprocess the text report corresponding to each radiology image sample to obtain several candidate discrete words; Mapping the candidate discrete words into candidate continuous semantic space vectors using a word embedding function; Stepwise adding noise to the candidate continuous semantic space vector to obtain random noise; The random noise and the sample visual fusion features are input into the decoder of BART, and the text diffusion model loss and word embedding function parameters are jointly optimized to obtain the text diffusion loss function.

6. The method according to claim 2 or 4, characterized in that: After fusing the sample image features and the sample knowledge graph features using the graph attention mechanism to obtain the sample visual fusion features, and before taking the sample visual fusion features as the input of a prediction model and taking the text report as the output of the prediction model and training the prediction model, the method further includes: Performing a mean pooling operation on the sample visual fusion features to obtain mean pooled visual fusion features; Perform linear classification on the visual fusion features after mean pooling to obtain the corresponding multi-label binary classification probability values; According to the multi-label binary classification probability values ​​and pseudo labels, the auxiliary lesion classification loss function is calculated using the binary classification cross entropy function.

7. A medical imaging diagnosis report generating device, characterized in that: The device comprises: An extraction module, used to extract radiological images of preset parts using a pre-trained radiological image encoder to obtain radiological image features; An encoding module, used to encode a pre-constructed medical knowledge graph using a pre-trained medical graph encoder to obtain medical knowledge graph features, wherein the medical knowledge graph includes a number of disease keywords matching a preset part; A fusion module, used for fusing the radiological image features with the medical knowledge graph features using a graph attention mechanism to obtain visual fusion features; An input module, used for inputting the visual fusion feature into a pre-trained target prediction model to obtain a corresponding target continuous semantic space vector; A mapping module, used to map the target continuous semantic space vector into a target discrete word using a preset mapping function; A generating module is used to generate a medical imaging diagnosis report according to the target discrete words.

8. An electronic device, characterized in that: The device comprises: a processor and a memory storing computer program instructions; when the processor calls the computer program instructions, the method for generating a medical imaging diagnosis report as described in any one of claims 1 to 6 is implemented.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer program instructions, and when the computer program instructions are called by the processor, the method for generating a medical imaging diagnostic report according to any one of claims 1 to 6 is implemented.

10. A computer program product, characterized in that When the instructions in the computer program product are executed by a processor of an electronic device, the electronic device executes the medical imaging diagnostic report generating method as described in any one of claims 1 to 6.

Images