Theme-guided chest image diagnosis report generation method and system

Abstract

The invention provides a theme-guided chest image diagnosis report generation method and system, and the method comprises the steps: extracting class tags from a diagnosis report through a theme construction module, and constructing a theme according to the class tags; clustering the diagnosis reports in the theme to obtain a theme report; processing the topic report by using a topic knowledge module to obtain a topic text representation and a topic map representation; processing the diagnosis report by utilizing a vision and text coding module and obtaining global vision representation and global text representation; generating a task based on the diagnosis report and performing theme modeling; obtaining related theme knowledge based on the global visual representation and the global text representation; and a theme-related diagnostic report is generated through a theme decoder module by utilizing knowledge. According to the method, theme modeling is carried out on the chest image report generation task, related themes are obtained through theme inference, and a smooth diagnosis report with important clinical value is generated.

Description

Technical Field

[0001] The present invention relates to the technical field of computer image processing, and in particular to a subject-guided chest imaging diagnosis report generation method and system. Background Art

[0002] With the significant progress in image description generation technology, automated chest imaging report generation technology has reduced the heavy workload of radiologists. A large number of studies in this field have adopted the "encoder-decoder" architecture to encode images into visual representations and then decode these representations into diagnostic reports. Multiple studies have shown that this method significantly improves performance in generating chest imaging diagnostic reports, marking a significant progress in this field.

[0003] Existing work has the following drawbacks: most models ignore the visual and textual biases in the medical field, and the generated reports lack accurate descriptions of abnormal lesion areas; most models only use the information of the image itself and ignore the expert knowledge in the field, which leads to an upper limit on the model effect and insufficient clinical value of the generated reports; existing knowledge-enhanced models focus on general knowledge and specific knowledge. Although general knowledge is extensive, it is too common and cannot provide accurate and detailed clinical knowledge. Although specific knowledge can provide detailed clinical knowledge, it is affected by the retrieval accuracy and the retrieved specific knowledge is usually not comprehensive enough. Summary of the invention

[0004] In view of the above situation, the main purpose of the present invention is to propose a subject-guided chest imaging diagnostic report generation method and system to solve the above technical problems.

[0005] The present invention proposes a topic-guided chest imaging diagnosis report generation method, which comprises the following steps: Step 1: Use the CheXBert network in the topic building module to extract class labels from the diagnosis report and build topics based on these class labels; Step 2: Cluster the diagnostic reports in the topic to obtain the topic report; The ClinicalBert network in the subject knowledge module is used to embed the features of the subject report to obtain the subject text features; the K-Means algorithm in the subject knowledge module is used to cluster the subject text features to obtain the subject center clusters; the subject center clusters are processed by multi-head attention mechanism, feedforward neural network and average operation in turn to obtain the subject text representation; Extract meta-phrases from the subject report, and calculate the point-by-point mutual information values ​​between adjacent language units in the phrases; select a preset number of phrases as candidate phrases based on the point-by-point mutual information values ​​in descending order, calculate the point-by-point mutual information values ​​between the candidate phrases and each topic, and select phrases as subject phrases based on the point-by-point mutual information values ​​between the candidate phrases and each topic; use the ClinicalBert network to embed features of the subject phrases to obtain subject phrase features; extract triplets from the diagnosis report using radiology atlases, and calculate the point-by-point mutual information values ​​between the triplets and each topic; select a preset number of triplets in descending order based on the point-by-point mutual information values ​​between the triplets and each topic to form a subject map, and form an adjacency matrix adjacent to the subject map; The ClinicalBert network is used to embed the features of nodes in the topic graph and obtain the node features. The graph encoder built on the basis of the multi-head attention module is used to decode the topic graph by combining the node features and the adjacency matrix adjacent to the topic graph to obtain the topic graph representation. Step 3: Extract chest images from the diagnosis report, and extract and obtain local visual features from the chest images based on the ResNet-101 network; use the local visual features to obtain global visual features by averaging, and then use the visual encoder built on the basis of the multi-head attention module to encode the global visual features and local visual features to obtain local visual representation and global visual representation; Based on the topic report, the text features of the diagnosis report are converted through the embedding function; based on the text features of the diagnosis report, the text encoder built on the basis of the multi-head attention module is used to encode the local text representation; and the local text representation is then averaged to obtain the global text representation; Step 4: Generate tasks based on the diagnostic report and perform topic modeling; Step 5: Based on the global visual representation and the global text representation, a probability model is constructed by constructing a topic inference module, and the probability model is used to infer the topic to obtain relevant topic knowledge; Step 6: Based on the relevant subject knowledge, a subject-related diagnostic report is generated through the subject decoder module.

[0006] The present invention also provides a subject-guided chest imaging diagnostic report generation system, the system comprising: Theme building blocks for: The CheXBert network in the topic building module is used to extract class labels from the diagnosis reports and build topics based on these class labels; Subject knowledge modules for: Cluster the diagnostic reports in the topic to obtain the topic report; The ClinicalBert network in the subject knowledge module is used to embed the features of the subject report to obtain the subject text features; the K-Means algorithm in the subject knowledge module is used to cluster the subject text features to obtain the subject center clusters; the subject center clusters are processed by multi-head attention mechanism, feedforward neural network and average operation in turn to obtain the subject text representation; Extract meta-phrases from the subject report, and calculate the point-by-point mutual information values ​​between adjacent language units in the phrases; select a preset number of phrases as candidate phrases based on the point-by-point mutual information values ​​in descending order, calculate the point-by-point mutual information values ​​between the candidate phrases and each topic, and select phrases as subject phrases based on the point-by-point mutual information values ​​between the candidate phrases and each topic; use the ClinicalBert network to embed features of the subject phrases to obtain subject phrase features; extract triplets from the diagnosis report using radiology atlases, and calculate the point-by-point mutual information values ​​between the triplets and each topic; sort in descending order, select a preset number of triplets based on the point-by-point mutual information values ​​between the triplets and each topic to form a subject map, and form an adjacency matrix adjacent to the subject map; The ClinicalBert network is used to embed the features of nodes in the topic graph and obtain the node features. The graph encoder built on the basis of the multi-head attention module is used to decode the topic graph by combining the node features and the adjacency matrix adjacent to the topic graph to obtain the topic graph representation. Visual and text encoding module for: Extract chest images from the diagnosis report, and extract and obtain local visual features from the chest images based on the ResNet-101 network; use the local visual features to obtain global visual features by averaging, and then use the visual encoder built on the basis of the multi-head attention module to encode the global visual features and local visual features to obtain local visual representation and global visual representation; Based on the topic report, the text features of the diagnosis report are converted through the embedding function; based on the text features of the diagnosis report, the text encoder built on the basis of the multi-head attention module is used to encode the local text representation; and the local text representation is then averaged to obtain the global text representation; Topic inference module, used to: Generate tasks based on diagnostic reports and perform topic modeling; Based on the global visual representation and global text representation, a probability model is constructed by constructing a topic inference module, and the probability model is used to infer the topic to obtain relevant topic knowledge; Theme decoder module for: Based on the relevant topic knowledge, a topic-related diagnostic report is generated through the topic decoder module.

[0007] Compared with the prior art, the present invention has the following beneficial effects: 1. This paper proposes a theme-guided chest imaging diagnostic report generation framework, which can infer the theme and align the visual and textual modalities, so as to pay more attention to the lesion area in report generation; 2. The present invention constructs a topic diagnosis report, a topic phrase and a topic map based on the topic to provide topic-level expert knowledge, which is both extensive and can provide detailed text and clinical information; 3. The present invention can inject relevant subject knowledge into the diagnostic report generation process by designing a subject-guided chest imaging diagnostic report generation decoder module, thereby improving the clinical accuracy of the generated diagnostic report; 4. The present invention can provide additional topic information in the decoding stage by designing the word bag planning task, and at the same time perform implicit word bag planning on the diagnostic report generation process to improve the quality of the generated diagnostic report.

[0008] Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description or learned through embodiments of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1 A flowchart of a subject-guided chest imaging diagnostic report generation method proposed by the present invention; Figure 2 This is a general architecture diagram of a subject-guided chest imaging diagnostic report generation method proposed by the present invention; Figure 3 This is a schematic diagram of the overall framework of a subject-guided chest imaging diagnostic report generation system proposed by the present invention. DETAILED DESCRIPTION

[0010] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be understood as limiting the present invention.

[0011] These and other aspects of the embodiments of the present invention will be apparent with reference to the following description and accompanying drawings. In these descriptions and accompanying drawings, some specific implementations of the embodiments of the present invention are specifically disclosed to represent some ways of implementing the principles of the embodiments of the present invention, but it should be understood that the scope of the embodiments of the present invention is not limited thereto.

[0012] See also Figure 1 and Figure 2The embodiment of the present invention provides a subject-guided chest imaging diagnosis report generation method, the method comprising the following steps: Step 1: Use the CheXBert network in the topic building module to extract class labels from the diagnosis report and build topics based on these class labels; Furthermore, in step 1, the CheXBert network was used to extract 14-class class labels from the chest imaging diagnosis report. The classification consisted of 12 diseases (e.g., “atelectasis”) and 2 specific conditions (e.g., “no findings”), where each class contained four labels: “positive,” “negative,” “uncertain,” and “not present.” Themes are constructed based on these 14 classification labels, with each category corresponding to two topics, namely "positive" and "negative", and each topic corresponding to two topic labels "appeared" and "not appeared". For example, the "atelectasis" category corresponds to the two topics "atelectasis positive" and "atelectasis negative", among which the "atelectasis positive: appeared" topic label corresponds to the "positive" and "uncertain" class labels, the "atelectasis negative: appeared" topic label corresponds to the "negative" class label, and the "atelectasis positive: did not appear" and "atelectasis negative: did not appear" correspond to the "not appeared" class label.

[0013] Step 2: Cluster the diagnostic reports in the topic to obtain the topic report; The ClinicalBert network in the subject knowledge module is used to embed the features of the subject report to obtain the subject text features; the K-Means algorithm in the subject knowledge module is used to cluster the subject text features to obtain the subject center clusters; the subject center clusters are processed by multi-head attention mechanism, feedforward neural network and average operation in turn to obtain the subject text representation; Extract meta-phrases from the subject report, calculate the point-by-point mutual information values ​​between adjacent language units in the phrases; select a preset number of phrases as candidate phrases based on the point-by-point mutual information in descending order, calculate the point-by-point mutual information values ​​between the candidate phrases and each topic, and select phrases as subject phrases based on the point-by-point mutual information values ​​between the candidate phrases and each topic; use the ClinicalBert network to embed features of the subject phrases to obtain subject phrase features; extract triplets from the diagnosis report using radiology atlases, calculate the point-by-point mutual information values ​​between the triplets and each topic; select a preset number of triplets in descending order based on the point-by-point mutual information values ​​between the triplets and each topic to form a subject map, and form an adjacency matrix adjacent to the subject map; The ClinicalBert network is used to embed the features of nodes in the topic graph and obtain the node features. The graph encoder built on the basis of the multi-head attention module is used to decode the topic graph by combining the node features and the adjacency matrix adjacent to the topic graph to obtain the topic graph representation. In step 2, the topic center points are processed by multi-head attention mechanism, feedforward neural network and average operation in turn to obtain the topic text representation. The relationship between the corresponding process is: ; in, represents the average operation, It means processed by feedforward neural network. It means that it is processed by a multi-head attention mechanism. Indicates the central point of the subject. represents the subject text representation; Extract meta-phrases from the topic report and calculate the point-by-point mutual information values ​​between adjacent language units in the phrases. The corresponding process has the following relationship: ; in, and Both represent adjacent language units in a phrase. represents the point-wise mutual information of adjacent language units, Indicates the probability of occurrence of a language unit; Using the graph encoder built on the basis of the multi-head attention module, the topic graph is decoded by combining the node features and the adjacency matrix adjacent to the topic graph to obtain the topic graph representation. The relationship between the corresponding process is: ; in, Represents the topic graph representation, It means that it has been processed by the graph encoder. Indicates that it has been processed by the ClinicalBert network. Represents the theme map, Represents the adjacency matrix.

[0014] Step 3: Extract chest images from the diagnosis report, and extract and obtain local visual features from the chest images based on the ResNet-101 network; use the local visual features to obtain global visual features by averaging, and then use the visual encoder built on the basis of the multi-head attention module to encode the global visual features and local visual features to obtain local visual representation and global visual representation; Based on the topic report, the text features of the diagnosis report are converted through the embedding function; based on the text features of the diagnosis report, the text encoder built on the basis of the multi-head attention module is used to encode the local text representation; and the local text representation is then averaged to obtain the global text representation; In step 3, the chest image is extracted from the diagnosis report. Based on the ResNet-101 network, local visual features are extracted from the chest image. The corresponding relationship is: ; in, Indicates processing by convolutional network ResNet-101, It means processed by multi-layer perceptron. Represents local visual features, Indicates Local visual features, represents the number of local visual features, Indicates chest imaging picture; Using the visual encoder built on the basis of the multi-head attention module, the global visual features and local visual features are encoded to obtain the local visual representation and the global visual representation. The relationship between the corresponding processes is: ; in, represents the global visual representation, represents the local visual representation, represents the global visual features, Represents processing by the visual encoder, Indicates A local visual representation, represents the number of local visual representations; Based on the subject report, the text features of the diagnostic report are converted through the embedding function. The corresponding relationship in the process is: ; in, It means that it has been processed by the embedded function. represents the position embedding feature, Text features representing diagnostic reports, Indicates the diagnostic report. Indicates text features, Indicates the number of report text features; Based on the text features of the diagnostic report, the local text representation is obtained by encoding through a text encoder built on the basis of a multi-head attention module. The relationship between the corresponding process is: ; in, Indicates that it has been processed by the embedded function. represents a local text representation, Indicates processing through a text editor. Indicates A local text representation, Indicates the number of local text representations.

[0015] Step 4: Generate tasks based on the diagnostic report and perform topic modeling; Furthermore, in step 4, tasks are generated based on the diagnostic report and topic modeling is performed. The process of topic modeling also includes the following steps: Introducing the topic mixed variable and the topic variable into the conditional log-likelihood, we can get the maximized conditional log-likelihood. The corresponding process has the following relationship: ; in, represents the variational distribution, represents the posterior distribution, represents the maximization of the conditional log-likelihood, represents the KL divergence term, represents the subject mixed variable, represents the subject variable, The variable representing the integral symbol; By minimizing the KL divergence term in maximizing the conditional log-likelihood, we can obtain the lower bound of the evidence for the log-likelihood. The relationship between the corresponding process is: ; in, represents the evidential lower bound of the log-likelihood; Given the subject mixed variables and chest imaging images, the subject variables are conditionally independent, and the lower bound of the evidence for the log likelihood is expressed as: ; During the training process, the variational distribution is made dependent on the diagnostic report and conditioned, and then the objective function for the diagnostic report is derived based on the lower bound of the evidence of the log-likelihood. The relationship between the corresponding process is: ; in, represents the variational distribution, represents the posterior distribution, represents the hyperparameter, Indicates Monte Carlo samples, It means that the posterior distribution is given by The topic inferred by the Monte Carlo sample, represents the number of Monte Carlo sampling, Indicates that the report words, Indicates that the report words, Indicates the conditional probability of the first word in the report based on the condition, The loss function representing the evidence lower bound of the log-likelihood; The average value of the topic mixture variable sample is used to infer the topic and assist in generation, while simplifying the lower bound of the evidence of the log likelihood. The simplified expression is: ; in, represents the sample mean of the subject mixed variable.

[0016] Step 5: Based on the global visual representation and the global text representation, a probability model is constructed by constructing a topic inference module, and the probability model is used to infer the topic to obtain relevant topic knowledge; Furthermore, in step 5, based on the global visual representation and the global text representation, a probability model is constructed by constructing a topic inference module, and the topic is inferred using the probability model to obtain relevant topic knowledge, which specifically includes the following sub-steps: Parameterize the variational distribution and the posterior distribution, and the relationship between the corresponding processes is: ; in, and Both represent learnable standard deviation parameters, represents the global visual representation, and All represent normal distribution, represents the identity matrix; Introducing the probability distribution of the mixed subject variable into the Gaussian mixture distribution, the resulting calculation formula is: ; in, Represents the subject text representation, represents the mixing coefficient, represents the probability distribution of the mixed subject variables, represents the learnable standard deviation parameter, Indicates The distribution probability of the topics, represents the probability distribution of the subject variable, Indicates based on Under the conditions of The conditional probability distribution of Indicates Thematic variables for each subject; Based on the conditional probability calculation of topic mixing weights, the corresponding process has the following relationship: ; in, Indicates based on Inferred to be The probability of a topic, represents the number of topics, Represents the subject text representation, represents the mixing coefficient, represents the learnable standard deviation parameter; Based on the topic mixing weight, a preset number of topics are selected to determine the relevant topics according to the size of the topic mixing weight; After obtaining the relevant topics, the phrase features of the relevant topics are concatenated to obtain the phrase features; the icons of the relevant topics are concatenated to obtain the graph representation; the local visual features are aligned with the graph representation, and finally the aligned visual representation is obtained. The relationship between the corresponding process is: ; in, It means that it is processed by the cross attention mechanism. A visual representation of the graph alignment, Representation diagram representation; In the step of building a probability model based on global visual representation and global text representation by constructing a topic inference module and using the probability model to infer the topic to obtain relevant topic knowledge, a classification loss function is introduced. The expression of the classification loss function is: ; in, represents the hyperparameter controlling the unbalanced topic contribution, represents the classification loss function.

[0017] Step 6: Generate a topic-related diagnostic report through a topic decoder module based on the relevant topic knowledge; Further, in step 6, based on the relevant subject knowledge, a subject-related diagnostic report is generated through a subject decoder module. The step of generating a subject-related diagnostic report specifically includes the following sub-steps: Shift the diagnostic report right by one position, and then map the shifted diagnostic report through the Embed layer to obtain the initial hidden state; Before the initial hidden state enters the decoder, each initial hidden state is fused with the mean text representation of the relevant topic obtained by topic inference to obtain the intermediate hidden state in the decoding process. The relationship between the corresponding process is: ; in, represents the mean of the text representation of the related topics obtained by topic inference, represents the intermediate hidden state in the decoding process, represents the initial hidden state; The decoder is generated by using the intermediate hidden state in the decoding process; the relationship between the corresponding process is expressed as: ; in, Represents the multi-head self-attention operation through mask, Indicates that attention is assisted by the theme phrase. , and Both represent the intermediate hidden states in the decoding process. represents the weight matrix, represents the bias vector, Indicates phrase features. The hidden state Tokens; In the process of generating the decoder, the topic phrase is used to assist attention to avoid redundant descriptions caused by visual bias. The expression of the topic phrase assisted attention is: ; in, Indications are processed through cross-attention; In the process of generating topic-related diagnostic reports, the word bag planning task is introduced in the decoding stage. The category distribution expression in the word bag planning task is: ; in, It means processed by multi-layer perceptron. represents the processing of the diagnostic report decoder guided through the entire topic, Represents a mixed representation of topics, Represents the category distribution; In the step of generating topic-related diagnostic reports through the topic decoder module based on relevant topic knowledge, the bag-of-words planning loss function is introduced to maximize the log-likelihood of correctly predicting the occurrence of each word in the report. The expression of the bag-of-words planning loss function is: ; in, represents the bag-of-words planning loss, Indicates the first The estimated probability of a word.

[0018] Furthermore, the total loss is composed of ELBO loss, classification loss, and bag-of-words planning loss. The expression of the total loss is: ; in, represents the total loss, represents the hyperparameters that control the impact of the classification task, Represents the hyperparameters that control the impact of the bag-of-words planning task.

[0019] Please refer to Figure 3 The embodiment of the present invention further provides a subject-guided chest imaging diagnosis report generation system, the system comprising: Theme building blocks for: The CheXBert network is used to extract class labels from diagnostic reports and to construct topics based on these class labels. Subject knowledge modules for: Cluster the diagnostic reports in the topic to obtain the topic report; The ClinicalBert network is used to embed the features of the topic report to obtain the topic text features; the K-Means algorithm in the topic knowledge module is used to cluster the topic text features to obtain the topic center clusters; the topic center clusters are processed by multi-head attention mechanism, feedforward neural network and average operation in turn to obtain the topic text representation; Extract meta-phrases from the subject report, and calculate the point-by-point mutual information values ​​between adjacent language units in the phrases; select a preset number of phrases as candidate phrases based on the point-by-point mutual information values ​​in descending order, calculate the point-by-point mutual information values ​​between the candidate phrases and each topic, and select phrases as subject phrases based on the point-by-point mutual information values ​​between the candidate phrases and each topic; use the ClinicalBert network to embed features of the subject phrases to obtain subject phrase features; extract triplets from the diagnosis report using radiology atlases, and calculate the point-by-point mutual information values ​​between the triplets and each topic; select a preset number of triplets in descending order based on the point-by-point mutual information values ​​between the triplets and each topic to form a subject map, and form an adjacency matrix adjacent to the subject map; The ClinicalBert network is used to embed the features of nodes in the topic graph and obtain the node features. The graph encoder built on the basis of the multi-head attention module is used to decode the topic graph by combining the node features and the adjacency matrix adjacent to the topic graph to obtain the topic graph representation. Visual and text encoding module for: Extract chest images from the diagnosis report, and extract and obtain local visual features from the chest images based on the ResNet-101 network; use the local visual features to obtain global visual features by averaging, and then use the visual encoder built on the basis of the multi-head attention module to encode the global visual features and local visual features to obtain local visual representation and global visual representation; Based on the topic report, the text features of the diagnosis report are converted through the embedding function; based on the text features of the diagnosis report, the text encoder built on the basis of the multi-head attention module is used to encode the local text representation; and the local text representation is then averaged to obtain the global text representation; Topic inference module, used to: Generate tasks based on diagnostic reports and perform topic modeling; Based on the global visual representation and global text representation, a probability model is constructed by constructing a topic inference module, and the probability model is used to infer the topic to obtain relevant topic knowledge; Theme decoder module for: Generate topic-related diagnostic reports based on relevant topic knowledge.

[0020] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0021] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0022] The above-mentioned embodiments only express several implementation methods of the present invention, and the description thereof is relatively specific and detailed, but it cannot be understood as limiting the scope of the patent of the present invention. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the attached claims.

Claims

1. A topic-guided chest imaging diagnostic report generation method, the method is implemented based on a deep learning model, the deep learning model is composed of a topic construction module, a topic knowledge module, a visual and text encoding module, a topic inference module and a topic decoder module, the method comprises the following steps: Step 1: Use the CheXBert network in the topic building module to extract class labels from the diagnosis report and build topics based on the class labels; Step 2: Cluster the diagnostic reports in the topic to obtain the topic report; The ClinicalBert network in the subject knowledge module is used to embed the features of the subject report to obtain the subject text features; The K-Means algorithm in the topic knowledge module is used to cluster the topic text features to obtain the topic center clusters; the topic center clusters are processed by multi-head attention mechanism, feedforward neural network and average operation in turn to obtain the topic text representation; Extracting meta-phrases from the topic reports, calculating point-by-point mutual information values ​​between adjacent language units in the phrases; selecting a preset number of phrases as candidate phrases based on the point-by-point mutual information values ​​in descending order, calculating the point-by-point mutual information values ​​between the candidate phrases and each topic, and selecting phrases as topic phrases based on the point-by-point mutual information values ​​between the candidate phrases and each topic; The feature embedding of subject phrases is performed using the ClinicalBert network to obtain the subject phrase features; the triplets are extracted from the diagnosis report using the radiology atlas, and the point-by-point mutual information values ​​between the triplets and each subject are calculated; a preset number of triplets are selected in descending order based on the point-by-point mutual information values ​​between the triplets and each subject to form a subject graph, and an adjacency matrix adjacent to the subject graph is formed; Use ClinicalBert network to embed features of nodes in the topic graph and obtain node features; Using a graph encoder built on the basis of a multi-head attention module, the topic graph is decoded by combining node features and the adjacency matrix adjacent to the topic graph to obtain the topic graph representation; Step 3: Extract chest images from the diagnosis report, and extract and obtain local visual features from the chest images based on the ResNet-101 network; use the local visual features to obtain global visual features by averaging, and then use the visual encoder built on the basis of the multi-head attention module to encode the global visual features and local visual features to obtain local visual representation and global visual representation; Based on the topic report, the text features of the diagnosis report are converted through the embedding function; based on the text features of the diagnosis report, the text encoder built on the basis of the multi-head attention module is used to encode the local text representation; and the local text representation is then averaged to obtain the global text representation; Step 4: Generate tasks based on the diagnostic report and perform topic modeling; Step 5: Based on the global visual representation and the global text representation, a probability model is constructed by constructing a topic inference module, and the probability model is used to infer the topic to obtain relevant topic knowledge; Step 6: Based on the relevant subject knowledge, a subject-related diagnostic report is generated through the subject decoder module.

2. A subject-guided chest imaging diagnostic report generation method according to claim 1, characterized in that: In step 2, the topic center points are processed by the multi-head attention mechanism, the feedforward neural network and the average operation in sequence to obtain the topic text representation. The relationship between the corresponding processes is: ； in, represents the average operation, It means processed by feedforward neural network. It means that it is processed by a multi-head attention mechanism. Indicates the central point of the subject. represents the subject text representation; Extract meta-phrases from the topic report and calculate the point-by-point mutual information values ​​between adjacent language units in the phrases. The corresponding process has the following relationship: ； in, and Both represent adjacent language units in a phrase. represents the point-wise mutual information of adjacent language units, Indicates the probability of occurrence of a language unit; Using the graph encoder built on the basis of the multi-head attention module, the topic graph is decoded by combining the node features and the adjacency matrix adjacent to the topic graph to obtain the topic graph representation. The relationship between the corresponding process is: ； in, Represents the topic graph representation, It means that it has been processed by the graph encoder. Indicates that it has been processed by the ClinicalBert network. Represents the theme map, Represents the adjacency matrix.

3. A subject-guided chest imaging diagnostic report generation method according to claim 2, characterized in that: In step 3, a chest image is extracted from the diagnosis report, and local visual features are extracted and obtained from the chest image based on the ResNet-101 network. The relationship between the corresponding process is: ； in, Indicates processing by convolutional network ResNet-101, It means processed by multi-layer perceptron. Represents local visual features, Indicates Local visual features, represents the number of local visual features, Indicates chest imaging picture; Using the visual encoder built on the basis of the multi-head attention module, the global visual features and local visual features are encoded to obtain the local visual representation and the global visual representation. The relationship between the corresponding processes is: ； in, represents the global visual representation, represents the local visual representation, represents the global visual features, Represents processing by the visual encoder, Indicates A local visual representation, represents the number of local visual representations; Based on the subject report, the text features of the diagnostic report are converted through the embedding function. The corresponding relationship in the process is: ； in, Indicates that it has been processed by the embedded function. represents the position embedding feature, Text features representing diagnostic reports, Indicates the diagnostic report. Indicates text features, Indicates the number of report text features.

4. A subject-guided chest imaging diagnostic report generation method according to claim 3, characterized in that: In step 3, based on the text features of the diagnosis report, the local text representation is obtained by encoding through a text encoder built on the basis of a multi-head attention module. The relationship between the corresponding process is: ； in, Indicates that it has been processed by the embedded function. represents a local text representation, Indicates processing through a text editor. Indicates A local text representation, Indicates the number of local text representations.

5. A subject-guided chest imaging diagnostic report generation method according to claim 4, characterized in that: In step 4, tasks are generated based on the diagnostic report and topic modeling is performed. The process of topic modeling also includes the following steps: Introducing the topic mixed variable and the topic variable into the conditional log-likelihood, we can get the maximized conditional log-likelihood. The corresponding process has the following relationship: ； in, represents the variational distribution, represents the posterior distribution, represents the maximization of the conditional log-likelihood, represents the KL divergence term, represents the subject mixed variable, represents the subject variable, The variable representing the integral symbol; By minimizing the KL divergence term in maximizing the conditional log-likelihood, we can obtain the lower bound of the evidence for the log-likelihood. The relationship between the corresponding process is: ； in, represents the evidential lower bound of the log-likelihood; Given the subject mixed variables and chest imaging images, the subject variables are conditionally independent, and the lower bound of the evidence for the log likelihood is expressed as: ； During the training process, the variational distribution is made dependent on the diagnostic report and conditioned, and then the objective function for the diagnostic report is derived based on the lower bound of the evidence of the log-likelihood. The relationship between the corresponding process is: ； in, represents the variational distribution, represents the posterior distribution, represents the hyperparameter, Indicates MonteCarlo samples, It means that the posterior distribution is given by The topic inferred by the Monte Carlo sample, represents the number of Monte Carlo sampling, Indicates that the report words, Indicates that the report words, Indicates the conditional probability of the first word in the report based on the condition, The loss function representing the evidence lower bound of the log-likelihood; The average value of the topic mixture variable sample is used to infer the topic and assist in generation, while simplifying the lower bound of the evidence of the log likelihood. The simplified expression is: ； in, represents the sample mean of the subject mixed variable.

6. A subject-guided chest imaging diagnostic report generation method according to claim 5, characterized in that: In step 5, based on the global visual representation and the global text representation, a probability model is constructed by constructing a topic inference module, and the topic is inferred using the probability model to obtain relevant topic knowledge, which specifically includes the following sub-steps: Parameterize the variational distribution and the posterior distribution, and the relationship between the corresponding processes is: ； in, and Both represent learnable standard deviation parameters, represents the global visual representation, and All represent normal distribution, represents the identity matrix; Introducing the probability distribution of the mixed subject variable into the Gaussian mixture distribution, the resulting calculation formula is: ； in, Represents the subject text representation, represents the mixing coefficient, represents the probability distribution of the mixed subject variables, represents the learnable standard deviation parameter, Indicates The distribution probability of the topics, represents the probability distribution of the subject variable, Indicates based on Under the conditions of The conditional probability distribution of Indicates Thematic variables for each subject; Based on the conditional probability calculation of topic mixing weights, the corresponding process has the following relationship: ； in, According to Inferred to be The probability of a topic, represents the number of topics, Represents the subject text representation, represents the mixing coefficient, represents the learnable standard deviation parameter; Based on the topic mixing weight, a preset number of topics are selected to determine the relevant topics according to the size of the topic mixing weight; After obtaining the relevant topics, the phrase features of the relevant topics are concatenated to obtain the phrase features; the icons of the relevant topics are concatenated to obtain the graph representation; the local visual features are aligned with the graph representation, and finally the aligned visual representation is obtained. The relationship between the corresponding process is: ； in, It means that it is processed by the cross attention mechanism. A visual representation of the graph alignment, Representation diagram representation.

7. A subject-guided chest imaging diagnostic report generation method according to claim 6, characterized in that: In the step of building a probability model based on global visual representation and global text representation by constructing a topic inference module and using the probability model to infer the topic to obtain relevant topic knowledge, a classification loss function is introduced. The expression of the classification loss function is: ； in, represents the hyperparameter controlling the unbalanced topic contribution, represents the classification loss function.

8. A subject-guided chest imaging diagnostic report generation method according to claim 7, characterized in that: In step 6, based on the relevant subject knowledge, a subject-related diagnostic report is generated through a subject decoder module. The step of generating a subject-related diagnostic report specifically includes the following sub-steps: Shift the diagnostic report right by one position, and then map the shifted diagnostic report through the Embed layer to obtain the initial hidden state; Before the initial hidden state enters the decoder, each initial hidden state is fused with the mean text representation of the relevant topic obtained by topic inference to obtain the intermediate hidden state in the decoding process. The relationship between the corresponding process is: ； in, represents the mean of the text representation of the related topics obtained by topic inference, represents the intermediate hidden state in the decoding process, represents the initial hidden state; The decoder is generated by using the intermediate hidden state in the decoding process; the relationship between the corresponding process is expressed as: ； in, Represents the multi-head self-attention operation through mask, Indicates that attention is assisted by the theme phrase. , and Both represent the intermediate hidden states in the decoding process. represents the weight matrix, represents the bias vector, Indicates phrase features. The hidden state Tokens; In the process of generating the decoder, the topic phrase is used to assist attention to avoid redundant descriptions caused by visual bias. The expression of the topic phrase assisted attention is: ； in, Indications are processed through cross-attention; In the process of generating topic-related diagnostic reports, the word bag planning task is introduced in the decoding stage. The category distribution expression in the word bag planning task is: ； in, It means processed by multi-layer perceptron. represents the processing of the diagnostic report decoder guided through the entire topic, Represents a mixed representation of topics, Represents the category distribution.

9. A subject-guided chest imaging diagnostic report generation method according to claim 8, characterized in that: In the step of generating topic-related diagnostic reports through the topic decoder module based on relevant topic knowledge, the bag-of-words planning loss function is introduced to maximize the log-likelihood of correctly predicting the occurrence of each word in the report. The expression of the bag-of-words planning loss function is: ； in, represents the bag-of-words planning loss, Indicates the first The estimated probability of a word.

10. A subject-guided chest imaging diagnostic report generation system, characterized in that: The system applies the subject-guided chest imaging diagnostic report generation method according to any one of claims 1 to 9, and the system comprises: Theme building blocks for: The CheXBert network is used to extract class labels from diagnostic reports and to construct topics based on these class labels. Subject knowledge modules for: Cluster the diagnostic reports in the topic to obtain the topic report; The ClinicalBert network is used to embed the features of the topic report to obtain the topic text features; the K-Means algorithm in the topic knowledge module is used to cluster the topic text features to obtain the topic center clusters; the topic center clusters are processed by multi-head attention mechanism, feedforward neural network and average operation in turn to obtain the topic text representation; Extract meta-phrases from the subject report, and calculate the point-by-point mutual information values ​​between adjacent language units in the phrases; select a preset number of phrases as candidate phrases based on the point-by-point mutual information values ​​in descending order, calculate the point-by-point mutual information values ​​between the candidate phrases and each topic, and select phrases as subject phrases based on the point-by-point mutual information values ​​between the candidate phrases and each topic; use the ClinicalBert network to embed features of the subject phrases to obtain subject phrase features; extract triplets from the diagnosis report using radiology atlases, and calculate the point-by-point mutual information values ​​between the triplets and each topic; select a preset number of triplets in descending order based on the point-by-point mutual information values ​​between the triplets and each topic to form a subject map, and form an adjacency matrix adjacent to the subject map; The ClinicalBert network is used to embed the features of nodes in the topic graph and obtain the node features. The graph encoder built on the basis of the multi-head attention module is used to decode the topic graph by combining the node features and the adjacency matrix adjacent to the topic graph to obtain the topic graph representation. Visual and text encoding module for: Extract chest images from the diagnosis report, and extract and obtain local visual features from the chest images based on the ResNet-101 network; use the local visual features to obtain global visual features by averaging, and then use the visual encoder built on the basis of the multi-head attention module to encode the global visual features and local visual features to obtain local visual representation and global visual representation; Based on the topic report, the text features of the diagnosis report are converted through the embedding function; based on the text features of the diagnosis report, the text encoder built on the basis of the multi-head attention module is used to encode the local text representation; and the local text representation is then averaged to obtain the global text representation; Topic inference module, used to: Generate tasks based on diagnostic reports and perform topic modeling; Based on the global visual representation and global text representation, a probability model is constructed by constructing a topic inference module, and the probability model is used to infer the topic to obtain relevant topic knowledge; Theme decoder module for: Generate topic-related diagnostic reports based on relevant topic knowledge.

Images