Medical image report generation method and related equipment

Abstract

The invention provides a medical image report generation method and related equipment. The method comprises the following steps: acquiring medical image data; and inputting the medical image data into a pre-trained medical image report generation model, and outputting a medical image report corresponding to the medical image data by the medical image report generation model. The accuracy of the generated medical image report can be improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer image processing, and in particular relates to a medical imaging report generating method and related equipment. Background Art

[0002] Medical imaging report generation is similar to the image description task, which involves deriving a corresponding description from an image. Pretrained convolutional neural networks (such as ResNet-101) are typically used to extract image features, after which the initial features are fed into an encoder-decoder network to generate sentences. With the advent of the Transformer, many studies have begun using various attention mechanisms to improve performance. Recently, some methods have explored the use of additional radiological knowledge to assist in report generation. Among them, Yang et al. proposed a knowledge-enhancing attention mechanism to improve generation quality. Some studies have also employed multi-task learning and leveraged symptom classification information to assist in report generation.

[0003] Although some progress has been made, challenges still exist in the process of extracting radiological knowledge to assist in report generation. First, directly using disease or symptom classification results to assist in generation (for example, the presence of edema: edema-positive) often ignores the refinement of the degree of negative or positive. For example, "edema-positive" cannot reflect the severity of edema, while more detailed descriptions (such as "probable mild interstitialedema", which means there may be mild interstitial edema) can provide more accurate pathological information and reflect the specific extent of the disease. In addition, the relationship between disease and symptoms and between different diseases is ignored, which limits the model's understanding and expression of pathological characteristics.

[0004] Second, traditional attention-based methods fail to fully utilize valuable image-level global features, which may lead to missing abnormal information during report generation, e.g. Figure 1 In the figure, the upper image is a normal image, and the lower image has edema. It can be seen that the images labeled "edema" and "normal" are very similar in overall visual appearance. The edges of the fluid accumulation area in the edema image are blurred and difficult to distinguish clearly from the healthy tissue area. Ignoring the global features of the image may lead to the omission of key pathological features ("whether there is edema"), which will directly affect the accuracy of medical imaging reports. Finally, the computational cost of the Transformer network is high (O(N 2 )). When processing high-resolution raw X-ray medical images, its attention mechanism usually performs poorly in terms of computational speed and memory consumption. Summary of the Invention

[0005] The technical problem to be solved by the present invention is to provide a medical imaging report generation method and related equipment to improve the accuracy of medical imaging reports.

[0006] In a first aspect, the present invention provides a method for generating a medical imaging report, the method comprising the following steps:

[0007] Acquire medical imaging data; medical imaging data includes medical images and image description text;

[0008] Medical imaging data is input into a pre-trained medical imaging report generation model, and the medical imaging report generation model outputs a medical imaging report corresponding to the medical imaging data; the medical imaging report generation model includes a visual feature extractor, a fusion encoder, a topic classifier and a fusion decoder; the visual feature extractor is used to extract the visual features of the medical image and obtain a visual feature sequence; the fusion encoder is used to extract contextual semantic features from the visual feature sequence, and adaptively encode the contextual semantic features to obtain the semantic features of the medical image; the topic classifier is used to extract disease labels from the image description text, construct a disease semantic graph with dynamic edge weights based on the disease labels and semantic features, obtain the labeling probability corresponding to each disease label in the disease semantic graph based on Bayesian state embedding, and obtain the disease feature vector corresponding to the medical imaging data based on the labeling probability; the fusion decoder is used to generate a medical imaging report based on the semantic features and disease feature vector.

[0009] Optionally, contextual semantic features are extracted from the visual feature sequence, and the contextual semantic features are adaptively encoded to obtain semantic features of the medical image, including:

[0010] Use Mamba encoder to extract contextual semantic features;

[0011] Based on the contextual semantic features, the scores of multiple pre-configured candidate expert networks are calculated respectively, and the final expert network for semantic feature extraction is determined from the multiple candidate expert networks based on the scores; the candidate expert networks are trainable feedforward neural networks used to extract the semantic features of medical images, and the model parameter values ​​of different candidate expert networks are not exactly the same;

[0012] The final expert network is used to obtain semantic features.

[0013] Optionally, based on the contextual semantic features, scores of each of the pre-configured multiple candidate expert networks are calculated, and a final expert network for semantic feature extraction is determined from the multiple candidate expert networks based on the scores, including:

[0014] Calculate the score of each candidate expert network; for the input context semantic feature x, pre-configure N expertscandidate expert networks, each of which has a corresponding trainable weight matrix W i , calculated by formula h i (x) = W i x obtains the scores of each candidate expert network h i (x), where h i (x) represents the score of the i-th candidate expert network for the contextual semantic feature x, W i is a trainable parameter used to map input features into the score space;

[0015] According to the scores, the selection probability distribution of the multiple candidate expert networks is calculated; based on the scores h of each candidate expert network i (x), calculate the probability distribution p through the softmax function i (x), the formula is

[0016] According to the selection probability distribution, the candidate expert network with the highest selection probability is determined as the final expert network.

[0017] Optionally, extract disease labels from the image description text and construct a disease semantic graph with dynamic edge weights based on the disease labels and semantic features, including:

[0018] Use the automatic label generation tool to extract labels from the image description text to obtain multiple disease labels corresponding to the image description text;

[0019] Calculate the similarity between disease labels based on the number of occurrences of each disease label;

[0020] A disease semantic graph is constructed based on the graph attention mechanism and multiple disease labels. The disease semantic graph includes multiple nodes, which correspond one-to-one to multiple disease labels. The weight of the edge connecting two nodes is calculated by the similarity between the two nodes.

[0021] Optionally, obtain the labeling probability corresponding to each disease label in the disease semantic graph based on Bayesian state embedding, including:

[0022] By calculating the formula

[0023] M state =μ+∈·exp(0.5·logσ 2 ),∈~N(0,1)

[0024] Get the labeling probability M state ; Among them, μ, σ are the Gaussian distribution mean and Gaussian distribution variance corresponding to each disease label, and ∈ represents the learnable parameters.

[0025] Optionally, the disease feature vector corresponding to the medical imaging data is obtained based on the labeling probability, including:

[0026] Based on the cross-attention mechanism and semantic features, the correspondence between each disease label and the image region of the medical image is determined, and the most relevant image representation of each disease label is determined based on the correspondence;

[0027] Calculate the disease state matrix based on the cross-attention mechanism, labeling probability and correspondence;

[0028] Calculate the disease feature vector based on the disease state matrix and labeled probabilities.

[0029] Optionally, the disease state matrix can be expressed as:

[0030] M′ state =CAM(F′ g ,M state )=α disease M state W V

[0031] F′ g =CAM(M,F g )

[0032] CAM(X,Y)=αYW V

[0033]

[0034] Among them, CAM(X,Y) is the formula description of the attention mechanism, W q 、W K 、W V Represents the linear weight matrices used for trainable query, key, and value transformations, respectively, to map input features to the attention space. α represents the attention weight coefficient calculated by the softmax function, which is used to measure the correlation between different feature positions. M represents the features obtained by the image encoder. F g Indicates disease information, F g ′ represents the extraction of radiological regions of interest for each node in the disease semantic graph through the cross-attention mechanism, thereby retrieving the image region most relevant to each disease symptom information, which can be understood as the most relevant image feature for each disease label. statE is the probability information of the disease, α disease Represents the probability of different diseases, M′ state represents the disease state matrix.

[0035] Optionally, calculate the disease feature vector based on the disease state matrix and the labeled probabilities, including:

[0036] By calculating the formula

[0037] Z out =M′ state +y′M state

[0038]

[0039] Get the disease feature vector Z out ; Among them, y′ represents the labeling probability, y′=α disease , τ represents the preset threshold value, which is used to determine the disease status.

[0040] In a second aspect, the present invention provides a terminal device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned method when executing the computer program.

[0041] In a third aspect, the present invention provides a computer-readable storage medium storing a computer program, which implements the above method when executed by a processor.

[0042] The beneficial effects of the present invention are:

[0043] The medical image report generation method provided by the present invention discloses a medical image report generation model. The fusion encoder is used to extract contextual semantic features from a visual feature sequence and adaptively encode the contextual semantic features to obtain semantic features of the medical image. The fusion encoder can capture long-range dependencies and understand the relationship between lesions and surrounding tissues. Through adaptive encoding, more accurate semantic features can be obtained, which is conducive to improving the accuracy of the generated medical image report. A disease semantic graph with dynamic edge weights is constructed based on disease labels and semantic features to avoid the rigidity of fixed labels and distinguish similar diseases. The labeling probability corresponding to each disease label in the disease semantic graph is obtained based on Bayesian state embedding, and the uncertainty of the existence of the disease is expressed by probability distribution. Compared with binary classification labels, more fine-grained pathological information can be provided, which is conducive to improving the accuracy of the generated medical image report. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 This is an example of medical imaging in the background technology of this application;

[0045] Figure 2 This is a flowchart of a method for generating a medical imaging report in one embodiment of the present application;

[0046] Figure 3 This is an overall structural diagram of a medical imaging report generation model in one embodiment of the present application;

[0047] Figure 4aThis is a schematic diagram of a traditional hidden state calculation method in one of the embodiments of this application;

[0048] Figure 4b This is a schematic diagram of hidden state calculation of an enhanced spatial perception module in one embodiment of the present application;

[0049] Figure 5 This is a schematic diagram of the structure of a cross-context enhancement layer in one embodiment of the present application;

[0050] Figure 6 A comparison chart of reports generated by the traditional method and the medical imaging report generation method provided by the present invention in the comparative embodiment of this application;

[0051] Figure 7 This is a structural diagram of a terminal device in one of the embodiments of the present application. DETAILED DESCRIPTION

[0052] In response to the problem of low accuracy of traditional medical image report generation methods, the present invention provides a medical image report generation method and related equipment; wherein, the method discloses a medical image report generation model, which uses a fusion encoder for extracting contextual semantic features from a visual feature sequence and adaptively encoding the contextual semantic features to obtain semantic features of medical images. The model can capture long-range dependencies and understand the relationship between lesions and surrounding tissues. Through adaptive encoding, more accurate semantic features can be obtained, which is conducive to improving the accuracy of the generated medical image reports; constructing a disease semantic graph with dynamic edge weights based on disease labels and semantic features can avoid the rigidity of fixed labels and distinguish similar diseases; based on Bayesian state embedding, the labeling probability corresponding to each disease label in the disease semantic graph is obtained, and the uncertainty of the existence of the disease is represented by probability distribution. Compared with binary classification labels, more fine-grained pathological information can be provided, which is conducive to improving the accuracy of the generated medical image reports.

[0053] The following describes the method for generating a medical imaging report provided by the present invention.

[0054] like Figure 2 As shown, the medical imaging report generation method includes the following steps:

[0055] Step 21: Acquire medical imaging data.

[0056] In an embodiment of the present invention, medical imaging data includes medical images and image description text. Medical images are images showing the internal structure of the human body. In a feasible embodiment, the medical images are radiological images (such as X-ray images, CT images, and MRI images), which can be obtained from a general medical imaging dataset, such as MIMIC-CXR-JPG (derived from MIT and Beth Israel Deaconess Medical Center, which stores over 370,000 chest X-ray images). Image description text is used to describe the patient clinical data associated with the medical images. Typically, a medical imaging dataset contains both medical images and image description text.

[0057] In step 22 , the medical image data is input into a pre-trained medical image report generation model, and the medical image report generation model outputs a medical image report corresponding to the medical image data.

[0058] In the embodiment of the present invention, the medical image report generation model includes a visual feature extractor, a fusion encoder, a topic classifier, and a fusion decoder. The overall structure of the medical image report generation model is shown in FIG. Figure 3 shown.

[0059] The functions of each module are described as follows:

[0060] The visual feature extractor is used to extract the visual features of medical images and obtain a visual feature sequence.

[0061] The fusion encoder is used to extract contextual semantic features from the visual feature sequence and adaptively encode the contextual semantic features to obtain the semantic features of the medical image.

[0062] The topic classifier is used to extract disease labels from image description text, construct a disease semantic graph with dynamic edge weights based on disease labels and semantic features, obtain the labeling probability corresponding to each disease label in the disease semantic graph based on Bayesian state embedding, and obtain the disease feature vector corresponding to the medical image data based on the labeling probability.

[0063] The fusion decoder is used to generate medical imaging reports based on semantic features and disease feature vectors.

[0064] The functional content of each module is described in detail below.

[0065] As for the visual feature extractor, it is difficult to capture important information due to the small size and high noise of lesions in medical images. Therefore, in the embodiment of the present invention, a pre-trained CNN network is used to extract the visual features of medical images. In a feasible embodiment, the last average pooling layer and linear layer of DenseNet-121 (a deep convolutional neural network (CNN) architecture, belonging to the DenseNet series) can be removed and used as a visual feature extractor, and the convolution part can be used to directly extract the visual features of the medical image. This process can be expressed as f v (I)=V={v1,...,v n ,...,v N}; where f v (·) represents the visual feature extractor, v n Represents the nth visual feature, n=1,...,N, N represents the total number of visual features, N=h×w, h represents the height of the image feature, w represents the width of the image feature, V represents the visual feature set, c represents the number of channels of image features.

[0066] The following describes a process in which a fusion encoder extracts contextual semantic features from a visual feature sequence and adaptively encodes the contextual semantic features to obtain semantic features of a medical image, specifically including steps A to C.

[0067] In step A, the Mamba encoder is used to extract contextual semantic features.

[0068] It should be noted that to extract contextual semantic information, the Mamba architecture is used as the image encoder in this embodiment of the present invention. Unlike the quadratic computational complexity of traditional Transformer models, the Mamba architecture has linear complexity when processing sequential data. This significantly reduces computing resource requirements while accelerating model training and inference.

[0069] The core component of the Mamba model is the state-space model (SSM). The state-space model is a mathematical model used to describe the dynamic behavior of a system. It uses the current hidden state and input information to predict the state at the next moment and the output at the current moment. It combines the advantages of recurrent neural networks and convolutional neural networks, and can both extract long-distance dependencies and perform parallel operations using convolution. The selective state-space model is an improved version of the state-space model, enabling the model to dynamically focus on important inputs, thereby reducing the amount of computation and improving generalization capabilities. This mechanism allows the model to automatically identify and focus on key information during the encoding process, thereby improving its ability to capture complex features in medical data.

[0070] Specifically, through the potential state Map the visual feature sequence x(t) to the output response y(t), where d represents the size of the state dimension and L is the length of the visual feature sequence. This process can be expressed by a linear ordinary differential equation:

[0071]

[0072] Where t is the current input time; h′(t) represents the hidden state of the current input x(t); h(t) represents the hidden state of the previous time; is the evolution parameter; and is the projection parameter; is the dynamic parameter used for residual operation.

[0073] However, since SSM is a continuous-time model, it is difficult to directly use it for deep learning. To solve this problem, the data needs to be discretized first. Among them, the most commonly used method is zero-order hold (ZOH), which converts continuous parameters (A, B) into discrete parameters. The formula is:

[0074]

[0075] Where I represents the identity matrix and Δ is the discretization step size. The discretized SSM equation is:

[0076]

[0077] Finally, the model calculates the output M through global convolution. The calculation process is as follows:

[0078]

[0079] Where, is the Hadamard product operation.

[0080] Step B: Calculate the score of each of the pre-configured multiple candidate expert networks based on the contextual semantic features, and determine the final expert network for semantic feature extraction from the multiple candidate expert networks based on the score.

[0081] To improve the model's ability to understand complex medical image features, the present invention introduces a Mixture of Experts (MoE) system based on the existing Mamba encoder, ensuring computational efficiency without significantly increasing the number of model parameters. In this embodiment of the present invention, the candidate expert network is a trainable feedforward neural network used to extract semantic features of medical images. The model parameter values ​​of different candidate expert networks are not exactly the same, but each candidate expert network has the same parameter type.

[0082] In a feasible embodiment, step B includes steps B.1 to B.3.

[0083] Step B.1, calculate the score of each candidate expert network.

[0084] In a feasible implementation, for the input context semantic feature x, N is pre-configured experts candidate expert networks, each of which has a corresponding trainable weight matrix W v , calculated by formula h i (x) = W i x obtains the scores of each candidate expert network h i (x), where h i (x) represents the score of the i-th candidate expert network for the contextual semantic feature x, W i is a trainable parameter used to map input features into the score space.

[0085] In step B.2, based on the scores, the selection probability distribution of multiple candidate expert networks is calculated.

[0086] For example, by calculating the formula

[0087]

[0088] Get the selection probability P of the i-th candidate expert network i (x), h j (x) represents the score of the j-th candidate expert network for the contextual semantic feature x.

[0089] In step B.3, according to the selection probability distribution, the candidate expert network with the highest selection probability is determined as the final expert network.

[0090] In the traditional MoE method, the first K candidate expert networks with the highest selection probability are usually selected for processing. In the embodiment of the present invention, in order to further reduce the computational cost, referring to the selection strategy of Switch-MoE, only the candidate expert network with the highest probability is selected for forward calculation, that is: y = P I ·E I(x),I=argmax i P i (x).

[0091] Step C: Use the final expert network to obtain semantic features.

[0092] The following describes the process of extracting disease labels from image description text using a topic classifier, constructing a disease semantic graph with dynamic edge weights based on disease labels and semantic features, obtaining the labeling probability corresponding to each disease label in the disease semantic graph based on Bayesian state embedding, and obtaining the disease feature vector corresponding to the medical image data based on the labeling probability.

[0093] Disease labels are extracted from image description texts, and a disease semantic graph with dynamic edge weights is constructed based on disease labels and semantic features, including steps I to III.

[0094] Step I: Use a label automatic generation tool to extract labels from the image description text to obtain multiple disease labels corresponding to the image description text.

[0095] In a feasible embodiment, for radiological images, the radiological label automatic generation tool CheXbert can be used to analyze the image description text to obtain 14 chest disease labels corresponding to the image description text, such as "pneumonia" and "edema".

[0096] In another feasible embodiment, the frequency of each word in all image description texts can be counted, and the top 100 symptom-related words with the highest frequency of occurrence are used as supplementary labels. Then, the number of times these 114 labels appear in each sentence of the report is counted to obtain an N×114 matrix, where N is the total number of sentences. This matrix is ​​then transposed to obtain a 144×N label vector matrix, and let Represents the co-occurrence vector of label u in all sentences, whose kth element L uk Indicates the number of times label u appears in the kth sentence.

[0097] In step II, the similarity between disease labels is calculated based on the number of occurrences of each disease label.

[0098] Specifically, the similarity calculation formula is in,

[0099] In step III, a disease semantic graph is constructed based on the graph attention mechanism and multiple disease labels.

[0100] The above disease semantic graph includes multiple nodes, and the multiple nodes correspond one-to-one to multiple disease labels. The weight of the edge connecting two nodes is calculated by the similarity between the two nodes.

[0101] In this embodiment of the present invention, the relationship between diseases and symptoms is learned through the Graph Attention Network (GAT), and the process can be expressed as F g =GAT(M topic GAT is a graph neural network based on the attention mechanism. Its core idea is to perform weighted aggregation of neighbor nodes according to the edge weights in the adjacency matrix, so that each node can dynamically focus on the neighboring nodes that are most relevant to it, thereby more effectively capturing the semantic dependencies in the graph structure.

[0102] Different from the static edge weights based on statistical similarity in traditional graph neural networks, this paper introduces a trainable adjacency weight w u,v , which represents the correlation between two different disease labels u and v, and the weight w u,v It will be learned during the training process. The traditional GAT calculates an attention score e based on node features. u,v , the present invention modifies it and changes the weight w u,v Integrate it into the calculation of GAT's attention coefficient so that it takes into account both feature similarity and learned edge weight information:

[0103] e u,v ′=LeakyReLU(a T [Wh u ||Wh v ]+w u,v )

[0104] Among them, h u ,h v are the vector representations of nodes u and v in the graph structure, W represents the learnable matrix, a represents the parameter vector of the attention mechanism, || represents the concatenation operation of the tensor, LeakyReLU represents the activation function, and w u,v Represents a trainable parameter that controls the raw attention strength.

[0105] Then combine e u,v ' to calculate the final normalized attention coefficient a u,v ′, and use the characteristics of its aggregated neighbor nodes:

[0106]

[0107] in, N uRepresents the set of neighbor nodes of node u. Through this process, the weight value of each edge can be dynamically updated during the training process. u,v It reflects the potential relationship between diseases and symptoms. This approach enables the disease semantic graph to be adaptively optimized during continuous learning. The structure of its edges is no longer preset, but has context-awareness and can capture the implicit semantic associations between more complex disease topics.

[0108] The following describes a process in which a topic classifier in an embodiment of the present invention obtains the labeling probability corresponding to each disease label in a disease semantic graph based on Bayesian state embedding.

[0109] Since traditional state embedding methods usually directly map the predicted results of disease labels (such as 0 or 1) to a fixed embedding vector, ignoring the uncertainty in the classification process, especially when the predicted value is close to the threshold, it is difficult for the model to effectively determine whether the disease state exists. Therefore, the present invention introduces the Bayesian State Embedding mechanism. For each disease state, the present invention no longer learns a single, fixed embedding vector, but instead predicts a probability distribution through a neural network and samples it to obtain the state embedding.

[0110] Specifically, the present invention assumes that the embedding of each disease state follows a Gaussian distribution, which is composed of its mean vector μ and variance vector σ 2 Here, the mean vector μ represents the most expected value of the disease state embedding. The variance vector σ 2 This represents the range of variation in the embedding of the disease state. A larger variance indicates a more uncertain embedding of the state and a wider range of possible values. During calculation, the present invention cannot directly use a distribution as input; instead, it samples from this learned distribution to obtain a specific embedding vector. To enable this sampling process to be trained via backpropagation, the present invention employs a reparameterization technique. This method enables the state embedding to reflect not only the presence of a disease but also the probability of its occurrence.

[0111] Specifically, by calculating the formula

[0112] M state =μ+∈·exp(0.5·log σ 2 ),∈~N(0,1)

[0113] Get the labeling probability m state ; Among them, μ, σ are the Gaussian distribution mean and Gaussian distribution variance corresponding to each disease label, respectively, and ∈ represents a learnable parameter, which is used to enable the model to be optimized according to different label distributions, thereby enhancing the generalization ability and robustness of the system.

[0114] The classification part uses the cross entropy loss function to update the model parameters. In one embodiment of the present invention, the cross entropy loss function expression is: Among them, y uv ∈{0,1},α disease,uv ∈{0,1} represents the vth true label and predicted value of the uth semantic label respectively.

[0115] The following describes a process in which a topic classifier acquires a disease feature vector corresponding to medical image data according to the labeling probability in an embodiment of the present invention, specifically including steps ① to ③.

[0116] Step ①, based on the cross-attention mechanism and semantic features, determine the correspondence between each disease label and the image area of ​​the medical image, and determine the most relevant image representation of each disease label based on the correspondence.

[0117] In the actual workflow of a hospital, radiologists usually find corresponding visual evidence for each common radiological examination item, such as the lesion area in the chest X-ray image, and then make a diagnosis based on these clinical symptoms. In order to simulate this process, in the embodiment of the present invention, after extracting the disease information Finally, the cross-attention mechanism (CAM) is used to extract the radiological region of interest of each node in the disease semantic graph, thereby retrieving the image region most relevant to each disease symptom information. This process can be expressed as F′ g =CAM(M,F g ), where M represents the features obtained by the image encoder, F g ′ represents the comprehensive representation of the most relevant image features for each disease label.

[0118] Step ②, calculate the disease state matrix based on the cross-attention mechanism, labeling probability and correspondence.

[0119] Specifically,

[0120] CAM(X,Y)=αYW V

[0121]

[0122] In the attention formula, CAM(X,Y) is the formula description of the attention mechanism, where W Q 、W K 、W VRepresent the linear weight matrices used for trainable query, key, and value transformations, respectively, to map input features to the attention space. α represents the attention weight coefficient calculated by the softmax function, which is used to measure the correlation between different feature positions.

[0123] F′ g =CAM(M,F g )

[0124] In the second formula, M represents the features obtained by the image encoder. g Represents disease information, which is obtained by the following steps: Define the matrix n randomly initialized vectors The dataset is stacked to represent different disease-related topics, including disease labels and high-frequency words or phrases extracted from medical reports. It is vectorized using topic embedding, and then the relationship between diseases and symptoms is learned through the Graph Attention Network (GAT). This process can be expressed as F g =CAM(M topic ). F′ g Indicates that the cross-attention mechanism (CAM) is used to extract the radiological region of interest of each node in the disease semantic graph, thereby retrieving the image region most relevant to each disease symptom information; F g ′ can be understood as the most relevant image feature for each disease label.

[0125] M′ state =CAM(F′ g ,M state )=α disease M state W V

[0126] M state is the probability information of the disease extracted previously, α disease It can be understood as the weight based on the probability of different diseases, M′ state Represents the disease state matrix, which integrates the image information and disease information selected by the attention mechanism, so as to better model the state characteristics of the disease.

[0127] The fusion decoder is described below.

[0128] Specifically, the present invention designs a fusion decoder structure comprising five modules. These modules include an Enhance Spatial Perception Module (ESPM), a self-attention mechanism, a cross-attention mechanism (receiving output from the encoder), a Cross-Context Enhancement Layer (CCEL), and a cross-attention module for integrating graph structure information. Among them, the ESPM module is used to improve the spatial perception ability of the image area. The CCEL module comprehensively considers the interaction between visual features and global context information to generate a more refined feature representation.

[0129] The functions of each module are described below.

[0130] Aimed at enhancing spatial perception modules.

[0131] In traditional Transformer decoders, the hidden state is usually used as a query vector to generate attention features from input features. Figure 4a As shown in Figure 2, existing Transformer-based methods for automatic radiology report generation mostly use decoders to calculate the current hidden state by modeling the relationships between generated words. This approach relies on a masked multi-head self-attention mechanism, which ensures that the generated word depends only on previously generated words. However, during the calculation of the hidden state, the influence of the global features of the entire image is ignored, failing to effectively capture the global context of the image.

[0132] In order to solve this problem, the present invention proposes an enhanced spatial perception module, such as Figure 4b As shown in Figure 2, ESPM is placed before the self-attention mechanism to provide global visual context features for the subsequent self-attention mechanism. When generating hidden states, this module can explicitly consider the global features of the entire image, thereby enhancing the decoder's ability to perceive spatial features.

[0133] Specifically, in order to better utilize global context information in the decoding process, a global feature extractor is introduced to generate global features. First, the global feature is calculated from the intermediate state M output by the visual encoder. It is the result of taking the average of all grid features along the spatial dimension:

[0134]

[0135] Then, in order to align with the dimension of the output feature x of the text encoder, Expand and concatenate with x to obtain the enhanced feature representation:

[0136]

[0137] Then, based on the enhanced feature representation, the ESPM module performs nonlinear transformation on the features, further processes the fused features, and generates a new representation:

[0138] x′ v =ReLU(W1x v +b1)

[0139] Among them, W1 and b1 are the learnable weights and biases used to fuse global features with target word representation.

[0140] Finally, the output x′ of the ESPM module v It is passed to the masked multi-head self-attention mechanism to generate the current hidden state H, which is calculated as follows:

[0141] H=MHA(x′ v )

[0142] MHA stands for Multi-Head Attention. In this way, the enhanced spatial perception module can explicitly introduce global visual context information into the hidden state calculation process, thereby effectively improving the decoder's ability to comprehensively model global and local features.

[0143] For cross-context enhancement layers.

[0144] In the automatic radiology report generation system based on the encoder-decoder architecture, the cross-attention mechanism allows the model to establish cross-modal interactions between the encoded image grid features and the previously generated text words. Specifically, the cross-attention mechanism determines which image regions are most relevant to the currently generated words in the text by calculating the correlation between the image features and the current text generation sequence. Its output is usually used as the prediction basis for the next word to be generated. However, when directly applied to the task of automatic radiology report generation, existing methods have several limitations: (1) Imprecise feature selection: During the calculation process, the cross-attention mechanism always returns a weighted average vector as the result, regardless of whether the candidate features actually meet the requirements of the query vector. For example, in the report generation task, assume that the query vector is the currently generated word and the candidate features are different regions in the image. Even if some image regions have little relationship with the current word, the cross-attention mechanism may still assign non-zero weights to these regions and include them in the output. In this case, even if the weights of irrelevant regions are small, the accumulation may cause the generated words to not fully match the image content, thereby affecting the quality of report generation. (2) Ignoring global information: The traditional cross-attention mechanism only establishes interactions between the encoded grid features and previously generated words, while ignoring global information at the entire image level. In medical image analysis, global information (such as the overall distribution or location of lesions) is often important for diagnostic conclusions, so ignoring this information may weaken the quality and reliability of the generated report.

[0145] To address this issue, in an embodiment of the present invention, after performing cross-attention, a cross-context enhancement layer is constructed to capture the global information of interest. In this way, the model can better incorporate the overall structure of the image, thereby improving the accuracy and relevance of the generated report.

[0146] Specifically, such as Figure 5 As shown in Figure 1, the cross-context enhancement layer consists of two components: the Refined Unit and the Fusion Unit. The Refined Unit is responsible for removing redundant information from the original input and extracting more representative features to improve feature quality; the Fusion Unit combines these high-quality features with global information to provide richer visual context information for subsequent processing. The Refined Unit is defined as follows:

[0147]

[0148] in, b. is a learnable weight, σ refers to the Sigmoid activation function, ⊙ represents element-wise multiplication, and q is the query vector.

[0149] Then, given the output C of the cross-attention module and the global image features The hidden state obtained by the decoder is used as the query vector to generate refined attention features and global features in the following way:

[0150] C′=f r (H,C)

[0151]

[0152] H=Self-Attention(ESPM(M),y)

[0153] Based on the output of the refinement unit, a fusion unit is further constructed to integrate the newly obtained global features and attention feature C′. The fusion unit helps the model obtain more comprehensive visual information by calculating the importance weights of different features in the input image. To this end, a scoring function is designed to evaluate the attention weights of local features and global features, which is defined as follows:

[0154]

[0155] Where W s 、b s is a learnable weight. Then, according to the output of the scoring function, the attention weight α of the local features and the global features and the final context feature C are calculated. t :

[0156]

[0157] Among them, α∈[0,1] represents the local feature C′ and the global feature Specifically, if the local feature C′ is more important in the current task, the value of α will be biased towards 1, and the model will tend to rely more on local features; on the contrary, if the global feature If the local and global features are more important, α will be biased towards 0, making the model pay more attention to global information. In this way, the model can dynamically adjust the importance of local features and global features according to different contexts, thereby better capturing visual information.

[0158] In order to verify the effectiveness of the medical imaging report generation method provided by the present invention, in another embodiment of the invention, the medical imaging report generation method provided by the present invention is compared with the traditional method, as follows:

[0159] In the comparative example, two public medical chest image datasets were used, namely IU-Xray and MIMIC-CXR. The IU-Xray dataset was released by Indiana University and contains a total of 7,470 images and 3,955 corresponding diagnostic reports, where two chest radiographs of each patient correspond to one report. The MIMIC-CXR dataset is currently the largest public radiology dataset, containing 473,057 chest X-ray images and 206,563 reports of 63,478 patients, where one report corresponds to one or more X-ray images, generally a front view and a side view of the chest. For IU-Xray, the present invention uses a ratio of 7:1:2 to divide it into a training set, a validation set, and a test set. For the MIMIC-CXR dataset, the official division standard is followed to facilitate comparison with existing work. The specific division method of the dataset is shown in Table 1.

[0160] Table 1

[0161]

[0162] In order to objectively and comprehensively evaluate the text generation performance of the model and compare it with other models of the same task, the present invention adopts natural language generation (NLG) indicators, including BLEU-n, METEOR, ROUGE-L and CIDEr. Among them, BLEU-n is an indicator commonly used in machine translation tasks, which is used to measure the similarity between two sentences, where n is the number of consecutive words. BLEU-1 measures the accuracy at the word level, while the higher-order BLEU-n can evaluate the fluency of the generated sentences; METEOR's evaluation is based on the entire corpus. This method pays attention to the synonyms of candidate words in the corpus during the evaluation process, which is more consistent with the results of manual judgment than BLEU; ROUGE-L calculates the overlap rate of the longest common subsequence between two sentences, which can reflect the accuracy at the sentence level and is often used to evaluate the quality of machine translation tasks and the level of image subtitle generation tasks. The larger the value of the above indicators, the better the model generation effect.

[0163] For clinical efficacy metrics (CE metrics), CheXbert was used to annotate the reports generated by the model, obtaining labels for 14 categories related to chest diseases, and the results were compared with the true labels. The annotation results include: 1 (definitely present), 0 (definitely not present), -1 (uncertain whether it exists), and blank (not mentioned). Based on previous studies, 1 is considered a positive sample and the remaining results are considered negative samples. Afterwards, the labels extracted from the predicted reports and the true reports are compared to calculate the precision, recall, and F1-score of these 14 categories. Since CheXbert is not applicable to the IU-Xray dataset, its clinical accuracy was not evaluated.

[0164] In terms of parameter settings for the comparative example, a single image from the IU-Xray and MIMIC-CXR datasets was used as input. The original image size was 256×256 pixels, which was then reduced to 224×224 pixels via center cropping. Image features were then extracted using a pre-trained DenseNet-121. Furthermore, for the 114 extracted label features, the top k nodes with the highest similarity were selected as neighbors to construct a similarity graph, with k set to 5. The image encoder's hidden layer dimension was set to 256, the SSM state expansion factor was set to 16, and the local convolution kernel width and block expansion factor were set to 4 and 2, respectively. The decoder employed a multi-head attention mechanism with 8 attention heads and a hidden dimension of 512. The model was trained using the AdamW optimizer with a cross-entropy loss function and a weight decay of 5e-5. The initial learning rate was set to 1e-4. The MultiStepLR learning rate scheduler was used, with the learning rate decaying to one-tenth of the previous value every 25 epochs. The batch size was set to 32, and the search size for report generation was set to 3. The number of training rounds on the IU-Xray and MIMIC-CXR datasets was set to 100 and 30, respectively. The training device selected was an NVIDIA RTX 4060Ti graphics card with 16GB of video memory.

[0165] For IU-Xray, the comparative implementation results of different methods are shown in Table 2.

[0166] Table 2

[0167]

[0168]

[0169] Among them, the bold numbers represent the maximum values ​​of the data set, and the underlined parts represent the suboptimal values. As can be seen from Table 2, in the IU-Xray dataset, the medical imaging report generation method provided by the present invention achieved the best results in BLEU-2, BLEU-3 and METEOR indicators, which were 4.00%, 1.26% and 2.86% higher than the other optimal methods in the table, respectively, and achieved suboptimal results in BLEU-1 and BLEU-4 indicators. In the MIMIC-CXR dataset, the medical imaging report generation method provided by the present invention achieved the best results in BLEU-1 to BLEU-4 indicators, among which it was 2.85% higher than the other optimal methods in the table in BLEU-1 and 2.37% higher in BLEU-3, and achieved suboptimal results in the METEOR indicator. Among these compared methods, the traditional image captioning methods ATT2IN and ADAATT failed to achieve better results when directly applied to the field of medical imaging report generation, which shows that it is necessary and important to conduct field-specific radiology report generation research. The R2Gen and R2GenCMN models that use memory networks only consider using memory records to generate information. Although they have achieved good results, they have not solved the problem of data bias. Compared with the model HDGAN that introduces reinforcement learning and the large model CvT212DistilGPT2 that uses additional data, the medical imaging report generation model provided by the present invention has a simpler structure and achieves better results. Experimental results show that the medical imaging report generation method provided by the present invention performs better than most comparison methods in terms of NLG indicators, which indicates that the medical imaging report generation model provided by the present invention can better generate text descriptions corresponding to medical images, and the generated reports have certain improvements in word order logic and semantic matching, and their expression is more in line with the style of real reports.

[0170] The medical imaging report generation model provided by the present invention was further compared with other methods in terms of the CE index, and the comparison results are shown in Table 3. Among them, the CE index, as a clinical evaluation indicator, can directly reflect the clinical accuracy of the report generated by the model and is considered to be more clinically meaningful than traditional word overlap-based evaluation indicators (such as BLEU and ROUGE).

[0171] Table 3

[0172] Model Precision Recall F1-Score ATT2IN(2017) 0.322 0.239 0.249 ADAATT (2017) 0.268 0.186 0.181 R2Gen (2020) 0.333 0.273 0.276 R2GenCMN(2021) 0.334 0.275 0.278 RAMT(2023) 0.380 0.342 0.335 METransformer (2023) 0.364 0.309 0.311 CGFTrans(2024) 0.367 0.303 0.308 The present invention 0.394 0.355 0.348

[0173] As shown in Table 3, the medical imaging report generation model provided by this invention achieved the best results in terms of CE metrics, with a precision of 39.4%, a recall of 35.5%, and an F1-score of 34.8%, surpassing the model proposed by traditional methods by 3.68%, 3.80%, and 3.88%, respectively. These improvements demonstrate that this method can significantly improve the clinical accuracy of reports, thereby generating more clinically valuable medical reports.

[0174] To further demonstrate the effectiveness of the proposed module, ablation experiments were conducted on the MIMIC-CXR and IU-Xray datasets. The experimental results are shown in Tables 4 and 5. In the table, "base" represents the Transformer-based encoder-decoder model, and the remaining rows represent the metrics obtained by adding the corresponding modules to this model. Experiment (d) shows the results obtained using the medical imaging report generation model provided by this invention.

[0175] Table 4

[0176] Model GSTC Mamba Encoder ESPM CCEL BL-1 BL-2 BL-3 BL-4 ROUGE-L METEOR base 0.322 0.191 0.128 0.096 0.255 0.124 (a) √ 0.343 0.222 0.153 0.110 0.274 0.136 (b) √ √ 0.351 0.229 0.157 0.111 0.282 0.140 (c) √ √ √ 0.392 0.246 0.168 0.121 0.286 0.158 (d) √ √ √ √ 0.397 0.251 0.173 0.125 0.289 0.160

[0177] Table 5

[0178] Model GSTC Mamba Encoder ESPM CCEL BL-1 BL-2 BL-3 BL-4 ROUGE-L METEOR base 0.422 0.271 0.182 0.124 0.365 0.173 (a) √ 0.461 0.290 0.202 0.147 0.372 0.183 (b) √ √ 0.469 0.293 0.197 0.142 0.361 0.182 (c) √ √ √ 0.483 0.316 0.224 0.166 0.376 0.195 (d) √ √ √ √ 0.495 0.338 0.241 0.176 0.406 0.216

[0179] By comparing the results of the base and experiments (a) and (b) on the two datasets, we find that adding global visual context to the decoder of the Transformer model improves performance on most metrics. Traditional Transformer models rely solely on local feature interactions through a multi-head attention mechanism, lacking systematic integration of global image-level features. This leads to the model easily missing key diagnostic information when processing pathological areas with ambiguous visual features. Therefore, introducing global visual information into the decoder enables it to better integrate image features, providing the model with richer semantic information.

[0180] Experiment (c) adds a graph-structured topic classifier (GSTC) to the base model. It shows significant improvements across all evaluation metrics compared to the "base" results. This demonstrates that GSTC captures the underlying relationship between symptoms and pathological features, extracts disease-related classification information, and effectively integrates it into the decoder, effectively enhancing the model's understanding of medical images and the quality of report generation.

[0181] In the medical imaging report generation model provided by this invention, it can be seen that all indicators achieve optimal results. In addition, the model introduces a Mamba-structured encoder to replace the traditional Transformer encoder. This improvement significantly reduces the complexity of the model. The standard 6-layer Transformer encoder contains 2.375M parameters, while the Mamba encoder contains 0.438M parameters, a parameter reduction of 81.6%. This not only significantly reduces computational costs but also significantly improves the model's operational efficiency.

[0182] In order to further analyze the text generation of the model, Figure 6The results of a qualitative analysis of three imaging cases from the MIMIC-CXR dataset are presented. The colored areas in the figure represent text that appeared in the real reports and correctly appeared in the predicted reports. It can be seen that compared to the traditional Transformer network model R2Gen, the text generated by the medical imaging report generation model provided by this invention is closer to the reports written by radiologists.

[0183] In case (1), both reports correctly predicted that there was no pleural effusion and pneumothorax in the image. However, the R2Gen method incorrectly reported the presence of pneumonia in the left lower lobe. The medical imaging report generation method provided by the present invention accurately described that the heart size was within the normal range and that there was no consolidation, effusion or pneumothorax in the lungs, which was more consistent with the actual report. In case (2), R2Gen did not mention left lung base atelectasis and mild pulmonary edema, nor did it mention left diaphragm elevation. In contrast, the medical imaging report generation method provided by the present invention accurately described low lung volume, left diaphragm elevation, mild left lung base atelectasis, and mentioned that the fuzzy pulmonary vascular markings were consistent with mild pulmonary edema, and the description of the abnormal part was more accurate. In case (3), it can be seen that both reports accurately identified the pacemaker, pleural effusion and pulmonary edema that were clearly present in the image. Compared with the report generated by R2Gen, the medical imaging report generation model provided by the present invention also correctly predicted the enlargement of the heart. However, neither model predicted calcification of the aortic knob, primarily because the lesion typically appears as small calcifications on X-rays, which can be easily missed even by expert radiologists when image quality is poor or when the calcified area overlaps with mediastinal structures.

[0184] Experimental results show that compared to text generated by R2Gen, the reports generated by the medical imaging report generation model provided by this invention are closer to real reports, with complete and fluent sentences. The model provides a closer mapping relationship between organs and abnormalities, enabling more detailed and accurate descriptions of different disease characteristics. Furthermore, by learning text features for identically labeled text, the model generates text that is not identical to the target report. Instead, it uses alternative words and sentences with the same meaning, making the generated reports more flexible.

[0185] In summary, the present invention proposes a medical imaging report generation model, which aims to effectively enhance the utilization of disease symptom information and global image information, and improve the quality of report generation and clinical accuracy. By constructing a similarity graph between disease and symptom features and using a graph attention mechanism to guide the model to focus on the potential correlation between the disease and symptoms, the severity of the disease can be perceived. A multi-scale feature extraction scheme based on the encoder of the Mamba structure is adopted, which effectively makes up for the shortcomings of the traditional attention mechanism in capturing long-range dependencies and fuzzy pathological features. In the decoder part, by fusing global and local information layer by layer, the model can better understand subtle changes in the image, thereby solving the problem of traditional attention mechanisms ignoring global features and improving the accuracy and semantic consistency of the report text. Experiments on the MIMIC-CXR and IU-Xray datasets show that this method surpasses the existing state-of-the-art technology in terms of conventional language generation indicators and clinical accuracy.

[0186] like Figure 7 As shown, an embodiment of the present invention provides a terminal device, such as Figure 7 As shown, the terminal device D10 of this embodiment includes: at least one processor D100 ( Figure 7 Only one processor is shown in the figure), a memory D101, and a computer program D102 stored in the memory D101 and executable on the at least one processor D100, wherein the processor D100 implements the steps of any of the above method embodiments when executing the computer program D102.

[0187] Specifically, when the processor D100 executes the computer program D102, it obtains medical imaging data; inputs the medical imaging data into a pre-trained medical imaging report generation model, and the medical imaging report generation model outputs a medical imaging report corresponding to the medical imaging data. The disclosed medical imaging report generation model, through a fusion encoder for extracting contextual semantic features from a visual feature sequence and adaptively encoding the contextual semantic features to obtain semantic features of medical images, can capture long-range dependencies and understand the relationship between lesions and surrounding tissues. Through adaptive encoding, more accurate semantic features can be obtained, thereby facilitating the improvement of the accuracy of the generated medical imaging report. A disease semantic graph with dynamic edge weights constructed based on disease labels and semantic features can avoid the rigidity of fixed labels and distinguish similar diseases. The labeling probability corresponding to each disease label in the disease semantic graph is obtained based on Bayesian state embedding, and the uncertainty of the existence of the disease is represented by a probability distribution. Compared with binary classification labels, this model can provide more fine-grained pathological information, thereby facilitating the improvement of the accuracy of the generated medical imaging report.

[0188] The processor D100 may be a central processing unit (CPU), or may be another general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. A general-purpose processor may be a microprocessor or any conventional processor.

[0189] In some embodiments, the memory D101 may be an internal storage unit of the terminal device D10, such as a hard disk or memory of the terminal device D10. In other embodiments, the memory D101 may also be an external storage device of the terminal device D10, such as a plug-in hard disk, a smart memory card (SMC, SmartMedia Card), a secure digital (SD, Secure Digital) card, a flash card, etc. equipped on the terminal device D10. Furthermore, the memory D101 may also include both an internal storage unit of the terminal device D10 and an external storage device. The memory D101 is used to store an operating system, an application program, a boot loader (BootLoader), data, and other programs, such as the program code of the computer program. The memory D101 may also be used to temporarily store data that has been output or is to be output.

[0190] An embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the above-mentioned various method embodiments can be implemented.

[0191] An embodiment of the present application provides a computer program product. When the computer program product is run on a terminal device, the terminal device can implement the steps in the above-mentioned method embodiments when executing the computer program product.

[0192] Those skilled in the art should understand that the discussion of any of the above embodiments is merely illustrative and is not intended to imply that the scope of protection of the present application is limited to these examples. In line with the present application, the technical features in the above embodiments or different embodiments may be combined, the steps may be implemented in any order, and there are many other variations of different aspects of one or more embodiments of the present application as described above, which are not provided in detail for the sake of simplicity.

[0193] The one or more embodiments of this application are intended to encompass all such substitutions, modifications, and variations that fall within the broad scope of this application. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of one or more embodiments of this application should be included in the scope of protection of this application.

Claims

1. A method for generating a medical imaging report, characterized in that: include: Acquiring medical imaging data; the medical imaging data includes medical images and image description text; The medical image data is input into a pre-trained medical image report generation model, and the medical image report generation model outputs a medical image report corresponding to the medical image data; the medical image report generation model includes a visual feature extractor, a fusion encoder, a topic classifier, and a fusion decoder; the visual feature extractor is used to extract visual features of the medical image to obtain a visual feature sequence; the fusion encoder is used to extract contextual semantic features from the visual feature sequence, and adaptively encode the contextual semantic features to obtain semantic features of the medical image; The topic classifier is used to extract disease labels from the image description text, construct a disease semantic graph with dynamic edge weights based on the disease labels and the semantic features, obtain the labeling probability corresponding to each disease label in the disease semantic graph based on Bayesian state embedding, and obtain the disease feature vector corresponding to the medical image data based on the labeling probability; the fusion decoder is used to generate a medical imaging report based on the semantic features and the disease feature vector.

2. The method for generating a medical imaging report according to claim 1, wherein: The extracting of contextual semantic features from the visual feature sequence and adaptively encoding the contextual semantic features to obtain semantic features of the medical image includes: Extracting the contextual semantic features using a Mamba encoder; Calculating scores for each of a plurality of pre-configured candidate expert networks based on the contextual semantic features, and determining a final expert network for semantic feature extraction from the plurality of candidate expert networks based on the scores; wherein the candidate expert networks are trainable feedforward neural networks for extracting the semantic features of the medical image, and model parameter values ​​between different candidate expert networks are not completely the same; The semantic features are obtained using the final expert network.

3. The method for generating a medical imaging report according to claim 2, wherein: The step of calculating a score of each of a plurality of pre-configured candidate expert networks according to the contextual semantic features, and determining a final expert network for semantic feature extraction from the plurality of candidate expert networks according to the scores, comprises: Calculate the score of each candidate expert network; for the input context semantic feature x, pre-configure N experts candidate expert networks, each of which has a corresponding trainable weight matrix W i , calculated by formula h i (x) = W i x obtains the scores of each candidate expert network h i (x), where h i (x) represents the score of the i-th candidate expert network for the contextual semantic feature x, W i is a trainable parameter used to map input features into the score space; According to the scores, the selection probability distribution of the plurality of candidate expert networks is calculated; based on the scores h of each candidate expert network i (x), calculate the probability distribution p through the softmax function i (x), the formula is According to the selection probability distribution, the candidate expert network with the greatest selection probability is determined as the final expert network.

4. The method for generating a medical imaging report according to claim 3, wherein: The step of extracting disease labels from the image description text and constructing a disease semantic graph with dynamic edge weights based on the disease labels and the semantic features includes: Using an automatic label generation tool to extract labels from the image description text to obtain multiple disease labels corresponding to the image description text; Calculating the similarity between the disease labels according to the number of occurrences of each disease label; A disease semantic graph is constructed based on the graph attention mechanism and the multiple disease labels; the disease semantic graph includes multiple nodes, the multiple nodes correspond one-to-one to the multiple disease labels, and the weight of the edge connecting two nodes is calculated by the similarity between the two nodes.

5. The method for generating a medical imaging report according to claim 4, wherein: The method of obtaining the labeling probability corresponding to each disease label in the disease semantic graph based on Bayesian state embedding includes: By calculating the formula M state =μ+∈·exp(0.5·logσ 2 ),∈~N(0,1) Get the labeling probability M state ; Among them, μ, σ are the Gaussian distribution mean and Gaussian distribution variance corresponding to each disease label, and ∈ represents the learnable parameters.

6. The method for generating a medical imaging report according to claim 5, wherein: The obtaining of the disease feature vector corresponding to the medical image data according to the labeling probability includes: Determining a correspondence between each disease label and an image region of the medical image based on a cross-attention mechanism and the semantic features, and determining a most relevant image representation for each disease label according to the correspondence; Calculating a disease state matrix based on a cross-attention mechanism, the labeling probability, and the corresponding relationship; The disease feature vector is calculated according to the disease state matrix and the labeled probability.

7. The method for generating a medical imaging report according to claim 6, wherein: The expression of the disease state matrix is: M′ state =CAM(F′ g ,M state )=α disease M state W V F′ g =CAM(M,F g ) CAM(X,Y)=αYW V Among them, CAM(X,Y) is the formula description of the attention mechanism, W Q 、W K 、W V Represents the linear weight matrices used for trainable query, key, and value transformations, respectively, to map input features to the attention space. α represents the attention weight coefficient calculated by the softmax function, which is used to measure the correlation between different feature positions. M represents the features obtained by the image encoder. F g Indicates disease information, F′ g It means that the radiological region of interest of each node in the disease semantic graph is extracted through the cross attention mechanism, thereby retrieving the image region most relevant to each disease symptom information, which can be understood as the most relevant image feature for each disease label. state is the probability information of the disease extracted previously, α disease Represents the probability of different diseases, M′ state represents the disease state matrix.

8. The method for generating a medical imaging report according to claim 7, wherein: The calculating the disease feature vector according to the disease state matrix and the labeled probability includes: By calculating the formula Z out =M′ state +y′M state Get the disease feature vector Z out ; Among them, y′ represents the labeling probability, y′=α disease , τ represents the preset threshold value, which is used to determine the disease status.

9. A terminal device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the method according to any one of claims 1 to 8 is implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images