Medical report generation method based on visual manifold graph and generative adversarial network
By constructing a visual manifold graph and a generative adversarial network, and combining zero-center Wasserstein distance and gradient penalty terms, the problems of negative description and language style in medical report generation are solved, achieving efficient and accurate medical report generation.
Patent Information
- Application Number
- CN202510954300.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-07-11
- Publication Date
- 2025-10-31
AI Technical Summary
Existing methods for generating medical reports are inadequate in generating correct negative descriptions and medical language styles. Furthermore, GAN-based methods are difficult to converge, the use of natural image metrics is unreasonable, and they ignore the unique language style of medical reports.
We construct a visual manifold graph to capture the relationship between visual objects and text predicates, design zero-center Wasserstein distance and language style loss, generate accurate medical reports through generative adversarial networks, and use Faster R-CNN, graph parser, graph convolutional network and LSTM network for feature extraction and generation. We introduce zero-center Wasserstein distance and gradient penalty terms to ensure training convergence.
It enables the generation of accurate medical reports that conform to clinical expression habits, avoids gradient explosion and pattern collapse, and improves the authenticity and professionalism of the reports.
Smart Images
Figure CN120878023A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of image processing, and in particular relates to a method for generating medical reports based on visual manifold graphs and generative adversarial networks. Background Technology
[0002] Medical report generation is a current hot research area, aiming to automatically and accurately analyze radiological images using technologies such as image processing, machine learning, and deep learning, thereby achieving rapid and reliable computer-aided diagnosis. Therefore, it can help radiologists speed up the decision-making process, improve diagnostic efficiency, and reduce the time and financial burden on patients. Existing medical report generation methods mainly include three approaches: encoder-decoder architecture-based, Transformer-based, and GAN-based.
[0003] Despite the initial progress achieved by the aforementioned methods, several problems remain. First, almost all methods are based on non-standard language generation, ignoring proper negation descriptions. This means that existing methods produce unsatisfactory generation when using the commonly used <subject-verb-object> triplet. Second, it can be shown that all GAN-based medical report generation or potential natural title generation methods are non-convergent regardless of whether they use alternating or simultaneous training, and that each generator update followed by a finite discriminator update fails to converge to local equilibrium. Furthermore, medical reports have a unique linguistic style, making it unreasonable to use natural image metrics to measure the generated reports, as in previous works. Summary of the Invention
[0004] In light of this, we propose a medical report generation method based on visual manifold graphs and generative adversarial networks (GANs) to achieve efficient medical report generation. We first construct a visual manifold graph to capture the latent relationships between visual objects and textual predicates, generating reports with correct negation. Then, we design a convergent zero-center Wasserstein distance to achieve convergence, ensuring it accurately converges to local equilibrium during alternating or simultaneous training. Furthermore, we embed a linguistic style loss to balance fidelity and clinical naturalness (similar to a clinician's style) to ensure a clinical language style.
[0005] To achieve the above objectives, the present invention is implemented through the following technical solution:
[0006] A medical report generation method based on visual manifold graphs and generative adversarial networks includes the following steps:
[0007] S1: Configure the Faster R-CNN network and graph resolver. The Faster R-CNN network extracts visual features from medical images, and the graph resolver extracts semantic information from diagnostic report data. Visual features are used as object nodes, and semantic information is used as predicate nodes to construct a visual manifold graph.
[0008] S2: Configure a graph convolutional network to enhance the processing of the visual popularity graph;
[0009] S3: Use a two-layer LSTM and a joint attention module to build a report generator to generate reports;
[0010] S4: Design a discriminator network to guide the report generator to improve the authenticity and professional expression quality of medical reports through image-text matching judgment, style consistency learning, and comparison of diverse samples;
[0011] S5: Collect batch-paired medical image-diagnostic report data to train the system model constructed according to steps S1-S4. Input the medical images into the trained system model to generate medical diagnostic reports that are both accurate and in line with clinical expression habits.
[0012] The entire training process employs an adversarial learning mechanism. The generator strives to produce high-quality reports that can deceive the discriminator, while the discriminator continuously improves its recognition capabilities. By introducing zero-center Wasserstein distance and gradient penalty terms, the convergence of training is ensured, ultimately enabling the model to generate medical diagnostic reports that are both accurate and consistent with clinical expression habits.
[0013] Optionally, step S1 specifically includes:
[0014] S11: For a given medical image I, a pre-trained Fast R-CNN network is used to identify object instances in the medical image, thereby obtaining a set of regions V = {v1, v2, ..., v...} corresponding to each object in the image. N};
[0015] S12: An attribute classifier consisting of a small fully connected layer fc-ReLU-fc-Sofemax is used to classify the relationships in these regions;
[0016] S13: Visual features extracted by the Fast R-CNN network Embedded with the identified tags The fusion process is performed to obtain the final object node features u. oi ,Right now:
[0017] Where W1 and W2 are trainable parameters used when fusing features, o i Represented as the i-th object node;
[0018] S14: Using r ij Indicates the connection object node o i and o j The predicate nodes; the node set V includes object nodes and predicate nodes, and the edge set E represents the connection relationship between object nodes and predicate nodes, thus obtaining the visual manifold graph G = (V, E).
[0019] Optionally, step S2 specifically includes:
[0020] S21: Based on the visual manifold G = (V, E) constructed in step S1, perform cross-modal feature integration operation on each node;
[0021] For object node o i Visual features extracted from the Faster R-CNN network The word embedding text features corresponding to this object Perform feature concatenation operations and pass them through a feedforward network φ with ReLU activation function. o Perform nonlinear transformation to obtain fused features
[0022] For the predicate node r ij Embed its words into text features Through feedforward network φ r Perform feature transformation to obtain a feature representation with uniform dimension.
[0023] S22: Perform graph convolution enhancement processing on the fusion features obtained in step S21;
[0024] For object node o i The neighborhood information is aggregated using graph convolution operations:
[0025]
[0026] Among them, f in f out This represents a feedforward neural network; These represent object nodes o respectively i N is the set of adjacent nodes of the endpoint and the starting point; i Represents object node o i The total number of adjacent nodes;
[0027] For the predicate node r ij Enhancement is achieved by fusing the features of the two connected object nodes:
[0028]
[0029] in These are the fusion features of the connected object nodes, f r It is a feedforward network;
[0030] S23: Assign the enhanced features to different network modules according to functional requirements: assign object node features Predicate node features Along with a complete enhanced manifold structure input generator, it performs attention-guided visual encoding, semantic generation of medical terminology, and ensures logical consistency of diagnostic reports. Simultaneously, it inputs a multimodal representation composed of enhanced visual manifold features and text features into a discriminator, and uses enhanced semantic relation features to verify the medical accuracy of the generated text and determine the consistency of clinical language style.
[0031] Optionally, step S3 specifically includes:
[0032] S31: Construction of attention module and organization of candidate feature set;
[0033] The object node feature set enhanced by the graph convolutional network in step S2 and predicate node feature set The unified organization is a set of candidate feature vectors A, which serves as the input source for the attention mechanism;
[0034] S32: Attention weight calculation and context vector generation;
[0035] At each time step t when the report is generated, the attention score is calculated as follows:
[0036]
[0037] Among them, h t This represents the hidden state output by the LSTM module at time step t, reflecting the semantic context of the current decoding stage; h s W is the s-th element in the candidate feature vector set A; a U a A learnable linear transformation matrix used to align the feature spaces of queries and keys;
[0038] The attention score is then normalized using softmax to obtain the final attention weight α. i And generate context vectors by weighted averaging.
[0039] S33: Construction of a two-layer LSTM network architecture and first-layer decoding processing;
[0040] A report generator is constructed using a two-layer LSTM network structure to achieve sequence decoding conversion from image encoded features to medical diagnostic reports;
[0041] set up and W represents the hidden state and memory unit of the first LSTM layer, respectively. e The word embedding matrix is represented; at time step t, the computation process of the first LSTM layer is as follows: Where c t-1 This represents the context vector of the decoder at the previous time step; it is both the output of the attention module and the input of the output module. It is the average pooling result of feature set A, used to provide global information and compared with c. t-1 The sum is then fed into the first LSTM layer;
[0042] S34: Second-layer LSTM decoding and vocabulary probability distribution prediction;
[0043] By calling the attention module f att (·), using the output of the first LSTM layer to generate the attention context vector for the current time step: Then, the attention feature vector of the current time step With query vector Feature concatenation is performed, and the resulting data is fed into a second LSTM layer for further sequence modeling. The hidden state output As a context vector; ultimately, through a W p and b p A parameterized linear layer, combined with a softmax activation function, is used to output and predict the probability distribution of the vocabulary, i.e., p(y t ∣y 1:t-1 ) = softmax(c t W p +b p );
[0044] S35: Implementation using continuous differentiable approximation and soft sampling mechanism;
[0045] End-to-end optimization is achieved using a continuously differentiable approximation method for discrete text samples;
[0046] The word distribution vector p generated by the model at step t t Treated as "soft words," they are used as input for step t+1; in step t+1, the word embedding vector is calculated as: e t+1 =W e p t ,in, Word embedding matrix;
[0047] Subsequently, e t+1 The text will be input into a two-layer LSTM to generate soft text sentences, denoted as... The randomness of sampling is controlled by the temperature parameter τ: Among them o t The input is the logit vector to the softmax function, and τ is initially set to 1 and gradually annealed to 0 during training.
[0048] S36: Joint optimization of generation loss and adaptation loss;
[0049] A composite loss function for the generator is constructed to balance the quality of report generation with the degree of alignment with target attributes;
[0050] Adaptation loss through L Att =E p(z)p(c) [logD(z|G r The calculation of (x,z))] ensures that the generated report meets the target attribute requirements of medical diagnosis;
[0051] Combining the basic generation loss, the generator's total loss function is expressed as:
[0052] Where α is a weighting parameter that controls the ratio of generation loss to adaptation loss.
[0053] In practice, step S4 specifically includes:
[0054] S41: Cross-modal fusion operation of image features and sentence features
[0055] Based on the medical report text output by the generator in step S3 and the original medical image, the discriminator network first performs a combination operation of image features and sentence features, specifically as follows:
[0056]
[0057] Where x = CNN(I) represents the image features extracted by CNN, and W e This represents the embedding matrix that maps sentences to one-hot word vectors, z1, z2...z L Vector representations of each word in a sentence;
[0058] S42: Adaptive Group Convolution Kernel Feature Extraction and Convolution Calculation
[0059] Adaptive group convolutional kernels with different window sizes are used to process the fused features. The feature maps of the input image and sentence are passed through the convolutional kernels. A convolution operation is performed on a window of size d×l to generate a new feature map c = [c1, c2, ..., c T-l+2 ];
[0060] Specifically, a certain feature c i The calculation method is as follows: ci =ReLU(w*ε i：i+l-1 +b), where * represents the convolution operation, b is the bias term, and ReLU(·) is the modified linear unit activation function;
[0061] S43: Time-dimensional max pooling and eigenvector construction
[0062] Apply time-dimensional max pooling to the convolutional feature map c to extract the maximum activation value corresponding to each convolutional kernel, denoted as:
[0063] Then, all the pooled features are concatenated to form a fixed-dimensional feature vector. in m is the number of convolutional kernels;
[0064] S44: Highway Network Enhancement and Gating Mechanism
[0065] A Highway network structure is introduced before the final fully connected layer to enhance the feature vectors. The calculation process of the Highway layer is as follows:
[0066]
[0067] in, and {b T ,b H}∈R n These represent the weights and biases of the Highway layer, respectively. The symbol σ represents the sigmoid activation function, and ⊙ represents element-wise multiplication.
[0068] S45: Probability Output Calculation and Diverse Sample Classification Mechanism
[0069] Enhanced features of the Highway network output By applying a fully connected layer and a sigmoid transform, the probability that a sentence is a true sentence given an image is calculated: Among them W o b o These correspond to the weight matrix and bias term of the output layer, respectively, and p is the final output probability of the discriminator.
[0070] The discriminator processes three classes of training samples simultaneously to ensure text naturalness and consistency of language style:
[0071] The first category is positive samples, which are real pairs (I, x) consisting of a real image and its matching real sentence. 1:T The second category is negative samples, which include fake pairs consisting of generated sentences and real images. The third category is negative samples, consisting of error pairs composed of mismatched real sentences and images. Let the three sample sets be denoted as follows:
[0072] S46: Loss Function Design and Convergence Guarantee in Adversarial Training
[0073] We introduce the zero-center Wasserstein distance and gradient penalty term to construct the discriminator loss function:
[0074]
[0075] Where I represents medical image features, x 1:T This represents a sequence of words from a real diagnostic report of length T. Indicates the generated report sequence, x 1:T This indicates a sequence of reports indicating incorrect matches.
[0076] First item Fit loss, using real medical images I and matched real reports x 1:T Positive sample pairs are used to train the discriminator to correctly identify real samples;
[0077] Second item Generate adversarial loss using medical image I and a generator-generated report The discriminant's ability to identify generated samples is calculated using binary cross-entropy;
[0078] Third item Naturalness loss, using medical image I and mismatched real report x 1:T Ensure that the generated report conforms to clinical expression standards;
[0079] Fourth item Gradient penalty term, using mixed samples z 1:T Calculate the gradient norm of the discriminator;
[0080] Optionally, the zero-centered gradient penalty term is calculated as follows:
[0081] S461: The complete expression for the gradient penalty term in the discriminator loss function
[0082] The discriminator's loss function uses the zero-center Wasserstein distance, and its complete expression is:
[0083]
[0084] S462: Interpolation Construction Strategy for Mixed Distribution Samples
[0085] Construct a mixed distribution sample set S u For medical image x and diagnostic report sequence z 1:T The mixed sample is constructed as follows:
[0086] Medical image domain preservation: Where, x real These are real medical images;
[0087] Diagnostic report sequence interpolation: in This is a sequence of actual diagnostic reports. The report sequence generated by the generator, ∈ ~Uniform(0,1) are random interpolation coefficients;
[0088] S463: Backpropagation Calculation and Numerical Implementation of Gradient Penalty Term
[0089] Discriminator gradient calculation: for mixed samples The automatic differential calculation discriminator D φ Regarding the gradient of the input:
[0090] L2 calculation of gradient norm:
[0091] Among them ||·|| F is the Frobenius norm, used to calculate the norm of image feature gradients; ||·||2 is the L2 norm, used to calculate the norm of word feature gradients.
[0092] The loss function design ensures that the discriminator can effectively guide the generator to produce diagnostic reports that conform to medical language standards and are highly matched with the image content, thereby significantly improving the authenticity and professional expression quality of medical reports.
[0093] The beneficial effects of this invention are:
[0094] This invention studies a medical report generation method based on visual manifold graphs and generative adversarial networks, which differs from the existing WGAN-GP method by employing (|||▽) D ||2-1|) 2 Unlike the technical approach of forcibly constraining the gradient norm to 1, this invention uses a negative gradient penalty term -k||▽ D || 2 The innovative design achieves zero-centrality constraints, which encourage the gradient norm of the discriminator to tend towards 0 rather than a fixed value of 1, thereby obtaining a smoother discriminant function and more stable training dynamics. This zero-centrality approach theoretically guarantees that the optimization process can reach local Nash equilibrium at a linear convergence rate, effectively avoiding common technical problems in traditional methods such as gradient explosion, gradient vanishing, and mode collapse. It is particularly suitable for the technical requirements of long-sequence, high-precision text generation tasks such as medical reports.
[0095] Through the complete technical implementation path described above, this invention successfully transforms the zero-center Wasserstein distance from a mathematical theoretical concept into a practically deployable neural network training algorithm, providing a technical solution with strict theoretical convergence guarantees and practical engineering feasibility for the automatic generation of medical reports.
[0096] Other advantages, objectives, and features of the invention will be set forth in part in the description which follows, and in part will be apparent to those skilled in the art from the following examination, or may be learned from practice of the invention. The objectives and other advantages of the invention can be realized and obtained through the following description. Attached Figure Description
[0097] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will now be described in further detail with reference to the accompanying drawings, wherein:
[0098] Figure 1 This is a system architecture diagram provided in an embodiment of the present invention;
[0099] Figure 2 This is a subjective experimental result image on the IU Xray dataset provided in this embodiment of the invention;
[0100] Figure 3 This is a diagram showing the subjective experimental results on the MIMIC-CXR dataset provided in this embodiment of the invention. Detailed Implementation
[0101] The preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings. It should be understood that the preferred embodiments are for illustrative purposes only and are not intended to limit the scope of protection of the present invention.
[0102] like Figure 1 As shown, this embodiment provides a medical report generation method based on visual manifold graphs and generative adversarial networks, including the following steps:
[0103] S1: Configure the Faster R-CNN network and graph resolver. The Faster R-CNN network extracts visual features from medical images, and the graph resolver extracts semantic information from diagnostic report data. Visual features are used as object nodes, and semantic information is used as predicate nodes to construct a visual manifold graph.
[0104] S2: Configure a graph convolutional network to enhance the processing of the visual popularity graph;
[0105] S3: Use a two-layer LSTM and a joint attention module to build a report generator to generate reports;
[0106] S4: Design a discriminator network to guide the report generator to improve the authenticity and professional expression quality of medical reports through image-text matching judgment, style consistency learning, and comparison of diverse samples;
[0107] S5: Collect batch-paired medical image-diagnostic report data to train the system model constructed according to steps S1-S4. Input the medical images into the trained system model to generate medical diagnostic reports that are both accurate and in line with clinical expression habits.
[0108] Optionally, step S1 specifically includes:
[0109] S11: For a given medical image I, a pre-trained Fast R-CNN network is used to identify object instances in the medical image, thereby obtaining a set of regions V = {v1, v2, ..., v...} corresponding to each object in the image. N};
[0110] S12: An attribute classifier consisting of a small fully connected layer fc-ReLU-fc-Sofemax is used to classify the relationships in these regions;
[0111] S13: Visual features extracted by the Fast R-CNN network Embedded with the identified tags The fusion process is performed to obtain the final object node features u. oi ,Right now:
[0112] Where W1 and W2 are trainable parameters used when fusing features, o i Represented as the i-th object node;
[0113] S14: Using r ij Indicates the connection object node o i and o j The predicate nodes; the node set V includes object nodes and predicate nodes, and the edge set E represents the connection relationship between object nodes and predicate nodes, thus obtaining the visual manifold graph G = (V, E).
[0114] Optionally, step S2 specifically includes:
[0115] S21: Based on the visual manifold G = (V, E) constructed in step S1, perform cross-modal feature integration operation on each node;
[0116] For object node o i Visual features extracted from the Faster R-CNN network The word embedding text features corresponding to this object Perform feature concatenation operations and pass them through a feedforward network φ with ReLU activation function. o Perform nonlinear transformation to obtain fused features
[0117] For the predicate node r ij Embed its words into text features Through feedforward network φ r Perform feature transformation to obtain a feature representation with uniform dimension.
[0118] S22: Perform graph convolution enhancement processing on the fusion features obtained in step S21;
[0119] For object node o i The neighborhood information is aggregated using graph convolution operations:
[0120]
[0121] Among them, f in f out This represents a feedforward neural network; These represent object nodes o respectively i N is the set of adjacent nodes of the endpoint and the starting point; i Represents object node o i The total number of adjacent nodes;
[0122] For the predicate node r ij Enhancement is achieved by fusing the features of the two connected object nodes:
[0123]
[0124] in These are the fusion features of the connected object nodes, f r It is a feedforward network;
[0125] S23: Assign the enhanced features to different network modules according to functional requirements: assign object node features Predicate node features Along with a complete enhanced manifold structure input generator, it performs attention-guided visual encoding, semantic generation of medical terminology, and ensures logical consistency of diagnostic reports. Simultaneously, it inputs a multimodal representation composed of enhanced visual manifold features and text features into a discriminator, and uses enhanced semantic relation features to verify the medical accuracy of the generated text and determine the consistency of clinical language style.
[0126] Optionally, step S3 specifically includes:
[0127] S31: Construction of attention module and organization of candidate feature set;
[0128] The object node feature set enhanced by the graph convolutional network in step S2 and predicate node feature set The unified organization is a set of candidate feature vectors A, which serves as the input source for the attention mechanism;
[0129] S32: Attention weight calculation and context vector generation;
[0130] At each time step t when the report is generated, the attention score is calculated as follows:
[0131]
[0132] Among them, h t This represents the hidden state output by the LSTM module at time step t, reflecting the semantic context of the current decoding stage; h s W is the s-th element in the candidate feature vector set A; a U a A learnable linear transformation matrix used to align the feature spaces of queries and keys;
[0133] The attention score is then normalized using softmax to obtain the final attention weight α. i And generate context vectors by weighted averaging.
[0134] S33: Construction of a two-layer LSTM network architecture and first-layer decoding processing;
[0135] A report generator is constructed using a two-layer LSTM network structure to achieve sequence decoding conversion from image encoded features to medical diagnostic reports;
[0136] set up and W represents the hidden state and memory unit of the first LSTM layer, respectively. e The word embedding matrix is represented; at time step t, the computation process of the first LSTM layer is as follows: Where c t-1 This represents the context vector of the decoder at the previous time step; it is both the output of the attention module and the input of the output module. It is the average pooling result of feature set A, used to provide global information and compared with c. t-1 The sum is then fed into the first LSTM layer;
[0137] S34: Second-layer LSTM decoding and vocabulary probability distribution prediction;
[0138] By calling the attention module f att(·), using the output of the first LSTM layer to generate the attention context vector for the current time step: Then, the attention feature vector of the current time step With query vector Feature concatenation is performed, and the resulting data is fed into a second LSTM layer for further sequence modeling. The hidden state output As a context vector; ultimately, through a W p and b p A parameterized linear layer, combined with a softmax activation function, is used to output and predict the probability distribution of the vocabulary, i.e., p(y t ∣y 1:t-1 ) = softmax(c t W p +b p );
[0139] S35: Implementation using continuous differentiable approximation and soft sampling mechanism;
[0140] End-to-end optimization is achieved using a continuously differentiable approximation method for discrete text samples;
[0141] The word distribution vector p generated by the model at step t t Treated as "soft words," they are used as input for step t+1; in step t+1, the word embedding vector is calculated as: e t+1 =W e p t ,in, Word embedding matrix;
[0142] Subsequently, e t+1 The text will be input into a two-layer LSTM to generate soft text sentences, denoted as... The randomness of sampling is controlled by the temperature parameter τ: Among them o t The input is the logit vector to the softmax function, and τ is initially set to 1 and gradually annealed to 0 during training.
[0143] S36: Joint optimization of generation loss and adaptation loss;
[0144] A composite loss function for the generator is constructed to balance the quality of report generation with the degree of alignment with target attributes;
[0145] Adaptation loss through L Att =E p(z)p(c) [logD(z|G r The calculation of (x,z))] ensures that the generated report meets the target attribute requirements of medical diagnosis;
[0146] Combining the basic generation loss, the generator's total loss function is expressed as:
[0147] Where α is a weighting parameter that controls the ratio of generation loss to adaptation loss.
[0148] In practice, step S4 specifically includes:
[0149] S41: Cross-modal fusion operation of image features and sentence features
[0150] Based on the medical report text output by the generator in step S3 and the original medical image, the discriminator network first performs a combination operation of image features and sentence features, specifically as follows:
[0151]
[0152] Where x = CNN(I) represents the image features extracted by CNN, and W e This represents the embedding matrix that maps sentences to one-hot word vectors, z1, z2...z L Vector representations of each word in a sentence;
[0153] S42: Adaptive Group Convolution Kernel Feature Extraction and Convolution Calculation
[0154] Adaptive group convolutional kernels with different window sizes are used to process the fused features. The feature maps of the input image and sentence are passed through the convolutional kernels. A convolution operation is performed on a window of size d×l to generate a new feature map c = [c1, c2, ..., c T-l+2 ];
[0155] Specifically, a certain feature c i The calculation method is as follows: c i =ReLU(w*ε i：i+l-1 +b), where * represents the convolution operation, b is the bias term, and ReLU(·) is the modified linear unit activation function;
[0156] S43: Time-dimensional max pooling and eigenvector construction
[0157] Apply time-dimensional max pooling to the convolutional feature map c to extract the maximum activation value corresponding to each convolutional kernel, denoted as:
[0158] Then, all the pooled features are concatenated to form a fixed-dimensional feature vector. in m is the number of convolutional kernels;
[0159] S44: Highway Network Enhancement and Gating Mechanism
[0160] A Highway network structure is introduced before the final fully connected layer to enhance the feature vectors. The calculation process of the Highway layer is as follows:
[0161]
[0162] in, and {b T ,b H}∈R n These represent the weights and biases of the Highway layer, respectively. The symbol σ represents the sigmoid activation function, and ⊙ represents element-wise multiplication.
[0163] S45: Probability Output Calculation and Diverse Sample Classification Mechanism
[0164] Enhanced features of the Highway network output By applying a fully connected layer and a sigmoid transform, the probability that a sentence is a true sentence given an image is calculated: Among them W o b o These correspond to the weight matrix and bias term of the output layer, respectively, and p is the final output probability of the discriminator.
[0165] The discriminator processes three classes of training samples simultaneously to ensure text naturalness and consistency of language style:
[0166] The first category is positive samples, which are real pairs (I, x) consisting of a real image and its matching real sentence. 1:T The second category is negative samples, which include fake pairs consisting of generated sentences and real images. The third category is negative samples, consisting of error pairs composed of mismatched real sentences and images. Let the three sample sets be denoted as follows:
[0167] S46: Loss Function Design and Convergence Guarantee in Adversarial Training
[0168] We introduce the zero-center Wasserstein distance and gradient penalty term to construct the discriminator loss function:
[0169]
[0170] Where I represents medical image features, x 1:T This represents a sequence of words from a real diagnostic report of length T. Indicates the generated report sequence, x 1:T This indicates a sequence of reports indicating incorrect matches.
[0171] First item Fit loss, using real medical images I and matched real reports x 1:T Positive sample pairs are used to train the discriminator to correctly identify real samples;
[0172] Second item Generate adversarial loss using medical image I and a generator-generated report The discriminant's ability to identify generated samples is calculated using binary cross-entropy;
[0173] Third item Naturalness loss, using medical image I and mismatched real report x 1:T Ensure that the generated report conforms to clinical expression standards;
[0174] Fourth item Gradient penalty term, using mixed samples z 1:T Calculate the gradient norm of the discriminator;
[0175] In practice, the zero-centered gradient penalty term is calculated as follows:
[0176] S461: The complete expression for the gradient penalty term in the discriminator loss function
[0177] The discriminator's loss function uses the zero-center Wasserstein distance, and its complete expression is:
[0178]
[0179] S462: Interpolation Construction Strategy for Mixed Distribution Samples
[0180] Construct a mixed distribution sample set S u For medical image x and diagnostic report sequence z 1:T The mixed sample is constructed as follows:
[0181] Medical image domain preservation: Where, x real These are real medical images;
[0182] Diagnostic report sequence interpolation: in This is a sequence of actual diagnostic reports. The report sequence generated by the generator, ∈ ~Uniform(0,1) are random interpolation coefficients;
[0183] S463: Backpropagation Calculation and Numerical Implementation of Gradient Penalty Term
[0184] Discriminator gradient calculation: for mixed samples The automatic differential calculation discriminator D φ Regarding the gradient of the input:
[0185] L2 calculation of gradient norm:
[0186] Among them ||·|| F is the Frobenius norm, used to calculate the norm of image feature gradients; ||·||2 is the L2 norm, used to calculate the norm of word feature gradients.
[0187] This invention organically embeds the zero-center Wasserstein distance into the core optimization objective of the medical report generation model, realizing a complete technical path from abstract mathematical theory to specific neural network implementation.
[0188] In the specific implementation, given N training images I, these images I are first transmitted to the feature representation through a pre-trained encoder, with the goal of training the generator to predict and generate the current word Z. t The probability, which is conditioned on the preceding word of the image's visual features, i.e., maximized:
[0189]
[0190] Where x j For the features of the j-th image, G is the true description of the j-th image, and L is the length of the description of the j-th image. Specifically, G θ (·) is a function controlled by the parameter θ, representing the probability distribution of generated words given an image or previously generated words. This can also be rewritten in the form of negative log-likelihood.
[0191] To ensure that the loss value is exactly zero when the generated distribution is perfectly aligned with the true distribution, thereby achieving a more stable and efficient optimization process, this invention introduces the zero-center Wasserstein distance as a theoretical guide:
[0192] Definition 1: Zero-center Wasserstein distance, let P r and P g It is a continuous distribution defined on the set Ω, where It is an open, bounded, and connected set. For any integer k, the zero-center Wasserstein distance is defined as follows:
[0193]
[0194] Where E(·) represents the expectation, P r P represents the true data distribution. g P represents the distribution of generated data. u The mixed distribution is used for gradient penalty, f represents the discriminant function, and k represents the threshold that limits the gradient penalty. Let Ω be the function space of all first-order differentiable functions. For any real-valued function f, we impose the common regularization constraint f(t) = -log(1 + exp(-t)) on the continuous differentiable space Ω.
[0195] The core technical contribution of this invention lies in establishing a systematic mapping mechanism from the abstract definition of the zero-center Wasserstein distance theory to the objective function of a specific trainable neural network. Through a carefully designed technical solution, the probability distribution P within the theoretical framework is... r P g P u These correspond to the specific datasets used in the actual training process: the set of true matching pairs S r Bearing the real data distribution P r The sampling is used to synthesize a set of matching pairs S. f Bearer generates data distribution P g The sampling, mixed distribution sample set S u Gradient penalty calculation is supported by interpolation techniques. Meanwhile, the abstract discriminant function f is processed by a deep neural network discriminant D with parameters φ. φ To implement this concretely, the gradient term ||▽ in the theoretical formula... x f(x)|| 2 Transformed into the actual gradient norm of the discriminator network output with respect to the input ||▽ D (x,z 1:T )|| 2 .
[0196] To enable the discriminator to accurately distinguish between real and fake data during subsequent training phases, and to ensure that the generated medical reports reside on the data manifold composed of real clinical reports, this invention employs three types of training sample pairs for supervised learning of the discriminator, including real matching pairs S. r (Real images and their corresponding real reports), synthetic matching pairs S f (Real images and generator-generated reports) and mismatched pairs S w (Real images and mismatched real reports).
[0197] Based on the above-mentioned mapping relationship from theory to implementation, this invention fully embeds the zero-center Wasserstein distance into the optimization objective of generative adversarial networks. Through mathematical transformation and engineering implementation, the theoretical formula is transformed into the following practically executable training objective function:
[0198]
[0199] Where α is the weight controlling the ratio of adversarial loss to language loss, θ, S represents the parameters for the generator and discriminator, respectively. r S fS w S u Let represent true matching pairs, synthetic matching pairs, false matching pairs, and mixed distribution samples, respectively. Each component of the objective function establishes a strict one-to-one correspondence with the theoretical definition of the zero-center Wasserstein distance: the first two terms correspond to the following in the theoretical formula: The third item directly corresponds to different implementations of the item. The fourth term is the core innovation of zero-center Wasserstein distance—the gradient penalty term. The direct technical implementation of this, with its negative sign design, embodies the essential characteristic of "zero centralization".
[0200] Compared with the WGAN-GP method in the prior art, (|||▽) D ||2-1|) 2 Unlike the technical approach of forcibly constraining the gradient norm to 1, this invention uses a negative gradient penalty term -k||▽ D || 2 The innovative design achieves zero-centrality constraints, which encourage the gradient norm of the discriminator to tend towards 0 rather than a fixed value of 1, thereby obtaining a smoother discriminant function and more stable training dynamics. This zero-centrality approach theoretically guarantees that the optimization process can reach local Nash equilibrium at a linear convergence rate, effectively avoiding common technical problems in traditional methods such as gradient explosion, gradient vanishing, and mode collapse. It is particularly suitable for the technical requirements of long-sequence, high-precision text generation tasks such as medical reports.
[0201] Through the complete technical implementation path described above, this invention successfully transforms the zero-center Wasserstein distance from a mathematical theoretical concept into a practically deployable neural network training algorithm, providing a technical solution with strict theoretical convergence guarantees and practical engineering feasibility for the automatic generation of medical reports.
[0202] Figure 2 and Figure 3The visualization results generated by this invention on the IU X-ray and MIMIC-CXR lung X-ray image datasets are presented respectively. Bone lesions are marked in bold, and the highlighted parts in green, blue, orange, red, and purple correspond to the supporting device information, pleural effusion, lung condition, pneumothorax, and heart-related lesions correctly identified by the model, respectively. Compared with seven methods, R2Gen (Chen et al., published at EMNLP 2020), R2GenRL (Qin et al., published at ACL 2022), CMN (Chen et al., published at ACL 2021), M2KT (Yang et al., published at MIA 2023), CGFTrans (Xv et al., published at JBHI 2024), DMVF (Tang et al., published at MIA 2025), and DDL-GCN (Xv et al., published at TKDD 2025), the results of this method are closer to the actual results and diagnose more pathological information.
[0203] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.
Claims
1. A method for generating medical reports based on visual manifold graphs and generative adversarial networks, characterized in that, Includes the following steps: S1: Configure the Faster R-CNN network and graph resolver. The Faster R-CNN network extracts visual features from medical images, and the graph resolver extracts semantic information from diagnostic report data. Visual features are used as object nodes, and semantic information is used as predicate nodes to construct a visual manifold graph. S2: Configure a graph convolutional network to enhance the processing of the visual popularity graph; S3: Use a two-layer LSTM and a joint attention module to build a report generator to generate reports; S4: Design a discriminator network to guide the report generator to improve the authenticity and professional expression quality of medical reports through image-text matching judgment, style consistency learning, and comparison of diverse samples; S5: Collect batch-paired medical image-diagnostic report data to train the system model constructed according to steps S1-S4. Input the medical images into the trained system model to generate medical diagnostic reports that are both accurate and in line with clinical expression habits.
2. The medical report generation method based on visual manifold graphs and generative adversarial networks according to claim 1, characterized in that, Step S1 specifically includes: S11: For a given medical image I, a pre-trained FastR-CNN network is used to identify object instances in the medical image, thereby obtaining a set of regions V = {v1, v2, ..., v...} corresponding to each object in the image. N }; S12: An attribute classifier consisting of a small fully connected layer fc-ReLU-fc-Sofemax is used to classify the relationships in these regions; S13: Visual features extracted by the FastR-CNN network Embedded with the identified tags The fusion process is performed to obtain the final object node features u. oi ,Right now: Where W1 and W2 are trainable parameters used when fusing features, o i Represented as the i-th object node; S14: Using r ij Indicates the connection object node o i and o j The predicate nodes; the node set V includes object nodes and predicate nodes, and the edge set E represents the connection relationship between object nodes and predicate nodes, thus obtaining the visual manifold graph G = (V, E).
3. The medical report generation method based on visual manifold graphs and generative adversarial networks according to claim 2, characterized in that, Step S2 specifically includes: S21: Based on the visual manifold G = (V, E) constructed in step S1, perform cross-modal feature integration operation on each node; For object node o i Visual features extracted from the Faster R-CNN network The word embedding text features corresponding to this object Perform feature concatenation operations and pass them through a feedforward network φ with ReLU activation function. o Perform nonlinear transformation to obtain fused features For the predicate node r ij Embed its words into text features Through feedforward network φ r Perform feature transformation to obtain a feature representation with uniform dimension. S22: Perform graph convolution enhancement processing on the fusion features obtained in step S21; For object node o i The neighborhood information is aggregated using graph convolution operations: Among them, f in f out Represents a feedforward neural network; N in N out These represent object nodes o respectively i N is the set of adjacent nodes of the endpoint and the starting point; i Represents object node o i The total number of adjacent nodes; For the predicate node r ij Enhancement is achieved by fusing the features of the two connected object nodes: in These are the fusion features of the connected object nodes, f r It is a feedforward network; S23: Assign the enhanced features to different network modules according to functional requirements: assign object node features Predicate node features Along with a complete enhanced manifold structure input generator, it performs attention-guided visual encoding, semantic generation of medical terminology, and ensures logical consistency of diagnostic reports. Simultaneously, it inputs a multimodal representation composed of enhanced visual manifold features and text features into a discriminator, and uses enhanced semantic relation features to verify the medical accuracy of the generated text and determine the consistency of clinical language style.
4. The medical report generation method based on visual manifold graphs and generative adversarial networks according to any one of claims 1-3, characterized in that, Step S3 specifically includes: S31: Construction of attention module and organization of candidate feature set; The object node feature set enhanced by the graph convolutional network in step S2 and predicate node feature set The unified organization is a set of candidate feature vectors A, which serves as the input source for the attention mechanism; S32: Attention weight calculation and context vector generation; At each time step t when the report is generated, the attention score is calculated as follows: Among them, h t This represents the hidden state output by the LSTM module at time step t, reflecting the semantic context of the current decoding stage; h s W is the s-th element in the candidate feature vector set A; a U a A learnable linear transformation matrix used to align the feature spaces of queries and keys; The attention score is then normalized using softmax to obtain the final attention weight α. i And generate context vectors by weighted averaging. S33: Construction of a two-layer LSTM network architecture and first-layer decoding processing; A report generator is constructed using a two-layer LSTM network structure to achieve sequence decoding conversion from image encoded features to medical diagnostic reports; set up and W represents the hidden state and memory unit of the first LSTM layer, respectively. e The word embedding matrix is represented; at time step t, the computation process of the first LSTM layer is as follows: Where c t-1 This represents the context vector of the decoder at the previous time step; it is both the output of the attention module and the input of the output module. It is the average pooling result of feature set A, used to provide global information and compared with c. t-1 The sum is then fed into the first LSTM layer; S34: Second-layer LSTM decoding and vocabulary probability distribution prediction; By calling the attention module f att (·), using the output of the first LSTM layer to generate the attention context vector for the current time step: Then, the attention feature vector of the current time step With query vector Feature concatenation is performed, and the resulting data is fed into a second LSTM layer for further sequence modeling. The hidden state output As a context vector; ultimately, through a W p and b p A parameterized linear layer, combined with a softmax activation function, is used to output and predict the probability distribution of the vocabulary, i.e., p(y t ∣y 1:t-1 ) = softmax(c t W p +b p ); S35: Implementation using continuous differentiable approximation and soft sampling mechanism; End-to-end optimization is achieved using a continuously differentiable approximation method for discrete text samples; The word distribution vector p generated by the model at step t t Treated as "soft words," they are used as input for step t+1; in step t+1, the word embedding vector is calculated as: e t+1 =W e p t ,in, Word embedding matrix; Subsequently, e t+1 The text will be input into a two-layer LSTM to generate soft text sentences, denoted as... The randomness of sampling is controlled by the temperature parameter τ: Among them o t The input is the logit vector to the softmax function, and τ is initially set to 1 and gradually annealed to 0 during training. S36: Joint optimization of generation loss and adaptation loss; Construct a composite loss function for the generator to balance report generation quality with target attribute conformity; Adaptation loss through L Att =E p(z)p(c) [logD(z|G r The calculation of (x,z))] ensures that the generated report meets the target attribute requirements of medical diagnosis; Combining the basic generation loss, the generator's total loss function is expressed as: Where α is a weighting parameter that controls the ratio of generation loss to adaptation loss.
5. The medical report generation method based on visual manifold graphs and generative adversarial networks according to any one of claims 1-3, characterized in that, Step S4 specifically includes: S41: Cross-modal fusion operation of image features and sentence features Based on the medical report text output by the generator in step S3 and the original medical image, the discriminator network first performs a combination operation of image features and sentence features, specifically as follows: Where x = CNN(I) represents the image features extracted by CNN, and W e This represents the embedding matrix that maps sentences to one-hot word vectors, z1, z2...z L Vector representations of each word in a sentence; S42: Adaptive Group Convolution Kernel Feature Extraction and Convolution Calculation The fused features are processed using adaptive group convolutional kernels with different window sizes. The feature maps of the input image and sentence are processed by convolutional kernel w∈R. d×l A convolution operation is performed on a window of size d×l to generate a new feature map c = [c1, c2, ..., c...]. T-l+2 ]; Specifically, a certain feature c i The calculation method is as follows: c i =ReLU(w*ε i：i+l-1 +b), where * denotes the convolution operation, b is the bias term, ReLU(·) is the modified linear unit activation function, and ε i：i+l-1 This represents a continuous feature window region in the input feature map, starting from position i and ending at position i+l-1, with a length of l units, corresponding to the window size of the convolution kernel. When the convolution kernel slides on the fused image-text feature map, it performs element-wise multiplication and summation with this feature window. After adding the bias term b, it generates new feature values through the ReLU activation function, thereby achieving effective extraction of local correlation patterns in medical images and report texts. S43: Time-dimensional max pooling and eigenvector construction Apply time-dimensional max pooling to the convolutional feature map c to extract the maximum activation value corresponding to each convolutional kernel, denoted as: Then, all the pooled features are concatenated to form a fixed-dimensional feature vector. in m is the number of convolution kernels; S44: Highway Network Enhancement and Gating Mechanism A Highway network structure is introduced before the final fully connected layer to enhance the feature vector. The calculation process of the Highway layer is as follows: Among them, {W T W H }∈R n×n and {b T ,b H }∈R n These represent the weights and biases of the Highway layer, respectively. The symbol σ represents the sigmoid activation function, and ⊙ represents element-wise multiplication. S45: Probability Output Calculation and Diverse Sample Classification Mechanism Enhanced features of the Highway network output By applying a fully connected layer and a sigmoid transform, the probability that a sentence is a true sentence given an image is calculated: Among them W o b o These correspond to the weight matrix and bias term of the output layer, respectively, and p is the final output probability of the discriminator. The discriminator processes three classes of training samples simultaneously to ensure text naturalness and consistency of language style: The first category is positive samples, which are real pairs (I, x) consisting of a real image and its matching real sentence. 1:T The second category is negative samples, which include fake pairs consisting of generated sentences and real images. The third category is negative samples, consisting of error pairs composed of mismatched real sentences and images. Let S denote the three sample sets respectively. r S f S w ; S46: Loss Function Design and Convergence Guarantee in Adversarial Training We introduce the zero-center Wasserstein distance and gradient penalty term to construct the discriminator loss function: Where I represents medical image features, x 1:T This represents a sequence of words from a real diagnostic report of length T. This represents the generated report sequence. This indicates a sequence of reports indicating incorrect matches. First item To fit the loss, real medical images I are used and matched real reports x are used. 1:T Positive sample pairs are used to train the discriminator to correctly identify real samples; Second item To generate adversarial loss, a report is generated using medical images and a generator. The discriminant's ability to identify generated samples is calculated using binary cross-entropy; Third item To account for the loss of naturalness, medical images and mismatched real reports were used. Ensure that the generated report conforms to clinical expression guidelines; Fourth item As a gradient penalty term, using mixed samples z 1:T Calculate the gradient norm of the discriminator.
6. The medical report generation method based on visual manifold graphs and generative adversarial networks according to claim 5, characterized in that, The zero-centered gradient penalty term is calculated as follows: S461: The complete expression for the gradient penalty term in the discriminator loss function The discriminator's loss function uses the zero-center Wasserstein distance, and its complete expression is: S462: Interpolation Construction Strategy for Mixed Distribution Samples Construct a mixed distribution sample set S u For medical image x and diagnostic report sequence z 1:T The mixed sample is constructed as follows: Medical image domain preservation: Where, x real These are real medical images; Diagnostic report sequence interpolation: in This is a sequence of actual diagnostic reports. The report sequence generated by the generator, ∈ ~Uniform(0,1) are random interpolation coefficients; S463: Backpropagation Calculation and Numerical Implementation of Gradient Penalty Term Discriminator gradient calculation: for mixed samples The automatic differential calculation discriminator D φ Regarding the gradient of the input: L2 calculation of gradient norm: Among them ||·|| F is the Frobenius norm, used to calculate the norm of image feature gradients; ||·||2 is the L2 norm, used to calculate the norm of word feature gradients.