Chest x-ray diagnosis report generation method based on memory reinforcement transformer

Abstract

The application discloses a kind of chest X-ray diagnosis report generation methods based on memory reinforcement Transform, according to actual needs to collect a number of chest X-ray images and corresponding diagnosis report, construct vocabulary based on diagnosis report, obtain the word probability matrix and word vector matrix of the word table contained in each diagnosis report, together with chest X-ray image constitutes a training sample, to obtain training sample set;Construct diagnosis report word prediction model based on Transform and memory module;Training diagnosis report word prediction model using training sample, input the chest X-ray image of diagnosis report to be generated into the diagnosis report word prediction model trained, obtain word probability matrix, finally generate the final diagnosis report text using text sequence generation algorithm.The application introduces memory module in Transform model, to automatically generate corresponding diagnosis report in only chest X-ray image case, while improving the quality of diagnosis report text generation.

Description

Technical Field

[0001] This invention belongs to the field of computer vision technology, and more specifically, relates to a method for generating chest X-ray diagnostic reports based on memory-enhanced Transformer. Background Technology

[0002] With the continuous development of deep learning algorithms, artificial intelligence (AI) technologies have been widely applied across various industries, including medical diagnosis. Medical images are frequently used for diagnosis and treatment in clinical practice. Writing reports based on patients' medical images is a time-consuming and tedious process, and inexperienced doctors may produce erroneous reports. In China, high-quality medical resources are relatively scarce, and a large number of patients require corresponding reports based on their medical images. However, the number of doctors is insufficient compared to the number of patients, making it essential to utilize AI technology to assist doctors in generating diagnostic reports. Currently, AI technology cannot completely replace doctors in clinical practice, but it can help them perform their work better and reduce their workload, especially for less experienced doctors, assisting them in writing diagnostic reports and reducing omissions. Using AI to generate reports is an effective method. However, two main challenges remain: first, with the increase in the number of patients, the model needs regular fine-tuning, which wastes a significant amount of time; second, due to the complexity of medical reports, the quality of text generated by existing AI technologies needs improvement.

[0003] Furthermore, the ability to rapidly iterate on models is a crucial aspect of this field. Because text generation models typically employ large models, updating model parameters when new data arrives incurs significant costs. Additionally, unlike the interpretation of natural images, medical diagnostic reports contain longer texts requiring more precise descriptions to comprehensively and accurately depict abnormal regions within the images—another important aspect of this field. Summary of the Invention

[0004] The purpose of this invention is to overcome the shortcomings of the prior art and provide a method for generating chest X-ray diagnostic reports based on memory-enhanced Transformer. By introducing a memory module into the Transformer model, the invention enables the automatic generation of corresponding diagnostic reports when only chest X-ray images are available, while also improving the quality of the generated diagnostic report text.

[0005] To achieve the above-mentioned objectives, the present invention provides a method for generating chest X-ray diagnostic reports based on memory-enhanced Transformers, comprising the following steps:

[0006] S1: Collect several chest X-ray images as needed, and obtain the corresponding diagnostic report for each chest X-ray image.

[0007] Count the words appearing in all diagnostic reports and compile them into a vocabulary, denoted as K. Set the length T of the diagnostic reports according to actual needs, and generate a word probability matrix P of size D×K for each report. When the t-th position in the diagnostic report is the k-th word in the vocabulary, let p be the element in the t-th row and k-th column of the word probability matrix P. t,k =1, otherwise p t,k =0, t=1,2,…,T, k=1,2,…,K;

[0008] Randomly generate an initial word vector c for each word in the vocabulary. k Let the dimension of the word vector be L. Then, generate a word vector matrix C of size T×L for each diagnostic report. The t-th row vector in the word vector matrix C is the word vector of the t-th word in the diagnostic report.

[0009] The word vector matrix and word probability matrix of each chest X-ray image and the corresponding diagnostic report are used as training samples to obtain the training sample set;

[0010] S2: Construct a word prediction model for the diagnostic report, including a visual feature encoder, a Transformer encoder, a memory-enhanced decoder, a fully connected layer, and a Softmax layer, where:

[0011] A visual feature encoder is used to encode visual features from chest X-ray images, resulting in visual features X = {x1, x2, ..., x...} N The output is then fed to the Transformer encoder, where N represents the number of visual features;

[0012] The Transformer encoder is used to determine the received visual features X = {x1, x2, ..., x...} N Extract the hidden states to obtain each visual feature x. i The latent vector h i And output to the memory enhancement decoder;

[0013] Memory reinforcement decoder is used for N latent vectors h i Decoding is performed to obtain the decoded features, which are then sent to the fully connected layer. The memory-enhanced decoder includes an output embedding layer, L groups of network blocks, and a memory module, wherein:

[0014] The output embedding layer is used to embed the word vector matrix C in the training samples, resulting in an embedded word vector matrix Y of size T×L. Let y be the embedded word vector of the t-th word. tThe embedded word vector matrix Y is output to the memory module, the masked multi-head attention layer, and the first feature fusion module of the first group of network blocks;

[0015] The memory module is used to generate a memory matrix M based on the embedded word vector matrix Y, and outputs it to the first MCLN module, the second MLCN module, and the third MLCN module in each group of network blocks; wherein, the memory matrix M is generated by iterative update, specifically as follows:

[0016] 1) Let the word index t = 1, initialize the initial memory matrix M0 of size D × L, and initialize the word embedding vector y0, where D represents the size of the memory slot in the memory module, and D < T;

[0017] 2) The memory matrix M of the t-th word is obtained using the following formula. t :

[0018] M t =(1-Z) t )⊙C t +Z t ⊙M t-1

[0019] in:

[0020] C t =tanh(W mc M t-1 +W sc S t +b c )

[0021] Z t =sigmoid(W mz M t-1 +W sz S t +b z )

[0022] S t =MultiHeadAtt(M t-1 ,[M t-1 ;y t-1 ],[M t-1 ;y t-1 ])

[0023] ⊙ represents the Adama product, W mc W sc W mz W sz b represents the learnable weights. c b z Represents the learnable bias term; C t Z represents the internal state. tThese are update gates; tanh() and sigmoid() are activation functions.

[0024] MultiHeadAtt() represents a multi-head attention mechanism, in which the memory matrix M is used. t-1 As a query for multi-head attention, then the M of the previous word t-1 The word embedding vector y of the previous word t-1 The connection yields [M] t-1 ;y t-1 ], serving as the key and value for multi-head attention;

[0025] 3) Determine if t < T. If yes, let t = t + 1 and return to step 2); otherwise, proceed to step 4.

[0026] 4) Convert each memory matrix M t Convert the data into a memory vector m of length D×L using row-major order. t Then use T memory vectors m t As row vectors, they form a memory matrix M of size T×(D×L);

[0027] Each group of network blocks includes a multi-head attention module for occlusion, a first feature fusion module, a first MCLN module, a multi-head attention module, a second feature fusion module, a second MLCN module, a forward feedback module, a third feature fusion module, and a third MLCN module, wherein:

[0028] The occlusion multi-head attention module is used to process the received feature matrix f based on the occlusion multi-head attention mechanism. A,In After processing, a feature matrix f of size T×L is obtained. A,1 The feature matrix received by the occlusion multi-head attention module in the first network block is the embedded word vector matrix Y output by the output embedding layer, and the feature matrix received by the occlusion multi-head attention module in the second to Lth network blocks is the feature matrix output by the third MLCN module in the previous network block.

[0029] The first feature fusion module is used to process the feature matrix f received by the occlusion multi-head attention module. A,In and the output feature matrix f A,1 The features are added together to obtain a fusion feature matrix f1 of size T×L, which is then output to the first MCLN module.

[0030] The first MCLN module is used to process the fused feature matrix f1 based on the memory matrix M to obtain a feature matrix f of size T×L. m,1 The output is sent to the multi-head attention module and the second feature fusion module; the processing procedure of the MCLN module is as follows:

[0031] Let In be the fused feature matrix received by the MCLN module. The feature matrix f is obtained using the following method. m,1 The row vector f in row t is m,1 [t]:

[0032] Initialize the learnable parameters γ and β, and then update the parameter γ using the formula shown below:

[0033] Δγ t =f mlp (m t )

[0034]

[0035] The parameter β is updated using the following formula:

[0036] Δβ t =f mlp (m t )

[0037]

[0038] Among them, f mlp () represents a multilayer perceptron;

[0039] The row vector f in row t m,1 The formula for calculating [t] is shown below:

[0040]

[0041] Among them, In t is the t-th row vector of the input fusion feature matrix In, where μ and v represent the mean and standard deviation of the row vectors in the fusion feature matrix In;

[0042] The multi-head attention module is used to receive N latent vectors h. i and characteristic matrix f m,1 The latent vector h i As the query and key for multi-head attention, the feature matrix f m,1 As the value of multi-head attention, a feature matrix f of size T×L is obtained based on the multi-head attention mechanism. A,2 And send it to the second feature fusion module;

[0043] The second feature fusion module is used to process the feature matrix f m,1 and characteristic matrix f A,2 The features are added together to obtain a fusion feature matrix f2 of size T×L, which is then output to the second MCLN module.

[0044] The second MLCN module is used to process the fused feature matrix f2 based on the memory matrix M, resulting in a feature matrix f of size T×L. m,2 The output is sent to the forward feedback module and the third feature fusion module;

[0045] The forward feedback module is used to process the feature matrix f m,2 A linear transformation is performed to obtain a feature matrix f3 of size T×L, which is then output to the third feature fusion module.

[0046] The third feature fusion module will fuse the feature matrix f m,2 Add the feature matrix f3 to obtain a fused feature matrix f4 of size T×L, which is then output to the third MLCN module.

[0047] The third MLCN module is used to process the fused feature matrix f4 based on the memory matrix M, resulting in a feature matrix f of size T×L. m,3 The output is produced by the third MLCN module in the first to L-1 network blocks, which outputs the feature matrix f. m,3 The output is passed to the next network block. The third MLCN module in the Lth network block will output the feature matrix f. m,3 Output to a fully connected layer;

[0048] The fully connected layer is used to perform linear transformation on the received feature matrix and output the resulting features to the Softmax layer;

[0049] The Softmax layer is used to predict the probability of each word in the vocabulary appearing in each position in the diagnostic report based on the obtained features, thus obtaining a word probability matrix;

[0050] S3: Take each chest X-ray image in the training sample set in step S1 as input and the corresponding word probability matrix as the expected output to train the diagnostic report word prediction model and obtain the trained diagnostic report word prediction model.

[0051] S4: For the chest X-ray image to be used to generate a diagnostic report, input the chest X-ray image into the diagnostic report word prediction model trained in step S3 to obtain the word probability matrix corresponding to the chest X-ray image.

[0052] S5: Based on the word probability vectors generated in step S4, the final diagnostic report text is generated using a text sequence generation algorithm.

[0053] This invention relates to a method for generating chest X-ray diagnostic reports based on memory-enhanced Transformers. The method involves collecting several chest X-ray images and corresponding diagnostic reports as needed, constructing a vocabulary based on the diagnostic reports, obtaining the word probability matrix and word vector matrix for each word in the vocabulary of each diagnostic report, and combining these with the chest X-ray images to form a training sample set. A diagnostic report word prediction model based on Transformers and a memory module is then constructed. This model is trained using the training samples. The chest X-ray images of the diagnostic report to be generated are input into the trained diagnostic report word prediction model to obtain the word probability matrix. Finally, a text sequence generation algorithm is used to generate the final diagnostic report text.

[0054] The present invention has the following beneficial effects:

[0055] 1) The diagnostic report word prediction model of the present invention uses a memory reinforcement decoder to memorize specific patterns of text during the calculation process of the model, thereby improving the quality of diagnostic report text generation;

[0056] 2) In the diagnostic report word prediction model of the present invention, the visual feature encoder can adopt a pre-trained ViT module, which saves the training time of the diagnostic report word prediction model, improves the ability of visual features to represent chest X-ray images, and thus further improves the quality of diagnostic report text generation. Attached Figure Description

[0057] Figure 1 This is a flowchart illustrating a specific implementation method for generating chest X-ray diagnostic reports based on memory-enhanced Transformer according to the present invention.

[0058] Figure 2 This is a structural diagram of the diagnostic report word prediction model in this invention;

[0059] Figure 3 This is a structural diagram of the memory enhancement decoder of the present invention;

[0060] Figure 4 This is a structural diagram of the memory module in this embodiment. Detailed Implementation

[0061] The specific embodiments of the present invention will now be described with reference to the accompanying drawings to enable those skilled in the art to better understand the invention. It should be particularly noted that in the following description, detailed descriptions of known functions and designs that might obscure the main content of the invention will be omitted here.

[0062] Example

[0063] Figure 1This is a flowchart illustrating a specific implementation of the method for generating chest X-ray diagnostic reports based on memory-enhanced Transformer according to the present invention. Figure 1 As shown, the specific steps of the method for generating chest X-ray diagnostic reports based on memory-enhanced Transformer according to the present invention include:

[0064] S101: Obtain training samples:

[0065] Collect several chest X-ray images as needed, and obtain the corresponding diagnostic report for each chest X-ray image.

[0066] For the diagnostic reports in the training sample set, count the words that appear in all diagnostic reports and compile them into a word list, denoted as K. Set the length T of the diagnostic report according to actual needs, and generate a word probability matrix P of size T×K for each diagnostic report. When the t-th position in the diagnostic report is the k-th word in the word list, let p be the element in the t-th row and k-th column of the word probability matrix P. t,k =1, otherwise p t,k =0, t=1,2,…,T, k=1,2,…,K.

[0067] Randomly generate an initial word vector c for each word in the vocabulary. k Let the dimension of the word vector be L. Then, generate a word vector matrix C of size T×L for each diagnostic report. The row vector of the t-th row in the word vector matrix C is the word vector of the word at the t-th position in the diagnostic report.

[0068] The word vector matrix and word probability matrix of each chest X-ray image and the corresponding diagnostic report are used as training samples to obtain the training sample set.

[0069] To better extract features from chest X-ray images and improve the training samples for the model, it is usually necessary to preprocess the chest X-ray images and diagnostic reports using a preset method. In this embodiment, the preprocessing method for the chest X-ray images is as follows: the size of the chest X-ray image is normalized to a preset size (256x256 in this embodiment), then random image cropping is performed (the cropped size is 224x224 in this embodiment), the cropped image is randomly horizontally flipped, and finally, the image is regularized.

[0070] In this embodiment, the preprocessing method for the diagnostic report is as follows: the diagnostic report is cleaned, then all punctuation marks except for periods are removed, and words involving patient privacy are replaced with default characters (e.g., "XXX").

[0071] S102: Constructing a word prediction model for diagnostic reports:

[0072] This invention first predicts the probability of a word appearing in the diagnostic report based on a chest X-ray image, and then generates the diagnostic report based on the word's probability of appearance. Clearly, the accuracy of the word prediction probability is crucial to the quality of the final diagnostic report. Therefore, this invention focuses on improving the word prediction model for the diagnostic report by introducing a memory model, thereby increasing the accuracy of the word prediction probability. Figure 2 This is a structural diagram of the diagnostic report word prediction model in this invention. For example... Figure 2 As shown, the diagnostic report word prediction model in this invention includes a visual feature encoder, a Transformer encoder, a memory reinforcement decoder, a fully connected layer, and a Softmax layer. Each module will be described in detail below.

[0073] A visual feature encoder is used to encode visual features from chest X-ray images, resulting in visual features X = {x1, x2, ..., x...} N The output is then sent to the Transformer encoder, where N represents the number of visual features.

[0074] To improve the representational ability of visual features for chest X-ray images, a pre-trained CNN (Convolutional Neural Network) or ViT (Vision Transformer) module can be used as the visual feature encoder. In this embodiment, a pre-trained ViT module is used as the visual feature encoder. Furthermore, to improve the performance of the ViT module, the MAE (Masked Autoencoders) method is used for pre-training. The specific method is as follows:

[0075] A decoder is selected based on actual needs. The output of the ViT module is used as the input of the decoder. The decoder reconstructs the image based on the visual features of the ViT module, thus forming an encoder-decoder model. Chest X-ray images in the training sample set are divided into several image blocks of equal size. A portion of these image blocks is randomly occluded according to a preset occlusion ratio, resulting in occluded chest X-ray images. These occluded chest X-ray image blocks are used as input to the encoder-decoder model, and the original chest X-ray image is used as the expected output of the model. The encoder-decoder model is then trained, thus completing the training of the ViT module. The loss function is the image reconstruction loss, typically using the mean squared error function.

[0076] The Transformer encoder is used to determine the received visual features X = {x1, x2, ..., x...} N Extract the hidden states to obtain each visual feature x. i The latent vector h iThe output is then sent to the memory reinforcement decoder. Since the visual feature encoder extracts visual features, while the task of this invention is text generation, there is a certain gap between these two tasks. Therefore, this invention uses a Transformer encoder to extract the latent vectors of the visual features, thereby achieving adaptation between the two tasks. In this embodiment, a standard 3-layer Transformer encoder is used.

[0077] Memory reinforcement decoder is used for N latent vectors h i Decode the data, obtain the decoded features, and send them to the fully connected layer.

[0078] Because diagnostic reports exhibit certain similarities, meaning they share patterns that can be recorded by external memory, this pattern—"The heart is normal in size." and "The heart size within normal limits."—frequently appears in reports with similar images and shows consistency. Therefore, this invention incorporates a memory module into the Transformer decoder. This memory module can memorize specific patterns in the text during model computation, thereby improving the quality of generated diagnostic reports. Memory modules are commonly used in text generation, video interpretation, and medical diagnostic report generation. In this invention, to further enhance the quality of generated diagnostic reports, the memory module has been improved for better integration with the Transformer decoder. Figure 3 This is a structural diagram of the memory enhancement decoder of the present invention. (See diagram below.) Figure 3 As shown, the memory-enhanced decoder in this invention includes an output embedding layer, L groups of network blocks, and a memory module, wherein:

[0079] The output embedding layer is used to embed the word vector matrix C in the training samples, resulting in an embedded word vector matrix Y of size T×L. Let y be the embedded word vector of the t-th word. t The embedded word vector matrix Y is output to the memory module, the masked multi-head attention layer, and the first feature fusion module of the first group of network blocks.

[0080] The memory module generates a memory matrix M based on the embedded word vector matrix Y, and outputs it to the first MCLN module, the second MLCN module, and the third MLCN module in each group of network blocks. The memory matrix M is generated iteratively, specifically as follows:

[0081] 1) Let the word index t = 1, initialize an initial memory matrix M0 of size D × L, and initialize the word embedding vector y0, where D represents the size of the memory slot in the memory module, D < T. In this embodiment, the initial memory matrix M0 is composed of D randomly selected initial word vectors, and the word vector y0 is a randomly selected initial word vector.

[0082] 2) The memory matrix M of the t-th word is obtained using the following formula. t :

[0083] M t =(1-Z) t )⊙C t +Z t ⊙M t-1

[0084] in:

[0085] C t =tanh(W mc M t-1 +W sc S t +b c )

[0086] Z t =sigmoid(W mz M t-1 +W sz S t +b z )

[0087] S t =MultiHeadAtt(M t-1 ,[M t-1 ;y t-1 ],[M t-1 ;y t-1 ])

[0088] ⊙ represents the Hadamard product, W mc W sc W mz W sz b represents the learnable weights. c b z This represents the learnable bias term. (C) t Z represents the internal cell state. t The update gate determines which information from the previous memory state is retained, thereby reducing redundancy and maintaining the consistency of the generated paragraphs. `tanh()` and `sigmoid()` are activation functions.

[0089] MultiHeadAtt() represents a multi-head attention mechanism, in which the memory matrix M is used. t-1 As a query for multi-head attention, then the M of the previous word t-1 The word embedding vector y of the previous word t-1 The connection yields [M] t-1 ;y t-1 ], serving as the key and value for multi-head attention.

[0090] This update mechanism is very similar to LSTM and GRU, but the difference is that multi-head attention is used to encode the memory state, thus supporting multiple memory slots instead of a single memory slot in LSTM and GRU, which gives it a greater ability to model complex relationships.

[0091] 3) Determine if t < T. If yes, let t = t + 1 and return to step 2); otherwise, proceed to step 4.

[0092] 4) Convert each memory matrix M t Convert the data into a memory vector m of length D×L using row-major order. t Then use T memory vectors m t As row vectors, they form a memory matrix M of size T×(D×L).

[0093] Figure 4 This is a structural diagram of the memory module in this embodiment. For example... Figure 4 As shown, in this embodiment, the memory module includes a first multilayer perceptron, a second multilayer perceptron, a connection module, a multi-head attention module, a third multilayer perceptron, a fourth multilayer perceptron, a first summation module, a second summation module, a tanh activation layer, a sigmoid activation layer, and a memory matrix operation module, wherein:

[0094] The first multilayer perceptron is used to process the memory matrix M of the previous word calculated by the memory matrix operation module. t-1 Assign weight W mz , obtain W mz M t-1 And send it to the first summation module.

[0095] The second multilayer perceptron is used to process the memory matrix M of the previous word calculated by the memory matrix operation module. t-1 Assign weight W mc , obtain W mc M t-1 And send it to the second summation module.

[0096] The connection module is used to connect the memory matrix M of the previous word calculated by the memory matrix operation module.t-1 With embedded word vector y t-1 Perform the connection to obtain the connection matrix [M] t-1 ;y t-1 And send it to the multi-head attention module.

[0097] The multi-head attention module is used to integrate the memory matrix M t-1 As a query for multi-head attention, the connection matrix [M] t-1 ;y t-1 As the keys and values ​​of multi-head attention, the feature S is obtained through processing. t And send it to the third and fourth multilayer perceptrons.

[0098] The third multilayer perceptron is used to analyze feature S. t Assign weight W sz , obtain W sz S t And send it to the first summation module.

[0099] The fourth multilayer perceptron is used for feature S t Assign weight W sc , obtain W sc S t And send it to the second summation module.

[0100] The first summation module is used to calculate the received W. mz M t-1 W sz S t Together with deviation term b z Summing, we get W mz M t-1 +W sz S t +b z And send it to the sigmoid activation layer.

[0101] The second summation module is used to calculate the received W. mc M t-1 W sc S t Together with deviation term b c Summing, we get W mc M t-1 +W sc S t +b c And send it to the tanh activation layer.

[0102] The sigmoid activation layer is used to apply the activation function sigmoid() to W. mz M t-1 +W sz S t +b zProcessing yields the update gate Z. t And send it to the memory matrix operation module.

[0103] The tanh activation layer is used to apply the activation function tanh() to W. mc M t-1 +W sc S t +b c Processing yields the internal state C. t And send it to the memory matrix operation module.

[0104] The memory matrix operation module is used to perform operations based on the internal state C. t and Update Gate Z t For memory matrix M t-1 Perform the calculations to obtain the memory matrix M. t The calculation formula is as follows:

[0105] M t =(1-Z) t )⊙C t +Z t ⊙M t-1

[0106] And each memory matrix M t Convert the data into a memory vector m of length D×L using row-major order. t Then use T memory vectors m t This forms a memory matrix M of size T×(D×L).

[0107] Each group of network blocks includes a multi-head attention module for occlusion, a first feature fusion module, a first MCLN module, a multi-head attention module, a second feature fusion module, a second MLCN module, a forward feedback module, a third feature fusion module, and a third MLCN module, wherein:

[0108] The Masked Multi-head Attention module is used to process the received feature matrix f based on the masked multi-head attention mechanism. A,In After processing, a feature matrix f of size T×L is obtained. A,1The masked multi-head attention mechanism in the first network block receives the feature matrix Y, which is the word embedding matrix output by the output embedding layer. The masked multi-head attention modules in the second to Lth network blocks receive the feature matrices output by the third MLCN module in the previous network block. The difference between the masked multi-head attention mechanism and the ordinary multi-head attention mechanism is that before performing Softmax calculation on the intermediate features, the intermediate features need to be masked using a Mask matrix (when an element in the Mask matrix is ​​0, the corresponding element in the intermediate matrix is ​​changed to -1e9; when an element in the Mask matrix is ​​0, the corresponding element in the intermediate matrix remains unchanged) to mask some feature values. In the Mask matrix, values ​​above the main diagonal are 0, and values ​​at and below the main diagonal are 1.

[0109] The first feature fusion module is used to process the feature matrix f received by the occlusion multi-head attention module. A,In and the output feature matrix f A,1 The features are added together to obtain a fusion feature matrix f1 of size T×L, which is then output to the first MCLN module.

[0110] The first MCLN (Memory-driven Conditional Layer Normalization) module is used to process the fused feature matrix f1 based on the memory matrix M to obtain a feature matrix f of size T×L. m,1 The output is sent to the multi-head attention module and the second feature fusion module.

[0111] External memory is widely used in natural language processing tasks. However, it cannot be well integrated with the decoding process of the Transformer decoder because the decoding and memory update processes are highly coupled during text generation. To address this issue and improve the quality of long text generation, this invention introduces the MCLN module. MCLN combines layer regularization and external memory to enhance the decoding capabilities of the Transformer. To this end, MCLN introduces two key parameters, γ and β, to scale and shift the learned features, respectively. The processing procedure of the MCLN module is as follows:

[0112] Let In be the fused feature matrix received by the MCLN module. The feature matrix f is obtained using the following method. m,1 The row vector f in row t is m,1 [t]:

[0113] Initialize the learnable parameters γ and β, and then update the parameter γ using the formula shown below:

[0114] Δγ t=f mlp (m t )

[0115]

[0116] Similarly, the parameter β is updated using the following formula:

[0117] Δβ t =f mlp (m t )

[0118]

[0119] Among them, f mlp () represents a multilayer perceptron. That is, the MCLN module updates parameters γ and β based on the values ​​from external memory to obtain the parameters corresponding to the current word. and Finally, these two parameters are incorporated into the calculation of layer normalization, and the row vector f of row t is... m,1 The formula for calculating [t] is as follows:

[0120]

[0121] Among them, In t is the t-th row vector of the input fusion feature matrix In, where μ and v represent the mean and standard deviation of the row vectors in the fusion feature matrix In.

[0122] The multi-head attention module is used to receive N hidden vectors h. i and characteristic matrix f m,1 The latent vector h i As the query and key for multi-head attention, the feature matrix f m,1 As the value of multi-head attention, a feature matrix f of size T×L is obtained based on the multi-head attention mechanism. A,2 And send it to the second feature fusion module.

[0123] The second feature fusion module is used to process the feature matrix f m,1 and characteristic matrix f A,2 The features are added together to obtain a fusion feature matrix f2 of size T×L, which is then output to the second MCLN module.

[0124] The second MLCN module is used to process the fused feature matrix f2 based on the memory matrix M, resulting in a feature matrix f of size T×L. m,2 The output is sent to the forward feedback module and the third feature fusion module.

[0125] The feedforward module is used to process the feature matrix f. m,2 A linear transformation is performed to obtain a feature matrix f3 of size T×L, which is then output to the third feature fusion module.

[0126] The third feature fusion module will fuse the feature matrix f m,2 The fused feature matrix f4 is added to the feature matrix f3 to obtain a fused feature matrix f4 of size T×L, which is then output to the third MLCN module.

[0127] The third MLCN module is used to process the fused feature matrix f4 based on the memory matrix M, resulting in a feature matrix f of size T×L. m,3 The output is produced by the third MLCN module in the first to L-1 network blocks, which outputs the feature matrix f. m,3 The output is passed to the next network block. The third MLCN module in the Lth network block will output the feature matrix f. m,3 Output to the fully connected layer.

[0128] The fully connected layer is used to perform linear transformations on the received feature matrix and output the resulting features to the Softmax layer.

[0129] The Softmax layer is used to predict the probability of each word in the vocabulary appearing in each position in the diagnostic report based on the obtained features, thus obtaining a word probability matrix.

[0130] S103: Training the diagnostic report word prediction model:

[0131] Using each chest X-ray image in the training sample set in step S101 as input and the corresponding word probability matrix as the expected output, the diagnostic report word prediction model is trained to obtain the trained diagnostic report word prediction model.

[0132] In this embodiment, negative log-likelihood is used as the loss function for the diagnostic report word prediction model.

[0133] When the visual feature encoder uses a pre-trained ViT module, the parameters of the ViT module can be kept constant during the training of the diagnostic report word prediction model. This can greatly reduce the amount of computation during the model update process and significantly reduce the model's convergence time. Experimental results show that the convergence time can be shortened from 2 to 3 days to 1 day.

[0134] S104: Diagnostic Report Word Prediction:

[0135] For the chest X-ray image to be used to generate a diagnostic report, input the chest X-ray image into the diagnostic report word prediction model trained in step S103 to obtain the word probability matrix corresponding to the chest X-ray image.

[0136] S105: Generate diagnostic report:

[0137] Based on the word probability vectors generated in step S104, the final diagnostic report text is generated using a text sequence generation algorithm.

[0138] Currently, various text sequence generation algorithms have been proposed in the industry, such as Greedy Search, Viterbi algorithm, and Beam Search, which can be selected according to needs in practical applications. In this embodiment, the Beam Search algorithm is adopted.

[0139] To better illustrate the technical effects of this invention, specific examples were used to experimentally verify the invention. The selected datasets were the MIMIC-CXR dataset and the IU-Xray dataset. Simultaneously, some existing diagnostic report generation methods were selected for comparative verification in this experiment, namely:

[0140] Comparison Method 1: CNN-RNN model, which uses CNN as a visual feature extractor and RNN (LSTM) as a text generator.

[0141] Comparison Method 2: AdaAtt model, which uses CNN as the visual feature extractor and LSTM as the text generator of the model. However, the difference between it and the CNN-RNN model is that it introduces an adaptive attention mechanism in LSTM, which allows the model to pay more attention to certain regions in the image.

[0142] Comparison with method 3: Att2in model, which introduces reinforcement learning to improve the quality of text generation.

[0143] Comparison Method 4: Up-Down Model. In the visual feature extractor part of Fast R-CNN, a bottom-up attention mechanism is introduced, which calculates attention based on the target and salient regions in the image.

[0144] Comparison Method 5: Transformer model, using vanilla Transformer to generate diagnostic reports for medical images.

[0145] Comparison Method 5: The HRGR-Agent model uses CNN to extract visual features, RNN to generate text, and then uses reinforcement learning to reward the recall strategy module and the text generation module at the sentence level and word level, respectively.

[0146] Comparison Method 6: CMAS-RL model, which uses reinforcement learning to train the model and introduces a multi-agent cooperative system, can implicitly capture the imbalanced distribution between abnormal and normal regions in the image.

[0147] Comparison Method 7: R2Gen model, which introduces a memory module based on the vanilla Transformer.

[0148] Comparison Method 8: CMN model, which improves the external memory module based on R2Gen.

[0149] Comparison Method 9: PPKED model, whose basic architecture is Transformer. The Transformer encoder performs text and visual feature alignment, and also introduces external prior knowledge, knowledge graph, text information from existing reports, etc.

[0150] In this experimental verification, the visual feature extractor in the diagnostic report word prediction model proposed in this invention is ViT-base, which is pre-trained on the CheXpert dataset using the MAE method, with the hyperparameters of the pre-training model referring to MAE. The parameters of the visual feature extractor remain unchanged throughout the entire model training process. An 8-head multi-head attention mechanism is used, with the Adam optimizer, a batch size of 48, a learning rate of 3e-4, a weight decay of 5e-5, and amsgrad applied.

[0151] In this experimental verification, three evaluation methods—BLEU, METEOR, and ROUGE—were used to obtain the performance indicators for each method. Table 1 is a comparison table of the indicators of the present invention and the comparative methods in this embodiment.

[0152]

[0153] Table 1

[0154] As shown in Table 1, on the MIMIC-CXR dataset, the memory slot size of this invention is 2, and its BLEU score comprehensively outperforms other models. Its METEOR score is almost on par with PPKED, with only a slight difference in the ROUGE-L score. On the IU-Xray dataset, the memory slot size of this invention is 1, and its BLEU score shows an overwhelming advantage over other algorithms. Its METEOR score is also significantly better than other algorithms, with only a slight difference in the ROUGE-L score. Therefore, overall, the method proposed in this invention is quite effective and has significant advantages.

[0155] In addition, ablation experiments were conducted on the MIMIC-CXR dataset for this verification. Table 2 is a performance comparison table of the diagnostic report word prediction model of this invention in the ablation experiment in this embodiment.

[0156]

[0157] Table 2

[0158] In this experimental verification, the BASE model for the diagnostic report word prediction model is a ViT + Transformer encoder + Transformer decoder. Several phenomena can be observed from the ablation experiments: First, on most NLG (Natural Language Generation) metrics, the BASE + memory module and BASE + pre-trained ViT + memory module outperform the BASE version. This phenomenon proves the effectiveness of introducing an external memory module into the Transformer, because the text of chest X-ray diagnostic reports is highly patterned, and these patterns are reasonably modeled. Second, comparing the BASE model with BASE + pre-trained ViT, and BASE + memory module with BASE + pre-trained ViT + memory module, it can be concluded that using pre-trained ViT as the model's visual feature extractor is very effective, because pre-trained ViT has already learned to correctly extract the visual features of chest X-ray images. Third, on most NLG metrics, the diagnostic report word prediction model of this invention achieves the best performance, indicating that the method proposed in this invention is indeed effective.

[0159] Although the illustrative specific embodiments of the present invention have been described above to enable those skilled in the art to understand the invention, it should be understood that the invention is not limited to the scope of the specific embodiments. For those skilled in the art, various changes are obvious as long as they are within the spirit and scope of the invention as defined and determined by the appended claims, and all inventions utilizing the concept of the present invention are protected.

Claims

1. A method for generating chest X-ray diagnostic reports based on memory-enhanced Transformer, characterized in that, Includes the following steps: S1: Collect several chest X-ray images as needed, and obtain the corresponding diagnostic report for each chest X-ray image. Compile a list of words that appear in all diagnostic reports, and record the number of words in the list. ; Set the length of the diagnostic report according to actual needs. Generate a size of [size] for each diagnostic report. Word probability matrix When the diagnostic report is in the first The position is the first in the vocabulary. When there are 100 words, let the word probability matrix be... The Middle Line number Column elements ,otherwise , , ; Randomly generate an initial word vector for each word in the vocabulary. The dimension of the word vector is . Then generate a size of [size missing] for each diagnostic report. word vector matrix Word vector matrix The Middle The row vector is the first row in the diagnostic report. Word vectors for each position of a word; The word vector matrix and word probability matrix of each chest X-ray image and the corresponding diagnostic report are used as training samples to obtain the training sample set; S2: Construct a word prediction model for the diagnostic report, including a visual feature encoder, a Transformer encoder, a memory-enhanced decoder, a fully connected layer, and a Softmax layer, where: A visual feature encoder is used to encode visual features from chest X-ray images to obtain visual features. And output to the Transformer encoder, where Indicates the number of visual features; Transformer encoders are used to determine the received visual features. Extract the hidden states to obtain each visual feature. latent vectors And output to the memory enhancement decoder; Memory enhancement decoder is used for Hidden vectors Decoding is performed to obtain the decoded features, which are then sent to the fully connected layer; the memory-enhanced decoder includes an output embedding layer. Group network blocks and memory modules, wherein: The output embedding layer is used to process the word vector matrix in the training samples. After embedding, a size of [size missing] is obtained. Embedded word vector matrix , record the The word embedding vectors of each word are Embed the word vector matrix The output is sent to the memory module, the masked multi-head attention layer, and the first feature fusion module of the first group of network blocks; The memory module is used to determine the word vector matrix. Generate memory matrix The output is sent to the first MCLN module, the second MLCN module, and the third MLCN module in each group of network blocks; where, Memory Matrix It is generated using an iterative update method, specifically as follows: 1) Let the word sequence number Initial size is Initial memory matrix Initialize word embedding vectors ,in This indicates the size of the memory slots in the memory module. ; 2) The following calculation formula is used to obtain the first... A memory matrix of words : ， in: ， ， ， Represents Adama's product. , , , Represents the learnable weights. , This represents a learnable bias term; Indicates the internal state. It's an update door; , For activation functions; This represents a multi-head attention mechanism, in which the memory matrix is ​​used. As a query for multi-head attention, then the previous word Embedded word vectors of the previous word Connection obtained , serving as the key and value for multi-head attention; 3) Determine whether If so, let (Return to step 2), otherwise proceed to step 4). 4) Convert each memory matrix Converted to length in row-major order memory vector Then memory vectors As a row vector, it forms a structure of size memory matrix ; Each group of network blocks includes a multi-head attention module for occlusion, a first feature fusion module, a first MCLN module, a multi-head attention module, a second feature fusion module, a second MLCN module, a forward feedback module, a third feature fusion module, and a third MLCN module, wherein: The occlusion multi-head attention module is used to process the received feature matrix based on the occlusion multi-head attention mechanism. After processing, a size of Feature matrix The feature matrix received by the masking multi-head attention module in the first group of network blocks is the embedded word vector matrix output by the output embedding layer. The second to the third The feature matrix received by the masked multi-head attention module in the group of network blocks is the feature matrix output by the third MLCN module in the previous group of network blocks; The first feature fusion module is used to process the feature matrix received by the occluded multi-head attention module. and the output feature matrix Adding them together, we get a value of fusion feature matrix Output to the first MCLN module; The first MCLN module is used to determine the memory matrix. For the fusion feature matrix After processing, a size of Feature matrix The output is sent to the multi-head attention module and the second feature fusion module; the processing procedure of the MCLN module is as follows: Let the fused feature matrix received by the MCLN module be... The feature matrix is ​​obtained using the following method. The Middle Row vector : Initialize learnable parameters and Then for the parameters The update is performed using the formula shown below: ， ， For parameters The update is performed using the formula shown below: ， ， in, This represents a multilayer perceptron; No. Row vector The calculation formula is as follows: ， in, It is the input fusion feature matrix The Row vectors and Represents the fusion feature matrix The mean and standard deviation of the row vectors; Multi-head attention module is used for receiving Hidden vectors and characteristic matrix , will the latent vector As the query and key for multi-head attention, the feature matrix As the value of multi-head attention, it is obtained by processing based on the multi-head attention mechanism, with a size of... Feature matrix And send it to the second feature fusion module; The second feature fusion module is used to process the feature matrix. and characteristic matrix Adding them together, we get a value of fusion feature matrix Output to the second MCLN module; The second MLCN module is used for processing based on the memory matrix. For the fusion feature matrix After processing, a size of Feature matrix The output is sent to the forward feedback module and the third feature fusion module; The forward feedback module is used for the feature matrix Perform a linear transformation to obtain a value of Feature matrix Output to the third feature fusion module; The third feature fusion module will integrate the feature matrix and characteristic matrix Adding them together, we get a value of fusion feature matrix Output to the third MLCN module; The third MLCN module is used for processing based on the memory matrix. For the fusion feature matrix After processing, a size of Feature matrix Output the first to the second... The third MLCN module in the group network block will use the feature matrix Output to the next group of network blocks, the first The third MLCN module in the group network block will use the feature matrix Output to a fully connected layer; The fully connected layer is used to perform linear transformation on the received feature matrix and output the resulting features to the Softmax layer; The Softmax layer is used to predict the probability of each word in the vocabulary appearing in each position in the diagnostic report based on the obtained features, thus obtaining a word probability matrix; S3: Take each chest X-ray image in the training sample set in step S1 as input and the corresponding word probability matrix as the expected output to train the diagnostic report word prediction model and obtain the trained diagnostic report word prediction model. S4: For the chest X-ray image to be used to generate a diagnostic report, input the chest X-ray image into the diagnostic report word prediction model trained in step S3 to obtain the word probability matrix corresponding to the chest X-ray image. S5: Based on the word probability vectors generated in step S4, the final diagnostic report text is generated using a text sequence generation algorithm.

2. The method for generating a chest X-ray diagnostic report according to claim 1, characterized in that, In step S1, the chest X-ray image and diagnostic report are preprocessed using a preset method, wherein: The preprocessing method for chest X-ray images is as follows: normalize the size of the chest X-ray image to a preset size, then perform random image cropping, randomly flip the cropped image horizontally, and finally perform image regularization. The preprocessing method for the diagnostic report is as follows: the diagnostic report is cleaned, then all punctuation marks except periods are removed, and words involving patient privacy are replaced with default characters.

3. The method for generating a chest X-ray diagnostic report according to claim 1, characterized in that, The visual feature encoder in step S3 uses a pre-trained ViT module. The pre-training method of the ViT module is as follows: select a decoder according to actual needs, use the output of the ViT module as the input of the decoder, and the decoder reconstructs the image according to the visual features of the ViT module, thereby forming an encoding and decoding model. The chest X-ray images in the training sample set are divided into several image blocks of the same size. A portion of the image blocks are randomly occluded according to a preset occlusion ratio to obtain the occluded chest X-ray images. The occluded chest X-ray image blocks are used as the input to the encoding and decoding model, and the original chest X-ray image is used as the expected output of the encoding and decoding model. The encoding and decoding model is trained to complete the training of the ViT module.

4. The method for generating a chest X-ray diagnostic report according to claim 1, characterized in that, The memory module in step S3 includes a first multilayer perceptron, a second multilayer perceptron, a connection module, a multi-head attention module, a third multilayer perceptron, a fourth multilayer perceptron, a first summation module, a second summation module, a tanh activation layer, a sigmoid activation layer, and a memory matrix operation module, wherein: The first multilayer perceptron is used to process the memory matrix of the previous word calculated by the memory matrix operation module. Assign weights ,get And send it to the first summation module; The second multilayer perceptron is used to process the memory matrix of the previous word calculated by the memory matrix operation module. Assign weights ,get And send it to the second summation module; The connection module is used to process the memory matrix of the previous word calculated by the memory matrix operation module. With embedded word vectors Perform the connection to obtain the connection matrix. And send it to the multi-head attention module; The multi-head attention module is used to integrate the memory matrix. As a query for multi-head attention, the connection matrix will be used. As keys and values ​​for multi-head attention, features are obtained through processing. And send it to the third and fourth multilayer perceptrons; The third multilayer perceptron is used to analyze features. Assign weights ,get And send it to the first summation module; The fourth multilayer perceptron is used for feature analysis. Assign weights ,get And send it to the second summation module; The first summation module is used to sum the received data. , Together with the deviation term Summing, we get And send it to the sigmoid activation layer; The second summation module is used to sum the received data. , Together with the deviation term Summing, we get And send it to the tanh activation layer; The sigmoid activation layer is used to employ an activation function. right Processing is performed to obtain the update gate. And send it to the memory matrix operation module; The tanh activation layer is used to employ an activation function. right Processing is performed to obtain the internal state. And send it to the memory matrix operation module; The memory matrix operation module is used to perform operations based on the internal state. and Update Gate For memory matrix Perform the calculations to obtain the memory matrix. The calculation formula is as follows: ， and each memory matrix Converted to length in row-major order memory vector Then memory vectors The size of the composition is memory matrix .

5. The method for generating a chest X-ray diagnostic report according to claim 1, characterized in that, In step S3, negative log-likelihood is used as the loss function for the diagnostic report word prediction model.

6. The method for generating a chest X-ray diagnostic report according to claim 1, characterized in that, In step S3, during the training of the diagnostic report word prediction model, the parameters of the ViT module are kept constant.

7. The method for generating a chest X-ray diagnostic report according to claim 1, characterized in that, The text sequence generation algorithm in step S5 uses the Beam Search algorithm.

Images