Method for automatically generating medical image report based on memory ability learning

Abstract

The application discloses a medical image report automatic generation method based on memory ability learning, uses a visual encoder pre-trained and fine-tuned on a medical data set to extract grid visual features of a medical image, reduces spatial information loss of the grid visual features through a spatial information enhancement module, and then sends the grid visual features into an encoder to obtain visual signals, sends the visual signals into each memory decoding layer in a memory decoder, learns characteristics of text mode in an image report and obtains hidden states; a language feature representation module takes a diagnosis report as input to obtain language signals; finally, the language signals, the visual signals and the hidden states are sent into an adaptive attention module to generate a report, and the adaptive attention module can automatically measure contributions of the visual signals and the language signals to generated content during report generation. The method can significantly improve the accuracy of medical image report generation.

Description

Technical Field

[0001] This invention belongs to the field of deep learning technology, specifically relating to a method for automatically generating medical image reports based on memory learning, which can be applied to computer-aided diagnosis and smart healthcare. Background Technology

[0002] Medical imaging, as a non-invasive technology, interacts with the human body through a medium to capture images of internal organs and structures, displaying them in a visual format. Radiologists use the information provided by these images to assess a patient's health and write diagnostic reports, which are crucial for disease screening and subsequent clinical treatment. As of 2018, the annual growth rate of medical imaging data in my country was approximately 30%, while the annual growth rate of the number of radiologists was only 4.1%, leading to a heavy workload and increasing pressure on radiologists. This increased workload and pressure can result in missed diagnoses and misdiagnoses. With the rapid development of artificial intelligence and the widespread application of smart healthcare, the automated generation of diagnostic reports by machines is becoming a reality. In situations of scarce medical resources and insufficient experience or high workload for radiologists, automated report generation can alleviate their workload, reduce the probability of missed diagnoses and misdiagnoses, and allow patients to quickly access information about their health. Therefore, leveraging deep learning technology to automate the generation of medical reports can significantly improve the efficiency of clinical diagnosis.

[0003] With the development of deep learning in computer vision and natural language processing, natural image description tasks, as a multimodal task connecting computer vision and natural language processing, have attracted increasingly widespread research. Natural image description tasks involve generating a single-sentence text description of the visual content of a given natural image. With the widespread application of encoder-decoder structures in image description tasks, the derivative task of automatic medical image report generation has also received increasing attention and research. Automatic medical image report generation involves generating long paragraphs of medical terminology descriptions given a medical image. Its difference from image description tasks lies in the highly patterned nature of medical reports and the lower resolution of medical images. Most existing automatic medical image report generation tasks follow the encoder-decoder framework, which first uses a convolutional neural network to encode the medical image into an intermediate vector representation, and then uses a recurrent neural network as a decoder to take the encoded vector as input to generate the final text report. To improve the accuracy of the generated report, the encoder-decoder framework often employs an attention mechanism to focus on abnormal lesion areas in the medical image and generate relevant disease descriptions. Most existing methods focus on designing novel attention mechanisms to focus on the visual features of abnormal lesion areas in medical images, without taking advantage of the highly patterned nature of diagnostic reports and the fact that all words are treated equally during the decoding process, which affects the accuracy of the generated reports. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this invention provides an automatic medical image report generation method based on memory learning. This method solves the problems of insufficient report generation accuracy caused by existing methods failing to utilize the patterned nature of diagnostic reports and treating all words equally during the decoding process.

[0005] The present invention achieves the above-mentioned technical objectives through the following technical means.

[0006] An automatic generation method for medical image reports based on memory learning:

[0007] Extracting the grid visual features from the processed medical images;

[0008] The grid visual features are calculated by the spatial information enhancement module and then sent to the encoder for encoding to obtain a visual signal;

[0009] The visual signal is sent to the memory decoder to obtain the hidden state;

[0010] The diagnostic report is processed by a language feature representation module to obtain language signals;

[0011] Construct an adaptive attention module: including a masked multi-head attention layer A and a multi-head attention layer A'. The hidden state is fed into the masked multi-head attention layer A to obtain a new hidden state. The language signal and visual signal are concatenated together and mapped into vectors respectively. These vectors, along with the vector mapped from the new hidden state at the same time step, are then fed into the multi-head attention layer A to obtain the distribution probability of the predicted word. The index corresponding to the word with the highest distribution probability is the required generated report.

[0012] A further technical solution involves the memory decoder comprising six structurally identical memory decoding layers. Each memory decoding layer includes a masked multi-head attention layer B, three memory-driven constraint normalization layers, another multi-head attention layer B, a feedforward neural network layer A, and a relational memory module. Specifically, the memory-driven constraint normalization layer involves: processing the memory matrix C at time t... t Expand to c t Then, the change in parameters γ and β is predicted through the linear layer, Δγ. t Δβ t The change and the parameter are added together to obtain the new parameter. and It is then used in conjunction with r, the mean and variance of the memory-driven constraint normalization layer, where r represents the output Y1′ of the masked multi-head attention layer and the word vector y with added positional information. t-1 The result of residual connections; the relational memory module includes a multi-head attention layer, a linear layer, and a gating module, which stores the memory matrix C at time t-1. t-1 C t-1 With y t-1 The concatenated result is fed into the multi-head attention layer, which outputs C′1. C′1 is then compared with C... t-1 The summation is fed into the linear layer and C is added. t-1 And C′1, output The C t-1 and After passing through the gating module, the memory matrix C at time t is output. t .

[0013] A further technical solution involves mapping the words to vectors, then performing scaling dot product attention calculations to obtain the attention result of the p-th attention head for the word at time step t. The attention results of all words at time steps in the p-th attention head are then concatenated to obtain the attention result of the word sequence in the p-th attention head. The attention results of the word sequences in all attention heads are then concatenated together and linearly mapped to obtain the output of the multi-head attention layer A. After linear transformation, the distribution probability of the predicted words is finally obtained.

[0014] In a further technical solution, the spatial information enhancement module is used to calculate the relative geometric features between two grids in the grid visual features.

[0015] In a further technical solution, the relative geometric features are calculated based on the grid center coordinates, relative width, and relative height, which are calculated from a pair of two-dimensional relative positions of the grid.

[0016] In a further technical solution, the language feature representation module includes a pre-trained text encoder and a masked multi-head attention layer C. The pre-trained text encoder encodes the word sequence of the diagnostic report to extract language features. The extracted language features are linearly transformed and then fed into the masked multi-head attention layer C to obtain the language signal.

[0017] A further technical solution involves extracting the grid visual features using a visual encoder. Specifically, the visual encoder involves changing the output channels of the fully connected layer in a ResNet50 pre-trained on ImageNet to 14, then fine-tuning it on the CheXpert medical dataset, and finally removing the last pooling layer and fully connected layer from the fine-tuned ResNet50 and replacing them with 512 1×1 convolutional layers.

[0018] A further technical solution describes an encoder consisting of multiple structurally identical coding layers. Each coding layer includes a multi-head attention layer C, a feedforward neural network layer B, and two normalization layers, connected by residual connections. The multi-head attention layer C maps the grid visual features into n different vectors. Relative geometric features are incorporated into the scaled dot product attention of the multi-head attention layer C, resulting in n different attention head output features. These n attention head output features are concatenated and mapped through a linear layer to obtain the output of the multi-head attention layer C. The output of the multi-head attention layer C is residually connected to the grid visual features and fed into the first normalization layer. The output of the first normalization layer is fed into the feedforward neural network layer B. The output of the feedforward neural network layer B is residually connected to the output of the first normalization layer and fed into the second normalization layer to obtain the output of the coding layer. Other coding layers take the output of the previous coding layer as input, and the output of the last coding layer is the visual signal.

[0019] A further technical solution involves obtaining the hidden state as follows: the output of the masked multi-head attention layer B is residually concatenated with the word vectors incorporating positional information, and then combined with the output c transformed by the relational memory module. t Together, they are input into the first memory-driven constraint normalization layer, outputting Y2′. Y2′ and the aforementioned visual signal are fed into the multi-head attention layer B, outputting Y3′. Y3′ and Y2′ are residually connected, and then connected to c. tTogether, they are input into the second memory-driven constraint normalization layer to obtain Y4′. Y4′ is fed into the feedforward neural network layer, which outputs Y5′. Y4′ and Y5′ are then residually connected and connected to c. t Together they are fed into the third memory-driven constraint normalization layer; other memory decoding layers take the output of the previous memory decoding layer as input, and the last memory decoding layer outputs the hidden state.

[0020] A further technical solution is that the memory matrix C output by the gating module... t It is calculated using the following formula:

[0021]

[0022] Where σ is the sigmoid function, and ⊙ is the Hardman product. Represents the Gate of Oblivion This indicates the input gate.

[0023] The beneficial effects of this invention are as follows:

[0024] (1) This invention reduces the loss of spatial information caused by flattening the visual features of the grid by adding a spatial information enhancement module.

[0025] (2) By adding a relational memory module and a memory-driven constraint normalization layer, this invention can learn the textual pattern characteristics present in image reports and obtain a vector representation of a deeper hidden state.

[0026] (3) This invention improves the accuracy of the generated report by adding an adaptive attention module to automatically measure the contribution of visual and linguistic signals to the generated content based on the hidden state during the report generation process, and treats all word generation equally during the decoding process. Attached Figure Description

[0027] Figure 1 This is an overall block diagram of the medical image report automatic generation method based on memory ability learning described in this invention;

[0028] Figure 2 This is a schematic diagram of the structure of the coding layer described in this invention;

[0029] Figure 3 This is a schematic diagram of the memory decoding layer described in this invention;

[0030] Figure 4 This is a schematic diagram of the gate control module described in this invention;

[0031] Figure 5 This is a schematic diagram of the adaptive attention module structure described in this invention. Detailed Implementation

[0032] The present invention will be further described below with reference to the accompanying drawings and specific embodiments, but the scope of protection of the present invention is not limited thereto.

[0033] like Figure 1 As shown, the present invention provides a method for automatically generating medical image reports based on memory learning, which specifically includes the following steps:

[0034] Step (1): Randomly select a medical image from the training set and scale it to 256×256 size, then randomly crop it to 224×224 size, then randomly flip the image horizontally, and randomly perform brightness and contrast transformations. Finally, convert the image into tensor form and perform standardization. The training set is selected from the IU-Xray dataset.

[0035] Step (2): First, change the output channels of the fully connected layers in the ResNet50 pre-trained on ImageNet to 14, then fine-tune it on the CheXpert medical dataset. Then, remove the last pooling layer and fully connected layer from the fine-tuned ResNet50 and replace them with 512 1×1 convolutional layers as the visual encoder. Use the visual encoder to extract the grid visual features U1′ of the medical image processed in step (1):

[0036] U1′=ResNet50(I)(1)

[0037] in: The input image is the medical image processed in step (1). These are the extracted grid visual features.

[0038] Step (3): In order to reduce the loss of spatial position information caused by flattening the mesh visual feature U1′ and feeding it into the encoder, a spatial information enhancement module is constructed. The spatial information enhancement module is used to calculate the relative geometric features between the two meshes.

[0039] Step (3.1): Calculate a pair of 2D relative positions for each grid cell in the grid visual feature U′1: in This represents the relative position coordinates of the top-left corner of grid i. This represents the relative position coordinates of the lower right corner of grid i; the calculation of the 2D relative position was implemented programmatically in PyThon software, and the specific process is existing technology.

[0040] Step (3.2): Calculate the relative center coordinates (cx) of grid i. i ,cy i Relative width w i and relative height h i :

[0041]

[0042]

[0043]

[0044] Step (3.3): Calculate the relative geometric features between meshes i and j:

[0045]

[0046] G = lW G (6)

[0047] λ g =ReLU(GW) λ (7)

[0048] in: It refers to the relative geometric relationships between the grids. and These are learnable weight parameters. It is a high-dimensional representation of l It is a relative geometric feature; the ReLU function, as a zero-point correction operation, only considers mesh features with geometric relationships.

[0049] Step (4): The grid visual features U1′ are calculated by the spatial information enhancement module and then fed into the encoder to obtain the visual signal; the encoder consists of six identical coding layers, each containing a multi-head attention layer, a feedforward neural network layer, and two normalization layers, with residual connections in between, such as Figure 2 As shown; the operation flow of coding layer 1 is as follows:

[0050] Step (4.1): The multi-head attention layer consists of 8 heads, which can acquire information from multiple angles; after changing the size of the grid visual feature U1′ to 49×512, it is fed into the multi-head attention layer. The multi-head attention layer uses 8 different linear layers to map the grid visual feature U1′ into 8 different query vectors Q. h Key vector K h Sum vector V h ;

[0051]

[0052] in: This indicates that the h-th header is linearly mapped to the query vector Q. h Learnable weight parameters, This indicates that the h-th head is linearly mapped to the key vector K. h Learnable weight parameters, This represents the linear mapping of the h-th head to the value vector V.h The learnable weight parameters, h = 1, 2…8;

[0053] Step (4.2): The relative geometric feature λ obtained in step (3.3) g By incorporating scaled dot product attention into a multi-head attention layer, eight different attention head output features are calculated:

[0054]

[0055] Where: d k Represents the key vector K h The dimension of the feature head output by each attention head. h The dimensions are 49×64;

[0056] Step (4.3): Output the feature heads from the 8 attention heads h The concatenations, after being mapped through a linear layer, yield the output Z of the multi-head attention layer:

[0057] Z=MHA(U′1,U′1,U′1)=Concat(head1,...,head8)W Z (10)

[0058] Where: MHA represents the multi-head attention layer. This means that the output of the concatenated attention head is mapped to the learnable weight parameters of the vector Z. The dimension of the output Z of the multi-head attention layer is 49×512.

[0059] Step (4.4): Perform a residual connection between the output Z of the multi-head attention layer and the grid visual feature U′1, and then feed it into the first normalization layer to obtain the output.

[0060] U′2=LayerNorm(U′1+Z) (11)

[0061] Step (4.5): Feed the output of the first normalization layer into the feedforward neural network layer to obtain...

[0062]

[0063] in: and These are learnable weight parameters;

[0064] Step (4.6): Convert the output of the feedforward neural network layer Compared with the output of the first normalized layer Perform residual connections and feed the results into the second normalization layer to obtain the output of coding layer 1.

[0065] U′4=LayerNorm(U′2+U′3) (13)

[0066] Other coding layers take the output of the previous coding layer as input and repeat the above process. The only difference is in the calculation of the scaled dot product attention in step (4.2), relative to the geometric feature λ. g It no longer participates in the calculation; the calculation process for other steps remains the same. The output of coding layer 6 is the visual signal. As shown in the following formula:

[0067] U = f en6 (f en5 (f en4 (f en3 (f en2 (U′4))))) (14

[0068] Where: f en2 (·)-f en6 (·) indicates the calculation process between steps (4.1) and (4.6).

[0069] Step (5): The visual signal output from coding layer 6 The hidden state is fed into a memory decoder, which contains six structurally identical memory decoding layers. Each memory decoding layer includes a masked multi-head attention layer, three memory-driven constraint normalization layers, a multi-head attention layer, a feedforward neural network layer, and a relational memory module, as shown below. Figure 3 As shown; at time t, the first memory decoding layer (corresponding to Figure 1 The specific implementation of the encoding layer memory decoder 1) is as follows:

[0070] Step (5.1): The predicted word sequence X = {x0, x1, ..., xt} before time t... t-1 Word embedding and positional encoding are used as inputs to the memory decoding layer. The calculation process for positional encoding is as follows:

[0071] PE(pos,2e)=sin(pos / 10000 2e / d (15)

[0072] PE(pos,2e+1)=cos(pos / 10000 2e / d (16)

[0073] y t-1 =X+PE (17)

[0074] Where: pos represents the position of the word in the sentence, e represents the dimension of the word embedding vector, and the dimension d of the word embedding vector is 512. This represents the word vector with added positional information, where M is the maximum length of the word sequence; each dimension of the positional encoding vector corresponds to a sine curve, and each odd / even position has a corresponding PE code;

[0075] Step (5.2): The calculation process of the masked multi-head attention layer is similar to that of steps (4.1)-(4.3), and will not be repeated here. The only difference is the calculation of the scaled dot product attention in step (4.2). The masked multi-head attention layer introduces a lower triangular matrix for the calculation of the scaled dot product attention in each attention head. The matrix size is 49×49, the values ​​of the upper triangle are all 0, and the values ​​of the main diagonal and below are all 1. Then the attention operation can be performed, as shown in the following formula:

[0076]

[0077] Y1′=MMHA(y t-1 ,y t-1 ,y t-1 (19)

[0078] Where: ⊙ is the Hartmann product, and MMHA represents a masked multi-head attention layer. This is the output of a masked multi-head attention layer;

[0079] Step (5.3): The output Y1′ of the masked multi-head attention layer and y t-1 Perform a residual join to obtain r, and then combine it with the output c after transformation by the relational memory module. t The first memory-driven constraint normalization layer is input together to obtain...

[0080] r = Y1′ + y t-1 (20)

[0081] Y2′=MDCNL(r,c t ) (twenty one)

[0082] Step (5.4): Input the result from the previous step, along with the visual signal output from encoding layer 6, into the multi-head attention layer to obtain...

[0083] Y3′=MHA(Y2′,U,U) (22)

[0084] Step (5.5): Perform a residual connection between the output Y3′ of the multi-head attention layer and Y2′ from step (5.3), and connect it with c. t Together, input into the second memory-driven constraint normalization layer to obtain...

[0085] Y4′=MDCNL(Y2′+Y3′,c t ) (twenty three)

[0086] Step (5.6): Feed the output Y4′ from the previous step into the feedforward neural network layer to obtain the output result.

[0087]

[0088] in: and Represents the learnable weight parameters;

[0089] Step (5.7): Perform a residual connection between the output Y4′ of the second memory-driven constrained normalization layer and the output Y5′ of the feedforward neural network layer, and connect it with c. t Together they are fed into the third memory-driven constraint normalization layer to obtain...

[0090] Y6′=MDCNL(Y′4+Y′5,c t (25)

[0091] Other memory decoding layers use the output of the previous memory decoding layer as input, repeating the process between steps (5.2) and (5.7). The last memory decoding layer (corresponding to...) Figure 1 The memory decoding layer 6 in the memory decoder outputs the hidden state. As shown in the following formula:

[0092] H W =f de6 (f de5 (f de4 (f de3 (f de2 (Y′6))))) (26)

[0093] Where: f de2 (·)-f de6 (·) indicates the calculation process between steps (5.2) and (5.7).

[0094] Step (6): Related medical images can share similar writing patterns in diagnostic reports, serving as mutual references and aiding the generation process; a relational memory module is constructed to fully utilize these characteristics, calculating the interactions between writing patterns and the generation process; the relational memory module includes a multi-head attention layer, a linear layer, and a gating module, such as Figure 3 As shown; the specific process of the relational memory module at time t is as follows:

[0095] Step (6.1): Obtain the memory matrix C at time t-1. t-1 , and Ct-1 y in step (5.1) t-1 The components are pieced together and sent into the multi-head attention layer:

[0096] C′1=MHA(C t-1 ,[C t-1 ;y t-1 ],[C t-1 ;y t-1 (27)

[0097] Step (6.2): ​​Obtain the memory matrix C at time t-1. t-1 The output C′1 of the multi-head attention layer in step (6.1) is summed and fed into the linear layer. The output of the linear layer is then added to the memory matrix C at time t-1. t-1 The output C′1 of the multi-head attention layer is obtained

[0098]

[0099] Step (6.3): Construct a gating module to mitigate the vanishing and exploding gradients caused by the relational memory module during the loop decoding process. The detailed structure of the gating module is as follows: Figure 4 As shown, in order to ensure y t-1 Can be used in C t-1 The calculation is performed by copying it into multiple rows and expanding it into a matrix Y. t-1 The memory matrix C at time t-1 t-1 Perform hyperbolic tangent activation, and apply it to Y. t-1 The activation result is sent to the input gate to obtain... Y t-1 and activated C t-1 Send to the Gate of Oblivion, and obtain Forgotten Gate and input gate Used to balance the sources from C t-1 and y t-1 The input, specifically calculated as follows:

[0100]

[0101]

[0102] Among them: W f W i Y is the forget gate and input gate. t-1 Trainable weights, U f U i It is C in the forget gate and input gate t-1 Trainable weights;

[0103] Step (6.4): Perform sigmoid function activation, and use the hyperbolic tangent function on... Calculations will be performed on C. t The result of the calculation and the activation Multiplying will activate The memory matrix C at time t-1 t-1 The two products are multiplied, and then the results are added together to obtain the final output C of the gating module. t The specific formula is as follows:

[0104]

[0105] Where: σ is the sigmoid function, C t It is the output of the entire relational memory module at time t.

[0106] Step (7): Utilize the output C of the relational memory module t The memory-driven constraint normalization layer affects two key parameters, γ and β, representing scaling and mobile learning representations in the memory decoding layer. This further integrates memory information to improve decoding performance. The specific processing flow is as follows:

[0107] Step (7.1): Obtain the memory matrix C at time t. t Expand to c t The change in γ, Δγ, is predicted after a linear layer. t To obtain new parameters The calculation formula is as follows:

[0108] Δγ t =Linear(c t (32)

[0109]

[0110] Step (7.2): Predict the change in β Δβ through the linear layer. t To obtain new parameters The calculation formula is as follows:

[0111] Δβ t =Linear(c t (34)

[0112]

[0113] Step (7.3): Add new parameters and The mean and variance of the memory-driven constrained normalization layer are then processed as follows:

[0114]

[0115] Where: μ and v are the mean and standard deviation of the memory-driven constraint normalization layer, respectively; f is the output of the first and second memory-driven constraint normalization layers in the memory decoding layer. MDCNL (r) is fed into the next module as input, and the output of the third memory-driven constraint normalization layer is used as input to the next memory decoding layer.

[0116] Step (8): Construct a language feature representation module to obtain the language features of a given sequence. The language feature representation module includes a text encoder BERT pre-trained on a large corpus and a masked multi-head attention layer; the specific process is as follows:

[0117] Step (8.1): Encode the word sequences of the diagnostic report using the pre-trained text encoder BERT to extract linguistic features:

[0118] lf=BERT(W) (37)

[0119] in: These are the language features output by the BERT model;

[0120] Step (8.2): The language features lf extracted in the previous step are linearly transformed and then fed into a masked multi-head attention layer after adding the positional encoding from step (5.1) to obtain the language signal.

[0121] S = MMHA(Linear(lf) + PE) (38)

[0122] Step (9): Construct an adaptive attention module that does not directly use the hidden state H output by the last memory decoding layer. W Instead of predicting words, the language signal S output in step (8), the visual signal U output from the last coding layer in step (4), and the hidden state H are used. W Together, they measure the contribution of visual and linguistic signals to the prediction of each word; the adaptive attention module consists of a masked multi-head attention layer, three linear layers, and one multi-head attention layer. Figure 5 The specific structure of this module is described, and the calculation process is as follows:

[0123] Hide state Feed it into a masked multi-head attention layer to obtain

[0124] H=MaskedMultiHeadAttention(H w (39)

[0125] At each time step, the corresponding hidden state and linguistic signal are selected, and the linguistic and visual signals are concatenated together and fed into the multi-head attention layer along with the hidden state to obtain the probability distribution of the predicted word. The index corresponding to the word with the highest probability distribution is the required generated report. Taking time step t as an example, the specific calculation process is as follows:

[0126] First, hide the state h at time step t. t Mapped to query vector q via a linear layer t The language signal s at time step t t The visual signal U is concatenated with the visual signal U and mapped to the key vector k through a linear layer. t Sum vector v t :

[0127]

[0128] Where: q p，t Let k be the query vector of the t-th word in the p-th attention head of the multi-head attention layer. p，t v p，t Let be the key vector and value vector of the word at time step t in the p-th attention head of the multi-head attention layer, respectively. This indicates that the p-th attention head is linearly mapped to the query vector q. p，t Learnable weight parameters, This indicates that the linear mapping in the p-th attention head is to the key vector k. p，t Learnable weight parameters, This represents the linear mapping of the p-th attention head to the value vector v. p，t The learnable weight parameters, p = 1, 2…8;

[0129] Then, the scaled dot product attention is calculated to obtain the attention result of the p-th attention head for the word at time step t. j,t :

[0130]

[0131] The attention results of all words at all time steps in the p-th attention head are concatenated to obtain the attention result head of the word sequence in the p-th attention head. p :

[0132] head p =Concate(head) p，1 head p，2 , ..., head P，M (42)

[0133] The attention results of the word sequences from all attention heads are concatenated together and then linearly mapped to obtain the output att of the multi-head attention layer:

[0134] att=Concate(head1,head2,...,head8)W O (43)

[0135] After linear transformation, the probability distribution of the predicted words is finally obtained.

[0136]

[0137] Where: b represents the bias term, W O W represents the learnable weight parameters that linearly map to att. att Represents a linear mapping to The learnable weight parameters are represented by the log-softmax function.

[0138] In this embodiment, the specific structures of the multi-head attention layer, feedforward neural network layer, normalization layer, masked multi-head attention layer, linear layer, and text encoder are all existing technologies.

[0139] The embodiments described above are preferred embodiments of the present invention, but the present invention is not limited to the above embodiments. Any obvious improvements, substitutions or modifications that can be made by those skilled in the art without departing from the essence of the present invention shall fall within the protection scope of the present invention.

Claims

1. A method for automatically generating medical image reports based on memory learning, characterized in that: Extracting the grid visual features from the processed medical images; The grid visual features are calculated by the spatial information enhancement module and then sent to the encoder for encoding to obtain a visual signal; The visual signal is sent to the memory decoder to obtain the hidden state; The diagnostic report is processed by a language feature representation module to obtain language signals; Construct an adaptive attention module: including a masked multi-head attention layer A and a multi-head attention layer A'. The hidden state is fed into the masked multi-head attention layer A to obtain a new hidden state. The language signal and the visual signal are concatenated together and mapped into vectors respectively. These vectors are then fed into the multi-head attention layer A along with the vector mapped from the new hidden state at the same time step to obtain the distribution probability of the predicted word. The index corresponding to the word with the highest distribution probability is the required generated report. Time step The process of obtaining the probability distribution is as follows: First, set the time step Hidden state Mapped to query vector q via a linear layer t Time step Language signals and visual signals The concatenated vectors are mapped to key vectors through a linear layer. Sum value vector : in: In a multi-head attention layer, the p-th attention head is the first... The query vector of each word. These are the p-th attention heads in the multi-head attention layer. The key vector and value vector of each word at each time step This indicates that the p-th attention head is linearly mapped to the query vector. Learnable weight parameters, This indicates that the linear mapping in the p-th attention head is converted into a key vector. Learnable weight parameters, This represents the linear mapping of the p-th attention head to a value vector. The learnable weight parameters, p=1, 2…8; Then, the scaled dot product attention is calculated to obtain the p-th attention head at the p-th position. Attention results for words at each time step : By concatenating the attention results of all words at all time steps in the p-th attention head, we obtain the attention result of the word sequence in the p-th attention head. : The attention results of the word sequences from all attention heads are concatenated together and then linearly mapped to obtain the output att of the multi-head attention layer: After linear transformation, the probability distribution of the predicted words is finally obtained. : Where: b represents the bias term, W O W represents the learnable weight parameters that linearly map to att. att Represents a linear mapping to The learnable weight parameters are represented by the log-softmax function.

2. The method for automatically generating medical image reports according to claim 1, characterized in that, The memory decoder comprises six structurally identical memory decoding layers. Each memory decoding layer includes a masked multi-head attention layer B, three memory-driven constraint normalization layers, another multi-head attention layer B, a feedforward neural network layer A, and a relational memory module. Specifically, the memory-driven constraint normalization layer... Memory matrix of moments Expand to become Then, the parameters are predicted through a linear layer. and Change , The change and the parameter are added together to obtain the new parameter. and and with The mean and variance of the memory-driven constraint normalization layer are calculated, where This represents the output of a masked multi-head attention layer. Word vectors with added location information The result of residual connections; the relational memory module includes a multi-head attention layer, a linear layer, and a gating module, which will... Memory matrix of moments , and The concatenated results are fed into the multi-head attention layer and output. , will the and The summation is fed into the linear layer and added. and Output The and After passing through the gating module, the output... Memory matrix of moments .

3. The method for automatically generating medical image reports according to claim 1, characterized in that, After mapping to a vector, the scaled dot product attention is calculated to obtain the p-th attention head at the p-th position. The attention results of words at each time step are concatenated together to obtain the attention results of the word sequence in the p-th attention head. The attention results of the word sequences in all attention heads are concatenated together and linearly mapped to obtain the output of the multi-head attention layer A. After linear transformation, the distribution probability of the predicted words is finally obtained.

4. The method for automatically generating medical image reports according to claim 2, characterized in that, The spatial information enhancement module is used to calculate the relative geometric features between two grids in the grid visual features.

5. The method for automatically generating medical image reports according to claim 4, characterized in that, The relative geometric features are calculated based on the grid center coordinates, relative width, and relative height, which are obtained from a pair of 2D relative positions of the grid.

6. The method for automatically generating medical image reports according to claim 1, characterized in that, The language feature representation module includes a pre-trained text encoder and a masked multi-head attention layer C. The pre-trained text encoder encodes the word sequence of the diagnostic report to extract language features. The extracted language features are linearly transformed and then fed into the masked multi-head attention layer C to obtain the language signal.

7. The method for automatically generating medical image reports according to claim 1, characterized in that, The grid visual features are extracted using a visual encoder, which specifically involves changing the output channels of the fully connected layers in a ResNet50 pre-trained on ImageNet to 14, then fine-tuning it on the CheXpert medical dataset, and finally removing the final pooling and fully connected layers from the fine-tuned ResNet50, using 512 channels. The convolutional layer is replaced.

8. The method for automatically generating medical image reports according to claim 4, characterized in that, The encoder consists of multiple identical coding layers. Each coding layer includes a multi-head attention layer C, a feedforward neural network layer B, and two normalization layers, connected by residual connections. The multi-head attention layer C maps the grid visual features into n different vectors. Relative geometric features are incorporated into the scaled dot product attention of the multi-head attention layer C to calculate n different attention head output features. These n attention head output features are concatenated together and mapped through a linear layer to obtain the output of the multi-head attention layer C. The output of the multi-head attention layer C is residually connected to the grid visual features and fed into the first normalization layer. The output of the first normalization layer is fed into the feedforward neural network layer B. The output of the feedforward neural network layer B is residually connected to the output of the first normalization layer and fed into the second normalization layer to obtain the output of the coding layer. Other coding layers take the output of the previous coding layer as input, and the output of the last coding layer is the visual signal.

9. The method for automatically generating medical image reports according to claim 8, characterized in that, The process of obtaining the hidden state is as follows: the output of the masked multi-head attention layer B is residually concatenated with the word vectors containing positional information, and then combined with the output transformed by the relational memory module. The first memory-driven constraint normalization layer is input together, and the output is... , The visual signal is sent to the multi-head attention layer B, and the output is... , and Perform residual join, and Together, input into the second memory-driven constraint normalization layer to obtain... , The data is fed into a feedforward neural network layer and outputs... ,Will and Perform residual join, and Together they are fed into the third memory-driven constraint normalization layer; other memory decoding layers take the output of the previous memory decoding layer as input, and the last memory decoding layer outputs the hidden state.

10. The method for automatically generating medical image reports according to claim 2, characterized in that, The memory matrix output by the gating module It is calculated using the following formula: in: yes function, It is the Hartmann product. Represents the Gate of Oblivion This indicates the input gate.

Images