A chest radiograph report generation method

Abstract

The application discloses a chest radiograph report generation method, and belongs to the technical field of medical information processing. In the application, if a corresponding past X image does not exist for a current X image to be processed, the current X image and corresponding clinical context knowledge thereof are input into a chest radiograph report generation network model, once forward propagation is performed, and a predicted chest radiograph report is obtained; if a corresponding past X image exists for the current X image, the current X image, corresponding clinical context knowledge thereof and the past X image are input into the chest radiograph report generation network model, once forward propagation is performed, and a predicted chest radiograph report is obtained; the model is obtained by training a chest radiograph report generation network. The predicted chest radiograph report generated by the method is more coherent in grammar and semantics, is better in higher-level phrase collocation, is better in semantic consistency, and is better in accuracy and flexibility of semantic expression.

Description

Technical Field

[0001] This invention belongs to the field of medical information processing technology, and specifically relates to a method for generating chest X-ray reports. Background Technology

[0002] Chest X-ray reports are crucial for physicians in making disease diagnoses, treatment decisions, and monitoring patient conditions. For patients with prior examinations, physicians typically need to compare current X-ray images with previous images to determine whether a lesion has newly developed, worsened, improved, or stabilized. Currently, most existing methods ignore potential positional differences between current and previous X-ray images, leading to the misinterpretation of non-pathological differences caused by positional variations as lesion changes. This results in poor semantic accuracy in the generated predicted chest X-ray reports. Therefore, this application designs a method for generating chest X-ray reports. Summary of the Invention

[0003] In view of the defects and deficiencies in the prior art, this application proposes a method for generating chest X-ray reports.

[0004] To achieve the above-mentioned objectives, the present invention adopts the following technical solution: A method for generating a chest X-ray report includes the following steps: If the current X-ray image to be processed does not have a corresponding previous X-ray image, the current X-ray image to be processed and its corresponding clinical context knowledge are input into the chest X-ray report generation network model, and the model is forward-propagated once to obtain a predicted chest X-ray report. If the current X-ray image to be processed has a corresponding previous X-ray image, the current X-ray image to be processed, its corresponding clinical context knowledge, and the previous X-ray image are input into the chest X-ray report generation network model, and the model is forward-propagated once to obtain a predicted chest X-ray report. The clinical context knowledge includes examination indications, patient history, comparative examination information, and examination technique information. The chest X-ray report generation network model is obtained by training a chest X-ray report generation network. The chest X-ray report generation network includes a visual feature extraction module, a clinical context encoding module, a spatiotemporal fusion unit, a change perception module, and a report generation module. The visual feature extraction module is connected to the spatiotemporal fusion unit and the change perception module, the clinical context encoding module is connected to the spatiotemporal fusion unit and the report generation module, the change perception module and the spatiotemporal fusion unit are both connected to the report generation module, and the report generation module outputs a predicted chest X-ray report.

[0005] Preferably, the visual feature extraction module includes a first visual encoder, a second visual encoder, a first Add layer, a second Add layer, a first visual projection layer, and a second visual projection layer. This application also constructs a shared body position embedding parameter matrix, and obtains the current body position embedding vector and previous body position embedding vectors based on the shared body position embedding parameter matrix. The first visual encoder is connected to the first Add layer, and the current body position embedding vector is input to the first Add layer. The second visual encoder is connected to the second Add layer, and the previous body position embedding vector is input to the second Add layer. The first Add layer is connected to the first visual projection layer, and the second Add layer is connected to the second visual projection layer. The first visual projection layer outputs the current visual feature V. c The second visual projection layer outputs the previous visual features V. p .

[0006] Preferably, the shared position embedding parameter matrix is ​​constructed as follows: First, establish a mapping relationship between all position category labels and numerical identifiers contained in the current X-image and previous X-images. Specifically, all position category labels contained in the current X-image and previous X-images include anteroposterior (AP) category labels, posteroanterior (PA) category labels, and lateral (Lateral) category labels. Then, set the numerical identifiers corresponding to the anteroposterior (AP) category labels to 0, the numerical identifiers corresponding to the posteroanterior (PA) category labels to 1, and the numerical identifiers corresponding to the lateral (Lateral) category labels to 2. Then, the number of position vectors is determined based on the types of position categories included in the mapping relationship. Next, a position vector is set for each position category, and all position vectors together constitute a shared position embedding parameter matrix. The number of position vectors in the shared position embedding parameter matrix is ​​the same as the number of position categories included in the mapping relationship. In this embodiment, there are three types of position categories: anterior-posterior (AP), posterior-anterior (PA), and lateral (Lateral). Each position vector has a feature dimension of 768, and the shared position embedding parameter matrix is ​​a 3×768 matrix.

[0007] Preferably, the workflow of the spatiotemporal fusion unit includes the following steps: First, determine the previous visual features V p Is it a zero tensor? If so, then the third Add layer in the spatiotemporal fusion unit combines the current temporal embedding vector and the current visual feature V. c Element-wise addition is performed to obtain the current temporal enhanced visual feature sequence; the layer normalization layer in the spatiotemporal fusion unit performs layer normalization operation on the current temporal enhanced visual feature sequence to obtain the spatiotemporal visual feature Z; the first latent variable fusion module in the spatiotemporal fusion unit combines the spatiotemporal visual feature Z with the clinical context representation Z.ctx Feature fusion is performed to obtain the spatiotemporal visual representation Z. st At this point, the spatiotemporal visual representation Z st Output to the report generation module; If not, then the fourth Add layer in the spatiotemporal fusion unit will process the previous temporal embedding vector and the previous visual feature V. p Element-wise addition is performed to obtain the previous temporal enhanced visual feature sequence; the fifth Add layer in the spatiotemporal fusion unit adds the current temporal embedding vector and the current visual feature V. c Element-wise addition is performed to obtain the current temporal enhanced visual feature sequence; the cross-temporal feature fusion module in the spatiotemporal fusion unit fuses the current temporal enhanced visual feature sequence and the previous temporal enhanced visual feature sequence to obtain the spatiotemporal visual feature Z'; the second latent variable fusion module in the spatiotemporal fusion unit fuses the spatiotemporal visual feature Z' and the clinical context representation Z. ctx Feature fusion is performed to obtain a spatiotemporal visual representation. At this point, spatiotemporal visual representation Output to the report generation module; where clinical context representation Z ctx The acquisition method is as follows: input clinical context knowledge into the clinical context coding module to obtain the clinical context representation Z. ctx .

[0008] Preferably, the cross-temporal feature fusion module includes three cross-temporal fusion blocks and a layer normalization layer connected in sequence. The three cross-temporal fusion blocks have the same structure and are respectively called the first cross-temporal fusion block, the second cross-temporal fusion block, and the third cross-temporal fusion block. Each cross-temporal fusion block includes a multi-head cross-attention layer, a first residual connection network, a second residual connection network, and a feedforward neural network. The multi-head cross-attention layer is connected to the first residual connection network, the first residual connection network is connected to the feedforward neural network, and the first residual connection network and the feedforward neural network are connected to the second residual connection network.

[0009] Preferably, the current time embedding vector and the previous time embedding vector are obtained in the following ways: First, two temporal embedding vectors are pre-defined, both with dimensions equal to the current visual feature V. c The feature dimensions are the same; the two time embedding vectors are called the current time embedding vector and the past time embedding vector, respectively. The current time embedding vector is used to represent the time information corresponding to the current X image, and the past time embedding vector is used to represent the time information corresponding to the past X images. Then, the current time embedding vector is respectively coupled with the current visual feature V. c The features at each position of each visual token are summed element by element to make the current visual feature V. cIt contains time information corresponding to the current X image; the past time embedding vector is respectively coupled with the past visual features V. p The features at corresponding positions of each visual token are added element-wise to make the previous visual feature V p It contains time information corresponding to previous X images.

[0010] Preferably, the change perception module includes a positional difference processing module, an explicit change encoding module, a comparison inference module, a change classifier, and a max pooling layer connected in sequence. The max pooling layer outputs sample-level change category scores. The first visual projection layer is connected to the positional difference processing module, the explicit change encoding module, and the comparison inference module. The second visual projection layer is connected to the positional difference processing module, and the positional difference processing module is connected to the comparison inference module. The comparison query can also be input to the comparison inference module.

[0011] Preferably, the body position difference processing module includes a query normalization layer, a key-value normalization layer, and a multi-head cross-attention layer. The query normalization layer and the key-value normalization layer are essentially layer normalization layers. The first visual projection layer is connected to the query normalization layer, and the second visual projection layer is connected to the key-value normalization layer. Both the query normalization layer and the key-value normalization layer are connected to the multi-head cross-attention layer, which is connected to the subtraction unit. The body position difference processing module also includes a Sigmoid layer. In this application, a learnable scalar parameter is input into the Sigmoid layer. The Sigmoid layer and the subtraction unit are both connected to the element-wise multiplication unit. The element-wise multiplication unit and the second visual projection layer are both connected to the Add layer. The Add layer outputs aligned with previous visual features V. pa .

[0012] Preferably, the explicit change encoding module includes a directional difference information acquisition unit and an element-by-element interactive feature construction unit; the first visual projection layer and the body position difference processing module are both connected to the directional difference information acquisition unit, and the first visual projection layer and the body position difference processing module are also both connected to the element-by-element interactive feature construction unit; the directional difference information acquisition unit is respectively connected to the absolute difference feature construction unit and the Concat layer, the element-by-element interactive feature construction unit is connected to the Concat layer, and the Concat layer is sequentially connected to the normalization layer, the first linear layer, the GELU activation layer, and the second linear layer; the second linear layer of the explicit change encoding module is connected to the comparison inference module.

[0013] Preferably, the comparison reasoning module includes three normalization layers (i.e., the current feature normalization layer, the change feature normalization layer, and the past feature normalization layer). The first visual projection layer in the visual feature extraction module is connected to the current feature normalization layer. The explicit change encoding module and the body position difference processing module in the change perception module are respectively connected to the change feature normalization layer and the past feature normalization layer. The change feature normalization layer and the past feature normalization layer are respectively connected to the change masking unit and the past masking unit. The current feature normalization layer, the change masking unit, and the past masking unit are all connected to the Concat layer. Type embedding vector input. The Add layer is connected to the Concat layer, and the Add layer is connected to the first cross-attention block and the second cross-attention block respectively. In this application, the learnable comparison query is also input to the first cross-attention block, the first cross-attention block is connected to the second cross-attention block, and the second cross-attention block outputs a change-aware representation C. The network structures of the first cross-attention block and the second cross-attention block are the same. The two inputs of the first cross-attention block are the output of the Add layer and the learnable comparison query, respectively. The two inputs of the second cross-attention block are the output of the Add layer and the output of the first cross-attention block, respectively.

[0014] Preferably, the method for obtaining the type embedding vector includes the following steps: First, let's consider the current visual features V. c Align with previous visual features V pa and explicit change characteristics V Δ Pre-set distinct type identifiers; among them, the current visual feature V c The corresponding type identifier is set to the first preset integer identifier, and the previous visual feature V is... pa The corresponding type identifier is set to the second preset integer identifier, and the explicit change feature V is set. Δ The corresponding type identifier is set to the third preset integer identifier; in this embodiment, the first preset integer identifier, the second preset integer identifier, and the third preset integer identifier are set to 0, 1, and 2, respectively; Then, based on the current visual feature V c Align with previous visual features V pa and explicit change characteristics V Δ Based on the first preset integer identifier, the second preset integer identifier, and the third preset integer identifier, construct a current visual feature type identifier subsequence, an aligned previous visual feature type identifier subsequence, and an explicit change feature type identifier subsequence, each with a sequence length of S; The reason why the sequence length of the current visual feature type identifier subsequence, the aligned previous visual feature type identifier subsequence, and the explicit change feature type identifier subsequence in this application is S is: the current visual feature V in this application cAlign with previous visual features V pa and explicit change characteristics V Δ The dimensions are all B×S×D, where B represents the batch size set during the training of the pre-trained chest X-ray report generation network. In this embodiment, B is 16, and S represents the current visual feature V. c Align with previous visual features V pa and explicit change characteristics V Δ The corresponding visual feature sequence length, where the current visual feature V c Align with previous visual features V pa and explicit change characteristics V Δ The corresponding visual feature sequences all have the same length. In this embodiment, the value of S is 1370, and D represents the current visual feature V. c Align with previous visual features V pa and explicit change characteristics V Δ The corresponding visual feature dimension; in this embodiment, the value of D is 768; In this embodiment, the sequence length of the current visual feature type identifier subsequence is S, and all elements in the current visual feature type identifier subsequence are 0; the sequence length of the aligned previous visual feature type identifier subsequence is S, and all elements in the aligned previous visual feature type identifier subsequence are 1; the sequence length of the explicit change feature type identifier subsequence is S, and all elements in the explicit change feature type identifier subsequence are 2. Then, the current visual feature type identifier subsequence, the aligned previous visual feature type identifier subsequence, and the explicit change feature type identifier subsequence are sequentially concatenated to obtain a type identifier sequence with a sequence length of 3S. Then, the first preset integer identifier, the second preset integer identifier, and the third preset integer identifier in the type identifier sequence with a sequence length of 3S are mapped to obtain a type embedding vector with a sequence length of 3S and a dimension of D.

[0015] Preferably, the report generation module includes a first normalization layer, a second normalization layer, a third normalization layer, a Concat layer, and a report decoder; the second cross-attention block of the comparison inference module is connected to the first normalization layer of the report generation module, and the second normalization layer receives the spatiotemporal visual representation Z output by the spatiotemporal fusion unit. st Or spatiotemporal visual representation The clinical context encoding module is connected to the third normalization layer; the first, second, and third normalization layers are all connected to the Concat layer, which is connected to the report decoder. The report decoder outputs the predicted chest X-ray report.

[0016] Preferably, training the chest X-ray report generation network includes the following steps: Step 1) The chest X-ray report generation network is pre-trained using the image-text global alignment pre-training method to obtain the pre-trained chest X-ray report generation network; Step 2) Train the pre-trained chest X-ray report generation network based on the MIMIC-CXR training set to obtain the chest X-ray report generation network model.

[0017] Preferably, step 2) specifically includes the following steps: When the current X-ray image in the MIMIC-CXR training set to be input into the pre-trained chest X-ray report generation network does not have a corresponding previous X-ray image, the current X-ray image and the corresponding clinical context knowledge are directly input into the pre-trained chest X-ray report generation network for forward propagation. The total loss of the pre-trained chest X-ray report generation network is calculated based on the sample-level change category score, the reference report, and the predicted chest X-ray report. Then, guided by the total loss, backpropagation is performed to update the network parameters in the pre-trained chest X-ray report generation network. This completes one epoch of training. During the training of the pre-trained chest X-ray report generation network, the batch size is set to 16. When a corresponding previous X-ray image exists in the MIMIC-CXR training set of the pre-trained chest X-ray report generation network, the current X-ray image, previous X-ray image, clinical context knowledge, corresponding reference report sample, and change category label are input into the pre-trained chest X-ray report generation network for forward propagation to calculate the total loss of the pre-trained chest X-ray report generation network. Then, guided by the total loss, backpropagation is performed to update the network parameters in the pre-trained chest X-ray report generation network. This completes the training process of the pre-trained chest X-ray report generation network for one epoch. During the training of the pre-trained chest X-ray report generation network, the batch size is set to 16. After training the pre-trained chest X-ray report generation network for 30 epochs, the training of the pre-trained chest X-ray report generation network is completed, and the chest X-ray report generation network model is obtained.

[0018] Compared with the prior art, the beneficial effects of the present invention are as follows: In this application, the body position difference processing module can effectively reduce the possible body position differences between the current X-ray image and previous X-ray images, and obtain aligned previous visual features V. pa The explicit change encoding module is based on the current visual feature V. c Align with previous visual features V pa Obtain explicit change features V that contain information on the direction of difference, the magnitude of difference, and interaction information. Δ The comparison and reasoning module is based on the current visual feature V c Align with previous visual features V paExplicit change characteristics V Δ The learnable comparison query obtains the change perception representation C used to distinguish four change categories: new occurrence, aggravation, improvement, and stability. The change classifier performs a linear classification mapping on the features output by the second cross attention block. The max pooling layer performs a max pooling operation on the features output by the change classifier to obtain the sample-level change category score. Furthermore, the configuration of the chest X-ray report generation network in this application enables the application to characterize Z based on clinical context even when there is no corresponding previous X-ray image for the current X-ray image, that is, when there is no previous X-ray image input to the second visual encoder. ctx The spatiotemporal visual representation Z corresponding to the current X image st Generate a predicted chest X-ray report.

[0019] Tests show that the predicted chest X-ray reports generated by the method described in this application are more grammatically and semantically coherent, and are better in terms of higher-level phrase collocation, semantic consistency, and the accuracy and flexibility of semantic expression. Attached Figure Description

[0020] Figure 1 This is a schematic diagram of the structure of the chest X-ray report generation network in this application; Figure 2 for Figure 1 Workflow diagram of the mid-temporal fusion unit; Figure 3 for Figure 2 A schematic diagram of the structure of the mid-span temporal feature fusion module; Figure 4 for Figure 1 A schematic diagram of the mid-body position difference processing module; Figure 5 for Figure 1 A schematic diagram of the structure of the explicit change coding module; Figure 6 for Figure 1 A schematic diagram of the structure of the comparison reasoning module; Figure 7 for Figure 6 A schematic diagram of the structure of the first cross attention block. Detailed Implementation

[0021] A method for generating a chest X-ray report includes the following steps: If the current X-ray image to be processed does not have a corresponding previous X-ray image, the current X-ray image to be processed and its corresponding clinical context knowledge are input into the chest X-ray report generation network model, and the model is forward-propagated once to obtain a predicted chest X-ray report. If the current X-ray image to be processed has a corresponding previous X-ray image, the current X-ray image to be processed, its corresponding clinical context knowledge, and the previous X-ray image are input into the chest X-ray report generation network model, and the model is forward-propagated once to obtain a predicted chest X-ray report. The clinical context knowledge includes examination indications, patient history, comparative examination information, and examination technique information. In this application, according to the time sequence of chest X-ray examinations for the same patient, the chest X-ray image corresponding to the current examination is called the current X-ray image, and the chest X-ray examination that is closest to the current examination in time before the current examination is called the previous examination, and the chest X-ray image corresponding to the previous examination is called the previous X-ray image. The chest X-ray report generation network model is obtained by training a chest X-ray report generation network. The network structure of the chest X-ray report generation network is as follows: Figure 1 As shown, the chest X-ray report generation network includes a visual feature extraction module, a clinical context coding module, a spatiotemporal fusion unit, a change perception module, and a report generation module; the visual feature extraction module is connected to the spatiotemporal fusion unit and the change perception module respectively, the clinical context coding module is connected to the spatiotemporal fusion unit and the report generation module, and the change perception module and the spatiotemporal fusion unit are both connected to the report generation module; In this application, the visual feature extraction module includes a first visual encoder, a second visual encoder, a first Add layer, a second Add layer, a first visual projection layer, and a second visual projection layer. This application also constructs a shared posture embedding parameter matrix, and obtains the current posture embedding vector and previous posture embedding vectors based on the shared posture embedding parameter matrix. The first visual encoder is connected to the first Add layer, and the current posture embedding vector is input to the first Add layer. The second visual encoder is connected to the second Add layer, and the previous posture embedding vector is input to the second Add layer. The first Add layer is connected to the first visual projection layer, and the second Add layer is connected to the second visual projection layer. The first visual projection layer outputs the current visual feature V. c The second visual projection layer outputs the previous visual features V. p ; In this application, the shared body position embedding parameter matrix is ​​constructed as follows: First, establish a mapping relationship between all position category labels and numerical identifiers contained in the current X-image and previous X-images. Specifically, all position category labels contained in the current X-image and previous X-images include anteroposterior (AP) category labels, posteroanterior (PA) category labels, and lateral (Lateral) category labels. Then, set the numerical identifiers corresponding to the anteroposterior (AP) category labels to 0, the numerical identifiers corresponding to the posteroanterior (PA) category labels to 1, and the numerical identifiers corresponding to the lateral (Lateral) category labels to 2. Then, the number of position vectors is determined according to the types of position categories included in the mapping relationship; then, a position vector is set for each position category, and all position vectors together constitute a shared position embedding parameter matrix; the number of position vectors included in the shared position embedding parameter matrix is ​​the same as the number of position categories included in the mapping relationship. In this embodiment, there are 3 types of position categories, including anterior-posterior (AP), posterior-anterior (PA), and lateral (Lateral). Each position vector has a feature dimension of 768, and the shared position embedding parameter matrix is ​​a 3×768 matrix. Before training the chest X-ray report generation network, this application first randomly initializes the shared position embedding parameter matrix.

[0022] In this application, the method for obtaining the current position embedding vector based on the shared position embedding parameter matrix is ​​as follows: obtain the position category label corresponding to the current X image, and then, based on the mapping relationship between the position category label and the digital identifier established during the construction of the shared position embedding parameter matrix, determine the digital identifier corresponding to the position category label of the current X image, and use the digital identifier as an index to extract the position vector corresponding to the position category from the shared position embedding parameter matrix, and use the extracted position vector as the current position embedding vector; The method for obtaining the previous posture embedding vector based on the shared posture embedding parameter matrix is ​​as follows: obtain the posture category label corresponding to the previous X image, and then, based on the mapping relationship between the posture category label and the digital identifier established during the construction of the shared posture embedding parameter matrix, determine the digital identifier corresponding to the posture label of the previous X image, and use the digital identifier as an index to extract the posture vector corresponding to the posture category from the same shared posture embedding parameter matrix, and use the extracted posture vector as the previous posture embedding vector.

[0023] In this application, if there is a corresponding previous X image for the current X image, the current X image is input into the first visual encoder and the previous X image is input into the second visual encoder. In this process, the first visual encoder extracts visual tokens from the current X image; the first Add layer adds the features of the current body position embedding vector and all visual tokens output by the first visual encoder to obtain all current body position enhancement visual tokens; the current body position enhancement visual tokens are input into the first visual projection layer, which performs a linear projection transformation on each current body position enhancement visual token and outputs a D-dimensional visual feature vector corresponding to each visual token, which is called the current visual feature V. c In this application, the value of D is 768. The second visual encoder extracts visual tokens from the previous X image; the second Add layer adds the features of the previous posture embedding vector and all visual tokens output by the second visual encoder to obtain all previous posture enhancement visual tokens; the previous posture enhancement visual tokens are input into the second visual projection layer, which outputs a D-dimensional visual feature vector corresponding to each visual token, and the D-dimensional visual feature vector is called the previous visual feature V. p In this application, the current visual feature V c And previous visual features V p They have the same feature dimensions.

[0024] In this application, if there is no corresponding previous X image for the current X image, then no previous X image is input to the second visual encoder; only the current X image is input to the first visual encoder. This application will input the previous visual features V... p Initialize to the feature vector output by the first visual projection layer (i.e., the current visual feature V) c The zero tensor has the same shape; in this application, the zero tensor is used as a previous visual feature V. p The data is input into the body position difference processing module and the spatiotemporal fusion unit.

[0025] In this application, the first visual encoder and the second visual encoder use the same network structure and share network parameters, so as to make the current visual feature V c And previous visual features V pLocated in the same visual feature space and having the same feature dimensions, the first and second visual encoders in this embodiment adopt the RAD-DINO visual encoder (Radiology Domain-specific DINO) disclosed in the paper "Exploring Scalable Medical Image Encoders Beyond Text Supervision". The first and second visual projection layers have the same structure and function as the token-level nonlinear projection layer disclosed in the paper "Multi-Granularity Cross-modal Alignment for Generalized Medical Visual Representation Learning".

[0026] In this application, the current visual feature V c And previous visual features V p The data is transmitted to the spatiotemporal fusion unit; specifically, the current visual feature V... c The data is respectively sent to the third Add layer and the fifth Add layer in the spatiotemporal fusion unit, and the previous visual features V p It is then transported to the fourth Add layer in the spatiotemporal fusion unit; The workflow diagram of the spatiotemporal fusion unit, as follows: Figure 2 As shown, the workflow of the spatiotemporal fusion unit includes the following steps: First, determine the previous visual features V p Is it a zero tensor? If so, then the third Add layer combines the current temporal embedding vector and the current visual feature V. c Element-wise addition is performed to obtain the current temporal enhanced visual feature sequence; a layer normalization layer performs layer normalization on the current temporal enhanced visual feature sequence to obtain the spatiotemporal visual feature Z; the first latent variable fusion module combines the spatiotemporal visual feature Z with the clinical context representation Z. ctx Feature fusion is performed to obtain the spatiotemporal visual representation Z. st At this point, the spatiotemporal visual representation Z st Output to the report generation module; If not, then the fourth Add layer will process the previous time embedding vector and the previous visual features V. p Element-wise addition is performed to obtain the previous temporal enhanced visual feature sequence; the fifth Add layer adds the current temporal embedding vector and the current visual feature V. cElement-wise addition is performed to obtain the current temporal enhanced visual feature sequence; the cross-temporal feature fusion module fuses the current temporal enhanced visual feature sequence and the previous temporal enhanced visual feature sequences to obtain the spatiotemporal visual feature Z'; the second latent variable fusion module fuses the spatiotemporal visual feature Z' and the clinical context representation Z. ctx Feature fusion is performed to obtain a spatiotemporal visual representation. At this point, spatiotemporal visual representation Output to the report generation module; where clinical context representation Z ctx The acquisition method is as follows: input clinical context knowledge into the clinical context coding module to obtain the clinical context representation Z. ctx .

[0027] The clinical context encoding module used in this application is consistent with the text encoder CXR-BERT disclosed in the paper "Making the Most of TextSemantics to Improve Biomedical Vision-Language Processing"; the latent variable fusion module is consistent with the Perceiver architecture disclosed in the paper "Perceiver: GeneralPerception with Iterative Attention". In this application, a schematic diagram of the cross-temporal feature fusion module is shown below. Figure 3As shown, the cross-temporal feature fusion module includes three cross-temporal fusion blocks and a layer normalization layer connected sequentially. The three cross-temporal fusion blocks have the same structure and are respectively called the first cross-temporal fusion block, the second cross-temporal fusion block, and the third cross-temporal fusion block. Each cross-temporal fusion block includes a multi-head cross-attention layer, a first residual connection network, a second residual connection network, and a feedforward neural network. The multi-head cross-attention layer is connected to the first residual connection network, the first residual connection network is connected to the feedforward neural network, and the first residual connection network and the feedforward neural network are connected to the second residual connection network. The multi-head cross-attention layer used in this application has eight attention heads, and each attention head performs attention calculation separately. The method of attention calculation for each attention head is the same as the multi-head attention calculation method disclosed in the paper "Attention Is All You Need". The method of residual connection for the first residual connection module and the second residual connection module is the same as the residual connection method disclosed in the paper "Attention Is All You Need". The feedforward neural network is the same as the method of residual connection disclosed in the paper "Attention Is All You Need". The feedforward network disclosed in "Is All You Need" has the same structure and function.

[0028] Functions of the cross-temporal feature fusion module: In the first cross-temporal fusion block, the multi-head cross-attention layer uses the current temporal enhanced visual feature sequence as the query feature and the previous temporal enhanced visual feature sequence as the key and value features to perform multi-head cross-attention calculation; the first residual connection network performs residual connections on the current temporal enhanced visual feature sequence and the features output by the multi-head cross-attention layer; the feedforward neural network performs nonlinear transformation on the features output by the first residual connection network; the second residual connection network performs residual connections on the features output by the feedforward neural network and the features output by the first residual connection network; the features output by the second residual connection network are the features output by the first cross-temporal fusion block. In the second cross-temporal fusion block, the multi-head cross-attention layer uses the features output by the first cross-temporal fusion block as query features and the previous temporal enhanced visual feature sequence as key and value features to perform multi-head cross-attention calculation; the first residual connection network performs residual connection on the features output by the first cross-temporal fusion block and the features output by the multi-head cross-attention layer; the feedforward neural network performs nonlinear transformation on the features output by the first residual connection network; the second residual connection network performs residual connection on the features output by the feedforward neural network and the features output by the first residual connection network; the features output by the second residual connection network are the features output by the second cross-temporal fusion block. In the third cross-temporal fusion block, the multi-head cross-attention layer uses the features output by the second cross-temporal fusion block as query features and the previous temporal enhanced visual feature sequence as key and value features to perform multi-head cross-attention calculation; the first residual connection network performs residual connection on the features output by the second cross-temporal fusion block and the features output by the multi-head cross-attention layer; the feedforward neural network performs nonlinear transformation on the features output by the first residual connection network; the second residual connection network performs residual connection on the features output by the feedforward neural network and the features output by the first residual connection network; the features output by the second residual connection network are the features output by the third cross-temporal fusion block. The features output by the third cross-temporal fusion block are fed into the layer normalization layer. The layer normalization layer performs layer normalization operation on the features output by the third cross-temporal fusion block to obtain spatiotemporal visual features.

[0029] In this application, the current time embedding vector and the previous time embedding vector are obtained in the following ways: First, two temporal embedding vectors are pre-defined, both with dimensions equal to the current visual feature V. c The feature dimensions are the same; the two time embedding vectors are called the current time embedding vector and the past time embedding vector, respectively. The current time embedding vector is used to represent the time information corresponding to the current X image, and the past time embedding vector is used to represent the time information corresponding to the past X images. Then, the current time embedding vector is respectively coupled with the current visual feature V. c The features at each position of each visual token are summed element by element to make the current visual feature V. c It contains time information corresponding to the current X image; the past time embedding vector is respectively coupled with the past visual features V. p The features at corresponding positions of each visual token are added element-wise to make the previous visual feature V p It contains time information corresponding to previous X images; In this application, both the current time embedding vector and the previous time embedding vector are used as trainable parameters. Before training the chest X-ray report generation network, the current time embedding vector and the previous time embedding vector are randomly initialized. That is, the randomly initialized current time embedding vector and the randomly initialized previous time embedding vector are used as the original current time embedding vector and the previous time embedding vector in the chest X-ray report generation network. During the training of the chest X-ray report generation network, the current time embedding vector and the previous time embedding vector are updated through backpropagation based on the total loss of the pre-trained chest X-ray report generation network.

[0030] In this application, the current visual feature V c And previous visual features V pIt is also fed to the change perception module; the change perception module includes a body position difference processing module, an explicit change encoding module, a comparison reasoning module, a change classifier and a max pooling layer connected in sequence. The max pooling layer outputs sample-level change category scores; the first visual projection layer is connected to the body position difference processing module, the explicit change encoding module and the comparison reasoning module respectively, the second visual projection layer is connected to the body position difference processing module, the body position difference processing module is connected to the comparison reasoning module, and the learning comparison query is also input to the comparison reasoning module; In this application, the network structure of the body position difference processing module is as follows: Figure 4 As shown, the body position difference processing module includes a query normalization layer, a key-value normalization layer, and a multi-head cross-attention layer. The query normalization layer and the key-value normalization layer are essentially layer normalization layers. The first visual projection layer is connected to the query normalization layer, and the second visual projection layer is connected to the key-value normalization layer. Both the query normalization layer and the key-value normalization layer are connected to the multi-head cross-attention layer, which is connected to the subtraction unit. The body position difference processing module also includes a Sigmoid layer. This application inputs learnable scalar parameters into the Sigmoid layer. The Sigmoid layer and the subtraction unit are both connected to the element-wise multiplication unit. The element-wise multiplication unit and the second visual projection layer are both connected to the Add layer. The Add layer outputs aligned with previous visual features V. pa In this application, the learnable scalar parameters are updated via backpropagation under the constraint of the total loss of the pre-trained chest radiograph report generation network; The functions of the body position difference processing module in this application are as follows: Current visual feature V c And previous visual features V p All are input into the body position difference processing module. Specifically, the current visual feature V c Input into the query normalization layer, previous visual features V p The input is fed into the key-value normalization layer. Both the query normalization layer and the key-value normalization layer use the same set of learnable scaling and bias parameters to process the front-vision feature V. c And previous visual features V p Normalization is performed to obtain the current visual feature V. c The corresponding current visual normalized features and previous visual features V p The corresponding prior visual normalized features; the multi-head cross-attention layer uses the current visual normalized features as the query features, and the prior visual normalized features as key and value features to perform cross-attention calculation, resulting in the soft-aggregated prior features V. ps ;Soft aggregation of previous features V from the output of the multi-head cross-attention layer ps Previous visual features V output by the second visual projection layer p Input subtraction unit, the subtraction unit performs soft aggregation on previous features V psAnd previous visual features V p Element-wise subtraction is performed to obtain the alignment difference feature between the soft-aggregated previous features and the previous visual features. In this application, a learnable scalar parameter α is pre-set and input into the Sigmoid layer. The Sigmoid layer performs a non-linear mapping on the learnable scalar parameter α to obtain the alignment strength coefficient γ, γ∈(0,1). The element-wise multiplication unit performs element-wise multiplication on the alignment difference feature and the alignment strength coefficient γ to obtain the alignment difference feature controlled by the alignment strength coefficient γ. The Add layer then applies the previous visual features V output by the second visual projection layer. p The alignment difference features, controlled by the alignment intensity coefficient γ, output from the element-wise multiplication unit are added together to obtain the alignment-previously-visual features V. pa In this application, the body position difference processing module can effectively reduce the possible body position differences between the current X-ray image and the previous X-ray image. In this application, the network structure of the explicit change coding module is as follows: Figure 5 As shown, the explicit change encoding module includes a directional difference information acquisition unit and an element-wise interaction feature construction unit. The first visual projection layer and the body position difference processing module are both connected to the directional difference information acquisition unit, and both are also connected to the element-wise interaction feature construction unit. The directional difference information acquisition unit is connected to the absolute difference feature construction unit and the Concat layer, respectively. The element-wise interaction feature construction unit is connected to the Concat layer, which in turn is connected to the normalization layer, the first linear layer, the GELU activation layer, and the second linear layer. The second linear layer of the explicit change encoding module is connected to the comparison inference module. The directional difference information acquisition unit is used for element-wise subtraction, and the method of element-wise subtraction is the same as the difference calculation method disclosed in the paper "Enhanced LSTM for Natural Language Inference". The element-wise interaction feature construction unit is used for element-wise multiplication, and the method of element-wise multiplication is the same as the element-wise product method disclosed in the paper "Enhanced LSTM for Natural Language Inference". The method of calculating the absolute value of the difference is the same as that of calculating the absolute value of the difference (i.e., the absolute value of their difference) disclosed in the paper "FullyConvolutional Siamese Networks for Change Detection". In this application, the explicit change coding module has the following functions: In the explicit change encoding module, the orientation difference information acquisition unit obtains the current visual feature V output by the first visual projection layer. c Alignment with previous visual features V output by the body position difference processing module pa An element-wise subtraction operation is performed to obtain a feature that contains information about the direction and magnitude of the difference between the current visual feature and the aligned previous visual feature in each feature dimension. This feature is the one that includes information about the magnitude of the directional difference. The element-wise interactive feature construction unit constructs the current visual feature V output by the first visual projection layer. c Alignment with previous visual features V output by the body position difference processing module pa Element-wise multiplication of corresponding feature positions yields features containing interactive information between the current visual features and aligned previous visual features in the corresponding feature dimensions. The absolute difference feature construction unit calculates the absolute value of the difference between the features output by the direction difference information acquisition unit, resulting in a feature that eliminates the influence of difference direction while retaining difference magnitude information; this feature contains no direction difference magnitude information. The Concat layer concatenates the features output by the direction difference information acquisition unit, the absolute difference feature construction unit, and the element-wise interaction feature construction unit, resulting in a feature that simultaneously contains interaction information, direction difference magnitude information, and no direction difference magnitude information. The normalization layer normalizes the features output by the Concat layer. The first linear layer performs linear mapping and feature transformation on the features output by the normalization layer, mapping the feature dimension from 3D to 2D. The GELU activation layer performs non-linear activation on the features output by the first linear layer. The second linear layer performs linear mapping on the features output by the GELU activation layer, mapping the feature dimension from 2D to D, outputting an explicit change feature V containing difference direction information, difference magnitude information, and interaction information. Δ .

[0031] In this application, the network structure of the comparison reasoning module is as follows: Figure 6As shown, the comparison reasoning module includes three normalization layers (i.e., the current feature normalization layer, the change feature normalization layer, and the past feature normalization layer). The first visual projection layer in the visual feature extraction module is connected to the current feature normalization layer. The explicit change encoding module and the body position difference processing module in the change perception module are respectively connected to the change feature normalization layer and the past feature normalization layer. The change feature normalization layer and the past feature normalization layer are respectively connected to the change masking unit and the past masking unit. The current feature normalization layer, the change masking unit, and the past masking unit are all connected to the Concat layer. Type embedding vector input. The Add layer is connected to the Concat layer, and the Add layer is connected to the first cross-attention block and the second cross-attention block respectively. In this application, the learnable comparison query is also input to the first cross-attention block, which is connected to the second cross-attention block. The second cross-attention block outputs a change-aware representation C. The network structures of the first and second cross-attention blocks are the same. The two inputs of the first cross-attention block are the output of the Add layer and the learnable comparison query, respectively. The two inputs of the second cross-attention block are the output of the Add layer and the output of the first cross-attention block, respectively. In this application, the network structure of the first cross-attention block is as follows: Figure 7 As shown, the first cross-attention block includes a key-value normalization layer and a query normalization layer, both of which are essentially normalization layers. The Add layer of the comparison inference module is connected to the key-value normalization layer, which is the learnable comparison query input to the query normalization layer. Both the key-value normalization layer and the query normalization layer are connected to the multi-head cross-attention layer, which is connected to the first residual connection unit. The learnable comparison query is also input to the first residual connection unit, which is sequentially connected to the feedforward normalization layer and the feedforward neural network. Both the first residual connection unit and the feedforward neural network are connected to the second residual connection unit. The feedforward normalization layer is essentially a normalization layer. The two inputs of the first cross-attention block are the output of the Add layer and the learnable comparison query. The two inputs of the second cross-attention block are the output of the Add layer and the output of the first cross-attention block. Specifically, the output of the Add layer is sent to the key-value normalization layer of the second cross-attention block, and the output of the first cross-attention block is sent to the query normalization layer and the first residual connection unit of the second cross-attention block, respectively. The residual connection method of the first residual connection unit and the second residual connection unit is the same as the residual connection method disclosed in the paper "Attention Is All You Need"; the feedforward neural network has the same structure and function as the feedforward network disclosed in the paper "Attention Is All You Need".

[0032] In this application, the method for obtaining the type embedding vector includes the following steps: First, let's consider the current visual features V. c Align with previous visual features V pa and explicit change characteristics V Δ Pre-set distinct type identifiers; among them, the current visual feature V c The corresponding type identifier is set to the first preset integer identifier, and the previous visual feature V is... pa The corresponding type identifier is set to the second preset integer identifier, and the explicit change feature V is set. Δ The corresponding type identifier is set to the third preset integer identifier; in this embodiment, the first preset integer identifier, the second preset integer identifier, and the third preset integer identifier are set to 0, 1, and 2, respectively; Then, based on the current visual feature V c Align with previous visual features V pa and explicit change characteristics V Δ Based on the first preset integer identifier, the second preset integer identifier, and the third preset integer identifier, construct a current visual feature type identifier subsequence, an aligned previous visual feature type identifier subsequence, and an explicit change feature type identifier subsequence, each with a sequence length of S; The reason why the sequence length of the current visual feature type identifier subsequence, the aligned previous visual feature type identifier subsequence, and the explicit change feature type identifier subsequence in this application is S is: the current visual feature V in this application c Align with previous visual features V pa and explicit change characteristics V Δ The dimensions are all B×S×D, where B represents the batch size set during the training of the pre-trained chest X-ray report generation network. In this embodiment, B is 16, and S represents the current visual feature V. c Align with previous visual features V pa and explicit change characteristics V Δ The corresponding visual feature sequence length, where the current visual feature V c Align with previous visual features V pa and explicit change characteristics V Δ The corresponding visual feature sequences all have the same length. In this embodiment, the value of S is 1370, and D represents the current visual feature V. c Align with previous visual features V pa and explicit change characteristics V Δ The corresponding visual feature dimension; in this embodiment, the value of D is 768; In this embodiment, the sequence length of the current visual feature type identifier subsequence is S, and all elements in the current visual feature type identifier subsequence are 0; the sequence length of the aligned previous visual feature type identifier subsequence is S, and all elements in the aligned previous visual feature type identifier subsequence are 1; the sequence length of the explicit change feature type identifier subsequence is S, and all elements in the explicit change feature type identifier subsequence are 2. Then, the current visual feature type identifier subsequence, the aligned previous visual feature type identifier subsequence, and the explicit change feature type identifier subsequence are sequentially concatenated to obtain a type identifier sequence with a sequence length of 3S. Then, the first preset integer identifier, the second preset integer identifier, and the third preset integer identifier in the type identifier sequence with a sequence length of 3S are mapped to obtain a type embedding vector with a sequence length of 3S and a dimension of D. In this application, the learnable comparison query is obtained as follows: M learnable queries are pre-set, each of which is a D-dimensional parameter vector, thus forming a learnable parameter matrix of size M×D. This M×D learnable parameter matrix is ​​the learnable comparison query, where M represents the number of learnable queries, and D is the same size as the feature dimension of the current visual feature, aligned past visual features, and explicit change features. In this embodiment, M=16 and D=768. Before training the chest X-ray report generation network, the parameters in the learnable comparison query are randomly initialized; during the training process of the chest X-ray report generation network, the learnable comparison query is iteratively updated under the constraint of the total loss L.

[0033] The functions of the comparison reasoning module in this application are as follows: The current visual feature V output by the current feature normalization layer to the first visual projection layer c Normalization is performed; the change feature normalization layer normalizes the explicit change features V output by the explicit change encoding module. Δ Normalization is performed; the previous feature normalization layer aligns with the previous visual features V output by the body position difference processing module. paNormalization processing is performed; the change masking unit masks the features output by the change feature normalization layer based on whether there is a corresponding previous X image in the current X image. This achieves the goal of preserving explicit change features when there is a corresponding previous X image in the current X image, and setting the explicit change features to zero when there is no corresponding previous X image in the current X image. Specifically, the change masking unit performs the masking process as follows: the change masking unit multiplies the features output by the change feature normalization layer with the previous mask m. If there is a corresponding previous X image in the current X image, then the previous mask m is set to zero. If the current X image does not have a corresponding previous X image, then the previous mask m=0. The previous masking unit performs masking processing on the features output by the previous feature normalization layer based on whether the current X image has a corresponding previous X image. This achieves the purpose of preserving the aligned previous visual features when the current X image has a corresponding previous X image, and setting the aligned previous visual features to zero when the current X image does not have a corresponding previous X image. Specifically, the masking process of the previous masking unit is as follows: the previous masking unit performs masking processing on the features output by the previous feature normalization layer and the previous... The masks m are multiplied. If a corresponding previous X image exists for the current X image, the previous mask m is set to 1; otherwise, the previous mask m is set to 0. The Concat layer concatenates the features output from the current feature normalization layer, the features output from the previous mask unit, and the features output from the change mask unit to obtain features that simultaneously contain current visual information, aligned previous visual information, and explicit change information. The Add layer adds the features output from the Concat layer element-wise to the type embedding vector with a sequence length of 3S and a dimension of D to obtain comparative evidence features. The first cross-attention block performs cross-attention calculation, residual connection, and nonlinear transformation on the features output by the learnable comparison query and the Add layer; the second cross-attention block performs cross-attention calculation, residual connection, and nonlinear transformation on the features output by the first cross-attention block and the Add layer, obtaining a change-aware representation C used to distinguish four change categories: new occurrence, aggravation, improvement, and stabilization; the change classifier performs linear classification mapping on the features output by the second cross-attention block; and the max-pooling layer performs max-pooling operation on the features output by the change classifier to obtain sample-level change category scores. In this application, in the first cross-attention block, the key-value normalization layer normalizes the comparison evidence features output by the Add layer, and the output of the key-value normalization layer serves as the key and value features input to the multi-head cross-attention layer; the query normalization layer normalizes the learnable comparison query input, and the output of the query normalization layer serves as the query feature input to the multi-head cross-attention layer; the multi-head cross-attention layer performs cross-attention calculation on the features output by the key-value normalization layer and the features output by the query normalization layer, and performs a weighted aggregation operation on the value features according to the attention weight between the query features and the key features, thereby realizing the utilization of learnable comparison features. The purpose of the comparison query is to read the comparison evidence features related to the learnable comparison query from the comparison evidence features; the first residual connection unit performs residual connection on the features output by the learnable comparison query and the multi-head cross-attention layer to obtain features with learnable comparison query information and comparison evidence feature information related to the learnable comparison query; the feedforward normalization layer normalizes the features output by the first residual connection unit, and the feedforward neural network performs nonlinear transformation on the features output by the feedforward normalization layer; the second residual connection unit performs residual connection on the features output by the feedforward neural network and the features output by the first residual connection unit.

[0034] In this application, the report generation module includes a first normalization layer, a second normalization layer, a third normalization layer, a Concat layer, and a report decoder; the second cross-attention block of the comparison inference module is connected to the first normalization layer of the report generation module, and the second normalization layer receives the spatiotemporal visual representation Z output by the spatiotemporal fusion unit. st Or spatiotemporal visual representation The clinical context encoding module is connected to the third normalization layer; the first, second, and third normalization layers are all connected to the Concat layer, which is connected to the report decoder. The report decoder outputs the predicted chest X-ray report. The report decoder in this application has the same structure and function as the DistilGPT2 report decoder (i.e., DistilGPT2) in the CvT2DistilGPT2 model disclosed in the paper "Improving Chest X-Ray Report Generation by Leveraging WarmStarting". The report generation module in this application has the following functions: In the report generation module, the first normalization layer normalizes the change perception representation C; the second normalization layer normalizes the spatiotemporal visual representation Z output by the spatiotemporal fusion unit. st Or spatiotemporal visual representation Normalization is performed; the third normalization layer normalizes the clinical context representation Z output by the clinical context encoding module. ctxNormalization is performed; the Concat layer concatenates the features output from the first, second, and third normalization layers to obtain features with clinical context information, spatiotemporal visual information, and change perception information; the report decoder decodes the features output from the Concat layer and predicts the next report token token by token based on the generated report tokens and the decoding prefix P (the decoding prefix P serves as the encoding end conditional feature of the report decoder); until an end token is generated or the preset maximum report length is reached, all predicted report tokens are arranged in the order of generation to form a predicted chest X-ray report. In this embodiment, the preset maximum report length is set to 100.

[0035] The configuration of the chest X-ray report generation network in this application enables it to represent the Z-axis based on clinical context even when there is no corresponding previous X-ray image for the current X-ray image, that is, when there is no previous X-ray image input to the second visual encoder. ctx The spatiotemporal visual representation Z corresponding to the current X image st Generate a predicted chest X-ray report.

[0036] In this application, the chest X-ray report generation network is trained, specifically: the chest X-ray report generation network is trained based on the MIMIC-CXR training set to obtain the chest X-ray report generation network model; the MIMIC-CXR training set refers to the training set in the MIMIC-CXR dataset, and training the chest X-ray report generation network based on the MIMIC-CXR training set includes the following steps: Step 1) The chest X-ray report generation network is pre-trained using an image-text global alignment pre-training method to obtain a pre-trained chest X-ray report generation network. The image-text global alignment pre-training method used in this application is the same as the prior-guided contrastive pre-training method disclosed in the paper "PriorRG: Prior-Guided Contrastive Pre-training and Coarse-to-Fine Decoding for Chest X-ray Report Generation". The batch size is set to 16 during the pre-training process of the chest X-ray report generation network. Step 2) Train the pre-trained chest X-ray report generation network based on the MIMIC-CXR training set to obtain the chest X-ray report generation network model. This includes the following steps: When the current X-ray image in the MIMIC-CXR training set to be input into the pre-trained chest X-ray report generation network does not have a corresponding previous X-ray image, the current X-ray image and the corresponding clinical context knowledge are directly input into the pre-trained chest X-ray report generation network for forward propagation. The total loss of the pre-trained chest X-ray report generation network is calculated based on the sample-level change category score, the reference report, and the predicted chest X-ray report. Then, guided by the total loss, backpropagation is performed to update the network parameters in the pre-trained chest X-ray report generation network. This completes one epoch of training. During the training of the pre-trained chest X-ray report generation network, the batch size is set to 16. When a corresponding previous X-ray image exists in the MIMIC-CXR training set of the pre-trained chest X-ray report generation network, the current X-ray image, previous X-ray image, clinical context knowledge, corresponding reference report sample, and change category label are input into the pre-trained chest X-ray report generation network for forward propagation to calculate the total loss of the pre-trained chest X-ray report generation network. Then, guided by the total loss, backpropagation is performed to update the network parameters in the pre-trained chest X-ray report generation network. This completes the training process of the pre-trained chest X-ray report generation network for one epoch. During the training of the pre-trained chest X-ray report generation network, the batch size is set to 16. In this embodiment, the training of the pre-trained chest X-ray report generation network is completed after iterating for 30 epochs, thus obtaining the chest X-ray report generation network model.

[0037] In this application, the total loss L of the pre-trained chest X-ray report generation network includes the report generation loss L. gen And change category prediction loss L tc The total loss L of the pre-trained chest X-ray report generation network is calculated as shown in equation (1): L=L gen +λ tc L tc (1) In equation (1), L gen To report the generated loss, L tc For predicting loss based on the category of change, λ tc In this embodiment, λ is the weighting coefficient for the loss prediction of the change category. tc The value is 0.1.

[0038] In this application, the report generation loss L is calculated based on the prediction results of each report token by the report decoder and the target report token in the corresponding reference report. gen The report generates a loss L genThe calculation method is the same as that of the cross-entropy loss published in the paper "Generating Radiology Reports via Memory-driven Transformer".

[0039] In this application, the change category prediction loss L tc The calculation method includes the following steps: First, obtain the change category tags corresponding to the reference report, which includes the following steps: The Findings and Impressions from the reference report corresponding to the current X image are concatenated, with a space inserted between them, to obtain the comparison text T. cmp And compare text T cmp Convert all uppercase letters in the text to lowercase to get the lowercase text R. cmp Then, the lowercase text R is sequentially processed using the newly generated term set, the stressed term set, the improved term set, and the stable term set. cmp Perform a match; The construction methods for the newly developed term set, the stressed term set, the improved term set, and the stable term set are as follows: Terms used to characterize new changes in the current X image relative to previous X images, including words or phrases such as "newly appeared" and "not seen on the prior" to express new changes, are called new terms. All new terms constitute the new term set. The terms used to characterize the intensified changes in the current X-image relative to previous X-images include words or phrases such as worse and interval progression used to express intensified changes. These words or phrases are called intensification terms, and all intensification terms constitute an intensification term set. Terms used to characterize improvements in the current X-image relative to previous X-images include words or phrases such as "improvement" and "interval decrease" to express improvements. These words or phrases are called improvement terms, and all improvement terms constitute the improvement term set. Terms used to characterize the state of the current X image relative to previous X images, including words or phrases such as "unchanged" and "unchanged in appearance" to express stability, are called stable terms. All stable terms constitute a set of stable terms. If the current X image has a corresponding previous X image and the lowercase text R cmp When a newly generated term set is hit, let the lowercase text R... cmp The corresponding change category label r=1; if the current X image has a corresponding previous X image and the lowercase text R cmp When the set of stressed terms is hit, let the lowercase text R cmp The corresponding change category label r=2; if the current X image has a corresponding previous X image and the lowercase text R cmp When the improved term set is hit, let the lowercase text R... cmp The corresponding change category label r=3; if the current X image has a corresponding previous X image and the lowercase text R cmp When a stable term set is hit, let the lowercase text R... cmp The corresponding change category label r=4; If the current X image has a corresponding previous X image, but the lowercase text R... cmp If no change term set is found in the newly created term set, the stressed term set, the improved term set, or the stable term set, let the lowercase text R... cmp The corresponding change category label r=5; If the current X image has a corresponding previous X image and the lowercase text R cmp When multiple sets of changed terms are matched simultaneously, the lowercase text R is determined in the following order: newly emerging term set, stressed term set, improved term set, and stable term set. cmp The corresponding unique change category label r; for example, if the current X image has a corresponding previous X image with lowercase text R. cmp When the newly generated term set, the stressed term set, and the improved term set are all matched simultaneously, then the lowercase text R... cmp The corresponding unique change category label r=1; If there is no corresponding previous X image for the current X image, let the change category label r=0; Then, determine whether the reference text has a corresponding change category label r∈{1,2,3,4}; When there is no change category label r∈{1,2,3,4}, that is, when there is no reference report corresponding to the current X image with change category label r∈{1,2,3,4}, this application directly sets the change category prediction loss L. tc =0; When there exists a change category label r∈{1,2,3,4}, i.e., when there is a reference report corresponding to the current X image with change category label r∈{1,2,3,4}, then the change category prediction loss L is performed based on the sample-level change category score and the corresponding reference report corresponding to the current X image with effective change category label r∈{1,2,3,4}. tc The calculation includes the following steps: The change category labels corresponding to the reference reports with r=1, r=2, r=3, and r=4 are all referred to as the valid change category labels r for the reference reports. b Then label each valid change category r b The current X image and its corresponding reference report are taken as a valid training sample, and all valid training samples constitute the valid training sample set Ω; the valid change category label r corresponding to each valid training sample is... b Mapping to effective change classification target t b Among them, the effective change category label r b and effective change classification target t b The relationship is shown in equation (2): t b =r b -1 (2) In equation (2), t b Indicates the effective change classification objective; r b Indicates the effective change category label; Then, based on the valid change category label r corresponding to the reference report. b The effective change classification target t corresponding to the reference report b And the sample-level change category score z obtained from predicting chest X-ray reports b Calculate the change category prediction loss L tc As shown in equation (3): (3) In equation (3), Ω represents the effective change category label r b The corresponding reference report is the set consisting of the current X image and its corresponding reference report, where |Ω| represents the number of valid training samples in set Ω, and b represents the sample index of the valid training samples in set Ω; z b t represents the sample-level change category score. b The effective change classification objective is represented by softmax(z). b ) represents the sample-level change category score z corresponding to the b-th valid training sample. b The probability of change category prediction is obtained by performing Softmax normalization; Indicates the classification objective t based on effective change.b As a category index, from The index t is selected from the predicted probabilities of the four change categories: new occurrence, aggravation, improvement, and stabilization. b The probability of predicting the category of change.

[0040] In this embodiment, the chest X-ray report generation network is trained using the AdamW optimizer. During training, the batch size is set to 16. The learning rate of the trainable parameters in the pre-trained chest X-ray report generation network that participated in optimization in step 1) and continue to participate in optimization in step 2) is set to 5×10. -6 The learning rate for the trainable parameters of the positional difference processing module, explicit change encoding module, comparative inference module, and report decoder in the pre-trained chest radiograph report generation network was set to 5 × 10⁻⁶. -5 The initial value of the alignment strength coefficient γ is set to 0.1.

[0041] In this application, network parameters that participate in optimization during the corresponding training phase and are updated by the optimizer are collectively referred to as trainable parameters. Among them, trainable scalar parameters, trainable embedding parameters, and learnable comparison queries are all trainable parameters. Network parameters that do not participate in optimization during the corresponding training phase remain unchanged during that training phase.

[0042] test: To verify the technical effectiveness of the chest X-ray report generation method (hereinafter referred to as the OURS method) described in this application compared with other chest X-ray report generation methods, this application uses the test set in the MIMIC-CXR dataset to test and compare the OURS method with six existing methods: R2Gen method (from "Generating Radiology Reports via Memory-driven Transformer"), CMN method (from "Cross-modal Memory Networks for Radiology Report Generation"), CA method (from "Contrastive Attention for Automatic Chest X-ray Report Generation"), CMM+RL method (from "Reinforced Cross-modal Alignment for Radiology Report Generation"), HERGen method (from "HERGen: Elevating Radiology Report Generation with Longitudinal Data"), and MLRG method (from "Enhanced Contrastive Learning with Multi-view Longitudinal Data for Chest X-ray Report Generation"). The test results are shown in Table 1. During testing, this application first pre-trains the six existing methods using the training set of the MIMIC-CXR dataset to obtain pre-trained network models corresponding to each of the six existing methods. Then, the test set of the MIMIC-CXR dataset is input into the pre-trained network models corresponding to the six existing methods for testing. For testing the OURS method, the test set of the MIMIC-CXR dataset is directly input into the chest X-ray report generation network model obtained in step 2) of the OURS method for testing.

[0043] Table 1. Test results of different methods on the test set of the MIMIC-CXR dataset.

[0044] In Table 1, the BLEU-1 metric is used to evaluate the matching of individual characters between the predicted chest X-ray report and the reference report; the BLEU-2 metric is used to evaluate the matching of two consecutive word pairs between the predicted chest X-ray report and the reference report; the BLEU-3 metric is used to evaluate the matching of three consecutive word pairs between the predicted chest X-ray report and the reference report; and the BLEU-4 metric is used to evaluate the matching of four consecutive word pairs between the predicted chest X-ray report and the reference report. The METEOR metric comprehensively considers the precision, recall, and word matching between the predicted chest X-ray report and the reference report. A higher METEOR value indicates that the predicted chest X-ray report is semantically closer to the reference report. The ROUGE-L metric is used to evaluate the similarity between the predicted chest X-ray report and the reference report in terms of sentence structure and content coherence. A higher ROUGE-L value indicates that the predicted chest X-ray report is closer to the reference report in terms of overall sentence structure and content coherence.

[0045] To more intuitively analyze the technical effects of the OURS method, this application selects the MLRG method, which is relatively similar to the OURS method, for in-depth comparison; As can be seen from Table 1, when conducting tests based on the test set in the MIMIC-CXR dataset: The BLEU-1 index obtained by the OURS method is 1.74% higher than that obtained by the MLRG method. This indicates that the OURS method is more accurate in matching individual characters, generates more content that is consistent with individual characters of the reference text, and improves the generation quality at the individual character level. The BLEU-2 index obtained by the OURS method is 4.36% higher than that obtained by the MLRG method, which indicates that the OURS method is more accurate in matching tuples and can generate more precise phrase combinations. The BLEU-3 index obtained by the OURS method is 8.96% higher than that obtained by the MLRG method, which indicates that the OURS method is more accurate in matching triples and the predicted chest X-ray report generated by the OURS method is more grammatically and semantically coherent. The BLEU-4 index obtained by the OURS method is 15.58% higher than that obtained by the MLRG method. This indicates that the OURS method is more accurate in matching quadruplets. The predicted chest X-ray report generated by the OURS method performs better in terms of higher-level phrase collocation and semantic consistency. The METEOR index obtained by the OURS method is 10.40% higher than that obtained by the MLRG method, which indicates that the predicted chest X-ray report generated by the OURS method is better in terms of the accuracy and flexibility of semantic expression. The ROUGE-L index obtained by the OURS method is 2.83% higher than that obtained by the MLRG method, indicating that the predicted chest X-ray report generated by the OURS method is closer to the reference text in terms of sentence structure and coherence.

Claims

1. A method for generating a chest X-ray report, characterized in that: Includes the following steps: If there is no corresponding previous X image for the current X image to be processed, the current X image and its corresponding clinical context knowledge are input into the chest X-ray report generation network model, and the forward propagation is performed once to obtain the predicted chest X-ray report. If the current X-image has a corresponding previous X-image, the current X-image, its corresponding clinical context knowledge, and the previous X-image are input into the chest X-ray report generation network model, and the forward propagation is performed once to obtain the predicted chest X-ray report. The model was obtained by training a chest X-ray report generation network. In the generative network, the visual feature extraction module is connected to the spatiotemporal fusion unit and the change perception module, the clinical context encoding module is connected to the spatiotemporal fusion unit and the report generation module, the change perception module and the spatiotemporal fusion unit are both connected to the report generation module, and the report generation module outputs the predicted chest X-ray report.

2. The method for generating a chest X-ray report according to claim 1, characterized in that: A shared posture embedding parameter matrix is ​​constructed, and current and past posture embedding vectors are obtained based on the shared posture embedding parameter matrix. The visual feature extraction module includes first and second visual encoders, first and second Add layers, and first and second visual projection layers. The first visual encoder is connected to the first Add layer, and the current posture embedding vector is input to the first Add layer. The second visual encoder is connected to the second Add layer, and the past posture embedding vector is input to the second Add layer. The first and second Add layers are respectively connected to the first and second visual projection layers. The first projection layer outputs the current visual feature V. c The second visual projection layer outputs the previous visual features V. p .

3. The method for generating a chest X-ray report according to claim 2, characterized in that: The shared position embedding parameter matrix is ​​constructed as follows: First, establish a mapping relationship between all body position category labels and numerical identifiers contained in the current X image and previous X images; Then, the number of position vectors is determined according to the types of position categories included in the mapping relationship; then, a position vector is set for each position category, and all position vectors together constitute a shared position embedding parameter matrix; the number of position vectors included in the shared position embedding parameter matrix is ​​the same as the number of position categories included in the mapping relationship.

4. The method for generating a chest X-ray report according to claim 2, characterized in that: The workflow of the spatiotemporal fusion unit includes the following steps: First, determine the previous visual features V p Is it a zero tensor? If so, then the third Add layer in the spatiotemporal fusion unit combines the current temporal embedding vector and the current visual feature V. c Element-wise addition is performed to obtain the current temporal enhanced visual feature sequence; the layer normalization layer in the unit performs layer normalization operation on the current temporal enhanced visual feature sequence to obtain spatiotemporal visual feature Z; the first latent variable fusion module in the unit combines the spatiotemporal visual feature Z and the clinical context representation Z. ctx Feature fusion is performed to obtain the spatiotemporal visual representation Z. st Spatiotemporal visual representation Z st Output to the report generation module; If not, then the fourth Add layer in the unit will process the previous time embedding vector and the previous visual feature V. p Element-wise addition is performed to obtain the previous temporal enhanced visual feature sequence; the fifth Add layer in the unit adds the current temporal embedding vector and the current visual feature V. c Element-wise addition is performed to obtain the current temporal enhanced visual feature sequence; the cross-temporal feature fusion module in the unit fuses the current temporal enhanced visual feature sequence and the previous temporal enhanced visual feature sequence to obtain the spatiotemporal visual feature Z'; the second latent variable fusion module in the unit fuses the spatiotemporal visual feature Z' and the clinical context representation Z. ctx Feature fusion is performed to obtain a spatiotemporal visual representation. Spatiotemporal visual representation Output to the report generation module; where clinical context representation Z ctx It is obtained by inputting clinical context knowledge into the clinical context coding module.

5. A method for generating a chest X-ray report according to claim 4, characterized in that: The current time embedding vector and previous time embedding vectors are obtained in the following ways: First, two parameters are pre-set that are both related to the current visual feature V. c Temporal embedding vectors with the same feature dimension are called the current and previous temporal embedding vectors, respectively. Then, the current time embedding vector is respectively coupled with the current visual feature V. c The features at each position of each visual token are summed element by element to make the current visual feature V. c It contains time information corresponding to the current X image; the past time embedding vector is respectively coupled with the past visual features V. p The features at corresponding positions of each visual token are added element-wise to make the previous visual feature V p It contains time information corresponding to previous X images.

6. The method for generating a chest X-ray report according to claim 1, characterized in that: The change perception module includes a positional difference processing module, an explicit change encoding module, a comparison reasoning module, a change classifier, and a max pooling layer connected in sequence. The max pooling layer outputs sample-level change category scores. The first visual projection layer is connected to the positional difference processing module, the explicit change encoding module, and the comparison reasoning module. The second visual projection layer is connected to the positional difference processing module, and the positional difference processing module is connected to the comparison reasoning module. The module can learn comparison queries and inputs to the comparison reasoning module.

7. A method for generating a chest X-ray report according to claim 6, characterized in that: The body position difference processing module includes a query layer, a key-value normalization layer, and a multi-head cross-attention layer. The first and second visual projection layers are connected to the query and key-value normalization layers, respectively. Both the query and key-value normalization layers are connected to the multi-head cross-attention layer, which in turn is connected to the subtraction unit. The module also includes a Sigmoid layer, which can learn scalar parameter inputs. The Sigmoid layer and the subtraction unit are both connected to the element-wise multiplication unit. The element-wise multiplication unit and the second visual projection layer are both connected to the Add layer. The Add layer outputs aligned with previous visual features V. pa .

8. A method for generating a chest X-ray report according to claim 6, characterized in that: The explicit change encoding module includes a direction difference information acquisition unit and an element-by-element interactive feature construction unit; the first visual projection layer and the body position difference processing module are both connected to the direction difference information acquisition unit, and the first visual projection layer and the body position difference processing module are both connected to the element-by-element interactive feature construction unit. The directional difference information acquisition unit is connected to the absolute difference feature construction unit and the Concat layer respectively. The element-wise interactive feature construction unit is connected to the Concat layer. The Concat layer is connected to the normalization layer, the first linear layer, the GELU activation layer, and the second linear layer in sequence.

9. A method for generating a chest X-ray report according to claim 6, characterized in that: The comparison reasoning module includes current, change, and past feature normalization layers. A first visual projection layer is connected to the current feature normalization layer. An explicit change encoding module and a body position difference processing module are connected to the change and past feature normalization layers, respectively. The change and past feature normalization layers are connected to the change and past masking units, respectively. The current feature normalization layer, change, and past masking units are all connected to the Concat layer. A type embedding vector is input to the Add layer. The Concat layer is connected to the Add layer. The Add layer is connected to the first cross-attention block and the second cross-attention block, respectively. A learnable comparison query is also input to the first cross-attention block. The first cross-attention block is connected to the second cross-attention block, and the second cross-attention block outputs a change-perceived representation C.

10. A method for generating a chest X-ray report according to claim 9, characterized in that: The method for obtaining the type embedding vector includes the following steps: First, let's consider the current visual features V. c Align with previous visual features V pa and explicit change characteristics V Δ Pre-set distinct first, second, and third preset integer identifiers; Then, construct a current visual feature type identifier subsequence with a sequence length of S, an aligned previous visual feature type identifier subsequence, and an explicit change feature type identifier subsequence; Then, the current visual feature type identifier subsequence, the aligned previous visual feature type identifier subsequence, and the explicit change feature type identifier subsequence are sequentially concatenated to obtain a type identifier sequence with a sequence length of 3S. Then, the first, second, and third preset integer identifiers in the type identifier sequence with a sequence length of 3S are mapped to obtain a type embedding vector with a sequence length of 3S and a dimension of D.

Images