Radiology report generation method and system based on cross-modal fusion and memory supervision

Abstract

The application discloses a radiology report generation method and system based on cross-modal fusion and memory supervision, and belongs to the technical field of natural language processing. The cross-modal fusion module is used to promote the information interaction between medical image visual features and medical report text features, and solve the problem of insufficient multi-modal feature interaction. The memory supervision module is used to merge text information into the normalization layer of the decoder, and provide targeted guidance and reference for radiology report generation. The multi-class contrast loss is used to enhance the learning of multi-modal features. The application solves the problem of data deviation between vision and text in the current radiology report generation, and the problem of difficulty in generating long text. Not only the accuracy of the performance on the task is improved, but also the radiology report generation task with similar problems can be used as a reference.

Description

Technical Field

[0001] This invention belongs to the technical field of natural language processing, and relates to a radiology report generation technology, particularly a radiology report generation method and system based on cross-modal fusion and memory supervision. Background Technology

[0002] With the development of science and technology and medical standards, hospitals generate a large amount of medical imaging data every day, which increases the burden on doctors in medical diagnosis and report writing. The emergence, development and application of computer-aided diagnostic systems and automatic medical imaging report generation systems can reduce the time doctors spend reading images and writing reports to a certain extent, and at the same time, can avoid missed diagnoses and misdiagnoses to a certain extent.

[0003] Radiographic images, such as X-rays and MRI, are widely used in medicine to support disease diagnosis. However, traditional clinical practice is time-consuming and labor-intensive because it requires medical experts, such as radiologists, to carefully analyze the images and then generate a medical report, which typically takes more than five minutes. This process is also prone to errors due to subjective factors such as fatigue and distraction.

[0004] Traditionally, radiology report writing was a manual task performed by experienced physicians. However, with advancements in computer hardware and deep learning theory, deep neural network models have been widely applied to automated radiology report generation. Radiology report generation remains challenging, primarily in the following two aspects:

[0005] (1) Visual data bias: In the dataset, abnormal areas in medical images account for only a small portion of radiographic images, while normal areas make up the majority of the data. This imbalanced visual distribution will hinder the model's accurate capture of the features of rare and diverse abnormal areas.

[0006] (2) Text data bias: Radiologists tend to describe all symptoms found in medical images, causing descriptions of normal areas to dominate the entire report. Furthermore, many similar sentences are used to describe the same normal areas. Due to this imbalanced text distribution, training with this dataset results in the generation of normal sentences being dominant, making the model unable to describe specific key anomalies.

[0007] (3) Long text generation problem: Most previous radiology report generation tasks were based on general image description models. However, image description models are good at generating short sentences. Radiology report generation tasks need to generate some accurate long sentences to accurately describe the symptoms in medical images. Summary of the Invention

[0008] In view of this, the present invention provides a method and system for generating radiology reports based on cross-modal fusion and memory supervision, for generating corresponding medical reports for medical images. This invention addresses three challenges faced by most current models in the radiology report generation process: visual data bias, text data bias, and the problem of generating long texts.

[0009] The technical solution adopted in this invention is as follows:

[0010] On one hand, the present invention provides a method for generating radiological reports based on cross-modal fusion and memory supervision, the method comprising:

[0011] S1: Obtain the currently generated medical image, input the medical image into the medical image extractor, and extract the medical image features;

[0012] S2: Input the word embedding features of the medical report and the medical image features into the cross-modal fusion module to obtain text feature representation and visual feature representation; the cross-modal fusion module generates pseudo-classes for each image-text pair and initializes the cross-modal fusion matrix; the cross-modal fusion matrix serves as a medium connecting visual and text features to obtain visual feature representation and text feature representation;

[0013] S3: Input the word embedding features of the medical report into the memory supervision matrix generator, and use the output of the memory supervision matrix generator as the input to the normalization layer in the decoder; the word embedding of the medical report is the text features of the previous time step; the output of the memory supervision matrix generator is the memory supervision matrix of the current time step;

[0014] S4: The data after passing through a fully connected layer with the extracted medical image features and the visual feature representation obtained from the cross-modal fusion module is used as the input to the encoder to obtain the hidden state; the data after passing through a fully connected layer with the word embedding of the medical report and the text feature representation obtained from the cross-modal fusion module, as well as the hidden state obtained from the encoder, are used as the input to the decoder to obtain the radiology report prediction result at the current time step, and the final prediction result is optimized using multi-class contrastive loss.

[0015] Furthermore, the medical image extractor is a pre-trained convolutional neural network ResNet101.

[0016] Furthermore, the decoder layer includes: Masked MHA, multi-head attention layer, feedforward layer, fully connected linear layer and Softmax layer, wherein a guidance normalization layer is included between Masked MHA and multi-head attention layer, between multi-head attention layer and feedforward layer, and between feedforward layer and fully connected linear layer; the guidance normalization layer takes the memory supervision matrix of the current time step output by the memory supervision matrix generator as input.

[0017] Further, the cross-modal fusion matrix is ​​initialized, including:

[0018] For each historical radiology report sample, extract global visual and textual features;

[0019] By combining the category labels of each radiology report sample, the text features and visual features are concatenated to obtain the feature set for each category;

[0020] The K-Means algorithm is used to divide each feature set into P groups, and the average value of the features in each group is used to initialize the cross-modal fusion matrix.

[0021] Furthermore, the cross-modal fusion matrix serves as a medium connecting visual and textual features, yielding visual feature representations and textual feature representations, including:

[0022] The similarity between the medical image features, the corresponding report text features, and the cross-modal fusion vectors in the same category is estimated using an initialized cross-modal fusion matrix; each vector in the cross-modal fusion matrix is ​​called a cross-modal fusion vector.

[0023] Under each category, select the K most similar cross-modal fusion vectors, use their similarity to visual features and text features as weights, and perform a weighted sum on the K cross-modal fusion vectors to finally obtain new representations of visual features and text features.

[0024] Furthermore, the word embedding features of the medical report are input into the memory supervision matrix generator, which outputs the memory supervision matrix at the current time step, including:

[0025] In the memory supervision matrix generator, based on the multilayer perceptron, the memory supervision matrix of the previous time step is used as the query;

[0026] The memory supervision matrix of the previous time step and the text features of the previous time step are concatenated as key and value, and multi-head attention is performed, resulting in Z;

[0027] Residual connections, forgetting mechanisms, and input gates are introduced to prevent gradient vanishing and exploding, ultimately yielding the memory supervision matrix for the current time step.

[0028] In another aspect, the present invention also discloses a system for generating radiological reports based on cross-modal fusion and memory supervision, comprising:

[0029] The medical image feature extractor module takes a medical image as input and extracts its features.

[0030] The cross-modal fusion module takes the word embedding features of the medical report and the medical image features as input, generates pseudo-classes for each image-text pair, and initializes the cross-modal fusion matrix; the cross-modal fusion matrix serves as a medium connecting visual and text features to obtain visual feature representations and text feature representations;

[0031] The memory matrix supervision matrix generator module takes the word embedding features of the medical report as input, uses the memory matrix to record entity information in the text report, and outputs the memory supervision matrix generated at the current time step; the word embeddings of the medical report are the text features of the previous time step.

[0032] The encoder module, comprising multiple encoder layers, takes the medical image features extracted from the medical image features and the visual feature representation obtained from the cross-modal fusion module as input after passing through the fully connected layer to obtain the hidden state;

[0033] The decoder module, comprising multiple decoder layers, takes the data after the fully connected layer of the word embedding of the medical report and the text feature representation obtained by the cross-modal fusion module, as well as the hidden state obtained by the encoder module, as input to obtain the radiology report prediction result at the current time step;

[0034] The multi-class contrastive loss module uses multi-class contrastive loss to optimize the final prediction results.

[0035] Furthermore, the decoder layer includes: Masked MHA, multi-head attention layer, feedforward layer, fully connected linear layer and Softmax layer, wherein a guidance normalization layer is included between Masked MHA and multi-head attention layer, between multi-head attention layer and feedforward layer, and between feedforward layer and fully connected linear layer; the guidance normalization layer takes the memory supervision matrix of the current time step output by the memory supervision matrix generator as input.

[0036] Furthermore, the cross-modal fusion module includes:

[0037] For each historical radiology report sample, extract global visual and textual features;

[0038] By combining the category labels of each radiology report sample, the text features and visual features are concatenated to obtain the feature set for each category;

[0039] The K-Means algorithm is used to divide each feature set into P groups, and the average value of the features in each group is used to initialize the cross-modal fusion matrix.

[0040] The similarity between the medical image features, the corresponding report text features, and the cross-modal fusion vectors in the same category is estimated using an initialized cross-modal fusion matrix; each vector in the cross-modal fusion matrix is ​​called a cross-modal fusion vector.

[0041] Under each category, select the K most similar cross-modal fusion vectors, use their similarity to visual features and text features as weights, and perform a weighted sum on the K cross-modal fusion vectors to finally obtain new representations of visual features and text features.

[0042] Furthermore, in the memory supervision matrix generator, based on a multilayer perceptron, the memory supervision matrix of the previous time step is used as the query; the memory supervision matrix of the previous time step and the text features of the previous time step are concatenated as the key and value, and multi-head attention is performed to obtain Z; residual connections, forgetting and input gates are introduced to prevent gradient vanishing and exploding, and finally the memory supervision matrix of the current time step is obtained.

[0043] Compared with the prior art, the beneficial effects of the present invention are:

[0044] 1) This invention introduces a cross-modal fusion module to enhance the interaction between visual features and text features, alleviate the bias between visual and text data, reduce the isolation between visual features and text features, and alleviate the problem of bias between visual and text data.

[0045] 2) This invention designs a memory-supervised matrix generator module and a guidance normalization layer module to record entity information in text features, thereby promoting the generation of long texts.

[0046] 3) This invention enhances the learning of cross-modal features by introducing a multi-class contrast loss module to balance the contributions of positive and negative samples. Attached Figure Description

[0047] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0048] Figure 1 This is a flowchart illustrating a radiology report generation method based on cross-modal fusion and memory supervision in an embodiment of the present invention.

[0049] Figure 2 This is a schematic diagram of the CFMS neural network in an embodiment of the present invention. Detailed Implementation

[0050] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0051] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0052] This invention extracts medical image features from a medical image extractor, inputs medical report word embedding features and medical image features into a cross-modal fusion module to obtain new representations of visual and textual features; then, the report word embedding features are input into a memory-supervised matrix generator to record valuable text entity information, and input into the guided normalization layer of the decoder; the new representations of visual features are used as inputs to the encoder, and the new representations of textual features are used as inputs to the decoder, and multi-class contrastive loss is used to ultimately improve the performance of radiology report generation.

[0053] like Figure 1 As shown in the figure, an embodiment of the present invention provides a radiology report generation method based on cross-modal fusion and memory supervision, comprising the following steps:

[0054] S1: Input the medical image into the medical image extractor to extract the medical image features;

[0055] In practical implementation, the medical image feature extractor can use a pre-trained convolutional neural network ResNet101, represented as:

[0056] {x1,x2,..xi..,xM}=fv(I)#(1)

[0057] Where fv(·) is a medical image extractor, and in this embodiment of the invention, a pre-trained ResNet 101 is used, xi represents the i-th medical image feature, and I represents the medical image.

[0058] S2: Input the word embedding features of the medical report and the medical image features in S1 into the cross-modal fusion module to obtain text feature representation and visual feature representation.

[0059] In practice, word embedding features in medical reports can be obtained using PyTorch's built-in function nn.Embedding.

[0060] In practice, the cross-modal fusion module generates pseudo-classes for each image-text pair and initializes the cross-modal fusion matrix. The steps for initializing the cross-modal fusion matrix include:

[0061] S21. For each historical medical radiology report sample, extract global visual and textual features.

[0062] In practice, ResNet 101 and BERT can be used to extract global visual and textual features.

[0063] S22. Extract features from each sample and combine them with the category label of each sample to connect the text features and visual features to obtain the feature set of each category.

[0064] S23. Finally, the K-Means algorithm is used to divide each feature set into P groups, and the average value of the features in each group is used to initialize the cross-modal fusion matrix. Each vector in the cross-modal fusion matrix is ​​named a cross-modal fusion vector.

[0065] The specific process can be formalized as follows:

[0066] R={y1,y2,..yi..,yN}#(2)

[0067] {t1,t2,..ti..,tS}=fc(R)#(3)

[0068]

[0069]

[0070]

[0071]

[0072]

[0073] Where R is the feature of the report corresponding to the image, yi is the feature of the i-th word, and fc(R) is the category corresponding to the report. and These are the feature sets corresponding to the k-th category, representing visual features and text features, respectively. fKM is the K-Means algorithm. is the i-th group of the k-th category returned by fKM, Ou is the visual and text feature set of sample u concatenated, Rk is the feature set of the k-th category. CFM(k,i) is the i-th vector in the cross-modal feature set of the k-th category.

[0074] The cross-modal fusion matrix serves as a medium connecting visual and textual features, yielding both visual and textual feature representations. Specific steps include:

[0075] S24. For a given image, estimate the similarity between visual features and corresponding report text features and cross-modal fusion vectors in the same category using the initialized cross-modal fusion matrix.

[0076] S25. Select the K most similar cross-modal fusion vectors under each category, and use their similarity with visual features and text features as weights to perform a weighted sum on the K cross-modal fusion vectors, finally obtaining the new visual feature and text feature representations.

[0077] The entire process can be represented as:

[0078]

[0079]

[0080]

[0081]

[0082]

[0083] LV = FCN(Concat(x,r) V ),LT=FCN(Concat(y,r T ))#(14)

[0084] Where xi is the i-th vector in the medical image feature sequence obtained in S1, yi is the word embedding feature of the i-th word in the report, and pi is the i-th cross-modal fusion vector. In this embodiment of the invention, xi, yi, and pi are projected onto the same dimension to facilitate subsequent calculations. and There are two learnable parameters: C is the dimension of xi, D is the dimension of pi, and d is the dimension after projection. It uses cosine similarity to calculate the similarity between cross-modal fused vectors and visual features. It uses cosine similarity to calculate the similarity between cross-modal fusion vectors and text features. Yes The result of normalization Yes The result of normalization. This is the j-th cross-modal fusion vector in the most similar cross-modal feature set for the i-th image-text pair. A weighted sum of these transformed cross-modal fusion vectors yields the responses of the image and text features. and Finally, the original image features x and the response r of the image features are combined. V The concatenation is then fed into a fully connected layer (FCN) to obtain a new visual feature representation (LV), as well as a response (r) that combines word embedding features (y) and text features. T The features are then connected and fed into a fully connected layer (FCN) to obtain a new representation of the text features (LT).

[0085] S3: Input the word embedding features from the input of S2 into the memory supervision matrix generator, output the memory supervision matrix at the current time step, and use the output as the input to guide the normalization layer in the decoder.

[0086] The memory-supervised matrix generator, specifically, uses the memory-supervised matrix from the previous time step as the query, and concatenates the memory-supervised matrix from the previous time step with the text features from the previous time step as the key and value. Multi-head attention is applied to the query, key, and value, and the result is represented by Z. Residual connections, forgetting mechanisms, and input gates are introduced to prevent gradient vanishing and exploding, finally yielding the memory-supervised matrix for the current time step.

[0087] The entire process can be represented as:

[0088] Q=MSMt-1×Wq, K=[MSMt-1; yt-1]×Wk, V=[MSMt-1; yt-1]×Wv#(15)

[0089]

[0090]

[0091]

[0092] Where Q is the query, K is the key, and V is the value, and the result of multi-head attention computation of Q, K, and V is represented by Z. This is the result after using residual connections. MSMt is the memory supervision matrix generated at the current time step. MLP refers to Multilayer Perceptron, and th refers to the tanh function. Uf and Ui are the learnable parameters for MSMt-1.

[0093] The memory supervision matrix at the current time step is input into the decoder's guidance normalization layer module to provide textual guidance for the normalization operation, further improving the decoder's decoding capability. The specific process is as follows:

[0094] Δγt=MLP(msmt),γt*=γ+Δγt#(19)

[0095] Δβt=MLP(msmt),βt*=β+Δβt#(20)

[0096]

[0097] Here, MSMt is the memory supervision matrix generated at the current time step. To input the memory supervision matrix into the normalization layer, each row of MSMt is concatenated and expanded into a vector msmt. fGLN(r) is the output of the normalization layer, where μ and v represent the mean and standard deviation of r, and r is the result of the previous module.

[0098] S4: The new visual feature representation from the cross-modal fusion module in S2 is used as the input to the encoder, and the new text feature representation is used as the input to the decoder to obtain the radiological report prediction result; a multi-class contrastive loss is then used to optimize the final prediction result. The encoder refers to the encoder structure in the Transformer, which is composed of multiple identical encoder layers stacked together. The encoder layer consists of multi-head attention, residual connections, and feedforward layers stacked together. The decoder refers to a modified version of the decoder in the Transformer, composed of multiple identical decoder layers stacked together. The decoder layer consists of multi-head attention, feedforward layers, guided normalization layers, fully connected linear layers, and softmax layers stacked together.

[0099] In this embodiment of the invention, at least one sample of the same class (excluding class 0) is considered positive. If two samples do not belong to the same class, they constitute a negative pair. The input and output process of the encoder and decoder can be represented as follows:

[0100]

[0101]

[0102] in This is a new representation of visual features, which serves as input to the encoder to obtain the hidden state hi. The hidden state and the new representation of text features... As input to the decoder, the prediction result pt at time step t is obtained. The formula for the multi-class contrastive loss is as follows:

[0103]

[0104] hd=|sum(ti-tj)|, ht=|sum(ti+tj)|#(25)

[0105]

[0106] in It is a multi-class loss related to visual features. It is a multi-class loss related to text features, L ce It is the cross-entropy loss, L fin This is the final loss. A sample is considered positive if it has at least one sample of the same class (excluding class 0). If two samples do not belong to the same class, they are considered negative pairs. The final loss L is obtained by weighting and summing the multi-class contrastive loss and the cross-entropy loss. fin B represents the number of training samples in the batch. It is a dot product function. make sure and There must be at least one class in the same category (excluding 0). fa(.) and fs(.) are the mean function and cosine similarity function, respectively.

[0107] A constant threshold α ensures that only negative pairs with similarity greater than the boundary value contribute. Furthermore, Θ is used as a controller to tolerate a certain range of differences between positively similar pairs, rather than forcing them to be identical. Finally, the multi-class contrastive loss and cross-entropy loss are weighted and summed to obtain the final loss function.

[0108] In another embodiment, the present invention also discloses a system for generating radiological reports based on cross-modal fusion and memory supervision, comprising:

[0109] The medical image feature extractor module takes a medical image as input and extracts its features.

[0110] The cross-modal fusion module takes the word embedding features of the medical report and the medical image features as input, generates pseudo-classes for each image-text pair, and initializes the cross-modal fusion matrix; the cross-modal fusion matrix serves as a medium connecting visual and text features to obtain visual feature representations and text feature representations;

[0111] The cross-modal fusion module specifically includes:

[0112] For each historical radiology report sample, extract global visual and textual features;

[0113] By combining the category labels of each radiology report sample, the text features and visual features are concatenated to obtain the feature set for each category;

[0114] The K-Means algorithm is used to divide each feature set into P groups, and the average value of the features in each group is used to initialize the cross-modal fusion matrix.

[0115] The similarity between the medical image features, the corresponding report text features, and the cross-modal fusion vectors in the same category is estimated using an initialized cross-modal fusion matrix; each vector in the cross-modal fusion matrix is ​​called a cross-modal fusion vector.

[0116] Under each category, select the K most similar cross-modal fusion vectors, use their similarity to visual features and text features as weights, and perform a weighted sum on the K cross-modal fusion vectors to finally obtain new representations of visual features and text features.

[0117] The memory matrix supervision matrix generator module takes the word embedding features of the medical report as input, uses the memory matrix to record entity information in the text report, and outputs the memory supervision matrix generated at the current time step; the word embeddings of the medical report are the text features of the previous time step.

[0118] In the memory supervision matrix generator, based on a multilayer perceptron, the memory supervision matrix of the previous time step is used as the query; the memory supervision matrix of the previous time step and the text features of the previous time step are concatenated as the key and value, and multi-head attention is performed to obtain the result Z; residual connections, forgetting and input gates are introduced to prevent gradient vanishing and exploding, and finally the memory supervision matrix of the current time step is obtained.

[0119] The encoder module, comprising multiple encoder layers, takes the medical image features extracted from the medical image features and the visual feature representation obtained from the cross-modal fusion module as input after passing through the fully connected layer to obtain the hidden state;

[0120] The decoder module, comprising multiple decoder layers, takes the data after the fully connected layer of the word embedding of the medical report and the text feature representation obtained by the cross-modal fusion module, as well as the hidden state obtained by the encoder module, as input to obtain the radiology report prediction result at the current time step;

[0121] The decoder layer includes: Masked MHA, multi-head attention layer, feedforward layer, fully connected linear layer and Softmax layer. Each of the Masked MHA and multi-head attention layers, the multi-head attention layer and the feedforward layer, and the feedforward layer and the fully connected linear layer includes a guidance normalization layer. The guidance normalization layer takes the memory supervision matrix of the current time step output by the memory supervision matrix generator as input.

[0122] The multi-class contrastive loss module uses multi-class contrastive loss to optimize the final prediction results.

[0123] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for generating radiological reports based on cross-modal fusion and memory-supervised radiology, characterized in that, The method includes: S1: Obtain the currently generated medical image, input the medical image into the medical image extractor, and extract the medical image features; S2: Input the word embedding features of the medical report and the medical image features into the cross-modal fusion module to obtain text feature representation and visual feature representation; the cross-modal fusion module generates pseudo-classes for each image-text pair and initializes the cross-modal fusion matrix; the cross-modal fusion matrix serves as a medium connecting visual and text features to obtain visual feature representation and text feature representation; wherein, initializing the cross-modal fusion matrix includes: for each historical radiology report sample, extracting global visual and text features; combining the category label of each radiology report sample, concatenating the text features and visual features to obtain the feature set of each category; using the K-Means algorithm to divide each feature set into P groups, and using the average value of the features in each group to initialize the cross-modal fusion matrix; S3: The word embedding features of the medical report are input into the memory-supervised matrix generator, and the output of the memory-supervised matrix generator is used as the input to the normalization layer in the decoder; the word embeddings of the medical report are the text features of the previous time step; the output of the memory-supervised matrix generator is the memory-supervised matrix of the current time step; the decoder includes multiple decoder layers, including: Masked MHA, multi-head attention layer, feedforward layer, fully connected linear layer, and Softmax layer, wherein Masked... A guidance normalization layer is included between MHA and the multi-head attention layer, between the multi-head attention layer and the feedforward layer, and between the feedforward layer and the fully connected linear layer. The guidance normalization layer takes the memory supervision matrix of the current time step output by the memory supervision matrix generator as input. Specifically, the word embedding features of the medical report are input into the memory supervision matrix generator, and the memory supervision matrix of the current time step is output. This includes: in the memory supervision matrix generator, based on the multilayer perceptron, the memory supervision matrix of the previous time step is used as the query; the memory supervision matrix of the previous time step and the text features of the previous time step are concatenated as key and value, and multi-head attention is performed, resulting in Z; residual connections, forgetting, and input gates are introduced to prevent gradient vanishing and exploding, and finally the memory supervision matrix of the current time step is obtained. S4: The data after passing through a fully connected layer with the extracted medical image features and the visual feature representation obtained from the cross-modal fusion module is used as the input to the encoder to obtain the hidden state; the data after passing through a fully connected layer with the word embedding of the medical report and the text feature representation obtained from the cross-modal fusion module, as well as the hidden state obtained from the encoder, are used as the input to the decoder to obtain the radiology report prediction result at the current time step; and multi-class contrastive loss is used to optimize the final prediction result.

2. The method according to claim 1, characterized in that, The medical image extractor is a pre-trained convolutional neural network, ResNet101.

3. The method according to claim 1, characterized in that, The cross-modal fusion matrix serves as a medium connecting visual and textual features, yielding visual feature representations and textual feature representations, including: The similarity between the medical image features, the corresponding report text features, and the cross-modal fusion vectors in the same category is estimated using an initialized cross-modal fusion matrix; each vector in the cross-modal fusion matrix is ​​called a cross-modal fusion vector. Under each category, select the K most similar cross-modal fusion vectors, use their similarity to visual features and text features as weights, and perform a weighted sum on the K cross-modal fusion vectors to finally obtain new representations of visual features and text features.

4. A system for generating radiological reports based on cross-modal fusion and memory supervision, characterized in that, include: The medical image feature extractor module takes a medical image as input and extracts its features. The cross-modal fusion module takes the word embedding features of the medical report and the medical image features as input. For each image-text pair, it generates a pseudo-class and initializes the cross-modal fusion matrix. The cross-modal fusion matrix serves as a medium connecting visual and text features, resulting in visual feature representations and text feature representations. Initializing the cross-modal fusion matrix includes: extracting global visual and text features for each historical radiology report sample; combining the category label of each radiology report sample to concatenate the text features and visual features to obtain the feature set for each category; using the K-Means algorithm to divide each feature set into P groups, and using the average value of the features in each group to initialize the cross-modal fusion matrix. The memory-supervised matrix generator module takes the word embedding features of the medical report as input, uses a memory matrix to record entity information in the text report, and outputs the memory-supervised matrix for the current time step. This includes: using a multilayer perceptron as the query, the memory-supervised matrix from the previous time step is used; concatenating the memory-supervised matrix and the text features from the previous time step as key and value, performing multi-head attention to obtain Z; introducing residual connections, forgetting mechanisms, and input gates to prevent gradient vanishing and exploding, finally obtaining the memory-supervised matrix for the current time step; the word embeddings of the medical report are the text features from the previous time step. The encoder module, comprising multiple encoder layers, takes the medical image features extracted from the medical image features and the visual feature representation obtained from the cross-modal fusion module as input after passing through the fully connected layer to obtain the hidden state; The decoder module comprises multiple decoder layers. It takes as input the text feature representation obtained from the word embeddings of the medical report and the cross-modal fusion module, after passing through a fully connected layer, and the hidden state obtained from the encoder module, to obtain the radiology report prediction result for the current time step. The decoder layers include: Masked MHA, a multi-head attention layer, a feedforward layer, a fully connected linear layer, and a Softmax layer. Guided normalization layers are included between the Masked MHA and the multi-head attention layer, between the multi-head attention layer and the feedforward layer, and between the feedforward layer and the fully connected linear layer. The guided normalization layer takes as input the memory supervision matrix for the current time step output by the memory supervision matrix generator module. The multi-class contrastive loss module uses multi-class contrastive loss to optimize the final prediction results.

5. The system according to claim 4, characterized in that, In the cross-modal fusion module, the cross-modal fusion matrix serves as a medium connecting visual and textual features, resulting in visual feature representations and textual feature representations, including: The similarity between the medical image features, the corresponding report text features, and the cross-modal fusion vectors in the same category is estimated using an initialized cross-modal fusion matrix; each vector in the cross-modal fusion matrix is ​​called a cross-modal fusion vector. Under each category, select the K most similar cross-modal fusion vectors, use their similarity to visual features and text features as weights, and perform a weighted sum on the K cross-modal fusion vectors to finally obtain new representations of visual features and text features.

Images