Cross-modal information enhancement method for chest radiation report generation
Through the cross-modal information enhancement method, the visual and text feature extractor, alignment module and attention module are used, combined with the contrast learning and memory matrix, the problems of lesion recognition and semantic complexity processing in the generation of medical image reports are solved, which significantly improves the Accuracy and validity of the report.
Patent Information
- Application Number
- CN202411106529.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-13
- Publication Date
- 2024-12-06
AI Technical Summary
The existing medical image report generation model lacks the ability to identify lesions, process semantic complexity and characterize cross-modal information, resulting in insufficient accurate report generation.
Using the cross-modal information enhancement method, multi-scale features are extracted through the visual extractor and text feature extractor, and feature mapping and enhancement is used for cross-modal alignment module and attention module. Combined with the comparison learning and memory matrix, the input report generation module is generated. Final diagnostic report.
It significantly improves the accuracy and effectiveness of chest radiation imaging diagnostic reports, and reduces noise and generates more accurate reports through effective fusion of cross-modal information.
Smart Images

Figure CN119092033A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to a cross-modal information enhancement method for chest radiology report generation, and belongs to the technical fields of computer vision, natural language processing, and cross-modal fusion. Background Art
[0002] At present, medical diagnosis reports are the main medium for communication between doctors and patients. The descriptions in the reports are often about the size, location and possible abnormalities of the lesions. Writing medical reports requires radiologists to have strong medical knowledge and long-term medical work experience. Automatically generating medical diagnosis reports for radiological images can relieve doctors' work pressure and significantly reduce the burden on the medical system.
[0003] The difficulties of report generation tasks are: radiological image features are difficult to extract; radiological images are difficult to match with corresponding medical reports; generated medical reports contain a large number of complex words, etc. In recent years, the automatic generation of medical image reports based on deep learning has achieved certain success, proving that there is great potential in the field of report generation. However, the existing report generation model method still has three shortcomings: 1) The lesion area in medical images accounts for a very small proportion, and it is difficult for the model to accurately identify and distinguish all lesions. 2) Medical report texts have semantic complexity, and the current model is insufficient to obtain deep semantic information of medical report texts. 3) The model's ability to represent cross-modal information is insufficient. The key factor in the effectiveness of report generation is to learn the modal invariance features at the semantic level, which can ensure that the least noise is introduced in the cross-modal conversion process from image to text, thereby generating more accurate reports. Summary of the invention
[0004] The technical problem solved by the present invention is: the present invention relates to a cross-modal information enhancement method for report generation, which effectively integrates report text features and visual image features for better cross-modal information fusion, thereby making the generated chest radiography diagnosis report more accurate.
[0005] The technical solution of the present invention is: a cross-modal information enhancement method for chest radiology report generation, firstly, multi-scale image visual features and multi-scale report text features are extracted by a visual extractor and a text feature extractor respectively. Secondly, a cross-modal alignment module is used to perform two cross-modal alignments of different scales by introducing contrast learning to strengthen the mapping relationship between visual and text features. Then, a cross-modal attention module is used to process the fused information, and the cross-modal information is further enhanced with the help of a memory matrix. Finally, the encoded image feature information and the information of the memory matrix at the last moment are input into the report generation module to output the final imaging diagnosis report.
[0006] The method specifically includes the following:
[0007] Step 1: Use a visual feature extractor and a text feature extractor to extract multi-scale image visual features and multi-scale report text features respectively.
[0008] Step 2: Use a cross-modal alignment module to align multi-scale visual features and multi-scale text features twice at different scales to strengthen the mapping relationship between visual and text features.
[0009] Step 3: Input the aligned cross-modal features into the cross-modal attention module, obtain the pattern information and store it in the memory matrix. With the help of the memory matrix, the cross-modal information is further enhanced.
[0010] Step 4: Use the report generation module to input the encoded visual features and enhanced cross-modal information into the decoder to obtain the final diagnosis report.
[0011] Furthermore, the step 1 specifically includes the following:
[0012] Given a patient's chest image, it is input into the visual feature extractor with Swin-Transformer as the backbone network, and four visual features of different scales are obtained, from low dimension to high dimension. 56×56 , X 28×28 , X 14×14 , X 7×7 , where X i×j The i and j in represent the length and width of the feature map of the feature.
[0013] Then the four multi-scale visual features obtained are fused from top to bottom in pairs through the channel attention network to achieve multi-scale visual fusion. The calculation process of this step is shown in the formula:
[0014]
[0015] X ayer_d =s1X 28×28 +X 56×56
[0016] X layer_s =s2X 14×14 +X 28×28
[0017] X layer_r =s3X 7×7 +X 14×14
[0018] in, represents the visual features of the jth layer, GAP(·) represents the global average pooling operation on the visual features, Conv1D(·) represents the convolution operation, δ(·) represents the sigmoid function, and X layer_dis the instance feature, X layer_s is the intermediate feature, X layer_r is a semantic feature.
[0019] Given a diagnostic report corresponding to an image, the report, the sentences in the report, and the disease-related words are input into the Glove model to be converted into embedding vectors, and then input into the encoder to obtain multi-scale text feature representations y r ,y s and d , the calculation process of this step is expressed by the formula:
[0020] y r =f e (Glove(Report))
[0021] y s =f e (Glove(Sentence))
[0022] y d =f e (Glove(DiseaseWord))
[0023] Report represents the input report associated with the image, Sentence represents the sentence in the report, DiseaseWord represents the disease-related word in the report, Glove(·) represents the word embedding operation, and f e (·) indicates an encoding operation.
[0024] Furthermore, the step 2 specifically includes the following:
[0025] First, a text-visual alignment network is used to map multi-scale image features and multi-scale text features into a unified dimensional space through a linear layer. Then, the minimum Euclidean distance between feature vectors at corresponding scale levels is calculated, and finally the semantic feature X layer_r With report r Correspondingly, the intermediate feature X layer_s With sentence y s Corresponding to, instance feature X layer_d With label y d Correspondingly, the initial cross-modal alignment is completed.
[0026] Then, a contrastive learning network is used to jointly train image features and report features using triplet loss. The multi-scale impact features and multi-scale text features after initial alignment are mapped to the same high-dimensional space to obtain img and txt respectively. p , treat other reported samples in the same batch as negative samples txt n, and conduct joint training. In order to better model the mutual mapping relationship between images and reports, this paper uses the input image and the real report as anchor points respectively, and adjusts the distance from image to report and report to image, so that the paired image report pairs are closer in high-dimensional space. The training process can be expressed as:
[0027]
[0028] L triplet =L imgtotxt +L txttoimg
[0029] Among them, d(·) is the cosine similarity calculation, and the calculation formula is:
[0030]
[0031] in, Indicates vectorizing the image. Indicates vectorizing the positive sample report. Indicates vectorizing the negative sample report. It represents the distance between the input image and the true report. Indicates the distance between the input image and other reports. [·] + It means that when the value in the bracket is greater than zero, the parameter is updated, and when it is less than or equal to zero, the current parameter is not updated. α refers to the minimum interval distance parameter, which is a preset threshold.
[0032] After the contrastive learning network, the distance between paired image reports becomes closer, and the distance between negative sample reports that are extremely similar to the real reports is pulled farther, thereby achieving a more refined alignment between the image visual features and the report text features.
[0033] Furthermore, the step 3 specifically includes the following:
[0034] Assume that the current time is t, and the memory matrix M of the previous time is t-1 As the key and value of multi-head attention, the visual features in the vector pairs of its modules are cross-modal. As Query. Then the result after multi-head attention is combined with the visual features The result of addition is X mulAtt As the query of multi-head cross attention. The output of multi-head cross attention is the text feature Y adaptively adjusted according to the weight of the visual feature crossAtt , and then the obtained text feature Y crossAtt With visual features X mulAtt By establishing associations through the memory matrix, cross-modal pattern information is obtained and stored in the memory matrix. The process is represented by the formula:
[0035]
[0036] Among them, MultiHead(·) represents multi-head attention, concatenate(·) represents concatenation operation, and crossAttention(·) represents cross attention. The final output of the memory matrix is expressed as:
[0037]
[0038] in, is the forget gate of the current moment, is the output gate at the current moment, is the memory matrix after residual connection at time t, M t is the final memory matrix output, δ(·) is the sigmoid function, and ⊙ is the Hadamard product.
[0039] Since the memory matrix records and updates the correlation information between visual features and text features, the memory matrix is initialized and updated through triplet loss, making the input image report pairs closer, thereby improving the quality of the generated report.
[0040] Furthermore, the step 4 specifically includes the following:
[0041] Use the extracted multi-scale visual features to query the most relevant pattern information m through the memory matrix i , through m i Get visual response r x and text response y , and are input into the encoder and decoder to generate the final report Y target . The formula is shown below.
[0042] r x =w x ·m i ·X
[0043] r y =w y ·m i ·(m i ·X)
[0044] p=f e (r x )⊕r y
[0045] Y target =softmax(linear(f d (p)))
[0046] where w x and w yis the weight that can be obtained through training, the symbol ⊕ represents the addition operation, and f e (·) represents the encoder, f d (·) represents the decoder, linear(·) represents the fully connected layer, and softmax(·) is the activation function.
[0047] The beneficial effects of the present invention are as follows: the present invention effectively utilizes the cross-modal information between the report text features and the image visual features, and proposes an enhancement method for the cross-modal information. The ablation experiment and comparative experiment on two public medical report generation data sets IU X-ray and MIMIC-CXR prove that the description of the chest radiology diagnosis report generated by the present invention has achieved significant improvement in effectiveness and accuracy. BRIEF DESCRIPTION OF THE DRAWINGS
[0048] Figure 1 The overall framework diagram of the cross-modal information enhancement method for generating a face report of the present invention;
[0049] Figure 2 This is a diagram of the comparative learning network structure of the present invention;
[0050] Figure 3 This is a structural diagram of the cross-modal attention module of the present invention;
[0051] Figure 4 This is a schematic diagram of the visual analysis of the present invention. DETAILED DESCRIPTION
[0052] Embodiment 1, as Figure 1-Figure 4 As shown, a cross-modal information enhancement method for chest radiology report generation, the specific steps of the method are as follows:
[0053] Step 1: Use a visual feature extractor and a text feature extractor to extract multi-scale image visual features and multi-scale report text features respectively.
[0054] Step 2: Use a cross-modal alignment module to align multi-scale visual features and multi-scale text features twice at different scales to strengthen the mapping relationship between visual and text features.
[0055] Step 3: Input the aligned cross-modal features into the cross-modal attention module, obtain the pattern information and store it in the memory matrix. With the help of the memory matrix, the cross-modal information is further enhanced.
[0056] Step 4: Use the report generation module to input the encoded visual features and enhanced cross-modal information into the decoder to obtain the final diagnosis report.
[0057] Furthermore, the step 1 specifically includes the following:
[0058] Given a patient's chest image, it is input into the visual feature extractor with Swin-Transformer as the backbone network, and four visual features of different scales are obtained, from low dimension to high dimension. 56×56 , X 28×28 , X 14×14 , X 7×7 , where X i×j The i and j in represent the length and width of the feature map of the feature.
[0059] Then the four multi-scale visual features obtained are fused from top to bottom in pairs through the channel attention network to achieve multi-scale visual fusion. The calculation process of this step is shown in the formula:
[0060]
[0061] X layer_d =s1X 28×28 +X 56×56
[0062] X layer_s =s2X 14×14 +X 28×28
[0063] X layer_r =s3X 7×7 +X 14×14
[0064] in, represents the visual features of the jth layer, GAP(·) represents the global average pooling operation on the visual features, Conv1D(·) represents the convolution operation, δ(·) represents the sigmoid function, and X layer_d is the instance feature, X layer_s is the intermediate feature, X layer_r is a semantic feature.
[0065] Given a diagnostic report corresponding to an image, the report, the sentences in the report, and the disease-related words are input into the Glove model to be converted into embedding vectors, and then input into the encoder to obtain multi-scale text feature representations y r ,y s and d , the calculation process of this step is expressed by the formula:
[0066] y r =f e (Glove(Report))
[0067] y s =f e (Glove(Sentence))
[0068] y d =f e (Glove(DiseaseWord))
[0069] Report represents the input report associated with the image, Sentence represents the sentence in the report, DiseaseWord represents the disease-related word in the report, Glove(·) represents the word embedding operation, and f e (·) indicates an encoding operation.
[0070] Furthermore, the step 2 specifically includes the following:
[0071] First, a text-visual alignment network is used to map multi-scale image features and multi-scale text features into a unified dimensional space through a linear layer. Then, the minimum Euclidean distance between feature vectors at corresponding scale levels is calculated, and finally the semantic feature X layer_r With report r Correspondingly, the intermediate feature X layer_s With sentence y s Corresponding to, instance feature X layer_d With label y d Correspondingly, the initial cross-modal alignment is completed.
[0072] Then, a contrastive learning network is used to jointly train image features and report features using triplet loss. The multi-scale impact features and multi-scale text features after initial alignment are mapped to the same high-dimensional space to obtain img and txt respectively. p , treat other reported samples in the same batch as negative samples txt n , and conduct joint training. In order to better model the mutual mapping relationship between images and reports, the present invention uses the input image and the real report as anchor points respectively, and adjusts the distance from image to report and from report to image, so that the paired image report pairs are closer in high-dimensional space. The training process can be expressed as:
[0073]
[0074] L triplet =L imgtotxt +L txttoimg
[0075] Among them, d(·) is the cosine similarity calculation, and the calculation formula is:
[0076]
[0077] in, Indicates vectorizing the image. Indicates vectorizing the positive sample report. Indicates vectorizing the negative sample report. It represents the distance between the input image and the true report. Indicates the distance between the input image and other reports. [·] + It means that when the value in the bracket is greater than zero, the parameter is updated, and when it is less than or equal to zero, the current parameter is not updated. α refers to the minimum interval distance parameter, which is a preset threshold.
[0078] After the contrastive learning network, the distance between paired image reports becomes closer, and the distance between negative sample reports that are extremely similar to the real reports is pulled farther, thereby achieving a more refined alignment between the image visual features and the report text features.
[0079] Furthermore, the step 3 specifically includes the following:
[0080] Assume that the current time is t, and the memory matrix M of the previous time is t-1 As the key and value of multi-head attention, the visual features in the vector pairs of its modules are cross-modal. As Query. Then the result after multi-head attention is combined with the visual features The result of addition is X mulAtt As the query of multi-head cross attention. The output of multi-head cross attention is the text feature Y adaptively adjusted according to the weight of the visual feature cRossAtt , and then the obtained text feature Y crossAtt With visual features X mulAtt By establishing associations through the memory matrix, cross-modal pattern information is obtained and stored in the memory matrix. The process is represented by the formula:
[0081]
[0082] Among them, MultiHead(·) represents multi-head attention, concatenate(·) represents concatenation operation, and crossAttention(·) represents cross attention. The final output of the memory matrix is expressed as:
[0083]
[0084] in, is the forget gate of the current moment, is the output gate at the current moment, is the memory matrix after residual connection at time t, M t is the final memory matrix output, δ(·) is the sigmoid function, and ⊙ is the Hadamard product.
[0085] Since the memory matrix records and updates the correlation information between visual features and text features, the memory matrix is initialized and updated through triplet loss, making the input image report pairs closer, thereby improving the quality of the generated report.
[0086] Furthermore, the step 4 specifically includes the following:
[0087] Use the extracted multi-scale visual features to query the most relevant pattern information m through the memory matrix i , through m i Get visual response r x and text response y , and are input into the encoder and decoder to generate the final report Y target . The formula is shown below.
[0088] r x =w x ·m i ·X
[0089] r y =w y ·m i ·(m i ·X)
[0090]
[0091] Y target =softmax(linear(f d (p)))
[0092] where w x and w y is the weight that can be obtained through training, symbol represents the addition operation, f e (·) represents the encoder, f d (·) represents the decoder, linear(·) represents the fully connected layer, and softmax(·) is the activation function.
[0093] Furthermore, in order to verify the effect of the present invention, the model trained by the above steps was input into the medical report generation datasets IU X-ray and MIMIC-CXR for ablation experiments. The IU X-ray dataset is a widely used publicly available chest radiography dataset, which contains chest imaging reports of 3955 patients and 7470 matching radiographic images. The MIMIC-CXR dataset is the largest public dataset containing chest X-rays and imaging reports to date, including a total of 377110 chest X-ray images and 227835 reports. This experiment uses GPU RTX 3080Ti as the hardware platform, with a running memory of 12G. During the experiment and testing process, this paper uses the PyTorch deep learning framework, and the compilation language is Python3.8.
[0094] In the present invention, a cross-modal information enhancement method for chest radiology report generation is provided. Figure 1 First, the visual feature extractor and the text feature extractor are used to extract multi-scale image visual features and multi-scale report text features respectively; then, the cross-modal alignment module aligns the multi-scale visual features and multi-scale text features twice at different scales to strengthen the mapping relationship between visual and text features, as shown in Figure 1. Figure 2 As shown; secondly, the aligned cross-modal features are input into the cross-modal attention module, and the cross-modal information is further enhanced with the help of the memory matrix, as shown in Figure 3 As shown; finally, the encoded visual features and the enhanced cross-modal information are input into the decoder together to obtain the final diagnosis report.
[0095] In order to verify the effectiveness of the cross-modal information enhancement method for chest radiology report generation proposed in this invention, a series of experiments were conducted on the public medical report generation datasets IU X-ray and MIMIC-CXR to compare and verify the effectiveness of the method of this invention. The comparison of experimental results is shown in Table 1:
[0096] Table 1 Comparison of experimental results of different models on IU X-ray and MIMIC-CXR
[0097]
[0098] The results in Table 1 show that the experimental results of the mainstream medical report generation model methods in medical visual question answering research in recent years on the IU X-ray dataset and MIMIC-CXR were selected and compared with the experimental results of the present invention. In the model of the present invention, the cross-modal feature information is enhanced by introducing contrast learning and memory matrix, so as to better perform cross-modal feature fusion and improve the model's chest image recognition and diagnosis report generation capabilities. As shown in the table above, this method effectively improves the overall answer accuracy of the medical report generation model.
[0099] In order to verify the effectiveness of each module in the model method of this chapter, the present invention conducted a series of ablation experiments on the IU X-ray dataset and the MIMIC-CXR dataset. The experimental results are shown in Table 2:
[0100] Table 2 Ablation experiment results
[0101]
[0102] As shown in Table 2, BASE in the table is a benchmark model that uses Transformer as the backbone network, CMA is a cross-modal attention module, and CL is a contrastive learning module. This paper gradually adds the proposed modules to verify the effectiveness of the proposed model. The results show that both BLEU-n and ROUGE scores have been improved. The proposed cross-modal information enhancement method can effectively alleviate data bias, and enhance the feature representation of key lesions through contrastive learning, thereby enhancing the ability of the model training process to obtain visual and text feature mapping. In the prediction process, the model can generate reports that are more consistent with real reports.
[0103] The above experiments verified the performance of the proposed method through quantitative indicators and proved the superiority of the diagnosis report generation model. At the same time, the present invention qualitatively analyzed the effect of the model by intuitively comparing the generated diagnosis report results with the original medical diagnosis results. Figure 4 As shown;
[0104] It can be clearly seen from the comparison of the two reports in the figure that the sentences in the diagnostic report generated by the present invention are close to or even equivalent to the real report. Some of the statements about lesions are different from the real report, but the semantics expressed are the same. For example, in the real report corresponding to the second radiological image, "The contours of the mediastinum and the hilum are normal." and the generated "The contours of the heart, mediastinum and hilum are normal." have the same semantics, both indicating that "the contours of the heart, mediastinum and hilum are normal", which shows that the model can mine the deep correlation between image features and text features through comparative learning and memory matrix during the training process, so that accurate and diversified description reports can be generated during prediction, further verifying the effectiveness of the proposed model.
[0105] The specific implementation modes of the present invention are described in detail above in conjunction with the accompanying drawings, but the present invention is not limited to the above implementation modes, and various changes can be made within the knowledge scope of ordinary technicians in this field without departing from the purpose of the present invention.
Claims
1. A cross-modal information enhancement method for chest radiology report generation, characterized in that: The method specifically includes the following: Step 1: Use a visual feature extractor and a text feature extractor to extract multi-scale image visual features and multi-scale report text features respectively; Step 2: Using a cross-modal alignment module, the multi-scale visual features and multi-scale text features are aligned twice at different scales to strengthen the mapping relationship between visual and text features. Step 3: Input the aligned cross-modal features into the cross-modal attention module, obtain the mode information and store it in the memory matrix, and further enhance the cross-modal information with the help of the memory matrix; Step 4: Use the report generation module to input the encoded visual features and enhanced cross-modal information into the decoder to obtain the final diagnosis report.
2. The cross-modal information enhancement method for chest radiology report generation according to claim 1, characterized in that: The step 1 includes the following: Given a patient's chest image, it is input into the visual feature extractor with Swin-Transformer as the backbone network, and four visual features of different scales are obtained, from low dimension to high dimension. 56×56 , X 28×28 , X 14×14 , X 7×7 , where X i×j The i and j in it represent the length and width of the feature map of the feature; Then the four multi-scale visual features obtained are fused from top to bottom in pairs through the channel attention network to achieve multi-scale visual fusion. The calculation process of this step is shown in the formula: X layer_d =s1X 28×28 +X 56×56 X layer_s =s2X 14×14 +X 28×28 X layer_r =s3X 7×7 +X 14×14 Among them, s j express, represents the visual features of the jth layer, GAP(·) represents the global average pooling operation on the visual features, Conv1D(·) represents the convolution operation, δ(·) represents the sigmoid function, and X layer_d is the instance feature, X layer_s is the intermediate feature, X layer_r is a semantic feature; Given a diagnostic report corresponding to an image, the report, the sentences in the report, and the disease-related words are input into the Glove model to be converted into embedding vectors, and then input into the encoder to obtain multi-scale text feature representations y r ,y s and d , the calculation process of this step is expressed by the formula: y r =f e (Glove(Report)) y s =f e (Glove(Sentence)) y d =f e (Glove(DiseaseWord)) Report represents the input report associated with the image, Sentence represents the sentence in the report, DiseaseWord represents the disease-related word in the report, Glove(·) represents the word embedding operation, and f e (·) indicates an encoding operation.
3. The cross-modal information enhancement method for chest radiology report generation according to claim 1, characterized in that: The step 2 includes the following: First, a text-visual alignment network is used to map multi-scale image features and multi-scale text features into a unified dimensional space through a linear layer. Then, the minimum Euclidean distance between feature vectors at corresponding scale levels is calculated, and finally the semantic feature X layer_r With report r Correspondingly, the intermediate feature X layer_s With sentence y s Corresponding to, instance feature X layer_d With label y d Correspondingly, the initial cross-modal alignment is completed; Then, a contrastive learning network is adopted to jointly train the image features and report features using triplet loss; Map the multi-scale influence features and multi-scale text features after the initial alignment to the same high-dimensional space to obtain img and txt respectively. p , treat other reported samples in the same batch as negative samples txt n , and conduct joint training; in order to better model the mutual mapping relationship between images and reports, the input image and the real report are used as anchor points respectively, and the distances from image to report and report to image are adjusted at the same time, so that the paired image report pairs are closer in the high-dimensional space; the training process is expressed as: L triplet =L imgtotxt +L txttoimg Among them, d(·) is the cosine similarity calculation, and the calculation formula is: in, Indicates vectorizing the image. Indicates vectorizing the positive sample report. Indicates vectorizing the negative sample report. It represents the distance between the input image and the true report. Indicates the distance between the input image and other reports. [·] + It means that when the value in the bracket is greater than zero, the parameter is updated, and when it is less than or equal to zero, the current parameter is not updated. α refers to the minimum interval distance parameter, which is a preset threshold; After the contrastive learning network, the distance between paired image reports becomes closer, and the distance between negative sample reports that are extremely similar to the real reports is pulled farther, thereby achieving a more refined alignment between the image visual features and the report text features.
4. The cross-modal information enhancement method for chest radiology report generation according to claim 1, characterized in that: The step 3 specifically includes the following: Assume that the current time is t, and the memory matrix M of the previous time is t-1 As the key and value of multi-head attention, the visual features in the vector pairs of its modules are cross-modal. as Query; then the result after multi-head attention is combined with the visual features The result of addition is X mulAtt As the query of multi-head cross attention. The output of multi-head cross attention is the text feature Y adaptively adjusted according to the weight of the visual feature crossAtt , and then the obtained text feature Y crossAtt With visual features X mulAtt Establish associations through the memory matrix, obtain cross-modal pattern information and store it in the memory matrix; The process is represented by the formula: Among them, MultiHead(·) represents multi-head attention, concatenate(·) represents the concatenation operation, crossAttention(·) represents cross attention. The final output of the memory matrix is expressed as: in, is the forget gate of the current moment, is the output gate at the current moment, is the memory matrix after residual connection at time t, M t is the final memory matrix output, δ(·) is the sigmoid function, and ⊙ is the Hadamard product; Since the memory matrix records and updates the correlation information between visual features and text features, the memory matrix is initialized and updated through triplet loss, making the input image report pairs closer, thereby improving the quality of the generated report.
5. The cross-modal information enhancement method for chest radiology report generation according to claim 1, characterized in that: The step 4 specifically includes the following: Use the extracted multi-scale visual features to query the most relevant pattern information m through the memory matrix i , through m i Get visual response r x and text response y , and are input into the encoder and decoder to generate the final report Y target ; The formula is as follows. r x =w x ·m i ·X r y =w y ·m i ·(m i ·X) Y target =softmax(linear(f d (p))) where w x and w y is the weight obtained through training, symbol represents the addition operation, f e (·) represents the encoder, f d (·) represents the decoder, linear(·) represents the fully connected layer, and softmax(·) is the activation function.
Citation Information
Cited By
Medical form data identification method and system based on OCR and MLLM
CN119672743A
Three-dimensional image data processing method, data processing method, equipment and storage medium
CN119723004A
Similarity report prompt cross-modal enhancement medical report automatic generation method
CN120954613A
Similarity report prompt cross-modality augmented medical report generation method
CN120954613B
