A medical report automatic generation method based on cross-modal contrast attention

Abstract

The application relates to a medical report automatic generation method based on cross-modal contrast attention, which is specifically divided into two stages of encoding and decoding. The encoding stage comprises three sub-modules: a visual attention module VAM, a visual contrast attention module VCAM and a cross-modal attention module CAM. The VAM extracts visual features of an input image. In addition, by contrasting the visual difference and similarity between a current image and a similar case image in the past, the VCAM guides an attention model to focus on the unique abnormal visual area of the current image by using the difference, and the CMAM guides the attention model to realize cross-modal matching of effective words in the similar case report in the past by using the similarity. The decoding stage is composed of a parallel attention module PAM, which calculates the encoding features obtained in the encoding stage in parallel to guide the generation of a medical report. The application has carried out sufficient experiments on natural language evaluation indexes and clinical evaluation indexes, and the experimental results show the effectiveness of the proposed method.

Description

TECHNICAL FIELD

[0001] The present application relates to the fields of medical images, computer vision and natural language processing, and is directed to a medical report automatic generation method based on cross-modal contrast attention for the goal of automatic generation of lung X-ray image medical reports. BACKGROUND

[0002] Medical images and reports are widely used in clinical disease diagnosis and treatment, and have become an indispensable part of many disease diagnosis processes today. In practice, radiologists need to carefully read medical images, find possible lesion information in them, accurately diagnose disease types, and write reports to describe disease information in detail. With the massive growth of patients and related medical images, this has become a huge burden for radiologists in clinical diagnosis. Therefore, computer-aided radiologists to automatically generate medical reports has become a task that needs to be solved urgently and has received extensive attention. At the same time, due to the publication and promotion of lung X-ray image report related public data sets, the medical report generation task for lung X-ray images has become a hot spot of related research.

[0003] Medical report generation is a new research direction in the intersection of medicine and computer in recent years, and its goal is to automatically generate coherent and logically consistent medical reports for medical images. The knowledge used covers three fields of medical images, computer vision and natural language processing. Most current medical report generation methods follow the encoder-decoder (Encoder-Decoder) framework of natural image description and make improvements, and have made some progress. However, as shown in the following formula (1), compared with natural images, medical images have the following two major data biases: 1) The abnormal areas in medical images are usually small, the disease types are diverse, some disease types are rare, and the image features are monotonous and homogeneous. These problems make it difficult to identify lesions in medical images; 2) The corresponding medical report is a complete description of the information possessed by the image, and the content contains normal and abnormal image information, while the same medical image usually only contains a small amount of abnormal information, in which normal information dominates, so the description of normal information in the corresponding report occupies the majority of the text. The above two data biases make it unreliable to simply apply traditional natural image description methods to medical report generation tasks. Figure 1

[0004] ​Related research has proposed solutions to the above problems, but most of these methods only consider starting from the input data itself, designing relevant modules to narrowly find internal abnormalities of their own, thereby ignoring the mutual correlation between cases: according to the knowledge and writing specifications of radiologists, for similar cases of medical images, the medical reports written by doctors are basically the same. Therefore, when designing a medical report automatic generation method, making full use of similar past cases of the current input case can effectively help generate the medical report of the current case. SUMMARY

[0005] In order to effectively mine the effective information of the image and report in the past similar cases in the medical report automatic generation task to help the generation of the current medical image report, the application discloses a medical report automatic generation method based on cross-modal contrastive attention (CMCA). The application can effectively mine the effective information in the medical image and the corresponding text report of the past similar cases from the aspects of vision and semantics respectively, and assist in generating the medical report of the current image. The application can not only effectively improve the performance of the model on the natural language index, but also more effectively mine the disease information to improve the performance of the model on the clinical index.

[0006] To achieve the above purpose, the application provides a medical report automatic generation method based on cross-modal contrastive attention, characterized in that the method comprises two stages of encoding and decoding. The specific steps are as follows:

[0007] Step (1) data preparation: obtaining a medical report generation data set and preprocessing to obtain a training data set, and constructing a case library Among them, the data set contains medical image data and its corresponding medical report data, and the preprocessing is used to obtain the image space feature X and the global feature of the medical image data And the medical report data is mapped into a vector by word embedding Case library Including a plurality of cases;

[0008] Step (1.1) data preprocessing specifically includes: using a feature extractor model CNN to complete the pre-encoding of all medical image data Img in the data set, to obtain the image space feature X and the global feature The medical report data R in the data set is mapped into a vector by word embedding Wherein X, And The calculation formula is as follows:

[0009] X=CNN(Img),

[0010]

[0011]

[0012] Where CNN(·) represents a convolutional neural network, max_pooling(·) represents max pooling downsampling, and Embedding(·) represents word embedding.

[0013] Step (1.2) Case library construction specifically includes: the case library is obtained from all the data in the training set, and the spatial feature X of the d-th data in the training set is extracted through preprocessing. d Global features and its corresponding medical report term embedding vector Together they form a triplet Here, cased will be used as the d-th case; subsequently, the above operations will be performed on all data in the training set to form the case library. Where N represents the amount of data in the training set.

[0014] Step (2) Encoding stage: The encoding stage is used to extract the visual attention features V of the image to be processed I. a Visual contrast attention features of image I to be processed and similar cases, and cross-modal attention features of image I to be processed and similar cases, wherein visual attention feature V a Visual contrast attention features are used to characterize lesion information in image I itself. Attention feature Cr is used to characterize the feature comparison between image I and the case image. a Used to characterize cross-modal matching of valid words in case reports;

[0015] The method for selecting similar cases is as follows: calculate the global features of the image I to be processed. With case library The cosine similarity of the global features of each case is used to select the top N cases with the highest similarity. K The cases corresponding to the cosine similarity scores are N, which are the most similar to the image I to be processed. K Case

[0016] The visual attention features V of the image to be processed in step (2.1) a The calculation process is as follows: input the visual features of the image to be processed into the visual attention module (VAM) to obtain the visual attention features V of the image to be processed. a The calculation method is further expressed as the following formula:

[0017] V a =VAM(X) I ).

[0018] Where VAM(·) represents the calculation formula for the visual attention module,

[0019] VAM(X I ) = FFN(MHA(X I X I X I )),

[0020] MHA(q,k,v)=[head1,head2,…,head n W O ,

[0021]

[0022] FFN(x)=max(0,xW1+b1)W2+b2.

[0023] In the above formula, MHA(·) represents the calculation of multi-head attention features, which divides the input data into multiple subspaces and focuses on them by multiple attention heads respectively; FFN(·) represents a feedforward neural network; [·, ..., ·] represents the concatenation operation on tensors, head1, ..., head2, ..., head3, ..., head4, ..., head5, ..., head6, ..., head7, ..., head8, ..., head9, ..., head9, head1 ... n This represents the first to nth attention heads, used to focus on n subspaces of the input data. q, k, and v represent the input of the formula, and W... O These are the parameters for learning; head i This represents the i-th attention head, which will focus on the i-th subspace of the input data. Attn i (·) represents the calculation of its attention features, softmax(·) is a commonly used activation function, q i k i v i It is the i-th attention head i The input, i.e., the i-th subspace divided from the input data, d k is a fixed constant; x represents the input of the formula, W1 and W2 are the learned parameters, and b1 and b2 are the bias values. The MHA(·) and FFN(·) introduced here will continue to be used in subsequent calculations, where the learned parameters are obtained from the training data of different modules.

[0024] Step (2.2) Retrieve similar past cases and calculate visual contrast features V c First, based on the global features v of image I I Using cosine similarity from the case library Retrieve the most similar N from K Case Among them, case I，drepresents the dth case most similar to the image I; subsequently, the spatial feature X I of the similar case case I，d is calculated I，d , d e {1, …, N K}, and the visual contrast feature V c is calculated c , which will realize the enhancement of the lesion features unique to the image I, the weakening of the lesion features unique to the case I，d , and the preservation of the lesion information common to both, and its calculation formula is as follows:

[0025] D I = X I - X I，d ,

[0026] V c = X I + D I ;

[0027] wherein the spatial feature X I of the image I and the spatial feature X I，d of the case case I，d have three possible disease conditions: 1) unique in X I and not in X I，d ; 2) not in X I and unique in X I，d ; 3) common to both. Therefore, for the difference feature D I and the contrast feature V c obtained, wherein: 1) the difference feature D I will realize the preservation of the lesion information unique in X I and the removal of the lesion information of X I in X I，d by calculating the difference between the spatial features of the image I and the case case I，d ; 2) since the difference feature D will remove the lesion information common to both, the contrast feature V c is proposed to realize the recovery of the lesion information common to both removed in D on the basis of D, and V c will realize the enhancement of the lesion information unique in X I .

[0028] The calculation step of the visual contrast attention feature V mentioned in step (2.3) is: inputting the contrast attention feature V c into the visual contrast attention module VCAM to obtain the visual contrast attention feature V , and its calculation method is further represented as the following formula:

[0029]

[0030] where VCAM(·) represents the calculation formula of the visual contrast attention module, and the specific calculation steps are as follows:

[0031]

[0032] The calculation steps of the cross-modal attention feature Cr a described in step (2.4) are as follows: for the contrast attention feature V c and the report of the similar case case I，d Using the cross-modal contrast attention module CAM, the cross-modal matching of V c and is realized by using the multi-head attention mechanism, and the cross-modal attention feature Cr a is obtained. The specific calculation formula is as follows:

[0033]

[0034] where CMAM(·) represents the calculation formula of the cross-modal attention module, d∈{1,…,N K}, and the specific calculation steps are as follows:

[0035]

[0036] Step (3) decoding stage: generate a complete medical report of the image I to be processed using each similar case, input the encoding features extracted in the encoding stage into the parallel attention module, and generate a word y t at each time t, then concatenate all the generated words and finally obtain the complete report Y:

[0037] Step (3.1) calculate the decoding result at each time: for each time t in the decoding stage, input the word embedding vector generated at the previous time y 1：t-1 , then use the parallel attention module PAM to calculate the three encoding features obtained in the encoding stage to obtain the hidden state h t at this time, and further obtain the word embedding vector y t of the word generated at this time, and the calculation formula is as follows:

[0038]

[0039]

[0040] where is the learned parameter, ​is the bias value, PAM(·) represents the calculation formula of the parallel attention module, and the specific calculation steps are as follows:

[0041]

[0042]

[0043]

[0044]

[0045]

[0046] wherein Norm(·) represents a normalization operation.

[0047] Step (3.2) generates a complete medical report: all the word embeddings generated at all times are taken as the output complete report Y:

[0048] Y={y1,y2,...,y T}.

[0049] wherein T represents the word quantity of the generated complete medical report.

[0050] Step (3.3) selects the optimal medical report: N K cases obtained by searching in step (2) are made to participate in the above operation respectively and N K complete medical reports are finally obtained, evaluation indexes obtained by comparing the complete medical reports with the true values are calculated respectively, and an optimal report is selected as the final result.

[0051] Beneficial effects

[0052] The method has the following characteristics: 1) a medical report generation method based on a cross-modal contrastive attention mechanism (CMCA) is designed, and effective information is extracted from similar past cases to guide the generation of more accurate medical reports for the current case through a cross-modal contrastive method based on an attention mechanism, so as to solve the data distribution deviation problem of the task public data set and reduce the influence of the noise problem in the data on the performance of the task method. 2) CMCA is divided into two stages of encoding and decoding, wherein the encoding stage includes three sub-modules: a visual attention module (VAM), a visual contrastive attention module (VCAM) and a cross-modal attention module (CAM). Among them, VAM extracts visual features of the input image; in addition, by comparing the visual differences and similarities between the current image and the similar past case image: VCAM uses the differences to guide the attention model to focus on the unique abnormal visual area of the current image; CMAM uses the similarity to guide the attention model to realize the cross-modal matching of effective words in the past similar case report. The decoding stage is composed of a parallel attention module (PAM), which calculates the encoding features obtained in the encoding stage in parallel to guide the generation of medical reports. 3) The present application makes full experiments on natural language evaluation indicators and clinical evaluation indicators, and the experimental results show the effectiveness of the proposed method.

[0053] Compared with other medical report automatic generation methods, the method of the present application has better natural language evaluation indicators and clinical evaluation indicators. This shows that the present method can effectively mine effective information in medical images and reports in similar past cases, help to more accurately identify abnormal lesion information in the current input medical image, and accurately match effective text information in the case report to assist the report generation of the current input medical image. BRIEF DESCRIPTION OF DRAWINGS

[0054] Figure 1 : A lung X-ray medical image and its corresponding report.

[0055] Figure 2 : A medical report automatic generation method framework based on a cross-modal contrastive attention mechanism.

[0056] Figure 3 : A contrast feature extraction method structure diagram.

[0057] Figure 4 : A visual text cross-modal matching schematic diagram.

[0058] Figure 5 generating a report visualization.

[0059] Figure 6 generating a report versus real report visualization. DETAILED DESCRIPTION

[0060] The following takes the IUX-Ray lung X-ray medical report automatic generation public data set as an example to specifically explain the implementation steps and effectiveness of the method:

[0061] Step (1) first acquires the medical report generation data set IU X-Ray and pre-processes the data, and then divides the data set and constructs a case base:

[0062] Step (1.1) data preprocessing: the present application uses the IU X-Ray data set, which is a public medical report generation data set, containing 7470 lung X-ray images (each patient can have a front image and a side image) and 3955 medical reports corresponding thereto from 3955 patients. The data preprocessing steps include: using a feature extractor model to complete the pre-coding of the image data in the data set to obtain image space features and global features Word embedding mapping of the text data R of the medical report into a vector where R L represents the number of words contained in R. Wherein the calculation formula of X, and is as follows:

[0063] X=CNN(Img),

[0064]

[0065]

[0066] Then, all the pre-processed data is randomly divided into a training set, a validation set and a test set according to a ratio of 7:1:2. Among them, the training set is used to learn the parameters of the neural network, the validation set is used to determine the hyperparameters, and the test set is used to verify the effectiveness of the present application.

[0067] Step (1.2) constructing a case base: the case base of the present application is obtained from the training set. Taking the dth data of the training set as an example: the above steps extract the spatial features X d , global features and the corresponding medical report word embedding vector together constitute a triple Cased will be used as one case. Subsequently, the triples formed from the data in the training set will collectively constitute the case library. Where N represents the amount of data in the training set.

[0068] Step (2) Encoding stage: Extracting the lesion information V from the image to be processed a Based on this, we will make full use of the effective information in similar case images and reports, and utilize the contrast feature V. c Extracting visual contrast attention features V c a and cross-modal attention features Cr a Two-part coding features:

[0069] Step (2.1) Calculate the visual attention features V of the image to be processed. a For the spatial features of the input image obtained in step (1.2) Using the Visual Attention Module (VAM) proposed in this invention, X is extracted. I Visual attention features The specific calculation formula is as follows:

[0070] V a =VAM(X) I ).

[0071] Step (2.2) Retrieve similar past cases and calculate visual contrast features: First, based on the global features v of image I... I Using cosine similarity from the case library Retrieve the most similar N from K Case Among them, case I，d This represents the d-th case most similar to image I; subsequently, the spatial features X of image I are used... I Similar cases I，d Spatial features X I，d d∈{1,…,N K}, calculate the visual contrast features V of the two. c The visual contrast feature V c This will enhance the unique lesion features of image I and weaken the case. I，d The unique lesion characteristics, while maintaining the shared lesion information between the two, are calculated using the following formula:

[0072] D I =X I -X I，d ,

[0073] V c =X I +D I ;

[0074] Step (2.3) Calculate visual contrast attention feature V For the contrast attention feature V c obtained in step (2.3), the visual contrast attention module (VCAM) proposed in the present application is used to further extract the attention feature of the abnormal area of the enhanced input image V c The enhancement of the unique lesion area of the input image I is realized:

[0075]

[0076] Step (2.4) Calculate cross-modal attention feature Cr a : For the contrast attention feature V c obtained in step (2.3) and the report of the similar case case I，d obtained in step (2.1) Using the cross-modal contrast attention module (CAM) proposed in the present application, the cross-modal matching of V c and is realized by using the multi-head attention mechanism, and the cross-modal attention feature Cr is obtained. This module will realize the cross-modal alignment of the lesion area common to both V c and the effective words in the case report, so as to extract the effective words to help the generation of the input image report, and the specific calculation formula is as follows:

[0077]

[0078] V a , and Cr a obtained in the encoding stage will be jointly provided as encoding features to the decoding stage.

[0079] Step (3) Decoding stage: The decoding stage will realize the generation of a complete medical report of the input image I to be processed by using each similar case, and the encoding features V a , and Cr a obtained in the encoding stage are used to generate words yt one by one at each time t, and finally a complete report Y is obtained:

[0080] Step (3.1) Calculate the decoding result at each time: for each time t of the decoding stage, the word embedding vector generated at the previous time is used as input y 1：t-1 , and then the parallel attention module (PAM) proposed in the present application is used to calculate the three encoding features (i.e. visual attention feature V​a , visual contrast attention feature and cross-modal attention feature Cr a ), to obtain the hidden state of the moment and further obtain the word embedding vector of the word generated at the moment The calculation formula is as follows:

[0081]

[0082]

[0083] Step (3.2) generates a complete medical report. The word embedding generated at all moments is taken as the complete report output by the present application

[0084] Y = {y1, y2,..., y T}.

[0085] Step (3.3) selects the optimal medical report: N K cases obtained by step (2) are made to participate in the above operation respectively and N K complete medical reports are finally obtained, the BLEU-1 evaluation index obtained by comparing each complete medical report with the true value is calculated, and the optimal one is selected as the final result.

[0086] In order to verify the effectiveness of the method described in the present application, we carried out experiments on the above-mentioned method on the public data sets of lung X-ray medical report automatic generation IU X-ray and MIMIC-CXR, and carried out sufficient comparative analysis with other existing advanced methods. In the specific implementation process, we also compared the most advanced related work in the field of medical report generation and image description, including: CNN-RNN, AdaAtt, Att2in, HRNN, CoAtt, HRGR-Agent, CMAS-RL, KERP, R2Gen, CMN, CA, Top-Down and Transformer. Among them, the Transformer model is the comparison baseline used in the present application. The experiment adopts natural language evaluation indexes for natural language processing tasks such as text generation, including BLEU, METEOR, ROUGE, and also uses clinical rating indexes for evaluating disease recognition accuracy, including precision, recall and F1 score. The evaluation indexes from the two angles can realize the comprehensive consideration of the natural language coherence, accuracy and disease objective situation recognition accuracy of the generated text, and are the currently recognized and widely used performance evaluation method. Among them, the results of the comparison test of the natural language evaluation indexes are shown in Table 1.

[0087] Table 1 comparative experimental results of the method proposed in the application and different methods on natural language evaluation indexes

[0088]

[0089]

[0090] Table 1 shows the comparative results of the performance of the method of the application and other advanced methods on natural language evaluation indexes. It can be seen that the scores of the method proposed in the application are almost better than those of the most advanced related methods in two data sets, which means that the method proposed in the application can generate more accurate and fluent natural language expressions on the evaluation criteria of natural language. In addition, it can be seen from Table 1 that the scores of the method proposed in the application on the two indexes of BLEU-3 and BLEU-4 are much higher than those of other methods, which means that the method proposed in the application can generate more natural and fluent natural language expressions compared with other methods.

[0091] In addition, we also conducted an ablation experiment on the IUX-Ray data set to prove the effectiveness of the visual contrast attention module (VCAM), the cross-modal contrast attention module (CAM) and the parallel attention module (PAM) in the method proposed in the application, and the ablation experiment still takes the Transformer model as the baseline. The results of the ablation experiment are shown in Table 2.

[0092] Table 2 ablation experiment of different modules of the method proposed in the application

[0093]

[0094] As shown in Table 2, the three modules proposed by the method make the model have different degrees of improvement in the performance of natural language evaluation indicators. By comparing Transformer and (a), it can be seen that VCAM plays a very positive role in improving the performance of the model, and (a) has made the model reach the performance level shown by the most advanced method in Table 1, which fully embodies that VCAM can effectively extract the effective visual features of similar past cases to assist the disease recognition and report generation of the current input image. By comparing Transformer and (b), it can be seen that CAM also effectively improves the performance of the model. By comparing (a) and (b), it can be seen that CAM has a more obvious improvement on the ROUGE indicator, which means that there is a longer common subsequence between the medical report generated by CAM and the real report, which fully embodies that CAM can effectively extract the effective words in the medical report of similar past cases to assist the report generation of the current input image. By comparing (a), (b) and (c), it can be seen that the combination of VCAM and CAM can make the performance of the model improve as a whole. By comparing (c) and Ours, it can be seen that using the PAM parallel module can more effectively guide the model to generate accurate reports than directly using the encoded features after splicing.

[0095] In addition to evaluating the experimental results of the present research on natural language evaluation indicators, clinical evaluation indicators are also used to evaluate the disease recognition of the model, including precision, recall and F1 score. The experimental results of the clinical evaluation indicators are shown in Table 3.

[0096] Table 3 Comparison of the experimental results of the method of the present application and different methods on clinical evaluation indicators

[0097]

[0098]

[0099] As shown in Table 3, the performance of the method proposed in this paper on the clinical evaluation indicators is higher than that of other advanced comparison methods. Among them, the method proposed in the present application shows strong performance on the two indicators of precision and F1 score, which means that the method proposed in the present application can more effectively and accurately identify the disease information existing in the image.

[0100] In summary, through comparison experiments and ablation experiments, the effectiveness of the method proposed in this paper in the medical report automatic generation task is verified. This shows that the method proposed in the present application can more effectively and accurately identify the abnormal lesion information existing in the input medical image, and the generated report is more fluent and natural, and closer to the writing result of human beings. The above performance improvement is because the method proposed in the present application can effectively use the visual and semantic information in past cases to help the report generation of the current input image.

[0101] Figure 5 The qualitative results of the proposed method are shown. The second and third columns of the figure show the retrieved similar past cases, the fourth column is the true report of the input image, the fifth column is the report result generated by the Transformer baseline model, and the last column is the medical report generated by the proposed method. The red marked words represent the generated report and the true report match, and the underlined words represent the generated report and the case report match. From Figure 5 it can be seen that the proposed method can generate more accurate medical reports, and the generated report and the true report have higher matching degree. From Figure 5 it can be seen that the proposed method can effectively use the effective words in the similar case report to assist the generation of the current report, and can generate sentences according to the information of the input image itself, such as “the lungs are clear” and “the skeletal structures are normal” in the third example.

[0102] Figure 6 The comparison between the report generated by the proposed method and the true report written by the doctor is shown. Among them, the underlined words represent that the generated report has more complete sentences than the true report, for example, in the first example, “the cardiomediasinal silhouette and vasculature are within normal limits for size and contour”, which is a correct expression of the image but is not written in the true report by the doctor. We believe that this is because in reality, radiologists may pay more attention to abnormal lesion areas displayed in the image and write related reports, thereby ignoring the recording and writing of other normal conditions, and the proposed method will effectively solve this problem and generate more complete and rich medical reports.

[0103] In summary, the proposed medical report automatic generation method based on cross-modal contrast attention has achieved good results in both quantitative natural language evaluation indicators and clinical evaluation indicators, and the comparison between the generated report and the true report in the qualitative analysis, which effectively utilizes the visual and text information of the past similar cases and has good application prospects in future practical applications.

Claims

1. A method for medical report automatic generation based on cross-modal contrastive attention, characterized in that: The method comprises an encoding and decoding stage, and the specific steps are as follows: Step (1) data preparation: obtain a medical report generation dataset and preprocess to obtain a training dataset, and construct a case library Wherein, the dataset contains medical image data and its corresponding medical report data, the preprocessing is used to obtain image space features X and global features of the medical image data And the medical report data is mapped to a vector by word embedding Case library Including a plurality of cases; Step (2) encoding stage, used to extract visual attention features V of the image I to be processed a , visual contrast attention features of the image I to be processed and similar cases, and cross-modal attention features of the image I to be processed and similar cases, wherein the visual attention features V a are used to represent the lesion information of the image I itself, the visual contrast attention features are used to represent the feature contrast of the image I and the case image, and the cross-modal attention features Cr a are used to represent the cross-modal matching of the effective words in the case report; The method for selecting similar cases is as follows: calculate the global features of the image I to be processed. With case library The cosine similarity of the global features of each case is used to select the top N cases with the highest similarity. K The cases corresponding to the cosine similarity scores are N, which are the most similar to the image I to be processed. K Case Step (3) decoding stage: generate a complete medical report of the image I to be processed for each similar case; input the encoded features extracted in the encoding stage into the parallel attention module to generate a word y at each time t t Splice the words generated at all times and finally get the complete report Y; select the medical report with the optimal natural language evaluation index as the final report.

2. The method for automatically generating a medical report based on cross-modal contrast attention according to claim 1, further comprising the following steps in data preprocessing: Using the feature extractor model CNN, pre-encoding is completed on all medical image data Img in the data set to obtain image space features X and global features Word embedding mapping is performed on the medical report data R in the data set to obtain a vector Wherein X, and The calculation formula is as follows: X=CNN(Img) Wherein, CNN(·) represents a convolutional neural network, max_pooling(·) represents maximum pooling downsampling, and Embedding(·) represents word embedding.

3. The method for automatically generating a medical report based on cross-modal contrast attention according to claim 2, further comprising the following steps in constructing a case library: The case library is obtained from all data of the training set, and the spatial features X of the dth data in the training set are extracted by preprocessing d , global features , and corresponding medical report word embedding vectors which together constitute a triple wherein the case d will be the dth case; subsequently, all data in the training set are operated as above to collectively constitute a case library wherein N represents the data amount of the training set.

4. The method for automatic generation of medical reports based on cross-modal contrastive attention according to claim 1, characterized in that: a visual attention feature V of the image to be processed a The calculation method is as follows: image space features X of the image to be processed I a visual attention module VAM is inputted to obtain visual attention features V of the image to be processed a as follows: V a = VAM(X I ) Wherein, VAM(·) represents the calculation formula of the visual attention module, VAM(X I ) = FFN(MHA(X I , X I , X I )) MHA(q, k, v) = [head1, head2,..., head n ]W O FFN(x)=max(0,xW1+b1)W2+b2 In the above formula, MHA(·) represents the calculation of multi-head attention features, which divides the input data into multiple subspaces and focuses on them by multiple attention heads respectively; FFN(·) represents a feedforward neural network; [·,…,·] represents the concatenation operation on tensors, head1,…,head n This represents the first to nth attention heads, used to focus on n subspaces of the input data. q, k, and v represent the input of the formula, and W... O These are the parameters for learning; head i This represents the i-th attention head, which will focus on the i-th subspace of the input data. Attn i (·) represents the calculation of its attention features, softmax(·) is a commonly used activation function, q i k i v i It is the i-th attention head i The input, i.e., the i-th subspace divided from the input data, d k is a fixed constant; x represents the input of the formula, W1 and W2 are the learned parameters, and b1 and b2 are the bias values. The MHA(·) and FFN(·) introduced here will continue to be used in subsequent calculations, where the learned parameters are obtained from the training data of different modules.

5. The method for automatic generation of medical reports based on cross-modal contrastive attention according to claim 4, characterized in that: The method for calculating the visual contrast attention feature of the similar case of the image to be processed I is as follows: 1), calculate the visual contrast features of the image I to be processed and each similar case, strengthen the lesion features unique to the image I, weaken the lesion features common to the image I and the similar case I,d , and keep the lesion information common to both unchanged, wherein the visual contrast features V I,d of the image I to be processed and the dth similar case case c are calculated according to the following formula: D I = X I - X I,d V c = X I + D I wherein X I,d represents a spatial feature of a case case I,d ; 2), the visual contrast feature is input into a visual contrast attention module VCAM to obtain a visual contrast attention feature, wherein the visual contrast feature V c The visual contrast attention feature obtained by inputting the visual contrast feature into the visual contrast attention module VCAM is represented by the following formula: Wherein, VCAM(·) represents the calculation formula of the visual contrast attention module.

6. The method for automatic generation of medical reports based on cross-modal contrastive attention according to claim 5, characterized in that: The calculation formula of the visual contrast attention module is as follows: VCAM(V c ) = FFN(MHA(V c , V c , V c )) 7. The method of claim 4, further comprising: obtaining a cross-modal attention feature Cr of the medical report of the d-th similar case case I,d a The calculation method is as follows: for visual contrast feature V c and the report of the similar case case I,d Using the cross-modal contrast attention module CAM, the multi-head attention mechanism is used to realize the cross-modal matching of V c and The cross-modal attention feature Cr a is obtained, and the specific calculation formula is as follows:​​ wherein CAM(·) denotes the computational formula of the cross-modal attention module, d e {1, …, N K}, the specific calculation steps are as follows:

8. The method for automatic generation of medical reports based on cross-modal contrastive attention according to claim 1, characterized in that: Further, the specific steps for generating a complete medical report of the image to be processed I by using each similar case include: Step (3.1) calculates the decoding result at each time: for each time t of the decoding stage, the word embedding vector generated at the previous time is taken as the input y 1:t-1 , and then the three encoding features obtained in the encoding stage are calculated in parallel using the parallel attention module PAM to obtain the hidden state h t of this time, and further obtain the word embedding vector y t of the word generated at this time, and the calculation formula is as follows: wherein, is a parameter of learning, is a bias value, PAM(·) represents the calculation formula of the parallel attention module, and the specific calculation steps are as follows: Wherein, Norm(·) represents a normalization operation; Step (3.2) generates a complete medical report: all the word embeddings generated at all times are taken as the output complete report Y: Y = {yl, y2,..., y T} Wherein, T represents the number of words of the generated complete medical report; Step (3.3) Select the best medical report: Select the best medical report from the N K cases obtained in step (2) by making them participate in the above operations and finally obtaining N K complete medical reports, respectively calculating the evaluation indexes obtained by comparing them with the true value, and selecting the best one as the final result.

Images