Medical image report generation method and system based on multi-granularity cross-modal alignment

Abstract

The invention discloses a medical image report generation method and system based on multi-granularity cross-modal alignment, and relates to the technical field of medical image analysis. Image and text related features are aligned through local-global-disease three granularities, the local granularity improves the discovery ability of the model for local lesions through a cross attention mechanism, and the global and disease granularities guide an encoder to focus on and align disease related visual features and text features through a similarity matrix and prototype clustering; according to the medical image report generation system, the modal difference between the image features and the text features is reduced, so that the image features are better associated with related diseases, the medical text containing related disease category inference is generated according to disease conditions, the medical image report generation system is realized, and the accuracy and practicability of the generated report are improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of medical image analysis, and in particular to a method and system for generating a medical image report based on multi-granularity cross-modality alignment. Background Art

[0002] In 1895, German physicist Wilhelm Roentgen first discovered X-rays, and imaging technology developed rapidly in the following period. The emergence of medical imaging is for medical research. It takes the human body or a part of the human body as the research object, and obtains the technology and processing process of internal tissue imaging in a non-invasive way. Among them, computed tomography (CT) and magnetic resonance imaging (MRI) have been widely used. However, with the development of technology, medical imaging has developed from fixed direction and fixed angle inspection to multi-angle and all-round inspection. Therefore, medical imaging technology has also become a common technology for medical examination, and medical imaging reports have become an important part of patients' electronic health records (EHR).

[0003] Medical imaging reports are documents that record the results of diagnosing and monitoring patients using imaging technology. The report contains the doctor's interpretation and diagnosis of the imaging results, and can provide clinicians with important information about the patient's health status. The task of automatically generating medical imaging reports aims to generate clear, correct, concise, complete, consistent, and coherent reports, i.e., 6C features, from given medical images. This usually requires the doctor who writes the report to have a solid understanding of the relevant profession and imaging diagnosis. Writing medical reports is also a part that is prone to errors for doctors, and may even result in misdiagnosis and other consequences. In addition, the proportion of medical technicians is declining, and the geographical distribution of professional medical technicians also tends to be in larger cities.

[0004] Medical imaging report generation technology is designed to solve diagnostic problems in medical imaging technology, improve diagnostic efficiency, narrow the gap in medical resources, and improve the application effect of medical imaging technology. In the case of scarce medical resources and insufficient experience or energy consumption of radiologists, using machines to automatically generate diagnostic reports can reduce the workload of radiologists and reduce the probability of missed diagnosis and misdiagnosis. It can also allow patients to quickly obtain their own health conditions, shorten the time patients wait for diagnostic results, reduce the workload of radiologists, improve clinical efficiency, and expand to underserved areas, which has significant practical significance. Therefore, it has become a hot research subject in the field of artificial intelligence and smart medicine.

[0005] At present, the research in the field of automatic generation of medical imaging reports is developing rapidly, and there have been many related research results. These research results show that the automatic generation of medical imaging reports using deep learning technology can generate relatively concise and consistent reports, providing valuable information for doctors. However, most of the existing research is based on a single-modal method such as an encoding-decoding structure. This single-modal method cannot take into account the matching relationship between images and text reports in the dataset in real situations, ignores the natural gap between image features and text features, lacks alignment of text reports and medical images, and the accuracy and reliability of the generated reports still exist. Summary of the invention

[0006] In view of the deficiencies in the prior art, the present invention provides a method and system for generating medical image reports based on multi-granularity cross-modality alignment, so as to solve the problem that the existing method for automatically generating medical image reports adopts a single-modality method and the generated reports are insufficient in accuracy and reliability.

[0007] A first aspect of the present invention provides a method for generating a medical imaging report based on multi-granularity cross-modality alignment, comprising the following steps:

[0008] Step 1: Construct a medical image dataset; the medical image dataset includes a number of image-text pairs, each of which includes a medical image and its corresponding medical image report;

[0009] Step 2: Divide the medical imaging dataset into training set, validation set and test set according to the set ratio;

[0010] Step 3: Construct a medical report generation model, and use the medical image data set to train and adjust the medical report generation model to obtain a trained medical report generation model;

[0011] The medical report generation model includes a visual feature extraction network, a text feature extraction network, a cross-modal alignment network based on a bidirectional cross-attention mechanism, a cross-modal alignment network based on a similarity matrix, a cross-modal alignment network based on prototype clustering, and a text generation network based on a Transformer;

[0012] During the training process, the input of the medical report generation model is an image-text pair in the training set, and the output is a medical image report generated according to the medical image in the input image-text pair. The training process is specifically as follows:

[0013] The visual feature extraction network uses a ResNet network to extract local and global features of the input medical image and obtain visual token sequences and the global image feature V k, where k is the number of the image-text pairs in the training set, and k = 0, 1, ..., N, N is the number of image-text pairs in the training set;

[0014] The visual token sequence is represented as: Among them, x i is the i-th visual token, i is the number of the visual token, and m is the number of visual tokens;

[0015] The text feature extraction network uses the BERT network to perform local and global feature extraction on the input medical imaging report to obtain text token sequences respectively. and the global text feature T k ;

[0016] The text token sequence is represented as Among them, y j is the jth text token, j is the number of the text token, and n is the number of text tokens;

[0017] The cross-modal alignment network based on the bidirectional cross-attention mechanism learns visual token sequences and a sequence of text tokens The soft matching between them is done by bidirectional cross-attention mechanism on the input visual token sequence and a sequence of text tokens Perform alignment;

[0018] Specifically, the visual token sequence output by the visual feature extraction network is and the text token sequence output by the text feature extraction network First, the dimension is reduced by the projection layer, and then the bidirectional cross-attention mechanism is used to calculate the cross-modal embedding: for the visual token x in the visual token sequence i , calculate the visual token x through the query matrix Q, key matrix K and value matrix V i The attention weights of all text tokens are combined and weighted to obtain the corresponding cross-modal text embedding Symmetrically, for a text token y in a text token sequence j , calculate its attention weight with all visual tags, and then obtain the cross-modal visual embedding Finally, the InfoNCE loss function is used to calculate each visual token x i and cross-modal text embeddings Compare the similarity and get the loss L av ; At the same time, the InfoNCE loss function is used for each text token y j and cross-modal visual embeddings Compare the similarity and get the loss L at, and finally the total loss L of the cross-modal alignment network based on the bidirectional cross-attention mechanism is obtained a =(L av +L at ) / 2;

[0019] The cross-modal alignment network based on the similarity matrix aligns global image features and global text features through the similarity matrix, maximizes the consistency between positive samples and random negative samples to achieve cross-modal alignment of medical images and medical image reports; the positive samples are real image-text pairs, and the random negative samples are random image-text pairs;

[0020] Specifically, first, the global image feature V k and the global text feature T k The dimension is reduced through the nonlinear projection layer, and the global image features V after dimension reduction are calculated respectively. k To the global text feature T k Cosine similarity of direction S v and the global text feature T k To the global image feature V k Cosine similarity of direction S t , the InfoNCE loss function is used to perform contrastive learning on the image-text pairs during training, and two symmetrical contrastive learning losses L are obtained. bv and L bt , and the total loss of the cross-modal alignment network based on the similarity matrix is ​​obtained by summing and averaging

[0021] The cross-modal alignment network based on prototype clustering realizes cross-modal alignment of global image features and global text features at the disease level through prototype clustering;

[0022] Specifically, we first define a cross-modal prototype set R = {r 1 ,r 2 ,…,r l ,…,r K}, where each learnable cross-modal prototype corresponds to a disease category, r l is the lth learnable cross-modal prototype, l is the number of the learnable cross-modal prototype; then the global image feature V is iterated by the Sinkhorn-Knopp algorithm. k and the global text feature T k Clustering, global image features V k and the global text feature T k is assigned to K learnable cross-modal prototypes to generate a global image feature V k The probability distribution P v and the global text feature Tk The probability distribution P t At the same time, the global image features V are calculated separately k and the global text feature T k The softmax probability Q of the cosine similarity with all learnable cross-modal prototypes v and Q t Finally, the cross-entropy loss function is used to achieve cross-modal alignment at the disease level, and the image cross-entropy loss L is obtained. cv =P v *logQ v and text cross entropy loss L ct =P t *logQ t , the total loss of the cross-modal alignment network based on prototype clustering is

[0023] The Transformer-based text generation network includes several layers of Transformer decoders that transform visual token sequences As image features, text is generated based on the input image features to obtain a medical imaging report;

[0024] The cross entropy loss function is used to calculate the difference between the generated medical image report G and the real medical image report A, and the loss L is obtained. d ;

[0025] The total loss function during training is:

[0026] L=αL a +βL b +θL c +ΩL d

[0027] Among them, α, β, θ and Ω are hyperparameters;

[0028] Step 4: Input the medical images in the test set into the visual feature extraction network in the trained medical report generation model, and generate a medical imaging report through the Transformer-based text generation network;

[0029] Specifically, the visual feature extraction network extracts the visual token sequence of the medical image, and the Transformer-based text generation network generates the medical imaging report according to the visual token sequence.

[0030] A second aspect of the present invention provides a medical image report generation system based on multi-granularity cross-modality alignment, which is used to implement the medical image report generation method based on multi-granularity cross-modality alignment, including:

[0031] A medical image acquisition module, used for acquiring medical images;

[0032] The medical report generation model is used to generate corresponding medical imaging reports based on the acquired medical images.

[0033] The third aspect of the present invention provides an electronic device, comprising: a processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor communicates with the memory via the bus, and when the machine-readable instructions are executed by the processor, the steps of the medical imaging report generation method based on multi-granularity cross-modality alignment are performed;

[0034] A fourth aspect of the present invention provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the steps of the medical imaging report generation method based on multi-granularity cross-modality alignment are executed.

[0035] Compared with the prior art, the beneficial effects of the present invention are:

[0036] The present invention proposes a medical image report generation method based on multi-granularity cross-modal alignment. The method proposes a multi-granularity cross-modal alignment provided in the second aspect of the present invention to solve the problem of difficulty in aligning disease-related medical descriptions in medical report generation tasks and the problem of not being able to focus well on abnormal information. The main idea is to align image and text-related features at three granularities of local-global-disease. The local granularity improves the model's ability to detect local lesions through a cross-attention mechanism. The global and disease granularity guide the encoder to focus on aligning disease-related visual features and text features through a similarity matrix and prototype clustering, reduce the inter-modal differences between image features and text features, and make image features better associated with related diseases. A medical text containing inferences of related disease categories is generated according to the disease condition, and a medical image report generation system is implemented to improve the accuracy and practicality of the generated report. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 is a flow chart of a method for generating a medical imaging report based on multi-granularity cross-modality alignment in an embodiment of the present invention;

[0038] Figure 2 This is a model structure diagram of a medical imaging report generation method based on multi-granularity cross-modality alignment in an embodiment of the present invention. DETAILED DESCRIPTION

[0039] The present invention aligns image and text-related features at three granularities: local, global, and disease, to solve the problem of difficulty in aligning disease-related medical descriptions and the problem of not being able to focus well on abnormal information in the task of generating medical reports. The local granularity improves the model's ability to discover local diseases, while the global and disease granularities focus on aligning disease-related visual features and text features, so that disease features are better associated with related diseases, and medical text containing disease-related disease category inferences is generated based on the disease situation, thereby improving the accuracy of generated reports.

[0040] Medical imaging report generation method based on multi-granularity cross-modality alignment, such as Figure 1 As shown, the following steps are included:

[0041] Step 1: Construct a medical image dataset; the medical image dataset includes a number of image-text pairs, each of which includes a medical image and its corresponding medical image report;

[0042] In this implementation, the process of collecting the medical image report data set is often affected by various external factors due to issues such as data privacy, resulting in the inability of medical images and medical image reports to form a correct image-text pair. Therefore, it is necessary to remove medical images that lack corresponding medical image reports and medical image reports that lack corresponding medical images.

[0043] Step 2: Divide the medical imaging dataset into training set, validation set and test set according to the set ratio;

[0044] Step 3: Build Figure 2 The medical report generation model shown in the figure is used to train and adjust the medical report generation model using the medical image data set to obtain a trained medical report generation model;

[0045] The medical report generation model includes a visual feature extraction network, a text feature extraction network, a cross-modal alignment network based on a bidirectional cross-attention mechanism, a cross-modal alignment network based on a similarity matrix, a cross-modal alignment network based on prototype clustering, and a text generation network based on a Transformer;

[0046] During the training process, the input of the medical report generation model is an image-text pair in the training set, and the output is a medical image report generated according to the medical image in the input image-text pair. The training process is specifically as follows:

[0047] The visual feature extraction network uses a ResNet network to extract local and global features of the input medical image and obtain visual token sequences and the global image feature V k, where k is the number of the image-text pairs in the training set, and k = 0, 1, ..., N, N is the number of image-text pairs in the training set;

[0048] The visual token sequence is represented as: Among them, x i is the i-th visual token, i is the number of the visual token, and m is the number of visual tokens;

[0049] The text feature extraction network uses the BERT network to perform local and global feature extraction on the input medical imaging report to obtain text token sequences respectively. and the global text feature T k ;

[0050] The text token sequence is represented as Among them, y j is the jth text token, j is the number of the text token, and n is the number of text tokens;

[0051] The cross-modal alignment network based on the bidirectional cross-attention mechanism learns visual token sequences and a sequence of text tokens The soft matching between them is done by bidirectional cross-attention mechanism on the input visual token sequence and a sequence of text tokens Perform alignment to achieve cross-modal alignment of local area information in medical images and keywords in medical imaging reports;

[0052] Specifically, the visual token sequence output by the visual feature extraction network is and the text token sequence output by the text feature extraction network First, the dimension is reduced by the projection layer, and then the bidirectional cross-attention mechanism is used to calculate the cross-modal embedding: for the visual token x in the visual token sequence i , calculate the visual token x through the query matrix Q, key matrix K and value matrix V i The attention weights of all text tokens are combined and weighted to obtain the corresponding cross-modal text embedding Symmetrically, for a text token y in a text token sequence j , calculate its attention weight with all visual tags, and then obtain the cross-modal visual embedding Finally, the InfoNCE (Information Noise Contrastive Estimation) loss function is used to estimate the loss of each visual token x i and cross-modal text embeddings Compare the similarity and get the loss Lav ; At the same time, the InfoNCE loss function is used for each text token y j and cross-modal visual embeddings Compare the similarity and get the loss L at , and finally the total loss L of the cross-modal alignment network based on the bidirectional cross-attention mechanism is obtained a =(L av +L at ) / 2;

[0053] The cross-modal alignment network based on the similarity matrix aligns global image features and global text features through the similarity matrix, maximizes the consistency between positive samples and random negative samples to achieve cross-modal alignment of medical images and medical image reports; the positive samples are real image-text pairs, and the random negative samples are random image-text pairs;

[0054] Specifically, first, the global image feature V k and the global text feature T k The dimension is reduced through the nonlinear projection layer, and the global image features V after dimension reduction are calculated respectively. k To the global text feature T k Cosine similarity of direction S v and the global text feature T k To the global image feature V k Cosine similarity of direction S t , the InfoNCE loss function is used to perform contrastive learning on the image-text pairs during training, and two symmetrical contrastive learning losses L are obtained. bv and L bt , to penalize the wrong matching of medical images and medical image reports, and to obtain the total loss of the cross-modal alignment network based on the similarity matrix by summing and averaging By maximizing the similarity of positive sample pairs and minimizing the similarity of negative sample pairs, the model is forced to learn global semantic alignment, ensuring that the model accurately associates medical images with their medical imaging reports at the overall semantic level;

[0055] The cross-modal alignment network based on prototype clustering focuses on cross-modal semantic consistency at the disease level, captures high-level disease associations through prototype clustering, and achieves cross-modal alignment of global image features and global text features at the disease level;

[0056] Specifically, we first define a cross-modal prototype set R = {r 1 ,r 2 ,…,r l ,…,r K}, where each learnable cross-modal prototype corresponds to a disease category, rl is the lth learnable cross-modal prototype, l is the number of the learnable cross-modal prototype; then the global image feature V is iterated by the Sinkhorn-Knopp algorithm. k and the global text feature T k Clustering, global image features V k and the global text feature T k is assigned to K learnable cross-modal prototypes to generate a global image feature V k The probability distribution P v and the global text feature T k The probability distribution P t At the same time, the global image features V are calculated separately k and the global text feature T k The softmax probability Q of the cosine similarity with all learnable cross-modal prototypes v and Q t Finally, the cross-entropy loss function is used to achieve cross-modal alignment at the disease level, and the image cross-entropy loss L is obtained. cv =P v *logQ v and text cross entropy loss L ct =P t *logQ t , the total loss of the cross-modal alignment network based on prototype clustering is

[0057] The Transformer-based text generation network includes several layers of Transformer decoders that transform visual token sequences As image features, text is generated based on the input image features to obtain a medical imaging report;

[0058] Specifically, first, position encoding is added to the input image features, and then several layers of Transformer decoders gradually generate the next word of the text. At each generation step, the word output in the previous step and the input image features are used to generate the probability distribution of the next word until the end symbol is generated to obtain the medical image report G. Finally, the cross entropy loss function is used to calculate the difference between the generated medical image report G and the real medical image report A to obtain the loss L. d ;

[0059] The total loss function during training is:

[0060] L=αL a +βL b +θL c +ΩL d

[0061] Among them, α, β, θ and Ω are hyperparameters;

[0062] Step 4: Input the medical images in the test set into the visual feature extraction network in the trained medical report generation model, and generate a medical imaging report through the Transformer-based text generation network;

[0063] Specifically, the visual feature extraction network extracts the visual token sequence of the medical image, and the Transformer-based text generation network generates the medical image report based on the visual token sequence;

[0064] In this embodiment, the medical image report generation method based on multi-granularity cross-modal alignment uses the public IU-XRay data set for training and testing, and selects representative medical image report generation models in recent years such as CoAtt, R2Gen, SSVE, etc. for comparative experiments. The results show that the best results are achieved in BLEU-1, BLEU-2, METEOR, and ROUGE indicators, and the second best results are also achieved in BLEU-3 and BLEU-4, indicating that the method proposed in the present invention can generate medical image reports with high word matching rates, not only can it match single words well, but also can match longer phrases well, and has achieved good text generation capabilities in the direction of medical image reports. The present invention is simple to deploy, effective in detection, and meets the requirements of the application.

[0065] Table 1 Comparison of experimental results of this patent method and mainstream methods in recent years on the IU-XRay dataset

[0066]

[0067]

[0068] This embodiment provides a medical image report generation system based on multi-granularity cross-modality alignment, which is used to implement the medical image report generation method based on multi-granularity cross-modality alignment, including:

[0069] A medical image acquisition module, used for acquiring medical images;

[0070] The medical report generation model is used to generate corresponding medical imaging reports based on the acquired medical images.

[0071] This embodiment provides an electronic device, including: a processor, a memory, and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor communicates with the memory via the bus, and when the machine-readable instructions are executed by the processor, the steps of the medical imaging report generation method based on multi-granularity cross-modality alignment are performed;

[0072] This embodiment provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the steps of the method for generating a medical image report based on multi-granularity cross-modality alignment are performed.

Claims

1. A method for generating medical image reports based on multi-granularity cross-modality alignment, characterized in that: The following steps are involved: Step 1: Construct a medical image dataset; the medical image dataset includes a number of image-text pairs, each of which includes a medical image and its corresponding medical image report; Step 2: Divide the medical imaging dataset into training set, validation set and test set according to the set ratio; Step 3: Construct a medical report generation model, and use the medical image data set to train and adjust the medical report generation model to obtain a trained medical report generation model; Step 4: Input the medical images in the test set into the visual feature extraction network in the trained medical report generation model, and generate medical imaging reports through the Transformer-based text generation network.

2. The method for generating a medical imaging report based on multi-granularity cross-modality alignment according to claim 1, characterized in that: The medical report generation model in step 3 includes a visual feature extraction network, a text feature extraction network, a cross-modal alignment network based on a bidirectional cross-attention mechanism, a cross-modal alignment network based on a similarity matrix, a cross-modal alignment network based on prototype clustering, and a text generation network based on Transformer; During the training process, the input of the medical report generation model is an image-text pair in the training set, and the output is a medical image report generated according to the medical image in the input image-text pair. The training process is specifically as follows: The visual feature extraction network uses a ResNet network to extract local and global features of the input medical image and obtain visual token sequences and the global image feature V k , where k is the number of the image-text pairs in the training set, and k = 0, 1, ..., N, N is the number of image-text pairs in the training set; The visual token sequence is represented as: Among them, x i is the i-th visual token, i is the number of the visual token, and m is the number of visual tokens; The text feature extraction network uses the BERT network to perform local and global feature extraction on the input medical imaging report to obtain text token sequences respectively. and the global text feature T k ; The text token sequence is represented as Among them, y j is the jth text token, j is the number of the text token, and n is the number of text tokens; The cross-modal alignment network based on the bidirectional cross-attention mechanism learns visual token sequences and a sequence of text tokens The soft matching between them is done by bidirectional cross-attention mechanism on the input visual token sequence and a sequence of text tokens Perform alignment; The cross-modal alignment network based on the similarity matrix aligns global image features and global text features through the similarity matrix, maximizes the consistency between positive samples and random negative samples to achieve cross-modal alignment of medical images and medical image reports; the positive samples are real image-text pairs, and the random negative samples are random image-text pairs; The cross-modal alignment network based on prototype clustering realizes cross-modal alignment of global image features and global text features at the disease level through prototype clustering; The Transformer-based text generation network includes several layers of Transformer decoders that transform visual token sequences As image features, text is generated according to the input image features to obtain a medical imaging report.

3. The method for generating a medical imaging report based on multi-granularity cross-modality alignment according to claim 2, characterized in that: The cross-modal alignment network based on the bidirectional cross-attention mechanism is specifically: the visual token sequence output by the visual feature extraction network and the text token sequence output by the text feature extraction network First, the dimension is reduced by the projection layer, and then the bidirectional cross-attention mechanism is used to calculate the cross-modal embedding: for the visual token x in the visual token sequence i , calculate the visual token x through the query matrix Q, key matrix K and value matrix V i The attention weights of all text tokens are combined and weighted to obtain the corresponding cross-modal text embedding Symmetrically, for a text token y in a text token sequence j , calculate its attention weight with all visual tags, and then obtain the cross-modal visual embedding Finally, the InfoNCE loss function is used to calculate each visual token x i and cross-modal text embeddings Compare the similarity and get the loss L av ; At the same time, the InfoNCE loss function is used for each text token y j and cross-modal visual embeddings Compare the similarity and get the loss L at , and finally the total loss L of the cross-modal alignment network based on the bidirectional cross-attention mechanism is obtained a =(L av +L at ) / 2.

4. The method for generating a medical imaging report based on multi-granularity cross-modality alignment according to claim 2, characterized in that: The cross-modal alignment network based on similarity matrix is ​​specifically as follows: First, the global image feature V k and the global text feature T k The dimension is reduced through the nonlinear projection layer, and the global image features V after dimension reduction are calculated respectively. k To the global text feature T k Cosine similarity of direction S v and the global text feature T k To the global image feature V k Cosine similarity of direction S t , the InfoNCE loss function is used to perform contrastive learning on the image-text pairs during training, and two symmetrical contrastive learning losses L are obtained. bv and L bt , and the total loss of the cross-modal alignment network based on the similarity matrix is ​​obtained by summing and averaging 5. The method for generating a medical imaging report based on multi-granularity cross-modality alignment according to claim 2, characterized in that: The cross-modal alignment network based on prototype clustering is specifically as follows: first, a cross-modal prototype set R = {r1, r2, ..., r l ,…,r K }, where each learnable cross-modal prototype corresponds to a disease category, r l is the lth learnable cross-modal prototype, l is the number of the learnable cross-modal prototype; then the global image feature V is iterated by the Sinkhorn-Knopp algorithm. k and the global text feature T k Clustering, global image features V k and the global text feature T k is assigned to K learnable cross-modal prototypes to generate a global image feature V k The probability distribution P v and the global text feature T k The probability distribution P t At the same time, the global image features V are calculated separately k and the global text feature T k The softmax probability Q of the cosine similarity with all learnable cross-modal prototypes v and Q t Finally, the cross-entropy loss function is used to achieve cross-modal alignment at the disease level, and the image cross-entropy loss L is obtained. cv =P v *logQ v and text cross entropy loss L ct =P t *logQ t , the total loss of the cross-modal alignment network based on prototype clustering is 6. The method for generating a medical imaging report based on multi-granularity cross-modality alignment according to claim 2, characterized in that: The Transformer-based text generation network uses the cross entropy loss function to calculate the difference between the generated medical image report G and the real medical image report A, and obtains the loss L d ; The total loss function during training is: L=αL a +βL b +θL c +ΩL d Among them, α, β, θ and Ω are hyperparameters.

7. The method for generating a medical image report based on multi-granularity cross-modality alignment according to claim 1, characterized in that: The step 4 specifically includes: the visual feature extraction network extracts the visual token sequence of the medical image, and the Transformer-based text generation network generates a medical image report according to the visual token sequence.

8. A medical image report generation system based on multi-granularity cross-modality alignment, used to implement the medical image report generation method based on multi-granularity cross-modality alignment according to any one of claims 1 to 7, characterized in that: include: A medical image acquisition module, used for acquiring medical images; The medical report generation model is used to generate corresponding medical imaging reports based on the acquired medical images.

9. An electronic device, characterized in that: include: A processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor and the memory communicate via the bus, and when the machine-readable instructions are executed by the processor, the steps of the method for generating a medical image report based on multi-granularity cross-modality alignment as described in any one of claims 1 to 7 are performed.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which, when executed by a processor, executes the steps of the method for generating a medical imaging report based on multi-granularity cross-modality alignment as described in any one of claims 1 to 7.

Images