Method and device for automatically generating MRI medical image report

Abstract

The invention discloses an MRI medical image report automatic generation method and device, and belongs to the technical field of medical image intelligent analysis. The MRI medical image report automatic generation method comprises the following steps: acquiring an MRI image; inputting the MRI image into a target multi-modal large model, and obtaining a target image report output by the target multi-modal large model; wherein the target multi-modal large model is obtained by training based on the following steps: training to obtain an image encoder based on a sample image and a sample text report corresponding to the sample image; based on the trained projection layer, mapping a vector space of an image encoder to a vector space of a large language model to obtain an initial multi-modal large model; and adjusting the initial multi-modal large model to obtain a target multi-modal large model. According to the MRI medical image report automatic generation method, the semantic coherence of the generated image report is improved, the clinical practice requirement can be fully met, and the clinical diagnosis efficiency and accuracy are improved.

Description

Technical Field

[0001] This application belongs to the technical field of intelligent medical image analysis, and particularly relates to a method and device for automatically generating MRI medical image reports. Background Art

[0002] Medical images are an indispensable tool in modern medical diagnosis. Especially in the diagnosis and prognosis evaluation of breast cancer, magnetic resonance imaging (MRI) has become an important imaging method due to its high soft tissue resolution and non-invasiveness. However, due to the complexity of breast tissue and the diversity of lesion types, it is difficult to interpret images, which easily leads to misdiagnosis or missed diagnosis. Moreover, doctors still need to manually write reports after completing image diagnosis, which is time-consuming and inefficient. In related technologies, there is a method of generating image reports through a model trained with a small-scale single-center dataset. The reports generated by this method lack semantic coherence and cannot fully meet the needs of clinical practice, and cannot effectively improve the diagnostic efficiency and accuracy. Summary of the Invention

[0003] This application aims to at least solve one of the technical problems existing in the related technologies. For this reason, this application provides a method and device for automatically generating MRI medical image reports, which improves the semantic coherence of the generated image reports, can fully meet the needs of clinical practice, and improves the clinical diagnostic efficiency and accuracy.

[0004] In a first aspect, this application provides a method for automatically generating an MRI medical image report, which includes:

[0005] Obtain an MRI image;

[0006] Input the MRI image into a target multi-modal large model to obtain a target image report output by the target multi-modal large model; wherein, the target multi-modal large model is trained based on the following steps:

[0007] Train an image encoder based on a sample image and a sample text report corresponding to the sample image;

[0008] Based on the trained projection layer, map the vector space of the image encoder to the vector space of the large language model to obtain an initial multi-modal large model;

[0009] Adjust the initial multi-modal large model to obtain the target multi-modal large model.

[0010] According to the MRI medical image report automatic generation method provided by the embodiments of the present application, an image encoder is trained through sample images and sample text reports, and a projection layer is introduced to map the vector space of the image encoder to the vector space of the large language model, so as to train the target multi-modal large model. By adding a visual layer and a projection layer to the large language model, cross-modal alignment of 3D image features and text reports is achieved, enabling the model to accurately understand image features, thereby generating high-quality and clinically valuable medical image reports, improving the semantic coherence of the generated image reports, fully meeting the needs of clinical practice, and enhancing the efficiency and accuracy of clinical diagnosis.

[0011] The MRI medical image report automatic generation method according to an embodiment of the present application, based on the sample images and the corresponding sample text reports of the sample images, training an image encoder, includes:

[0012] Input the sample images into the image encoder to obtain the first visual features corresponding to the sample images output by the image encoder;

[0013] Input the sample text reports corresponding to the sample images into the text encoder to obtain the first text features corresponding to the sample text reports output by the text encoder;

[0014] Based on the first visual features and the first text features, using the contrastive learning method, train the image encoder.

[0015] The MRI medical image report automatic generation method according to an embodiment of the present application, based on the first visual features and the first text features, using the contrastive learning method, training the image encoder, includes:

[0016] Process the first visual features based on the weight matrix of the visual projection layer to obtain second visual features; process the first text features based on the weight matrix of the text projection layer to obtain second text features;

[0017] Based on the second visual features and the second text features, using the loss function of contrastive learning, train the image encoder to obtain the trained image encoder.

[0018] The MRI medical image report automatic generation method according to an embodiment of the present application, adjusting the initial multi-modal large model to obtain the target multi-modal large model, includes:

[0019] Adopt the low-rank adaptation technology, and based on the original weight matrix in the initial multi-modal large model, decompose the updated weight matrix in the initial multi-modal large model to obtain the decomposed updated weight matrix;

[0020] Based on the decomposed updated weight matrix and the original weight matrix, obtain the target weight matrix corresponding to the initial multi-modal large model;

[0021] Based on the sample image, the image encoder, and the projection layer, train the target weight matrix to obtain the target multi-modal large model.

[0022] For the MRI medical image report automatic generation method according to an embodiment of the present application, the training of the target weight matrix based on the sample image, the image encoder, and the projection layer to obtain the target multi-modal large model includes:

[0023] Input the sample image into the image encoder to obtain the second visual feature output by the image encoder;

[0024] Input the second visual feature into the projection layer to obtain the third visual feature output by the projection layer;

[0025] Input the third visual feature into the updated initial multi-modal large model to reversely train the target weight matrix to obtain the target multi-modal large model.

[0026] For the MRI medical image report automatic generation method according to an embodiment of the present application, the trained projection layer includes a pooling layer and a fully connected layer, and the output end of the pooling layer is connected to the input end of the fully connected layer; the mapping of the vector space of the image encoder to the vector space of the large language model based on the trained projection layer includes:

[0027] Input the sample image into the image encoder to obtain the first visual feature corresponding to the sample image output by the image encoder;

[0028] Input the first visual feature into the pooling layer to obtain the fourth visual feature;

[0029] Input the fourth visual feature into the fully connected layer to obtain the fifth visual feature, so as to map the vector space of the image encoder to the vector space of the large language model.

[0030] In a second aspect, the present application provides an MRI medical image report automatic generation device, including:

[0031] A first processing module for acquiring an MRI image;

[0032] A second processing module for inputting the MRI image into the target multi-modal large model to obtain the target image report output by the target multi-modal large model; wherein, the target multi-modal large model is trained based on the following steps:

[0033] Training an image encoder based on a sample image and a sample text report corresponding to the sample image;

[0034] Based on the trained projection layer, the vector space of the image encoder is mapped to the vector space of the large language model to obtain an initial multimodal large model;

[0035] The initial multimodal large model is adjusted to obtain the target multimodal large model.

[0036] According to the automatic generation device of MRI medical imaging reports provided in the embodiment of the present application, an image encoder is obtained by training sample images and sample text reports, and a projection layer is introduced to map the vector space of the image encoder to the vector space of the large language model, thereby training a target multimodal large model. The visual layer and the projection layer are added to the large language model to achieve cross-modal alignment of 3D image features and text reports, so that the model can accurately understand the image features, thereby generating high-quality and clinically valuable medical imaging reports, improving the semantic coherence of the generated imaging reports, fully meeting the needs of clinical practice, and improving the efficiency and accuracy of clinical diagnosis.

[0037] In a third aspect, the present application provides an electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the computer program, the method for automatically generating MRI medical imaging reports as described in the first aspect above is implemented.

[0038] In a fourth aspect, the present application provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method for automatically generating an MRI medical imaging report as described in the first aspect above.

[0039] In a fifth aspect, the present application provides a computer program product, comprising a computer program, which, when executed by a processor, implements the method for automatically generating MRI medical imaging reports as described in the first aspect above.

[0040] The above one or more technical solutions in the embodiments of the present application have at least one of the following technical effects:

[0041] An image encoder is obtained by training with sample images and sample text reports. A projection layer is introduced to map the vector space of the image encoder to the vector space of the large language model, thereby training the target multimodal large model. The visual layer and projection layer are added to the large language model to achieve cross-modal alignment of 3D image features and text reports, enabling the model to accurately understand image features, thereby generating high-quality and clinically valuable medical imaging reports, improving the semantic coherence of the generated imaging reports, fully meeting the needs of clinical practice, and improving the efficiency and accuracy of clinical diagnosis.

[0042] Furthermore, by using the low-rank adaptation technique to decompose the updated weight matrix in the initial multi-modal large model, the updated part can be fine-tuned, which can significantly reduce the parameters required for updating, retain the quality of the original model, increase the fine-tuning speed, improve the adaptability of the model, maintain the generalization ability of the pre-trained model, broaden the applicable scenarios, and further enhance the practicality of the model.

[0043] Additional aspects and advantages of the present application will be given in part in the following description, become apparent in part from the following description, or be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] The above and / or additional aspects and advantages of the present application will become apparent and be readily understood from the description of the embodiments in conjunction with the following drawings, wherein:

[0045] Figure 1 is one of the schematic flowcharts of the method for automatically generating an MRI medical image report provided by an embodiment of the present application;

[0046] Figure 2 is another schematic flowchart of the method for automatically generating an MRI medical image report provided by an embodiment of the present application;

[0047] Figure 3 is yet another schematic flowchart of the method for automatically generating an MRI medical image report provided by an embodiment of the present application;

[0048] Figure 4 is the schematic diagram of the model architecture of the method for automatically generating an MRI medical image report provided by an embodiment of the present application;

[0049] Figure 5 is the schematic diagram of the result comparison of the method for automatically generating an MRI medical image report provided by an embodiment of the present application;

[0050] Figure 6 is the fourth schematic flowchart of the method for automatically generating an MRI medical image report provided by an embodiment of the present application;

[0051] Figure 7 is the schematic diagram of the structure of the device for automatically generating an MRI medical image report provided by an embodiment of the present application;

[0052] Figure 8 is the schematic diagram of the structure of the electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0053] Next, the technical solutions in the embodiments of the present application will be clearly described in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art belong to the scope of protection of the present application.

[0054] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, rather than to describe a specific order or sequence. It should be understood that such data can be interchanged under appropriate circumstances so that the embodiments of the present application can be implemented in an order other than those illustrated or described herein, and the objects distinguished by "first", "second", etc. are usually of the same category, and do not limit the number of objects. For example, the first object can be one or more. In addition, "and / or" in the specification and claims means at least one of the connected objects, and the character " / " generally means an "or" relationship between the associated objects before and after.

[0055] Next, in conjunction with the accompanying drawings, the MRI medical image report automatic generation method, the MRI medical image report automatic generation device, the electronic device, and the readable storage medium provided by the embodiments of the present application will be described in detail through specific embodiments and their application scenarios.

[0056] Among them, the MRI medical image report automatic generation method can be applied to a terminal, and can be specifically executed by hardware or software in the terminal.

[0057] The terminal includes, but is not limited to, portable communication devices such as mobile phones or tablet computers having a touch-sensitive surface (for example, a touch screen display and / or a touchpad). It should also be understood that in some embodiments, the terminal may not be a portable communication device, but a desktop computer having a touch-sensitive surface (for example, a touch screen display and / or a touchpad).

[0058] In the following various embodiments, a terminal including a display and a touch-sensitive surface is described. However, it should be understood that the terminal may include one or more other physical user interface devices such as a physical keyboard, a mouse, and a joystick.

[0059] The MRI medical image report automatic generation method provided by the embodiments of the present application. The execution subject of the MRI medical image report automatic generation method may be an electronic device or a functional module or functional entity in the electronic device that can implement the MRI medical image report automatic generation method. The electronic devices mentioned in the embodiments of the present application include, but are not limited to, mobile phones, tablet computers, computers, cameras, and wearable devices, etc. Here, taking the electronic device as the execution subject as an example, the MRI medical image report automatic generation method provided by the embodiments of the present application will be described.

[0060] As Figure 1 shown, the method for automatically generating an MRI medical image report includes: step 110 and step 120.

[0061] Step 110: Obtain an MRI image;

[0062] In this step, the MRI image is magnetic resonance imaging. MRI scanning can generate a three-dimensional voxel grid, and spatial context is extracted through 3D convolution or hierarchical processing.

[0063] The MRI image can be magnetic resonance imaging corresponding to parts or tissues such as the breast, thyroid, nervous system, or cardiovascular system.

[0064] In some embodiments, the MRI image can be normalized. For example, when the size of the MRI image is 32 - 256 - 256, the pixel values can be normalized to 0 - 1 by dividing by 255.0.

[0065] Step 120: Input the MRI image into a target multi-modal large model to obtain a target image report output by the target multi-modal large model;

[0066] In this step, the target multi-modal large model is a pre-trained model that can perform feature extraction and other processing on the MRI image to generate a target image report corresponding to the MRI image.

[0067] The target image report can include information such as the lesion location, size, and grading in the MRI image. [[ID=2⑥]]

[0068] In some embodiments, after the MRI image is input into the target multi-modal large model, the target multi-modal large model can receive a first instruction and, in response to the first instruction, output a target image report corresponding to the MRI image.

[0069] In this embodiment, the first instruction can include a voice instruction.

[0070] For example, the user can input a report generation instruction, such as, "Please generate a medical report." After receiving the report generation instruction, the target multi-modal large model can output a target image report in response to the report generation instruction.

[0071] As Figure 2 shown, the target multi-modal large model can be trained through the following steps:

[0072] Step 130: Based on a sample image and a sample text report corresponding to the sample image, train an image encoder;

[0073] In this step, the sample image is image data used to train the image encoder, and the sample image corresponds to the sample text report.

[0074] The sample image may include magnetic resonance imaging corresponding to parts or tissues such as the breast, thyroid, nervous system, or cardiovascular system.

[0075] The sample text report includes a detailed description of the content of the sample image or semantic information related to the image.

[0076] By learning the features of the sample image, the image encoder can extract visual information related to the image content. The text report provides semantic guidance for the image encoder, and the image encoder can learn how to align the visual information with the text information, thus realizing the fusion of multimodal information.

[0077] After obtaining the dataset, it can be divided into a training set, a validation set, and a test set according to the target ratio (such as 8:1:1).

[0078] Among them, the training set is used for training the model, the validation set can be used to optimize the model parameters, and the test set can be used to evaluate the performance of the model.

[0079] Step 140: Based on the trained projection layer, map the vector space of the image encoder to the vector space of the large language model to obtain an initial multimodal large model;

[0080] In this step, the projection layer can align the output features of the image encoder to the vector space of the large language model, so that the large language model can understand the image features in the sample image, that is, obtain a new large language model (initial multimodal large model).

[0081] By processing the vector space (output features) of the image encoder through the projection layer, the image features can be aligned with the text features in the vector space of the large language model.

[0082] Methods such as contrastive learning or supervised learning can be used to ensure that the image features and text features have a similar semantic structure in the vector space of the large language model.

[0083] The projection layer can be constructed by two layers of multi-layer perceptrons to achieve the alignment of visual data and text data. During the training of the projection layer, the parameters of the image encoder and the large language model can be frozen, and only the weight matrix of the projection layer is trained.

[0084] Step 150: Adjust the initial multimodal large model to obtain the target multimodal large model.

[0085] In this step, the initial multimodal large model includes an image encoder, a projection layer, and a large language model.

[0086] Adjust the initial multi-modal large model so that it can adapt to specific tasks and data distributions.

[0087] For example, the initial multi-modal large model can be fine-tuned using a dataset for a specific task, which can include new image and text pairs, or annotation data for a specific task.

[0088] According to the task requirements, the loss function and optimization strategy of the model can be adjusted, and the hyperparameters of the model (such as learning rate and regularization term, etc.) can be adjusted to obtain the target multi-modal large model.

[0089] During the process of adjusting the initial multi-modal large model, the large language model, image encoder, and projection layer can be unfrozen, and supervised fine-tuning can be performed on the training set to further learn knowledge and interaction capabilities from manual annotations.

[0090] The inventors found during the R & D process that related technologies mainly focus on 2D images or 3D-CT data (such as M3D and 3D-CT-GPT, etc.). The differences between the imaging principles of MRI and CT lead to difficulties in model migration: the high soft tissue resolution of MRI is more suitable for breast cancer detection, but its dynamic enhancement characteristics and complex artifacts increase the difficulty of feature extraction.

[0091] In this application, a target multi-modal large model is trained through sample MRI images corresponding to other tissues such as the breast, and an imaging report of the MRI image is obtained based on the target multi-modal large model, solving the problem of difficult model migration between MRI and CT.

[0092] According to the MRI medical imaging report automatic generation method provided by the embodiments of this application, an image encoder is trained through sample images and sample text reports, and a projection layer is introduced to map the vector space of the image encoder to the vector space of the large language model, thereby training the target multi-modal large model. By adding a visual layer and a projection layer to the large language model, cross-modal alignment of 3D imaging features and text reports is achieved, enabling the model to accurately understand imaging features, thereby generating high-quality and clinically valuable medical imaging reports, improving the semantic coherence of the generated imaging reports, fully meeting the needs of clinical practice, and enhancing the efficiency and accuracy of clinical diagnosis.

[0093] Figure 4 The overall framework of the target multi-modal large model is exemplified. The target multi-modal large model adopts a visual and language joint framework, and the core components of the target multi-modal large model include an image encoder, a projection layer, and an initial multi-modal large model.

[0094] The training of the model mainly includes three parts: pre-training the 3D image encoder, training the projection layer, and fine-tuning the initial multi-modal large model.

[0095] The training process of the target multi-modal large model will be specifically described below from the above three training parts respectively.

[0096] In some embodiments, step 130 may include:

[0097] Input the sample image into the image encoder to obtain the first visual feature corresponding to the sample image output by the image encoder;

[0098] Input the sample text report corresponding to the sample image into the text encoder to obtain the first text feature corresponding to the sample text report output by the text encoder;

[0099] Based on the first visual feature and the first text feature, use the contrastive learning method to train the image encoder.

[0100] In this embodiment, the image encoder is a deep learning model, which can be based on a convolutional neural network or a Vision Transformer (ViT) architecture. The image encoder can convert the input image into a feature vector that can capture the key visual information in the image.

[0101] After inputting the sample image into the image encoder, the image encoder can gradually extract the features of the image through convolutional layers, pooling layers or fully connected layers (or the self-attention mechanism of the Transformer), and finally output the first visual feature.

[0102] The text encoder can be a language model based on the Transformer architecture (such as BERT or RoBERTa, etc.).

[0103] The text encoder can convert the input text into a feature vector that can capture the semantic information of the text.

[0104] After inputting the sample text report into the text encoder, the text encoder can gradually extract the features of the text through word embedding, multi-head self-attention mechanism and feed-forward network, and finally output the first text feature.

[0105] Contrastive learning is an unsupervised or weakly supervised learning method. By using the contrastive learning method to train the image encoder, the image features and text features can be aligned in the feature space.

[0106] Contrastive learning can learn feature representations through the comparison of positive sample pairs and negative sample pairs. In multi-modal learning, the positive sample pair is the image feature and the corresponding text feature, and the negative sample pair is the image feature and the non-corresponding text feature.

[0107] In some embodiments, based on the first visual feature and the first text feature, using the contrastive learning method to train the image encoder may include:

[0108] Process the first visual feature based on the weight matrix of the visual projection layer to obtain a second visual feature; process the first text feature based on the weight matrix of the text projection layer to obtain a second text feature;

[0109] Based on the second visual feature and the second text feature, use the loss function of contrastive learning to train the image encoder to obtain a trained image encoder.

[0110] In this embodiment, the visual projection layer is a linear layer or a fully connected layer. The weight matrix of the visual projection layer is used to convert the first visual feature into a second visual feature. For example, the second visual feature can be obtained based on the product between the weight matrix of the visual projection layer and the first visual feature.

[0111] The text projection layer is a linear layer or a fully connected layer. The weight matrix of the text projection layer is used to convert the first text feature into a second text feature. For example, the second text feature can be obtained based on the product between the weight matrix of the text projection layer and the first text feature.

[0112] Train the image encoder through the loss function of contrastive learning so that the image features and text features can be aligned in the feature space.

[0113] The loss function of contrastive learning can include contrast loss and InfoNCE loss. The objective of the loss function is to minimize the distance between positive sample pairs and maximize the distance between negative sample pairs.

[0114] The positive sample pair is the second visual feature and the corresponding second text feature, and the negative sample pair is the second visual feature and the second text feature that does not correspond.

[0115] The following combines Figure 3 , and further illustrates the pre-training process of the image encoder.

[0116] The image encoder can be a 3DVIT of 12-layer Transformer. 3DVIT is a 3D vision model based on the Transformer architecture. By extending the Transformer architecture to 3D space, it can effectively extract complex features in 3D images.

[0117] Contrastive learning can be used to obtain a 3D image encoder that can well extract MRI features.

[0118] The text encoder can be Bert.

[0119] The 3D sample MRI image I ∈ R D×H×W can be input into the image encoder, where D, H, and W are the depth, height, and width of the sample image respectively.

[0120] The sample image can be divided into a series of non-overlapping 3D patches (small blocks) to adapt to the Transformer architecture, and the size of each patch is P D ×P H ×P W , where P D 、P H and P W are the depth, height and width of the patch respectively.

[0121] Then, after extraction from 3DVIT, the first visual feature f = 3DViT(I) can be obtained, and the text report corresponding to the sample image is input into Bert to obtain the first text feature t = BERT(T), where BERT represents processing by the text encoder Bert, and T is the description of the input text.

[0122] Align the two feature vectors: f′=W f f, where W f is the weight matrix of the visual projection layer, f′ is the second visual feature, f is the first visual feature, t′=W t t, where W t is the weight matrix of the text projection layer, t′ is the second text feature, and t is the first text feature.

[0123] The loss function of contrastive learning can be used to perform gradient descent calculation to obtain a good weight for the 3D image encoder.

[0124] Among them, the formula of the loss function is as follows:

[0125]

[0126] Among them, L is the value of the loss function, N is the number of samples in the batch, i is the index of the current sample, j is the index of the comparison sample, sim is the similarity function, f i ′ is the second visual feature corresponding to the i-th sample image, t′ i is the second text feature corresponding to the i-th sample image, t′ j is the second text feature corresponding to the j-th sample image, and τ is a temperature hyperparameter used to adjust the sharpness of the similarity distribution.

[0127] For each sample i, its second visual feature f can be calculated i ′ and the second text feature t′ of all samples j The similarity between sim(f i ′,t′ j) Calculate the InfoNCE loss using the above formula. The loss value can be used to characterize the distance difference between the positive sample pairs and the negative sample pairs. Then, through the backpropagation algorithm, the parameters of the image encoder and the weight matrix can be adjusted according to the loss value, so that the image features and the corresponding text features are closer in the feature space and farther from other irrelevant features. Repeat the above steps until the model converges.

[0128] During the pre-training process, the learning rate can be set to 1×10 -4 , and the batch size is set to 8×4 (i.e., 8 validation sets and 4 training sets). Parallel training can be performed based on four GPUs to accelerate the convergence of the model and improve the training efficiency.

[0129] In some embodiments, based on the trained projection layer, mapping the vector space of the image encoder to the vector space of the large language model may include:

[0130] Input the sample image into the image encoder to obtain the first visual feature corresponding to the sample image output by the image encoder;

[0131] Input the first visual feature into the pooling layer to obtain the fourth visual feature;

[0132] Input the fourth visual feature into the fully connected layer to obtain the fifth visual feature, so as to map the vector space of the image encoder to the vector space of the large language model.

[0133] In this embodiment, the trained projection layer includes a pooling layer and a fully connected layer, and the output end of the pooling layer is connected to the input end of the fully connected layer.

[0134] The first visual feature is the feature corresponding to the sample image output by the image encoder.

[0135] Input the first visual feature into the 3D pooling layer to obtain the fourth visual feature, which can retain the original image information while effectively reducing the number of tokens.

[0136] Input the fourth visual feature into the fully connected layer to obtain the fifth visual feature, that is, the fourth visual feature can be mapped to the vector space of the large language model through a mapping.

[0137] In the actual execution process, a fully connected layer can be used as the projection layer, and a 3D pooling layer can be added before the fully connected layer to reduce the number of tokens while retaining the original image information, thereby reducing the training cost.

[0138] Input the first visual feature f obtained from the image encoder 3DVIT into the 3D pooling layer to obtain a pooled fourth visual feature f p : f p=Pool(f), where f p is the fourth visual feature, f is the first visual feature, and then the fourth visual feature f p Mapped to the vector space f of the large language model through the fully connected layer l :

[0139] f l =W×f p +b

[0140] Among them, W is the weight matrix of the fully connected layer, b is the bias vector of the fully connected layer, and f p is the fourth visual feature, f l It is the fifth visual feature.

[0141] During the projection layer training phase, only the projection layer consisting of two layers of multi-layer perceptrons is trained. For example, the cross entropy function can be used to compare the difference between the predicted tokens and the true label labels to train the weight matrix of the projection layer.

[0142] The trained projection layer can align visual data and text data, and the parameters of the image encoder and the large language model can be kept frozen during training.

[0143] During training, the batch size can be set to 4×2 and the learning rate can be set to 1×10 -4 , to ensure the stability and efficiency of the training process.

[0144] In some embodiments, step 150 may include:

[0145] Low-rank adaptation technology is used to decompose the update weight matrix in the initial multimodal large model based on the original weight matrix in the initial multimodal large model to obtain the decomposed update weight matrix;

[0146] Based on the decomposed update weight matrix and the original weight matrix, the target weight matrix corresponding to the initial multimodal large model is obtained;

[0147] Based on the sample image, image encoder and projection layer, the target weight matrix is trained to obtain the target multimodal large model.

[0148] In this embodiment, low-rank adaptation technology (LoRA) can reduce the number of parameters by decomposing the weight matrix, thereby reducing computational cost and memory usage while maintaining model performance.

[0149] For the initial multimodal large model, its original weight matrix can be kept unchanged, and only the training update part can be fine-tuned. The low-rank adaptation technology can be used to decompose the updated weight matrix into the product of two low-rank matrices.

[0150] The decomposed updated weight matrix can be combined with the original weight matrix to obtain the target weight matrix.

[0151] The initial multi-modal large model can be Phi3-4B. Phi3-4B achieves an optimized balance between computational efficiency and model performance through a compact architecture with 4B parameters. While maintaining strong language understanding capabilities, it significantly reduces the video memory occupancy and hardware threshold during inference, and is especially suitable for deployment in resource-constrained scenarios. Secondly, the model adopts designs such as dynamic sparse attention mechanism and hierarchical parameter sharing, making its performance in tasks such as text generation and logical reasoning relatively high.

[0152] In some embodiments, based on the sample image, the image encoder, and the projection layer, training the target weight matrix to obtain the target multi-modal large model may include:

[0153] Input the sample image into the image encoder to obtain the second visual feature output by the image encoder;

[0154] Input the second visual feature into the projection layer to obtain the third visual feature output by the projection layer;

[0155] Input the third visual feature into the updated initial multi-modal large model to backtrain the target weight matrix and obtain the target multi-modal large model.

[0156] In this embodiment, input the third visual feature into the updated initial multi-modal large model, update the target weight matrix through the backpropagation algorithm, and can adjust the target weight matrix by optimizing the loss function to make the model perform better in multi-modal tasks. Repeat the above steps until the model converges to obtain the target multi-modal large model.

[0157] During the actual execution process, when adjusting the initial multi-modal large model, the parameters of the large model, the image encoder, and the projection layer can be unfrozen and supervised fine-tuning can be performed on the training set.

[0158] The low-rank adaptation technique can achieve fine-tuning of the model by only training low-rank matrices and then injecting these parameters into the original model.

[0159] The updated weight matrix ΔW can be decomposed into two low-rank matrices A and B:

[0160] ΔW = BA

[0161] where B ∈ R d×r , A ∈ R r×k , ΔW ∈ R d×k , and r << d, d is the number of rows of the original weight matrix, k is the number of columns of the original weight matrix, r is the rank of the low-rank matrix, and is usually much smaller than d.

[0162] Based on the updated weight matrix and the original weight matrix, the target weight matrix can be obtained:

[0163] W = W0 + ΔW

[0164] where W is the target weight matrix, W0 is the original weight matrix, and ΔW is the decomposed updated weight matrix.

[0165] Before decomposing the updated weight, the low-rank matrices A and B can be initialized as zero matrices. During the training process, the cross-entropy loss function can be used to calculate the loss value, and the weights of the model can be updated through backpropagation.

[0166] During the training process, the weights of the model are updated through backpropagation, including the updated weight matrix of the large model, the weights of the visual encoder, and the weights of the projection layer.

[0167] That is, all linear layers of the model can be fine-tuned, including Q (query layer), K (key layer), V (value layer) in the attention mechanism and the final output layer. And set the rank r of the low-rank matrix to 16, the scaling factor lora_alpha to 32, the dropout ratio to 0.1, and the maximum length to 512.

[0168] Through the image encoder, the second visual feature corresponding to the sample image can be obtained, then fp can be obtained through the 3D pooling layer, and then the third visual feature can be obtained through the fully connected layer. Finally, the third visual feature is input into the large model for backpropagation training to obtain the target multi-modal large model.

[0169] During the process of fine-tuning the initial multi-modal large model, the batch size can be adjusted to 2×1, and the learning rate can be reduced to 5×10^(-5) to achieve more refined model adjustment.

[0170] According to the MRI medical image report automatic generation method provided by the embodiments of the present application, by decomposing the updated weight matrix in the initial multi-modal large model through the low-rank adaptation technology to fine-tune the updated part, the parameters required for updating can be significantly reduced, the quality of the original model can be retained, the fine-tuning speed can be increased, the adaptability of the model can be improved, the generalization ability of the pre-trained model can be maintained, the applicable scenarios can be broadened, and the practicality of the model can be further improved.

[0171] During the training process of the above three stages, the Adam optimizer can be used, and the cosine annealing learning rate adjustment strategy can be adopted to dynamically adjust the learning rate and improve the convergence performance of the model. At the same time, DeepSpeed can be used for parallel training to accelerate the training process and improve the training efficiency. And the BF16 mixed precision training strategy can be adopted to further improve the training efficiency, reduce the training time and resource consumption.

[0172] The above training process can be implemented under the Pytorch framework and carried out on four 48GB A40 GPUs to ensure the efficiency and stability of model training.

[0173] As Figure 6 shown, the target multimodal large model can be trained through the training set, and then its performance can be evaluated through the test set. For example, BLEU, MEMTOR, and ROUGE scores can be used as the core criteria for measuring the model's performance. Among them, the BLEU score focuses on evaluating the matching degree and accuracy between the generated text and the reference text at the lexical level, the MEMTOR score focuses on the professionalism and accuracy of medical texts, and the ROUGE score focuses on whether the generated text contains as many important information points in the reference text as possible.

[0174] As Figure 5 illustrates the comparison between the output example (Answer) corresponding to the target multimodal large model provided by the embodiments of the present application and the original imaging report (Ground Truth). Through comparison, it can be seen that the prediction accuracy and precision of the target multimodal large model provided by the embodiments of the present application are relatively high.

[0175] The model architecture and training method of the target multimodal large model provided by the embodiments of the present application have high flexibility and can adapt to different dataset and task requirements. For example, it can be applied to breast datasets, thyroid datasets, or datasets corresponding to other tissues.

[0176] By using 3DViT as the image encoder, combining with the Phi3-4B large language model, and introducing the LoRA fine-tuning technology, it is possible to effectively process 3D medical image data.

[0177] After training, the target multimodal large model can be used as an auxiliary tool to automatically generate medical imaging reports, providing accurate and efficient diagnostic reports for doctors, reducing diagnostic errors caused by fatigue or lack of experience, and thus significantly improving the diagnostic efficiency and accuracy.

[0178] The following describes the MRI medical imaging report automatic generation device provided by the present application. The MRI medical imaging report automatic generation device described below can be mutually referred to the MRI medical imaging report automatic generation method described above.

[0179] For the MRI medical imaging report automatic generation method provided by the embodiments of the present application, the execution subject can be the MRI medical imaging report automatic generation device. In the embodiments of the present application, taking the MRI medical imaging report automatic generation device to execute the MRI medical imaging report automatic generation method as an example, the MRI medical imaging report automatic generation device provided by the embodiments of the present application is described.

[0180] The embodiment of the present application also provides an automatic MRI medical image report generation device.

[0181] As Figure 7 shown, the automatic MRI medical image report generation device includes: a first processing module 710 and a second processing module 720.

[0182] The first processing module 710 is used to obtain MRI images;

[0183] The second processing module 720 is used to input the MRI images into a target multimodal large model to obtain a target image report output by the target multimodal large model; wherein, the target multimodal large model is trained based on the following steps:

[0184] Based on the sample images and the corresponding sample text reports, an image encoder is trained;

[0185] Based on the trained projection layer, the vector space of the image encoder is mapped to the vector space of the large language model to obtain an initial multimodal large model;

[0186] The initial multimodal large model is adjusted to obtain the target multimodal large model.

[0187] According to the automatic MRI medical image report generation device provided by the embodiment of the present application, an image encoder is trained through the sample images and the sample text reports, and a projection layer is introduced to map the vector space of the image encoder to the vector space of the large language model, thereby training the target multimodal large model. By adding a visual layer and a projection layer to the large language model, cross-modal alignment of 3D image features and text reports is achieved, enabling the model to accurately understand image features, thus generating high-quality and clinically valuable medical image reports, improving the semantic coherence of the generated image reports, fully meeting the needs of clinical practice, and enhancing the efficiency and accuracy of clinical diagnosis.

[0188] In some embodiments, the second processing module 720 can also be used to:

[0189] Input the sample images into the image encoder to obtain the first visual features corresponding to the sample images output by the image encoder;

[0190] Input the sample text reports corresponding to the sample images into the text encoder to obtain the first text features corresponding to the sample text reports output by the text encoder;

[0191] Based on the first visual features and the first text features, a contrast learning method is used to train the image encoder.

[0192] In some embodiments, the second processing module 720 can also be used to:

[0193] Process the first visual feature based on the weight matrix of the visual projection layer to obtain a second visual feature; process the first text feature based on the weight matrix of the text projection layer to obtain a second text feature;

[0194] Based on the second visual feature and the second text feature, use the loss function of contrastive learning to train the image encoder to obtain a trained image encoder.

[0195] In some embodiments, the second processing module 720 can also be used for:

[0196] Adopt the low-rank adaptation technique to decompose the updated weight matrix in the initial multi-modal large model based on the original weight matrix in the initial multi-modal large model to obtain the decomposed updated weight matrix;

[0197] Based on the decomposed updated weight matrix and the original weight matrix, obtain the target weight matrix corresponding to the initial multi-modal large model;

[0198] Based on the sample image, the image encoder and the projection layer, train the target weight matrix to obtain the target multi-modal large model.

[0199] In some embodiments, the second processing module 720 can also be used for:

[0200] Input the sample image into the image encoder to obtain the second visual feature output by the image encoder;

[0201] Input the second visual feature into the projection layer to obtain the third visual feature output by the projection layer;

[0202] Input the third visual feature into the updated initial multi-modal large model to reversely train the target weight matrix to obtain the target multi-modal large model.

[0203] In some embodiments, the trained projection layer includes a pooling layer and a fully connected layer, and the output end of the pooling layer is connected to the input end of the fully connected layer; the second processing module 720 can also be used for:

[0204] Input the sample image into the image encoder to obtain the first visual feature corresponding to the sample image output by the image encoder;

[0205] Input the first visual feature into the pooling layer to obtain the fourth visual feature;

[0206] Input the fourth visual feature into the fully connected layer to obtain the fifth visual feature, so as to map the vector space of the image encoder to the vector space of the large language model.

[0207] The MRI medical image report automatic generation device in the embodiments of the present application may be an electronic device or a component in an electronic device, such as an integrated circuit or a chip. The electronic device may be a terminal or other devices other than terminals. Exemplarily, the electronic device may be a mobile phone, a tablet computer, a laptop computer, a handheld computer, an in-vehicle electronic device, a Mobile Internet Device (MID), an augmented reality (AR) / virtual reality (VR) device, a robot, a wearable device, an ultra-mobile personal computer (UMPC), a netbook, or a personal digital assistant (PDA), etc. It may also be a server, a Network Attached Storage (NAS), a personal computer (PC), a television (TV), a teller machine, or a self-service machine, etc. The embodiments of the present application do not make specific limitations.

[0208] The MRI medical image report automatic generation device in the embodiments of the present application may be a device with an operating system. The operating system may be an Android operating system, an IOS operating system, or other possible operating systems. The embodiments of the present application do not make specific limitations.

[0209] The MRI medical image report automatic generation device provided by the embodiments of the present application can implement Figures 1 to 6 each process implemented by the method embodiments. To avoid repetition, it will not be elaborated here. <s

[0210] In some embodiments, as Figure 8 shown, the embodiments of the present application further provide an electronic device 800, including a processor 801, a memory 802, and a computer program stored on the memory 802 and executable on the processor 801. When the program is executed by the processor 801, it implements each process of the above-mentioned MRI medical image report automatic generation method embodiment and can achieve the same technical effect. To avoid repetition, it will not be elaborated here.

[0211] It should be noted that the electronic devices in the embodiments of the present application include the above-mentioned mobile electronic devices and non-mobile electronic devices. <s

[0212] On the other hand, the present application also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium. The computer program includes program instructions. When the program instructions are executed by a computer, the computer can execute each process of the above-mentioned embodiment of the automatic MRI medical image report generation method and achieve the same technical effect. To avoid repetition, it will not be elaborated here.

[0213] On another aspect, the present application also provides a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it can execute each process of the above-mentioned embodiment of the automatic MRI medical image report generation method and achieve the same technical effect. To avoid repetition, it will not be elaborated here.

[0214] On another aspect, an embodiment of the present application further provides a chip, which includes a processor and a communication interface. The communication interface is coupled to the processor. The processor is used to run programs or instructions to implement each process of the above-mentioned embodiment of the automatic MRI medical image report generation method and achieve the same technical effect. To avoid repetition, it will not be elaborated here.

[0215] It should be understood that the chip mentioned in the embodiment of the present application may also be referred to as a system-on-chip, system chip, chip system, or system-on-chip, etc.

[0216] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. Those of ordinary skill in the art can understand and implement it without creative efforts.

[0217] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, and of course, also by hardware. Based on this understanding, the essence of the above technical solution, or the part that contributes to the related technology, can be embodied in the form of a software product. The computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions for causing a computer device (which can be a personal computer, server, or network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0218] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements on some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the various embodiments of the present application.

Claims

1. An automatic generation method for MRI medical image reports, characterized in that, Including: Obtain an MRI image; Input the MRI image into a target multi-modal large model to obtain a target image report output by the target multi-modal large model; wherein, the target multi-modal large model is trained based on the following steps: Train an image encoder based on a sample image and a sample text report corresponding to the sample image; Based on a trained projection layer, map the vector space of the image encoder to the vector space of a large language model to obtain an initial multi-modal large model; Adjust the initial multi-modal large model to obtain the target multi-modal large model.

2. The automatic generation method of the MRI medical image report according to claim 1, wherein, The training of the image encoder based on the sample image and the sample text report corresponding to the sample image includes: Input the sample image into the image encoder to obtain a first visual feature corresponding to the sample image output by the image encoder; Input the sample text report corresponding to the sample image into a text encoder to obtain a first text feature corresponding to the sample text report output by the text encoder; Based on the first visual feature and the first text feature, use a contrastive learning method to train the image encoder.

3. The automatic generation method of the MRI medical image report according to claim 2, characterized in that, The training of the image encoder based on the first visual feature and the first text feature using a contrastive learning method includes: Process the first visual feature based on the weight matrix of a visual projection layer to obtain a second visual feature; process the first text feature based on the weight matrix of a text projection layer to obtain a second text feature; Based on the second visual feature and the second text feature, use a loss function of contrastive learning to train the image encoder to obtain a trained image encoder.

4. The automatic generation method of the MRI medical image report according to any one of claims 1-3, characterized in that, The adjustment of the initial multi-modal large model to obtain the target multi-modal large model includes: Adopt a low-rank adaptation technique to decompose an updated weight matrix in the initial multi-modal large model based on an original weight matrix in the initial multi-modal large model to obtain a decomposed updated weight matrix; Based on the decomposed updated weight matrix and the original weight matrix, obtain a target weight matrix corresponding to the initial multi-modal large model; Based on the sample image, the image encoder, and the projection layer, train the target weight matrix to obtain the target multi-modal large model.

5. The automatic generation method of MRI medical image report according to claim 4, characterized in that, The training of the target weight matrix based on the sample image, the image encoder, and the projection layer to obtain the target multi-modal large model includes: Input the sample image into the image encoder to obtain a second visual feature output by the image encoder; Input the second visual feature into the projection layer to obtain a third visual feature output by the projection layer; Input the third visual feature into the updated initial multi-modal large model to reversely train the target weight matrix to obtain the target multi-modal large model.

6. The automatic generation method of MRI medical image report according to any one of claims 1-3, characterized in that, The trained projection layer includes a pooling layer and a fully connected layer, and the output end of the pooling layer is connected to the input end of the fully connected layer; the mapping of the vector space of the image encoder to the vector space of the large language model based on the trained projection layer includes: Input the sample image into the image encoder to obtain the first visual feature corresponding to the sample image output by the image encoder; Input the first visual feature into the pooling layer to obtain the fourth visual feature; Input the fourth visual feature into the fully connected layer to obtain the fifth visual feature, so as to map the vector space of the image encoder to the vector space of the large language model.

7. An automatic MRI medical image report generation device, characterized in that, Comprising: A first processing module for acquiring an MRI image; A second processing module for inputting the MRI image into a target multi-modal large model to obtain a target image report output by the target multi-modal large model; wherein, the target multi-modal large model is trained based on the following steps: Train an image encoder based on a sample image and the sample text report corresponding to the sample image; Based on the trained projection layer, map the vector space of the image encoder to the vector space of the large language model to obtain an initial multi-modal large model; Adjust the initial multi-modal large model to obtain the target multi-modal large model.

8. An electronic device, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the MRI medical image report automatic generation method according to any one of claims 1-6.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the MRI medical image report automatic generation method according to any one of claims 1-6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the MRI medical image report automatic generation method according to any one of claims 1-6.

Images