Brain magnetic resonance angiography report generation method and device

Abstract

The invention provides a brain magnetic resonance angiography report generation method and device, and the method comprises the steps: obtaining a brain magnetic resonance angiography (MRA) image of a patient; performing image preprocessing on the brain MRA image to obtain a center MRA image and a plurality of maximum density projection MIP images corresponding to the brain MRA image; respectively inputting the central MRA image and the plurality of MIP images into a pre-trained visual language model to obtain a first target image semantic feature corresponding to the central MRA image and a second target image semantic feature corresponding to the plurality of MIP images output by the visual language model; and inputting the first target image semantic feature and the second target image semantic feature into a pre-trained large language model to obtain a prediction structured report for the brain of the patient.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a method and apparatus for generating brain magnetic resonance angiography reports. Background Technology

[0002] Magnetic resonance angiography (MRA), a non-invasive, high-resolution vascular imaging technique, is widely used for the screening and diagnosis of cerebrovascular diseases such as cerebral aneurysms, arterial stenosis, and vascular malformations. With the increasing clinical reliance on MRA examinations, the amount of MRA image data generated is growing rapidly. Doctors need to manually analyze a large number of MRA images and write reports. This not only increases workload but also easily leads to problems such as interpretation discrepancies, missed diagnoses, and misdiagnoses. Summary of the Invention

[0003] In view of this, the purpose of this application is to provide at least one method and apparatus for generating brain magnetic resonance angiography (MRA) reports. This method involves preprocessing the patient's brain MRA images to obtain a central MRA image and multiple MIP images. A pre-trained visual language model is then used to obtain semantic features of a first target image corresponding to the central MRA image and semantic features of second target images corresponding to the multiple MIP images. These semantic features are then input into a pre-trained large language model to obtain a predicted structured report of the patient's brain. This solves the technical problems of low efficiency and high error rate caused by the need for doctors to manually observe MRA images to generate reports in existing technologies, achieving the technical effect of increasing work efficiency and reducing the recognition error rate.

[0004] This application mainly includes the following aspects: In a first aspect, embodiments of this application provide a method for generating a brain magnetic resonance angiography report. The method includes: acquiring a brain magnetic resonance angiography (MRA) image of a patient; performing image preprocessing on the brain MRA image to obtain a corresponding central MRA image and multiple maximum intensity projection (MIP) images; inputting the central MRA image and multiple MIP images into a pre-trained visual language model to obtain a first target image semantic feature corresponding to the central MRA image and a second target image semantic feature corresponding to the multiple MIP images output by the visual language model; and inputting the first target image semantic feature and the second target image semantic feature into a pre-trained large language model to obtain a predicted structured report for the patient's brain.

[0005] Optionally, the visual language model is trained as follows: Image preprocessing is performed on brain MRA samples corresponding to multiple historical patients to obtain a central MRA sample and multiple MIP samples; and the original text reports corresponding to multiple historical patients are structured to obtain original structured text samples. For each brain MRA sample, the central MRA sample, the multiple MIP samples, and the original structured text samples corresponding to that brain MRA sample are combined as an image-text dataset for that brain MRA sample. The image-text dataset corresponding to each brain MRA sample is used as training samples to train the visual language model, so that the visual language model outputs a first training image semantic feature corresponding to the central MRA sample and a second training image semantic feature corresponding to the multiple MIP samples. The image semantic feature refers to an image feature that incorporates the semantic features corresponding to the original structured text samples.

[0006] Optionally, the visual language model is trained by performing an image reconstruction task on each central MRA sample, fusing the original structured text sample and the multiple MIP samples corresponding to each central MRA sample to perform a text reconstruction task, performing an image alignment task on each central MRA sample and its corresponding multiple MIP samples, and performing an image-text alignment task on the multiple MIP samples and the original structured text sample corresponding to each central MRA sample.

[0007] Optionally, the image reconstruction task for each central MRA sample is performed as follows: multiple voxel blocks of each central MRA sample are randomly occluded to obtain occluded MRA samples, and the unoccluded parts of the occluded MRA samples are processed by an image encoder; the processing result of the image encoder, the occluded parts of the occluded MRA samples, and the position encoding of each voxel block in the occluded MRA samples are input to an image decoder, so that the image decoder reconstructs the occluded MRA samples corresponding to each central MRA sample to obtain reconstructed MRA samples; a first loss function is calculated between each central MRA sample and its corresponding reconstructed MRA sample, and the image reconstruction task ends when the first loss function reaches the training completion condition corresponding to the image reconstruction task.

[0008] Optionally, the text reconstruction task is performed as follows: The original structured text sample corresponding to each central MRA sample is randomly masked to obtain a masked structured sample, and the masked structured sample is input into a text encoder to obtain masked structured text features; the MIP sample image features of the multiple MIP samples are determined by an image encoder; the masked structured text features are used as a query vector, and the MIP sample image features are used as key and value vectors for image-text fusion to obtain a mixed image-text vector; the mixed image-text vector and the full-text position encoding in the masked structured sample are input into a text decoder to obtain a reconstructed structured sample; a second loss function is calculated between the reconstructed portion of the masked structured sample and the original structured text sample in the reconstructed structured sample, and the text reconstruction task ends when the second loss function reaches the training completion condition corresponding to the text reconstruction task.

[0009] Optionally, the image alignment task for each central MRA sample and its corresponding plurality of MIP samples is performed in the following manner: extracting MRA sample image features of each central MRA sample and extracting MIP sample image features of the plurality of MIP samples corresponding to each central MRA sample using an image encoder; calculating a first cosine similarity between each MRA sample image feature and each MIP sample image feature; calculating a third loss function between the MRA sample image features and the MIP sample image features according to the first cosine similarity, wherein the third loss function is used to narrow the distance between the MRA sample image features and the MIP sample image features in the same case and widen the distance between the MRA sample image features and the MIP sample image features in different cases, so that the image alignment task ends when the third loss function reaches the training completion condition corresponding to the image alignment task.

[0010] Optionally, the image-text alignment task is performed on the plurality of MIP samples and the original structured text samples corresponding to each central MRA sample in the following manner: extracting MIP sample image features of the plurality of MIP samples corresponding to each central MRA sample through an image encoder, and extracting structured sample features of the original structured text samples corresponding to each central MRA sample through a text encoder; calculating a second cosine similarity between each MIP sample image feature and each structured sample feature; calculating a fourth loss function between the MIP sample image features and the structured sample features according to the second cosine similarity, wherein the fourth loss function is used to narrow the distance between the MRA sample image features and the structured sample features under the same case and widen the distance between the MIP sample image features and the structured sample features between different cases, so that the image-text alignment task ends when the fourth loss function reaches the training completion condition corresponding to the image-text alignment task.

[0011] Optionally, the large language model is trained by: inputting central MRA samples and multiple MIP samples from multiple historical patients into the visual language model to obtain first training image semantic features corresponding to the central MRA samples and second training image semantic features corresponding to the multiple MIP samples; inputting the first training image semantic features, the second training image semantic features, and original structured text samples corresponding to multiple historical patients into the large language model, and performing low-rank adaptation training on the large language model by calculating language modeling loss and medical entity weighted loss to train the large language model.

[0012] Optionally, the central MRA image and multiple MIP images corresponding to the brain MRA image are obtained by the following methods: converting the brain MRA image into a target format image, and performing bias field correction processing on the target format image to obtain a corrected image; resampling the corrected image according to the specifications of a preset voxel block to obtain multiple voxel blocks corresponding to the corrected image; standardizing the grayscale of each voxel block corresponding to the corrected image to obtain a standard image; extracting voxel blocks within a preset coordinate range from the standard image to obtain the central MRA image corresponding to the brain MRA image; and performing maximum density projection on the central MRA image at preset angle intervals to obtain multiple MIP images corresponding to the brain MRA image.

[0013] Secondly, embodiments of this application also provide a brain magnetic resonance angiography report generation device, the device comprising: an acquisition module for acquiring brain magnetic resonance angiography (MRA) images of a patient; a preprocessing module for performing image preprocessing on the brain MRA images to obtain corresponding central MRA images and multiple maximum density projection (MIP) images; a textual feature extraction module for inputting the central MRA images and multiple MIP images into a pre-trained visual language model to obtain a first target image semantic feature corresponding to the central MRA image and a second target image semantic feature corresponding to the multiple MIP images output by the visual language model; and a report generation module for inputting the first target image semantic feature and the second target image semantic feature into a pre-trained large language model to obtain a predicted structured report for the patient's brain.

[0014] This application provides a method and apparatus for generating a brain magnetic resonance angiography report. The method includes: acquiring a brain magnetic resonance angiography (MRA) image of a patient; performing image preprocessing on the brain MRA image to obtain a corresponding central MRA image and multiple maximum intensity projection (MIP) images; inputting the central MRA image and multiple MIP images into a pre-trained visual language model to obtain a first target image semantic feature corresponding to the central MRA image and a second target image semantic feature corresponding to the multiple MIP images output by the visual language model; and inputting the first target image semantic feature and the second target image semantic feature into a pre-trained large language model to obtain a predicted structured report for the patient's brain. This application obtains a central MRA image and multiple MIP images by preprocessing the patient's brain MRA images. Then, it obtains the semantic features of the first target image corresponding to the central MRA image and the semantic features of the second target images corresponding to the multiple MIP images through a pre-trained visual language model. The semantic features of the first and second target images are then input into a pre-trained large language model to obtain a predicted structured report of the patient's brain. This solves the technical problems of low work efficiency and high error rate caused by doctors having to manually observe MRA images to generate reports in the prior art, and achieves the technical effect of increasing work efficiency and reducing recognition error rate.

[0015] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0016] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 A flowchart of a method for generating a brain magnetic resonance angiography report provided in an embodiment of this application is shown.

[0018] Figure 2 A flowchart illustrating the steps for training a visual language model provided in an embodiment of this application is shown.

[0019] Figure 3 A schematic diagram of the training visual language model provided in an embodiment of this application is shown.

[0020] Figure 4 A schematic diagram of the training of a large language model provided in an embodiment of this application is shown.

[0021] Figure 5 This illustration shows a functional block diagram of a brain magnetic resonance angiography report generation device provided in an embodiment of this application.

[0022] Figure 6 A schematic diagram of the structure of an electronic device provided in an embodiment of this application is shown. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. It should be understood that the drawings in this application are for illustrative and descriptive purposes only and are not intended to limit the scope of protection of this application. Furthermore, it should be understood that the schematic drawings are not drawn to scale. The flowcharts used in this application illustrate operations implemented according to some embodiments of this application. It should be understood that the operations in the flowcharts may not be implemented in sequence, and steps without logical contextual relationships may be reversed or implemented simultaneously. In addition, those skilled in the art, guided by the content of this application, may add one or more other operations to the flowcharts, or remove one or more operations from the flowcharts.

[0024] Furthermore, the described embodiments are merely some, not all, of the embodiments of this application. The components of the embodiments of this application described and illustrated herein can typically be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of the application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0025] Magnetic resonance angiography (MRA), a non-invasive, high-resolution vascular imaging technique, is widely used for the screening and diagnosis of cerebrovascular diseases such as cerebral aneurysms, arterial stenosis, and vascular malformations. With the increasing clinical reliance on MRA examinations, the amount of MRA image data generated is growing rapidly. Doctors need to manually analyze a large number of MRA images and write reports. This not only increases workload but also easily leads to problems such as interpretation discrepancies, missed diagnoses, and misdiagnoses.

[0026] To alleviate these problems, automated medical report generation technologies have emerged in recent years. Existing research often employs convolutional neural networks to extract image features, combined with recurrent neural networks or Transformer models to generate natural language descriptions. These methods have achieved some success in two-dimensional medical image analysis. However, their application in three-dimensional medical images remains limited, primarily because the high dimensionality of three-dimensional images significantly increases the computational complexity of the models, and there is a lack of large-scale, publicly available datasets that pair three-dimensional images with reports. Furthermore, current AI (Artificial Intelligence) generated medical reports are typically free text, lacking structured medical terminology, making it difficult to meet clinical needs for standardized and consistent medical reports.

[0027] Existing visual and language pre-trained models are generally used to enhance the alignment between images and text, such as Contrastive Language-Image Pre-training (CLIP) and Bootstrapping Language-Image Pre-training (BLIP), to apply multimodal pre-trained models in the medical field to improve cross-modal semantic understanding. However, in medical report generation tasks, these methods still suffer from insufficient image semantic modeling and crude image-text fusion mechanisms. Especially in brain MRA scenarios, the complex vascular structure and significant differences in detail place higher demands on the model's semantic understanding. Most current methods struggle to fully exploit the fine-grained features of blood vessels and have not effectively integrated the diagnostic logic of combining multiple modal images relied upon by clinicians in actual image interpretation, resulting in significant deficiencies in clinical interpretability and the model's overall performance. Therefore, designing pre-training strategies tailored to the anatomical features of brain blood vessels and integrating multimodal image modeling methods that align with doctors' image interpretation habits is a crucial issue that current research urgently needs to address.

[0028] Based on this, this application provides a method and apparatus for generating brain magnetic resonance angiography (MRA) reports. The method involves preprocessing the patient's brain MRA images to obtain a central MRA image and multiple MIP images. A pre-trained visual language model is then used to obtain semantic features of a first target image corresponding to the central MRA image and semantic features of second target images corresponding to the multiple MIP images. These semantic features are then input into a pre-trained large language model to obtain a predicted structured report of the patient's brain. This solves the technical problems of low efficiency and high error rate caused by the need for doctors to manually observe MRA images to generate reports in existing technologies, achieving the technical effect of increasing work efficiency and reducing the recognition error rate, as detailed below: Please see Figure 1 , Figure 1 This is a flowchart illustrating a method for generating a brain magnetic resonance angiography report, provided as an embodiment of this application. Figure 1 As shown in the embodiments of this application, the method for generating a brain magnetic resonance angiography report includes the following steps: S101: Acquire MRI images of the patient's brain.

[0029] In other words, magnetic resonance angiography is performed on the patient's intracranial blood vessels to obtain MRA images of the patient's brain. Furthermore, the brain MRA images may show abnormalities in at least one anatomical region.

[0030] S102: Perform image preprocessing on the brain MRA images to obtain the corresponding central MRA images and multiple maximum density projection MIP images.

[0031] Among them, the central MRA image refers to the image obtained by cropping the central part of the brain MRA image, and multiple MIP images refer to the images obtained by performing maximum density projection at different angles on the brain MRA image.

[0032] The central MRA image and multiple MIP images corresponding to the brain MRA image are obtained through the following methods: the brain MRA image is converted into a target format image, and the target format image is subjected to bias field correction processing to obtain a corrected image; the corrected image is resampled according to the specifications of a preset voxel block to obtain multiple voxel blocks corresponding to the corrected image; the grayscale of each voxel block corresponding to the corrected image is standardized to obtain a standard image; voxel blocks within a preset coordinate range are extracted from the standard image to obtain the central MRA image corresponding to the brain MRA image; and maximum density projection is performed on the central MRA image at preset angle intervals to obtain multiple MIP images corresponding to the brain MRA image.

[0033] In other words, the DICOM sequence of the brain MRA image is converted into NIFIT format to obtain the target format image; the target format image is then subjected to bias field correction to eliminate the influence of the MRA device on the image to obtain the corrected image; the corrected image is then resampled, for example, by dividing the corrected image into multiple voxel blocks according to the specification of 0.5mm (millimeters) × 0.5mm × 0.5mm, to redefine the voxel block specification of the corrected image. The voxel block specification should be considered so as not to increase the computational load of subsequent model processing and not to affect the model's recognition of image features; the pixel values ​​of each voxel block of the corrected image are then standardized to obtain the standard image, and the pixel values ​​of each voxel block are standardized; in this way, voxel blocks within the preset coordinate range of the standard image are selected to form the central MRA image. Specifically, for each of the three directions of the standard image, 224 voxel blocks located at the center position in that direction are selected to select the central MRA image with a specification of 224×224×224. Furthermore, the preset coordinate range refers to the 224 voxel blocks located at the center position in each direction of the standard image.

[0034] Furthermore, after obtaining the central MRA image, a maximum density projection is performed every 12 degrees along the vertical central axis of the central MRA image. This involves compressing the highest density voxels from top to bottom onto a two-dimensional horizontal plane, resulting in 15 images (i.e., MIP images (Maximum Intensity Projection) from different angles.

[0035] Furthermore, after acquiring the patient's brain MRA images, image preprocessing was performed to extract the central MRA image of the central region, and multiple MIP images were obtained through maximum density projection. The central MRA image is a three-dimensional image, while the multiple MIP images are two-dimensional images.

[0036] S103: Input the central MRA image and multiple MIP images into a pre-trained visual language model to obtain the first target image semantic features corresponding to the central MRA image and the second target image semantic features corresponding to the multiple MIP images output by the visual language model.

[0037] Among them, the visual language model is a basic model that integrates three-dimensional images, two-dimensional images and structured text. By performing joint training on the visual language model through multiple tasks, the ability of the visual language model to model brain MRA images and semantic information is enhanced, so that the visual language model can identify image features containing semantic features from brain MRA images.

[0038] For details, please refer to Figure 2 , Figure 2 This is a flowchart illustrating the steps involved in training a visual language model, as provided in an embodiment of this application. Figure 2 As shown, the visual language model is trained in the following way: S201: Perform image preprocessing on brain MRA samples corresponding to multiple historical patients to obtain central MRA samples and multiple MIP samples, and perform structuring processing on the original text reports corresponding to multiple historical patients to obtain original structured text samples.

[0039] In other words, brain MRA samples are obtained for each historical patient. Then, the aforementioned image preprocessing methods are used to obtain the central MRA sample and multiple MIP samples corresponding to each historical patient's brain MRA sample. Furthermore, it is also necessary to obtain the original text report made by the doctor for each historical patient's brain MRA image, and to obtain the original structured text sample by structuring the original text report.

[0040] Each historical patient's brain MRA sample may show abnormalities in at least one anatomical region. Furthermore, the brain MRA samples and original text reports from multiple historical patients should be accurate and correspond to each other to prevent errors in the training samples from affecting the model's recognition accuracy.

[0041] For example, a structured report template is pre-defined, and the original text report of each historical patient is structured by deploying a large language model based on prompting engineering to obtain the original structured text sample corresponding to each original text report.

[0042] This involves cleaning multiple original text reports, removing those containing errors, as well as irrelevant and incomplete information from each original text report, and standardizing the Chinese and English descriptions within them.

[0043] For example, the structured report template includes the following four parts: left-right orientation, anatomical location, specific location, and imaging manifestations. Left-right orientation is used to indicate the location of the lesion or abnormality, anatomical location is used to indicate the vascular anatomical name of the lesion or abnormality in the brain, specific location is used to indicate the precise location of the anatomical location within the vascular segment, and imaging manifestations are used to indicate the abnormal manifestations of the anatomical location in MRA images.

[0044] The left-right orientation generally includes left, right, and bilateral. Anatomical locations include the anterior cerebral artery, middle cerebral artery, posterior cerebral artery, vertebral artery, internal carotid artery, superior cerebellar artery, and basilar artery. Specific locations include: the horizontal, vertical, genu, and pericallosal segments of the anterior cerebral artery; the horizontal, circumflex, lateral fissure, and bifurcation segments of the middle cerebral artery; the pre-communicating, ambient cistern, tetradymal, and talus segments of the posterior cerebral artery; the extraosseous, intervertebral foramen, extravertebral, and intracranial segments of the vertebral artery; and the petrous, communicating, cavernous sinus, ocular, clinoid, foramen lacerum, and cervical segments of the internal carotid artery. Imaging features include tortuous and thin, bulging, elongated, thickened, poorly visualized, fenestrated, unevenly smooth walls, elevation, heterogeneous signal, suspicious small protrusions, arcuate bulges, sparse branching, common trunk, thin lumen, weakened blood flow signal at origin, three-branch variation, uneven thickness, luminal stenosis, etc.

[0045] For example, a large language model is trained based on cue engineering. The original text report is input into the large language model, which then outputs a sample of the original structured text according to the four parts required by the structured report template described above. Specifically, the following cue words are input into the large language model: "Prompt=You are the Medical Report Structure Assistant. Please extract the MRA report into the following four fields:" Left and right directions (e.g., left, right, both sides; leave blank if not specified); Anatomical locations (such as the anterior cerebral artery, middle cerebral artery, posterior cerebral artery, internal carotid artery, superior cerebellar artery, vertebral artery, and basilar artery). Specific location (e.g., horizontal segment, vertical segment, knee segment, rotation segment, lateral fissure segment, etc.; leave blank if none); Imaging features (such as tortuous, narrow, poor imaging, rough tube walls, no visible, common stem, bifurcation variation, embryonic type, etc.)).

[0046] Furthermore, detailed requirements are set, stipulating that the left and right orientation of the basilar artery must be empty; when anatomical sites and specific locations appear side by side, the anatomical sites and specific locations must be displayed in a structured manner, and the imaging manifestations of both must be identical; when the original text report shows "not significant," it needs to be replaced with the most recently observed imaging manifestation. For example, if the original text report shows stenosis of the left anterior cerebral artery and "not significant" on the right, it means that the right anterior cerebral artery is also stenotic, and the stenosis on the right is more severe than on the left; when common imaging manifestations (such as slightly tortuous major blood vessels) appear, they should be added to all extracted imaging manifestations; the identified original structured text samples must be completely faithful to the original text report, and self-creation, modification, or omission is prohibited.

[0047] The output requirements are as follows: output only a single JSON (JavaScript Object Notation) array without any additional explanation; each record should sequentially include left and right orientation, anatomical location, specific location, and imaging features, and the order of the fields must be consistent.

[0048] For example, the original text report and manually processed structured samples are input into the large language model as examples. The unprocessed original text report is also input into the large language model, allowing it to iteratively generate a structured report of the original text report. For instance, the original text report "Bilateral vertebral arteries and basilar arteries are tortuous, and the left posterior cerebral artery is stenotic in the amphitheater segment" and its corresponding manually processed structured samples are input into the large language model. The manually processed structured samples include the following: "Left-right orientation: bilateral, anatomical location: vertebral artery, specific location: none, imaging appearance: tortuous course; left-right orientation: none, anatomical location: basilar artery, specific location: none, imaging appearance: tortuous course; left-right orientation: left, anatomical location: posterior cerebral artery, specific location: amphitheater segment, imaging appearance: stenosis;". Based on this, the unprocessed original text report is input into the large language model, allowing it to iteratively train according to the above example and output the original structured text sample corresponding to the original text report.

[0049] In other words, the large language model extracts the descriptions of each part of the structured report template from the original text report according to the structured report template, thereby obtaining the original structured text sample corresponding to the original text report.

[0050] S202: For each brain MRA sample, the central MRA sample corresponding to the brain MRA sample, the multiple MIP samples, and the original structured text sample are combined to form the image text dataset of the brain MRA sample.

[0051] Furthermore, for each historical patient, the central MRA sample, multiple MIP samples, and the original structured text sample corresponding to that historical patient are combined to determine the image-text dataset corresponding to the brain MRA sample of that historical patient. The image-text dataset further includes three-dimensional samples, two-dimensional samples, and the original structured text sample.

[0052] S203: Use the image-text dataset corresponding to each brain MRA sample as a training sample to train the visual language model, so that the visual language model outputs the first training image semantic features corresponding to the central MRA sample and the second training image semantic features corresponding to the multiple MIP samples.

[0053] The visual language model is trained by performing image reconstruction on each central MRA sample, fusing the original structured text sample and the multiple MIP samples corresponding to each central MRA sample to perform text reconstruction, performing image alignment on each central MRA sample and its corresponding multiple MIP samples, and performing image-text alignment on the multiple MIP samples and the original structured text sample corresponding to each central MRA sample.

[0054] Here, image semantic features refer to image features that incorporate the semantic features corresponding to the original structured text samples. The visual language model includes an image encoder, an image decoder, a text encoder, and a text decoder. For an example, please refer to... Figure 3 , Figure 3 This is a schematic diagram of a training visual language model provided in an embodiment of this application. Figure 3 As shown, joint training is performed on an image-text dataset around four core tasks, enabling the visual language model to identify image features containing semantic features from both 3D and 2D images.

[0055] For example, the image encoder refers to a shallow-shared, deep-branched bimodal coding architecture proposed for simultaneously modeling the central MRA and MIP samples of medical images. The shallow-shared encoder uses the underlying common features corresponding to the central MRA and MIP samples respectively, providing a unified basic representation for subsequent branch processing. The image coding branch corresponding to the MIP samples utilizes a Transformer module (a deep learning model based on an attention mechanism) to capture the consistency between the context and cross-projection semantics of the MIP sequence. The image coding branch corresponding to the central MRA samples introduces the Swin UNETR structure (3D medical image segmentation network) suitable for 3D medical images to model spatial and volumetric information. Finally, the branch outputs corresponding to the central MRA and MIP samples are projected onto the same dimensional space through a Multilayer Perceptron (MLP), with each MLP layer including two linear transformation layers and a linear rectified function (ReLU activation function) to ensure that the image features corresponding to the central MRA and MIP samples have the same dimensionality. The image decoder consists of a set of shared fully connected layers that share image features across all inputs. These layers are used to recover masked image patches from each image feature in order to reconstruct the image.

[0056] For example, the text encoder employs a frozen bidirectional encoder (BERT). Before being input into the text encoder, the original structured text samples need to be segmented by a tokenizer (BERT Tokenizer) to obtain a sample sequence. This allows the text encoder to convert the sample sequence into context-sensitive hidden vectors. The text decoder, based on the Transformer structure, receives the fused multimodal features and performs contextual modeling on the masked features to reconstruct the text.

[0057] Specifically, the image reconstruction task for each central MRA sample is performed as follows: multiple voxel blocks of each central MRA sample are randomly occluded to obtain occluded MRA image samples, and the unoccluded parts of the occluded MRA samples are processed by an image encoder; the processing result of the image encoder, the occluded parts of the occluded MRA samples, and the position encoding of each voxel block in the occluded MRA samples are input into an image decoder, so that the image decoder reconstructs the occluded MRA samples corresponding to each central MRA sample to obtain reconstructed MRA samples; a first loss function is calculated between each central MRA sample and its corresponding reconstructed MRA sample, and the image reconstruction task ends when the first loss function reaches the training completion condition corresponding to the image reconstruction task.

[0058] For example, each central MRA sample The image is divided into 2744 voxel blocks according to a 16×16×16 specification. Within each voxel block group, 50% of the voxel blocks are randomly selected for occlusion. The occluded central MRA sample is then input into the image encoder for feature extraction to obtain the occluded MRA sample. The occluded MRA sample includes both the occluded and unoccluded voxel blocks, with the occluded voxel blocks serving as the reconstruction target. voxel blocks of the unmasked parts The input is fed into an image encoder to obtain an implicit representation of the unmasked portion. .

[0059] The implicit representation of the unmasked portion is obtained using the following formula: (1) In formula (1), This refers to the implicit representation of the unmasked portion corresponding to the nth central MRA sample. This refers to the voxel block of the unmasked portion corresponding to the nth central MRA sample. This refers to the feature processing performed by the image encoder.

[0060] Furthermore, the voxel blocks of the unmasked portion The input is fed into an image encoder for processing to obtain an implicit representation of the unmasked portion. Then, the implicit representation of the unmasked portion is... The voxel blocks of the occluded portion are replaced by a learnable mask representation. and the position encoding of each voxel block in the masked MRA sample The input is fed into the image decoder, which uses a fully connected network with shared weights to reconstruct the occluded portion and combines it with the unoccluded portion to output a complete reconstructed MRA sample. In other words, the reconstructed MRA sample can be understood as a complete set combining voxel blocks of the unoccluded portion and voxel blocks reconstructed from the occluded portion.

[0061] The first loss function can be the Mean Squared Error (MSE). The MSE minimizes the difference between the predicted reconstructed MRA sample and the central MRA sample, thus encouraging the model to learn fine-grained structural information in the central MRA sample, especially detailed features such as the edges, orientation, and continuity of the vascular region. The first loss function is calculated using the following formula: (2) In formula (2), This refers to the first loss function. This refers to the implicit representation of the unmasked portion corresponding to the nth central MRA sample. This refers to the learnable substitution representation of the voxel block corresponding to the occluded portion of the nth central MRA sample. This refers to the positional encoding of each voxel block in the occluded MRA sample corresponding to the nth central MRA sample. This refers to the feature processing performed by the image decoder. This refers to the voxel block of the unmasked portion corresponding to the nth central MRA sample. It refers to the voxel block of the occluded part corresponding to the nth central MRA sample, and N refers to the total number of central MRA samples in a certain training batch.

[0062] Furthermore, each central MRA sample is randomly occluded to obtain occluded MRA samples. The image encoder extracts features from the unoccluded parts and inputs the extraction results into the image decoder to reconstruct the occluded parts. This is done to perform a self-supervised learning task. The first loss function between the reconstructed MRA sample and the central MRA sample image is calculated. The model parameters are optimized through iterative loops until the first loss function reaches the training completion condition corresponding to the iteration stop. This completes the image reconstruction task, enabling the visual language model to learn local details and global spatial distribution features in the vascular structure.

[0063] Specifically, the text reconstruction task is performed as follows: The original structured text sample corresponding to each central MRA sample is randomly masked to obtain a masked structured sample, and this masked structured sample is input into a text encoder to obtain masked structured text features; the MIP sample image features of the multiple MIP samples are determined by an image encoder; the masked structured text features are used as a query vector, and the MIP sample image features are used as key and value vectors for image-text fusion to obtain a mixed image-text vector; the mixed image-text vector and the full-text position encoding in the masked structured sample are input into a text decoder to obtain a reconstructed structured sample; a second loss function is calculated between the reconstructed portion of the masked structured sample and the original structured text sample in the reconstructed structured sample, and the text reconstruction task ends when the second loss function reaches the training completion condition corresponding to the text reconstruction task.

[0064] Furthermore, multiple MIP samples are used to guide the reconstruction of masked structured samples. A one-way cross-modal fusion module (MRCFA, MIP-Report Cross-modal Fusion and Attention) is introduced to integrate MIP image report text. Multiple MIP samples are processed by an image encoder to extract features, and the extracted features are then introduced into the text reconstruction process to improve the model's ability to complete semantically missing parts.

[0065] First, the original structured text sample The original structured text samples are converted into word units that the model can process by using BERT Tokenizer for word segmentation. Then, for the sub-word units 50% of the data is randomly selected for masking. The masked portion is then replaced with a special marker [MASK]. The masked portion and the unmasked portion are used as structured masking samples. Masking the masked parts in a structured sample. This serves as the target for reconstruction. In this way, the obscured structured samples will be... The input is fed into a text encoder to obtain masked structured text features. These features include text features corresponding to both the unmasked and masked portions of the masked structured sample. The masked structured text features are obtained through the following method: (3) In formula (3), This refers to the masked structured text features corresponding to the nth original structured text sample. This refers to the masked structured sample corresponding to the nth original structured text sample. This refers to the feature processing performed by the text encoder.

[0066] Furthermore, the MIP sample image features of multiple MIP samples are determined using the following formula: (4) In formula (4), This refers to the MIP sample image features of multiple MIP samples corresponding to the nth central MRA sample. This refers to the multiple MIP samples corresponding to the nth central MRA sample. This refers to the feature processing performed by the image encoder.

[0067] Therefore, the hybrid image-text vector can be obtained using the following formula: (5) In formula (5), This refers to the mixed image-text vector corresponding to multiple MIP samples and the original structured text sample corresponding to the nth central MRA sample. This refers to the projection matrix of each attention head. This refers to the feature dimension. This refers to the query vector, which is the masked structured text feature. , This refers to the key vector. This refers to the value vector, where both the key vector and the value vector correspond to the features of the MIP sample image. , This refers to the multi-head linear projection matrix. Formula (5) is an extension of the standard multi-head self-attention (MHSA) mechanism, specifically designed for cross-modal input, capable of simultaneously processing text and multi-view image embeddings. The query vector originates from the text's token embedding, i.e., masked structured text features, representing the semantic information of the text; the key vector and value vector are formed by concatenating the embedded image features of multiple MIP samples corresponding to the central MRA sample, covering the multi-view information of the image. During attention calculation, the attention weights of text features on each part of the image are calculated through the dot product of the query vector and the key vector, thereby achieving cross-modal information alignment. Subsequently, these attention weights are applied to the value vector, weighting and aggregating the image features to generate image feature representations related to text semantics. This mechanism enables the model to flexibly handle different numbers of text and image inputs, achieving deep fusion of two modal information, and ultimately generating semantically related cross-modal embedding vectors. Furthermore, residual connections (i.e., adding the original query vector Q to the output) help stabilize the training process and preserve the original semantic features of the text.

[0068] Furthermore, the hybrid image and text vectors Full-text location encoding in masked structured samples The input is fed into a lightweight text decoder (Transformer decoder), which fully utilizes multi-source modal information to predict the masked portion to be replaced by [MASK]. During training, a second loss function is calculated only for the masked portion; the second loss function is cross-entropy loss, calculated using the following formula: (6) In formula (6), This refers to the second loss function. This refers to the mixed image-text vector corresponding to multiple MIP samples and the original structured text sample corresponding to the nth central MRA sample. This refers to the full-text position encoding in the masked structured sample corresponding to the original structured text sample associated with the nth central MRA sample. This refers to the processing performed by the text decoder. This refers to the obscured portion of the obscured structured sample corresponding to the original structured text sample associated with the nth central MRA sample. This refers to calculating the cross-entropy. The cross-entropy is calculated using the following formula: (7) In formula (7), This refers to calculating the cross-entropy between the first parameter p and the second parameter q. This refers to the v-th second parameter q. This refers to the v-th first parameter p, where V refers to the total number of cross-entropies being calculated.

[0069] Furthermore, by partially masking the original structured text samples, a supervised learning task based on masked text reconstruction is constructed. In this process, MIP sample image features corresponding to the original structured text samples are introduced. A multimodal cross-fusion mechanism guides the text reconstruction process, and a second loss function is calculated between the reconstructed masked portion and the masked portion in the original structured text sample. The model parameters are optimized through iterative iteration until the second loss function reaches the training completion condition corresponding to the iteration stop, thus completing the text reconstruction task. Therefore, the model can integrate the contextual semantic information of images and text, enhancing its ability to model the semantic structure of reports and medical terminology, improving the accuracy of missing semantic completion, and achieving effective fusion of image and structured text information.

[0070] Specifically, the image alignment task is performed on each central MRA sample and its corresponding plurality of MIP samples in the following manner: MRA sample image features of each central MRA sample are extracted using an image encoder, and MIP sample image features of the plurality of MIP samples corresponding to each central MRA sample are extracted using an image encoder; a first cosine similarity is calculated between each MRA sample image feature and each MIP sample image feature; a third loss function is calculated between the MRA sample image features and the MIP sample image features based on the first cosine similarity. The third loss function is used to narrow the distance between the MRA sample image features and the MIP sample image features in the same case and to widen the distance between the MRA sample image features and the MIP sample image features in different cases, so that the image alignment task ends when the third loss function reaches the training completion condition corresponding to the image alignment task.

[0071] In other words, after inputting the central MRA sample and multiple corresponding MIP samples for the same historical patient into an image encoder with shared weights, each central MRA sample is extracted. MRA sample image features ,Right now And extracting the image features of one MIP sample corresponding to multiple MIP samples. .

[0072] To align the feature spaces of MRA and MIP samples, an Information Noise Contrastive Estimation (InfoNCE) loss function is introduced as a third loss function. The model parameters are modified through iterative iterations to gradually narrow the distance between MRA and MIP samples in the same case and widen the feature distance between MRA and MIP samples in different cases. This enables the encoder to learn the potential structural information for cross-modal alignment, thereby improving the model's ability to model vascular structures.

[0073] The third loss function is calculated using the following formula: (8) In formula (8), This refers to the third loss function. This refers to the contrastive loss function used to align MIP samples to MRA samples. This refers to the contrastive loss function used to align MRA samples to MIP samples. This refers to the MIP sample image features of multiple MIP samples corresponding to the nth central MRA sample. This refers to the MRA sample image features of the nth central MRA sample. This refers to adjusting the temperature coefficient of contrast loss. This refers to cosine similarity. N refers to the total number of center MRA samples in a training batch. i refers to the MRA sample image features of the center MRA sample that are always compared when aligning MIP samples with MRA samples. j refers to the MIP sample image features that are always compared when aligning MRA samples with MIP samples. The values ​​of i and j may be different. This is an index traversal matching process (dynamically changing).

[0074] Furthermore, MRA samples and their generated multi-angle MIP samples from the same case are modeled uniformly and input into a shared-weight image encoder to extract the sample features corresponding to the MRA and MIP samples, respectively. A contrastive learning strategy is used to minimize the feature distance between MRA and MIP samples from the same case, while simultaneously increasing the feature distance between MRA and MIP samples from different cases. Thus, as the third loss function gradually decreases and stabilizes, it signifies that positive and negative samples can be distinguished, thereby concluding the image alignment task. This enables the model to learn potential image representations with consistent semantic meaning across different image modalities, improving the model's robust understanding of vascular morphology and structure.

[0075] Specifically, the image-text alignment task is performed on the multiple MIP samples and the original structured text samples corresponding to each central MRA sample in the following manner: Image encoders extract the MIP sample image features of the multiple MIP samples corresponding to each central MRA sample, and text encoders extract the structured sample features of the original structured text samples corresponding to each central MRA sample; a second cosine similarity is calculated between each MIP sample image feature and each structured sample feature; a fourth loss function is calculated based on the second cosine similarity between the MIP sample image features and the structured sample features. This fourth loss function is used to narrow the distance between the MRA sample image features and the structured sample features in the same case and widen the distance between the MIP sample image features and the structured sample features in different cases. The image-text alignment task ends when the fourth loss function reaches the training completion condition corresponding to the image-text alignment task.

[0076] In other words, multiple MIP samples corresponding to the central MRA samples of the same historical patient are input into the image encoder to obtain the MIP sample image features. The original structured text sample corresponding to the central MRA sample of this historical patient. Input is fed into a text encoder to obtain structured sample features. ,Right now Furthermore, to align the feature spaces of the images and text, CLIP (Contrastive Language Image Pretraining) contrastive learning loss is used as the fourth loss function. This brings the MIP samples and their corresponding original structured text samples from the same case closer together in the embedding space, while simultaneously increasing the feature distance between MIP samples and original structured text samples from different cases, thereby further improving the model's cross-modal understanding capability. The fourth loss is calculated using the following formula: (9) In formula (9), This refers to the fourth loss function. This refers to the contrastive loss function used to align MIP samples to the original structured text samples. This refers to the contrastive loss function used to align original structured text samples to MIP samples. This refers to the MIP sample image features of multiple MIP samples corresponding to the nth central MRA sample. This refers to the structured sample features corresponding to the nth central MRA sample. This refers to the temperature coefficient that adjusts the sharpness of the fourth loss function. This refers to cosine similarity.

[0077] Furthermore, by inputting the MIP samples and their corresponding original structured text samples into the image encoder and text encoder respectively, and minimizing the contrastive learning loss between them in the shared embedding space, the MIP sample image features and structured sample features of the same historical patient are brought close together in the vector space, while the MIP sample image features of one historical patient are far apart from the structured sample features of other historical patients, and vice versa. Thus, as the fourth loss function gradually decreases and stabilizes, it means that positive and negative samples can be distinguished, thereby concluding the image-text alignment task and improving the model's cross-modal semantic understanding and expressive capabilities.

[0078] For example, the method further includes: calculating the total loss of the visual language model using a first loss function, a second loss function, a third loss function, and a fourth loss function to balance the loss terms of the joint training task. The total loss of the visual language model is calculated using the following formula: (10) In formula (10), This refers to the total loss of the visual language model. This refers to the third loss function. This refers to the hyperparameters corresponding to the fourth loss function. This refers to the fourth loss function. This refers to the hyperparameters corresponding to the first loss function. This refers to the first loss function. This refers to the hyperparameters corresponding to the second loss function. This refers to the second loss function.

[0079] For example, when the total loss of the visual language model gradually decreases and stabilizes, and each loss function also stabilizes within its respective task, the visual language model can be considered trained successfully. Furthermore, by changing the hyperparameters corresponding to different loss functions, the contribution of each task to the visual language model can be set to balance the influence of each loss function. After the visual language model training is complete, the parameters in the visual language model are frozen. During subsequent training of the larger language model, the parameters in the visual language model are not modified synchronously; training is only performed on the larger language model.

[0080] return Figure 1 S104: Process the semantic features of the first target image and the semantic features of the second target image using a large language model to obtain a predicted structured report corresponding to the patient's brain.

[0081] In other words, the central MRA image is input into a trained visual language model to obtain the first target image semantic features corresponding to the central MRA image, and multiple MIP images are input into the trained visual language model to obtain the second target image semantic features corresponding to the multiple MIP images. Furthermore, the first and second target image semantic features refer to image features possessing semantic characteristics. Therefore, the subsequently trained large language model can identify semantic features based on the first and second target image semantic features to generate a structured report.

[0082] For example, the semantic features of the first target image and the semantic features of the second target image, as well as the text prompt, output by the visual language model need to be input into the trained large language model, so that the large language model can output a structured report.

[0083] For example, please refer to Figure 4 , Figure 4 This is a schematic diagram of the large language model provided in an embodiment of this application. Figure 4As shown, the central MRA image is input into a trained visual language model to obtain the first target image semantic features corresponding to the central MRA image. Multiple MIP images are input into the trained visual language model to obtain the second target image semantic features corresponding to the multiple MIP images. The first and second target image semantic features are concatenated along the channel dimension to obtain fused image semantic features. Next, the fused image semantic features are input into a linear mapping layer (LinearProjection) to convert the image semantic features into a visual embedding representation aligned with the embedding space of the large language model (LLM). The visual embedding representation includes the image features resulting from the concatenation of the MRA and MIP images.

[0084] The linear mapping layer is essentially a fully connected layer, defined as a weight matrix. To fuse image semantic features This is converted into the feature dimensions of a large language model. Here, T represents the number of visual features (i.e., the number of semantic features of the first target image and the number of semantic features of the second target image), and the dimension of each visual feature is... Visual feature embeddings are obtained after dimensionality transformation. This embeds visual features With text prompts Both together constitute the input sequence. , This is to enable the trained large language model to generate structured reports.

[0085] For example, the text prompt is set as follows: "[INST]" <imagefeature>[reportGeneration] Please generate a structured report in the following format based on the MRA images above: Anatomical location: Left and right directions: Specific location: Image performance: [ / INST]”.

[0086] like Figure 4 As shown, the large language model selected in this embodiment is DeepSeek-VL-7B. Based on this, a low-rank adaptation (LoRA) mechanism is introduced into the key network modules of the large language model. This mechanism inserts a trainable low-rank matrix into the weight matrix of the query vector and value vector in the self-attention layer, and trains the low-rank matrix of the linear projection layer and the low-rank adaptation using the image semantic features output by the visual language model and the original structured text samples.

[0087] Therefore, in the training process of the large language model, the training samples of the large language model need to be processed first through a visual language model. That is, the brain MRA images of multiple historical patients are processed through the aforementioned image preprocessing steps to obtain central MRA images, and multiple MIP images are obtained by performing multi-angle maximum density projection on the central MRA images of multiple historical patients. Then, the central MRA images of multiple historical patients are called central MRA samples, and the multiple MIP images corresponding to each of the multiple historical patients are called MIP samples. Each central MRA sample and its multiple MIP samples are processed by a pre-trained visual language model, so that the semantic features of the processed images and the original structured text samples are used as training samples for training the large language model.

[0088] Furthermore, the original structured text samples should be pre-converted into text sample features acceptable to the large language model before training. Then, the large language model is trained using the image semantic features and text sample features obtained by the visual language model through the central MRA sample and its corresponding multiple MIP samples. During the training process, the loss function is optimized by optimizing the various parameters of the large language model until the training is completed.

[0089] Specifically, the large language model is trained as follows: central MRA samples and multiple MIP samples from multiple historical patients are input into the visual language model to obtain first training image semantic features corresponding to the central MRA samples and second training image semantic features corresponding to the multiple MIP samples; the first training image semantic features, the second training image semantic features, and original structured text samples corresponding to multiple historical patients are input into the large language model, and low-rank adaptation training is performed on the large language model by calculating language modeling loss and medical entity weighted loss to train the large language model.

[0090] Specifically, inputting the central MRA sample of each historical patient into the trained visual language model yields the first training image semantic features, and inputting multiple MIP samples of each historical patient into the trained visual language model yields the second training image semantic features. At this point, both the first and second training image semantic features are image features containing the structured sample features of each historical patient.

[0091] The training primarily involves language modeling cross-entropy loss and medical entity-weighted loss. For example, the loss function for the large language model is calculated using the following formula: (11) In formula (11), This refers to the loss function of a large language model. This refers to the cross-entropy loss function in language modeling. This refers to hyperparameters, where the structured sample features of the original structured text sample of the i-th historical patient are represented as follows: The predicted structured sample features, which are used by the large language model to predict based on the semantic features of the first training image and the semantic features of the second training image, are represented as follows: , This refers to a large language model with a set of model parameters. According to the known preceding text Predict the j-th word under certain conditions The probability of. This refers to the medical entity weighted loss function, which is used to increase the weight of low-frequency medical terms to prevent them from being ignored. This represents the total number of training samples. Representing entities Frequency of occurrence in the training set This refers to the minimum frequency of all medical terms. The structured sample feature representation... The entity set is represented as ,in Indicates the index position of the entity in the text.

[0092] Furthermore, training a large language model by only considering the language modeling cross-entropy loss function will ignore low-frequency medical terms, resulting in poor predictive ability for these terms. Therefore, by... To assign higher weights to low-frequency medical terms, the loss function value is strengthened when low-frequency medical terms are predicted incorrectly, forcing the model to strengthen its learning of low-frequency medical terms and improve its ability to recognize and generate terms in professional scenarios.

[0093] Furthermore, this application constructs the input for a large language model by linearly mapping the extracted semantic features of the training images and concatenating them with structured samples and their corresponding text prompts. LoRA technology is then used to fine-tune the large language model, enabling it to combine the input image information to generate accurate and clinically compliant predicted structured reports.

[0094] Furthermore, this application introduces the LoRA mechanism into the large language model to adjust the linear mapping layer, enabling the large language model to adapt to specific medical application scenarios without large-scale parameter updates. Since the large language model itself lacks specific medical image perception capabilities, the generated reports often fail to accurately cover key medical details and anatomical features. Therefore, it is necessary to construct a visual language model that integrates medical images and structured text, and then combine this with fine-tuning strategies of the large language model to automatically predict structured reports for MRA images.

[0095] This application integrates 3D MRA images, multi-angle MIP images, and structured reports, employing masking reconstruction, contrastive learning, and cross-attention mechanisms to deeply model a visual language model, generating image features containing semantic features of vascular images. Simultaneously, the LoRA fine-tuning method is used to efficiently fine-tune the parameters of the large language model, enabling it to automatically generate structured reports from semantic image features. This effectively assists physicians in image interpretation and report writing, demonstrating broad clinical application prospects.

[0096] Based on the same application concept, this application also provides a brain magnetic resonance angiography report generation device corresponding to the brain magnetic resonance angiography report generation method provided in the above embodiments. Since the principle of the device in this application is similar to the brain magnetic resonance angiography report generation method in the above embodiments of this application, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be described again.

[0097] like Figure 5 As shown, Figure 5 This is a functional block diagram of a brain magnetic resonance angiography report generation device provided in an embodiment of this application. The brain magnetic resonance angiography report generation device 10 includes: an acquisition module 101, which acquires brain magnetic resonance angiography (MRA) images of a patient; a preprocessing module 102, which performs image preprocessing on the brain MRA images to obtain corresponding central MRA images and multiple maximum intensity projection (MIP) images; an image and text feature extraction module 103, which inputs the central MRA images and multiple MIP images into a pre-trained visual language model to obtain a first target image semantic feature corresponding to the central MRA image and a second target image semantic feature corresponding to the multiple MIP images output by the visual language model; and a report generation module 104, which inputs the first target image semantic feature and the second target image semantic feature into a pre-trained large language model to obtain a predicted structured report for the patient's brain.

[0098] Based on the same application concept, see [link / reference] Figure 6 The diagram shown is a structural schematic of an electronic device provided in an embodiment of this application. The electronic device 20 includes a processor 201, a memory 202, and a bus 203. The memory 202 stores machine-readable instructions executable by the processor 201. When the electronic device 20 is running, the processor 201 and the memory 202 communicate through the bus 203. When the machine-readable instructions are executed by the processor 201, they perform the steps of the brain magnetic resonance angiography report generation method as described in any of the above embodiments.

[0099] Specifically, when the machine-readable instructions are executed by the processor 201, they can perform the following processes: acquiring the patient's brain magnetic resonance angiography (MRA) images; performing image preprocessing on the brain MRA images to obtain the corresponding central MRA image and multiple maximum density projection (MIP) images; inputting the central MRA image and multiple MIP images into a pre-trained visual language model to obtain the first target image semantic features corresponding to the central MRA image and the second target image semantic features corresponding to the multiple MIP images output by the visual language model; inputting the first target image semantic features and the second target image semantic features into a pre-trained large language model to obtain a predicted structured report for the patient's brain.

[0100] Based on the same concept, this application also provides a computer-readable storage medium storing a computer program, which, when run by a processor, executes the steps of the brain magnetic resonance angiography report generation method provided in the above embodiments.

[0101] Specifically, the storage medium can be a general-purpose storage medium, such as a portable disk or hard disk. When the computer program on the storage medium is run, it can execute the aforementioned brain magnetic resonance angiography report generation method. This method involves preprocessing the patient's brain MRA images to obtain a central MRA image and multiple MIP images. A pre-trained visual language model is then used to obtain the semantic features of a first target image corresponding to the central MRA image and the semantic features of second target images corresponding to the multiple MIP images. These semantic features are then input into a pre-trained large language model to obtain a predicted structured report of the patient's brain. This method solves the technical problems of low efficiency and high error rate caused by the need for doctors to manually observe MRA images to generate reports in existing technologies, achieving the technical effect of increasing work efficiency and reducing the recognition error rate.

[0102] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems and devices described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here. In the several embodiments provided in this application, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division; in actual implementation, there may be other division methods. Furthermore, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Another point is that the displayed or discussed mutual coupling or direct coupling or communication connection may be through some communication interfaces; the indirect coupling or communication connection of devices or units may be electrical, mechanical, or other forms.

[0103] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0104] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0105] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a processor-executable, non-volatile, computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0106] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.< / imagefeature>

Claims

1. A method for generating a brain magnetic resonance angiography report, characterized in that, The method includes: Obtain MRI images of the patient's brain; Brain MRA images are preprocessed to obtain their corresponding central MRA images and multiple maximum density projection MIP images; The central MRA image and multiple MIP images are respectively input into a pre-trained visual language model to obtain the first target image semantic features corresponding to the central MRA image and the second target image semantic features corresponding to the multiple MIP images output by the visual language model. The semantic features of the first target image and the semantic features of the second target image are input into a pre-trained large language model to obtain a structured predictive report of the patient's brain.

2. The method according to claim 1, characterized in that, The visual language model is trained using the following methods: Image preprocessing was performed on brain MRA samples corresponding to multiple historical patients to obtain central MRA samples and multiple MIP samples. The original text reports corresponding to multiple historical patients were structured to obtain original structured text samples. For each brain MRA sample, the central MRA sample, the multiple MIP samples, and the original structured text sample corresponding to the brain MRA sample are combined to form the image-text dataset of the brain MRA sample; The image-text dataset corresponding to each brain MRA sample is used as a training sample to train the visual language model, so that the visual language model outputs the first training image semantic features corresponding to the central MRA sample and the second training image semantic features corresponding to the multiple MIP samples. The image semantic features refer to image features that fuse the semantic features corresponding to the original structured text samples.

3. The method according to claim 2, characterized in that, The visual language model is trained using the following methods: The visual language model is trained by performing image reconstruction on each central MRA sample, fusing the original structured text sample and the multiple MIP samples corresponding to each central MRA sample to perform text reconstruction, performing image alignment on each central MRA sample and its corresponding multiple MIP samples, and performing image-text alignment on the multiple MIP samples and the original structured text sample corresponding to each central MRA sample.

4. The method according to claim 3, characterized in that, The image reconstruction task for each central MRA sample is performed using the following method: Multiple voxel blocks of each central MRA sample are randomly occluded to obtain occluded MRA samples, and the unoccluded parts of the occluded MRA samples are processed by an image encoder. The processing result of the image encoder, the occluded part in the occluded MRA sample, and the position encoding of each voxel block in the occluded MRA sample are input into the image decoder so that the image decoder can reconstruct the occluded MRA sample corresponding to each central MRA sample to obtain the reconstructed MRA sample. Calculate a first loss function between each central MRA sample and its corresponding reconstructed MRA sample, and terminate the image reconstruction task when the first loss function reaches the training completion condition corresponding to the image reconstruction task.

5. The method according to claim 3, characterized in that, The text reconstruction task can be performed using the following methods: After randomly masking the original structured text sample corresponding to each central MRA sample, masked structured samples are obtained, and the masked structured samples are input into the text encoder to obtain masked structured text features; The MIP sample image features of the multiple MIP samples are obtained through an image encoder; The masked structured text features are used as the query vector, and the MIP sample image features are used as the key vector and value vector for image-text fusion to obtain a hybrid image-text vector. The hybrid image-text vector and the full-text position encoding in the masked structured sample are input into the text decoder to obtain the reconstructed structured sample; Calculate a second loss function between the reconstructed portion of the occluded portion of the occluded structured sample and the original structured text sample, and terminate the text reconstruction task when the second loss function reaches the training completion condition corresponding to the text reconstruction task.

6. The method according to claim 3, characterized in that, The image alignment task is performed on each central MRA sample and its corresponding plurality of MIP samples using the following method: The MRA sample image features of each central MRA sample are extracted using an image encoder, and the MIP sample image features of the multiple MIP samples corresponding to each central MRA sample are also extracted using an image encoder. Calculate the first cosine similarity between each MRA sample image feature and each MIP sample image feature; A third loss function is calculated between MRA sample image features and MIP sample image features based on the first cosine similarity. The third loss function is used to narrow the distance between the MRA sample image features and the MIP sample image features in the same case and widen the distance between the MRA sample image features and the MIP sample image features in different cases, so that the image alignment task ends when the third loss function reaches the training completion condition corresponding to the image alignment task.

7. The method according to claim 3, characterized in that, The image-text alignment task is performed on the multiple MIP samples and the original structured text samples corresponding to each central MRA sample using the following method: The MIP sample image features of the multiple MIP samples corresponding to each central MRA sample are extracted by an image encoder, and the structured sample features of the original structured text sample corresponding to each central MRA sample are extracted by a text encoder. Calculate the second cosine similarity between each MIP sample image feature and each structured sample feature; A fourth loss function is calculated between MIP sample image features and structured sample features based on the second cosine similarity. The fourth loss function is used to narrow the distance between the MRA sample image features and the structured sample features under the same case and widen the distance between the MIP sample image features and the structured sample features between different cases, so that the image-text alignment task ends when the fourth loss function reaches the training completion condition corresponding to the image-text alignment task.

8. The method according to claim 1, characterized in that, The large language model is trained using the following methods: Multiple central MRA samples and multiple MIP samples from historical patients are respectively input into the visual language model to obtain the first training image semantic features corresponding to the central MRA samples and the second training image semantic features corresponding to the multiple MIP samples; The semantic features of the first training image, the semantic features of the second training image, and the original structured text samples corresponding to multiple historical patients are input into the large language model. The large language model is trained by low-rank adaptation by calculating language modeling loss and medical entity weighted loss.

9. The method according to claim 1 or 2, characterized in that, The central MRA image and multiple MIP images corresponding to the brain MRA images are obtained in the following way: Brain MRA images are converted into target format images, and bias field correction is applied to the target format images to obtain corrected images; The corrected image is resampled according to the preset voxel block specifications to obtain multiple voxel blocks corresponding to the corrected image; The grayscale of each voxel block corresponding to the corrected image is standardized to obtain a standard image; Extract voxel blocks within a preset coordinate range from the standard image to obtain the central MRA image corresponding to the brain MRA image; Maximum density projection is performed on the central MRA image at preset angle intervals to obtain multiple MIP images corresponding to the brain MRA image.

10. A brain magnetic resonance angiography report generation device, characterized in that, The device includes: The acquisition module acquires the patient's brain magnetic resonance angiography (MRA) images; The preprocessing module performs image preprocessing on the brain MRA images to obtain the corresponding central MRA images and multiple maximum density projection MIP images; The image and text feature extraction module inputs the central MRA image and multiple MIP images into a pre-trained visual language model to obtain the first target image semantic features corresponding to the central MRA image and the second target image semantic features corresponding to the multiple MIP images output by the visual language model. The report generation module inputs the semantic features of the first target image and the semantic features of the second target image into a pre-trained large language model to obtain a predicted structured report of the patient's brain.

Images