Text completion processing method and device for medical image report

Abstract

The invention provides a text completion processing method and device for a medical image report, and relates to the technical field of artificial intelligence. The method comprises the following steps: acquiring a medical image and a report text thereof; checking each report text based on a text generation model of a preset medical image report generation model, and if it is determined that the report text is missing, generating text completion task prompt information according to the missing content; and performing feature extraction on each medical image based on the image encoder to obtain an image feature vector, obtaining an existing report text corresponding to the image feature vector based on the text generation model, and complementing the missing report text according to the existing report text and the text complementing task prompt information to obtain a medical image report. The device executes the method. According to the method and the device provided by the embodiment of the invention, the complete and high-quality medical image report can be efficiently and timely generated.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a text completion processing method and device for medical imaging reports. Background Art

[0002] In recent years, artificial intelligence technologies, particularly deep learning algorithms, have demonstrated powerful capabilities in computer vision and natural language processing tasks. However, existing deep learning models remain insufficient for processing multimodal data in the medical field. First, data labeling, which machine learning relies on, has become a major bottleneck for intelligent assisted diagnosis models due to its high professional requirements, difficulty in obtaining data privacy, high costs, and low efficiency. Historical imaging reports, which reflect the description and diagnosis of medical images and are widely stored electronically in medical institutions, can serve as an effective method for supervised learning. Second, in the past, most medical imaging report generation tasks relied on text training. Medical imaging report generation requires comprehensive analysis of multimodal data such as the patient's medical images (such as CT, MRI) and clinical history, while generating logically rigorous and medically professional reports. This process requires not only the model's language generation capabilities but also a deep understanding of the high-dimensional spatial structure of medical images and the associated professional knowledge.

[0003] Although multimodal models combined with visual models have gradually emerged in recent years, such as BLIP and LLaVA, their application in the field of medical imaging still has significant limitations. On the one hand, medical imaging data has complex spatial structural characteristics. Medical imaging data not only consists of multiple slices, but also needs to capture the spatial context information between slices to construct a complete representation of the three-dimensional anatomical structure. In addition, pathological features in medical images often manifest as subtle morphological or density changes. These features not only rely on the information of a single slice, but also need to be comprehensively judged in combination with the global structure of the context. Therefore, extremely high requirements are placed on the accuracy and understanding ability of the model. On the other hand, the task of generating medical imaging reports requires the model to have the ability to comprehensively process and logically reason multimodal information (including medical images, clinical history, etc.), and requires complete training samples. This technical bottleneck limits the widespread application of deep learning models in the field of medical imaging. Summary of the Invention

[0004] In response to the problems in the prior art, embodiments of the present invention provide a method and apparatus for text completion processing of medical imaging reports, which can at least partially solve the problems in the prior art.

[0005] In one aspect, the present invention provides a method for text completion processing of a medical imaging report, comprising:

[0006] Obtain medical images corresponding to various organs, and report texts corresponding to each medical image;

[0007] The text generation model based on the preset medical imaging report generation model checks each report text. If the inspection result determines that the report text is missing, a text completion task prompt is generated based on the missing content.

[0008] The preset medical imaging report generation model includes an image encoder and a text generation model connected in sequence, and the preset medical imaging report generation model is obtained by progressively training an initial multimodal generation model based on a plurality of report text type data with decreasing data quality in advance;

[0009] Based on the image encoder, feature extraction is performed on each medical image to obtain an image feature vector. Based on the text generation model, an existing report text corresponding to the image feature vector is obtained, and the missing report text is completed according to the existing report text and the text completion task prompt information to obtain a medical image report.

[0010] The step of extracting features from each medical image based on the image encoder to obtain an image feature vector includes:

[0011] Perform organ region cropping on each medical image to obtain the key anatomical regions of the organ regions corresponding to each medical image;

[0012] Based on the residual neural network, the features of each key anatomical region are extracted to obtain the organ region features corresponding to each medical image;

[0013] The organ region features corresponding to each medical image are fused respectively to obtain the image feature vectors corresponding to each medical image.

[0014] The organ region clipping of each medical image is performed to obtain key anatomical regions of the organ regions corresponding to each medical image, including:

[0015] Normalizing each medical image separately to obtain each standardized medical image;

[0016] Based on the pre-trained convolutional neural network, each standardized medical image is segmented to obtain the organ anatomical mask;

[0017] The key regions of each organ anatomical mask are clipped according to the size of the organ anatomical mask and the priority weight of each organ, and the key anatomical regions of the organ regions corresponding to each medical image are obtained.

[0018] The multiple report text types of data of decreasing data quality include clinical description text, image description text, and image diagnosis text in order; accordingly, training the preset medical image report generation model includes:

[0019] Using medical images of multiple organs and their corresponding clinical description texts, image description texts, and image diagnosis texts as samples to train the initial multimodal generation model, thereby obtaining a first multimodal generation model;

[0020] Using medical images of multiple organs and their corresponding clinical description texts and image description texts as samples to train the first multimodal generation model, thereby obtaining a second multimodal generation model;

[0021] Using medical images of multiple organs and their corresponding clinical description texts as samples to train the second multimodal generative model, thereby obtaining a third multimodal generative model;

[0022] Medical images of multiple organs are used as samples to train the third multimodal generation model to obtain the preset medical image report generation model.

[0023] The initial multimodal generation model includes an initial image encoder and an initial text generation model; accordingly, the initial multimodal generation model is trained using medical images of multiple organs and their corresponding clinical description texts, image description texts, and image diagnosis texts as samples to obtain a first multimodal generation model, including:

[0024] Extracting features of medical images of multiple organs based on the initial image encoder to obtain a first image feature vector, and extracting features of clinical description text, image description text, and image diagnosis text based on the initial text generation model to obtain a first text feature vector;

[0025] The first image feature vector and the first text feature vector are associated and aligned through a multimodal cross-attention mechanism, and the initial multimodal generation model is trained through a first loss function constructed based on the multimodal alignment loss to obtain the first multimodal generation model.

[0026] The first multimodal generation model includes a first image encoder and a first text generation model; accordingly, the first multimodal generation model is trained using medical images of multiple organs and their corresponding clinical description texts and image description texts as samples to obtain a second multimodal generation model, including:

[0027] Extracting features from medical images of multiple organs based on the first image encoder to obtain a second image feature vector, and extracting features from clinical description text and image description text based on the first text generation model to obtain a second text feature vector;

[0028] The second image feature vector and the second text feature vector are associated and aligned through a multimodal cross-attention mechanism, and the first multimodal generation model is trained through a second loss function constructed based on a combination of multimodal alignment loss and text generation loss to obtain the second multimodal generation model.

[0029] The second multimodal generative model includes a second image encoder and a second text generation model; accordingly, the second multimodal generative model is trained using medical images of multiple organs and their corresponding clinical description texts as samples to obtain a third multimodal generative model, including:

[0030] Extracting features from medical images of multiple organs based on the second image encoder to obtain a third image feature vector, and extracting features from clinical description text based on the second text generation model to obtain a third text feature vector;

[0031] The third image feature vector and the third text feature vector are associated and aligned through a multimodal cross-attention mechanism, the weight of the second loss function is adjusted to obtain a third loss function, and the second multimodal generation model is trained through the third loss function to obtain the third multimodal generation model.

[0032] The third multimodal generation model includes a third image encoder and a third text generation model; accordingly, the third multimodal generation model is trained using medical images of multiple organs as samples to obtain the preset medical imaging report generation model, including:

[0033] Extracting features from medical images of multiple organs based on the third image encoder to obtain a fourth image feature vector;

[0034] The fourth image feature vectors are associated and aligned through the self-attention mechanism, and the third multimodal generation model is trained through the fourth loss function constructed based on the combination of regional feature aggregation loss and text generation loss to obtain the preset medical imaging report generation model.

[0035] In one aspect, the present invention provides a text completion processing device for a medical imaging report, comprising:

[0036] an acquisition unit, configured to acquire medical images corresponding to a plurality of organs, and report texts corresponding to the respective medical images;

[0037] A generating unit is configured to check each report text based on a text generation model of a preset medical imaging report generation model, and if it is determined that the report text is missing, generate a text completion task prompt information based on the missing content;

[0038] The preset medical imaging report generation model includes an image encoder and a text generation model connected in sequence, and the preset medical imaging report generation model is obtained by progressively training an initial multimodal generation model based on a plurality of report text type data with decreasing data quality in advance;

[0039] The completion unit is used to extract features of each medical image based on the image encoder to obtain an image feature vector, obtain an existing report text corresponding to the image feature vector based on the text generation model, and complete the missing report text according to the existing report text and the text completion task prompt information to obtain a medical image report.

[0040] In another aspect, an embodiment of the present invention provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the following method is implemented:

[0041] Obtain medical images corresponding to various organs, and report texts corresponding to each medical image;

[0042] The text generation model based on the preset medical imaging report generation model checks each report text. If the inspection result determines that the report text is missing, a text completion task prompt is generated based on the missing content.

[0043] The preset medical imaging report generation model includes an image encoder and a text generation model connected in sequence, and the preset medical imaging report generation model is obtained by progressively training an initial multimodal generation model based on a plurality of report text type data with decreasing data quality in advance;

[0044] Based on the image encoder, feature extraction is performed on each medical image to obtain an image feature vector. Based on the text generation model, an existing report text corresponding to the image feature vector is obtained, and the missing report text is completed according to the existing report text and the text completion task prompt information to obtain a medical image report.

[0045] An embodiment of the present invention provides a computer-readable storage medium, including:

[0046] The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the following method is implemented:

[0047] Obtain medical images corresponding to various organs, and report texts corresponding to each medical image;

[0048] The text generation model based on the preset medical imaging report generation model checks each report text. If the inspection result determines that the report text is missing, a text completion task prompt is generated based on the missing content.

[0049] The preset medical imaging report generation model includes an image encoder and a text generation model connected in sequence, and the preset medical imaging report generation model is obtained by progressively training an initial multimodal generation model based on a plurality of report text type data with decreasing data quality in advance;

[0050] Based on the image encoder, feature extraction is performed on each medical image to obtain an image feature vector. Based on the text generation model, an existing report text corresponding to the image feature vector is obtained, and the missing report text is completed according to the existing report text and the text completion task prompt information to obtain a medical image report.

[0051] An embodiment of the present invention further provides a computer program product, comprising a computer program. When the computer program is executed by a processor, the computer program implements the following method:

[0052] Obtain medical images corresponding to various organs, and report texts corresponding to each medical image;

[0053] The text generation model based on the preset medical imaging report generation model checks each report text. If the inspection result determines that the report text is missing, a text completion task prompt is generated based on the missing content.

[0054] The preset medical imaging report generation model includes an image encoder and a text generation model connected in sequence, and the preset medical imaging report generation model is obtained by progressively training an initial multimodal generation model based on a plurality of report text type data with decreasing data quality in advance;

[0055] Based on the image encoder, feature extraction is performed on each medical image to obtain an image feature vector. Based on the text generation model, an existing report text corresponding to the image feature vector is obtained, and the missing report text is completed according to the existing report text and the text completion task prompt information to obtain a medical image report.

[0056] The text completion processing method and device for medical imaging reports provided by the embodiments of the present invention obtain medical images corresponding to multiple organs, and report texts corresponding to each medical image; each report text is checked by a text generation model based on a preset medical imaging report generation model, and if the inspection result determines that the report text is missing, a text completion task prompt information is generated according to the missing content; wherein the preset medical imaging report generation model includes an image encoder and a text generation model connected in sequence, and the preset medical imaging report generation model is obtained by progressively training an initial multimodal generation model based on multiple report text type data with decreasing data quality in sequence; based on the image encoder, feature extraction is performed on each medical image to obtain an image feature vector, based on the text generation model, an existing report text corresponding to the image feature vector is obtained, and the missing report text is completed according to the existing report text and the text completion task prompt information to obtain a medical imaging report. By combining the model trained with image and text data, it is possible to remove the complete dependence on text, realize the completion of missing content in the report text, and efficiently and timely generate a complete and high-quality medical imaging report. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative work. In the drawings:

[0058] Figure 1 The figure is a flowchart of a text completion processing method for a medical imaging report provided by an embodiment of the present invention.

[0059] Figure 2 It is a structural diagram of a text completion processing device for medical imaging reports provided by one embodiment of the present invention.

[0060] Figure 3 A schematic diagram of the physical structure of a computer device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0061] To make the purpose, technical solutions and advantages of the embodiments of the present invention more clear, the embodiments of the present invention are further described in detail below with reference to the accompanying drawings. Here, the illustrative embodiments of the present invention and their descriptions are used to explain the present invention, but are not intended to limit the present invention. It should be noted that, unless there is a conflict, the embodiments and features in the embodiments of this application can be combined with each other in any manner.

[0062] Figure 1FIG. 1 is a flow chart of a text completion processing method for a medical imaging report provided by an embodiment of the present invention. Figure 1 As shown, the text completion processing method for medical imaging reports provided by an embodiment of the present invention includes:

[0063] Step S1: Obtain medical images corresponding to a plurality of organs, and report texts corresponding to each medical image.

[0064] Step S2: Checking each report text based on the text generation model of the preset medical imaging report generation model. If the inspection result determines that the report text is missing, generating a text completion task prompt information based on the missing content;

[0065] Among them, the preset medical imaging report generation model includes an image encoder and a text generation model connected in sequence, and the preset medical imaging report generation model is obtained by progressively training an initial multimodal generation model based on multiple report text type data with decreasing data quality in sequence.

[0066] Step S3: Extract features of each medical image based on the image encoder to obtain an image feature vector, obtain the existing report text corresponding to the image feature vector based on the text generation model, and complete the missing report text according to the existing report text and the text completion task prompt information to obtain a medical imaging report.

[0067] In the above step S1, the device obtains medical images corresponding to a plurality of organs, and report texts corresponding to each medical image. The device can be a computer device that executes the method, such as a server. The medical images corresponding to a plurality of organs, and the report texts corresponding to each medical image are the medical images and report texts in the model usage phase. Medical images (such as CT, MRI, and X-rays) are an important basis for subsequent diagnosis. Radiologists need to analyze massive image data frame by frame and combine them with clinical history, which is a heavy workload and time-consuming.

[0068] In the above step S2, the device checks each report text based on the text generation model of the preset medical imaging report generation model. If the inspection result determines that the report text is missing, a text completion task prompt information is generated according to the missing content;

[0069] The preset medical imaging report generation model includes an image encoder and a text generation model connected in sequence. The preset medical imaging report generation model is obtained by progressively training an initial multimodal generation model based on multiple report text types of data of decreasing data quality. The multiple report text types of data of decreasing data quality are clinical description text, image description text, and image diagnosis text.

[0070] The instructions are as follows:

[0071] Clinical descriptions are typically provided by physicians based on the patient's chief complaint, medical history, and initial examination, and include non-imaging-specific descriptive information. For example, a patient reports right lower quadrant pain and suspected appendicitis.

[0072] Image description text provides a technical description of the anatomical structure and appearance of the image, often an objective description without conclusion. For example: CT scan shows increased fat density around the ileocecal region in the right lower abdomen, but no perforation is found.

[0073] Diagnostic imaging texts draw clear medical conclusions or hypotheses based on the combined diagnostic results of imaging and other information. For example, the imaging manifestations of appendicitis are recommended to be combined with clinical examinations.

[0074] In the past, most medical imaging report generation tasks relied on text training, without integrating multimodal data to build models. This method, which relies solely on text training, can easily lead to inaccurate training results and output diagnostic results if some text is missing. In addition, the medical imaging report generation task requires the model to have the ability to comprehensively process and logically reason about multimodal information (including medical images, clinical history, etc.). This requires complete medical images and their corresponding text descriptions, and at the same time, the medical images and corresponding text descriptions can be trained together.

[0075] Text completion task prompt information can include:

[0076] One or more of completing image diagnosis, generating image diagnosis, completing image description + generating image diagnosis, generating image description + generating image diagnosis.

[0077] The automated generation of medical imaging reports involves multiple tasks (such as completing image descriptions, generating image diagnoses, etc.), but not all scenarios require all subtasks. For example: if the report text already contains a complete image description and only the image diagnosis is missing, then the image diagnosis can be generated directly; if the text is completely missing, both the image description and the image diagnosis need to be generated at the same time. The generation task prompt ensures that generation tasks are only performed where necessary by determining the missing situation, avoiding the processing of redundant tasks and saving computing time and resources. Clarifying the type of generation task (such as "complete image diagnosis", "generate image description + generate image diagnosis") enables the model to generate corresponding reports according to more specific goals, rather than generating globally.

[0078] If it is determined that the inspection result shows that there is no missing report text, the currently acquired report text corresponding to each medical image can be used as the medical image report.

[0079] The multiple report text types of data of decreasing data quality include clinical description text, image description text and image diagnosis text in sequence; accordingly, training the preset medical image report generation model includes:

[0080] Using medical images of multiple organs and their corresponding clinical description texts, image description texts, and image diagnosis texts as samples to train the initial multimodal generation model, thereby obtaining a first multimodal generation model;

[0081] Using medical images of multiple organs and their corresponding clinical description texts and image description texts as samples to train the first multimodal generation model, thereby obtaining a second multimodal generation model;

[0082] Using medical images of multiple organs and their corresponding clinical description texts as samples to train the second multimodal generative model, thereby obtaining a third multimodal generative model;

[0083] Medical images of multiple organs are used as samples to train the third multimodal generation model to obtain the preset medical image report generation model.

[0084] Medical images (such as CT and MRI) are crucial for clinical diagnosis, but their complexity and multidimensional nature make it difficult to fully understand the condition based solely on images. Clinical report text (including clinical descriptions, image descriptions, and imaging diagnoses) summarizes the physician's experience and provides a highly condensed picture of the condition. Combining these two allows models to better understand the multimodal context within the data, thereby improving diagnostic accuracy and depth.

[0085] The initial multimodal generation model includes an initial image encoder and an initial text generation model; accordingly, the initial multimodal generation model is trained using medical images of multiple organs and their corresponding clinical description texts, image description texts, and image diagnosis texts as samples to obtain a first multimodal generation model, including:

[0086] Extracting features of medical images of multiple organs based on the initial image encoder to obtain a first image feature vector, and extracting features of clinical description text, image description text, and image diagnosis text based on the initial text generation model to obtain a first text feature vector;

[0087] The first image feature vector and the first text feature vector are associated and aligned using a multimodal cross-attention mechanism, and the initial multimodal generative model is trained using a first loss function constructed based on a multimodal alignment loss to obtain the first multimodal generative model. Specifically, the initial multimodal generative model is iteratively trained for multiple rounds, and the multimodal generative model parameters are updated using the first loss function and the ADAM optimization algorithm to obtain the first multimodal generative model.

[0088] Image and text data have modal differences in representation (for example, images are pixel-level data, while text is sequence data). It is difficult to ensure information consistency by directly fusing the data. A multimodal cross-attention mechanism is used to effectively associate and align the image feature vector with the text feature vector, thereby capturing the potential semantic relationship between the two modal data. Text information in clinical scenarios may be incomplete, noisy, or have inconsistent levels (for example, some cases only have descriptive text but no diagnostic conclusions). Using a progressive training strategy, samples are introduced one by one in order of decreasing text completeness, so that the model learns basic representations from "high-quality data" and then gradually adapts to "low-quality data", thereby improving the robustness of the model.

[0089] The first multimodal generation model includes a first image encoder and a first text generation model; accordingly, the first multimodal generation model is trained using medical images of multiple organs and their corresponding clinical description texts and image description texts as samples to obtain a second multimodal generation model, including:

[0090] Extracting features from medical images of multiple organs based on the first image encoder to obtain a second image feature vector, and extracting features from clinical description text and image description text based on the first text generation model to obtain a second text feature vector;

[0091] The second image feature vector and the second text feature vector are associated and aligned through a multimodal cross-attention mechanism, and the first multimodal generation model is trained through a second loss function constructed based on a combination of multimodal alignment loss and text generation loss to obtain the second multimodal generation model.

[0092] The calculation formula of the second loss function L2 is:

[0093] L2=ɑ·L m +β·L t

[0094] Among them, L m is the multimodal alignment loss, α is the weight corresponding to the multimodal alignment loss, L t is the text generation loss, and β is the weight corresponding to the text generation loss.

[0095] In the first stage, the first multimodal generative model with preliminary fusion capabilities for medical images and texts has been trained, but basic alignment alone may not be sufficient to handle the deep mapping relationship between more complex multimodal semantics. In the second stage, the first multimodal generative model is trained to gradually adapt to scenarios where there is a lack of textual information on image diagnosis, and to learn to rely on image and text generation capabilities to complete the missing image diagnosis part. Therefore, the training model needs to optimize the image-text alignment task and the text generation task at the same time, which may lead to excessive emphasis or weakening of a certain task. Through the weighted combination of two loss functions (modal alignment loss and text generation loss), the coordinated optimization of multi-task objectives is achieved to ensure the balance between alignment and generation.

[0096] The second multimodal generative model includes a second image encoder and a second text generation model; accordingly, the second multimodal generative model is trained using medical images of multiple organs and their corresponding clinical description texts as samples to obtain a third multimodal generative model, including:

[0097] Extracting features from medical images of multiple organs based on the second image encoder to obtain a third image feature vector, and extracting features from clinical description text based on the second text generation model to obtain a third text feature vector;

[0098] The third image feature vector and the third text feature vector are associated and aligned using a multimodal cross-attention mechanism, the weight of the second loss function is adjusted to obtain a third loss function, and the second multimodal generative model is trained using the third loss function to obtain the third multimodal generative model. The second multimodal generative model is iteratively trained for multiple rounds, and the multimodal generative model parameters are updated using the third loss function and the ADAM optimization algorithm to obtain the third multimodal generative model.

[0099] The third stage further reduces the text type dimension in the training data (using only clinical description text) and adjusts the loss weight more finely to further improve the generalization ability and robustness of the model.

[0100] The third stage removes image description text and trains the model performance under simplified input conditions, reducing the model's excessive reliance on multimodal joint input and achieving better versatility.

[0101] In medical scenarios, image modalities are relatively fixed and standardized, while text modalities are diverse and asymmetric. This is especially true when the text is incomplete or of low quality, such as when only clinical descriptions are provided, or only clinical descriptions and partial image descriptions. We focus on aligning the input image with the clinical description text, strengthening image-driven multimodal analysis capabilities while adapting the model to simplified or even oversimplified text modalities.

[0102] The third stage requires readjusting the priority weights between the alignment task and the text generation task under streamlined input conditions to adapt to the simplified characteristics of the data. By adjusting the weight parameters (α, β) in the loss function, it is ensured that the multimodal association alignment task (key task) remains the dominant goal, while the text generation task is stabilized and not completely weakened. After multiple rounds of training, especially after the joint optimization of the image encoder and the text generation model, the model has the ability to understand medical image features and align image features with text semantics. The third stage of training reduced the types of report texts and focused on simple input scenarios of "image + clinical description", enabling the model to infer and complete other missing texts based on a small amount of input.

[0103] The third multimodal generation model includes a third image encoder and a third text generation model; accordingly, the third multimodal generation model is trained using medical images of multiple organs as samples to obtain the preset medical imaging report generation model, including:

[0104] Extracting features from medical images of multiple organs based on the third image encoder to obtain a fourth image feature vector;

[0105] The fourth image feature vectors are aligned using a self-attention mechanism, and the third multimodal generative model is trained using a fourth loss function constructed based on a combination of a regional feature aggregation loss and a text generation loss to obtain the preset medical imaging report generation model. The third multimodal generative model is iteratively trained for multiple rounds, and the parameters of the multimodal generative model are updated using the fourth loss function and the ADAM optimization algorithm to obtain a fourth multimodal generative model.

[0106] In the first three stages, the model relies on the multimodal alignment logic of image and text input; without sufficient text supplementation, the model may find it difficult to generate accurate semantic content. The fourth stage is directly based on image input training, emphasizing the use of the self-attention mechanism of image features to make the internal correlation of image features closer, and independently generate text from a single image modality. Medical images usually contain multi-region, complex local information (such as lesion location, range, density, etc.), and existing global feature methods may not be able to capture this regional relationship. In the fourth stage, the self-attention mechanism is introduced to associate and align image feature vectors (that is, focus on the semantic relationship between different regions in the same image) to build a deeper feature understanding and association goal.

[0107] In the first three phases (especially the second and third phases), the text generation task was repeatedly optimized by aligning medical images with corresponding text. By the fourth phase, the model's text generation capabilities had built on the foundations of the previous phases, enabling it to generate corresponding image descriptions or diagnostic text after inputting images.

[0108] Through multi-stage iterative training, the model has learned the alignment between medical images and text. Simultaneously, it has strengthened its capabilities in image feature extraction, text generation, and image-modality-driven generation, enabling it to comprehensively process input images and report text. Based on this foundation, the model can not only distinguish the completeness of text input but also generate prompts and subsequent completion strategies that match the input.

[0109] In step S3, the device extracts features from each medical image using the image encoder to obtain image feature vectors. The device then retrieves existing report text corresponding to the image feature vectors based on the text generation model. The device then completes any missing report text based on the existing report text and the text completion task prompts to generate a medical imaging report. The text generation model not only uses image features and task prompts but also references existing report text as contextual basis for generating the completed content, ensuring semantic coherence and increased reliability of the generated medical imaging report.

[0110] The extracting features of each medical image based on the image encoder to obtain an image feature vector includes:

[0111] Performing organ region clipping on each medical image to obtain key anatomical regions of the organ region corresponding to each medical image; performing organ region clipping on each medical image to obtain key anatomical regions of the organ region corresponding to each medical image includes:

[0112] Each medical image is normalized to obtain each standardized medical image I seg The calculation formula is as follows:

[0113]

[0114] Among them, I raw is medical imaging data, μ seg is the standardized mean, σ seg is the standardized standard deviation.

[0115] Based on the pre-trained convolutional neural network, each standardized medical image is segmented to obtain the organ anatomical mask; the convolutional neural network can be specifically a 3D U-Net model, and the calculation formula is as follows:

[0116] M c =f θ (I seg ),c∈{1,2,…,k}

[0117] Among them, M c is the anatomical mask of the organ corresponding to type c (such as liver, lung, etc.), used to mark the organ area, fθ is a pre-trained 3D U-Net model, and k is the number of organ types.

[0118] According to the size of the organ anatomical mask and the priority weight of each organ, the key area of each organ anatomical mask is clipped to obtain the key anatomical area of the organ area corresponding to each medical image. The priority weight of the cth organ is calculated using the following formula:

[0119] p(c)=softmax(-α·Area(M c ))

[0120] Among them, p(c) is the priority weight of the cth organ, which indicates the importance of the organ in regional cropping. Area(M c ) represents the size of the anatomical mask of the cth organ, α is the temperature coefficient, which is used to control the sensitivity of the area difference of the anatomical masks of different organs to the weight, and softmax is the soft attention function.

[0121] Medical images often contain a large amount of background information (such as normal tissue and non-lesion areas). In diagnostic tasks, this information can interfere with model judgment and reduce its accuracy. Using a pre-trained 3D U-Net model for organ segmentation focuses on key anatomical regions (such as the liver and lungs) in the image, helping the model extract more meaningful pathological features from these anatomical regions.

[0122] Each organ has a different importance in medical imaging diagnosis. For example, the kidney region is clearly the primary focus in the diagnosis of kidney tumors, while the lung region requires particular attention in the diagnosis of lung nodules. Using a priority weight formula for key region clipping, each organ's weight is calculated, ensuring that the most important organ regions for medical diagnosis are prioritized in feature extraction and subsequent decision-making. The introduction of area and temperature coefficients dynamically adjusts the importance of different organ features, allowing for more intelligent, focused processing of key regions.

[0123] In addition, the image feature vector extraction process here is the same as that in the training process.

[0124] Based on the residual neural network, features are extracted from each key anatomical region to obtain organ region features corresponding to each medical image. The residual neural network can be a ResNet-34 model. The feature extraction content can include texture features, shape features, and lesion features.

[0125] The organ region features corresponding to each medical image are fused respectively to obtain the image feature vectors corresponding to each medical image.

[0126] Pathological features in medical images often include texture features (such as irregular density changes), shape features (such as organ deformation or growth), and lesion region features (such as tumor location, size, or morphology). Using a deep convolutional neural network (ResNet-34) to extract features from key anatomical regions effectively captures complex texture, shape, and lesion characteristics, making feature representation more detailed and accurate. Features extracted from a single organ region can only reflect the local characteristics of that organ and cannot fully express the potential correlations between multiple organs. Fusion of features from all organ regions to form a global image feature vector can empower the model with stronger comprehensive analysis capabilities, thereby improving the accuracy of diagnostic decisions. Certain medical diagnoses require consideration of the relationships between multiple organs. For example, abnormal liver function may be associated with both the pancreas and gallbladder. Integrating texture, shape, and lesion information from different organs into a unified feature space can capture cross-organ pathological associations and improve the ability to diagnose complex symptoms.

[0127] The text completion processing method for medical imaging reports provided by the embodiment of the present invention has the following beneficial technical effects:

[0128] First, by collecting medical images of several different organs and their corresponding different types of report text, and by realistically combining image and text data, we ensure that the model's learning content closely matches the needs of medical scenarios. The diversity of report text provides the model with learning opportunities from shallow descriptions (image description) to deep semantics (image diagnosis), improving the model's multi-dimensional expressive capabilities.

[0129] Secondly, by gradually reducing the completeness of the reports, the model can gradually adapt to scenarios with incomplete text input, increasing its robustness in processing complex data in real clinical scenarios. This progressive design allows the model to gradually divest itself of its complete reliance on text, thereby improving the level of image-driven natural language generation.

[0130] Third, task prompts enable the model to perform targeted generation tasks, avoiding inefficient or redundant text generation and conserving computing resources. The model can flexibly output corresponding imaging reports based on different scenario requirements (partially missing or incomplete input text), integrating image and text features to provide high-quality auxiliary support for medical imaging diagnosis.

[0131] The text completion processing method for medical imaging reports provided in an embodiment of the present invention obtains medical images corresponding to multiple organs, and report texts corresponding to each medical image; checks each report text based on a text generation model of a preset medical imaging report generation model, and if it is determined that the inspection result is that the report text is missing, generates text completion task prompt information based on the missing content; wherein the preset medical imaging report generation model includes an image encoder and a text generation model connected in sequence, and the preset medical imaging report generation model is obtained by progressively training an initial multimodal generation model based on multiple report text type data with decreasing data quality in sequence; based on the image encoder, feature extraction is performed on each medical image to obtain an image feature vector, based on the text generation model, an existing report text corresponding to the image feature vector is obtained, and the missing report text is completed according to the existing report text and the text completion task prompt information to obtain a medical imaging report. By combining the model trained with image and text data, it is possible to remove the complete dependence on text, realize the completion of missing content in the report text, and efficiently and timely generate a complete and high-quality medical imaging report.

[0132] Furthermore, the feature extraction of each medical image based on the image encoder to obtain an image feature vector includes:

[0133] Organ regions are cropped for each medical image to obtain key anatomical regions of the organ regions corresponding to each medical image. This can be described with reference to the above embodiment and will not be repeated here.

[0134] Based on the residual neural network, feature extraction is performed on each key anatomical region to obtain organ region features corresponding to each medical image. Please refer to the above embodiment for description and no further details will be given.

[0135] The organ region features corresponding to each medical image are fused to obtain the image feature vectors corresponding to each medical image.

[0136] Furthermore, the organ region clipping of each medical image is performed to obtain the key anatomical regions of the organ regions corresponding to each medical image, including:

[0137] Normalization processing is performed on each medical image to obtain each standardized medical image; the above description can be referred to in the embodiment, which will not be repeated here.

[0138] Based on the pre-trained convolutional neural network, each standardized medical image is subjected to organ segmentation to obtain an organ anatomical mask; the above description can be referred to in the embodiment and will not be repeated here.

[0139] The key regions of each organ anatomical mask are clipped according to the size of the organ anatomical mask and the priority weight of each organ to obtain the key anatomical regions of the organ regions corresponding to each medical image.

[0140] Furthermore, the multiple report text types of data with decreasing data quality include clinical description text, image description text and image diagnosis text in sequence; accordingly, training the preset medical image report generation model includes:

[0141] The medical images of multiple organs and their corresponding clinical description texts, image description texts and image diagnosis texts are used as samples to train the initial multimodal generation model to obtain a first multimodal generation model; the above embodiment can be referred to for description and will not be repeated here.

[0142] Medical images of multiple organs and their corresponding clinical description texts and image description texts are used as samples to train the first multimodal generation model to obtain the second multimodal generation model; the above embodiment can be referred to for description and will not be repeated here.

[0143] Medical images of multiple organs and their corresponding clinical description texts are used as samples to train the second multimodal generation model to obtain a third multimodal generation model; the above embodiment can be referred to for description and will not be repeated here.

[0144] The third multimodal generation model is trained using medical images of multiple organs as samples to obtain the preset medical image report generation model.

[0145] Furthermore, the initial multimodal generation model includes an initial image encoder and an initial text generation model; accordingly, the initial multimodal generation model is trained using medical images of multiple organs and their corresponding clinical description texts, image description texts, and image diagnosis texts as samples to obtain a first multimodal generation model, including:

[0146] Based on the initial image encoder, feature extraction is performed on medical images of multiple organs to obtain a first image feature vector, and based on the initial text generation model, feature extraction is performed on clinical description text, image description text and image diagnosis text to obtain a first text feature vector; please refer to the above embodiment for description and will not repeat them here.

[0147] The first image feature vector and the first text feature vector are associated and aligned using a multimodal cross-attention mechanism, and the initial multimodal generative model is trained using a first loss function constructed based on a multimodal alignment loss to obtain the first multimodal generative model. This can be explained with reference to the above embodiment and will not be repeated here.

[0148] Furthermore, the first multimodal generative model includes a first image encoder and a first text generation model; accordingly, the first multimodal generative model is trained using medical images of multiple organs and their corresponding clinical description texts and image description texts as samples to obtain a second multimodal generative model, including:

[0149] Based on the first image encoder, feature extraction is performed on medical images of multiple organs to obtain a second image feature vector, and based on the first text generation model, feature extraction is performed on clinical description text and image description text to obtain a second text feature vector; please refer to the above embodiment for description and will not repeat it here.

[0150] The second image feature vector and the second text feature vector are aligned using a multimodal cross-attention mechanism, and the first multimodal generative model is trained using a second loss function constructed based on a combination of a multimodal alignment loss and a text generation loss to obtain the second multimodal generative model. This can be explained with reference to the above embodiment and will not be repeated here.

[0151] Furthermore, the second multimodal generative model includes a second image encoder and a second text generation model; accordingly, the second multimodal generative model is trained using medical images of multiple organs and their corresponding clinical description texts as samples to obtain a third multimodal generative model, including:

[0152] Based on the second image encoder, feature extraction is performed on medical images of multiple organs to obtain a third image feature vector, and based on the second text generation model, feature extraction is performed on the clinical description text to obtain a third text feature vector; please refer to the above embodiment for description and will not repeat it here.

[0153] The third image feature vector and the third text feature vector are associated and aligned using a multimodal cross-attention mechanism, the weight of the second loss function is adjusted to obtain a third loss function, and the second multimodal generative model is trained using the third loss function to obtain the third multimodal generative model. This description can be made with reference to the above embodiment and will not be repeated here.

[0154] Furthermore, the third multimodal generation model includes a third image encoder and a third text generation model; accordingly, the third multimodal generation model is trained using medical images of multiple organs as samples to obtain the preset medical imaging report generation model, including:

[0155] Based on the third image encoder, feature extraction is performed on medical images of multiple organs to obtain a fourth image feature vector; reference may be made to the above embodiment for explanation, which will not be repeated here.

[0156] The fourth image feature vectors are aligned through a self-attention mechanism, and the third multimodal generation model is trained using a fourth loss function constructed based on a combination of regional feature aggregation loss and text generation loss to obtain the preset medical imaging report generation model. This can be explained with reference to the above embodiment and will not be repeated here.

[0157] Figure 2 FIG. 1 is a schematic diagram of a text completion processing device for a medical imaging report according to an embodiment of the present invention. Figure 2 As shown, the text completion processing device for medical imaging reports provided by an embodiment of the present invention includes an acquisition unit 201, a generation unit 202 and a completion unit 203, wherein:

[0158] The acquisition unit 201 is used to acquire medical images corresponding to multiple organs, and report texts corresponding to each medical image; the generation unit 202 is used to check each report text based on a text generation model of a preset medical image report generation model. If the inspection result is determined to be missing in the report text, a text completion task prompt information is generated according to the missing content; wherein the preset medical image report generation model includes an image encoder and a text generation model connected in sequence, and the preset medical image report generation model is obtained by progressively training an initial multimodal generation model based on multiple report text type data with decreasing data quality in sequence; the completion unit 203 is used to extract features of each medical image based on the image encoder to obtain an image feature vector, obtain an existing report text corresponding to the image feature vector based on the text generation model, and complete the missing report text according to the existing report text and the text completion task prompt information to obtain a medical image report.

[0159] Specifically, the acquisition unit 201 in the device is used to acquire medical images corresponding to multiple organs, and report texts corresponding to each medical image; the generation unit 202 is used to check each report text based on a text generation model of a preset medical image report generation model. If the inspection result is determined to be missing in the report text, a text completion task prompt information is generated according to the missing content; wherein the preset medical image report generation model includes an image encoder and a text generation model connected in sequence, and the preset medical image report generation model is obtained by progressively training an initial multimodal generation model based on multiple report text type data with decreasing data quality in advance; the completion unit 203 is used to extract features of each medical image based on the image encoder to obtain an image feature vector, obtain an existing report text corresponding to the image feature vector based on the text generation model, and complete the missing report text according to the existing report text and the text completion task prompt information to obtain a medical image report.

[0160] The text completion processing device for medical image reports provided in an embodiment of the present invention obtains medical images corresponding to multiple organs and report texts corresponding to each medical image; checks each report text based on a text generation model of a preset medical image report generation model, and if it is determined that the inspection result is that the report text is missing, generates text completion task prompt information based on the missing content; wherein the preset medical image report generation model includes an image encoder and a text generation model connected in sequence, and the preset medical image report generation model is obtained by progressively training an initial multimodal generation model based on multiple report text type data with decreasing data quality in sequence; based on the image encoder, feature extraction is performed on each medical image to obtain an image feature vector, based on the text generation model, existing report text corresponding to the image feature vector is obtained, and the missing report text is completed according to the existing report text and the text completion task prompt information to obtain a medical image report. The model trained by combining image and text data can remove the complete dependence on text, realize the completion of missing content in the report text, and can efficiently and timely generate a complete and high-quality medical image report.

[0161] The embodiment of the present invention provides an embodiment of a text completion processing device for medical imaging reports, which can be specifically used to execute the processing flow of the above-mentioned method embodiments. Its functions are not described in detail here, and reference can be made to the detailed description of the above-mentioned method embodiments.

[0162] Figure 3 A schematic diagram of the physical structure of a computer device provided in an embodiment of the present invention is shown in FIG. Figure 3 As shown, the computer device includes: a memory 301, a processor 302, and a computer program stored in the memory 301 and executable on the processor 302. When the processor 302 executes the computer program, the following method is implemented:

[0163] Obtain medical images corresponding to various organs, and report texts corresponding to each medical image;

[0164] The text generation model based on the preset medical imaging report generation model checks each report text. If the inspection result determines that the report text is missing, a text completion task prompt is generated based on the missing content.

[0165] The preset medical imaging report generation model includes an image encoder and a text generation model connected in sequence, and the preset medical imaging report generation model is obtained by progressively training an initial multimodal generation model based on a plurality of report text type data with decreasing data quality in advance;

[0166] Based on the image encoder, feature extraction is performed on each medical image to obtain an image feature vector. Based on the text generation model, an existing report text corresponding to the image feature vector is obtained, and the missing report text is completed according to the existing report text and the text completion task prompt information to obtain a medical image report.

[0167] This embodiment discloses a computer program product, which includes a computer program. When the computer program is executed by a processor, the following method is implemented:

[0168] Obtain medical images corresponding to various organs, and report texts corresponding to each medical image;

[0169] The text generation model based on the preset medical imaging report generation model checks each report text. If the inspection result determines that the report text is missing, a text completion task prompt is generated based on the missing content.

[0170] The preset medical imaging report generation model includes an image encoder and a text generation model connected in sequence, and the preset medical imaging report generation model is obtained by progressively training an initial multimodal generation model based on a plurality of report text type data with decreasing data quality in advance;

[0171] Based on the image encoder, feature extraction is performed on each medical image to obtain an image feature vector. Based on the text generation model, an existing report text corresponding to the image feature vector is obtained, and the missing report text is completed according to the existing report text and the text completion task prompt information to obtain a medical image report.

[0172] This embodiment provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program. When the computer program is executed by a processor, the following method is implemented:

[0173] Obtain medical images corresponding to various organs, and report texts corresponding to each medical image;

[0174] The text generation model based on the preset medical imaging report generation model checks each report text. If the inspection result determines that the report text is missing, a text completion task prompt is generated based on the missing content.

[0175] The preset medical imaging report generation model includes an image encoder and a text generation model connected in sequence, and the preset medical imaging report generation model is obtained by progressively training an initial multimodal generation model based on a plurality of report text type data with decreasing data quality in advance;

[0176] Based on the image encoder, feature extraction is performed on each medical image to obtain an image feature vector. Based on the text generation model, an existing report text corresponding to the image feature vector is obtained, and the missing report text is completed according to the existing report text and the text completion task prompt information to obtain a medical image report.

[0177] Compared with the technical solutions in the prior art, the embodiments of the present invention provide a text completion processing method for medical imaging reports, which obtains medical images corresponding to multiple organs and report texts corresponding to each medical image; checks each report text based on a text generation model of a preset medical imaging report generation model, and if the inspection result determines that the report text is missing, generates text completion task prompt information based on the missing content; wherein the preset medical imaging report generation model includes an image encoder and a text generation model connected in sequence, and the preset medical imaging report generation model is obtained by progressively training an initial multimodal generation model based on multiple report text type data with decreasing data quality in sequence; based on the image encoder, feature extraction is performed on each medical image to obtain an image feature vector, based on the text generation model, an existing report text corresponding to the image feature vector is obtained, and the missing report text is completed according to the existing report text and the text completion task prompt information to obtain a medical imaging report. By combining the model trained with image and text data, it is possible to remove the complete dependence on text, realize the completion of missing content in the report text, and efficiently and timely generate a complete and high-quality medical imaging report.

[0178] It will be understood by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0179] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0180] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0181] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0182] Throughout this specification, reference to terms such as "one embodiment," "a specific embodiment," "some embodiments," "for example," "example," "specific example," or "some examples" means that the specific features, structures, materials, or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, schematic representations of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

[0183] The specific embodiments described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A text completion processing method for medical imaging reports, characterized in that: include: Obtain medical images corresponding to various organs, and report texts corresponding to each medical image; The text generation model based on the preset medical imaging report generation model checks each report text. If the inspection result determines that the report text is missing, a text completion task prompt is generated based on the missing content. The preset medical imaging report generation model includes an image encoder and a text generation model connected in sequence, and the preset medical imaging report generation model is obtained by progressively training an initial multimodal generation model based on a plurality of report text type data with decreasing data quality in advance; Based on the image encoder, feature extraction is performed on each medical image to obtain an image feature vector. Based on the text generation model, an existing report text corresponding to the image feature vector is obtained, and the missing report text is completed according to the existing report text and the text completion task prompt information to obtain a medical image report.

2. The text completion processing method for medical imaging reports according to claim 1, characterized in that: The extracting features of each medical image based on the image encoder to obtain an image feature vector includes: Perform organ region cropping on each medical image to obtain the key anatomical regions of the organ regions corresponding to each medical image; Based on the residual neural network, the features of each key anatomical region are extracted to obtain the organ region features corresponding to each medical image; The organ region features corresponding to each medical image are fused respectively to obtain the image feature vectors corresponding to each medical image.

3. The text completion processing method for medical imaging reports according to claim 2, characterized in that: The organ region clipping of each medical image is performed to obtain key anatomical regions of the organ region corresponding to each medical image, including: Normalizing each medical image separately to obtain each standardized medical image; Based on the pre-trained convolutional neural network, each standardized medical image is segmented to obtain the organ anatomical mask; The key regions of each organ anatomical mask are clipped according to the size of the organ anatomical mask and the priority weight of each organ, and the key anatomical regions of the organ regions corresponding to each medical image are obtained.

4. The text completion processing method for medical imaging reports according to any one of claims 1 to 3, characterized in that: The multiple report text types of data of decreasing data quality include clinical description text, image description text and image diagnosis text in sequence; accordingly, training the preset medical image report generation model includes: Using medical images of multiple organs and their corresponding clinical description texts, image description texts, and image diagnosis texts as samples to train the initial multimodal generation model, thereby obtaining a first multimodal generation model; Using medical images of multiple organs and their corresponding clinical description texts and image description texts as samples to train the first multimodal generation model, thereby obtaining a second multimodal generation model; Using medical images of multiple organs and their corresponding clinical description texts as samples to train the second multimodal generative model, thereby obtaining a third multimodal generative model; Medical images of multiple organs are used as samples to train the third multimodal generation model to obtain the preset medical image report generation model.

5. The text completion processing method for medical imaging reports according to claim 4, characterized in that: The initial multimodal generation model includes an initial image encoder and an initial text generation model; accordingly, the initial multimodal generation model is trained using medical images of multiple organs and their corresponding clinical description texts, image description texts, and image diagnosis texts as samples to obtain a first multimodal generation model, including: Extracting features of medical images of multiple organs based on the initial image encoder to obtain a first image feature vector, and extracting features of clinical description text, image description text, and image diagnosis text based on the initial text generation model to obtain a first text feature vector; The first image feature vector and the first text feature vector are associated and aligned through a multimodal cross-attention mechanism, and the initial multimodal generation model is trained through a first loss function constructed based on the multimodal alignment loss to obtain the first multimodal generation model.

6. The text completion processing method for medical imaging reports according to claim 5, characterized in that: The first multimodal generation model includes a first image encoder and a first text generation model; accordingly, the first multimodal generation model is trained using medical images of multiple organs and their corresponding clinical description texts and image description texts as samples to obtain a second multimodal generation model, including: Extracting features from medical images of multiple organs based on the first image encoder to obtain a second image feature vector, and extracting features from clinical description text and image description text based on the first text generation model to obtain a second text feature vector; The second image feature vector and the second text feature vector are associated and aligned through a multimodal cross-attention mechanism, and the first multimodal generation model is trained through a second loss function constructed based on a combination of multimodal alignment loss and text generation loss to obtain the second multimodal generation model.

7. The text completion processing method for medical imaging reports according to claim 6, characterized in that: The second multimodal generative model includes a second image encoder and a second text generation model; accordingly, the second multimodal generative model is trained using medical images of multiple organs and their corresponding clinical description texts as samples to obtain a third multimodal generative model, including: Extracting features from medical images of multiple organs based on the second image encoder to obtain a third image feature vector, and extracting features from clinical description text based on the second text generation model to obtain a third text feature vector; The third image feature vector and the third text feature vector are associated and aligned through a multimodal cross-attention mechanism, the weight of the second loss function is adjusted to obtain a third loss function, and the second multimodal generation model is trained through the third loss function to obtain the third multimodal generation model.

8. The text completion processing method for medical imaging reports according to claim 7, characterized in that: The third multimodal generation model includes a third image encoder and a third text generation model; accordingly, the third multimodal generation model is trained using medical images of multiple organs as samples to obtain the preset medical imaging report generation model, including: Extracting features from medical images of multiple organs based on the third image encoder to obtain a fourth image feature vector; The fourth image feature vectors are associated and aligned through the self-attention mechanism, and the third multimodal generation model is trained through the fourth loss function constructed based on the combination of regional feature aggregation loss and text generation loss to obtain the preset medical imaging report generation model.

9. A text completion processing device for medical imaging reports, characterized in that: include: an acquisition unit, configured to acquire medical images corresponding to a plurality of organs, and report texts corresponding to the respective medical images; A generating unit is configured to check each report text based on a text generation model of a preset medical imaging report generation model, and if it is determined that the report text is missing, generate a text completion task prompt information based on the missing content; The preset medical imaging report generation model includes an image encoder and a text generation model connected in sequence, and the preset medical imaging report generation model is obtained by progressively training an initial multimodal generation model based on a plurality of report text type data with decreasing data quality in advance; The completion unit is used to extract features of each medical image based on the image encoder to obtain an image feature vector, obtain an existing report text corresponding to the image feature vector based on the text generation model, and complete the missing report text according to the existing report text and the text completion task prompt information to obtain a medical image report.

10. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the method according to any one of claims 1 to 8 is implemented.

11. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

12. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images