Multi-agent-based medical image report generation method

Abstract

The invention discloses a medical image report generation method based on multiple agents. The method comprises the following steps: step 1, constructing an image segmentation agent, a cue word retrieval agent, a report generation agent and an image database; 2, inputting the medical image into an image segmentation agent, and generating a focus label through the image segmentation agent; 3, inputting the focus label into a cue word retrieval intelligent agent, and obtaining an example report related to the focus label by the cue word retrieval intelligent agent; and 4, taking the focus label and the example report as input of a report generation agent, outputting a preliminary image report by the report generation agent, and then introducing a terminology condition constraint and reinforcement mechanism to correct the preliminary image report to generate a final image report. According to the invention, the workload of doctors can be reduced while the image analysis report which is accurate and meets the medical standard is generated.

Description

Technical Field

[0001] The present invention relates to the field of medical imaging technology, and specifically to a medical imaging report generation method based on multi-agents. Background Art

[0002] With the rapid development of medical imaging technology, the amount of medical imaging data generated and processed by the medical industry is increasing, including CT, MRI, ultrasound, and X-ray. These imaging data are not only huge in size, but also contain rich clinical information, which is crucial for early diagnosis and treatment decisions of diseases. However, the current analysis and report generation of medical images still mainly rely on the manual judgment and experience of doctors, which is not only time-consuming and labor-intensive, but also easily affected by subjective factors, resulting in the inability to guarantee the accuracy and consistency of the analysis results.

[0003] The traditional medical image analysis process usually includes image preprocessing, lesion identification, diagnostic analysis, and final report writing. This series of steps requires a lot of time and human resources, especially when facing high-complication clinical needs, it is often difficult to respond efficiently. In addition, doctors need to refer to a large amount of standard literature and past cases when writing reports, which further increases the workload. Summary of the invention

[0004] The purpose of the present invention is to provide a method for generating medical image reports based on multi-agents. The present invention can generate accurate image analysis reports that meet medical standards and reduce the workload of doctors.

[0005] The technical solution of the present invention is a method for generating a medical imaging report based on a multi-agent, comprising the following steps:

[0006] Step 1: Construct image segmentation agent, prompt word retrieval agent, report generation agent and image database;

[0007] Step 2: Input the medical image into the image segmentation agent, and generate the lesion label through the image segmentation agent;

[0008] Step 3: Input the lesion label into the prompt word retrieval agent, and the prompt word retrieval agent retrieves sample reports related to the lesion label;

[0009] Step 4: The lesion label and sample report are used as input to the report generation agent, which outputs a preliminary imaging report. The mechanism of professional terminology condition constraints and reinforcement is then introduced to correct the preliminary imaging report and generate the final imaging report.

[0010] In the aforementioned multi-agent-based medical image report generation method, in step 2, the process of the image segmentation agent generating lesion labels is: extracting original image metadata from the input medical image and preprocessing the medical image; constructing an image segmentation model, segmenting the preprocessed medical image, and extracting segmented image metadata from the segmented medical image; constructing and training a large language model to obtain a fine-tuned large language model, inputting the original image metadata and segmented image metadata into the fine-tuned large language model to generate lesion labels.

[0011] In the aforementioned multi-agent-based medical image report generation method, the specific process of image segmentation model segmentation is to use the projection layer to extract layer-by-layer features as the input of the encoder, so as to provide a suitable basic data format for subsequent feature extraction and processing; use deep convolution for convolution, move the deep convolution upward to calculate the features, introduce deep convolution scaling, and reduce the redundancy of cross-channel context; apply residual connection between the input and the last scaling layer, so that the image segmentation model can learn more complex feature representation; use layer normalization in the encoder block to normalize the convolution representation, enhance the convergence of the image segmentation model and reduce overfitting, perform downsampling operations at different stages, gradually reduce the image resolution of the medical image, and increase the number of feature channels to extract higher-level semantic features; perform upsampling in the decoder part, gradually restore the resolution of the medical image, splice the features of different levels of the encoder stage with the features of the corresponding levels of the decoder, fuse multi-scale feature information, and improve the accuracy of segmentation; finally generate the segmentation result of the medical image, and accurately segment different tissues and organs in the medical image.

[0012] In the aforementioned multi-agent-based medical imaging report generation method, the training process of the large language model is specifically to widely collect medical imaging-related text data and form a data set; select an open source large model, construct a loss function to dynamically adjust the learning rate in real time to achieve efficient parameter optimization; the loss function is constructed as follows:

[0013]

[0014] Among them, y i is the true value of the i-th category label, 0 or 1, p i is the predicted probability of the large language model for the i-th category label, and n is the number of category label samples;

[0015] Then the loss correction factor is introduced, the formula is as follows:

[0016] C=|p2-P|

[0017] Where p1 is the probability of the presence of pulmonary nodules predicted by the large language model, and p2 is the probability of the presence of hemoptysis;

[0018] Get the final loss function value:

[0019] L'=L×(1+C).

[0020] In the aforementioned multi-agent-based medical imaging report generation method, in step three, the process of the prompt word retrieval agent retrieving an example report related to the lesion label is: using the fine-tuned large language model to search the imaging database based on the lesion label to obtain an example report related to the lesion label.

[0021] In the aforementioned multi-agent-based medical imaging report generation method, in step four, in step four: the process of the report generation agent outputting the preliminary imaging report is: based on the fine-tuned large language model, using the example report as the prompt word and the lesion label as the input, screening out the preliminary imaging report that best matches the lesion label.

[0022] In the aforementioned multi-agent-based medical imaging report generation method, the process of screening out the preliminary imaging report is specifically: converting the lesion label and the sample report into a vector representation:

[0023] Let the vocabulary be V, and map the words in the lesion label or example report to a d-dimensional vector space. The word vector of word w is:

[0024] v w = {v w1 ,v w1 ,…,v wd};

[0025] For lesion label text T 病灶 Contains n words, and its vector representation is:

[0026]

[0027] Sample Report Text T 示例 Similarly, we can get:

[0028]

[0029] Then the similarity between the lesion label and the example report is obtained by correlation calculation:

[0030]

[0031]

[0032] Where m is the vector dimension;

[0033] Based on the calculated similarity, the most relevant imaging reports are retrieved and extracted from the imaging database.

[0034] In the aforementioned multi-agent-based medical imaging report generation method, the mechanism of conditional constraints and reinforcement of professional terms includes constructing and integrating professional terminology dictionaries, setting terminology constraint rules and terminology reinforcement learning strategies; the process of constructing and integrating professional terminology dictionaries is to collect multi-source professional medical terminology data, build a comprehensive and authoritative medical terminology dictionary, and integrate this medical terminology dictionary into the large language model; the process of setting terminology constraint rules is to set mandatory usage requirements for professional terms with key diagnostic significance or lesion feature descriptions; the process of the terminology reinforcement learning strategy is that during the training of the large language model, if the large language model accurately and completely uses professional terms in accordance with medical standards to generate report fragments, positive rewards are given, and if the large language model has errors, omissions or non-standard conditions in the use of professional terms, reverse penalties are given.

[0035] Compared with the prior art, the present invention has the following beneficial effects: the present invention combines the constructed image segmentation agent, prompt word retrieval agent, report generation agent and image database, from the image segmentation agent to process the medical image to generate the lesion label, to the prompt word retrieval agent to retrieve the sample report based on the lesion label, and then to the report generation agent to generate the preliminary report and revise it into the final report, which can integrate the advantages of different functional modules, thereby improving the accuracy of the final image report and reducing the workload of doctors. The present invention amends the preliminary image report by introducing the mechanism of professional terminology condition constraint and reinforcement, which can ensure that the final image report meets the specifications and requirements of the medical profession, can improve the professionalism and credibility of the report in the medical field, and facilitate medical personnel to make accurate diagnosis and treatment decisions. In addition, in the process of generating lesion labels by the image segmentation agent, the present invention extracts the original image metadata and the segmented image metadata, and uses a large language model for processing. This multi-level utilization of data helps to more accurately identify the characteristics of the lesion, so as to accurately describe it in the image report and improve the accuracy of the image report. The prompt word retrieval agent of the present invention retrieves the image database based on the fine-tuned large language model to obtain example reports, so that the large language model can better understand the semantic information of the lesion label, thereby obtaining more relevant example reports and providing more reference basis for the accuracy of the final report. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 is a flow chart of the method of the present invention;

[0037] Figure 2 This is a flow chart of human-machine collaborative optimization of the present invention;

[0038] Figure 3 This is the image segmentation flow chart of the present invention;

[0039] Figure 4 Searching for flow charts for the present invention;

[0040] Figure 5 Evaluation feedback optimization flow chart for the present invention.

[0041] Figure 6 The X-axis segmentation image of the abdominal multiple organs of the present invention;

[0042] Figure 7 The Y-axis segmentation image of the abdominal multiple organs of the present invention;

[0043] Figure 8 The Z-axis segmentation image of multiple abdominal organs of the present invention;

[0044] Fig. 9 The 3D segmentation image of multiple abdominal organs of the present invention;

[0045] Fig.10 Image reports output by the reporting agent are generated for the present invention. DETAILED DESCRIPTION

[0046] The present invention is further described below in conjunction with the accompanying drawings and embodiments, but they are not intended to limit the present invention.

[0047] Embodiment: A method for generating medical imaging reports based on multi-agents, the main process is as follows Figure 1 As shown, the following steps are included:

[0048] Step 1: Construct image segmentation agent, prompt word retrieval agent, report generation agent and image database;

[0049] Step 2: Input the medical image into the image segmentation agent, and generate the lesion label through the image segmentation agent;

[0050] In this embodiment, the DICOM image parsing tool is used to parse the specific medical image format, which has the functions of extracting the original image metadata and preprocessing the medical image, such as denoising, contrast enhancement, etc. Metadata refers to the attributes of the medical image file and related additional information. The original image metadata and the segmented image metadata contain information such as lesion size, lesion shape, lesion location and lesion boundary.

[0051] In this embodiment, the image segmentation model used is based on an algorithm for automatically segmenting different regions or objects (such as organs, lesions) in medical images. It identifies and marks the target area by analyzing each pixel in the medical image.

[0052] The image segmentation model uses a projection layer to extract layer-by-layer features as the input of the encoder, providing a suitable basic data format for subsequent feature extraction and processing, and then uses a deep convolution of size L*K (for example, starting from 7×7×7) to compress the window shift feature so that each kernel channel is convolved with the corresponding input channel, ensuring that the output feature has the same channel dimension as the input, which helps to capture the global feature information in medical images.

[0053] The image segmentation model uses a similar design to the ResNet block to calculate features by moving the deep convolution upwards, introducing a deep convolution scaling (DCS) with a 1×1×1 kernel, which can linearly scale each channel feature independently. In this way, the redundancy of cross-channel context is minimized while enriching the feature representation, enhancing the cross-channel feature correspondence with the downsampling block in each stage, and further reducing the complexity of the image segmentation model by 5%, and achieving results comparable to those of the block architecture using MLP.

[0054] A residual connection is applied between the input and the last scaling layer, which is different from the standard 3D U-Net block (using small convolution kernels to extract local features) and the SegResNet block (applying residuals similar to transformer blocks). This residual connection method helps to better transfer information during model training and promotes the effective propagation of gradients, allowing the image segmentation model to learn more complex feature representations.

[0055] In convolutional networks, batch normalization is common, but it may affect the generalizability of image segmentation models. Therefore, the image segmentation model directly uses layer normalization instead of batch normalization in the encoder block. Layer normalization normalizes the convolutional representation, which helps to enhance the convergence of the image segmentation model and reduce overfitting, so that the image segmentation model can maintain more stable performance when processing different samples.

[0056] The image segmentation model includes multiple downsampling blocks in the architecture design, which perform downsampling operations at different stages (such as Stage 1, Stage 2, Stage 3, Stage 4, etc.), gradually reduce the image resolution of medical images, and increase the number of feature channels to extract higher-level semantic features. In this process, through the collaborative work of components such as 3DUX-Net Block, as well as related convolution operations (such as 3D Conv, etc.), normalization layers (such as Layer Norm), activation layers (such as GELU) and residual connections, the features at different levels are effectively transmitted and fused, so that the image segmentation model can learn the feature information of medical images at different scales, so as to more accurately segment different tissues and organs in medical images.

[0057] After the image segmentation model has undergone feature extraction and downsampling in the encoder stage, it is upsampled in the decoder part through a series of operations (such as transpose convolution, etc.) to gradually restore the resolution of the medical image. In this process, the features at different levels in the encoder stage are spliced ​​with the features at the corresponding levels of the decoder, and multi-scale feature information is integrated to improve the accuracy of segmentation. For example, at a certain stage, the features of 16×16×16×384 dimensions are spliced ​​with other related features, and then processed by subsequent 3D UX-Net Block, Residual Block, etc., to finally generate the segmentation results of the medical image, accurately segmenting the different tissues and organs in the medical image. In this embodiment, multiple organs in the abdomen are segmented, and the segmentation results are as follows: Figure 6-9 As shown, it is convenient for subsequent medical applications such as diagnosis and treatment.

[0058] like Figure 3 As shown in the figure, the medical image agent process starts with the input of medical images, uses image parsing tools to extract the original image metadata, and performs necessary preprocessing on the image. The preprocessed image is input into the image segmentation model to perform segmentation operations on the lesion area to generate a segmented image. Then, the segmented image is used to extract the segmented image metadata through the parsing tool. Finally, the original image metadata and the segmented image metadata are input into the fine-tuned large language model to generate the lesion label.

[0059] In this embodiment, the large language model is trained using high-quality imaging reports reviewed by medical experts to ensure that the model follows medical standards and professional terminology when generating reports.

[0060] Specifically, the large language model training process includes the following steps:

[0061] Step 1: Medical imaging-related text data are widely collected from professional medical imaging databases, archives of imaging departments of well-known medical institutions, and public medical research datasets to form a dataset. These data cover a wealth of image descriptions, diagnostic reports, and lesion feature annotation information, which constitute the basic material for model learning. At the same time, multimodal medical data is innovatively integrated to assist in text data generation. Specifically, key feature data such as quantitative imaging indicators of lesions in medical images are accurately converted into text description information and integrated into the training data, thereby significantly enriching the dimension and information content of the data and providing a more comprehensive learning basis for the model. The collected text data is deeply cleaned to effectively remove text noise, including interference factors such as redundant punctuation, non-medical terms, and repeated sentences. At the same time, spelling, grammar, and medical terminology errors are carefully corrected to ensure the accuracy and standardization of the data. Subsequently, the data is uniformly converted into a specific text encoding format (UTF-8), and the paragraph structure and sentence length are standardized so that the data format meets the requirements of large language model training and facilitates efficient processing of large language models. Professional medical personnel annotate the lesion information in the text data in detail based on strict standards and rich clinical experience. The annotation content covers key aspects such as the location, size, morphology, density or signal characteristics of the lesion and the preliminary diagnosis conclusion. On this basis, the lesion evolution trajectory annotation is innovatively introduced, that is, the changes in the lesions corresponding to the images of the same patient at different times are systematically recorded, so that the large language model can deeply learn the dynamic development characteristics of the lesions, thereby greatly improving the accuracy and completeness of the description of the disease process, and laying a solid foundation for generating high-quality medical imaging reports.

[0062] Step 2: After comprehensive evaluation and screening, we selected the open source large model Qwen-72B, which is suitable for natural language processing and has excellent generalization capabilities. This model is built on an advanced neural network architecture and has a large-scale pre-trained parameter system. It can efficiently and accurately capture and represent the semantic and grammatical information in general text data, providing a powerful basic framework for subsequent fine-tuning of medical imaging report generation tasks.

[0063] On the basis of making full use of the selected open source big model, closely combined with the unique characteristics of the medical imaging report generation task, some key parameters are initialized and fine-tuned in a targeted manner. For example, for the layer structure parameters that are closely related to semantic understanding and text generation, the initial values ​​are carefully re-extracted from the normal distribution (mean 0, standard deviation 0.005) for assignment based on the professionalism and complexity of the vocabulary distribution in the medical field and the unique semantic logic complexity. In this way, the initial learning direction of the model in medical text processing is effectively guided, so that it can quickly adapt to the special language environment in the medical field, while retaining the inherent advantages of the open source big model in general language understanding to the greatest extent, and realizing the organic integration of general language capabilities and medical professional language processing capabilities.

[0064] Step 3: Use a multi-label classification loss function, such as Binary Cross-Entropy Loss. For samples with n class labels, the loss function calculation formula is:

[0065]

[0066] Among them, y i is the true value of the i-th category label (0 or 1), p i Predict the probability of the i-th category label for the large language model.

[0067] On this basis, a loss correction factor based on the medical knowledge graph is innovatively introduced. As a highly structured medical knowledge database, the medical knowledge graph comprehensively covers various medical entities (such as diseases, symptoms, examination methods, body parts, etc.) and the intricate relationships between them (such as causal relationships, correlation relationships, etc.). When calculating the loss, if the relationship between the lesion labels generated by the image segmentation agent does not match the professional knowledge in the medical knowledge graph, the loss correction factor will come into play and increase the loss value. For example, for a lung imaging report training sample, if the large language model predicts that the probability of the presence of lung nodules is p1 and the probability of hemoptysis is p2, and in the medical knowledge graph, the prior probability of hemoptysis when lung nodules are present is P, which is determined based on a large amount of clinical data and medical research. The loss correction factor C = |p2-P|, then the final loss function value is adjusted to:

[0068] L' = L × (1 + C);

[0069] Through this mechanism, the large language model is effectively urged to adjust parameters in time according to medical professional cognition in subsequent training, so that the learning process of the large language model is closely aligned with the logic of medical profession, ensuring that the reports generated by the large language model are highly professional and accurate. This loss correction factor can be widely used in the deep mining and learning of complex relationships between multiple lesion label categories, effectively improving the understanding and application ability of the large language model to the medical knowledge system. The Adam optimization algorithm is adopted, which cleverly combines the momentum method with the adaptive learning rate mechanism. During the training process, the learning rate is dynamically adjusted according to the real-time changes of the parameter gradient of the large language model to achieve efficient parameter optimization. In the early stage of training, a smaller learning rate is used. This strategy can effectively avoid the damage to the original knowledge structure of the open source large language model due to excessive learning rate, ensure that the large language model smoothly transitions during the fine-tuning process, and gradually adapts to the task of generating medical imaging reports.

[0070] As the fine-tuning training progresses steadily, when the convergence trend of the large language model tends to be stable, the learning rate is appropriately increased in a timely manner, thereby effectively promoting the large language model to deeply learn and master the specific knowledge of the medical imaging report task, accelerate the convergence speed of the large language model, and improve the training efficiency. Innovatively integrate the idea of ​​simulated annealing algorithm into the optimization process. The simulated annealing algorithm is inspired by the principle of physical annealing. In the later stage of training, when the large language model falls into the dilemma of local optimal solution, it accepts the inferior solution with a certain probability (the probability is based on the Metropolis criterion). Accurately calculate, where ΔE is the difference between the objective function values ​​in the new state and the current state, and T is the current temperature), thereby cleverly jumping out of the local optimal trap and continuing to explore better solutions in a wider parameter space, providing a strong guarantee for large language models to break through performance bottlenecks.

[0071] The data set is precisely divided into training set, validation set and test set in a ratio of 70%:15%:15%, and the training set data is sequentially input into the large language model in the form of small batches of data with a batch size of 16. After receiving the input data, the large language model performs forward propagation to calculate the prediction probability, and strictly calculates the loss value according to the pre-defined loss function. Then, the Adam optimization algorithm is used to accurately update the parameters of the large language model based on the calculated gradient information. After each training epoch is completed, the validation set is used in a timely manner to conduct a comprehensive evaluation of the large language model. The evaluation indicators cover accuracy:

[0072]

[0073] In the formula, TP is the number of true positive examples, TN is the number of true negative examples, FP is the number of false positive examples, and FN is the number of false negative examples), recall rate, F1-score and other important dimensions. Based on the evaluation results, accurately judge whether the large language model is overfitting or underfitting. If the large language model is overfitting, effective measures such as stopping training in advance, increasing the amount of training data, and using regularization technology can be taken to deal with it; if the large language model is underfitting, you can consider adjusting the structural parameters of the large language model in a targeted manner or appropriately extending the training time to optimize the performance of the large language model. An innovative adaptive training round adjustment mechanism is established to dynamically and flexibly adjust the training rounds according to the performance improvement speed of the large language model on the validation set, effectively avoiding invalid training, greatly improving training efficiency, and making the training process more intelligent and efficient.

[0074] Step 4: After the training of the large language model is completed, the test set is used to conduct a comprehensive and in-depth evaluation of the large language model. In addition to conventional evaluation indicators such as accuracy, recall, and F1-score, the semantic similarity evaluation method (using a professional semantic similarity algorithm to evaluate the semantic similarity between the lesion description generated by the large language model and the real report) and key information integrity evaluation (carefully checking whether the report generated by the large language model contains complete key lesion information) are innovatively adopted, and the performance of the large language model on different types of lesions and samples of different difficulty levels is deeply analyzed. The evaluation enhancement mechanism based on medical expert review feedback is innovatively adopted to quantify the professional review opinions of medical experts on the reports generated by the large language model, and accurately feed them back into the evaluation indicator system, so as to more accurately measure the clinical practicality and reliability of the large language model, and ensure that the reports generated by the large language model can meet the strict requirements of actual clinical diagnosis.

[0075] If the evaluation results do not meet the expected standards, more data can be collected for specific lesion types for secondary training, and the structural parameters of the large language model can be further adjusted (such as fine-tuning the number of neural network layers or nodes) and re-training and evaluation. Alternatively, hyperparameter tuning techniques can be used to fine-tune hyperparameters such as learning rate, batch size, number of training epochs, and regularization coefficient to comprehensively improve the performance of the large language model. An innovative large language model evolution system is constructed. Based on the idea of ​​genetic algorithms, cross-combination and mutation operations are performed on the excellent genes of different versions of large language models (such as excellent parameter combinations and network structure characteristics) to generate new large language model candidates, and then these candidate large language models are retrained and evaluated. This iterative evolution method accelerates the optimization process of the large language model and continuously improves the accuracy and professionalism of the large language model in the task of generating medical imaging reports.

[0076] After training, the large language model is adapted to specific scenarios or tasks. The scenario here is to generate lesion labels by comparing the segmented image metadata with the original image metadata.

[0077] Lesion labels are annotation information for lesion areas (such as tumors, inflammation, hemorrhage, etc.) found in medical images, and contain key features of the lesions, such as lesion size, lesion shape, lesion location, and lesion boundaries.

[0078] Step 3: Input the lesion label into the prompt word retrieval agent, and the prompt word retrieval agent retrieves sample reports related to the lesion label;

[0079] In this embodiment, starting from the lesion label, firstly, a report related to the label is searched in the image report library and the image database by using a data retrieval tool, and image records containing or describing the lesion label are screened out, such as Figure 4 This preliminary retrieval process obtains a batch of imaging reports related to the label, laying the foundation for subsequent semantic processing. In this way, the system can quickly collect imaging information associated with lesion labels and form a preliminary candidate report set.

[0080] In this embodiment, the process of screening out the preliminary imaging report is specifically: converting the lesion label and the example report into a vector representation.

[0081] Specifically, we use a pre-trained word embedding model (such as Word2Vec) or a word vector model self-trained on medical text. Let the vocabulary be V, and map the words in the lesion label or example report to a d-dimensional vector space. For example, the word vector of word w is v w = {v w1 ,v w1 ,…,v wd}.

[0082] For lesion label text T 病灶 Contains n words, and its vector representation is:

[0083]

[0084] Sample Report Text T 示例 Similarly, we can get:

[0085]

[0086] The similarity between the lesion label and the example report is calculated by algorithms such as cosine similarity. The formula is as follows:

[0087]

[0088] Where m is the vector dimension. The similarity is calculated to screen the preliminary imaging reports. The higher the similarity, the more the example report matches the lesion label.

[0089] Based on the calculated similarity, the most relevant imaging report is retrieved and extracted from the imaging database through a data retrieval tool, and the lesion label containing information such as the lesion location and size is input to generate a preliminary imaging report.

[0090] Step 4: The lesion label and sample report are used as input to the report generation agent, which outputs a preliminary imaging report. The mechanism of professional terminology condition constraints and reinforcement is then introduced to correct the preliminary imaging report and generate the final imaging report.

[0091] In this embodiment, after generating a preliminary report, the intelligent agent introduces a mechanism for conditional constraints and reinforcement of professional terminology, which includes constructing and integrating a professional terminology dictionary, setting terminology constraint rules, and a terminology reinforcement learning strategy.

[0092] The process of constructing and integrating the professional terminology dictionary is to collect multi-source professional medical terminology data, such as authoritative medical textbooks, clinical diagnosis and treatment guidelines, and terminology information in professional medical databases, to construct a comprehensive and authoritative medical professional terminology dictionary D. Each term t in the dictionary has precise definition, part-of-speech tagging, scope of application and other information. This medical professional terminology dictionary is integrated into the large language model, so that the large language model can directly call the dictionary information during training and report generation, ensuring the accuracy and standardization of terminology application.

[0093] The process of setting the term constraint rules is to set mandatory usage requirements for professional terms with key diagnostic significance or lesion feature descriptions. For example, if the image clearly shows "cavitary lesions" in the lungs, the large language model must use this term and describe its location (such as the specific location of the lung lobe), size (length, width, height), and wall characteristics (thickness, smoothness, etc.) in accordance with medical standards. The formula is expressed as: If P(t) (t is a specific professional term and meets the image feature conditions), then R(t) (R means the use of the term and related standard descriptions in the report), and then set up prohibition rules for ambiguous or non-medical professional standard expressions. For example, it is prohibited to use everyday colloquial vocabulary to describe medical imaging results. If Q(t') (t' is a non-professional or ambiguous term), then S(t') (S means that the term is not used in the report).

[0094] The process of the terminology reinforcement learning strategy is that during the training of the large language model, if the large language model accurately and completely generates report fragments using professional terms in accordance with medical standards, a positive reward is given. For example, if the large language model describes "breast lump" as "a lump was found in the upper outer quadrant of the breast, about x×y×z centimeters in size, hard in texture, unclear in boundaries, and poor in mobility", which meets the professional terminology usage and description standards, the large language model is given a reward value r+ The reward value can be determined based on multiple factors such as the accuracy and completeness of term usage and the degree of image feature matching. The formula is as follows:

[0095] r + =α×A+β×B+γ×C;

[0096] In the formula, A is the term accuracy score, B is the completeness score, C is the image matching score, and α, β, and γ are the corresponding weight coefficients), which encourages the large language model to use professional terms correctly; if the large language model uses professional terms incorrectly, omits, or is not standardized, a reverse penalty will be given. For example, if the large language model mistakenly writes "fracture" as "bone fracture", a penalty value r will be given according to the severity of the error. - The size of the penalty value is associated with the degree of impact of the error on the accuracy of the report, and the large language model backtracks and analyzes the error after being punished, and adjusts the parameters to reduce the probability of making the same mistake again.

[0097] like Figure 2 As shown in the figure, the logic and structure of the report are optimized by using more professional medical terms and ensuring that the content of the imaging report covers key information about the lesion (such as size, location, and recommendations). After completing these enhancement steps, the enhanced imaging report is stored in the imaging database and self-reviewed to identify and correct potential deficiencies to improve the accuracy and consistency of the output.

[0098] like Figure 5 As shown in the figure, if the report meets the medical standards after evaluation, the final result will be returned; if it fails the evaluation, it will enter the iterative optimization stage. In this stage, multiple rounds of refinement and correction are carried out based on feedback to ensure that the generated report is both professional and accurate. Ultimately, through this series of self-feedback and optimization, a complete and standard medical imaging report can be generated to help doctors make accurate diagnosis and treatment decisions. Medical imaging reports are as follows: Fig.10 shown.

[0099] The present invention combines the constructed image segmentation agent, prompt word retrieval agent, report generation agent and image database, from the image segmentation agent to process the medical image to generate the lesion label, to the prompt word retrieval agent to retrieve the sample report based on the lesion label, and then to the report generation agent to generate the preliminary report and revise it to the final report, which can integrate the advantages of different functional modules, thereby improving the accuracy of the final image report and reducing the workload of doctors. The present invention amends the preliminary image report by introducing the mechanism of professional terminology condition constraint and reinforcement, which can ensure that the final image report meets the specifications and requirements of the medical profession, can improve the professionalism and credibility of the report in the medical field, and facilitate medical personnel to make accurate diagnosis and treatment decisions. In addition, in the process of generating lesion labels by the image segmentation agent, the present invention extracts the original image metadata and the image metadata after segmentation, and uses a large language model for processing. This multi-level utilization of data helps to more accurately identify the lesion features, thereby accurately describing them in the image report and improving the accuracy of the image report. The prompt word retrieval agent of the present invention retrieves the image database based on the fine-tuned large language model to obtain example reports, so that the large language model can better understand the semantic information of the lesion label, thereby obtaining more relevant example reports and providing more reference basis for the accuracy of the final report.

[0100] In summary, the present invention can generate accurate image analysis reports that meet medical standards while reducing the workload of doctors.

Claims

1. A multi-agent-based medical imaging report generation method, characterized in that: The following steps are involved: Step 1: Construct image segmentation agent, prompt word retrieval agent, report generation agent and image database; Step 2: Input the medical image into the image segmentation agent, and generate the lesion label through the image segmentation agent; Step 3: Input the lesion label into the prompt word retrieval agent, and the prompt word retrieval agent retrieves sample reports related to the lesion label; Step 4: The lesion label and sample report are used as input to the report generation agent, which outputs a preliminary imaging report. The mechanism of professional terminology condition constraints and reinforcement is then introduced to correct the preliminary imaging report and generate the final imaging report.

2. The multi-agent-based medical imaging report generation method according to claim 1, characterized in that: In step 2, the process of the image segmentation agent generating lesion labels is as follows: extracting original image metadata from the input medical image and preprocessing the medical image; constructing an image segmentation model, segmenting the preprocessed medical image, and extracting segmented image metadata from the segmented medical image; constructing and training a large language model to obtain a fine-tuned large language model, inputting the original image metadata and the segmented image metadata into the fine-tuned large language model to generate lesion labels.

3. The multi-agent-based medical imaging report generation method according to claim 2, characterized in that: The specific process of segmentation by the image segmentation model is to use the projection layer to extract layer-by-layer features as the input of the encoder, so as to provide a suitable basic data format for subsequent feature extraction and processing; use deep convolution to perform convolution, move the deep convolution upward to calculate the features, introduce deep convolution scaling, and reduce the redundancy of cross-channel context; apply residual connection between the input and the last scaling layer, so that the image segmentation model can learn more complex feature representation; use layer normalization in the encoder block to normalize the convolution representation, enhance the convergence of the image segmentation model and reduce overfitting, perform downsampling operations at different stages, gradually reduce the image resolution of the medical image, and increase the number of feature channels at the same time to extract higher-level semantic features; perform upsampling in the decoder part, gradually restore the resolution of the medical image, splice the features of different levels in the encoder stage with the features of the corresponding levels of the decoder, fuse multi-scale feature information, and improve the accuracy of segmentation; finally generate the segmentation result of the medical image, and accurately segment different tissues and organs in the medical image.

4. The multi-agent-based medical imaging report generation method according to claim 2, characterized in that: The training process of the large language model is specifically to collect medical imaging related text data extensively and form a data set; select an open source large model, construct a loss function to dynamically adjust the learning rate in real time to achieve efficient parameter optimization; construct the loss function as follows: Among them, y i is the true value of the i-th category label, 0 or 1; p i is the predicted probability of the large language model for the i-th category label, and n is the number of category label samples; Then the loss correction factor is introduced, the formula is as follows: C=|p2-P| Where p1 is the probability of the presence of lung nodules predicted by the large language model, and p2 is the probability of the presence of hemoptysis; Get the final loss function value: L'=L×(1+C).

5. The multi-agent-based medical imaging report generation method according to claim 2, characterized in that: In step three, the process of the prompt word retrieval agent retrieving example reports related to lesion labels is: using the fine-tuned large language model to search the image database based on the lesion labels to obtain example reports related to the lesion labels.

6. The multi-agent-based medical imaging report generation method according to claim 5, characterized in that: In step 4: the process of the report generation agent outputting the preliminary imaging report is: based on the fine-tuned large language model, using the example report as the prompt word and the lesion label as the input, the preliminary imaging report that best matches the lesion label is screened out.

7. The multi-agent-based medical imaging report generation method according to claim 6, characterized in that: The process of screening out the preliminary imaging reports is specifically as follows: converting the lesion labels and sample reports into vector representations: Let the vocabulary be V, and map the word w in the lesion label or example report to a d-dimensional vector space. The word vector of word w is: v w ={v w1 ,v w1 ,…,v wd }; For lesion label text T 病灶 Contains n words, and its vector representation is: Sample Report Text T 示例 Similarly, we can get: Then the similarity between the lesion label and the example report is obtained by correlation calculation: Where m is the vector dimension; Based on the calculated similarity, the most relevant imaging reports are retrieved and extracted from the imaging database.

8. The multi-agent-based medical imaging report generation method according to claim 7, characterized in that: The mechanism of conditional constraint and reinforcement of professional terminology includes constructing and integrating professional terminology dictionaries, setting terminology constraint rules and terminology reinforcement learning strategies; the process of constructing and integrating professional terminology dictionaries is to collect multi-source professional medical terminology data, construct a comprehensive and authoritative medical terminology dictionary, and integrate this medical terminology dictionary into the large language model; the process of setting terminology constraint rules is to set mandatory use requirements for professional terms with key diagnostic significance or description of lesion characteristics; the process of the terminology reinforcement learning strategy is that during the training of the large language model, if the large language model accurately and completely uses professional terminology in accordance with medical standards to generate report fragments, positive rewards will be given, and if the large language model has errors, omissions or non-standard conditions in the use of professional terminology, reverse penalties will be given.

Images