A multi-agent based medical image report generation method

Abstract

The application discloses a kind of medical image report generation methods based on multi-agent, comprising the following steps: step one: constructing image segmentation agent, cue retrieval agent, report generation agent and image database;Step two: medical image is input into image segmentation agent, and lesion label is generated by image segmentation agent;Step three: the lesion label is input into cue retrieval agent, and the example report related to lesion label is obtained by cue retrieval agent;Step four: the lesion label and example report are used as the input of report generation agent, and the preliminary image report is output by report generation agent, then the mechanism of professional terminology condition constraint and reinforcement is introduced to modify preliminary image report, and the final image report is generated.The application generates accurate and medical standard image analysis report, while the work burden of doctor can be reduced.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of medical imaging technology, and specifically relates to a medical image report generation method based on multi-agent. BACKGROUND

[0002] With the rapid development of medical imaging technology, the amount of medical image data generated and processed by the medical industry is increasing, including CT, MRI, ultrasound and X-ray, etc. These image data not only have a large volume, but also contain rich clinical information, which is crucial for early diagnosis and treatment decision of diseases. However, the current analysis and report generation of medical images still mainly rely on the artificial judgment and experience of doctors, which not only consumes time and effort, but also is easily affected by subjective factors, resulting in the accuracy and consistency of the analysis results cannot be guaranteed.

[0003] The traditional medical image analysis process usually includes image preprocessing, lesion identification, diagnosis analysis and final report writing, which requires a lot of time and human resources, especially when facing high concurrency of clinical demand, it is often difficult to respond efficiently. In addition, doctors also need to refer to a large number of standard literature and past cases when writing reports, further increasing the workload. SUMMARY

[0004] The purpose of the present application is to provide a medical image report generation method based on multi-agent. The present application can not only generate accurate and medically standard image analysis reports, but also reduce the workload of doctors.

[0005] The technical solution of the present application: a medical image report generation method based on multi-agent, comprising the following steps:

[0006] Step one: constructing an image segmentation agent, a cue retrieval agent, a report generation agent and an image database;

[0007] Step two: inputting the medical image into the image segmentation agent, and generating the lesion label through the image segmentation agent;

[0008] Step three: inputting the lesion label into the cue retrieval agent, and retrieving the example report related to the lesion label by the cue retrieval agent;

[0009] Step four: taking the lesion label and the example report as the input of the report generation agent, outputting the preliminary image report by the report generation agent, and then introducing the mechanism of professional term condition constraint and reinforcement to modify the preliminary image report, to generate the final image report.

[0010] In the aforementioned multi-agent-based medical image report generation method, step two, the process of the image segmentation agent generating lesion labels, is as follows: extracting original image metadata from the input medical image and preprocessing the medical image; constructing an image segmentation model to segment the preprocessed medical image, and extracting segmented image metadata from the segmented medical image; constructing and training a large language model to obtain a fine-tuned large language model, and inputting the original image metadata and segmented image metadata into the fine-tuned large language model to generate lesion labels.

[0011] In the aforementioned multi-agent-based medical image report generation method, the specific process of image segmentation is as follows: A projection layer is used to extract layer-by-layer features as input to the encoder, providing a suitable basic data format for subsequent feature extraction and processing; depthwise convolution is used to perform convolution, and the depthwise convolution is shifted upwards to calculate features, introducing depthwise convolution scaling to reduce redundancy across channel contexts; a residual connection is applied between the input and the last scaling layer, enabling the image segmentation model to learn more complex feature representations; layer normalization is used in the encoder block to normalize the convolutional representation, enhancing the convergence of the image segmentation model and reducing overfitting; downsampling is performed at different stages to gradually reduce the image resolution of the medical image while increasing the number of feature channels to extract higher-level semantic features; upsampling is performed in the decoder to gradually restore the resolution of the medical image; features from different levels in the encoder stage are concatenated with features from corresponding levels in the decoder to fuse multi-scale feature information and improve segmentation accuracy; finally, the segmentation result of the medical image is generated, accurately segmenting different tissues and organs in the medical image.

[0012] In the aforementioned multi-agent-based medical image report generation method, the training process of the large language model specifically involves extensively collecting medical image-related text data and forming a dataset; selecting an open-source large model; constructing a loss function; and dynamically adjusting the learning rate in real time to achieve efficient parameter optimization. The loss function is constructed as follows:

[0013]

[0014] Among them, y i p represents the true value of the i-th category label, either 0 or 1. i Let n be the probability predicted by the large language model for the i-th category label, and n be the number of category label samples.

[0015] Then, a loss correction factor is introduced, as shown in the following formula:

[0016] C = |p² - P|

[0017] In the formula, p1 is the probability of the presence of pulmonary nodules predicted by the large language model, and p2 is the probability of hemoptysis.

[0018] The final loss function value is obtained:

[0019] L' = L × (1 + C).

[0020] In the aforementioned multi-agent-based medical image report generation method, in step three, the process of the prompt word retrieval agent retrieving example reports related to lesion labels is as follows: using a fine-tuned large language model to retrieve image databases based on lesion labels to obtain example reports related to lesion labels.

[0021] In the aforementioned multi-agent-based medical image report generation method, in step four, the process of the report generation agent outputting a preliminary image report is as follows: based on the fine-tuned large language model, using example reports as prompt words and lesion labels as input, the preliminary image report that best matches the lesion labels is selected.

[0022] In the aforementioned multi-agent-based medical image report generation method, the process of selecting preliminary image reports specifically involves converting lesion labels and example reports into vector representations:

[0023] Let the vocabulary be V. Map the words in the lesion label or example report to a d-dimensional vector space. The word vector of word w is:

[0024] v w ={v w1 ,v w1 ,…,v wd};

[0025] For the lesion label text T 病灶 A vector representation of n words:

[0026]

[0027] Example report text T 示例 Similarly, we can conclude that:

[0028]

[0029] Then, the similarity between the lesion label and the example report is calculated using relevance analysis:

[0030]

[0031]

[0032] In the formula, m is the vector dimension;

[0033] Based on the calculated similarity, the image reports with the highest relevance are retrieved from the image database.

[0034] In the aforementioned multi-agent-based medical image report generation method, the mechanism for constraining and reinforcing professional terminology includes constructing and integrating a professional terminology dictionary, setting terminology constraint rules, and a terminology reinforcement learning strategy. The process of constructing and integrating the professional terminology dictionary involves collecting multi-source professional medical terminology data, constructing a comprehensive and authoritative medical terminology dictionary, and integrating this medical terminology dictionary into a large language model. The process of setting terminology constraint rules involves setting mandatory usage requirements for professional terms with key diagnostic significance or lesion feature descriptions. The process of the terminology reinforcement learning strategy involves giving positive rewards if the large language model accurately and completely uses professional terminology to generate report fragments in accordance with medical standards during the training of the large language model, and giving negative penalties if the large language model uses professional terminology incorrectly, omitting, or in non-standard situations.

[0035] Compared with existing technologies, this invention has the following beneficial effects: By combining an image segmentation intelligent agent, a prompt word retrieval intelligent agent, a report generation intelligent agent, and an image database, this invention integrates the advantages of different functional modules. From the image segmentation intelligent agent processing medical images to generate lesion labels, to the prompt word retrieval intelligent agent retrieving example reports based on lesion labels, and finally to the report generation intelligent agent generating a preliminary report and revising it into a final report, this invention improves the accuracy of the final image report and reduces the workload of doctors. Furthermore, by introducing a mechanism of professional terminology constraints and reinforcement to revise the preliminary image report, this invention ensures that the final image report meets medical professional standards and requirements, improving the professionalism and credibility of the report in the medical field and facilitating accurate diagnostic and treatment decisions by medical personnel. In addition, during the process of generating lesion labels by the image segmentation intelligent agent, this invention extracts original image metadata and segmented image metadata and processes them using a large language model. This multi-layered utilization of data helps to more accurately identify lesion features, thereby accurately describing them in the image report and improving the accuracy of the image report. The prompt word retrieval agent of this invention retrieves example reports from an image database based on a finely tuned large language model. This enables the large language model to better understand the semantic information of lesion labels, thereby obtaining more relevant example reports and providing more reference for the accuracy of the final report. Attached Figure Description

[0036] Figure 1 This is a flowchart of the method of the present invention;

[0037] Figure 2 This is a flowchart of the human-machine collaboration optimization process of the present invention;

[0038] Figure 3 This is a flowchart of the image segmentation process of the present invention;

[0039] Figure 4 This is a flowchart of the retrieval process of the present invention;

[0040] Figure 5 This is a flowchart illustrating the evaluation, feedback, and optimization process of this invention.

[0041] Figure 6 This is an X-axis segmentation image of multiple organs in the abdomen according to the present invention;

[0042] Figure 7 This is a Y-axis segmentation image of multiple abdominal organs according to the present invention;

[0043] Figure 8 This is a Z-axis segmentation image of multiple organs in the abdomen according to the present invention;

[0044] Figure 9 This is a 3D segmentation image of multiple organs in the abdomen according to the present invention;

[0045] Figure 10 This invention generates an image report output by the intelligent agent. Detailed Implementation

[0046] The present invention will be further described below with reference to the accompanying drawings and embodiments, but this should not be construed as limiting the present invention.

[0047] Example: A method for generating medical image reports based on multi-agent systems, the main process of which is as follows: Figure 1 As shown, it includes the following steps:

[0048] Step 1: Construct an image segmentation agent, a prompt word retrieval agent, a report generation agent, and an image database;

[0049] Step 2: Input the medical images into the image segmentation AI, and generate lesion labels through the image segmentation AI;

[0050] In this embodiment, the DICOM image parsing tool is used to parse a specific medical image format. It has functions such as extracting original image metadata and preprocessing medical images, including noise reduction and contrast enhancement. Metadata refers to the attributes and related additional information of a medical image file. The original image metadata and the segmented image metadata contain information such as lesion size, lesion shape, lesion location, and lesion boundaries.

[0051] In this embodiment, the image segmentation model used is based on an algorithm for automatically segmenting different regions or objects (such as organs or lesions) in medical images. It identifies and marks the target region by analyzing each pixel in the medical image.

[0052] The image segmentation model uses a projection layer to extract layer-by-layer features as input to the encoder, providing a suitable basic data format for subsequent feature extraction and processing. Then, a depthwise convolution of size L*K (e.g., starting from 7×7×7) is used to compress the window shift characteristics so that each kernel channel is convolved with the corresponding input channel, ensuring that the output features have the same channel dimension as the input, which helps to capture global feature information in medical images.

[0053] The image segmentation model utilizes a similar design to ResNet blocks, computing features by shifting depthwise convolutions upwards and introducing depthwise convolutional scaling (DCS) with 1×1×1 kernels, which can independently and linearly scale the features of each channel. In this way, it enriches feature representations while minimizing redundancy across channel contexts, enhances the cross-channel feature correspondence with the downsampled block in each stage, and further reduces the complexity of the image segmentation model by 5%, achieving results comparable to block architectures using MLPs.

[0054] Applying residual connections between the input and the last scaling layer differs from standard 3D U-Net blocks (which use small convolutional kernels to extract local features) and SegResNet blocks (which apply residuals similar to transformer blocks). This residual connection approach helps to better transfer information during model training, promotes the effective propagation of gradients, and enables the image segmentation model to learn more complex feature representations.

[0055] While batch normalization is common in convolutional networks, it may affect the generalizability of image segmentation models. Therefore, image segmentation models directly use layer normalization instead of batch normalization in the encoder block. Layer normalization normalizes the convolutional representation, which helps to enhance the convergence of the image segmentation model and reduce overfitting, enabling the image segmentation model to maintain more stable performance when processing different samples.

[0056] The image segmentation model incorporates multiple downsampling blocks in its architecture. Downsampling operations are performed at different stages (e.g., Stage 1, Stage 2, Stage 3, Stage 4) to progressively reduce the image resolution of medical images while increasing the number of feature channels to extract higher-level semantic features. During this process, through the collaborative work of structures such as the 3DUX-Net Block, and related convolutional operations (e.g., 3D Conv), normalization layers (e.g., Layer Norm), activation layers (e.g., GELU), and residual connections, features at different levels are effectively transferred and fused. This allows the image segmentation model to learn medical image feature information at different scales, thereby more accurately segmenting different tissues and organs in medical images.

[0057] After feature extraction and downsampling in the encoder stage, the image segmentation model performs upsampling in the decoder stage through a series of operations (such as transpose convolution) to gradually restore the resolution of the medical image. During this process, features from different levels in the encoder stage are concatenated with corresponding features in the decoder, fusing multi-scale feature information to improve segmentation accuracy. For example, at a certain stage, 16×16×16×384-dimensional features are concatenated with other relevant features, and then processed by subsequent 3D UX-Net Blocks, Residual Blocks, etc., to finally generate the segmentation result of the medical image, accurately segmenting different tissues and organs in the medical image. In this embodiment, multiple organs in the abdomen are segmented, and the segmentation result is as follows: Figures 6-9 As shown, this is to facilitate subsequent medical applications such as diagnosis and treatment.

[0058] like Figure 3 As shown, the workflow of this medical imaging intelligent agent begins with the input medical image. It uses an image parsing tool to extract the original image metadata and performs necessary preprocessing. The preprocessed image is then input into an image segmentation model to segment lesion regions, generating segmented images. Next, the segmented images are parsed to extract metadata. Finally, the original image metadata and the segmented image metadata are input into a finely tuned large language model to generate lesion labels.

[0059] In this embodiment, the large language model is trained using high-quality image reports reviewed by medical experts to ensure that the model follows medical standards and professional terminology when generating reports.

[0060] Specifically, the large language model training process includes the following steps:

[0061] Step 1: A comprehensive dataset was created by extensively collecting medical image-related text data from professional medical image databases, archives of radiology departments in renowned medical institutions, and publicly available medical research datasets. This dataset encompasses rich image descriptions, diagnostic reports, and lesion feature annotations, forming the foundational material for model learning. Simultaneously, multimodal medical data was innovatively integrated to assist in text data generation. Specifically, key feature data from medical images, such as quantitative imaging indicators of lesions, were accurately converted into textual descriptions and incorporated into the training data, significantly enriching the data's dimensions and information content, providing the model with a more comprehensive learning basis. The collected text data underwent deep cleaning to effectively remove textual noise, including redundant punctuation, non-medical terminology, and repetitive sentences. Furthermore, spelling, grammatical, and medical terminology errors were carefully corrected to ensure data accuracy and standardization. Subsequently, the data was uniformly converted to a specific text encoding format (UTF-8), and paragraph structure and sentence length were standardized to ensure the data format met the requirements of large language model training, facilitating efficient processing by the large language model. Professional medical personnel meticulously annotate lesion information in text data based on rigorous standards and extensive clinical experience. The annotations cover key aspects such as lesion location, size, morphology, density, signal characteristics, and preliminary diagnostic conclusions. Building upon this foundation, an innovative approach is introduced to annotate lesion evolution trajectories, systematically recording changes in lesions at different times for the same patient. This allows the large language model to deeply learn the dynamic development characteristics of lesions, significantly improving the accuracy and completeness of disease progression descriptions and laying a solid foundation for generating high-quality medical imaging reports.

[0062] Step 2: After comprehensive evaluation and screening, the open-source large model Qwen-72B, which is suitable for natural language processing and has excellent generalization ability, was selected. This model is built on an advanced neural network architecture and has a large-scale pre-trained parameter system. It can efficiently and accurately capture and represent semantic and syntactic information in general text data, providing a powerful basic framework for subsequent fine-tuning for medical image report generation tasks.

[0063] By fully utilizing the selected open-source large model and closely considering the unique characteristics of the medical image report generation task, targeted initialization fine-tuning was performed on some key parameters. For example, for the layer structure parameters closely related to semantic understanding and text generation, initial values ​​were carefully re-sampled from a normal distribution (mean 0, standard deviation 0.005) based on the professional and complex characteristics of the medical vocabulary distribution and its unique semantic logic complexity. In this way, the initial learning direction of the model in medical text processing is effectively guided, enabling it to quickly adapt to the special language environment of the medical field, while preserving the inherent advantages of the open-source large model in general language understanding to the greatest extent, thus achieving an organic integration of general language capabilities and medical professional language processing capabilities.

[0064] Step 3: Use a multi-label classification loss function, such as Binary Cross-Entropy Loss. For samples with n class labels, the loss function calculation formula is as follows:

[0065]

[0066] Among them, y i p represents the true value (0 or 1) of the i-th category label. i This represents the probability predicted by the large language model for the i-th category label.

[0067] Building upon this foundation, a loss correction factor based on a medical knowledge graph is innovatively introduced. The medical knowledge graph, as a highly structured medical knowledge database, comprehensively covers various medical entities (such as diseases, symptoms, examination methods, body parts, etc.) and the complex relationships between them (such as causal relationships, correlational relationships, etc.). When calculating the loss, if the relationships between lesion labels generated by the image segmentation agent do not match the professional knowledge in the medical knowledge graph, the loss correction factor will come into play, increasing the loss value. For example, for a lung image report training sample, if the large language model predicts the probability of the presence of lung nodules as p1 and the probability of hemoptysis as p2, while in the medical knowledge graph, based on a large amount of clinical data and medical research, the prior probability of hemoptysis occurring when lung nodules are present is P, and the loss correction factor C = |p2 - P|, then the final loss function value is adjusted as follows:

[0068] L' = L × (1 + C);

[0069] This mechanism effectively encourages the large language model to adjust its parameters in a timely manner based on medical professional knowledge during subsequent training. This ensures that the learning process of the large language model closely aligns with medical professional logic, guaranteeing the high professionalism and accuracy of the reports generated by the model. This loss correction factor can be widely applied to the deep mining and learning of complex relationships between multiple lesion label categories, effectively enhancing the large language model's understanding and application of medical knowledge systems. The Adam optimization algorithm is employed, which cleverly combines momentum and adaptive learning rate mechanisms. During training, the learning rate is dynamically adjusted based on the real-time changes in the gradient of the large language model's parameters to achieve efficient parameter optimization. In the initial training phase, a smaller learning rate is used. This strategy effectively avoids damaging the original knowledge structure of the open-source large language model due to an excessively large learning rate, ensuring a smooth transition during fine-tuning and gradually adapting to the medical image report generation task.

[0070] As fine-tuning training progresses steadily, and the convergence trend of the large language model stabilizes, the learning rate is appropriately increased in a timely manner. This effectively promotes the large language model's in-depth learning and mastery of specific knowledge related to the medical image reporting task, accelerating the convergence speed and improving training efficiency. The simulated annealing algorithm, inspired by the principle of physical annealing, is innovatively integrated into the optimization process. In the later stages of training, when the large language model gets stuck in a local optimum, it accepts a suboptimal solution with a certain probability (this probability is based on the Metropolis criterion). The precise calculation (where ΔE is the difference between the objective function value in the new state and the current state, and T is the current temperature) cleverly escapes the local optimum trap and continues to explore better solutions in a broader parameter space, providing a strong guarantee for large language models to break through performance bottlenecks.

[0071] The dataset was precisely divided into training, validation, and test sets in a 70%:15%:15% ratio. The training set data was then sequentially input into the large language model in mini-batch format (batch size 16). Upon receiving the input data, the large language model performed forward propagation to calculate predicted probabilities, rigorously calculated the loss value according to a predefined loss function, and then used the Adam optimization algorithm to precisely update the large language model parameters based on the calculated gradient information. After each training epoch, the large language model was comprehensively evaluated using the validation set. Evaluation metrics included accuracy.

[0072]

[0073] In the formula, TP represents the number of true positives, TN represents the number of true negatives, FP represents the number of false positives, FN represents the number of false negatives, recall rate, F1-score, and other important dimensions. Based on the evaluation results, it accurately determines whether the large language model exhibits overfitting or underfitting. If the large language model is overfitting, effective measures such as stopping training early, increasing the amount of training data, and using regularization techniques can be taken to address this. If the large language model is underfitting, targeted adjustments to the large language model's structural parameters or appropriate extension of training time can be considered to optimize its performance. An innovative adaptive training epoch adjustment mechanism is established, dynamically and flexibly adjusting the training epochs based on the performance improvement rate of the large language model on the validation set, effectively avoiding ineffective training, significantly improving training efficiency, and making the training process more intelligent and efficient.

[0074] Step 4: After the large language model training is completed, a comprehensive and in-depth evaluation of the model is conducted using a test set. In addition to conventional evaluation metrics such as accuracy, recall, and F1-score, an innovative semantic similarity evaluation method is employed (using a professional semantic similarity algorithm to assess the semantic similarity between the lesion descriptions generated by the large language model and the actual reports) and a key information completeness evaluation (carefully checking whether the reports generated by the large language model completely contain key lesion information). Furthermore, the performance of the large language model on samples of different lesion types and difficulty levels is analyzed in depth. An innovative evaluation reinforcement mechanism based on medical expert review feedback is adopted. This quantifies the professional review opinions of medical experts on the reports generated by the large language model and accurately feeds them into the evaluation indicator system, thereby more accurately measuring the clinical applicability and reliability of the large language model and ensuring that the reports generated by the large language model can meet the stringent requirements of actual clinical diagnosis.

[0075] If the evaluation results do not meet the expected standards, more data can be collected for specific lesion types for secondary training. This allows for further adjustments to the structural parameters of the large language model (such as fine-tuning the number of neural network layers or nodes), followed by retraining and evaluation. Alternatively, hyperparameter tuning techniques can be used to finely adjust hyperparameters such as learning rate, batch size, number of training epochs, and regularization coefficients to comprehensively improve the performance of the large language model. An innovative evolutionary system for the large language model is constructed. Based on the genetic algorithm, superior genes (such as excellent parameter combinations and network structure features) from different versions of the large language model are cross-combined and mutated to generate entirely new candidate large language models. These candidate models are then retrained and evaluated. This iterative evolutionary approach accelerates the optimization process of the large language model, continuously improving its accuracy and professionalism in medical image report generation tasks.

[0076] After training, the large language model is adapted to specific scenarios or tasks. In this case, the scenario is to compare the segmented image metadata with the original image metadata to generate lesion labels.

[0077] Lesion labels are annotation information for lesion areas (such as tumors, inflammation, hemorrhage, etc.) found in medical images. They include key features of the lesion, such as lesion size, lesion shape, lesion location, and lesion boundaries.

[0078] Step 3: Input the lesion label into the prompt word retrieval agent, which will then retrieve example reports related to the lesion label.

[0079] In this embodiment, starting with the lesion label, a data retrieval tool is first used to search for reports related to the label in the image report library and image database. Image records containing or describing the lesion label are then selected, such as... Figure 4 As shown, this initial retrieval process yielded a batch of image reports related to the tags, laying the foundation for subsequent semantic processing. In this way, the system can quickly collect image information associated with lesion tags, forming a preliminary candidate report set.

[0080] In this embodiment, the process of selecting preliminary image reports specifically involves converting lesion labels and example reports into vector representations.

[0081] Specifically, this involves using a pre-trained word embedding model (such as Word2Vec) or a word vector model self-trained on medical text. Let the vocabulary be V, and map words in lesion labels or example reports to a d-dimensional vector space. For example, the word vector for word w is v. w ={v w1 ,v w1 ,…,v wd}

[0082] For the lesion label text T 病灶 A vector representation of n words:

[0083]

[0084] Example report text T 示例 Similarly, we can conclude that:

[0085]

[0086] The similarity between the lesion label and the example report is calculated using algorithms such as cosine similarity, as shown in the following formula:

[0087]

[0088] In the formula, m is the vector dimension; the preliminary image reports are filtered by calculating this similarity. The higher the similarity, the better the example report matches the lesion label.

[0089] Based on the calculated similarity, the most relevant image reports are retrieved from the image database using a data retrieval tool. A preliminary image report is generated by inputting lesion labels containing information such as lesion location and size.

[0090] Step 4: Use lesion labels and example reports as input to the report generation agent. The report generation agent outputs a preliminary image report. Then, a mechanism of technical terminology constraints and reinforcement is introduced to revise the preliminary image report and generate the final image report.

[0091] In this embodiment, after generating the preliminary report, the agent introduces a mechanism for constraining and reinforcing technical terms. This mechanism includes constructing and integrating a technical terminology dictionary, setting terminology constraint rules, and implementing a terminology reinforcement learning strategy.

[0092] The process of constructing and integrating a professional terminology dictionary involves collecting multi-source professional medical terminology data, such as authoritative medical textbooks, clinical practice guidelines, and terminology information from professional medical databases, to construct a comprehensive and authoritative medical terminology dictionary D. Each term t in the dictionary has a precise definition, part-of-speech tagging, and scope of application. This medical terminology dictionary is integrated into a large language model, allowing the large language model to directly access the dictionary information during training and report generation, ensuring the accuracy and standardization of terminology usage.

[0093] The process of setting terminology constraint rules involves establishing mandatory usage requirements for professional terms that have key diagnostic significance or describe lesion characteristics. For example, if an image clearly presents a "cavitary lesion" in the lung, the large language model must use this term and describe its location (e.g., the specific location in the lung lobe), size (length, width, and height), and wall characteristics (thickness, smoothness, etc.) according to medical standards. The formula is expressed as: if P(t) (t is a specific professional term that meets the imaging characteristic conditions), then R(t) (R indicates the use of this term and related standard description in the report). Then, prohibition rules are established for expressions that are prone to ambiguity or are not in accordance with medical professional standards. For example, the use of everyday colloquial terms to describe medical imaging results is prohibited. If Q(t') (t' is a non-professional or ambiguous term), then... S(t') (S indicates that the term is not used in the report).

[0094] The terminology reinforcement learning strategy involves providing a positive reward during the training of the large language model if it accurately and completely uses medical terminology to generate report segments according to medical standards. For example, if the large language model describes a "breast lump" as "a lump was found in the upper outer quadrant of the breast, approximately x×y×z cm in size, hard in texture, with indistinct borders and poor mobility," which conforms to the standards for the use and description of medical terminology, the large language model is given a reward value r.+ The reward value can be determined based on multiple factors, including the accuracy and completeness of terminology usage and the degree of image feature matching, as shown in the following formula:

[0095] r + =α×A+β×B+γ×C;

[0096] In the formula, A represents the terminology accuracy score, B represents the completeness score, C represents the image matching score, and α, β, and γ represent the corresponding weighting coefficients. This incentivizes the large language model to correctly use professional terminology. If the large language model uses professional terminology incorrectly, omitting, or in a non-standard manner, a reverse penalty is imposed. For example, if the large language model mistakenly writes "bone fracture" as "bone fissure," a penalty value r is given according to the severity of the error. - The penalty value is related to the degree of impact of the error on the accuracy of the report, and the large language model backtracks and analyzes the error after being penalized, adjusting parameters to reduce the probability of making the same error again.

[0097] like Figure 2 As shown, the logic and structure of the images are optimized by using more specialized medical terminology and ensuring that the content of the imaging report covers key information about the lesions (such as size, location, and recommendations). After these enhancement steps are completed, the enhanced imaging report is stored in the imaging database and self-examined to identify and correct potential deficiencies, thereby improving the accuracy and consistency of the output.

[0098] like Figure 5 As shown, if the report meets medical standards after evaluation, the final result is returned; if it fails the evaluation, it enters the iterative optimization phase. In this phase, based on feedback, multiple rounds of refinement and correction are performed to ensure the generated report is both professional and accurate. Ultimately, through this series of self-feedback and optimization processes, a complete and standard-compliant medical imaging report can be generated to help doctors make accurate diagnostic and treatment decisions. A medical imaging report is shown below. Figure 10 As shown.

[0099] This invention combines an image segmentation intelligent agent, a prompt word retrieval intelligent agent, a report generation intelligent agent, and an image database. From the image segmentation intelligent agent processing medical images to generate lesion labels, to the prompt word retrieval intelligent agent retrieving example reports based on the lesion labels, and finally to the report generation intelligent agent generating a preliminary report and revising it into a final report, this invention integrates the advantages of different functional modules, thereby improving the accuracy of the final image report and reducing the workload of physicians. This invention introduces a mechanism of professional terminology constraints and reinforcement to revise the preliminary image report, ensuring that the final image report meets medical professional standards and requirements, improving the report's professionalism and credibility in the medical field, and facilitating accurate diagnostic and treatment decisions by medical personnel. Furthermore, during the lesion label generation process of the image segmentation intelligent agent, this invention extracts original image metadata and segmented image metadata, and processes them using a large language model. This multi-layered utilization of data helps to more accurately identify lesion features, thereby accurately describing them in the image report and improving the accuracy of the image report. The prompt word retrieval agent of this invention retrieves example reports from an image database based on a finely tuned large language model. This enables the large language model to better understand the semantic information of lesion labels, thereby obtaining more relevant example reports and providing more reference for the accuracy of the final report.

[0100] In summary, this invention generates accurate and medically compliant image analysis reports while reducing the workload of doctors.

Claims

1. A multi-agent based medical image reporting generation method, characterized by, The method comprises the following steps: Step 1: constructing an image segmentation agent, a prompt word retrieval agent, a report generation agent and an image database; Step 2: inputting the medical image into the image segmentation agent to generate a lesion label through the image segmentation agent; Step 3: inputting the lesion label into the prompt word retrieval agent to retrieve an example report related to the lesion label through the prompt word retrieval agent; Step 4: taking the lesion label and the example report as inputs of the report generation agent, outputting a preliminary image report through the report generation agent, and then introducing a mechanism of professional term condition constraint and reinforcement to modify the preliminary image report to generate a final image report; In step 2, the process of generating the lesion label by the image segmentation agent comprises the following steps: extracting original image metadata from the input medical image and preprocessing the medical image; constructing an image segmentation model, segmenting the preprocessed medical image, and extracting segmented image metadata from the segmented medical image; constructing and training a large language model to obtain a fine-tuned large language model, inputting the original image metadata and the segmented image metadata into the fine-tuned large language model, and generating the lesion label; The specific process of the image segmentation model for segmentation comprises the following steps: using a projection layer to extract layer-by-layer features as the input of an encoder, using deep convolution for convolution, and moving the deep convolution upward to calculate the features; applying residual connection between the input and the last scaling layer, then normalizing the convolution representation in the encoder block by layer normalization to enhance the convergence of the image segmentation model and reduce overfitting, and performing down-sampling operation at different stages to gradually reduce the image resolution of the medical image and increase the number of feature channels to extract higher-level semantic features; performing up-sampling in the decoder part to gradually restore the resolution of the medical image, splicing the features at different levels of the encoder stage with the features at the corresponding levels of the decoder to fuse multi-scale feature information; and finally generating a segmentation result of the medical image. 2.The multi-agent based medical image reporting generation method of claim 1, wherein: The training process of the large language model comprises the following steps: collecting medical image related text data extensively to form a data set; selecting an open source large model, constructing a loss function, and dynamically adjusting the learning rate in real time to realize efficient parameter optimization; and constructing the loss function as follows: wherein y i is the true value of the i-th category label, 0 or 1; p i is the prediction probability of the i-th category label by the large language model, and s is the number of category label samples. Then, a loss correction factor is introduced, and the formula is as follows: C = |p2-p1|; In the formula, p1 is the probability of the existence of lung nodules predicted by the large language model, and p2 is the probability of the existence of hemoptysis; The final loss function value is obtained as follows: L' = L x (1+C). 3.The multi-agent based medical image reporting generation method of claim 1, wherein: In step 3, the process of retrieving the example report related to the lesion label by the prompt word retrieval agent comprises the following steps: using the fine-tuned large language model to retrieve the image database based on the lesion label to obtain the example report related to the lesion label. 4.The multi-agent based medical image reporting generation method of claim 3, wherein: In step 4, the process of outputting the preliminary image report by the report generation agent comprises the following steps: using the fine-tuned large language model to select the preliminary image report most matched with the lesion label based on the example report as the prompt word and the lesion label as the input.

5. The multi-agent based medical image reporting generation method of claim 4, wherein: The process of selecting the preliminary image report most matched with the lesion label comprises the following steps: converting the lesion label and the example report into vector representations Let the vocabulary be V, map the words w in the lesion label or example report to the m-dimensional vector space, and the word vector of the word w is: v w = {v w1 , v w2 ,..., v wm}; where v w1 representing the word vector v w the value in the 1st dimension, v w2 representing the word vector v w the value in the 2nd dimension, v wm representing the word vector v w the value in the mth dimension; For the lesion label text T 病灶 containing n words, the vector representation is: Example report text T 示例 By analogy: Then the similarity between the lesion label and the example report is calculated by the correlation degree: In the formula, m is the vector dimension; According to the calculated similarity, the image report with the highest correlation degree is retrieved and extracted from the image database. 6.The multi-agent based medical image reporting generation method of claim 5, wherein: The mechanism of professional term condition constraint and strengthening includes constructing and integrating a professional term dictionary, setting term constraint rules, and setting term reinforcement learning strategies; the process of constructing and integrating the professional term dictionary is to collect multi-source professional medical term data, construct a comprehensive and authoritative medical professional term dictionary, and integrate the medical professional term dictionary into the large language model; the process of setting term constraint rules is to set mandatory use requirements for professional terms with key diagnostic significance or lesion feature description; the process of setting term reinforcement learning strategies is to give positive rewards if the large language model accurately and completely generates report segments using professional terms according to medical standards during the training of the large language model, and to give negative punishments if the large language model makes errors, omissions or non-standard conditions in the use of professional terms.

Images