A gastrointestinal stromal tumor patient diagnosis and treatment method and device based on multiple agents

Abstract

The application provides a gastrointestinal stromal tumor patient diagnosis and treatment method and device based on multiple agents, the method comprising: acquiring multi-modal patient data, and acquiring a multi-modal base model based on the multi-modal patient data; performing supervised fine-tuning on the multi-modal base model, and performing reinforcement alignment using a preset group relative strategy optimization algorithm to obtain a multi-modal base model after two-stage reasoning capability training; and generating a standardized reasoning result based on the trained multi-modal base model. According to the application, the perceived multi-modal base model is trained for two-stage reasoning capability, and the multi-modal base model is used to generate a reasoning result, so that the perception task is no longer isolated, and perception, reasoning and tracking diagnosis and treatment are integrated into a unified process. In addition, longitudinal decision training is realized, the comprehensiveness of decision training is improved, and the accuracy of reasoning is improved.

Description

Technical Field

[0001] This application relates to the field of medical diagnostic technology, and more specifically, to a method and apparatus for diagnosing and treating patients with gastrointestinal stromal tumors based on multi-agent systems. Background Technology

[0002] Gastrointestinal stromal tumors (GISTs) present unique multimodal and time-span challenges to clinical decision-making: the patient’s accumulated imaging data (CT / MRI / ultrasound), endoscopic images, whole-slice pathology images (WSI), genomic test results, and textual records are still fragmented across different systems.

[0003] Currently, multimodal data is mainly processed through AI solutions, with most models addressing isolated sub-problems such as image classification, risk assessment, or report generation.

[0004] However, existing AI solutions deal with isolated perception tasks, making it difficult to realize mechanisms for perception, reasoning, and continuous patient diagnosis and treatment. Furthermore, the comprehensiveness of decision-making training is generally limited, resulting in mediocre decision accuracy. Summary of the Invention

[0005] In view of this, the purpose of this application is to provide a method and device for diagnosing and treating patients with gastrointestinal stromal tumors based on multi-agent systems. This method trains the reasoning ability of a perceived multimodal basic model in two stages and generates reasoning results from the multimodal basic model. It moves beyond isolated perception tasks, integrating perception, reasoning, and follow-up diagnosis into a unified process. Furthermore, it achieves longitudinal decision-making training, improving the comprehensiveness of decision-making training and thus enhancing the accuracy of reasoning.

[0006] In a first aspect, embodiments of this application provide a method for diagnosing and treating patients with gastrointestinal stromal tumors based on multi-agent systems, the method comprising: Acquire multimodal patient data and obtain a multimodal basic model based on the multimodal patient data; wherein the multimodal patient data includes at least text and visual data; The multimodal base model is subjected to supervised fine-tuning and reinforced alignment using a preset group relative strategy optimization algorithm to obtain a multimodal base model trained with two-stage reasoning ability; wherein, the trained multimodal base model generates expert reasoning trajectories and clinically effective answers; Standardized inference results are generated based on the trained multimodal base model; the inference results represent the treatment plan for the patient.

[0007] In one possible implementation, obtaining the multimodal base model based on the multimodal patient data includes: Each modality in the multimodal patient data is encoded, and the latent feature vector corresponding to each modality is extracted based on the modality-specific encoder corresponding to each modality. The latent feature vector of each modality is projected into a shared embedding space to align the information of different modalities, thereby obtaining the modal representation of the modality in the shared embedding space; Based on a preset contrastive learning objective, the target distance is adjusted in the shared embedding space to align visual and text data, ultimately resulting in a multimodal base model; wherein, the target distance is the distance between matched visual-text sample pairs and the distance between mismatched visual-text sample pairs, and the sample pair is a visual-text sample pair; Based on the contrastive learning objective, backpropagation is performed to update the modality-specific encoders and modality representations corresponding to each modality, so as to obtain a multimodal base model based on the updated modality-specific encoders and modality representations.

[0008] In one possible implementation, the supervised fine-tuning of the multimodal base model includes: Training samples are determined based on the multimodal patient data; wherein, the training samples include multimodal context, target text corresponding to the multimodal context, and may also include reasoning steps and a final answer; the reasoning steps represent expert reasoning trajectories, and the final answer represents clinically effective answers; Based on the training samples, the multimodal base model is supervised and fine-tuned to maximize the probability of generating expert reasoning trajectories and clinically effective answers, resulting in a supervised and fine-tuned multimodal base model; wherein, the supervised and fine-tuned multimodal base model maximizes the probability of generating expert reasoning trajectories and clinically effective answers.

[0009] In one possible implementation, the enhanced alignment using a preset group-relative strategy optimization algorithm includes: For each multimodal context in the multimodal patient data, candidate outputs for the target number are sampled from the current group relative strategy based on the multimodal base model, and a corresponding scalar reward is calculated for each candidate output; Based on the candidate outputs and the corresponding scalar rewards, the relative advantage between groups is calculated within the group, and the relative strategy between groups is updated by weighting based on the relative advantage between groups. A supervised fine-tuning checkpoint and a reference model for the supervised fine-tuning checkpoint are determined, and the divergence of the reference model is regularized based on a preset tag index set to bring the group relative strategy closer to the reference model in order to align the reasoning behavior of the multimodal base model; wherein, the tag index set includes tags for the reasoning steps and / or the final answer; Based on the training process of supervised fine-tuning and reinforcement alignment of relative advantages between groups, a multimodal basic model after two-stage reasoning ability training is obtained.

[0010] In one possible implementation, the step of calculating the corresponding scalar reward for each candidate output further includes: Based on the candidate output and the target text, determine the corresponding first reward signal and second reward signal; A scalar reward is obtained based on the first reward signal and the second reward signal; The determination of the corresponding first reward signal includes: Calculate the standardized semantic similarity between the candidate output and the target text, and determine the first reward signal based on the standardized semantic similarity.

[0011] In one possible implementation, the generation of standardized inference results based on the trained multimodal base model includes: For each target step, the corresponding maintenance status is determined, and a corresponding decision action is generated based on the maintenance status; wherein, generating an action means generating the decision output for the current step, and the decision action is a reasoning statement or a tool call; Based on the decision-making action, the corresponding decision trajectory is obtained, and the relative strategy of the group is adjusted based on the decision trajectory to maximize the expected cumulative clinical benefit, so as to obtain the reasoning result.

[0012] In one possible implementation, the method further includes: Dialogue is conducted through the human-computer interaction interface provided by the patient-side agent, and patient information is reported. The patient information is processed by a lightweight encoder aligned with the multimodal base model to obtain a corresponding 24 / 7 follow-up plan, and a corresponding follow-up schedule is generated and displayed based on the 24 / 7 follow-up plan.

[0013] Secondly, embodiments of this application also provide a multi-agent-based diagnostic and treatment device for patients with gastrointestinal stromal tumors, the device comprising: An acquisition module is used to acquire multimodal patient data and acquire a multimodal basic model based on the multimodal patient data; wherein the multimodal patient data includes at least text and visual data; The training module is used to supervise and fine-tune the multimodal base model and perform reinforcement alignment using a preset group relative strategy optimization algorithm to obtain a multimodal base model after two-stage reasoning ability training; wherein, the trained multimodal base model generates expert reasoning trajectories and clinically effective answers; The inference module is used to generate standardized inference results based on the trained multimodal base model; the inference results represent the treatment plan for the patient.

[0014] In one possible implementation, the acquisition module is specifically used for: Each modality in the multimodal patient data is encoded, and the latent feature vector corresponding to each modality is extracted based on the modality-specific encoder corresponding to each modality. The latent feature vector of each modality is projected into a shared embedding space to align the information of different modalities, thereby obtaining the modal representation of the modality in the shared embedding space; Based on a preset contrastive learning objective, the target distance is adjusted in the shared embedding space to align visual and text data, ultimately resulting in a multimodal base model; wherein, the target distance is the distance between matched visual-text sample pairs and the distance between mismatched visual-text sample pairs, and the sample pair is a visual-text sample pair; Based on the contrastive learning objective, backpropagation is performed to update the modality-specific encoders and modality representations corresponding to each modality, so as to obtain a multimodal base model based on the updated modality-specific encoders and modality representations.

[0015] In one possible implementation, the training module is specifically used for: Training samples are determined based on the multimodal patient data; wherein, the training samples include multimodal context, target text corresponding to the multimodal context, and may also include reasoning steps and a final answer; the reasoning steps represent expert reasoning trajectories, and the final answer represents clinically effective answers; Based on the training samples, the multimodal base model is supervised and fine-tuned to maximize the probability of generating expert reasoning trajectories and clinically effective answers, resulting in a supervised and fine-tuned multimodal base model; wherein, the supervised and fine-tuned multimodal base model maximizes the probability of generating expert reasoning trajectories and clinically effective answers.

[0016] In one possible implementation, the training module is specifically used for: For each multimodal context in the multimodal patient data, candidate outputs for the target number are sampled from the current group relative strategy based on the multimodal base model, and a corresponding scalar reward is calculated for each candidate output; Based on the candidate outputs and the corresponding scalar rewards, the relative advantage between groups is calculated within the group, and the relative strategy between groups is updated by weighting based on the relative advantage between groups. A supervised fine-tuning checkpoint and a reference model for the supervised fine-tuning checkpoint are determined, and the divergence of the reference model is regularized based on a preset tag index set to bring the group relative strategy closer to the reference model in order to align the reasoning behavior of the multimodal base model; wherein, the tag index set includes tags for the reasoning steps and / or the final answer; Based on the training process of supervised fine-tuning and reinforcement alignment of relative advantages between groups, a multimodal basic model after two-stage reasoning ability training is obtained.

[0017] In one possible implementation, the training module is specifically used for: Based on the candidate output and the target text, determine the corresponding first reward signal and second reward signal; A scalar reward is obtained based on the first reward signal and the second reward signal; The training module is also specifically used for: Calculate the standardized semantic similarity between the candidate output and the target text, and determine the first reward signal based on the standardized semantic similarity.

[0018] In one possible implementation, the inference module is specifically used for: For each target step, the corresponding maintenance status is determined, and a corresponding decision action is generated based on the maintenance status; wherein, generating an action means generating the decision output for the current step, and the decision action is a reasoning statement or a tool call; Based on the decision-making action, the corresponding decision trajectory is obtained, and the relative strategy of the group is adjusted based on the decision trajectory to maximize the expected cumulative clinical benefit, so as to obtain the reasoning result.

[0019] In one possible implementation, the device further includes: The reporting module is used to communicate and report patient information through the human-computer interaction interface provided by the patient-side agent; The generation module is used to process the patient information through a lightweight encoder aligned with the multimodal base model to obtain the corresponding 24 / 7 follow-up plan, and generate and display the corresponding follow-up schedule based on the 24 / 7 follow-up plan.

[0020] Thirdly, embodiments of this application provide an electronic device, including: a processor, a storage medium, and a bus, wherein the storage medium stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor communicates with the storage medium via the bus, and the processor executes the machine-readable instructions to perform the steps of the multi-agent-based gastrointestinal stromal tumor patient diagnosis and treatment method as described in any of the first aspects.

[0021] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the steps of the multi-agent-based method for diagnosing and treating patients with gastrointestinal stromal tumors as described in any one of the first aspects.

[0022] This application provides a method and apparatus for diagnosing and treating patients with gastrointestinal stromal tumors based on multi-agent systems. The method involves acquiring multimodal patient data and obtaining a multimodal base model based on this data. The multimodal base model is then subjected to supervised fine-tuning and reinforced using a pre-defined group-relative strategy optimization algorithm, resulting in a multimodal base model trained with two-stage reasoning capabilities. Standardized reasoning results are generated based on the trained multimodal base model. This application, by training the perceived multimodal base model with two-stage reasoning capabilities and generating reasoning results from the multimodal base model, moves beyond isolated perception tasks, integrating perception, reasoning, and follow-up diagnosis into a unified process. Furthermore, it achieves longitudinal decision training, improving the comprehensiveness of decision training and thus enhancing the accuracy of reasoning.

[0023] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0024] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0025] Figure 1 This is a flowchart of a multi-agent-based method for the diagnosis and treatment of patients with gastrointestinal stromal tumors, provided according to an embodiment of this application. Figure 2 This is a schematic diagram of the structure of a multi-agent-based diagnostic and treatment device for gastrointestinal stromal tumor patients provided according to an embodiment of this application; Figure 3 This is a schematic diagram of the structure of an electronic device provided according to an embodiment of this application. Detailed Implementation

[0026] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. It should be understood that the accompanying drawings in this application are for illustrative and descriptive purposes only and are not intended to limit the scope of protection of this application. Furthermore, it should be understood that the schematic drawings are not drawn to scale. The flowcharts used in this application illustrate operations implemented according to some embodiments of this application. It should be understood that the operations in the flowcharts may not be implemented in sequence, and steps without logical contextual relationships may be reversed or implemented simultaneously. In addition, those skilled in the art, guided by the content of this application, may add one or more other operations to the flowcharts, or remove one or more operations from the flowcharts.

[0027] Furthermore, the described embodiments are merely some, not all, of the embodiments of this application. The components of the embodiments of this application described and illustrated herein can typically be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of the application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0028] It should be noted that the term "comprising" will be used in the embodiments of this application to indicate the presence of the features declared thereafter, but does not exclude the addition of other features.

[0029] Given that gastrointestinal stromal tumors (GISTs) present unique multimodal and time-span challenges to clinical decision-making, the patient’s accumulated imaging data (CT / MRI / ultrasound), endoscopic images, whole-slice pathology images (WSI), genomic testing results, and textual records remain fragmented across different systems.

[0030] Currently, multimodal data is mainly processed through AI solutions, with most models addressing isolated sub-problems such as image classification, risk assessment, or report generation.

[0031] However, existing AI solutions deal with isolated perception tasks, making it difficult to realize mechanisms for perception, reasoning, and continuous patient diagnosis and treatment. Furthermore, the comprehensiveness of decision-making training is generally limited, resulting in mediocre decision accuracy.

[0032] To address this issue, this application provides a multi-agent-based method and apparatus for diagnosing and treating patients with gastrointestinal stromal tumors. By training the reasoning ability of a perceived multimodal basic model in two stages and generating reasoning results from the multimodal basic model, the method moves beyond isolated perception tasks, integrating perception, reasoning, and follow-up diagnosis into a unified process. Furthermore, it implements longitudinal decision-making training, improving the comprehensiveness of decision-making training and thus enhancing the accuracy of reasoning.

[0033] Figure 1 This is a flowchart of a multi-agent-based method for the diagnosis and treatment of gastrointestinal stromal tumor patients, provided according to embodiments of this application. Figure 1 As shown in the embodiments of this application, the multi-agent-based method for diagnosing and treating patients with gastrointestinal stromal tumors may specifically include: S101. Obtain multimodal patient data and obtain a multimodal basic model based on the multimodal patient data.

[0034] S102. Supervised fine-tuning of the multimodal base model and reinforcement alignment using a preset group relative policy optimization algorithm are performed to obtain the multimodal base model after two-stage inference ability training.

[0035] S103. Standardized inference results are generated based on the trained multimodal base model.

[0036] In the aforementioned multi-agent-based diagnostic and treatment method for gastrointestinal stromal tumors, a two-stage reasoning ability training process is implemented on the perceived multimodal basic model. The reasoning results are then generated using the multimodal basic model, moving beyond isolated perception tasks and integrating perception, reasoning, and follow-up diagnosis into a unified process. Furthermore, longitudinal decision-making training is implemented, improving the comprehensiveness of decision-making training and thus enhancing the accuracy of reasoning.

[0037] The exemplary steps described above in the embodiments of this application are illustrated below with specific examples: S101, acquire multimodal patient data, and acquire a multimodal basic model based on the multimodal patient data.

[0038] In this embodiment, the multimodal patient data includes at least text and visual data, and the categories of multimodal patient data include at least radiological, endoscopic, pathological, and textual data, i.e., CT images, pathological slides, or clinical reports, etc.; the multimodal basic model uniformly presents imaging and pathological features, characterizing the cross-modal consistency of tumor description and diagnosis; the multimodal basic model is obtained based on the collected multimodal patient data for subsequent processing.

[0039] Optionally, when obtaining a multimodal base model based on multimodal patient data, each modality in the multimodal patient data is encoded, and the latent feature vector corresponding to each modality is extracted based on the modality-specific encoder corresponding to each modality; the latent feature vector of each modality is projected into a shared embedding space to align the information of different modalities and obtain the modality representation of the modality in the shared embedding space; the target distance is adjusted in the shared embedding space based on a preset contrastive learning objective to align visual and textual data, and finally, a multimodal base model is obtained; backpropagation is performed based on the contrastive learning objective to update the modality-specific encoder and modality representation corresponding to each modality, so as to obtain the multimodal base model based on the updated modality-specific encoder and modality representation.

[0040] The target distance is the distance between matching visual-text sample pairs and the distance between non-matching visual-text sample pairs, and the sample pair is a visual-text sample pair; the temperature parameter is the temperature in the contrastive learning softmax.

[0041] Specifically, considering multimodal patient data X={x(1)'x(2)'...'x(M)}, each modality x (e.g., CT images, pathological slides, or clinical reports) is encoded by a modality-specific encoder f_{θ_m}^{(m)} to obtain z^(m)=, where z^(m) is a latent feature vector extracted from modality m. These features are projected into a shared embedding space (semantic space) through a learnable mapping W_m to obtain h^(m)=W_mz^(m), so as to align the information of different modalities to the same shared embedding space, achieve cross-modal alignment, and make semantics consistent between different modalities.

[0042] Continuing on, to ensure semantic alignment between different modalities (e.g., alignment between "enhanced gastric mass" and "suspected low-risk gastrointestinal stromal tumor"), this application employs a contrastive learning objective. This objective function achieves cross-modal consistency between tumor description and diagnosis by narrowing the distance between matching visual-text samples in the embedding space, while simultaneously narrowing the distance between matching visual-text sample pairs and widening the distance between mismatched visual-text sample pairs. Ultimately, the unified representation obtained by the multimodal base model becomes the perceptual core for downstream inference and treatment recommendations.

[0043] S102, supervised fine-tuning of the multimodal base model and reinforcement alignment using a preset group relative policy optimization algorithm to obtain the multimodal base model after two-stage inference ability training.

[0044] In this embodiment, the trained multimodal base model can generate expert reasoning trajectories and clinically valid answers. The multimodal base model is subjected to a first-stage supervised fine-tuning (SFT) and a second-stage reinforcement alignment (GRPO) is performed using a pre-grouped relative policy optimization algorithm to train the reasoning ability of the multimodal base model. The resulting multimodal base model with two-stage reasoning ability training is then used for subsequent processing.

[0045] It should be noted that although multimodal base models can uniformly represent imaging and pathological features, accurate clinical decision-making still requires explicit reasoning capabilities. Traditional supervised learning models often only capture the correlation between input data and output but lack the step-by-step reasoning process to support medical judgment. To achieve interpretable reasoning and reliable diagnostic and treatment recommendations, this application enhances large language models (LLMs) through a two-stage training strategy: first, supervised fine-tuning, and then using a group-relative strategy optimization to reinforce alignment.

[0046] Optionally, when performing supervised fine-tuning on the multimodal base model, training samples are determined based on multimodal patient data; the multimodal base model is then supervised fine-tuned based on the training samples to maximize the probability of generating expert reasoning trajectories and clinically valid answers, resulting in the supervised fine-tuned multimodal base model.

[0047] The training samples include multimodal context and target text corresponding to the multimodal context (structured output sequence of expert writing). The training samples may also include reasoning steps and final answers. Reasoning steps represent expert reasoning trajectories, and final answers represent clinically effective answers. The supervised fine-tuned multimodal base model maximizes the probability of generating expert reasoning trajectories and clinically effective answers.

[0048] Specifically, a group-relative policy model is first trained on paired multimodal cues and structured responses. Each instance (xi, yi) contains a multimodal context xi and its corresponding target text yi, which may simultaneously include reasoning steps and the final answer. The objective function follows a standard conditional language model loss. This stage aims to maximize the likelihood of expert-written reasoning trajectories and clinically valid answers. This stage builds the model's language generation on medical logic and domain-specific terminology, enabling it to simultaneously include reasoning steps and the final answer.

[0049] In some implementations, when using a pre-defined group-relative policy optimization algorithm for reinforcement alignment, for each multimodal context in the multimodal patient data, a target number of candidate outputs are sampled from the current group-relative policy based on the multimodal base model, and a corresponding scalar reward is calculated for each candidate output. Within each group, relative dominance between groups is calculated based on the candidate outputs and their corresponding scalar rewards, and the group-relative policy is updated weighted based on this relative dominance. Supervised fine-tuning checkpoints and a reference model for these checkpoints are determined, and the divergence of the reference model is regularized based on a pre-defined labeled index set to bring the group-relative policy closer to the reference model, thereby aligning the reasoning behavior of the multimodal base model. The training process, which balances supervised fine-tuning and reinforcement alignment based on the relative dominance between groups, yields a multimodal base model trained with two stages of reasoning ability. The labeled index set includes labels for reasoning steps and / or the final answer.

[0050] Specifically, after SFT, GRPO further optimizes the inference behavior to align with clinical preferences. GRPO is a stable policy optimization algorithm designed specifically for multi-sample relative reward comparison. For each multimodal cue x, the model samples a set of k runs from the current policy (i.e., candidate outputs for the target number). Each run receives a scalar reward, and the inter-group relative advantage is calculated. A relative advantage greater than 0 avoids numerical instability. GRPO aims to update the policy for generations with higher rewards while regularizing the divergence of the reference model at the ref (i.e., SFT checkpoint). The labeled index set is the set of labeled indices used for optimization (covering inference and / or answer labels), and it involves controlling the strength of the regularization to bring the policy close to its SFT initialization.

[0051] It should be noted that when calculating the scalar reward for each candidate output, a first reward signal and a second reward signal are determined based on the candidate output and the target text; the scalar reward is obtained based on the first reward signal and the second reward signal. Optionally, the normalized semantic similarity between the candidate output and the target text is calculated, and the first reward signal is determined based on the normalized semantic similarity.

[0052] The target text represents the structured output sequence of expert writing; the first reward signal is used to evaluate the correctness of the final clinical decision, and the second reward signal is used to measure whether the output structure conforms to the expected reasoning-action pattern; the reward function ensures that the multimodal base model follows the structured output template, which is the structured output template necessary to achieve ReAct-style reasoning consistency.

[0053] Specifically, to ensure that the improvements are both practical and easy to interpret, this application designs two reward signals that directly correspond to the expected behavior in the output of a large-scale reasoning-based language model. The first reward signal is used to evaluate the correctness of the final clinical decision, and the second reward signal is used to measure whether the output structure conforms to the expected reasoning-action pattern. The first reward signal can be determined by calculating the standardized semantic similarity (such as exact match, embedding similarity, or guideline compliance) between the generated output and the text. Furthermore, the model is ensured to follow structured output templates, such as: thinking steps → action steps → observation results... These templates are necessary to achieve ReAct-style reasoning consistency. Both reward metrics are standardized to the [0, 1] interval, and their relative weights are controlled by 0. In the training settings of this application, the weight can be 1.

[0054] In summary, this application combines a two-stage training optimization strategy, in which the relative advantage between groups is balanced between supervised learning and reinforcement learning. This two-step optimization strategy first lays the foundation for the model through expert demonstration, and then reinforces behaviors that produce accurate and structurally sound reasoning paths. Through this process, the agent learns to express medically plausible thought sequences in language and generate standardized ReAct-like outputs, thereby significantly improving the interpretability and reliability of reasoning in clinical decision-making.

[0055] S103 generates standardized inference results based on the trained multimodal base model.

[0056] In this embodiment of the application, the reasoning result represents the treatment plan for the patient. A standardized reasoning result is generated based on the multimodal basic model after the two-stage reasoning ability training in step S102, so as to help the patient be diagnosed and treated according to the reasoning result.

[0057] Optionally, when generating standardized inference results based on the trained multimodal base model, for the target step, the corresponding maintenance state is determined, and a corresponding decision action is generated based on the maintenance state; the corresponding decision trajectory is obtained based on the decision action, and the group relative strategy is adjusted based on the decision trajectory to maximize the expected cumulative clinical benefit to obtain the inference result. Here, the maintenance state includes the multimodal context, action observations prior to the target step, and tool observations; the generated action represents the generation of the decision output for the current step, and the decision action is either an inference statement or a tool call.

[0058] It should be noted that the management of gastrointestinal stromal tumors in the real world requires integrating continuously updated external knowledge, including the latest clinical guidelines, DRG reimbursement policies, and drug dosage regimens. Static inference models have limitations because they cannot dynamically query such external information. This application overcomes this limitation through a ReAct (Reasoning + Acting) framework that intertwines reasoning and calling tools, generating interpretable and fact-based decision-making trajectories.

[0059] Specifically, at each step t, the agent maintains a state st=(x, a<t, o<t), where a<t and o<t represent the previous actions and tool observations. The policy generates an action at the moment (at|st), which can be an inference statement (Thought) or a tool call (Action). A typical operation process is as follows: Thought: "The lesion size is 5 cm...", Action: Query (GIST_Guideline_Tool, tumor_size = 5 cm), Observation result: "Imatinib 400 mg / day for 3 - 6 months is recommended." Continuing, the trajectory of the agent is optimized to maximize the expected cumulative clinical benefit, and r(st, at, ot) is used to evaluate fact accuracy, reasoning consistency, and safety compliance. This architecture enables the intelligent agent to dynamically dock with real-world medical resources, obtain guideline updates in real time, verify reimbursement restrictions, and comprehensively formulate personalized treatment plans. By combining reasoning transparency with a factual basis, the intelligent agent provides evidence-based decision-making support for clinicians, helping them make wise choices that conform to the clinical context.

[0060] The multi-agent-based method for diagnosing and treating gastrointestinal stromal tumor patients provided by the embodiments of this application obtains multi-modal patient data and obtains a multi-modal base model based on the multi-modal patient data; performs supervised fine-tuning on the multi-modal base model and uses a preset group relative policy optimization algorithm for reinforcement alignment to obtain a multi-modal base model after two-stage reasoning ability training; generates a standardized reasoning result based on the trained multi-modal base model. The multi-agent-based method for diagnosing and treating gastrointestinal stromal tumor patients of this application, through two-stage reasoning ability training on the perceived multi-modal base model and generating a reasoning result with the multi-modal base model, is no longer an isolated perception task, integrating perception, reasoning, and tracking diagnosis and treatment into a unified process. In addition, longitudinal decision training is achieved, improving the comprehensiveness of decision training and thus the accuracy of reasoning.

[0061] Furthermore, a dialogue is carried out through the human-computer interaction interface provided by the patient-side agent to report patient information; the patient information is processed by a lightweight encoder aligned with the multi-modal base model to obtain a corresponding all-weather follow-up plan, and a corresponding follow-up schedule is generated and displayed based on the all-weather follow-up plan.

[0062] The patient information should include at least reported symptoms, examination images, or adverse reactions; the 24 / 7 follow-up plan should include at least adverse event triage (such as nausea, edema, hepatotoxicity), medication reminders, and lifestyle guidance.

[0063] It should be noted that, in addition to the decision support functions on the doctor's side, this application also integrates a dialogue agent on the patient's side, facilitating continuous follow-up and personalized monitoring. This module provides a human-computer interaction interface, allowing patients to report symptoms, upload examination images, or describe adverse reactions via natural language. The dialogue agent processes multimodal input (including text, voice, or photos) through a lightweight encoder aligned with the multimodal underlying model, ensuring consistent representation with the doctor's side.

[0064] Continuing on, the patient-side system enables 24 / 7 virtual follow-up, including adverse event triage (such as nausea, edema, and hepatotoxicity), medication reminders, and lifestyle guidance. All conversation records are synchronized to the clinical backend system for physician review, forming a closed-loop "human-AI-human" collaboration mechanism. This supports proactive management of long-term diagnosis and treatment of gastrointestinal stromal tumors and promotes patient participation in precision medicine.

[0065] Furthermore, based on patient information, corresponding prompts are generated and sent to enable patients to supplement the required patient information in the report; patient information is identified, and corresponding personalized suggestions are generated.

[0066] Specifically, the device utilizes built-in artificial intelligence technology to monitor simple symptoms. For example, when a patient reports leg swelling, the application prompts them to upload a photo of their leg. Using a lightweight visual model (trained for image-based edema detection), the assistant can identify peripheral edema and automatically generate personalized suggestions, such as instructing the patient to elevate the affected limb, reduce salt intake, and monitor weight daily. The assistant also cross-checks medication information against a local knowledge base (edema is a known side effect of imatinib). The system explains to the patient that mild edema is a common and manageable side effect. This interaction can be completed instantly without clinician intervention, but the system records the event and alerts the clinical team if the edema becomes severe or worsens. The patient side can also handle other routine tasks (such as medication adherence checks and symptom questionnaires), providing automated guidance or scheduling earlier follow-ups as needed. This closed-loop integration ensures that treatment plans are not only implemented but also dynamically adjusted based on patient feedback, all through the system. The resulting intelligent clinical assistant possesses continuous learning and responsiveness capabilities, providing full support to clinicians and patients throughout the treatment process.

[0067] The following case study, presented by Juyi, demonstrates the multi-agent-based diagnostic and treatment method for gastrointestinal stromal tumors (GIST) patients. The case illustrates how this application processes and analyzes patient data from multiple modal sources to guide clinical decision-making and provide customized recommendations for treatment plans (including neoadjuvant therapy, surgery, and follow-up planning).

[0068] Case 1 (Initial Diagnosis): A 29-year-old male patient presented with an abdominal mass discovered during a routine ultrasound examination. The initial CT scan revealed a 6.7 cm mass in the gastric body, which was heterogeneous and had irregular borders, with possible involvement of the left lobe of the liver. The patient had no significant medical history other than mild hypertension. It was necessary to determine whether neoadjuvant therapy (such as imatinib) was required before surgery, or whether direct surgery was appropriate.

[0069] Multimodal perception: The system integrates the patient's CT scan images, clinical history, and all available laboratory test results. Using DICOM image analysis technology, the system processes the CT scan data to accurately locate the mass and assess its size. Simultaneously, the agent retrieves the initial ultrasound report, indicating that the mass exhibits hypoechoic characteristics and irregular margins, potentially suggesting malignancy. Further system confirmation of no distant metastases was found in chest and abdominal CT slices. Comprehensive multimodal data analysis revealed that the mass is adjacent to vital organs (especially the liver), making direct surgery potentially too risky.

[0070] Diagnostic Reasoning and Treatment Plan Development: Given the tumor's proximity to the liver and the inconclusive biopsy results, the system assessed the risks and benefits of immediate surgery. The systematic analysis concluded that surgery at this stage might lead to combined organ resection due to unclear tumor boundaries or complications. Therefore, the system recommended a more cautious approach. The next step should be to obtain more tissue samples through biopsy for pathological examination and molecular testing. Based on the molecular testing results, if a gastrointestinal stromal tumor (GIST) is diagnosed with a ckit11 exon mutation, the system recommends initiating neoadjuvant imatinib therapy. This treatment aims to reduce tumor size, making the surgical procedure safer and less invasive.

[0071] Output: The system generates detailed treatment plans for the clinical team. It recommends more precise tumor characterization through contrast-enhanced CT and gastroscopy, and suggests a biopsy to confirm the tumor type, as well as KIT / PDGFRA gene mutation testing to guide treatment selection. If diagnosed with gastrointestinal stromal tumor (GIST), based on gene testing results, the system recommends neoadjuvant therapy with imatinib to shrink the tumor before considering surgery. This progressive treatment pathway ensures that treatment decisions are based on comprehensive and reliable data.

[0072] In summary, this application implements a unified multi-agent framework that integrates (1) a multimodal base model for perception, (2) tool-enhanced reasoning for dynamic decision-making, and (3) a patient-side agent for continuous monitoring. Unlike traditional AI systems that handle perception and reasoning separately, this system combines visual-text embedding with guideline-based agent reasoning and continuous patient-side monitoring. Through a combination of SFT and GRPO fine-tuning, a large language model can perform structured, stepwise reasoning, while a ReAct-style controller dynamically invokes external medical tools for evidence verification. This architecture enables the system to support a complete end-to-end clinical workflow, covering the entire process from initial diagnosis (Case 1), surgical planning to long-term follow-up, generating traceable reasoning trajectories for clinicians to review, thereby improving diagnostic credibility and supporting timely treatment planning.

[0073] Therefore, this application demonstrates that the synergistic effect between multimodal perception, reinforcement learning-driven reasoning, dynamic tool use, and closed-loop patient feedback mechanisms can bridge the gap between pattern recognition and longitudinal clinical diagnosis and treatment.

[0074] Figure 2 This is a schematic diagram of the structure of a multi-agent-based diagnostic and treatment device for gastrointestinal stromal tumor patients provided according to an embodiment of this application; as shown below. Figure 2 As shown, the multi-agent-based gastrointestinal stromal tumor patient diagnosis and treatment device 200 of this application embodiment may specifically include: The acquisition module 201 is used to acquire multimodal patient data and acquire a multimodal basic model based on the multimodal patient data; wherein the multimodal patient data includes at least text and visual data; Training module 202 is used to supervise and fine-tune the multimodal base model and perform reinforcement alignment using a preset group relative strategy optimization algorithm to obtain a multimodal base model after two-stage reasoning ability training; wherein, the trained multimodal base model generates expert reasoning trajectories and clinically effective answers; The reasoning module 203 is used to generate standardized reasoning results based on the trained multimodal base model; wherein the reasoning results represent the treatment plan for the patient.

[0075] In one possible implementation, the acquisition module is specifically used for: Each modality in the multimodal patient data is encoded, and the latent feature vector corresponding to each modality is extracted based on the modality-specific encoder corresponding to each modality. The latent feature vector of each modality is projected into a shared embedding space to align the information of different modalities, thereby obtaining the modal representation of the modality in the shared embedding space; Based on a preset contrastive learning objective, the target distance is adjusted in the shared embedding space to align visual and text data, ultimately resulting in a multimodal base model; wherein, the target distance is the distance between matched visual-text sample pairs and the distance between mismatched visual-text sample pairs, and the sample pair is a visual-text sample pair; Based on the contrastive learning objective, backpropagation is performed to update the modality-specific encoders and modality representations corresponding to each modality, so as to obtain a multimodal base model based on the updated modality-specific encoders and modality representations.

[0076] In one possible implementation, the training module is specifically used for: Training samples are determined based on the multimodal patient data; wherein, the training samples include multimodal context, target text corresponding to the multimodal context, and may also include reasoning steps and a final answer; the reasoning steps represent expert reasoning trajectories, and the final answer represents clinically effective answers; Based on the training samples, the multimodal base model is supervised and fine-tuned to maximize the probability of generating expert reasoning trajectories and clinically effective answers, resulting in a supervised and fine-tuned multimodal base model; wherein, the supervised and fine-tuned multimodal base model maximizes the probability of generating expert reasoning trajectories and clinically effective answers.

[0077] In one possible implementation, the training module is specifically used for: For each multimodal context in the multimodal patient data, candidate outputs for the target number are sampled from the current group relative strategy based on the multimodal base model, and a corresponding scalar reward is calculated for each candidate output; Based on the candidate outputs and the corresponding scalar rewards, the relative advantage between groups is calculated within the group, and the relative strategy between groups is updated by weighting based on the relative advantage between groups. A supervised fine-tuning checkpoint and a reference model for the supervised fine-tuning checkpoint are determined, and the divergence of the reference model is regularized based on a preset tag index set to bring the group relative strategy closer to the reference model in order to align the reasoning behavior of the multimodal base model; wherein, the tag index set includes tags for the reasoning steps and / or the final answer; Based on the training process of supervised fine-tuning and reinforcement alignment of relative advantages between groups, a multimodal basic model after two-stage reasoning ability training is obtained.

[0078] In one possible implementation, the training module is specifically used for: Based on the candidate output and the target text, determine the corresponding first reward signal and second reward signal; A scalar reward is obtained based on the first reward signal and the second reward signal; The training module is also specifically used for: Calculate the standardized semantic similarity between the candidate output and the target text, and determine the first reward signal based on the standardized semantic similarity.

[0079] In one possible implementation, the inference module is specifically used for: For each target step, the corresponding maintenance status is determined, and a corresponding decision action is generated based on the maintenance status; wherein, generating an action means generating the decision output for the current step, and the decision action is a reasoning statement or a tool call; Based on the decision-making action, the corresponding decision trajectory is obtained, and the relative strategy of the group is adjusted based on the decision trajectory to maximize the expected cumulative clinical benefit, so as to obtain the reasoning result.

[0080] In one possible implementation, the device further includes: The reporting module is used to communicate and report patient information through the human-computer interaction interface provided by the patient-side agent; The generation module is used to process the patient information through a lightweight encoder aligned with the multimodal base model to obtain the corresponding 24 / 7 follow-up plan, and generate and display the corresponding follow-up schedule based on the 24 / 7 follow-up plan.

[0081] The gastrointestinal stromal tumor (GIST) patient diagnosis and treatment device based on multi-agent technology provided in this application acquires multimodal patient data and obtains a multimodal basic model based on the multimodal patient data. The multimodal basic model is then subjected to supervised fine-tuning and reinforced alignment using a preset group-relative strategy optimization algorithm, resulting in a multimodal basic model trained with two-stage reasoning capabilities. Standardized reasoning results are generated based on the trained multimodal basic model. This multi-agent GIST patient diagnosis and treatment device, by training the perceived multimodal basic model with two-stage reasoning capabilities and generating reasoning results from the multimodal basic model, transcends isolated perception tasks, integrating perception, reasoning, and follow-up diagnosis and treatment into a unified process. Furthermore, it achieves vertical decision training, improving the comprehensiveness of decision training and thus enhancing the accuracy of reasoning.

[0082] like Figure 3As shown in the embodiment of this application, an electronic device 300 includes a processor 301, a memory 302, and a bus. The memory 302 stores machine-readable instructions executable by the processor 301. When the electronic device is running, the processor 301 communicates with the memory 302 via the bus. The processor 301 executes the machine-readable instructions to perform the steps of the multi-agent-based gastrointestinal stromal tumor patient diagnosis and treatment method described above.

[0083] Specifically, the memory 302 and processor 301 mentioned above can be general-purpose memory and processor, without any specific limitations. When the processor 301 runs the computer program stored in the memory 302, it can execute the above-mentioned multi-agent-based diagnosis and treatment method for patients with gastrointestinal stromal tumors.

[0084] Corresponding to the above-described multi-agent-based method for diagnosing and treating patients with gastrointestinal stromal tumors, this application also provides a computer-readable storage medium storing a computer program. When the computer program is run by a processor, it executes the steps of the above-described multi-agent-based method for diagnosing and treating patients with gastrointestinal stromal tumors.

[0085] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems and devices described above can be referred to the corresponding processes in the method embodiments, and will not be repeated here. In the several embodiments provided in this application, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. Furthermore, multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed mutual coupling or direct coupling or communication connection can be through some communication interfaces; the indirect coupling or communication connection of devices or modules can be electrical, mechanical, or other forms.

[0086] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0087] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0088] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a processor-executable, non-volatile, computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the deployment methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, ROM, RAM, magnetic disks, or optical disks.

[0089] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for diagnosing and treating patients with gastrointestinal stromal tumors based on multi-agent systems, characterized in that, The method includes: Acquire multimodal patient data and obtain a multimodal basic model based on the multimodal patient data; wherein the multimodal patient data includes at least text and visual data; The multimodal base model is subjected to supervised fine-tuning and reinforced alignment using a preset group relative strategy optimization algorithm to obtain a multimodal base model trained with two-stage reasoning ability; wherein, the trained multimodal base model generates expert reasoning trajectories and clinically effective answers; Standardized inference results are generated based on the trained multimodal base model; the inference results represent the treatment plan for the patient.

2. The method according to claim 1, characterized in that, The process of obtaining a multimodal basic model based on the multimodal patient data includes: Each modality in the multimodal patient data is encoded, and the latent feature vector corresponding to each modality is extracted based on the modality-specific encoder corresponding to each modality. The latent feature vector of each modality is projected into a shared embedding space to align the information of different modalities, thereby obtaining the modal representation of the modality in the shared embedding space; Based on a preset contrastive learning objective, the target distance is adjusted in the shared embedding space to align visual and text data, ultimately resulting in a multimodal base model; wherein, the target distance is the distance between matched visual-text sample pairs and the distance between mismatched visual-text sample pairs, and the sample pair is a visual-text sample pair; Based on the contrastive learning objective, backpropagation is performed to update the modality-specific encoders and modality representations corresponding to each modality, so as to obtain a multimodal base model based on the updated modality-specific encoders and modality representations.

3. The method according to claim 1, characterized in that, The supervised fine-tuning of the multimodal base model includes: Training samples are determined based on the multimodal patient data; wherein, the training samples include multimodal context, target text corresponding to the multimodal context, and may also include reasoning steps and a final answer; the reasoning steps represent expert reasoning trajectories, and the final answer represents clinically effective answers; Based on the training samples, the multimodal base model is supervised and fine-tuned to maximize the probability of generating expert reasoning trajectories and clinically effective answers, resulting in a supervised and fine-tuned multimodal base model; wherein, the supervised and fine-tuned multimodal base model maximizes the probability of generating expert reasoning trajectories and clinically effective answers.

4. The method according to claim 3, characterized in that, The enhanced alignment using a preset group-relative strategy optimization algorithm includes: For each multimodal context in the multimodal patient data, candidate outputs for the target number are sampled from the current group relative strategy based on the multimodal base model, and a corresponding scalar reward is calculated for each candidate output; Based on the candidate outputs and corresponding scalar rewards, the relative advantage between groups is calculated within the group, and the relative strategy between groups is updated by weighting based on the relative advantage between groups. A supervised fine-tuning checkpoint and a reference model for the supervised fine-tuning checkpoint are determined, and the divergence of the reference model is regularized based on a preset tag index set to bring the group relative strategy closer to the reference model in order to align the reasoning behavior of the multimodal base model; wherein, the tag index set includes tags for the reasoning steps and / or the final answer; Based on the training process of supervised fine-tuning and reinforcement alignment of relative advantages between groups, a multimodal basic model after two-stage reasoning ability training is obtained.

5. The method according to claim 4, characterized in that, The calculation of the corresponding scalar reward for each candidate output also includes: Based on the candidate output and the target text, determine the corresponding first reward signal and second reward signal; A scalar reward is obtained based on the first reward signal and the second reward signal; The determination of the corresponding first reward signal includes: Calculate the standardized semantic similarity between the candidate output and the target text, and determine the first reward signal based on the standardized semantic similarity.

6. The method according to claim 1, characterized in that, The standardized inference results generated based on the trained multimodal base model include: For each target step, the corresponding maintenance status is determined, and a corresponding decision action is generated based on the maintenance status; wherein, generating an action means generating the decision output for the current step, and the decision action is a reasoning statement or a tool call; Based on the decision-making action, the corresponding decision trajectory is obtained, and the relative strategy of the group is adjusted based on the decision trajectory to maximize the expected cumulative clinical benefit, so as to obtain the reasoning result.

7. The method according to claim 1, characterized in that, The method further includes: Dialogue is conducted through the human-computer interaction interface provided by the patient-side agent, and patient information is reported. The patient information is processed by a lightweight encoder aligned with the multimodal base model to obtain a corresponding 24 / 7 follow-up plan, and a corresponding follow-up schedule is generated and displayed based on the 24 / 7 follow-up plan.

8. A diagnostic and treatment device for patients with gastrointestinal stromal tumors based on multi-agent systems, characterized in that, The device includes: An acquisition module is used to acquire multimodal patient data and acquire a multimodal basic model based on the multimodal patient data; wherein the multimodal patient data includes at least text and visual data; The training module is used to supervise and fine-tune the multimodal base model and perform reinforcement alignment using a preset group relative strategy optimization algorithm to obtain a multimodal base model after two-stage reasoning ability training; wherein, the trained multimodal base model generates expert reasoning trajectories and clinically effective answers; The inference module is used to generate standardized inference results based on the trained multimodal base model; the inference results represent the treatment plan for the patient.

9. An electronic device, characterized in that, include: The device includes a processor, a memory, and a bus. The memory stores machine-readable instructions executable by the processor. When the electronic device is running, the processor communicates with the memory via the bus. When the machine-readable instructions are executed by the processor, they perform the steps of the multi-agent-based method for diagnosing and treating patients with gastrointestinal stromal tumors as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, performs the steps of the multi-agent-based method for diagnosing and treating patients with gastrointestinal stromal tumors as described in any one of claims 1 to 7.

Images