A Cross-Modal Reasoning Method and System for Multi-Task Expert Models in Medical Imaging

Abstract

This disclosure provides a cross-modal inference method and system for multi-task expert models in medical images, relating to the fields of artificial intelligence and medical information technology. The method includes: acquiring medical images and text data, and preprocessing them; allocating the preprocessed image and text data to embedded expert models adapted to the corresponding tasks via a multi-task expert controller, and outputting the inference result for the corresponding task; selecting the corresponding embedded expert model according to the task type, and the multi-task expert controller performing calculations to obtain the activation weight vector for each expert; using a Control-K sparse gating strategy to filter the activation weight vectors, obtaining sparse activation weight vectors; selecting and activating the corresponding embedded expert model based on the sparse activation weights, and fine-tuning the expert incremental parameters to achieve dynamic adaptation for the current task. This disclosure can dynamically select expert paths, improving the performance of different medical tasks.

Description

Technical Field

[0001] This disclosure relates to the fields of artificial intelligence and medical information technology, specifically to a cross-modal reasoning method and system for multi-task expert models used in medical images. Background Technology

[0002] The statements in this section are merely background information relating to this disclosure and do not necessarily constitute prior art.

[0003] In recent years, large-scale multimodal AI models (such as vision-language models) have shown significant potential in medical image analysis, clinical text understanding, and cross-modal collaborative diagnosis. However, in medical applications, existing technologies still face the following shortcomings:

[0004] (1) Huge computational resource consumption: The number of parameters in medical large models is huge, usually reaching billions, and complete fine-tuning requires a lot of computational resources and GPU memory;

[0005] (2) High task heterogeneity: Medical tasks are diverse, including classification, detection, segmentation, lesion description, report generation, etc. The coexistence of multiple tasks makes it difficult for the model to be uniformly adapted.

[0006] (3) Uneven data distribution and privacy barriers: The data distribution of different medical institutions varies greatly, and medical data is highly sensitive. The original images or texts cannot be shared between institutions, which makes model transfer difficult.

[0007] (4) Limitations of existing PEFT methods: LoRA and other parameter fine-tuning techniques have limited effectiveness in multimodal medical tasks and are unstable in image-text collaboration tasks;

[0008] (5) Lack of collaborative architecture: Existing methods mostly use large or small models independently, without effectively integrating the advantages of "strong representation of large models" and "high efficiency of expert models";

[0009] (6) High cost of cross-institutional migration: Traditional methods require fine-tuning in the new institution or sharing the full model weights, which is both time-consuming and violates privacy protection principles. Summary of the Invention

[0010] To address the aforementioned issues, this disclosure proposes a cross-modal reasoning method and system for multi-task expert models in medical images. By constructing a "large model-expert model collaborative architecture" and a "multi-task expert controller," it achieves efficient parameter tuning and deployment with task adaptability and privacy security while freezing the backbone of a general medical large model. This enables multi-task collaborative optimization processing of medical images, collaboratively optimizing heterogeneous tasks such as classification, segmentation, and generation under a unified objective, while balancing task balance and shared knowledge transfer.

[0011] According to some embodiments, the present disclosure adopts the following technical solutions:

[0012] Cross-modal reasoning methods for multi-task expert models of medical images include:

[0013] Acquire medical images and text data for specific task types and preprocess them;

[0014] The preprocessed image and text data are distributed to the embedded expert model adapted to the corresponding task through the multi-task expert controller, and the inference result of the corresponding task is output.

[0015] The adaptation and optimization process of the embedded expert model includes: identifying the task type, selecting the corresponding embedded expert model according to the task type, and extracting modality contribution and lesion prior features by processing image and text features; fusing the task type embedding, modality contribution, and lesion prior features to obtain a context vector; extracting the visual-text fusion features of the image and text, fusing the visual-text fusion features and context vector and inputting them into the multi-task expert controller to perform calculations to obtain the activation weight vector of each expert; using the Control-K sparse gating strategy to filter the activation weight vector to obtain a sparse activation weight vector; selecting and activating the corresponding embedded expert model according to the sparse activation weight, and fine-tuning the expert incremental parameters to achieve dynamic adaptation for the current task.

[0016] According to some embodiments, the present disclosure adopts the following technical solutions:

[0017] A cross-modal reasoning system for multi-task expert models of medical images includes:

[0018] The data acquisition module is used to acquire medical images and text data for specific task types and to preprocess them;

[0019] The multi-task inference module is used to distribute the preprocessed image and text data to the embedded expert model adapted to the corresponding task through the multi-task expert controller, and output the inference result of the corresponding task.

[0020] The adaptation and optimization process of the embedded expert model includes: identifying the task type, selecting the corresponding embedded expert model according to the task type, and extracting modality contribution and lesion prior features by processing image and text features; fusing the task type embedding, modality contribution, and lesion prior features to obtain a context vector; extracting the visual-text fusion features of the image and text, fusing the visual-text fusion features and context vector and inputting them into the multi-task expert controller to perform calculations to obtain the activation weight vector of each expert; using the Control-K sparse gating strategy to filter the activation weight vector to obtain a sparse activation weight vector; selecting and activating the corresponding embedded expert model according to the sparse activation weight, and fine-tuning the expert incremental parameters to achieve dynamic adaptation for the current task.

[0021] According to some embodiments, the present disclosure adopts the following technical solutions:

[0022] A computer program product includes a computer program that, when executed by a processor, implements the cross-modal reasoning method for a multi-task expert model for medical images.

[0023] According to some embodiments, the present disclosure adopts the following technical solutions:

[0024] A non-transitory computer-readable storage medium is provided for storing computer instructions, which, when executed by a processor, implement the cross-modal reasoning method for a multi-task expert model for medical images.

[0025] According to some embodiments, the present disclosure adopts the following technical solutions:

[0026] An electronic device includes a processor, a memory, and a computer program; wherein the processor is connected to the memory, the computer program is stored in the memory, and when the electronic device is running, the processor executes the computer program stored in the memory to cause the electronic device to execute the cross-modal reasoning method for implementing the multi-task expert model for medical images.

[0027] Compared with the prior art, the beneficial effects of this disclosure are as follows:

[0028] The cross-modal reasoning method for multi-task expert models for medical images disclosed herein achieves efficient parameter tuning and deployment, task adaptation, and privacy-preserving fine-tuning by constructing a "large model-expert model collaborative architecture" and a "multi-task expert controller" while freezing the backbone of a general medical large model.

[0029] This disclosed cross-modal inference method for multi-task expert models in medical images deploys multiple lightweight expert models, each tailored to a specific medical task or modality combination, as embedded expert models within the key network layers of a larger model, forming a structured adaptation module. This design "internalizes" the functionality of the expert models as components of the larger model, rather than as independent parallel pathways, thereby preserving the task-aware capabilities of the smaller models while avoiding the double computational overhead of the inference phase.

[0030] This disclosure presents a cross-modal reasoning method for multi-task expert models in medical images. It introduces a unified multi-task expert controller, dynamically generating activation weights for each expert based on contextual information extracted from the embedded expert model, including task type, modal contribution, and lesion priors, thus achieving data-driven expert path selection. This mechanism supports sparse activation, mitigating optimization conflicts between multiple tasks and enhancing the model's adaptability to heterogeneous tasks.

[0031] This disclosed cross-modal inference method for multi-task expert models in medical images trains only the embedded expert modules and controller parameters, while freezing the remaining large model parameters throughout the training process. The total number of expert parameters is constrained by a shared low-rank budget. A multi-task joint training strategy is employed to collaboratively optimize heterogeneous tasks such as classification, segmentation, and generation under a unified objective, balancing task balance and shared knowledge transfer. During model migration, only activated expert parameter fragments and corresponding controller configurations need to be exchanged; the original medical data, complete model weights, or expert model structure do not need to be transferred. This meets the security requirement of medical data remaining within its domain, significantly reducing the adaptation cost and deployment threshold for new institutions.

[0032] This disclosure presents a cross-modal inference method for multi-task expert models in medical images. It automatically selects the optimal expert path for different medical tasks, achieving performance improvements across multiple modal tasks including CT, MRI, ultrasound, and pathology. This method only requires exchanging expert weights, eliminating the need to share private data, thus enabling more efficient and secure cross-institutional model migration. By using a smaller model to filter tasks and generate hints, the larger model only processes necessary paths, improving inference speed. This method can be combined with any large model architecture (LLaMA, Qwen, SAM, MedViT, etc.) for rapid adaptation to medical scenarios. Attached Figure Description

[0033] The accompanying drawings, which form part of this disclosure, are used to provide a further understanding of this disclosure. The illustrative embodiments of this disclosure and their descriptions are used to explain this disclosure and do not constitute an undue limitation of this disclosure.

[0034] Figure 1 This is a schematic flowchart of a cross-modal reasoning method for a multi-task expert model for medical images according to an embodiment of this disclosure;

[0035] Figure 2 This is a schematic diagram of the structure of the multi-task expert controller (MEC) according to an embodiment of this disclosure. Detailed Implementation

[0036] The present disclosure will be further described below with reference to the accompanying drawings and embodiments.

[0037] It should be noted that the following detailed descriptions are illustrative and intended to provide further explanation of this disclosure. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure pertains.

[0038] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the exemplary embodiments according to this disclosure. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms “comprising” and / or “including” are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0039] Example 1

[0040] One embodiment of this disclosure provides a cross-modal reasoning method for a multi-task expert model of medical images, the method steps of which include:

[0041] Step 1: Acquire medical images and text data for the specific task type, and preprocess them;

[0042] Step 2: Distribute the preprocessed image and text data to the embedded expert model adapted to the corresponding task through the multi-task expert controller, and output the inference result of the corresponding task.

[0043] The adaptation and optimization process of the embedded expert model includes: identifying the task type, selecting the corresponding embedded expert model according to the task type, and extracting modality contribution and lesion prior features by processing image and text features; fusing the task type embedding, modality contribution, and lesion prior features to obtain a context vector; extracting the visual-text fusion features of the image and text, fusing the visual-text fusion features and context vector and inputting them into the multi-task expert controller to perform calculations to obtain the activation weight vector of each expert; using the Control-K sparse gating strategy to filter the activation weight vector to obtain a sparse activation weight vector; selecting and activating the corresponding embedded expert model according to the sparse activation weight, and fine-tuning the expert incremental parameters to achieve dynamic adaptation for the current task.

[0044] As one embodiment, this disclosed method for cross-modal inference of multi-task expert models for medical images aims to address core issues in medical scenarios, such as high model fine-tuning costs, strong task heterogeneity, difficulty in cross-institutional transfer, and weak multimodal collaboration. Multiple lightweight expert models, tailored to specific medical tasks or modal combinations, are deployed as embedded expert models within key network layers of a larger model, forming a structured adaptation module. This design "internalizes" the functionality of the expert models as components of the larger model, rather than as independent parallel pathways, thereby preserving the task-aware capabilities of smaller models while avoiding double computational overhead during the inference phase. A unified multi-task expert controller is introduced, dynamically generating activation weights for each expert based on contextual information such as task type, modal contribution, and lesion prior extracted from the embedded expert models, achieving data-driven expert path selection. This mechanism supports sparse activation, mitigating optimization conflicts between multiple tasks and improving the model's adaptability to heterogeneous tasks. This method, by constructing a "large model-expert model collaborative architecture" and a "multi-task expert controller," achieves efficient parameter tuning and deployment, task adaptation, and privacy-preserving fine-tuning while freezing the backbone of a general medical large model. The specific implementation process includes the following:

[0045] Step 1: Acquire medical images and text data for the specific task type, and preprocess them;

[0046] Retrieve CT, MRI, ultrasound, pathology images, and electronic medical record texts from medical datasets;

[0047] Perform denoising and normalization preprocessing operations on images and text.

[0048] Step 2: Distribute the preprocessed image and text data to the embedded expert model adapted to the corresponding task through a multi-task expert controller, and output the inference result of the corresponding task. Specific content includes:

[0049] (1) Construct a large-scale multimodal basic medical model framework;

[0050] The multimodal basic medical big model includes a visual encoder, a text encoder, a Transformer cross-modal fusion layer, and an output layer. It uses a visual encoder (such as ViT, EfficientNet) and a text encoder (such as an LLaMA-like architecture) to construct a multimodal joint feature space for processing CT, MRI, ultrasound, pathology images, and electronic medical record text.

[0051] Visual branch: Use ViT-B / 16 or MedSAM encoders to extract image features;

[0052] Text branch: Text features are extracted using an LLaMA-2-7B or Qwen-7B text encoder.

[0053] Image features and text features are aligned and fused in a cross-modal fusion layer to obtain visual-text fused features.

[0054] (2) An embedded expert model and a multi-task expert controller MEC-Adapter are introduced into the multimodal basic medical large model framework;

[0055] Specifically, embedded expert models are deployed as embedded experts in several Transformer layers of the large model. Each expert model is a "miniature LoRA adapter" for a specific medical task / organ / modality combination, hence called a "MEC-Adapter expert model," and is uniformly scheduled through a task controller. The embedded expert models are multiple lightweight expert models for specific medical tasks or modality combinations, deployed as embedded experts in the key network layers of the large model to form structured adaptation modules. This embeds the functionality of the embedded expert models into the large model, transforming them into components rather than independent parallel pathways, enabling multi-task perception.

[0056] The embedded expert models include modal expert models, namely CT expert, MRI expert, and ultrasound expert, used to process different data modalities; task expert models, namely report generation expert and image segmentation expert, corresponding to different medical tasks; and organ expert models, namely liver expert and fundus expert. The multi-task expert controller determines which embedded expert models to use. A LoRA / Adapter weight fine-tuning module is added to the local network layer for parallel processing, and the processing results are weighted and fused within the same layer.

[0057] Furthermore, the multi-task expert controller is embedded as follows: a MoE module is embedded in the feedforward network (FFN) or attention layer of the Transformer block, and each expert model is a pair of low-rank matrices (LoRA form), with the specific structure as follows:

[0058] The number of experts is N (e.g., N=8), and each expert corresponds to a specific task or modality combination; expert parameters include each expert's... Composed of two low-rank matrices and constitute;

[0059] in, i Indicates the first i An index of experts; No. i LoRA parameter increments provided by an expert; The low-rank projection matrix represents the input features. Descended to r dimension; For low-rank projection matrices, the features are transferred from... r Ascending back ; Input feature dimension; This is the output feature dimension.

[0060] As one embodiment, the adaptation optimization process of the embedded expert model includes: identifying the task type, selecting the corresponding embedded expert model according to the task type, and extracting modality contribution and lesion prior features by processing image and text features; fusing the task type embedding, modality contribution, and lesion prior features to obtain a context vector; extracting the visual-text fusion features of the image and text, fusing the visual-text fusion features and context vector, and inputting them into the multi-task expert controller to perform calculations to obtain the activation weight vector of each expert; using the Control-K sparse gating strategy to filter the activation weight vectors to obtain sparse activation weight vectors; selecting and activating the corresponding embedded expert model according to the sparse activation weights, and fine-tuning the expert incremental parameters to achieve dynamic adaptation for the current task. Specifically, this includes:

[0061] 1) The expert model output task type embedding in medicine includes task-related features such as disease category, organ category, and modality type.

[0062] 2) Simultaneously, modal contribution is extracted from the image branch to represent the importance of different modalities such as CT / MRI / ultrasound to the current task; lesion prior features are extracted from the lesion detection or attention module to reflect the strength of the correlation between regions in the image and the disease. The task type embedding, modal contribution, and lesion prior features are concatenated or linearly fused to obtain the context vector.

[0063] 3) Linearly combine or concatenate the visual-text fusion features with the context vector to form the input representation of the controller network. The input to the multi-task expert controller is the task ID or the task embedding output by the expert model. The context vector, fused from modal contribution and lesion prior features, along with visual-text fusion features, is mapped onto the above input representation to output an unnormalized expert score vector. A Softmax operation is performed on the unnormalized score vector to obtain the activation weight vector for each expert, used to measure the relevance of the expert to the current task. Expert weights are generated through linear transformation and Softmax.

[0064]

[0065] in, This is the parameter matrix of the controller network; Task embedding for output of medical mini-models.

[0066] 4) To avoid all experts being activated at the same time and introducing interference, a Control-K sparse gating strategy is adopted to select the top K experts with the largest weights from the activation weight vector; the weights of the unselected experts are reset to zero and renormalized to obtain a sparse activation weight vector.

[0067] Specifically, the sparse controller can be selected as follows: To mitigate optimization interference between tasks, a Control-K sparse controller is used to activate the one with the largest weight. One expert:

[0068] ;

[0069] in, Indicates the number of experts.

[0070] The complete expert forward communication process is as follows:

[0071]

[0072] in, The dimension is The input vector; The dimension is The output vector; This represents the rank of a trainable low-rank matrix, which determines the number of trainable parameters. These are constant hyperparameters used to... Optimize; This is a low-rank matrix projected downwards in LoRA; This is a low-rank matrix projected upwards in LoRA; These are the original parameters for freezing.

[0073] 5) Based on the obtained sparse activation weights, select and activate the corresponding expert embedding model. The outputs of the activated experts are weighted and summed to obtain the expert incremental parameters. These expert incremental parameters are then injected into the original Transformer layer via a LoRA structure to achieve dynamic adaptation for the current task.

[0074] Training is performed only on the embedded expert module and controller parameters, while all other large model parameters are frozen throughout the process. The total number of expert parameters is constrained by a shared low-rank budget. A multi-task joint training strategy is employed to collaboratively optimize heterogeneous tasks such as classification, segmentation, and generation under a unified objective, balancing task balance and shared knowledge transfer. During model migration, only activated expert parameter fragments and corresponding controller configurations need to be exchanged; the original medical data, complete model weights, or expert model structure do not need to be transferred. This meets the security requirement of medical data remaining within its domain, significantly reducing the adaptation cost and deployment threshold for new institutions. Specifically, this includes:

[0075] Trainable parameters are introduced only in the MEC expert module and controller network, while the parameters of the rest of the large model remain frozen, enabling efficient fine-tuning of parameters:

[0076] a. Trainable parameters: MECLoRA parameters Ai, Bii=1N, task-driven controller functions E, WT;

[0077] b. Total number of parameters: Same as standard LoRA, which is r×din+dout;

[0078] c. Training strategy: Randomly sample batches from all tasks and perform joint optimization to avoid the impact of data imbalance between tasks.

[0079] This disclosure selects based on the task category, modality type, and lesion-related features output by the small model, through a controller network:

[0080] a. Expert Activation: Select the expert combination that best matches the current task;

[0081] b. LoRA Enable Switch: Determines whether LoRA is enabled for certain layers based on task importance;

[0082] c. Modality fusion ratio: The fusion weights of image and text features are dynamically adjusted based on the modality contribution evaluated by the expert model.

[0083] (3) This disclosure adopts a combination of loss functions based on task differences, including classification loss, segmentation loss, attention region consistency loss, text generation loss, etc., and improves the generalization ability of the model through dynamic weight balancing.

[0084] a. Classification tasks: ;

[0085] b. Segmentation task: Combined segmentation loss ;

[0086] c. Generate task: ;

[0087] In summary, the combined losses from multiple tasks are as follows:

[0088]

[0089] in, The contrastive loss is used to align the representations of the large model and the expert model. , These are the model's predicted probabilities for large and small sample categories, respectively. , For class balance coefficients; Pixel-level cross-entropy loss; parameters Control the weight of each loss separately.

[0090] Example 2

[0091] One embodiment of this disclosure provides a cross-modal reasoning system for a multi-task expert model of medical images, comprising:

[0092] The data acquisition module is used to acquire medical images and text data for specific task types and to preprocess them;

[0093] The multi-task inference module is used to distribute the preprocessed image and text data to the embedded expert model adapted to the corresponding task through the multi-task expert controller, and output the inference result of the corresponding task.

[0094] The adaptation and optimization process of the embedded expert model includes: identifying the task type, selecting the corresponding embedded expert model according to the task type, and extracting modality contribution and lesion prior features by processing image and text features; fusing the task type embedding, modality contribution, and lesion prior features to obtain a context vector; extracting the visual-text fusion features of the image and text, fusing the visual-text fusion features and context vector and inputting them into the multi-task expert controller to perform calculations to obtain the activation weight vector of each expert; using the Control-K sparse gating strategy to filter the activation weight vector to obtain a sparse activation weight vector; selecting and activating the corresponding embedded expert model according to the sparse activation weight, and fine-tuning the expert incremental parameters to achieve dynamic adaptation for the current task.

[0095] Example 3

[0096] One embodiment of this disclosure provides a computer program product, including a computer program that, when executed by a processor, implements the cross-modal reasoning method for a multi-task expert model for medical images.

[0097] Example 4

[0098] One embodiment of this disclosure provides a non-transitory computer-readable storage medium for storing computer instructions, which, when executed by a processor, implement the cross-modal reasoning method for a multi-task expert model for medical images.

[0099] Example 5

[0100] One embodiment of this disclosure provides an electronic device, including: a processor, a memory, and a computer program; wherein the processor is connected to the memory, the computer program is stored in the memory, and when the electronic device is running, the processor executes the computer program stored in the memory to cause the electronic device to execute the cross-modal reasoning method for a multi-task expert model for medical images.

[0101] This disclosure is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create a machine for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0102] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0103] While the specific embodiments of this disclosure have been described above in conjunction with the accompanying drawings, this is not intended to limit the scope of protection of this disclosure. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of this disclosure are still within the scope of protection of this disclosure.

Claims

1. A cross-modal reasoning method for multi-task expert models of medical images, characterized in that, include: Acquire medical images and text data for specific task types and preprocess them; The preprocessed image and text data are distributed to the embedded expert model adapted to the corresponding task through the multi-task expert controller, and the inference result of the corresponding task is output. The embedded expert model consists of multiple lightweight expert models oriented towards specific medical tasks or modal combinations. These models are deployed in the key network layers of the large model as embedded experts, forming a structured adaptation module. This embeds the functions of the embedded expert models into the components of the large model, rather than as independent parallel pathways, enabling multi-task perception. The adaptation and optimization process of the embedded expert model includes: identifying the task type, selecting the corresponding embedded expert model according to the task type, extracting modality contribution and lesion prior features by processing image and text features; fusing the task type embedding, modality contribution, and lesion prior features to obtain a context vector; extracting the visual-text fusion features of the image and text, fusing the visual-text fusion features and context vector and inputting them into the multi-task expert controller, and performing calculations to obtain the activation weight vector of each expert; using the Control-K sparse gating strategy to filter the activation weight vector to obtain a sparse activation weight vector; selecting and activating the corresponding embedded expert model according to the sparse activation weight, and fine-tuning the expert incremental parameters to achieve dynamic adaptation for the current task. The embedded expert models include modal expert models, namely CT expert, MRI expert, and ultrasound expert, used to process different data modalities; task expert models, namely report generation expert and image segmentation expert, corresponding to different medical tasks; and organ expert models, namely liver expert and fundus expert. The multi-task expert controller determines which embedded expert models to use. A LoRA / Adapter weight fine-tuning module is added to the local network layer for parallel processing, and the processing results are weighted and fused within the same layer. Task type embedding, modality contribution, and lesion prior features are concatenated or linearly fused to obtain a context vector; visual-text fusion features are linearly combined or concatenated with the context vector to form the input representation of the multi-task expert controller; the multi-task expert controller maps the input representation and outputs an unnormalized expert score vector. A Softmax operation is performed on the unnormalized expert score vector to obtain the activation weight vector for each expert. Using the Control-K sparse gating strategy, the top K experts with the largest weights are selected from the activation weight vectors. The weights of the unselected experts are reset to zero and renormalized to obtain a sparse activation weight vector. Based on the obtained sparse activation weights, the corresponding embedded expert models are selected and activated. The outputs of the activated embedded expert models are weighted and summed according to their weights to obtain the expert incremental parameters. The expert incremental parameters are injected into the original Transformer layer through the LoRA structure to achieve dynamic adaptation for the current task.

2. The cross-modal reasoning method for multi-task expert models of medical images as described in claim 1, characterized in that, The identification task type is determined by selecting a corresponding embedded expert model based on the task type. This model extracts modality contribution and prior lesion features by processing image and text features, including: The medical large model is used to identify and output task type embeddings, which include multi-task related features of disease category, organ category, and modality type. At the same time, modality contribution is extracted on the image side to represent the importance of various modalities of CT / MRI / ultrasound to the current task. Lesion prior features are extracted in the lesion detection attention module to reflect the strength of the correlation between regions in the image and the disease.

3. A cross-modal reasoning system for a multi-task expert model for medical images, employing the cross-modal reasoning method for a multi-task expert model for medical images as described in any one of claims 1-2, characterized in that, include: The data acquisition module is used to acquire medical images and text data for specific task types and to preprocess them; The multi-task inference module is used to distribute the preprocessed image and text data to the embedded expert model adapted to the corresponding task through the multi-task expert controller, and output the inference result of the corresponding task. The adaptation and optimization process of the embedded expert model includes: identifying the task type, selecting the corresponding embedded expert model according to the task type, and extracting modality contribution and lesion prior features by processing image and text features; fusing the task type embedding, modality contribution, and lesion prior features to obtain a context vector; extracting the visual-text fusion features of the image and text, fusing the visual-text fusion features and context vector and inputting them into the multi-task expert controller to perform calculations to obtain the activation weight vector of each expert; using the Control-K sparse gating strategy to filter the activation weight vector to obtain a sparse activation weight vector; selecting and activating the corresponding embedded expert model according to the sparse activation weight, and fine-tuning the expert incremental parameters to achieve dynamic adaptation for the current task.

4. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the cross-modal reasoning method for a multi-task expert model for medical images as described in any one of claims 1-2.

5. A non-transitory computer-readable storage medium, characterized in that, The non-transitory computer-readable storage medium is used to store computer instructions, which, when executed by a processor, implement the cross-modal reasoning method for a multi-task expert model for medical images as described in any one of claims 1-2.

6. An electronic device, characterized in that, include: The device includes a processor, a memory, and a computer program; wherein the processor is connected to the memory, the computer program is stored in the memory, and when the electronic device is running, the processor executes the computer program stored in the memory to cause the electronic device to execute a cross-modal reasoning method for a multi-task expert model for medical images as described in any one of claims 1-2.

Images