Multi-task parameter adaptation reasoning system and method, electronic device, program product and storage medium

Abstract

A multi-task parameter adaptation reasoning system and method, an electronic device, a program product and a storage medium can eliminate the need for repeated training, and balance adaptation efficiency and engineering deployability. The system comprises: an input acquisition module configured to acquire task description text and task visual examples, and determine a target model and a target set of injectable modules thereof; a prompt generation module configured to generate structured multi-modal prompt content; a multi-modal fusion module configured to fuse and encode the structured multi-modal prompt content to generate a fused semantic representation; a conditional information construction module configured to construct a model parameter statistical quantity vector and a position encoding set; an adaptation parameter generation module configured to generate a low-rank adaptation parameter set and an adaptation configuration description based on the fused semantic representation, the model parameter statistical quantity vector and the position encoding set; and a parameter injection execution module configured to inject the low-rank adaptation parameter set into the target model according to the adaptation configuration description.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence, and more specifically to a multi-task parameter adaptation inference system and method, electronic device, program product and storage medium. Background Technology

[0002] As is well known, deep learning models have made significant progress in tasks such as visual understanding and multimodal analysis. In practical engineering applications, it is necessary to continuously adapt models, switch tasks, and iterate updates under conditions of multi-task, multi-domain, and multi-version parallelism. The system must not only meet the requirements of inference accuracy and real-time performance, but also have the efficiency of multi-task switching and engineering deployability.

[0003] Current technologies have their own technical limitations in terms of rapid multi-task adaptation and low-cost deployment. For example, the task-by-task independent training and deployment scheme trains and deploys an independent model version for each task or scenario. However, while this approach offers the advantage of deep optimization for a single task, it is prone to significantly high training, storage, and maintenance costs, and extends the model upgrade cycle, making it difficult to meet the engineering requirements of rapid delivery and continuous iteration.

[0004] As another existing solution, efficient parameter adaptation technology introduces a small number of trainable parameters into the base model, keeping the main parameters of the base model unchanged and only updating a few additional parameters to achieve task migration or performance improvement. However, while this type of solution reduces training costs to some extent, it still faces the problem of a lack of closed-loop production and management of adaptation parameters in scenarios of rapid switching between multiple tasks and large-scale deployment. When the number of tasks increases, the frequency of task switching rises, or the target device resources are limited, it is difficult to ensure the adaptation effect while achieving rapid generation, rapid loading, and sustainable iteration.

[0005] Furthermore, as another existing solution, pure natural language task conditional schemes rely solely on natural language descriptions as task condition inputs. However, while such schemes offer the advantage of flexible input, they are prone to instability in the adaptation process due to differences in expression, ambiguity, or incomplete information. Moreover, in some tasks, there are visual sample information that can more directly characterize task features, but there is a lack of a unified organization and fusion method, making it difficult to form a stable, consistent, and reusable multimodal task conditional representation.

[0006] In recent years, multimodal large model technology has been gradually applied to the field of visual task adaptation. Its semantic understanding and cross-modal reasoning capabilities can improve the performance of task condition representation. However, existing solutions directly use free natural language as task conditions, which inherently involves uncertainties in expression structure, word granularity, and information order. This can easily lead to ambiguity in adaptation output and instability in training signals, making it impossible to form a stable and alignable task condition representation. Furthermore, in multi-task switching scenarios, the generation, management, and iteration of adaptation parameters lack a closed-loop mechanism, making it difficult to reuse and continuously optimize accumulated task adaptation experience. In addition, the generation of adaptation parameters for lightweight models needs to consider task semantic information, target module structural information, and parameter state information simultaneously, but existing solutions lack a mechanism to unify and integrate the above-mentioned multi-source condition information. Summary of the Invention

[0007] This disclosure is based on the above circumstances and aims to provide a multi-task parameter adaptation inference system and method, electronic device, program product and storage medium that can realize the structured expression and unified semantic representation of multi-task conditions, eliminate the need for repetitive training with low-rank adaptation parameters, build a collaborative link between task conditions and parameter generation through multimodal prompts, and take into account both adaptation efficiency and engineering deployability.

[0008] This disclosure provides a multi-task parameter adaptation inference system, including: The input acquisition module is configured to acquire task description text and visual examples related to the target task, and to determine the target model and its set of injectable modules; The prompt generation module is configured to receive the task description text and the task visual example, perform template matching on the task description text and the task visual example based on a preset prompt word template library, and generate and output structured multimodal prompt content. The multimodal fusion module is configured to receive the structured multimodal prompt content, perform fusion encoding on the structured multimodal prompt content, and generate and output a fused semantic representation; The condition information construction module is configured to construct a model parameter statistics vector and a location encoding set based on the parameter state of the target model and the target injectable module set; The adaptation parameter generation module is configured to receive the fused semantic representation, the model parameter statistics vector, and the position encoding set, and generate and output a low-rank adaptation parameter set and an adaptation configuration description that correspond one-to-one with the target injectable module set; and The parameter injection execution module is configured to inject the low-rank adaptation parameter set into the target injectable module set of the target model according to the adaptation configuration description, so that the injected target model can be used to perform task inference to generate and output task output results.

[0009] According to the multi-task parameter adaptation and inference system described in this technical solution, the system acquires task description text and task visual examples through an input acquisition module, outputs structured multimodal prompt content through a prompt generation module, generates fused semantic representations through a multimodal fusion module, constructs model parameter statistical vectors and position encoding sets through a condition information construction module, generates low-rank adaptation parameter sets and adaptation configuration descriptions through an adaptation parameter generation module, and injects parameters and executes inference through a parameter injection and execution module. This system can construct a collaborative link from task condition acquisition to parameter injection and execution, realize the structured expression and unified semantic representation of multi-task conditions, and improve the accuracy of adaptation parameter generation and the efficiency of multi-task switching.

[0010] Optionally, the multi-task parameter adaptation inference system further includes a historical adaptation library management module, which is configured to manage historical task adaptation information. The historical task adaptation information includes at least the task identifier of the target task, the target model identifier of the target model, and a set of low-rank adaptation parameters associated with the task identifier and the target model identifier.

[0011] According to the multi-task parameter adaptation and inference system described in this technical solution, the historical adaptation library management module stores and retrieves historical task adaptation information, which enables the reuse of accumulated adaptation parameter experience, improves the adaptation efficiency of repetitive tasks and the traceability of parameter management, and enhances the system's continuous iteration capability.

[0012] Optionally, the multi-task parameter adaptation inference system further includes a closed-loop iterative control module, which is configured to: receive a performance index summary corresponding to the task output result; when the performance index summary meets a preset trigger condition, cause the historical adaptation library management module to write or update the historical task adaptation information; and trigger training or tuning of the adaptation parameter generation module.

[0013] According to the multi-task parameter adaptation inference system described in this technical solution, the closed-loop iterative control module triggers the update of the historical adaptation library and model tuning based on the performance index summary, which enables the adaptation parameter generation capability to be continuously optimized under new tasks and new scenarios, forming a closed-loop mechanism of "generation-verification-accumulation-iteration", thereby improving the system's generalization capability and long-term maintainability.

[0014] Optionally, the structured multimodal prompt content includes at least a structured text field and a structured visual field. The structured text field is generated by extracting and normalizing fields from the descriptive text of the target task, and the structured visual field is generated by processing the visual example of the task and its meta-information according to a template selected from the prompt word template library.

[0015] According to the multi-task parameter adaptation and inference system described in this technical solution, the task description text and task visual examples are organized into structured text fields and structured visual fields respectively by the prompt generation module. This can reduce ambiguity and inconsistent formatting caused by free input, and improve the stability and consistency of the system under different users and different description methods.

[0016] Optionally, the condition information construction module is further configured to construct the position code set based on the module type identifier and layer index identifier of each target module in the target injectable module set. Each position code in the position code set is generated by concatenating the module type embedding and the layer index embedding, wherein the module type embedding is a vector representation of the model type identifier, and the layer index embedding is a vector representation of the layer index identifier.

[0017] According to the multi-task parameter adaptation inference system described in this technical solution, the position encoding set is constructed by the condition information construction module based on the module type identifier and the layer index identifier. This enables the adaptation parameter generation module to generate differentiated adaptation parameters for different target modules, improves the matching between the generated low-rank adaptation parameters and the target modules, and reduces the risk of adaptation failure due to module mismatch.

[0018] Optionally, the low-rank adaptation parameter set includes low-rank parameter pairs and configuration items for rank and scaling factor. The low-rank parameter pairs correspond to each target module in the target injectable module set, and the adaptation configuration description includes at least the module type identifier and layer index identifier of the target module.

[0019] According to the multi-task parameter adaptation inference system described in this technical solution, the low-rank adaptation parameter set is made to correspond one-to-one with the target module by the adaptation parameter generation module, including low-rank parameter pairs and configuration items. This enables the adaptation parameters to be stored, loaded and managed in the form of independent parameter packages, supporting capability updates and delivery without changing the main parameters of the target model, and improving the engineering maintainability of multi-terminal deployment.

[0020] Optionally, the parameter injection execution module is further configured to encapsulate the low-rank adaptation parameter set and the adaptation configuration description into a parameter package.

[0021] Furthermore, this disclosure also provides a multi-task parameter adaptation inference method, including: The first step is to obtain the task description text and visual examples related to the target task, and to determine the target model and its set of injectable modules. The second step involves performing template matching on the task description text and the task visual example based on a preset prompt word template library, generating and outputting structured multimodal prompt content. The third step is to fuse and encode the structured multimodal prompt content to generate and output a fused semantic representation; The fourth step is to construct a model parameter statistics vector and a location encoding set based on the parameter state of the target model and the target injectable module set. The fifth step involves using the fused semantic representation, the model parameter statistics vector, and the position encoding set as conditional inputs to generate and output a low-rank adaptation parameter set and adaptation configuration description that correspond one-to-one with the target injectable module set; and The sixth step is to inject the low-rank adaptation parameter set into the target injectable module set of the target model according to the adaptation configuration description, so that the injected target model can be used to perform task inference to generate and output task output results. Attached Figure Description

[0022] Figure 1 This is a schematic diagram illustrating the main architecture of a multi-task parameter adaptation inference system according to an embodiment of the present disclosure.

[0023] Figure 2 This is a schematic diagram of the main architecture of a first variant of a multi-task parameter adaptation inference system according to an embodiment of the present disclosure.

[0024] Figure 3 This is a schematic diagram of the main architecture of a second variant of the multi-task parameter adaptation inference system according to an embodiment of the present disclosure.

[0025] Figure 4 This is a flowchart illustrating the main steps of a multi-task parameter adaptation reasoning method according to an embodiment of the present disclosure.

[0026] Figure 5 This is a flowchart illustrating the main steps of a first variation of the multi-task parameter adaptation reasoning method according to an embodiment of the present disclosure. Detailed Implementation

[0027] First, refer to Figure 1 The main architecture of the multi-task parameter adaptation inference system P according to one embodiment of the present disclosure will be described.

[0028] Figure 1 A schematic diagram of the main functional architecture of the parameter adaptation inference system P according to one embodiment of this disclosure is shown. Figure 1 As shown, the parameter adaptation reasoning system P mainly includes an input acquisition module N0, a prompt generation module N1, a multimodal fusion module N2, a condition information construction module N3, an adaptation parameter generation module N4, and a parameter injection execution module N5.

[0029] In this embodiment, the input acquisition module N0 functions as a component for acquiring the basic input information required for system operation. Specifically, the input acquisition module N0 is configured to acquire task description text and task visual examples, and determine the target model and its set of injectable modules. The task description text referred to here refers to textual information in free natural language that describes the requirements of the target task. The task description text may include one or more elements such as task type, task objective, detection object, and output format requirements. For example, the task description text might be "Perform defect detection on the surface of electronic components, including scratches, dents, and stains, and output the category and location of each defect." The task visual examples referred to here refer to a small amount of sample image data associated with the target task, which may include auxiliary information such as labels, task metadata, or graphic annotations. The number of task visual examples can be selected within a preset range based on input cost and effect requirements. The target model referred to here refers to a lightweight model with a small number of parameters, low inference overhead, and capable of performing target task inference after injecting low-rank adaptation parameters. The target injectable module set mentioned here refers to a pre-defined set of modules in the target model that can be affected by low-rank adaptable parameters, where each target module corresponds to a specific layer or sub-module in the target model. For example, the target injectable module set may include linear transformation modules, projection modules, or combinations thereof in the target model. The output of the input acquisition module N0 is a standardized input data packet, which includes at least the task description text, a task visual example, target model identification information, and the target injectable module set, and is sent to the corresponding interfaces in the prompt generation module N1, the condition information construction module N3, and the parameter injection execution module N5.

[0030] The prompt generation module N1 is configured to receive the task description text and task visual example output by the input acquisition module N0, and perform template matching and formatted filling on the task description text and task visual example based on the prompt word template library to generate and output structured multimodal prompt content. The prompt word template library mentioned here refers to a preset set of templates containing template identifiers, template slot / field definitions, field type constraints, and formatted output rules. Template matching refers to determining the matching template type in the prompt word template library based on the task description text. Formatted filling refers to, under the given template slot / field definitions, performing slot extraction, field normalization, and semantic normalization on the task description text, and organizing the task visual example and its meta-information according to the template requirements. The structured multimodal prompt content mentioned here refers to a multimodal task condition representation generated after template matching and formatted filling, organized with a normalized field structure, including at least structured text fields and structured visual fields. Structured text fields may include one or more fields such as task type, task objective, detection object, and output format. The structured visual fields can include one or more fields such as visual example image references, example labels, and example metadata. For example, the structured multimodal prompt content could be: "Task type: defect detection; Detection object: surface of electronic components; Defect category: scratches, dents, stains; Output format: category + location; Visual examples: [Example 1: scratch annotation image], [Example 2: dent annotation image]". Optionally, the prompt generation module N1 can be further configured to generate structured prompt content and adapt it based solely on the task description text when task visual examples are unavailable, or to generate structured multimodal prompt content and adapt it based on task visual examples and their metadata when task description text is limited. The prompt generation module N1 can, for example, employ a learning model formed by learning from a pre-defined dataset using machine learning or deep learning techniques. Specifically, a dataset is constructed by using a free natural language task description and a small set of task visual examples as samples, and the structured multimodal prompt content corresponding to each sample as a label. Based on this, the preset models (e.g., neural networks, decision trees, support vector machines, etc.) are trained using these datasets (adjusting the model's structure, parameters, hyperparameters, etc.) to form the prompt generation module N1, which serves as the model after learning.

[0031] The multimodal fusion module N2 is configured to receive structured multimodal prompt content output by the prompt generation module N1, perform fusion encoding on the structured multimodal prompt content, and generate and output a fused semantic representation. The fusion encoding here refers to the process of processing structured text fields and structured visual fields respectively through the text processing submodule and the visual processing submodule, and outputting a unified task semantic representation under the action of the fusion mechanism. The fused semantic representation here refers to the unified task representation obtained after fusion encoding, which carries all semantic information of the structured multimodal prompt content in the form of a sequence or vector, and can serve as the main conditional input for the subsequent low-rank adaptation parameter generation. For example, the fused semantic representation is an embedding sequence output in the form of a fused prompt sequence, the length of which corresponds to the number of fields or the number of field expansion tags. Optionally, the multimodal fusion module N2 is further configured to pool the fused prompt sequence to obtain a fused vector, and use the fused prompt sequence and / or the fused vector as one of the conditional inputs for the subsequent adaptation parameter generation module. Optionally, the multimodal fusion module N2 may also include a task type classification branch unit during the training phase. This branch unit is used to output task type prediction results based on the fused semantic representation to impose supervised constraints. This task type classification branch unit is deactivated or removed after training convergence. The multimodal fusion module N2 can be, for example, a learned model formed by learning from a pre-defined dataset using machine learning or deep learning techniques. Specifically, a dataset is constructed by using structured multimodal prompts as samples and the fused semantic representation corresponding to each sample as a label. Based on this, a pre-defined model (e.g., a neural network, a visual language model, etc.) is trained using these datasets (adjusting the model's structure, parameters, hyperparameters, etc.), thereby forming the multimodal fusion module N2 as a learned model.

[0032] The conditional information construction module N3 is configured to construct a model parameter statistics vector and a location encoding set based on the parameter states of the target model and the set of target injectable modules. The model parameter statistics vector refers to a statistical information vector calculated based on the parameter states of the target model. It can be composed of parameter statistical summaries (e.g., scale, norm, or distribution summaries) of each target module in the target injectable module set, used to characterize the parameter states of the target model. The location encoding set refers to a set of conditional information used to characterize the location and type information of each target module in the target injectable module set. Each location encoding in the location encoding set is generated by concatenating the module type embedding and the layer index embedding, where the module type embedding and the layer index embedding are learnable parameters. As an example, assuming the target injectable module set includes 3 target modules, where the first is a linear transformation module located in the 4th layer, its location encoding is generated by concatenating the module type embedding of "linear transformation" with the layer index embedding of "4th layer". Optionally, the model parameter statistics vector can also be further combined with forward representation statistical summaries from a small number of task visual examples. The output of the condition information construction module N3 is a vector of model parameter statistics and a set of location codes, which is then sent to the adaptation parameter generation module N4.

[0033] The adaptation parameter generation module N4 is configured to receive the fusion semantic representation output by the multimodal fusion module N2, the model parameter statistics vector output by the conditional information construction module N3, and the location encoding set. It combines these inputs to form the generation conditional input, generating and outputting a low-rank adaptation parameter set and an adaptation configuration description that correspond one-to-one with the target injectable module set. The low-rank adaptation parameter set here refers to the parameter set containing low-rank parameter pairs and their configuration items, each corresponding to a target module. Each low-rank parameter pair contains two low-rank matrices, and the configuration items include at least the target module index information and may include rank and scaling factor. The adaptation configuration description here refers to the configuration information used to support subsequent injection and loading, including at least the target module index information, which contains a module type identifier and a layer index identifier. For example, assuming the target injectable module set includes 3 target modules, the adaptation parameter generation module N4 outputs 3 sets of low-rank parameter pairs and corresponding configuration descriptions. Each configuration description records the module type identifier, layer index identifier, rank, scaling factor, and matrix dimension information corresponding to that set of parameters. Optionally, the adaptation parameter generation module N4 can be further configured to consider task semantic information, target module structural information, and parameter state information simultaneously when generating the low-rank adaptation parameter set, thereby improving the matching between the generated result and the target module. The adaptation parameter generation module N4 can, for example, be a learned model formed by learning from a pre-defined dataset using machine learning or deep learning techniques. Specifically, a dataset is constructed by using a set of samples that integrates semantic representations, model parameter statistics vectors, and positional encodings, and by using the low-rank adaptation parameter set corresponding to each sample as a label. Based on this, a pre-defined model (e.g., neural network, hypernetwork, generative model, etc.) is trained using these datasets (adjusting the model's structure, parameters, hyperparameters, etc.), thereby forming the adaptation parameter generation module N4 as the learned model.

[0034] The parameter injection execution module N5 is configured to inject the low-rank adaptation parameter set into the target injectable module set of the target model based on the adaptation configuration description output by the adaptation parameter generation module N4, and then use the injected target model to perform task inference, generating and outputting the task output result. The injection here refers to mapping each low-rank parameter pair in the low-rank adaptation parameter set to the corresponding preset target module in the target model according to the target module index information in the adaptation configuration description, completing the low-rank incremental loading. The injected target model achieves task adaptation through low-rank incremental terms while keeping the main parameters unchanged. The task output result here refers to the output obtained by performing inference on the target task input using the injected target model. For example, the task output result may be a defect detection result, including "Defect type: scratch; Location: (120, 45); Confidence: 0.92". Optionally, the parameter injection execution module N5 is further configured to encapsulate the low-rank adaptation parameter set and the adaptation configuration description into a parameter package, and perform signature verification and integrity verification on the parameter package before injection. The parameter package described here includes a manifest file and parameter data. The manifest file records the task identifier, target model identifier, target module index information, rank, scaling factor, and matrix dimension information. For example, the manifest file of the parameter package might record: "Task identifier: DEF_001; Target model identifier: LVM_v2; Number of modules: 3; Module 1: Type = Linear Transformation, Layer = 4, Rank = 8, Scaling Factor = 0.5; Module 2: Type = Projection, Layer = 7, Rank = 8, Scaling Factor = 0.5; Module 3: Type = Linear Transformation, Layer = 11, Rank = 8, Scaling Factor = 0.5". Optionally, the parameter injection execution module N5 is also configured to complete the task switch by simply replacing the new low-rank adaptation parameter set during task switching, without needing to change the main parameters of the target model.

[0035] According to the parameter adaptation inference system P described in this embodiment, through the mutual collaboration and data interaction among the input acquisition module, prompt generation module, multimodal fusion module, conditional information construction module, adaptation parameter generation module, and parameter injection execution module, the task description text and task visual examples can be uniformly organized into structured multimodal prompt content and generate fused semantic representations, providing a stable and reliable conditional input foundation for subsequent low-rank adaptation parameter generation. Based on this, by combining the fused semantic representation with the model parameter statistical vector and positional encoding set as generation conditions, the generated low-rank adaptation parameters can simultaneously consider task semantic information, target module structural information, and parameter state information, improving the matching between the adaptation parameters and the target module. Furthermore, through the injection and replacement of low-rank adaptation parameters, rapid switching inference across multiple tasks can be achieved without changing the main parameters of the target model, reducing the cost of repeated training and maintenance of multiple version models, and improving the system's adaptation efficiency and engineering deployability under different tasks and deployment scenarios.

[0036] Next, based on the above, referring to Figure 2 The main architecture of the parameter adaptation inference system P1 described in the first variation of the above-described embodiment will be explained. Here, to avoid repetition, only the differences between this first variation and the above-described embodiment will be described.

[0037] Figure 2 A schematic diagram of the main functional architecture of the parameter adaptation inference system P1 described in the first variation of the above embodiment is shown. Figure 2 As shown, the parameter adaptation inference system P1 differs from the parameter adaptation inference system P described in the above embodiment in that it further includes a historical adaptation library management module N6. The historical adaptation library management module N6 is configured to store, retrieve, and update historical task adaptation information. The historical task adaptation information referred to here refers to adaptation parameter-related data entries indexed by a joint index of task identifier and target model identifier. It includes at least the task identifier, the target model identifier, and a set of low-rank adaptation parameters associated with the task identifier and the target model identifier. It may also include structured multimodal prompts (or their summaries), fused semantic representations (or their summaries), model parameter statistics summaries, performance metric summaries, version numbers, and timestamps. For example, an entry stored in the historical adaptation library management module N6 might be: "Task identifier: DEF_001; Target model identifier: LVM_v2; Low-rank adaptation parameter set: [Module 1 parameter pair, Module 2 parameter pair, Module 3 parameter pair]; Performance metric: mAP=0.85; Version number: v1.0". Specifically, after the parameter injection execution module N5 generates the task output, the historical adaptation library management module N6 can write or update the task identifier, target model identifier, low-rank adaptation parameter set, and their metadata for the current task to the historical adaptation library. On the other hand, when a new task is received, the historical adaptation library management module N6 can retrieve historical entries based on the joint index of the task identifier and the target model identifier. If a matching entry is found, the stored low-rank adaptation parameter set can be directly reused without regenerating it. Optionally, the historical adaptation library management module N6 supports retaining multiple versions of the same joint index and recording effective flags to support version rollback and multi-version management.

[0038] According to the parameter adaptation inference system P1 described in the first variation, by adding a historical adaptation library management module to store and retrieve historical task adaptation information, the accumulated adaptation parameter experience can be reused, improving the adaptation efficiency of repetitive tasks. Simultaneously, by using version management based on composite indexes, version rollback and multi-version management of adaptation parameters can be supported, enhancing the system's continuous iteration capability and engineering maintainability.

[0039] Next, based on the above, refer to Figure 3The main architecture of the parameter adaptation inference system P2 described in the second variation of the above-described embodiment will be explained. Here, to avoid repetition, only the differences between this second variation and the above-described embodiments and the first variation will be described.

[0040] Figure 3 A schematic diagram of the main functional architecture of the parameter adaptation inference system P2 described in the second variation of the above embodiment is shown. Figure 3 As shown, the difference between this second variation and the parameter adaptation inference system described in the above embodiment and the first variation is that the parameter adaptation inference system P2 includes both the historical adaptation library management module N6 and the closed-loop iteration control module N7. Regarding the historical adaptation library management module N6, since it has already been described in the first variation, a repeat description is omitted here; only the closed-loop iteration control module N7 will be described.

[0041] The closed-loop iterative control module N7 is configured to receive a performance index summary corresponding to the task output result output by the parameter injection execution module N5, and when the performance index summary meets the preset trigger conditions, trigger the historical adaptation library management module N6 to selectively write or update the historical task adaptation information, and trigger the training or tuning of the adaptation parameter generation module N4. The performance index summary mentioned here refers to the summary of evaluation metrics of the task output result on the target task, such as the mean accuracy of the target detection task, the accuracy of the classification task, etc. The preset trigger conditions mentioned here refer to pre-set conditions used to determine whether to trigger the historical adaptation library update and model tuning, including but not limited to any one or more of the following: the number of entries in the library reaches a threshold, a preset time period is reached, the performance index summary is below the threshold or the fluctuation exceeds the threshold, and the new task matches the similarity of tasks in the library but the index does not meet the standard. Specifically, after receiving the performance index summary, the closed-loop iterative control module N7 compares it with the preset trigger conditions. When the triggering conditions are met, the closed-loop iterative control module N7 sends a write or update instruction to the historical adaptation library management module N6, so that the task identifier, target model identifier, low-rank adaptation parameter set and performance index summary of this task are written or updated to the historical adaptation library; at the same time, the closed-loop iterative control module N7 sends a training or tuning instruction to the adaptation parameter generation module N4, so that it can train or tune based on the expanded historical adaptation library, and the adaptation parameter generation module with improved generation capabilities is redeployed.

[0042] According to the parameter adaptation inference system P2 described in the second variation, by further adding a historical adaptation library management module and a closed-loop iteration control module, the system can automatically trigger the updating of the historical adaptation library and the optimization of the adaptation parameter generation module based on the performance index summary, forming a closed-loop mechanism of "generation—verification—accumulation—iteration". This enables the adaptation parameter generation capability to be continuously optimized under new tasks and scenarios, improving the system's generalization ability and long-term maintainability, while also enhancing its adaptability under changing task distribution conditions.

[0043] Next, based on the parameter adaptation inference system P described in the above embodiments, referring to... Figure 4 The main flow of the parameter adaptation reasoning method according to one embodiment of the present disclosure will be described.

[0044] Figure 4 A flowchart illustrating the main steps of a parameter adaptation reasoning method according to an embodiment of the present disclosure is shown.

[0045] First, in step ST100 (i.e., the first step), the task description text and task visual examples are obtained, and the target model and its set of target injectable modules are determined. Specifically, the task description text given in free natural language and a small number of task-related visual examples are obtained, and the target model to be adapted and its preset set of target injectable modules are determined.

[0046] Next, in step ST200 (i.e., the second step), template matching and formatting are performed on the task description text and the task visual example based on the prompt word template library to generate and output structured multimodal prompt content. Optionally, in step ST200, the structured multimodal prompt content includes at least structured text fields and structured visual fields. The structured text fields are generated by extracting and normalizing fields from the task description text, and the structured visual fields are generated by organizing the task visual example and its metadata according to template requirements.

[0047] Then, in step ST300 (i.e., the third step), the structured multimodal cue content is fused and encoded to generate and output a fused semantic representation. Alternatively, in step ST300, the fused cue sequence is pooled to obtain a fused vector, and the fused cue sequence and / or the fused vector are used as one of the conditional inputs for the subsequent generation of low-rank adaptation parameters.

[0048] Subsequently, in step ST400 (i.e., the fourth step), a model parameter statistics vector and a set of position codes are constructed based on the parameter state of the target model and the target injectable module set. Optionally, in step ST400, the set of position codes is constructed based on the module type identifier and layer index identifier of each target module in the target injectable module set, and each position code is generated by concatenating the module type embedding and the layer index embedding.

[0049] Next, in step ST500 (i.e., the fifth step), the fused semantic representation, the model parameter statistics vector, and the location encoding set are used as conditional inputs to generate and output a low-rank adaptation parameter set and an adaptation configuration description that correspond one-to-one with the target injectable module set. Optionally, in step ST500, the low-rank adaptation parameter set is configured to contain low-rank parameter pairs corresponding one-to-one with the target module, and includes configuration items for rank and scaling coefficients. The adaptation configuration description includes at least the target module index information.

[0050] Subsequently, in step ST600 (i.e., the sixth step), according to the adaptation configuration description, the low-rank adaptation parameter set is injected into the target injectable module set of the target model, and the injected target model is used to perform task inference, generating and outputting the task output results. Alternatively, in step ST600, the low-rank adaptation parameter set and the adaptation configuration description are encapsulated into a parameter package, and signature verification and integrity verification are performed on the parameter package before injection.

[0051] Next, based on the parameter adaptation inference system P2 described in the second variation above, referring to... Figure 5 The main flow of the first variation of the parameter adaptation reasoning method of the above-described implementation method will be explained.

[0052] Figure 5 A flowchart of a first variation of the parameter adaptation inference method described in the above embodiments is shown. The difference from the parameter adaptation inference method described in the above embodiments is that step ST700 is included after step ST600.

[0053] In step ST700, based on the performance metric summary corresponding to the task output, when the performance metric summary meets preset trigger conditions, selective writing or updating of the historical adaptation library is triggered, as well as training or tuning of the adaptation parameter generation module. Specifically, the evaluation metrics of the task output on the target task are compared with preset trigger conditions. When the trigger conditions are met, the task identifier, target model identifier, low-rank adaptation parameter set, and performance metric summary of the current task are written or updated to the historical adaptation library, and the adaptation parameter generation module is triggered to train or tune based on the expanded historical adaptation library.

[0054] Furthermore, it should be understood that this disclosure also provides an electronic device including a processor and a memory (or a computer-readable storage medium), the memory storing a computer program or instructions capable of implementing the parameter adaptation reasoning method described in the above embodiments and their first variations and any other suitable variations.

[0055] In addition, this disclosure also provides a server configured with the parameter adaptation inference system described in the first embodiment and its first to second modifications and other suitable modifications.

[0056] In addition, this disclosure also provides a computer program product comprising a computer program or instructions for implementing the parameter adaptation reasoning method described in the above embodiments and their first variations and any other suitable variations.

[0057] It should be understood that this disclosure is not limited to the systems or methods described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A multi-task parameter adaptation inference system, characterized in that, include: The input acquisition module is configured to acquire task description text and visual examples related to the target task, and to determine the target model and its set of injectable modules; The prompt generation module is configured to receive the task description text and the task visual example, perform template matching on the task description text and the task visual example based on a preset prompt word template library, and generate and output structured multimodal prompt content. The multimodal fusion module is configured to receive the structured multimodal prompt content, perform fusion encoding on the structured multimodal prompt content, and generate and output a fusion semantic representation; The condition information construction module is configured to construct a model parameter statistics vector and a location encoding set based on the parameter state of the target model and the target injectable module set; The adaptation parameter generation module is configured to receive the fused semantic representation, the model parameter statistics vector, and the position encoding set, and generate and output a low-rank adaptation parameter set and an adaptation configuration description corresponding to the target injectable module set. as well as The parameter injection execution module is configured to inject the low-rank adaptation parameter set into the target injectable module set of the target model according to the adaptation configuration description, so that the injected target model can be used to perform task inference to generate and output task output results.

2. The multi-task parameter adaptation inference system as described in claim 1, characterized in that, It also includes a historical adaptation library management module, which is configured to manage historical task adaptation information. The historical task adaptation information includes at least the task identifier of the target task, the target model identifier of the target model, and a set of low-rank adaptation parameters associated with the task identifier and the target model identifier.

3. The multi-task parameter adaptation inference system as described in claim 2, characterized in that, It also includes a closed-loop iterative control module, which is configured to: Receive a summary of the performance metrics corresponding to the task output results; When the performance metric summary meets the preset triggering conditions, the historical adaptation library management module writes or updates the historical task adaptation information. as well as The adaptation parameter generation module is trained or optimized.

4. The multi-task parameter adaptation inference system as described in claim 1, characterized in that, The structured multimodal prompt content includes at least structured text fields and structured visual fields. The structured text fields are generated by extracting and normalizing fields from the description text of the target task. The structured visual field is generated by processing the task visual example and its meta-information according to a template selected from the cue word template library.

5. The parameter adaptation inference system as described in claim 1, characterized in that, The condition information construction module is further configured to construct the position code set based on the module type identifier and layer index identifier of each target module in the target injectable module set. Each position code in the position code set is generated by concatenating module type embedding and layer index embedding. The module type embedding is a vector representation of the model type identifier, and the layer index embedding is a vector representation of the layer index identifier.

6. The multi-task parameter adaptation inference system as described in claim 1, characterized in that, The low-rank adaptation parameter set contains low-rank parameter pairs and configuration items for rank and scaling factor. The low-rank parameter pairs correspond to each target module in the target injectable module set. The adaptation configuration description includes at least the module type identifier and the layer index identifier of the target module.

7. The multi-task parameter adaptation inference system as described in claim 1, characterized in that, The parameter injection execution module is further configured to encapsulate the low-rank adaptation parameter set and the adaptation configuration description into a parameter package.

8. A multi-task parameter adaptation inference method, characterized in that, include: The first step is to obtain the task description text and visual examples related to the target task, and to determine the target model and its set of injectable modules. The second step involves performing template matching on the task description text and the task visual example based on a preset prompt word template library, generating and outputting structured multimodal prompt content. The third step is to fuse and encode the structured multimodal prompt content to generate and output a fused semantic representation; The fourth step is to construct a model parameter statistics vector and a location encoding set based on the parameter state of the target model and the target injectable module set. The fifth step involves taking the fused semantic representation, the model parameter statistics vector, and the location encoding set as conditional inputs to generate and output a low-rank adaptation parameter set and adaptation configuration description that correspond one-to-one with the target injectable module set. as well as The sixth step is to inject the low-rank adaptation parameter set into the target injectable module set of the target model according to the adaptation configuration description, so that the injected target model can be used to perform task inference to generate and output task output results.

9. The multi-task parameter adaptation inference method as described in claim 8, characterized in that, The sixth step is followed by: Obtain a summary of the performance metrics corresponding to the task output; and When the performance metric summary meets the preset triggering conditions, the historical task adaptation information is written or updated. The historical task adaptation information includes at least the task identifier of the target task, the target model identifier of the target model, and a set of low-rank adaptation parameters associated with the task identifier and the target model identifier.

10. The multi-task parameter adaptation inference method as described in claim 8, characterized in that, The structured multimodal prompt content includes at least structured text fields and structured visual fields. The structured text fields are generated by extracting and normalizing fields from the description text of the target task. The structured visual field is generated by processing the task visual example and its meta-information according to a template selected from the cue word template library.

11. The multi-task parameter adaptation inference method as described in claim 8, characterized in that, In the fourth step, the location code set is constructed based on the module type identifier and layer index identifier of each target module in the target injectable module set. Each location code is generated by concatenating the module type embedding and the layer index embedding. The module type embedding is a vector representation of the model type identifier, and the layer index embedding is a vector representation of the layer index identifier.

12. The multi-task parameter adaptation inference method as described in claim 8, characterized in that, The low-rank adaptation parameter set contains low-rank parameter pairs. The low-rank parameter pairs correspond to each target module in the target injectable module set. The adaptation configuration description includes at least the module type identifier and the layer index identifier of the target module.

13. The multi-task parameter adaptation inference method as described in claim 8, characterized in that, In the sixth step, the low-rank adaptation parameter set and the adaptation configuration description are encapsulated into a parameter package.

14. An electronic device comprising a processor and a memory, the memory storing a computer program, characterized in that, The computer program is executed by the processor to implement the multi-task parameter adaptation inference method according to any one of claims 8 to 13.

15. A server, characterized in that, The server is configured with the multi-task parameter adaptation inference system as described in any one of claims 1 to 7.

16. A computer program product, comprising a computer program, characterized in that, The computer program is executed by a processor to implement the multi-task parameter adaptation inference method according to any one of claims 8 to 13.

17. A computer-readable storage medium storing a computer program, characterized in that, The computer program is executed by a processor to implement the multi-task parameter adaptation inference method according to any one of claims 8 to 13.

Images