A multi-modal agent task allocation method and system for large model services

Abstract

The application provides a multi-modal agent task allocation method and system for large model services, belonging to the technical field of large models, comprising: receiving subjective evaluation for target output, obtaining a subjective preference vector and calculating a comprehensive score of fused subjective preference according to the subjective evaluation, taking the comprehensive score of fused subjective preference as the updated performance of each large model under different task types, obtaining the multi-modal task request of the next round of users, using a general large model to perform task type identification on the subjective preference vector, the updated context summary vector and the multi-modal task request of the next round of users, and obtaining the score of each large model for the task identification result from the updated performance of each large model under different task types. The application realizes dynamic optimal allocation of multi-modal tasks among different large models, effectively reduces the operation cost of users in model selection and parameter optimization, and significantly improves the efficiency and accuracy of multi-modal task processing.

Description

Technical Field

[0001] This invention belongs to the field of large model technology, specifically relating to a multimodal proxy task allocation method and system for large model services. Background Technology

[0002] Large models (LMs) refer to deep learning models with a large number of parameters and complex structures, such as Transformer and GPT (Generative Pre-trained Transformer). Trained on massive amounts of data, they can learn rich knowledge representations and feature extraction capabilities, and are widely used in tasks such as image classification, text analysis, and speech recognition. Large language models (LLMs), on the other hand, are large models specifically designed for the field of natural language processing. Trained on large-scale text data, they learn the syntax, semantics, and contextual information of a language, enabling them to generate natural language text, understand text meaning, and perform text classification, demonstrating powerful performance in natural language processing. Large language models have achieved groundbreaking results in multiple fields. For example, in image generation tasks, large language models such as DALL-E and Midjourney can accurately understand user-input text descriptions to generate detailed, high-quality images. In text analysis tasks, BERT (Bidirectional Encoder Representations from Transformers) and GPT-4 (Generative Pre-trained Transformer 4) can perform sentiment analysis, machine translation, and text summarization. Furthermore, large models and large language models possess strong generalization capabilities, maintaining good performance on unseen data, which gives them broad prospects and potential in practical applications.

[0003] The Multimodal Intelligent Task Allocation and Optimization System is an advanced artificial intelligence solution that integrates large-scale model resources from multiple modalities, such as image processing, text analysis, and speech recognition. Through intelligent analysis of task requirements, it automatically matches the most suitable large-scale model for task processing. This system not only has task receiving and parsing capabilities but also performs real-time optimization and adjustments during execution to ensure the quality and efficiency of the output results. It enables efficient collaborative work among multiple models, enhances overall processing capabilities, and provides a user-friendly interface. It is suitable for complex multimodal fusion tasks such as image classification, sentiment analysis, and speech-to-text conversion, aiming to provide users with more efficient and accurate services.

[0004] With the rapid development of artificial intelligence technology, various large-scale models have achieved remarkable results in fields such as image, text, speech, and multimodal fusion. However, the current technological system still suffers from many deep-seated contradictions: structural problems such as the fragmented capabilities of different models, the disjointed semantic representation, and the difficulty in cross-model context integration are still prevalent. Especially in complex multimodal tasks, the lack of effective collaboration between models makes it easy for information to be lost or distorted during cross-modal transmission, resulting in a lengthy overall system processing chain, low inference efficiency, and insufficient stability and accuracy of results. More importantly, this fragmented technological form makes it difficult to support users' expectations for a "unified intelligent experience," preventing models from achieving truly consistent, coherent, and efficient intelligent responses in continuous tasks and mixed-modal scenarios.

[0005] First, while numerous large open-source models exist in the market, no single model can achieve optimal performance across all tasks. Each large model is typically optimized for a specific domain or task, such as image processing, text analysis, or speech recognition, and may perform only moderately or insufficiently in other domains. This "single-model capability boundary" directly makes it difficult for a single model to independently complete multimodal and multi-type tasks. To achieve better results, users have to frequently switch between different models, manually determining which model to use and which parameters are suitable for the current input for a given task. This not only increases the complexity of the workflow but also incurs high time and learning costs, impacting overall efficiency.

[0006] Secondly, when users attempt to use different large models to handle different types of tasks, they often encounter the typical problem of "context not being carried over." Because these models are built from different architectures and training corpora, they inherently lack semantic continuity and sharing mechanisms. Even if the previous model has accumulated key background information in dialogue or task processing, this context usually cannot be inherited or reused after switching to another model. For multi-turn tasks, multimodal scenarios, or complex processing flows requiring cross-model collaboration, this context breakage can lead to comprehension biases and information loss, thereby weakening reasoning quality and task completion. Summary of the Invention

[0007] To address the shortcomings of existing technologies, this application proposes a multimodal agent task allocation method and system for large model services.

[0008] In a first aspect, the present invention provides a method for allocating multimodal proxy tasks for large model services, comprising:

[0009] Step S1: Obtain multiple large models of different types and calculate the performance of each large model under different task types;

[0010] Step S2: Obtain the user's multimodal task requests and use a general large model to identify the task type of the multimodal task requests;

[0011] Step S3: Based on the task type identification results, obtain the score of each large model for the task identification results from the performance of each large model under different task types;

[0012] Step S4: Select the largest model with the highest score as the target model;

[0013] Step S5: Encode the multimodal task request and its corresponding background using the context representation module of the general large model to obtain the context vector;

[0014] Step S6: Aggregate the context vectors to obtain the context summary vector;

[0015] Step S7: Fuse the context summary vector and the multimodal task request to obtain the enhanced input;

[0016] Step S8: Feed the augmented input into the target model to obtain the target output, and update the context summary vector using the target output;

[0017] Step S9: Receive subjective evaluations of the target output. Based on the subjective evaluations, obtain the subjective preference vector and calculate the comprehensive score of the fused subjective preferences. Use the comprehensive score of the fused subjective preferences as the updated performance of each large model under different task types. Proceed to step S2 to obtain the multimodal task requests of the next round of users. Use a general large model to identify the task type of the subjective preference vector, the updated context summary vector, and the multimodal task requests of the next round of users. In step S3, obtain the score of each large model for the task identification result from the updated performance of each large model under different task types.

[0018] The process of acquiring multiple large models of different types and calculating the performance of each large model under different task types includes:

[0019] Acquire multiple large models of different types to form a large model set;

[0020] For each type of task, a standardized set of evaluation tasks is generated;

[0021] Each evaluation task in the evaluation task set is input into each large model in the large model set to obtain the output result corresponding to each large model.

[0022] The output results of each large model are evaluated in multiple dimensions to obtain the performance score of each model under each evaluation task.

[0023] Based on the performance score of each model in each evaluation task, record the performance vector of each large model in different task types.

[0024] The aggregation of context vectors to obtain a context summary vector includes:

[0025] Set a context window, which includes multiple context vectors;

[0026] Average pooling is used to aggregate all context vectors in the context window to obtain a context summary vector.

[0027] The process of receiving subjective evaluations of the target output, obtaining a subjective preference vector based on the subjective evaluations, and calculating a comprehensive score that integrates subjective preferences includes:

[0028] Receive subjective evaluations of the target output, including: an overall satisfaction rating of the target output, a style rating of the target output, a creativity rating of the target output, and an accuracy rating of the target output;

[0029] The subjective evaluation is encoded using the preference modeling module of a general large model to obtain a subjective preference vector, which represents the user's preference profile in multiple dimensions such as style, creativity and accuracy.

[0030] Based on the overall satisfaction score of the target output and the performance scores of the target model under each evaluation task, a satisfaction function is constructed.

[0031] Based on the satisfaction function, a personalized satisfaction estimate is performed on the target output for the next round.

[0032] Based on the target output of the next round, a personalized satisfaction estimate is performed to obtain the comprehensive score of the next round that incorporates subjective preferences.

[0033] The satisfaction function is constructed based on the overall satisfaction score of the target output and the performance scores of the target model under each evaluation task. The calculation formula is as follows:

[0034] ;

[0035] in, Let k be the satisfaction function for the kth round. For the target model in round k, The quality score of the target model in the k-th round is... The computational cost score for the target model in the k-th round is given. Let be the response delay score of the target model in the k-th round. As the weight of the quality score, To calculate the weight of the cost score, As a weight for response latency score, The overall satisfaction score is output as the target. The weighting for the overall satisfaction score.

[0036] The next round of target output is estimated using a personalized satisfaction function, calculated as follows:

[0037] ;

[0038] in, For the personalized satisfaction estimate of the target output in the kth round, For the personalized satisfaction estimate of the target output in the (k-1)th round, The target model for round k-1 The satisfaction function, For the target model in round k-1, To control the weight of satisfaction.

[0039] The next round of personalized satisfaction estimation is performed based on the target output, resulting in a comprehensive score that incorporates subjective preferences. The calculation formula is as follows:

[0040] ;

[0041] in, For the next round of task types The overall score that integrates subjective preferences, To obtain the score of the i-th large model for task recognition results using a general large model in the next round, For the task type in round k Personalized satisfaction estimation of the target output. For weight fusion.

[0042] Secondly, the present invention also provides a multimodal agent task allocation system for large model services, comprising:

[0043] The performance calculation module is used to acquire multiple large models of different types and calculate the performance of each large model under different task types.

[0044] The type recognition module is used to acquire the user's multimodal task requests and use a general large model to identify the task type of the multimodal task requests;

[0045] The scoring calculation module is used to obtain the score of each large model for the task identification result based on the performance of each large model under different task types, according to the task type identification result.

[0046] The target model determination module is used to select the highest-scoring large model as the target model.

[0047] The context encoding module is used to encode the multimodal task request and its corresponding background using the context representation module of the general large model, so as to obtain the context vector;

[0048] The vector aggregation module is used to aggregate context vectors to obtain a context summary vector;

[0049] The vector fusion module is used to fuse the context summary vector and the multimodal task request to obtain enhanced input;

[0050] The vector update module is used to feed the augmented input into the target model, obtain the target output, and update the context summary vector using the target output;

[0051] The subjective evaluation module receives subjective evaluations of the target output. Based on these evaluations, it obtains a subjective preference vector and calculates a comprehensive score that integrates these subjective preferences. This comprehensive score is then used as the updated performance of each large model under different task types. The module then moves to the type recognition module to obtain the next round of users' multimodal task requests. A general large model is used to identify the task type of the subjective preference vector, the updated context summary vector, and the next round of users' multimodal task requests. Finally, in the score calculation module, the score for each large model's task recognition result is obtained from the updated performance of each large model under different task types.

[0052] Thirdly, this application proposes an electronic device, including: one or more processors, and a memory for storing instructions, which, when executed by the one or more processors, cause the one or more processors to perform the multimodal agent task allocation method for large model services.

[0053] Fourthly, this application proposes a computer-readable storage medium storing executable instructions that, when executed, cause a processor to perform the aforementioned multimodal agent task allocation method for large model services.

[0054] Beneficial effects:

[0055] This application proposes a multimodal proxy task allocation method and system for large model services. By constructing an intelligent routing mechanism based on multi-model performance evaluation, a cross-model inheritable contextual collaborative modeling method, and a personalized satisfaction optimization strategy that integrates user subjective preferences, the system achieves dynamic optimal allocation of multimodal tasks among different large models. This enables the system to output higher-quality results while ensuring semantic continuity and generation consistency. At the same time, it effectively reduces the operational costs for users in model selection and parameter tuning, significantly improves the efficiency, accuracy, and personalized experience of multimodal task processing, and achieves a synergistic improvement in performance, consistency, and user satisfaction. Attached Figure Description

[0056] Figure 1 Flowchart of a multimodal proxy task allocation method for large model services according to an embodiment of the present invention;

[0057] Figure 2 A flowchart of model selection and context vector aggregation processing according to an embodiment of the present invention;

[0058] Figure 3 A principle block diagram of a multimodal agent task allocation system for large model services according to an embodiment of the present invention. Detailed Implementation

[0059] The specific implementation methods of this application will be further described in detail below with reference to the accompanying drawings and embodiments.

[0060] This application aims to address key issues in current multimodal task processing, such as inaccurate model matching, inability to inherit contextual semantics across models, significant waste of computing resources, and low overall processing efficiency. By innovatively designing an intelligent task allocation and optimization system, this invention can automatically match the largest model best suited for the task based on task attributes, input features, and real-time resource status, achieving dynamic optimal task distribution among different models and significantly improving overall system performance. Simultaneously, this system overcomes the problem of information silos between traditional multi-model systems by structurally modeling user interaction history, task chains, and contextual information. This allows for the smooth transfer and effective utilization of context across different models, significantly improving comprehension, generation quality, and reasoning consistency in complex contexts. The implementation of this invention not only significantly improves the efficiency and accuracy of large models in multimodal task processing but also greatly reduces user operation complexity and resource consumption, providing strong technical support for the further expansion and deepening of artificial intelligence in practical applications. It has significant practical implications and broad application prospects.

[0061] The core idea of ​​this application is to address the problem of uneven performance of a single large model in multimodal tasks and the difficulty in balancing generation quality and context consistency. A multimodal proxy task allocation method for large model services is proposed. This method achieves intelligent matching and dynamic optimal allocation of different types of tasks by evaluating the performance of multiple large models, coordinating contexts, and fusing subjective preferences, thereby improving the overall effect and consistency of cross-modal generation.

[0062] Example 1:

[0063] This embodiment provides a method for allocating multimodal proxy tasks for large model services, such as... Figure 1 , Figure 2 As shown, it includes:

[0064] Step S1: Obtain multiple large models of different types and calculate the performance of each large model under different task types, including:

[0065] Step S1.1: Obtain multiple large models of different types to form a large model set;

[0066] In practice, multiple different types of large model APIs (Application Programming Interfaces) are integrated into the system to form a large model set. :

[0067] ;

[0068] in, It is a large collection of models, including text generation models, image generation models, translation models, video generation models, etc.

[0069] Step S1.2: Generate a standardized set of evaluation tasks for each type of task;

[0070] In this embodiment, various task types are addressed. where n is the number of each task type. For the nth task type, the system automatically generates a standardized set of evaluation tasks. Evaluation task set Each evaluation task includes multimodal input formats such as text, image, voice, and video, ensuring that the evaluation covers complex multi-round, multimodal task structures that may be involved in real-world business scenarios. Each task can be executed on different models to test performance.

[0071] Step S1.3: Input each evaluation task in the evaluation task set into each large model in the large model set, and obtain the output result corresponding to each large model;

[0072] Step S1.4: Evaluate the output results of each large model in multiple dimensions to obtain the performance score of each model under each evaluation task;

[0073] In this embodiment, the system calls each API interface to execute the corresponding evaluation task, collects its output results and multi-dimensional indicators (such as generation quality, cost, latency, etc.), and calculates the model's performance score under each task type:

[0074] ;

[0075] in, This represents the quality score of the m-th large model for the t-th task type. The computational cost of the m-th large model for the t-th task type is... Let be the response latency of the m-th large model for the t-th task type.

[0076] Step S1.5: Based on the performance score of each model in each evaluation task, record the performance vector of each large model in different task types.

[0077] In this embodiment, the system establishes a model performance profile, recording the performance of each model. The performance vectors for different task types are calculated as follows:

[0078] ;

[0079] in, For the first Performance vectors of large models across different task types For the first The large model in the first Performance vectors for each task type.

[0080] Step S2: Obtain the user's multimodal task requests and use a general large model to identify the task type of the multimodal task requests;

[0081] In this embodiment, when the system receives a multimodal task request input by the user... Then, the task recognition module of the general large model is used first. right Perform task comprehension and task type determination, and output the probability distribution of each task type. :

[0082] ;

[0083] in, The probability distribution for each task type. For multimodal task requests The later Probability distribution by task type This is a task recognition module for general-purpose large models, responsible for identifying tasks based on multimodal task requests. Confirm the task type.

[0084] The system determines the type of the current task based on the principle of maximum probability:

[0085] ;

[0086] in, The type of task to be identified.

[0087] Step S3: Based on the task type identification results, obtain the score of each large model for the task identification results from the performance of each large model under different task types;

[0088] Step S4: Select the largest model with the highest score as the target model;

[0089] In this embodiment, based on the identified task type The system reads the overall performance score of each model in the task type from the performance profile and selects the target model based on the highest overall performance score:

[0090] ;

[0091] in, For the target model.

[0092] Step S5: Encode the multimodal task request and its corresponding background using the context representation module of the general large model to obtain the context vector;

[0093] In this embodiment, upon receiving each task request... At that time, the context representation module of the general large model Encode the current task and its related context to generate a context vector representation:

[0094] ;

[0095] in, For task request The context vector is used to describe the semantic state and user intent in this round of tasks.

[0096] Step S6: Aggregate the context vectors to obtain a context summary vector, including:

[0097] Step S6.1: Set up a context window, which includes multiple context vectors;

[0098] In this embodiment, the system maintains a size of Context window:

[0099] ;

[0100] in, For context window data, For the first in the context window data A context vector, meaning the system always retains only the most recent one. A context vector is used to balance semantic continuity with computational cost. Window size. It can be dynamically configured based on user preferences or task complexity.

[0101] Step S6.2: Aggregate all context vectors in the context window using average pooling to obtain the context summary vector.

[0102] In this embodiment, to avoid all Directly inputting the vectors into the model leads to input redundancy. In this embodiment, the context is aggregated, and the set is subjected to average pooling to obtain a context summary vector:

[0103]

[0104] in, For vector average pooling operation, Let t be the context summary vector for the t-th task type, and finally obtain a context summary vector of controllable length. As a unified semantic background representation;

[0105] Step S7: Fuse the context summary vector and the multimodal task request to obtain the enhanced input;

[0106] In this embodiment, the system fuses the context summary vector with the current multimodal task request to construct the final input augmentation:

[0107]

[0108] in, This represents a vector concatenation operation. To enhance input.

[0109] Step S8: Feed the augmented input into the target model to obtain the target output, and update the context summary vector using the target output;

[0110] In this embodiment, the context summary vector is mainly used to store interaction information with the large model. When executing a new task, the context summary vector is passed in as auxiliary information to help the large model better understand the relationship between the user's past needs and the user's needs in the next round. This embodiment is used to assign tasks to different LLMs according to different task types and receive a return value. Since there is no communication between different large models, the user's needs on large model A cannot be understood by large model B. Therefore, it is necessary to summarize and refine past information as auxiliary knowledge and pass it to large model B. In other words, the context vector is auxiliary information given to the LLM after the system has already decided which LLM to assign the task to, and it is input along with the task requirements.

[0111] In this embodiment, the enhanced input Input target model To obtain output :

[0112]

[0113] in, This ensures that different large models share the same semantic context in cross-process and multimodal calls, achieving semantic continuity.

[0114] After the task is executed, the system will update the context summary vector based on the current output:

[0115]

[0116] in, This is the context window data for the (t+1)th task type. This is the context summary vector for the (t+1)th task type.

[0117] Step S9: Receive subjective evaluations of the target output. Based on the subjective evaluations, obtain the subjective preference vector and calculate the comprehensive score of the fused subjective preferences. Use the comprehensive score of the fused subjective preferences as the updated performance of each large model under different task types. Proceed to step S2 to obtain the multimodal task requests of the next round of users. Use a general large model to identify the task type of the subjective preference vector, the updated context summary vector, and the multimodal task requests of the next round of users. In step S3, obtain the score of each large model for the task identification result from the updated performance of each large model under different task types.

[0118] The process of receiving subjective evaluations of the target output, obtaining a subjective preference vector based on the subjective evaluations, and calculating a comprehensive score that integrates subjective preferences includes:

[0119] Step S9.1: Receive subjective evaluations of the target output, including: overall satisfaction score of the target output, style score of the target output, creativity score of the target output, and accuracy score of the target output;

[0120] In this embodiment, subjective evaluation is used to examine the user's preferences for various dimensions of the LLM output, and is used to form the subsequent subjective preference vector. There are many large models for the same task type. For example, the LLM for image generation integrates 5 large models. In addition to the objective performance of these 5 large models themselves, the user also needs to have a subjective preference. The user may prefer to generate comic-style images. Then, based on this preference vector, the system will consider that the large model A, which is good at comic-style images, should be the user's first choice for comic illustration image generation tasks.

[0121] In this embodiment, in the first In each round of tasks, the system selects the target model. As the output of the target model The system guides users to subjectively evaluate the results of this round through an interactive interface, recording the user's evaluation on the [number]th round. The overall subjective feedback from the wheel is as follows:

[0122] ;

[0123] in, The target model selected by the system in the k-th round of the task. The target output for the k-th round of the task. Indicates user's opinion Overall satisfaction score For user output Style rating For user output Creativity score For user output Accuracy score, This is the subjective evaluation for the kth round.

[0124] Step S9.2: Use the preference modeling module of the general large model to encode the subjective evaluation to obtain the subjective preference vector. The subjective preference vector represents the user's preference profile in multiple dimensions such as style, creativity and accuracy.

[0125] In this embodiment, the preference modeling module of the general large model Encode the user's historical ratings to obtain a subjective preference vector:

[0126] ;

[0127] in, This is a subjective preference vector, which represents a user's preference profile across multiple dimensions, including style, creativity, and accuracy. ( ) represents the preference modeling module of the general large model.

[0128] Step S9.3: Based on the overall satisfaction score of the target output and the performance scores of the target model under each evaluation task, construct the satisfaction function, calculated as follows:

[0129] ;

[0130] in, Let k be the satisfaction function for the kth round. For the target model in round k, The quality score of the target model in the k-th round is... The computational cost score for the target model in the k-th round is given. Let be the response delay score of the target model in the k-th round. As the weight of the quality score, To calculate the weight of the cost score, As a weight for response latency score, The overall satisfaction score is output as the target. The weighting for the overall satisfaction score.

[0131] In this embodiment, to incorporate user subjective preferences, a "personalized performance profile" and a satisfaction function are further constructed. The definition of the first... Wheelset Model Overall satisfaction .

[0132] Step S9.4: Based on the satisfaction function, perform personalized satisfaction estimation for the target output of the next round. The calculation formula is as follows:

[0133] ;

[0134] in, For the personalized satisfaction estimate of the target output in the kth round, For the personalized satisfaction estimate of the target output in the (k-1)th round, The target model for round k-1 The satisfaction function, For the target model in round k-1, To control the weight of satisfaction.

[0135] In this embodiment, for the user , for the first Round selection model Maintaining personalized satisfaction estimates And it is updated after each call, while other models not called in this round remain unchanged. Parameters Control the weight of new feedback.

[0136] Step S9.5: Based on the target output of the next round, perform personalized satisfaction estimation to obtain the comprehensive score of the next round that incorporates subjective preferences. The calculation formula is as follows:

[0137] ;

[0138] in, For the next round of task types The overall score that integrates subjective preferences, To obtain the score of the i-th large model for task recognition results using a general large model in the next round, For the task type in round k Personalized satisfaction estimation of the target output. For weight fusion.

[0139] In this embodiment, after completing the above modeling, Rotation task Upon arrival, the task type will be determined. Subsequently, model selection no longer depends solely on objective performance. Instead, it incorporates subjective preferences to construct a comprehensive score.

[0140] Based on this, the personalized target model is selected as follows:

[0141] ;

[0142] in, For personalized target models.

[0143] By repeating the above steps, the present invention achieves the goal of automatically selecting the best-performing large model that satisfies human preferences.

[0144] This embodiment proposes a multimodal proxy task allocation method for large model services. Firstly, existing large models often exhibit uneven performance across different task types, making it difficult to achieve optimal cross-modal output using a single model. To alleviate this problem, this embodiment proposes a multi-model task evaluation and intelligent allocation mechanism. The system automatically constructs a standardized evaluation task set covering multimodal formats such as text, image, speech, and video. It performs multi-dimensional index testing on different types of large model APIs and generates structured performance profiles, thereby achieving automatic matching of task types with the optimal model. Based on this mechanism, the system can dynamically select the optimal model interface during actual task execution based on task attributes, input features, and the actual performance of each model. This avoids performance bottlenecks caused by traditional manual judgment or fixed model calling patterns, achieving a better balance between accuracy, efficiency, and resource consumption in multimodal tasks.

[0145] Secondly, addressing the long-standing industry pain points such as semantic fragmentation and the inability to inherit context across models due to independent multi-model calls, this embodiment proposes a cross-model inheritable context management and collaborative modeling mechanism. The system generates task-level semantic vectors through a general large model, maintains a controllable-sized context window, and uses aggregation functions to compress semantic history into a unified context summary vector, enabling different models to share a consistent semantic background in multi-round, multi-modal task chains. This not only solves the semantic loss problem during model switching but also significantly improves the reasoning coherence and generation consistency of the task chain, fundamentally breaking through the structural limitations of existing large models' "information silos." This invention thus constructs a multi-model collaborative working mechanism with memory capabilities and the ability to continuously accumulate contextual information.

[0146] Finally, this embodiment constructs a satisfaction modeling mechanism that couples user subjective preferences with objective model performance. The system collects real-time scores for style, creativity, accuracy, and overall satisfaction through user interaction, encoding subjective feedback into preference vectors to build personalized performance profiles. This allows model selection for subsequent tasks to no longer solely rely on objective evaluation results, but rather achieve a dynamic trade-off between objective performance and user preferences. Furthermore, the system employs a round-by-round updated satisfaction function, enabling the model selection strategy to adaptively optimize as user preferences change. This allows the system to increasingly align with user style and preferences over continuous use, achieving truly "user-centric" intelligent task allocation and a breakthrough in unifying functionality and personalization.

[0147] Example 2:

[0148] This embodiment also provides a multimodal agent task allocation system for large model services, such as Figure 2 As shown, it includes:

[0149] The performance calculation module is used to acquire multiple large models of different types and calculate the performance of each large model under different task types.

[0150] The type recognition module is used to acquire the user's multimodal task requests and use a general large model to identify the task type of the multimodal task requests;

[0151] The scoring calculation module is used to obtain the score of each large model for the task identification result based on the performance of each large model under different task types, according to the task type identification result.

[0152] The target model determination module is used to select the highest-scoring large model as the target model.

[0153] The context encoding module is used to encode the multimodal task request and its corresponding background using the context representation module of the general large model, so as to obtain the context vector;

[0154] The vector aggregation module is used to aggregate context vectors to obtain a context summary vector;

[0155] The vector fusion module is used to fuse the context summary vector and the multimodal task request to obtain enhanced input;

[0156] The vector update module is used to feed the augmented input into the target model, obtain the target output, and update the context summary vector using the target output;

[0157] The subjective evaluation module receives subjective evaluations of the target output. Based on these evaluations, it obtains a subjective preference vector and calculates a comprehensive score that integrates these subjective preferences. This comprehensive score is then used as the updated performance of each large model under different task types. The module then moves to the type recognition module to obtain the next round of users' multimodal task requests. A general large model is used to identify the task type of the subjective preference vector, the updated context summary vector, and the next round of users' multimodal task requests. Finally, in the score calculation module, the score for each large model's task recognition result is obtained from the updated performance of each large model under different task types.

[0158] Example 3:

[0159] This embodiment proposes an electronic device, including: one or more processors, and a memory, wherein the memory is used to store instructions, and when the instructions are executed by the one or more processors, the one or more processors execute the multimodal proxy task allocation method for large model services.

[0160] The electronic device may be a mobile phone, computer, or tablet computer, etc., and includes a memory and a processor. The memory stores a computer program, which, when executed by the processor, implements a multimodal agent task allocation method for large model services as described in the embodiments. It is understood that the electronic device may also include input / output (I / O) interfaces and communication components.

[0161] The processor is used to execute all or part of the steps in the multimodal agent task allocation method for large model services as described in the above embodiments. The memory is used to store various types of data, which may include, for example, instructions for any application or method in an electronic device, as well as application-related data.

[0162] The processor can be implemented as an Application Specific Integrated Circuit (ASIC), Digital Signal Processor (DSP), Programmable Logic Device (PLD), Field Programmable Gate Array (FPGA), controller, microcontroller, microprocessor, or other electronic components, and is used to execute the multimodal proxy task allocation method for large model services described in the above embodiments.

[0163] Example 4:

[0164] This embodiment proposes a computer-readable storage medium that stores executable instructions. When these instructions are executed, if they are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium.

[0165] The computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the multimodal agent task allocation method for large model services described in various embodiments of this application.

[0166] The aforementioned storage media include: flash memory, hard disks, multimedia cards, card-type memory (e.g., SD (Secure Digital Memory Card) or DX (Memory Data Register, MDR) memory), random access memory (RAM), static random-access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic storage, disks, optical discs, servers, APP (Application) application stores, and other media capable of storing program verification codes. These media store computer programs, which, when executed by a processor, can implement the various steps of the aforementioned multimodal agent task allocation method for large-model services.

[0167] Example 5:

[0168] This embodiment proposes a computer program product, including a computer program or instructions, which, when executed by a processor, implements the aforementioned multimodal agent task allocation method for large model services.

[0169] Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or part of the technical solution, can be embodied in the form of a computer program product.

[0170] The various embodiments in this application are described in a progressive manner. The same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on describing the differences from other embodiments.

[0171] The scope of protection of this application is not limited to the embodiments described above. Obviously, those skilled in the art can make various modifications and variations to this disclosure without departing from the scope and spirit of this disclosure. If such modifications and variations fall within the scope of equivalent technology of this disclosure, then the intent of this disclosure also includes such modifications and variations.

Claims

1. A method for allocating multimodal proxy tasks for large model services, characterized in that, include: Step S1: Obtain multiple large models of different types and calculate the performance of each large model under different task types; Step S2: Obtain the user's multimodal task requests and use a general large model to identify the task type of the multimodal task requests; Step S3: Based on the task type identification results, obtain the score of each large model for the task identification results from the performance of each large model under different task types; Step S4: Select the largest model with the highest score as the target model; Step S5: Encode the multimodal task request and its corresponding background using the context representation module of the general large model to obtain the context vector; Step S6: Aggregate the context vectors to obtain the context summary vector; Step S7: Fuse the context summary vector and the multimodal task request to obtain the enhanced input; Step S8: Feed the augmented input into the target model to obtain the target output, and update the context summary vector using the target output; Step S9: Receive subjective evaluations of the target output. Based on the subjective evaluations, obtain the subjective preference vector and calculate the comprehensive score of the fused subjective preferences. Use the comprehensive score of the fused subjective preferences as the updated performance of each large model under different task types. Proceed to step S2 to obtain the multimodal task requests of the next round of users. Use a general large model to identify the task type of the subjective preference vector, the updated context summary vector, and the multimodal task requests of the next round of users. In step S3, obtain the score of each large model for the task identification result from the updated performance of each large model under different task types.

2. The multimodal proxy task allocation method for large model services according to claim 1, characterized in that, The process of acquiring multiple large models of different types and calculating the performance of each large model under different task types includes: Acquire multiple large models of different types to form a large model set; For each type of task, a standardized set of evaluation tasks is generated; Each evaluation task in the evaluation task set is input into each large model in the large model set to obtain the output result corresponding to each large model. The output results of each large model are evaluated in multiple dimensions to obtain the performance score of each model under each evaluation task. Based on the performance score of each model in each evaluation task, record the performance vector of each large model in different task types.

3. The multimodal proxy task allocation method for large model services according to claim 1, characterized in that, The aggregation of context vectors to obtain a context summary vector includes: Set a context window, which includes multiple context vectors; Average pooling is used to aggregate all context vectors in the context window to obtain a context summary vector.

4. The multimodal proxy task allocation method for large model services according to claim 1, characterized in that, The process of receiving subjective evaluations of the target output, obtaining a subjective preference vector based on the subjective evaluations, and calculating a comprehensive score that integrates subjective preferences includes: Receive subjective evaluations of the target output, including: an overall satisfaction rating of the target output, a style rating of the target output, a creativity rating of the target output, and an accuracy rating of the target output; The subjective evaluation is encoded using the preference modeling module of a general large model to obtain a subjective preference vector, which represents the user's preference profile in multiple dimensions such as style, creativity and accuracy. Based on the overall satisfaction score of the target output and the performance scores of the target model under each evaluation task, a satisfaction function is constructed. Based on the satisfaction function, a personalized satisfaction estimate is performed on the target output for the next round. Based on the target output of the next round, a personalized satisfaction estimate is performed to obtain the comprehensive score of the next round that incorporates subjective preferences.

5. A multimodal proxy task allocation method for large model services according to claim 4, characterized in that, The satisfaction function is constructed based on the overall satisfaction score of the target output and the performance scores of the target model under each evaluation task. The calculation formula is as follows: ； in, Let k be the satisfaction function for the kth round. For the target model in round k, The quality score of the target model in the k-th round. The computational cost score for the target model in the k-th round is... Let be the response delay score of the target model in the k-th round. As the weight of the quality score, To calculate the weight of the cost score, As a weight for response latency score, The overall satisfaction score is output as the target. The weighting for the overall satisfaction score.

6. The multimodal proxy task allocation method for large model services according to claim 4, characterized in that, The next round of target output is estimated using a personalized satisfaction function, calculated as follows: ； in, For the personalized satisfaction estimate of the target output in the kth round, For the personalized satisfaction estimate of the target output in the (k-1)th round, The target model for round k-1 The satisfaction function, For the target model in round (k-1), To control the weight of satisfaction.

7. The multimodal proxy task allocation method for large model services according to claim 4, characterized in that, The next round of personalized satisfaction estimation is performed based on the target output, resulting in a comprehensive score that incorporates subjective preferences. The calculation formula is as follows: ； in, For the next round of task types The overall score that integrates subjective preferences, To obtain the score of the i-th large model for task recognition results using a general large model in the next round, For the task type in round k Personalized satisfaction estimation of the target output For weight fusion.

8. A multimodal proxy task allocation system for large model services, used to implement the multimodal proxy task allocation method for large model services as described in any one of claims 1 to 7, comprising: The performance calculation module is used to acquire multiple large models of different types and calculate the performance of each large model under different task types. The type recognition module is used to acquire the user's multimodal task requests and use a general large model to identify the task type of the multimodal task requests; The scoring calculation module is used to obtain the score of each large model for the task identification result based on the performance of each large model under different task types, according to the task type identification result. The target model determination module is used to select the highest-scoring large model as the target model. The context encoding module is used to encode the multimodal task request and its corresponding background using the context representation module of the general large model, so as to obtain the context vector; The vector aggregation module is used to aggregate context vectors to obtain a context summary vector; The vector fusion module is used to fuse the context summary vector and the multimodal task request to obtain enhanced input; The vector update module is used to feed the augmented input into the target model, obtain the target output, and update the context summary vector using the target output; The subjective evaluation module receives subjective evaluations of the target output. Based on these evaluations, it obtains a subjective preference vector and calculates a comprehensive score that integrates these subjective preferences. This comprehensive score is then used as the updated performance of each large model under different task types. The module then moves to the type recognition module to obtain the next round of users' multimodal task requests. A general large model is used to identify the task type of the subjective preference vector, the updated context summary vector, and the next round of users' multimodal task requests. Finally, in the score calculation module, the score for each large model's task recognition result is obtained from the updated performance of each large model under different task types.

9. An electronic device, characterized in that, include: One or more processors, and a memory for storing instructions that, when executed by the one or more processors, cause the one or more processors to perform a multimodal proxy task allocation method for large model services as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, It stores executable instructions that, when executed, cause the processor to perform the multimodal proxy task allocation method for large model services as described in any one of claims 1 to 7.

Images