Multi-modal large model back-end cascade routing optimization method and system

Abstract

The invention relates to the technical field of artificial intelligence, and provides a multi-modal large model back-end cascade routing optimization method and system, and the method comprises the steps: constructing a task feature vector based on multi-modal input data and task metadata; selecting a cascade reasoning model based on the task feature vector and each model capability vector; generating a comprehensive uncertainty index based on a preliminary reasoning result output by the primary reasoning model; and generating a cascade routing decision based on the comprehensive uncertainty index. According to the multi-modal large model back-end cascade routing optimization method provided by the invention, dynamic selection of the cascade reasoning model is realized through matching of the task feature vector and the model capability vector, the cascade routing decision is generated by taking the comprehensive uncertainty index as a core control signal, and the multi-modal large model back-end cascade routing optimization is realized. According to the method, basic tasks can be quickly processed by using a lightweight model to improve the reasoning efficiency, and deep reasoning can be carried out through a model with stronger comprehensive uncertainty index triggering capability, so that multi-aspect balance of efficiency, cost and flexibility is achieved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a method and system for optimizing backend cascaded routing of multimodal large models. Background Technology

[0002] With the widespread deployment and application of multimodal large-scale models such as large language models, visual language models, speech models, and spatiotemporal models, enterprise-level artificial intelligence systems often use multiple models with different capabilities, costs, and inference performance simultaneously. However, existing multimodal large-scale model inference systems typically use a single model or fixed business flow to route user requests to specific backend models for processing. This simple routing method is difficult to use simple rules to select the appropriate model to handle complex business scenarios when processing multimodal input data. Moreover, when business needs require a multi-level inference process such as "visual model - data analysis model - text language model," existing systems lack the ability to automatically generate cascading business logic and cannot automatically construct a business flow that conforms to inference. In addition, when facing a large number of requests for simple tasks, complex large-scale models result in significant resource waste. Summary of the Invention

[0003] This application provides a multimodal large-model backend cascading routing optimization method and system to solve the problem that the lack of flexible business logic routing methods to cope with various business scenarios in the prior art leads to limited system performance and insufficient resource utilization.

[0004] Firstly, this application provides a method for optimizing backend cascading routes in a multimodal large model, including: Based on multimodal input data and task metadata, construct task feature vectors; Based on the task feature vector and the model capability vectors of each model in the backend of the multimodal large model, a cascaded inference model is selected from each of the models. The preliminary inference results output by the primary inference model for the current inference task are obtained, and a comprehensive uncertainty index is generated based on the preliminary inference results; the primary inference model is a lightweight model in the cascaded inference model whose capabilities cover basic requirements; the comprehensive uncertainty index is the main control signal for the cascaded inference upgrade. Based on the aforementioned comprehensive uncertainty index, a cascaded routing decision is generated.

[0005] In one embodiment, constructing the task feature vector based on multimodal input data and task metadata includes: Based on multimodal input data and scene metadata, identify the task type of the current inference task; Based on the task requirements of the task type and the modal quality of each modal input data, the weight value of each modal feature vector is determined; each modal feature vector is obtained by feature extraction based on its corresponding modal input data; Based on the weight values, the modal feature vectors are weighted and fused to obtain multimodal fusion features; Task metadata is embedded in the multimodal fusion features to generate a task feature vector.

[0006] In one embodiment, selecting a cascaded inference model from the models based on the task feature vector and the model capability vectors of each model in the multimodal large model backend includes: Determine the similarity between the task feature vector and the model capability vectors of each model in the backend of the multimodal large model; Select candidate models from among the models that meet the task constraints and whose similarity scores are among the top K largest; Based on the model capability level, the candidate models are hierarchically divided to construct a cascaded inference model.

[0007] In one embodiment, obtaining the preliminary inference result output by the first-level inference model for the current inference task, and generating a comprehensive uncertainty index based on the preliminary inference result, includes: Obtain the preliminary inference results output by the Level 1 inference model for the current inference task; The semantic consistency of the preliminary inference result with each modal input data is compared to generate the semantic consistency degree between the preliminary inference result and each modal input data; Based on the weight values, the semantic consistency of each value is weighted and fused to obtain a comprehensive uncertainty index.

[0008] In one embodiment, the cascaded inference model includes a first-level inference model, a second-level inference model, and a third-level inference model, wherein the model capability levels of the first-level inference model, the second-level inference model, and the third-level inference model are ranked from high to low; generating cascaded routing decisions based on the comprehensive uncertainty index includes: If the comprehensive uncertainty index is less than the first threshold, then the cascaded routing decision is determined to use the first-level inference model to analyze the current inference task; If the comprehensive uncertainty index is greater than or equal to the first threshold and less than or equal to the second threshold, then the cascaded routing decision is determined to be to use the first-level inference model and the second-level inference model to perform cascaded analysis on the current inference task; If the comprehensive uncertainty index is greater than the second threshold, then the cascaded routing decision is determined to be to use the first-level inference model, the second-level inference model and the third-level inference model to perform cascaded analysis on the current inference task; Wherein, the second threshold is greater than the first threshold.

[0009] In one embodiment, after generating the cascading routing decision, the method further includes: Based on the cascaded routing decision, the current inference task is sent to the target model in the cascaded inference model, and the inference result output by the target model is obtained; The inference results are then fused to obtain the target inference result; If the validity of the target inference result is successfully verified, it will be output.

[0010] In one embodiment, after generating the target inference result, the method further includes: Obtain user feedback data regarding the target inference result; Based on the feedback data and the operational monitoring indicators of the cascaded inference process, the first threshold and the second threshold are dynamically updated; Based on the actual performance of each model in different task dimensions of the multimodal large model backend, the model capability vector of each model is dynamically updated.

[0011] Secondly, this application also provides a multimodal large-model backend cascaded routing optimization system, including: The task feature vector construction module is used to construct task feature vectors based on multimodal input data and task metadata. The cascaded inference model construction module is used to select a cascaded inference model from each of the models based on the task feature vector and the model capability vector of each model in the backend of the multimodal large model. The comprehensive uncertainty index generation module is used to obtain the preliminary inference results output by the primary inference model for the current inference task, and generate a comprehensive uncertainty index based on the preliminary inference results; the primary inference model is a lightweight model in the cascaded inference model whose capabilities cover basic requirements; the comprehensive uncertainty index is the main control signal for the cascaded inference upgrade. The cascaded routing decision module is used to generate cascaded routing decisions based on the comprehensive uncertainty index.

[0012] Thirdly, this application provides an electronic device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the steps of any of the above-described multimodal large model backend cascaded routing optimization methods.

[0013] Fourthly, this application also provides a non-transitory computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of any of the above-described multimodal large model back-end cascaded routing optimization methods.

[0014] The multimodal large model backend cascading routing optimization method and system provided in this application constructs task feature vectors based on multimodal input data and task metadata to accurately characterize task requirements. Then, through precise matching of task feature vectors and model capability vectors, it achieves dynamic selection of cascading inference models. This not only utilizes lightweight models to quickly process basic tasks and improve inference efficiency, but also triggers more capable models for deep inference through comprehensive uncertainty indicators, significantly improving the system's ability to flexibly respond to complex business scenarios. Simultaneously, this cascading routing strategy avoids indiscriminate calls to high-computing-power models, effectively reducing redundant consumption of computing resources. While ensuring task processing accuracy, it significantly optimizes the overall system performance and resource utilization efficiency, achieving a multi-dimensional balance between efficiency, cost, and flexibility. Attached Figure Description

[0015] To more clearly illustrate the technical solutions in this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 This is a flowchart illustrating the multimodal large model backend cascaded routing optimization method provided in this application.

[0017] Figure 2 This is a schematic diagram of the structure of the multimodal large model backend cascaded routing optimization system provided in this application.

[0018] Figure 3 This is a schematic diagram of the structure of the electronic device provided in this application. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0020] The terms "first," "second," etc., used in this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein.

[0021] Existing systems lack the ability to automatically generate cascading business logic in scenarios requiring multi-level inference, and cannot automatically construct composite inference business flows. Furthermore, when dealing with numerous requests for simple, single-modal tasks, complex large models result in significant resource waste. Therefore, there is an urgent need for a method that supports multimodal input, dynamically selects models hierarchically, automatically assembles business flows, and achieves self-optimization based on feedback. This method should comprehensively consider multiple objectives such as inference quality, inference latency, computational cost, model risk (e.g., hallucination risk), and compliance to establish a routing method that automatically identifies task characteristics, intelligently constructs cascading model workflows, and optimally selects and schedules backend models in multimodal scenarios.

[0022] The following is combined with Figures 1-3 This application describes the multimodal large-model backend cascaded routing optimization method and system provided.

[0023] The multimodal large model backend cascaded routing optimization method provided in this application embodiment can be implemented based on the multimodal large model backend cascaded routing optimization system. Therefore, this application embodiment uses the multimodal large model backend cascaded routing optimization system as the execution subject to describe the multimodal large model backend cascaded routing optimization method.

[0024] Figure 1 This is a flowchart illustrating the multimodal large model backend cascaded routing optimization method provided in this application.

[0025] like Figure 1 As shown, the multimodal large model backend cascaded routing optimization method includes the following steps: Step 101: Construct task feature vectors based on multimodal input data and task metadata.

[0026] Specifically, when a user initiates a multimodal inference task request through the interactive interface, the system collects the multimodal data input by the user through a unified access interface. The multimodal large model supports input in various modal formats, including but not limited to text, images, video, audio, geographic and structured data, etc. After collecting the multimodal input data, the system also obtains metadata related to the current inference task. This metadata describes the specific requirements of the task and key information such as the scene context.

[0027] Furthermore, feature extraction algorithms are used to perform in-depth analysis and processing of multimodal input data and task metadata, extracting key feature information and integrating this information to construct a task feature vector. This task feature vector can comprehensively and accurately represent the various attributes and requirements of the current inference task, providing an important basis for subsequent model selection and inference processes, and ensuring the accuracy and relevance of the inference.

[0028] Step 102: Based on the task feature vector and the model capability vectors of each model in the backend of the multimodal large model, select a cascaded inference model from each of the models.

[0029] Specifically, after obtaining the task feature vector, it is compared and analyzed with the model capability vectors of each model in the multimodal large model backend. Each model is pre-defined and quantified according to its ability to handle inference tasks, constructing a capability profile vector to achieve a structured representation of the model's capabilities. In addition, during model operation, the capability profile vector can be continuously updated using user feedback data on inference results to reflect the model's latest performance.

[0030] By calculating the degree of matching between the task feature vector and the capability vectors of each model, the applicability of each model to the current inference task can be accurately evaluated. Based on this evaluation result, the most suitable cascaded inference model for the current inference task can be intelligently selected from the numerous models at the back end of the multimodal large model. This selection process not only considers the direct processing capability of the models but also takes into account the potential for collaborative work between models to ensure the efficiency and accuracy of cascaded inference.

[0031] Step 103: Obtain the preliminary inference results output by the primary inference model for the current inference task, and generate a comprehensive uncertainty index based on the preliminary inference results; the primary inference model is a lightweight model in the cascaded inference model whose capabilities cover basic requirements; the comprehensive uncertainty index is the main control signal for cascaded inference upgrade.

[0032] Specifically, the process begins by generating a model-understandable current inference task based on the multimodal input data. This task is then sent to a primary inference model within the cascaded inference model, which meets the requirements of low cost, low latency, and comprehensive capability coverage. The primary inference model receives the multimodal input data and utilizes its powerful algorithms and learning capabilities to perform preliminary processing and analysis, outputting preliminary inference results.

[0033] To quantitatively evaluate the reliability and accuracy of the preliminary inference results output by the first-level inference model, uncertainty indicators are extracted from dimensions such as modal consistency and cross-modal relationships based on the preliminary inference results. These indicators are then fused from multiple dimensions to form a comprehensive uncertainty indicator. This comprehensive uncertainty indicator reflects the confidence level of the first-level inference model in terms of complexity, modal alignment, and factual correctness. It serves as the main control signal for cascaded inference upgrades, determining whether an upgrade to the model's inference stage is necessary.

[0034] Step 104: Generate cascaded routing decisions based on the comprehensive uncertainty index.

[0035] Specifically, the cascaded routing decision-making mechanism adopts a two-layer framework of "static policy + learning router," meaning it combines static policy and learning router to jointly determine the model invocation path. The static policy ensures compliance and strong business constraints, which is reflected in intelligently selecting the most suitable cascaded inference model for the current inference task. The learning router automatically optimizes the decision path by analyzing historical inference quality, cost, and task characteristics to adapt to complex and ever-changing inference scenarios. This is reflected in dynamically generating cascaded routing decisions based on a comparison of comprehensive uncertainty indicators and preset thresholds.

[0036] The static strategy and the learning router work together. The static strategy provides a stable framework for the entire decision-making process, ensuring that the inference task is carried out in accordance with regulations and business requirements. The learning router, within this framework, flexibly optimizes the model call path based on the reliability and accuracy of the preliminary inference results reflected by the comprehensive uncertainty index.

[0037] The multimodal large-model backend cascading routing optimization method provided in this application constructs task feature vectors based on multimodal input data and task metadata to accurately characterize task requirements. Then, through precise matching of task feature vectors and model capability vectors, it achieves dynamic selection of cascading inference models. This method can utilize lightweight models to quickly process basic tasks to improve inference efficiency, while triggering more capable models for deep inference through comprehensive uncertainty indicators, significantly improving the system's ability to flexibly respond to complex business scenarios. Simultaneously, this cascading routing strategy avoids indiscriminate use of high-computing-power models, effectively reducing redundant consumption of computing resources. While ensuring task processing accuracy, it significantly optimizes the overall system performance and resource utilization efficiency, achieving a multi-dimensional balance between efficiency, cost, and flexibility.

[0038] In one embodiment, based on step 101, constructing the task feature vector based on multimodal input data and task metadata includes: Based on multimodal input data and scene metadata, identify the task type of the current inference task; Based on the task requirements of the task type and the modal quality of each modal input data, the weight value of each modal feature vector is determined; each modal feature vector is obtained by feature extraction based on its corresponding modal input data; Based on the weight values, the modal feature vectors are weighted and fused to obtain multimodal fusion features; Task metadata is embedded in the multimodal fusion features to generate a task feature vector.

[0039] Specifically, multimodal input data is adopted through a unified access interface, including but not limited to text, images, video, audio, geographic and structured data. After collecting multimodal input data, scenario metadata related to the current inference task is simultaneously acquired, including but not limited to contextual information such as tenant identifier, privacy level, cost budget, and latency requirements. The tenant identifier distinguishes the permissions and resource quotas of different tenants, ensuring that each tenant can only call the models they are authorized to, avoiding resource misuse or unauthorized access, and supporting tenant-level routing policy customization to improve service personalization and compliance. Models that meet data protection requirements are selected based on the tenant's privacy level, ensuring data security and compliance. The cost budget is a hard constraint for model selection, calling available models based on the cost budget to optimize resource allocation and cost control. Latency requirements are used to select model instances to improve system response efficiency. Format validation, permission verification, and storage reference association are performed on the acquired data, ensuring that the task has complete metadata and traceability before entering the backend.

[0040] Targeted preprocessing is performed on data of different modalities, including but not limited to text normalization, image resampling and enhancement, video frame extraction and scene detection, audio noise reduction and feature extraction, geographic data coordinate transformation, and structured data pattern verification, to ensure the consistency of multimodal input data in terms of quality, scale and format, and to reduce the impact of noise.

[0041] By utilizing multimodal input data and scene metadata, a lightweight task classification model is used to identify the task type of the current inference task. The identification results are then used as the basis for subsequent model selection and routing decisions, thus constraining the set of available models.

[0042] Digital encoding is used to encode each modal input data into feature vectors, and semantic alignment between modalities is achieved through cross-modal models. A modality weight learning and gating mechanism is introduced to dynamically adjust the contribution of each modality based on the task requirements and the quality of the modal data for this task type, determine the weight values ​​of each modality feature vector for the current inference task, and then perform weighted fusion of these modality feature vectors to obtain multimodal fusion features, ensuring that the fusion representation accurately reflects the core semantics of the task.

[0043] The multimodal fusion features are reduced to or projected to generate a feature vector of a unified dimension, and task metadata is embedded in it, including but not limited to task semantics, modal requirements, task complexity, domain attributes, and business constraints of scene metadata, and finally the task feature vector is generated.

[0044] This application's embodiments employ multimodal input signal normalization, internal cohesion, and modal-level preprocessing refinement to embed multimodal input data. Through automatic multimodal task type identification and cross-modal alignment, it constructs a unified task feature vector for multimodal input acquisition, preprocessing, and implementation, thereby achieving semantic, parameterized, and structured representation of tasks. This provides a reliable and comprehensive basis for subsequent backend filtering and routing decisions.

[0045] In one embodiment, based on step 102, selecting a cascaded inference model from each of the models based on the task feature vector and the model capability vectors of each model in the multimodal large model backend includes: Determine the similarity between the task feature vector and the model capability vectors of each model in the backend of the multimodal large model; Select candidate models from among the models that meet the task constraints and whose similarity scores are among the top K largest; Based on the model capability level, the candidate models are hierarchically divided to construct a cascaded inference model.

[0046] Specifically, each model in the multimodal large model backend is pre-defined and quantified based on its ability to handle inference tasks, constructing a capability profile vector. First, after training, the model undergoes pre-evaluation on a standard test set to obtain offline evaluation results. Then, based on these results, quantitative analysis is performed across dimensions such as inference ability, visual understanding, structured stability, domain performance, security, latency, and cost. This is then transformed into a fixed-dimensional vector through dimensionality reduction or embedding mapping. This vector serves as a unified representation of the model's capabilities, enabling vectorized matching between task vectors and model capabilities. Furthermore, the model capability vector is dynamically updated using metrics collected during model operation, such as quality feedback, latency performance, error types, and resource consumption, to reflect the model's latest capability status in the real-world environment. During the update process, time decay, robust statistics, and anomaly filtering strategies are employed to ensure that the capability profile adapts promptly to model version changes and task distribution shifts.

[0047] After generating the task feature vector, the latest model capability vector of each model is obtained. The similarity between the task feature vector and the capability vector of each model is calculated. Then, multiple rounds of filtering and similarity ranking are performed in combination with multiple task constraints such as privacy, cost, regional compliance, modal support and system load. Candidate models with higher similarity scores and satisfying multiple task constraints are selected.

[0048] After selecting candidate models, they are categorized into hierarchical levels based on predefined model capability standards. These standards can be set according to the comprehensive performance of models across key dimensions such as reasoning ability, visual understanding, structured stability, domain performance, security, latency, and cost. For example, models can be divided into three levels: beginner, intermediate, and advanced, each corresponding to a different capability range and applicable scenarios. This hierarchical division allows for the construction of a cascaded inference model, consisting of multiple model pools at different levels, each responsible for handling inference tasks of specific difficulty or type. During inference, the cascaded inference model sequentially calls models from lower to higher levels based on multimodal input data, until a reasoning result meeting the task requirements is obtained. This cascaded inference approach fully leverages the advantages of different model levels, improving the accuracy and efficiency of inference.

[0049] In this embodiment, a capability profile vector is constructed for each backend model to achieve a structured representation of the model's capabilities. During actual operation, the capability profile vector is continuously updated using feedback data to reflect the latest performance of the model. Subsequently, the task feature vector and the model capability vector are semantically similarly matched, and a candidate model set is selected by combining multiple constraints to build a model pool for the cascaded link, which can provide more accurate and suitable model resources for subsequent cascaded inference.

[0050] In one embodiment, based on step 103, obtaining the preliminary inference result output by the first-level inference model for the current inference task, and generating a comprehensive uncertainty index based on the preliminary inference result, includes: Obtain the preliminary inference results output by the Level 1 inference model for the current inference task; The semantic consistency of the preliminary inference result with each modal input data is compared to generate the semantic consistency degree between the preliminary inference result and each modal input data; Based on the weight values, the semantic consistency of each value is weighted and fused to obtain a comprehensive uncertainty index.

[0051] Specifically, the process begins by invoking the first-round inference model within the cascaded inference model. The first-level inference model receives multimodal input data and utilizes its powerful algorithms and learning capabilities to perform preliminary processing and analysis, outputting preliminary inference results. Simultaneously, it records intermediate information such as the attention matrix, inference latency, and token statistics during the inference process. This model's output serves as the basis for subsequent quality assessments, generating a comprehensive uncertainty evaluation index. This index is used to determine whether an upgrade to a stronger model is necessary.

[0052] After obtaining the initial inference results, to ensure the reliability and accuracy of the inference, it is necessary to further evaluate the consistency between the initial inference results and the input data. First, a cross-modal matching model is used to perform a detailed semantic consistency comparison between the initial inference results and the input data for each modality. This aims to quantify the degree of semantic fit between the initial inference results and the original data, thereby generating the semantic consistency score of the initial inference results across each modality, which serves as an uncertainty indicator. If inconsistencies in entities, attributes, or logic are found in the initial inference results, the consistency score will decrease, indicating a cross-modal risk in the first-level inference.

[0053] In addition, based on the preliminary inference results and intermediate information during the inference process, we can determine dimension data such as probability entropy, confidence level, inter-frame feature stability, and cross-modal semantic matching degree, and take these dimension data into account as uncertainty indicators.

[0054] Furthermore, based on the importance and influence of each modal input data in the overall inference process, corresponding weight values ​​are assigned to each modality. Simultaneously, corresponding weight values ​​can also be assigned to other dimensions of data. Then, based on the weight values ​​of each dimension, the uncertainty indicators are weighted and fused to form a comprehensive uncertainty indicator. This comprehensive uncertainty indicator serves as the main control signal for cascaded upgrades, determining whether to proceed to the advanced model inference stage.

[0055] This application extracts uncertainty indicators from multiple dimensions and integrates them into a comprehensive uncertainty index. This comprehensive uncertainty index can reflect the confidence level of the first-level inference model in various aspects of inference, and can more objectively evaluate the reliability and accuracy of the preliminary inference results, thereby making more reasonable model selection and adjustment in the cascade inference process.

[0056] In one embodiment, based on step 104, the cascaded inference model includes a first-level inference model, a second-level inference model, and a third-level inference model, wherein the model capability levels of the first-level inference model, the second-level inference model, and the third-level inference model are ranked from high to low; generating cascaded routing decisions based on the comprehensive uncertainty index includes: If the comprehensive uncertainty index is less than the first threshold, then the cascaded routing decision is determined to use the first-level inference model to analyze the current inference task; If the comprehensive uncertainty index is greater than or equal to the first threshold and less than or equal to the second threshold, then the cascaded routing decision is determined to be to use the first-level inference model and the second-level inference model to perform cascaded analysis on the current inference task; If the comprehensive uncertainty index is greater than the second threshold, then the cascaded routing decision is determined to be to use the first-level inference model, the second-level inference model and the third-level inference model to perform cascaded analysis on the current inference task; Wherein, the second threshold is greater than the first threshold.

[0057] Specifically, the cascaded inference model consists of multiple model pools at different levels. Taking this embodiment as an example, the cascaded inference model comprises a first-level model pool, a second-level model pool, and a third-level model pool. The first-level model pool stores first-level inference models, which are low-cost, low-latency basic models covering simple tasks. The second-level model pool stores second-level inference models, which are intermediate-level models used to handle tasks with higher uncertainty. Their specific task-processing capabilities are used to review the output of the first-level inference models. The third-level model pool stores third-level inference models, which are high-level models used to handle tasks with high uncertainty or high risk. Their specific task-processing capabilities are used to review the output of the first-level and / or second-level inference models. The first-level inference models are low-cost and highly efficient, used to offload most simple tasks; the second and third-level inference models have increasing costs and capabilities, and are only invoked when necessary to balance cost and quality.

[0058] Before adopting a cascaded routing decision mechanism, a first threshold (low threshold) and a second threshold (high threshold) are defined to determine whether to directly accept the inference results of the first-level inference model or to upgrade to an intermediate or advanced model for inference.

[0059] If the overall uncertainty index is less than the first threshold, then it is only necessary to send the current inference task to the first-level inference model. The first-level inference model analyzes the current inference task. Since the first-level inference model mainly handles simple tasks, it is fast and inexpensive, and can quickly provide preliminary analysis results. These preliminary results are sufficient for scenarios with simple single-modal tasks, thus achieving efficient resource utilization.

[0060] If the overall uncertainty index is greater than or equal to the first threshold and less than or equal to the second threshold, the current inference task needs to be sent to the first-level inference model and the second-level inference model. The first-level and second-level inference models then perform cascaded analysis on the current inference task. Specifically, the first-level inference model first processes the data, and then the results are passed to the second-level inference model. The second-level inference model reviews the results from the first-level inference model to improve the accuracy of the analysis. This cascaded analysis method improves the quality of analysis while maintaining a certain processing speed, making it suitable for scenarios where certain accuracy requirements are not particularly high.

[0061] If the overall uncertainty index exceeds the second threshold, the current inference task needs to be sent to the Level 1, Level 2, and Level 3 inference models. These models then perform a cascaded analysis: the Level 1 model performs initial processing, the Level 2 model reviews the Level 1 results, and the Level 3 model conducts in-depth analysis of the outputs from the first two levels. As a high-capacity model, the Level 3 model can handle the most complex and difficult tasks, ensuring high accuracy and reliability in the final analysis results. While this cascaded analysis method is more costly, it provides the highest quality analysis service and is suitable for scenarios with extremely high accuracy requirements.

[0062] Furthermore, the key-value cache state of each session is recorded, and upgrade inference is preferentially routed to instances with caches to reduce the computational cost of recoding. For cross-model upgrade scenarios, cache migration or compatible mapping strategies are employed to significantly reduce latency and improve inference efficiency for long-context tasks.

[0063] This application's embodiments achieve multimodal inference cost reduction by using a first-level small model to offload tasks, enabling simple tasks to call only lightweight models and complex tasks to trigger high-level models. Furthermore, lightweight models complete most tasks, and perceptual routing reduces redundant calculations, lowers overall system latency, and improves response speed. Through a cascaded routing decision mechanism, appropriate model combinations can be dynamically selected based on the uncertainty of the task, ensuring that system resources are maximized while meeting task requirements and improving inference efficiency. This provides an efficient and reliable solution for multimodal data processing.

[0064] In one embodiment, after generating the cascading routing decision, step 105 is further included: Based on the cascaded routing decision, the current inference task is sent to the target model in the cascaded inference model, and the inference result output by the target model is obtained; The inference results are then fused to obtain the target inference result; If the validity of the target inference result is successfully verified, it will be output.

[0065] Specifically, based on the determined cascaded routing decision, the corresponding target model in the cascaded inference model is invoked to perform cascaded analysis on the current inference task, and the inference results are fused to obtain the target inference result. If only a single-level inference model is used, the fused result of the cascaded inference results is the preliminary inference result itself; if a multi-level inference model is used, the multi-level inference results are fused.

[0066] The fusion method supports various strategies such as optimization, substitution, and integration, and can automatically select the optimal fusion method based on task difficulty, comprehensive uncertainty indicators, model capabilities, and historical performance. For example, simple tasks may prioritize the "optimization" strategy (directly using the output of a high-confidence model), while complex cross-modal tasks may choose the "integration" strategy (fusing the outputs of multiple models). During the fusion process, the semantic differences, confidence scores, and format regularity of the multi-model outputs are considered to improve the quality and stability of the results. The fusion decision model is trained using automatic evaluation and user feedback to learn the reliability and bias characteristics of different models in different tasks, dynamically adjusting fusion weights and strategy priorities to optimize the fusion results.

[0067] Furthermore, the target inference result undergoes validity checks, such as semantic consistency, cross-modal matching, structured formatting, and content security checks, to identify factual errors, pattern violations, or potential risks. When the validity check of the target inference result is successful, the result is output; when the check fails, for example, due to deviations in the fusion result, automatic regeneration of the inference result or downgrade processing is triggered. This validity check process provides reliable assurance for the output result, enabling timely triggering of correction mechanisms when potential problems are discovered, thereby significantly improving the overall performance and practicality of multimodal large model backend cascading routing optimization.

[0068] This application embodiment introduces a cascaded routing decision mechanism to enable the backend inference engine to dynamically select inference paths according to task requirements. By combining the complementary advantages of multi-level inference models, it effectively solves the limitations of a single model in complex tasks, reduces the illusion problem in system response, and significantly improves the system output quality through multi-level model collaboration.

[0069] In one embodiment, after generating the target inference result, step 106 is further included: Obtain user feedback data regarding the target inference result; Based on the feedback data and the operational monitoring indicators of the cascaded inference process, the first threshold and the second threshold are dynamically updated; Based on the actual performance of each model in different task dimensions of the multimodal large model backend, the model capability vector of each model is dynamically updated.

[0070] Specifically, after completing the task inference, user feedback data on the output results can be obtained. There are various ways to obtain user feedback data, such as collecting user satisfaction and accuracy evaluations of the target inference results through online questionnaires, user rating systems, or direct interactive interfaces. This feedback data is an important basis for optimizing model performance.

[0071] Based on explicit user feedback, implicit behavior, automatic quality assessment, and operational monitoring metrics, and associated with task feature vectors, model call paths, and comprehensive uncertainty metrics, a training data pool is formed that can be used for route optimization and capability updates.

[0072] Incremental training of the learning router is performed based on feedback data. Statistical analysis or optimization algorithms are used to automatically adjust the first and second update thresholds to achieve an optimal balance between upgrade ratio and quality improvement. The corresponding model capability vector is dynamically adjusted according to the model's real performance in different task dimensions, enhancing advantageous dimensions and reducing weak dimensions, ensuring that the capability profile continuously closely reflects actual capabilities. Based on feedback data and the model's actual capabilities, the fusion strategy and weight allocation are optimized. The optimization effect can be verified through canary releases and A / B testing mechanisms to ensure the security and controllability of updates.

[0073] Based on the latest feedback data and operation monitoring data, the embodiments of this application continuously train the learning router, automatically adjust the cascading threshold, update the model capability profile, and optimize the fusion strategy and weight allocation, etc., to achieve feedback closed-loop learning and optimization. This optimization mechanism enables the back-end cascading routing optimization of multimodal large models to better adapt to changes in user needs, improve the accuracy of inference results and user satisfaction.

[0074] In summary, the multimodal large-scale model backend cascading routing optimization method provided in this application automatically identifies the modality type and business objective of multimodal inputs through methods such as multimodal task feature vector representation, model capability profiling, and uncertainty-driven cascading routing and multi-model result fusion. It selects the optimal multi-model call path, providing enterprise-level multimodal large-scale model platforms with intelligent model selection and scheduling capabilities that integrate quality, cost, and latency optimization. It establishes a new routing system that automatically identifies task characteristics and intelligently constructs model cascading processes in multimodal scenarios, supporting scalable model registration and capability awareness, and enabling hot-plugging of multiple models and intelligent scheduling of models across multiple domains. This solves the problems of reliance on manual experience for model selection, high call costs, unstable service quality, and unexplainable routing strategies in current multi-model coexistence scenarios. It ensures a balance of interests among platform builders, model providers, and business users in terms of performance, cost, and risk control, improving the reliability, controllability, overall operational efficiency, and transparency of multimodal large-scale model application systems.

[0075] Figure 2 This is a schematic diagram of the structure of the multimodal large model backend cascaded routing optimization system provided in this application.

[0076] like Figure 2 As shown, the multimodal large model backend cascaded routing optimization system includes: The task feature vector construction module 210 is used to construct task feature vectors based on multimodal input data and task metadata; The cascaded inference model construction module 220 is used to select a cascaded inference model from each of the models based on the task feature vector and the model capability vector of each model in the backend of the multimodal large model. The comprehensive uncertainty index generation module 230 is used to obtain the preliminary inference results output by the primary inference model for the current inference task, and generate a comprehensive uncertainty index based on the preliminary inference results; the primary inference model is a lightweight model in the cascaded inference model whose capabilities cover basic requirements; the comprehensive uncertainty index is the main control signal for the cascaded inference upgrade. The cascaded routing decision module 240 is used to generate cascaded routing decisions based on the comprehensive uncertainty index.

[0077] The multimodal large-model backend cascading routing optimization system provided in this application constructs task feature vectors based on multimodal input data and task metadata to accurately characterize task requirements. Then, through precise matching of task feature vectors and model capability vectors, it achieves dynamic selection of cascading inference models. This allows for the rapid processing of basic tasks using lightweight models to improve inference efficiency, while also triggering more capable models for deep inference through comprehensive uncertainty indicators, significantly enhancing the system's flexibility in handling complex business scenarios. Simultaneously, this cascading routing strategy avoids indiscriminate use of high-computing-power models, effectively reducing redundant consumption of computing resources. While ensuring task processing accuracy, it significantly optimizes the overall system performance and resource utilization efficiency, achieving a multi-dimensional balance between efficiency, cost, and flexibility.

[0078] In one embodiment, the task feature vector construction module 210 is further configured to: Based on multimodal input data and scene metadata, identify the task type of the current inference task; Based on the task requirements of the task type and the modal quality of each modal input data, the weight value of each modal feature vector is determined; each modal feature vector is obtained by feature extraction based on its corresponding modal input data; Based on the weight values, the modal feature vectors are weighted and fused to obtain multimodal fusion features; Task metadata is embedded in the multimodal fusion features to generate a task feature vector.

[0079] In one embodiment, the cascaded inference model building module 220 is further configured to: Determine the similarity between the task feature vector and the model capability vectors of each model in the backend of the multimodal large model; Select candidate models from among the models that meet the task constraints and whose similarity scores are among the top K largest; Based on the model capability level, the candidate models are hierarchically divided to construct a cascaded inference model.

[0080] In one embodiment, the comprehensive uncertainty index generation module 230 is further configured to: Obtain the preliminary inference results output by the Level 1 inference model for the current inference task; The semantic consistency of the preliminary inference result with each modal input data is compared to generate the semantic consistency degree between the preliminary inference result and each modal input data; Based on the weight values, the semantic consistency of each value is weighted and fused to obtain a comprehensive uncertainty index.

[0081] In one embodiment, the cascaded routing decision module 240 is further configured to: If the comprehensive uncertainty index is less than the first threshold, then the cascaded routing decision is determined to use the first-level inference model to analyze the current inference task; If the comprehensive uncertainty index is greater than or equal to the first threshold and less than or equal to the second threshold, then the cascaded routing decision is determined to be to use the first-level inference model and the second-level inference model to perform cascaded analysis on the current inference task; If the comprehensive uncertainty index is greater than the second threshold, then the cascaded routing decision is determined to be to use the first-level inference model, the second-level inference model and the third-level inference model to perform cascaded analysis on the current inference task; Wherein, the second threshold is greater than the first threshold.

[0082] In one embodiment, the multimodal large model backend cascaded routing optimization system further includes an inference result fusion and output module, which is further used for: Based on the cascaded routing decision, the current inference task is sent to the target model in the cascaded inference model, and the inference result output by the target model is obtained; The inference results are then fused to obtain the target inference result; If the validity of the target inference result is successfully verified, it will be output.

[0083] In one embodiment, the multimodal large model backend cascaded routing optimization system further includes a feedback closed-loop learning module, which is further used for: Obtain user feedback data regarding the target inference result; Based on the feedback data and the operational monitoring indicators of the cascaded inference process, the first threshold and the second threshold are dynamically updated; Based on the actual performance of each model in different task dimensions of the multimodal large model backend, the model capability vector of each model is dynamically updated.

[0084] It should be noted that the multimodal large model backend cascaded routing optimization system provided in this application can execute the multimodal large model backend cascaded routing optimization method described in any of the above embodiments during actual operation, which will not be elaborated in this embodiment.

[0085] Figure 3 This is a schematic diagram of the structure of the electronic device provided in this application, such as... Figure 3 As shown, the electronic device may include: a processor 310, a communication interface 320, a memory 330, and a communication bus 340, wherein the processor 310, the communication interface 320, and the memory 330 communicate with each other through the communication bus 340. The processor 310 can call logical instructions in the memory 330 to execute a multimodal large model backend cascaded routing optimization method. This method includes: constructing a task feature vector based on multimodal input data and task metadata; selecting a cascaded inference model from each of the models based on the task feature vector and the model capability vectors of each model in the multimodal large model backend; obtaining the preliminary inference result output by the first-level inference model for the current inference task, and generating a comprehensive uncertainty index based on the preliminary inference result; the first-level inference model is a lightweight model whose capability covers basic requirements in the cascaded inference model; the comprehensive uncertainty index is the main control signal for cascaded inference upgrade; generating a cascaded routing decision based on the comprehensive uncertainty index; and generating a target inference result based on the cascaded routing decision.

[0086] Furthermore, the logical instructions in the aforementioned memory 330 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0087] On the other hand, this application also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium. The computer program includes program instructions, and when the program instructions are executed by a computer, the computer can execute the multimodal large model backend cascaded routing optimization method provided in the above embodiments. The method includes: constructing a task feature vector based on multimodal input data and task metadata; selecting a cascaded inference model from each of the models based on the task feature vector and the model capability vector of each model in the multimodal large model backend; obtaining the preliminary inference result output by the primary inference model for the current inference task, and generating a comprehensive uncertainty index based on the preliminary inference result; the primary inference model is a lightweight model whose capability covers basic requirements in the cascaded inference model; the comprehensive uncertainty index is the main control signal for cascaded inference upgrade; generating a cascaded routing decision based on the comprehensive uncertainty index; and generating a target inference result based on the cascaded routing decision.

[0088] In another aspect, this application also provides a non-transitory computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program is implemented to perform the multimodal large model backend cascaded routing optimization method provided in the above embodiments. The method includes: constructing a task feature vector based on multimodal input data and task metadata; selecting a cascaded inference model from each of the models based on the task feature vector and the model capability vectors of each model in the multimodal large model backend; obtaining the preliminary inference result output by the primary inference model for the current inference task, and generating a comprehensive uncertainty index based on the preliminary inference result; the primary inference model is a lightweight model whose capability covers basic requirements in the cascaded inference model; the comprehensive uncertainty index is the main control signal for cascaded inference upgrade; generating a cascaded routing decision based on the comprehensive uncertainty index; and generating a target inference result based on the cascaded routing decision.

[0089] The system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0090] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0091] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A method for optimizing backend cascaded routing in a multimodal large model, characterized in that, The multimodal large model backend cascade routing optimization method includes: Based on multimodal input data and task metadata, construct task feature vectors; Based on the task feature vector and the model capability vectors of each model in the backend of the multimodal large model, a cascaded inference model is selected from each of the models. The system obtains the preliminary inference results output by the primary inference model for the current inference task, and generates a comprehensive uncertainty index based on the preliminary inference results; the primary inference model is a lightweight model in the cascaded inference model whose capabilities cover basic requirements; the comprehensive uncertainty index is the main control signal for the cascaded inference upgrade. Based on the aforementioned comprehensive uncertainty index, a cascaded routing decision is generated.

2. The multimodal large model backend cascaded routing optimization method according to claim 1, characterized in that, The construction of task feature vectors based on multimodal input data and task metadata includes: Based on multimodal input data and scene metadata, identify the task type of the current inference task; Based on the task requirements of the task type and the modal quality of each modal input data, the weight value of each modal feature vector is determined; each modal feature vector is obtained by feature extraction based on its corresponding modal input data; Based on the weight values, the modal feature vectors are weighted and fused to obtain multimodal fusion features; Task metadata is embedded in the multimodal fusion features to generate a task feature vector.

3. The multimodal large model backend cascaded routing optimization method according to claim 1, characterized in that, The step of selecting a cascaded inference model from each of the models based on the task feature vector and the model capability vectors of each model in the multimodal large model backend includes: Determine the similarity between the task feature vector and the model capability vectors of each model in the backend of the multimodal large model; Select candidate models from among the models that meet the task constraints and whose similarity scores are among the top K largest; Based on the model capability level, the candidate models are hierarchically divided to construct a cascaded inference model.

4. The multimodal large model backend cascaded routing optimization method according to claim 2, characterized in that, The process of obtaining the preliminary inference results output by the first-level inference model for the current inference task, and generating a comprehensive uncertainty index based on the preliminary inference results, includes: Obtain the preliminary inference results output by the Level 1 inference model for the current inference task; The semantic consistency of the preliminary inference result with each modal input data is compared to generate the semantic consistency degree between the preliminary inference result and each modal input data; Based on the weight values, the semantic consistency of each value is weighted and fused to obtain a comprehensive uncertainty index.

5. The multimodal large model backend cascaded routing optimization method according to claim 1, characterized in that, The cascaded reasoning model includes a first-level reasoning model, a second-level reasoning model, and a third-level reasoning model, with the model capability levels corresponding to the first-level reasoning model, the second-level reasoning model, and the third-level reasoning model ranging from high to low. The step of generating cascaded routing decisions based on the comprehensive uncertainty index includes: If the comprehensive uncertainty index is less than the first threshold, then the cascaded routing decision is determined to use the first-level inference model to analyze the current inference task; If the comprehensive uncertainty index is greater than or equal to the first threshold and less than or equal to the second threshold, then the cascaded routing decision is determined to be to use the first-level inference model and the second-level inference model to perform cascaded analysis on the current inference task; If the comprehensive uncertainty index is greater than the second threshold, then the cascaded routing decision is determined to be to use the first-level inference model, the second-level inference model and the third-level inference model to perform cascaded analysis on the current inference task; Wherein, the second threshold is greater than the first threshold.

6. The multimodal large model backend cascaded routing optimization method according to claim 1, characterized in that, After generating the cascading routing decision, the following is also included: Based on the cascaded routing decision, the current inference task is sent to the target model in the cascaded inference model, and the inference result output by the target model is obtained; The inference results are then fused to obtain the target inference result; If the validity of the target inference result is successfully verified, it will be output.

7. The multimodal large model backend cascaded routing optimization method according to claim 5, characterized in that, After generating the target inference result, it also includes: Obtain user feedback data regarding the target inference result; Based on the feedback data and the operational monitoring indicators of the cascaded inference process, the first threshold and the second threshold are dynamically updated; Based on the actual performance of each model in different task dimensions of the multimodal large model backend, the model capability vector of each model is dynamically updated.

8. A multimodal large-model backend cascaded routing optimization system, characterized in that, The multimodal large model backend cascaded routing optimization system includes: The task feature vector construction module is used to construct task feature vectors based on multimodal input data and task metadata. The cascaded inference model construction module is used to select a cascaded inference model from each of the models based on the task feature vector and the model capability vector of each model in the backend of the multimodal large model. The comprehensive uncertainty index generation module is used to obtain the preliminary inference results output by the primary inference model for the current inference task, and generate a comprehensive uncertainty index based on the preliminary inference results; the primary inference model is a lightweight model in the cascaded inference model whose capabilities cover basic requirements; the comprehensive uncertainty index is the main control signal for cascaded inference upgrade; The cascaded routing decision module is used to generate cascaded routing decisions based on the comprehensive uncertainty index.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the multimodal large model back-end cascaded routing optimization method as described in any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium, wherein a computer program is stored on the non-transitory computer-readable storage medium, characterized in that, When the computer program is executed by the processor, it implements the steps of the multimodal large model back-end cascaded routing optimization method as described in any one of claims 1 to 7.

Images