Model reasoning service judgment system and method based on intention engine

Abstract

The invention discloses a model reasoning service judgment system and method based on an intention engine, which are used for sequentially completing service mode selection, model instance selection and routing execution, result output and closed-loop updating of strategy parameters by taking request characteristics and a system resource state as input for each reasoning request. According to the method, mechanisms such as request-level dual-mode switching, semantic and resource joint judgment, multi-dimensional result judgment and metadata-based closed-loop learning are integrated in the same edge inference system, so that the edge multi-model inference system can consider inference quality and resource utilization efficiency while ensuring a service level target.

Description

Technical Field

[0001] This invention belongs to the field of model reasoning service technology, specifically relating to a model reasoning service decision system and method based on an intent engine. Background Technology

[0002] In edge computing and distributed AI applications, to meet diverse business needs, systems typically deploy various AI models with different functions and complementary capabilities (such as large language models, speech models, and high-precision CV models) on heterogeneous edge nodes. However, how to accurately and efficiently assign the most suitable model to each inference request is a key technical problem in current system design. In early deployments, clients typically accessed specific service instances directly, relying heavily on hard-coded IP addresses or simple configurations for routing, lacking a unified gateway and intelligent routing mechanism, which resulted in inflexibility. As systems scale up, a more common deployment approach is for upstream inference requests to be processed through a static routing gateway, such as... Figure 1 As shown, the gateway hard-assigns requests to specific model instances in the edge computing environment based on a static rule base (e.g., If-Then rules based on the type of input requested).

[0003] In such Figure 1 In the static rule-based routing model shown, service deployment largely relies on the aforementioned static service routing strategy, which hard-maps requests to specific model services through predefined rules. It's important to note that "routing" here does not refer to network-layer packet forwarding based on IP addresses, but rather to application-layer model service distribution. This static routing strategy faces limitations in practical applications. When the intent of a user request is ambiguous or crosses the capability boundaries of multiple models, rigid rule matching may lead to improper route allocation, thus affecting service quality. Furthermore, as the model library expands and version iterations accelerate, the complexity and cost of maintaining If-Then rule sets increase, and it's difficult to cover long-tail scenarios. Looking further, even if existing intelligent routing solutions incorporate multiple factors (such as request type / origin, node load, GPU and memory usage, queue length, and estimated latency), their decision-making logic is usually fixed: either selecting an optimal instance (single endpoint) based on load balancing, or executing pre-configured A / B testing or model orchestration processes. These solutions all lack a dynamic service mode decision layer coupled with intent. In other words, it lacks the ability to adaptively switch between "pre-decision" (single-model direct access) and "post-decision" (multi-model adjudication) service modes based on real-time resource availability and the uncertainty of request intent; resource factors are typically only used as static thresholds or weighted scores in the decision-making process. Consequently, its ability to simultaneously satisfy latency constraints and result reliability is limited in scenarios with fluctuating loads or semantic ambiguity.

[0004] While the industry has introduced intent engines to optimize routing decisions, existing solutions have significant limitations in both the breadth of multimodal understanding and the depth of resource adaptation. On the one hand, existing intent engines are mostly limited to a single text modality, lacking the ability to deeply fuse cross-modal information such as images and speech. This leads to the system often incorrectly assigning requests to a general language model when processing text-based queries or multi-dimensional alarms in industrial settings, due to ignoring key visual or auditory features, severely impacting recognition accuracy. On the other hand, in environments with limited edge computing power and bandwidth, existing engines lack awareness and adaptive mechanisms for real-time resource status. They typically use pre-trained models for inference in a rigid manner, unable to dynamically adjust paths based on node load or latency budget. This rigid mechanism causes the system to experience increased latency during peak business periods due to the high overhead of intent recognition itself, while under computing power constraints, it is forced to passively degenerate into simple rule-based routing, resulting in drastic fluctuations in service quality.

[0005] Furthermore, existing systems generally lack a quality judgment step for inference results after request distribution: when the selected model is mismatched with the task and produces low-quality or incorrect answers, the system usually returns the results directly to the business side, lacking a mechanism to automatically trigger alternative strategies. For example, when it detects that the answer contains missing key information, factual inconsistencies, or does not meet business constraints, existing systems often cannot automatically switch to a more accurate model for re-inference, nor can they promptly initiate multi-model joint inference and adjudicate multiple results. This results in erroneous results being received by the business as normal responses, and the lack of a feedback loop makes it difficult to continuously calibrate routing boundaries and thresholds. Summary of the Invention

[0006] In view of the above-mentioned shortcomings in the prior art, the model reasoning service decision system and method based on intent engine provided by the present invention overcomes the rigidity of the existing static routing mechanism in resource-constrained and heterogeneous edge computing environments, solves the mismatch between the intent complexity of reasoning requests, real-time resource status and service instance selection, and the resulting imbalance between the accuracy of reasoning results and system resource consumption.

[0007] To achieve the aforementioned objectives, the present invention employs the following technical solution: a model reasoning service decision system based on an intent engine, comprising: The inference request input module is used to receive the user's original inference request and its associated SLO project field, and transmit them to the intent engine module and the inference execution module respectively. The intent engine module is used to perform intent analysis and feature extraction on the original inference request during the request phase, generating request-side feature vectors and a first-level candidate set of tasks to the model; and, in the post-decision mode, to review and score the multi-path candidate outputs generated by the inference execution module. The resource status awareness module is used to collect resource monitoring data from each edge node and model instance, construct resource status vectors, calculate risk metrics on the resource side, and transmit them to the mode selection and routing decision module. The mode selection and routing decision module is used to determine the decision mode based on the request side feature vector, first-level candidate set, resource status vector and risk metric, combined with policy parameters through two-level screening and adaptation calculation, and to send the corresponding routing plan to the inference execution module; and, in the post-decision mode, to call the review score to make a post-decision decision to select the optimal output and transmit it to the result output and recording module. The decision mode includes a pre-decision mode and a post-decision mode, and the corresponding routing plans are a single routing plan and a parallel / cascaded routing plan, respectively. The inference execution module is used to perform inference on the original inference request according to the routing plan and record the running statistics of each inference service path in real time. In the pre-decision mode, it transmits the single-path inference result and its running statistics to the result output and recording module. In the post-decision mode, it feeds back the generated multi-path candidate output and running statistics to the mode selection and routing decision module. The results output and recording module is used to construct a unified response from the received data and feed it back to the upstream business system, and to log the entire process of model inference service decision and update the strategy parameters through the feedback link.

[0008] Furthermore, the mode selection and routing decision module includes a policy parameter learning submodule; The strategy parameter learning submodule is used to estimate and calibrate the strategy parameters offline based on the system's historical operation logs, and update the strategy parameters by minimizing empirical risks.

[0009] A model reasoning service decision method based on an intent engine includes the following steps: S100: Receive the user's original inference request and its associated SLO project field through the inference request input module, and transmit them to the intent engine module and the inference execution module respectively. S200. In the intent engine module, during the request phase, intent analysis and feature extraction are performed on the original inference request to generate a request-side feature vector. And based on the adaptation function of task type and model capability label, construct the candidate mapping relationship from task to model, and then construct the first-level candidate set of functional adaptation; S300: Collect resource monitoring data of each edge node and model instance through the resource status perception module, and then construct resource status vector and calculate risk measurement on the resource side; S400. In the mode selection and routing decision module, the decision mode is determined by combining the request side feature vector, first-level candidate set, resource status vector and risk metric with the strategy parameters through two-level screening and adaptation calculation, and the corresponding routing plan is sent to the inference execution module. S500: The routing plan is executed through the inference execution module. In the pre-decision mode, the single-path inference result and its running statistics are transmitted to the result output and recording module. In the post-decision mode, the generated multi-path candidate output and running statistics are fed back to the mode selection and routing decision module. The decision mode includes a pre-decision mode and a post-decision mode, and the corresponding routing plans are a single routing plan and a parallel / cascaded routing plan, respectively. S600 In the result output and recording module, the single-path inference result of the pre-judgment mode or the optimal output selected by the post-judgment decision in the post-judgment mode and its judgment metadata are constructed into a unified response and fed back to the upstream business system. The entire process of model inference service judgment is logged and the strategy parameters are updated.

[0010] Further, in step S200, the request-side feature vector generated by intent analysis and feature extraction of the original inference request is represented as follows: ;in, Indicates the task type. Indicates the input mode. This indicates the latency budget in the SLO project field. This indicates the reliability preference in the SLO project field. Indicates the normalized request size. The uncertainty of intent is expressed as follows: In the formula, Indicates traversing variables, Indicates the original reasoning request, Represents the task type space, Indicates a request regarding the original reasoning. The conditional probability, These represent the original reasoning requests. The text length, number of images, and audio duration, Represents information entropy; In S200, the first-level candidate set for functional adaptation is represented as follows: ;in, Representing model instances For task types Functionality compatibility Indicates the pre-screening threshold. Represents a collection of model instances.

[0011] Furthermore, in S300, the resource state vector is a set of state sub-vectors of each model instance, where any model instance... State subvectors Represented as: In the formula, Indicates CPU / GPU utilization. Indicates memory / video memory usage. Indicates the current length of the inference request queue. Indicates available network bandwidth. This indicates the predicted response latency for the current load. Indicators representing service health status , This represents the average historical processing time at the instance level. Indicates the queuing delay coefficient. Indicates the historical correction factor; Risk measurement on the resource side Represented as: In the formula, This represents the activation function. Indicates the safe load threshold. Represents a collection of model instances.

[0012] Furthermore, S400 includes the following sub-steps: S401. Determine if the first-level candidate set for function adaptation is empty; if yes, trigger the degradation strategy and return feedback information that the request cannot be processed to the requesting end; if no, proceed to S402. S402. Based on the first-level candidate set, the resource state vector is combined to perform a second-level resource feasibility screening to obtain a subset of candidate instances that meet the local operating conditions as the resource constraint input space. S403. Based on the resource constraint input space points, calculate the fit of the decision mode according to the request side feature vector, resource state vector and risk metric, and select the decision mode; the decision mode includes pre-decision mode and post-decision mode. S404. When the decision mode is the pre-decision mode, the first-level candidate set is used as the candidate set. Combined with the resource state vector, a single-path routing plan is generated according to the policy parameters and sent to the inference execution module. When the decision mode is the post-decision mode, a subset of candidate instances is used as the candidate set that satisfies the resource constraints. Parallel or cascaded routing plans are generated according to the policy parameters and sent to the inference execution module. The system receives multiple candidate outputs and resource consumption data from the inference execution module when executing the routing plan. For each candidate output, it calls the review score calculated by the intent engine module and constructs a cost item by combining the resource consumption data. After execution, it makes a decision to select the optimal output and transmits it to the result output and recording module.

[0013] Furthermore, in S403, the formulas for calculating the fit between the pre-decision mode and the post-decision mode are as follows: The pattern decision rules are as follows: In the formula, Indicates the pre-decision mode Adaptability Post-judgment mode Adaptability Indicates the uncertainty of intent. This represents the feasibility score of the candidate model within the SLO latency budget for the current inference request. A risk metric representing a resource state vector. This indicates the reliability preference in the SLO project field. , and These represent the weighting coefficients for intent uncertainty, reliability preference, and resource risk measurement under the predictive model, respectively. , and These represent the weighting coefficients for intent uncertainty, reliability preference, and resource risk measurement under the post-judgment model, respectively.

[0014] Furthermore, in S404, when the decision mode is the pre-decision mode, the first-level candidate set is used as the candidate set, combined with the resource state vector, to meet the delay budget. Under the constraints, a single target model instance is selected, and execution parameters are configured to form a single-path routing plan, which is then sent to the inference execution module; whereby, the single target model instance... for: In the formula, This represents the resource overhead under the premise of satisfying latency constraints. This represents the end-to-end delay estimate; When the decision mode is post-decision mode, the candidate output with the highest score calculated by the post-decision decision is selected as the optimal output; the calculation formula for the post-decision decision is as follows: In the formula, Indicates the first The post-decision decision score of the path candidate output. The request-side feature vector calculated by the intent engine module With the Path candidate output Semantic consistency score, The representation intent engine module calculates the first Path candidate output Coverage score, Represents the resource state vector The cost item, This indicates the optimal output. Indicates the index of the optimal output.

[0015] Furthermore, in step S404, the strategy parameters are updated using an empirical risk minimization method, and the update formula is as follows: In the formula, This indicates the updated policy parameters. This indicates the policy parameters before the update. This indicates the latency budget in the SLO project field. Indicates in the strategy parameters Under its influence, targeting the first Service performance evaluation values ​​for each sample This indicates the number in the historical operation log. The request-side feature vector of each sample, This indicates the number in the historical operation log. The resource state vector of each sample, Represents the regularization coefficient. This represents the regularization term for the policy parameters, used to constrain the complexity of the policy parameters to prevent overfitting. This represents the loss function, used to quantify the performance and latency budget of service execution. The cost of the deviation between them.

[0016] This invention proposes a model reasoning service decision system and method based on an intent engine. By introducing a deep decision mechanism that coordinates "intent understanding" and "resource awareness," it significantly differs from existing static routing or simple load balancing schemes. Its specific beneficial effects are described below in order of importance: (1) Implement a request-level dual-mode switching mechanism for pre-decision mode / post-decision mode within the same system. The mode selection and routing decision module in the system takes the intent feature vector and resource state vector as input, calculates the fit between the pre-decision mode and the post-decision mode for each request, and selects the current service mode based on a preset threshold. When in mode A, inference is performed only on a single candidate model, and the result is quickly judged by quality gating. Once the gating fails, it is automatically upgraded to mode A, triggering multi-model inference and post-decision decision on a limited candidate set. With the help of this single-request-granular dual-mode selection mechanism and the automatic upgrade path triggered by quality gating, the system can maintain low latency and low computational overhead of the single-model path when the intent is clear and the risk is low, and can automatically switch to the multi-model collaborative path when semantic ambiguity or high-risk tasks occur, thereby improving overall reliability.

[0017] (2) The semantic intent, service level objectives, and edge resource constraints are unified into the same decision function to form a joint decision mechanism for edge environments. In the mode selection and routing decision module, this invention does not select routes solely based on request semantics or static configuration, but rather on a pre-deployed set of globally available models. A two-level candidate set is explicitly constructed: first, the intent engine module constructs a candidate set based on the request feature vector. from The selection process yields a set of candidate models that are functionally suitable for the current task. Secondly, the mode selection and routing decision module 104 in Based on resource state vectors Instances that do not meet the operating conditions, such as memory, computing power, and health status, are further eliminated to obtain a subset that meets the local operating constraints. and in The system generates a specific routing plan. The generated routing plan takes into account the task type, latency budget and reliability preference in the SLO, as well as the instantaneous load status of each edge node. It clearly distinguishes between the working intervals of "resource abundance, prioritizing result quality improvement" and "resource scarcity, prioritizing latency constraints." This allows semantic requirements, service level objectives, and resource feasibility to be jointly determined within the same system, avoiding the problem of the semantic routing layer and resource scheduling layer being isolated from each other in traditional solutions, which leads to large fluctuations in edge node load and response time.

[0018] (3) A multi-dimensional result judgment process, including semantic adjudication and cost constraints, is introduced under post-judgment mode B. When the system selects post-judgment mode B, the inference execution module 105 performs inference in parallel or cascaded on multiple candidate model instances. Then, the intent engine module 102 makes a two-stage judgment on each candidate output: first, it filters out outputs that contain sensitive violations, obvious illusions, or do not meet business constraints according to preset security and compliance rules; second, it calculates scores such as semantic consistency and key point coverage among the remaining candidates, and adds cost items such as latency, computing power, and power consumption, and selects the result with the best score as the final output. This multi-dimensional adjudication process is different from simple confidence comparison or majority voting, and is more suitable for scenarios with hard constraints on security and accuracy. Under the premise of keeping costs controllable, it reduces the probability of single-model misjudgment and leakage of high-risk content.

[0019] (4) A traceable and self-optimizing closed-loop decision system is formed through the decision metadata recording and strategy parameter learning submodule 1041. In the result output and recording module 106, the present invention records complete decision metadata for each inference request, including at least the service mode adopted, the result source model or model set, the resource status summary at the time of request, end-to-end latency, quality gating and adjudication score, etc. The strategy parameter learning submodule 1041 periodically or according to trigger conditions reads these records and updates the threshold and weight parameters in the mode selection and routing decision module 104 online, so that the decision boundaries such as "when to use mode A, when to switch to mode B, and how large the candidate set is" can gradually converge with the actual request distribution and resource fluctuations. Through this closed-loop design of complete metadata recording and adaptive parameter adjustment, the system can continuously optimize the service strategy without frequent manual parameter tuning, thereby improving the long-term stability and operation and maintenance efficiency.

[0020] In summary, by integrating mechanisms such as request-level dual-mode switching, semantic and resource joint decision-making, multi-dimensional result adjudication, and closed-loop learning based on metadata within the same edge inference system, this invention enables the edge multi-model inference system to ensure service level objectives while also taking into account inference quality and resource utilization efficiency. Attached Figure Description

[0021] Figure 1 This is a schematic diagram of the routing architecture for model inference services based on static rules in existing technologies.

[0022] Figure 2 The diagram shows the structure of the model reasoning service decision system based on the intent engine provided by this invention.

[0023] Figure 3 The flowchart of the model reasoning service decision method based on intent engine provided by the present invention is shown.

[0024] Figure 4This is a schematic diagram of the interaction process of each module in the pre-decision mode provided by the present invention.

[0025] Figure 5 This is a schematic diagram of the interaction flow of each module in the post-decision mode provided by the present invention.

[0026] Figure 6 This is a schematic diagram illustrating the application of the present invention in a smart door lock scenario. Detailed Implementation

[0027] The specific embodiments of the present invention are described below to enable those skilled in the art to understand the present invention. However, it should be understood that the present invention is not limited to the scope of the specific embodiments. For those skilled in the art, various changes are obvious as long as they are within the spirit and scope of the present invention as defined and determined by the appended claims. All inventions utilizing the concept of the present invention are protected.

[0028] Example 1: This invention provides a model reasoning service decision system based on an intent engine, such as... Figure 2 As shown, it includes: The inference request input module 101 is used to receive the user's original inference request and its associated SLO project field, and transmit them to the intent engine module and the inference execution module respectively. The intent engine module 102 is used to perform intent analysis and feature extraction on the original inference request during the request phase, generate request-side feature vectors and a first-level candidate set of tasks to the model; and, in the post-decision mode, to review and score the multi-path candidate outputs generated by the inference execution module. The resource status awareness module 103 is used to collect resource monitoring data from each edge node and model instance, construct resource status vectors, calculate risk metrics on the resource side, and transmit them to the mode selection and routing decision module. The mode selection and routing decision module 104 is used to determine the decision mode based on the request side feature vector, first-level candidate set, resource status vector and risk metric, combined with policy parameters through two-level screening and adaptation calculation, and to send the corresponding routing plan to the inference execution module; and, in the post-decision mode, to call the review score to make a post-decision decision to select the optimal output and transmit it to the result output and recording module. The decision modes include pre-decision mode and post-decision mode, and their corresponding routing plans are single routing plan and parallel / cascaded routing plan, respectively. The inference execution module 105 is used to perform inference on the original inference request according to the routing plan and record the running statistics of each inference service path in real time. In the pre-decision mode, it transmits the single-path inference result and its running statistics to the result output and recording module. In the post-decision mode, it feeds back the generated multi-path candidate output and running statistics to the mode selection and routing decision module. The result output and recording module 106 is used to construct a unified response from the received data and feed it back to the upstream business system, and to log the entire process of model inference service decision and update the strategy parameters through the feedback link.

[0029] In this embodiment, the intent engine module 102 can rely on pre-trained recognition and scoring models to achieve the above functions; when the request distribution changes significantly, it can combine online sampling results for periodic calibration to reduce judgment bias.

[0030] In this embodiment, the connection relationships and data flow between the modules in the above-mentioned model inference service decision system are as follows: The inference request input module serves as the user-side entry point, receiving the original inference request and its associated SLO project fields (such as latency budget). Reliability preference (etc.) performs traffic splitting: on the one hand, the request features are handed over to the intent engine module 102 for intent parsing and request-side feature vectors. and first-level candidate set On the one hand, it generates the original reasoning request content (or its preprocessed result) and on the other hand, it directly provides the reasoning execution module 105 with the original reasoning request content (or its preprocessed result) as the input material for subsequent actual reasoning.

[0031] Meanwhile, the resource status awareness module 103 continuously collects the utilization rate, memory usage, and queue status of each underlying instance to construct a resource status vector. and risk measurement And output to the mode selection and routing decision module 104.

[0032] The pattern selection and routing decision module 104 aggregates the request-side feature vector from the intent engine module 102 and the resource state vector from the resource state awareness module 103, and combines them with policy parameters. Perform two-level screening and fit calculation, and generate the judgment results (including the judgment pattern). Target Instance or candidate set (And concurrency / retry strategies) are sent to the inference execution module 105 through the control link.

[0033] The inference execution module 105 performs inference based on the decision result and records runtime statistics (latency, memory consumption, etc.) for each path in real time. Its output direction depends on the service mode: in the pre-decision mode (Mode A), after completing a single-path inference, the inference execution module 105 outputs the result... The runtime statistics are directly transmitted to the result output and recording module 106; while in the post-decision mode (mode B), the inference execution module 105 generates multiple candidate outputs. The operational statistics are fed back to the mode selection and routing decision module 104. The mode selection and routing decision module 104 then calls the intent engine module 102 to perform semantic consistency and coverage scoring, and calculates the comprehensive cost based on the statistical data. After completing the final decision, the best result is determined. Its metadata is sent to the result output and recording module 106.

[0034] Finally, the result output and recording module 106 receives the final result and the decision metadata of the entire link (including pattern tagging, end-to-end latency, and resource consumption summary), constructs a unified response output to the upstream business; at the same time, this module writes the complete log of "request-decision-result-feedback" into the history record, and provides it to the policy parameter learning submodule 1041 through the feedback link for offline estimation and periodic updating of weight and threshold parameters, and updates the parameters. Write back to the mode selection and routing decision module 104 to achieve closed-loop adaptive optimization of the system.

[0035] In this embodiment, the mode selection and routing decision module includes a policy parameter learning submodule; The strategy parameter learning submodule is used to estimate and calibrate the strategy parameters offline based on the system's historical operation logs, and update the strategy parameters through empirical risk minimization. Specifically, the historical operation logs include at least the request-side feature vector, resource status vector, feasibility and risk indicators, mode decision results, SLO achievement status, and decision score information in the post-decision mode.

[0036] In this embodiment, the aforementioned model inference service decision system is suitable for deployment in edge computing or local computing power cluster environments. The cluster consists of multiple types of computing power nodes, including general-purpose GPU nodes, low-power edge computing devices, and general-purpose CPU nodes, which are interconnected via a local area network or dedicated line. Multiple AI models with different performance and functions are pre-deployed on the aforementioned nodes, forming an available model set M. These models may include large language models, vertical domain question answering models, classification and retrieval models, and control and decision-making models, etc., and differ in terms of accuracy, inference latency, resource consumption, and domain adaptability.

[0037] In one specific deployment method, the intent engine module 102 and the mode selection and routing decision module 104 are deployed on a centralized control node or edge control plane to uniformly complete request parsing, service mode selection, and routing plan generation. The resource status awareness module 103 runs on each computing node in the form of an agent or sidecar, or interfaces with an existing orchestration system (such as a container orchestration platform), and periodically collects and transmits monitoring data such as node utilization, GPU / memory usage, queue length, network bandwidth, and model health status. The inference execution module 105 is deployed locally on each edge node, and initiates inference tasks on the local node or neighboring nodes and collects runtime metrics according to the routing plan issued by the mode selection and routing decision module 104. The intent engine module 102 integrates a result adjudication submodule, which is used to perform quality assessment and semantic adjudication on multi-path model outputs in post-decision mode. The result output and recording module 106 is deployed on the control node or on the same side as the business entry point, and is used to return inference results to the upstream business system and record decision metadata and runtime statistics to provide an update basis for the policy parameter learning submodule 1041.

[0038] Communication links are established between modules via Remote Procedure Call (RPC), message queues, or service gateways, supporting synchronous calls and asynchronous return with timeout control mechanisms. The model inference service decision system and its included functional modules support diverse deployment modes, covering enterprise private clouds, carrier edge computing clusters, and home or campus-level local environments. This system has broad hardware platform compatibility; as long as the operating environment integrates an edge node architecture, supports multi-model parallel coexistence, and possesses decision control logic and the necessary network connectivity, the various functions of this invention can be implemented.

[0039] Example 2: This embodiment provides a model reasoning service decision method based on an intent engine, building upon Embodiment 1. It is implemented using a reasoning service decision system, such as... Figure 3 As shown, it includes the following steps: S100: Receive the user's original inference request and its associated SLO project field through the inference request input module, and transmit them to the intent engine module and the inference execution module respectively. S200. In the intent engine module, during the request phase, intent analysis and feature extraction are performed on the original inference request to generate a request-side feature vector. And based on the adaptation function of task type and model capability label, construct the candidate mapping relationship from task to model, and then construct the first-level candidate set of functional adaptation; S300: Collect resource monitoring data of each edge node and model instance through the resource status perception module, and then construct resource status vector and calculate risk measurement on the resource side; S400. In the mode selection and routing decision module, the decision mode is determined by combining the request side feature vector, first-level candidate set, resource status vector and risk metric with the strategy parameters through two-level screening and adaptation calculation, and the corresponding routing plan is sent to the inference execution module. S500: The routing plan is executed through the inference execution module. In the pre-decision mode, the single-path inference result and its running statistics are transmitted to the result output and recording module. In the post-decision mode, the generated multi-path candidate output and running statistics are fed back to the mode selection and routing decision module. The decision modes include pre-decision mode and post-decision mode, and their corresponding routing plans are single routing plan and parallel / cascaded routing plan, respectively. S600 In the result output and recording module, the single-path inference result of the pre-judgment mode or the optimal output selected by the post-judgment decision in the post-judgment mode and its judgment metadata are constructed into a unified response and fed back to the upstream business system. The entire process of model inference service judgment is logged and the strategy parameters are updated.

[0040] In S200 of this embodiment, the request-side feature vector generated by intent analysis and feature extraction of the original inference request is represented as follows: ;in, Indicates the task type. Indicates the input mode. This indicates the latency budget in the SLO project field. This indicates the reliability preference in the SLO project field. Indicates the normalized request size. Indicates the uncertainty of intent.

[0041] The intent engine module calculates conditional probabilities based on a pre-trained classification model and determines the final task type through maximum a posteriori probability estimation. It is represented as: Similarly, input mode Represented as: Request size can be composed of request length and multimodal size features, and is expressed as follows: And can be further normalized to obtain ; Uncertainty of intent can be calculated using measures based on information entropy or the dispersion of probability distribution. It is represented as: In the formula, Indicates traversing variables, Indicates the original reasoning request, Represents the task type space, Indicates a request regarding the original reasoning. The conditional probability, These represent the original reasoning requests. The text length, number of images, and audio duration, Represents information entropy; Furthermore, using the request-side feature vector Task types Using an index, a candidate mapping relationship between tasks and models is constructed based on an adaptation function that matches task type and model capability labels. This leads to the construction of a first-level candidate set for functional adaptation, represented as follows: ;in, Representing model instances For task types Functionality compatibility Indicates the pre-screening threshold. Represents a collection of model instances.

[0042] In one specific implementation method Based on task type semantic description vector With model Capability Description Vector Similarity calculation between them: This allows the system to handle undefined task types, achieving zero-shot model matching. This first-level candidate set... This will serve as the base input space for the subsequent mode selection and routing decision module 104.

[0043] In this embodiment, the intent engine module 102, in addition to implementing intent parsing and candidate mapping generation during the request phase, can also reuse its semantic evaluation capabilities in the post-decision mode to review and score the multi-model candidate outputs to support the final decision. Assuming the system enters the post-decision mode (Mode B) according to the decision strategy, the system will analyze the multiple candidate outputs generated by the inference execution module 105. The system will reuse the intent engine module 102 to initiate the adjudication phase, performing semantic consistency checks and coverage scoring on the outputs of each candidate model. Specifically, the intent engine module 102 calculates the review score, including a semantic consistency score. and coverage score Among them, semantic consistency score To measure the semantic fit between output content and user intent, semantic representation similarity can be used, which is represented as follows: In the formula, This is a semantic representation function. To ensure real-time performance, Pre-trained lightweight encoders are preferred.

[0044] Coverage score Used to measure the output against the set of key points or constraints parsed from the request. The satisfaction status is expressed as follows: In the formula, For indicator functions, it means the first... Whether the constraints are satisfied.

[0045] In this embodiment, to achieve automated decision-making, the present invention uses hybrid validation logic: for explicit constraints (such as keywords, format requirements): character-level pattern matching is performed using regular expressions or inverted indexes; if... Include A specific pattern is determined to be satisfied; for implicit constraints (such as semantic logic and factual consistency): a lightweight natural language inference model is used. and The input model calculates the semantic implication probability. If the probability is higher than a preset threshold (such as 0.8), it is determined to be satisfied.

[0046] In S300 of this embodiment, resource monitoring data of each edge node and model instance is collected by the resource status awareness module, and normalized and aggregated to construct a resource status vector, which includes at least computing power resource utilization, video memory and memory usage, inference queue length, available bandwidth, response latency prediction, and node health marking.

[0047] Specifically, suppose there are |M| available model instances in the system, and the resource state vector is the set of state sub-vectors of each model instance. For any model instance... State subvectors Represented as: In the formula, Indicates CPU / GPU utilization. Indicates memory / video memory usage. Indicates the current length of the inference request queue. Indicates available network bandwidth. This indicates the predicted response latency for the current load. Indicators representing service health status , This represents the average historical processing time at the instance level. Indicates the queuing delay coefficient. Indicates the historical correction factor; Furthermore, the risk metric calculated by the resource status awareness module is a resource-side load / latency metric, used to suppress multi-path parallel overhead under high load conditions during mode selection and post-decision execution orchestration, thereby avoiding SLO defaults due to resource scarcity; therefore, the resource-side risk metric... Represented as: In the formula, This represents the activation function. Indicates the safe load threshold. Represents a collection of model instances.

[0048] In this embodiment, when the data is incomplete or abnormal, the resource status awareness module 103 also performs missing value backfilling and anomaly detection on the monitoring data, marking instances that clearly do not meet the preset operating conditions as unavailable, i.e., setting them as unavailable. .

[0049] Specifically, when an anomaly occurs in the monitoring link or sensor, the resource status perception module 103 can degenerate into generating a resource status vector based on a conservative estimate. To ensure end-to-end service quality objectives, this conservative estimation process can be implemented using a historical smoothing algorithm with a safety margin. Let... When data loss is detected, the resource status awareness module 103 can analyze the instance status vector. Conservative estimation is performed on a component-by-dimensional basis, and the estimated value is... Represented as: In the formula, express The exponentially weighted moving average at time t, A measure of volatility (such as standard deviation) within a historical time window. This represents the preset safety factor, which can be configured or dynamically adjusted based on business latency tolerance, historical fluctuation levels, and reliability preferences. By introducing... As a penalty, the system artificially overestimates the load status of the node, thus tending to avoid nodes with unclear information during routing decisions, minimizing the risk of service timeouts. Besides the example above, conservative estimation can also be achieved based on quantile regression, robust filtering, or prediction models with confidence intervals.

[0050] S400 of this embodiment includes the following sub-steps: S401. Determine if the first-level candidate set for function adaptation is empty; if yes, trigger the degradation strategy and return feedback information that the request cannot be processed to the requesting end; if no, proceed to S402. S402. Based on the first-level candidate set, the resource state vector is combined to perform a second-level resource feasibility screening to obtain a subset of candidate instances that meet the local operating conditions as the resource constraint input space. S403. Based on the resource constraint input space points, calculate the fit of the decision mode according to the request side feature vector, resource state vector and risk metric, and select the decision mode; the decision mode includes pre-decision mode and post-decision mode. S404. When the decision mode is the pre-decision mode, the first-level candidate set is used as the candidate set. Combined with the resource state vector, a single-path routing plan is generated according to the policy parameters and sent to the inference execution module. When the decision mode is the post-decision mode, a subset of candidate instances is used as the candidate set that satisfies the resource constraints. Parallel or cascaded routing plans are generated according to the policy parameters and sent to the inference execution module. The system receives multiple candidate outputs and resource consumption data from the inference execution module when executing the routing plan. For each candidate output, it calls the review score calculated by the intent engine module and constructs a cost item by combining the resource consumption data. After execution, it makes a decision to select the optimal output and transmits it to the result output and recording module.

[0051] In S401, to achieve joint decision-making based on intent understanding and resource awareness, a two-level candidate selection mechanism is established before pattern selection. Let the set of candidate models / model instances be... The intent engine module 102 is based on the request-side feature vector. First-level candidate set for output function adaptation And satisfy: Based on this, combined with the current resource state vector Secondary resource feasibility screening is performed to obtain a subset of candidate instances that meet local operating conditions; in, Representing model instances In resource status The feasibility of operation is as follows. This is the threshold for resource-side filtering; This can comprehensively represent factors such as computing power / GPU memory (or system memory) reserves, queue pressure, bandwidth, and health indicators, causing instances with insufficient resources or unready health states to be downgraded or eliminated. The resulting subset of candidate instances... This serves as the resource constraint input space for subsequent pattern adaptation calculations, routing plan generation, and execution orchestration.

[0052] Based on this, the choice of judgment mode is regarded as a binary decision variable: in These are strategy parameters.

[0053] In S402, to avoid unnecessary calculations for obviously infeasible scenarios, the mode selection and routing decision module 104 first performs a pre-veto: if the first-level candidate set is empty. If the condition is not met, the degradation strategy will be triggered directly (e.g., returning "Unable to process this request"), and the process will not proceed to the subsequent mode comparison process.

[0054] In S403, the fit calculation logic for the pre-decision mode is configured as follows: it increases as the uncertainty of intent decreases and decreases as the resource risk increases; the fit calculation logic for the post-decision mode is configured as follows: it increases as the uncertainty of intent increases and decreases as the resource risk increases; finally, the two fits are compared to determine the service mode.

[0055] In a preferred and easily implemented engineering embodiment, the fitness degree can be calculated using a linear weighted method, and the calculation formulas are as follows: In the formula, Indicates the pre-decision mode Adaptability Post-judgment mode Adaptability Indicates the uncertainty of intent. This represents the feasibility score of the candidate model for the current inference request within the SLO latency budget, reflecting the overall service capability of the candidate set under the current resource conditions. A risk metric representing a resource state vector. This indicates the reliability preference in the SLO project field. , and These represent the weighting coefficients for intent uncertainty, reliability preference, and resource risk measurement under the predictive model, respectively. , and These represent the weighting coefficients for intent uncertainty, reliability preference, and resource risk measurement under the post-judgment model, respectively.

[0056] In a preferred embodiment, Can be taken as The maximum feasibility score for satisfying the time delay budget constraint. As a measure of high load / high latency risk, For reliability preference; the fit of post-decision mode B varies. and This is to increase and improve, reflecting the strategic intent of using multi-path competition / post-decision to enhance output reliability and quality; at the same time, it introduces... As a resource-side suppression term, it avoids blindly increasing the overhead of multiple models under high load conditions. Used to adjust the weight relationship between intent certainty, resource feasibility, and risk measurement in pre-judgment mode A. This is used to adjust the relative importance of intent uncertainty, reliability preference, and resource risk mitigation terms in Mode B; both of these are parameters. Components of.

[0057] In this embodiment, the pattern decision rule is as follows: In S404, such as Figure 4 As shown, when the decision mode is the pre-decision mode, the first-level candidate set is used as the candidate set, combined with the resource state vector, to meet the latency budget. Under the constraints, a single target model instance is selected, and execution parameters are configured to form a single-path routing plan, which is then sent to the inference execution module; whereby, the single target model instance... for: In the formula, This represents the resource overhead under the premise of satisfying latency constraints. This represents the end-to-end delay estimate.

[0058] in, Used to measure the routing of requests to model instances. The resulting overall resource consumption; among which, resource overhead can be related to at least factors such as computing power consumption, video memory / memory consumption, queue pressure, and power consumption, and can be based on resource state vectors. The corresponding components are calculated.

[0059] In one implementation, The results were obtained by weighting the above resource indicators after normalization; For end-to-end latency estimation, the output of the resource state awareness module 103 can be directly reused. ; Budget for SLO latency; if multiple If the same minimum cost is achieved, then one of them will be executed according to the preset priority.

[0060] Furthermore, the mode selection and routing decision module further configures the concurrency, timeout and retry policy for the selected instance, and sends the relevant routing and configuration parameters to the inference execution module 105.

[0061] In S404, such as Figure 5 As shown, when the decision mode is post-decision mode, the candidate output with the highest score calculated by the post-decision decision is selected as the optimal output; whereby the calculation formula for the post-decision decision is: In the formula, Indicates the first The post-decision decision score of the path candidate output. The request-side feature vector calculated by the intent engine module With the Path candidate output Semantic consistency score, The representation intent engine module calculates the first Path candidate output Coverage score, Represents the resource state vector The cost item, This indicates the optimal output. Indicates the index of the optimal output.

[0062] Specifically, depending on the size of the request ,risk or reliability preference Trigger the selection of parallel / cascaded routing plans; further, through the above formula, a decision is made if multiple... If the same maximum score is obtained, then the preset preference (e.g., semantic consistency) will be used. Priority, then coverage Priority is given; if they are still tied, then the cost item is used. (Or, the one with the lower actual latency will be selected as the final output)

[0063] In the above process of mode selection and routing decision, policy parameters The system can be calibrated offline based on historical operational data, or dynamically updated by the strategy parameter learning submodule 1041 in conjunction with online feedback, to continuously optimize the balance between SLO achievement rate and resource cost. Historical operational data includes at least the following information: request feature I, resource status R, feasibility and risk indicators, mode decision results, SLO achievement status, and adjudication score under the post-decision mode. Parameter updates can be performed using an empirical risk minimization approach, with the update formula as follows: In the formula, This indicates the updated policy parameters. This indicates the policy parameters before the update. This indicates the latency budget in the SLO project field. Indicates in the strategy parameters Under its influence, targeting the first Service performance evaluation values ​​for each sample This indicates the number in the historical operation log. The request-side feature vector of each sample, This indicates the number in the historical operation log. The resource state vector of each sample, Represents the regularization coefficient. This represents the regularization term for the policy parameters, used to constrain the complexity of the policy parameters to prevent overfitting. This represents the loss function, used to quantify the performance and latency budget of service execution. The cost of the deviation between them.

[0064] In this embodiment, through statistical analysis of the above-mentioned record items, it is possible to... and Perform periodic reassessments to adapt mode selection and multi-model adjudication strategies to business load and resource fluctuations.

[0065] In S500 of this embodiment, during the execution of the routing plan by the inference execution module: when At that time, the inference execution module 105 selects the target instance specified by the mode selection and routing decision module 104. Performing single-path inference yields the following output: at this time, It is directly passed as the final result to the result output and recording module 106. Representing model instances For input Inference mapping.

[0066] when At that time, the inference execution module 105 selects candidates from the candidate set according to the candidate execution orchestration strategy issued by the mode selection and routing decision module 104. One or more model instances are selected for parallel or staged cascaded execution to obtain one or more candidate outputs: During the above execution process, the system monitors and records operational statistics for each path in real time. These statistics include at least actual execution latency, GPU / RAM usage, queue changes, and abnormal states. The inference execution module 105 will then process the candidate outputs. The corresponding operational statistics are fed back to the pattern selection and routing decision module 104, so that the pattern selection and routing decision module 104 can reuse the intent engine module 102 to obtain semantic consistency and coverage scores, and combine them with the operational statistics to construct cost items to complete the final decision and result selection.

[0067] In S600 of this embodiment, the result output and recording module 106 constructs a response to the upper-layer business system and performs closed-loop data recording. This module has a multi-source input interface and receives data from different upstream modules according to different service modes: in the pre-decision mode, it receives the single-model inference result and runtime statistics from the inference execution module 105; in the post-decision mode, it receives the final adjudicated result, adjudication score summary, and aggregated execution statistics from the mode selection and routing decision module 104.

[0068] The result output and recording module 106 encapsulates the received content into a unified format response message and returns it to the business side along with decision metadata information (including service mode tag, result source model ID, end-to-end latency, resource consumption summary, etc.). Simultaneously, this module writes the complete link data from "original request – decision feature – execution result – business feedback" into the historical log, providing it to the policy parameter learning submodule 1041 through the feedback link for offline estimation and dynamic updating of policy parameters, thereby achieving closed-loop adaptive optimization of the decision policy. Furthermore, this module also supports version management and anomaly identification of the recorded data to ensure traceability throughout the entire link.

[0069] Example 3: This embodiment provides an application example of the above-mentioned model inference service decision system in a smart home security scenario.

[0070] In this embodiment, as Figure 6 As shown, the model inference service decision system can be embedded into the smart lock motherboard, or it can run on a home gateway node and communicate with a local heterogeneous edge computing cluster formed by the door lock, edge computing box, and user host through a local area network. Multiple models with different capabilities are pre-deployed on these devices, forming the aforementioned globally available model set M. The smart lock runs a lightweight face recognition model locally, while the host or edge computing box runs a high-precision large-scale visual model and a voiceprint recognition model, which are then called by the inference execution module.

[0071] In actual operation, when a visitor triggers the door lock, the inference request input module 101 receives multimodal data collected by the camera and microphone and forwards it to the intent engine module 102. The intent engine module 102, combined with the output of the lightweight local visual model called by the inference execution module 105, performs feature analysis on the current scene, extracts and forms a request feature vector I, which includes information such as image clarity, presence of facial occlusion, and visitor identity category, and calculates the corresponding intent uncertainty index. At the same time, the resource status perception module 103 collects resource monitoring data from nodes such as the door lock body, battery management unit, home gateway, and edge computing box, and constructs a resource status vector R. R includes at least the door lock battery power, computing power and memory reserves of each node, and local area network bandwidth. After these two types of information are converged to the mode selection and routing decision module 104, it calculates the adaptability of the pre-decision mode A and the post-decision mode B according to the preset decision function, and completes the service mode selection for the current request.

[0072] When the intent engine module 102 determines, based on the request feature vector I, that the visitor is a locally registered family member, has low intent uncertainty, and a low scenario risk level, and the resource status awareness module 103 reports that the door lock battery power is within a healthy range and the door lock's local computing power resources meet the latency budget of the current SLO, the mode selection and routing decision module 104 determines the current request to be adapted to pre-decision mode A. At this time, the system, based on the resource and task matching logic, directly routes the request to the lightweight model running on the low-power DSP built into the door lock. In this mode, the system completes inference within millisecond-level latency. If the output confidence passes the quality gating, the unlocking command is directly issued, thereby maximizing the device's battery life while ensuring a good user experience.

[0073] Conversely, when I indicates the visitor's identity is a stranger or the intent description is ambiguous (e.g., claiming to be a property maintenance worker), and the scene detects factors that increase intent uncertainty, such as facial occlusion or insufficient lighting, or when R indicates that the door lock's local computing power is insufficient to complete high-quality recognition within the specified time delay, the mode selection and routing decision module 104 will select post-decision mode B. In this mode, the system no longer relies on a single model, but instead constructs a cross-device execution orchestration based on the current resource view: it sends high-resolution video frames through the local area network to a ready host, calling the large visual model deployed on it to perform fine-grained behavior analysis and uniform recognition; simultaneously, it distributes audio data to Jetson edge boxes for voiceprint verification. The inference execution module then aggregates multiple outputs from different computing power nodes and submits them to the intent engine for final decision. The system comprehensively compares multiple results, such as "the large visual model recognizes the uniform as a maintenance worker," "the voiceprint matches the property registration," and "the local model has low confidence," to calculate a semantic consistency score. The system will only push a confirmation request to the user's mobile phone or perform unlocking when the semantic conclusions of all high-weight models are consistent and meet the safety baseline. This achieves accurate coverage of complex long-tail intentions without having to occupy high-power computing power for a long time, thus achieving the optimal balance between edge-side energy efficiency and service quality.

[0074] During the above process, the result output and recording module 106 will also write the decision metadata, such as the service mode corresponding to each unlocking request, the model instance used, the door lock battery level and network status summary, and the final decision score, into the local or gateway-side log. The policy parameter learning submodule 1041 can periodically adjust the mode switching threshold and the size of the candidate set in different scenarios based on these historical records. For example, for unfamiliar visitors who have been marked by users multiple times as having a "high risk of misjudgment", the probability of them entering and being judged in mode B will be increased to further improve the overall security in the smart home security scenario.

[0075] Specific embodiments have been used to illustrate the principles and implementation methods of this invention. The descriptions of the embodiments above are only for the purpose of helping to understand the method and core ideas of this invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this invention. Therefore, the content of this specification should not be construed as a limitation of this invention.

[0076] Those skilled in the art will recognize that the embodiments described herein are intended to help the reader understand the principles of the invention, and should be understood that the scope of protection of the invention is not limited to such specific statements and embodiments. Those skilled in the art can make various other specific modifications and combinations based on the technical teachings disclosed in this invention without departing from the spirit of the invention, and these modifications and combinations are still within the scope of protection of this invention.

Claims

1. A model reasoning service decision system based on an intent engine, characterized in that, include: The inference request input module is used to receive the user's original inference request and its associated SLO project field, and transmit them to the intent engine module and the inference execution module respectively. The intent engine module is used to perform intent analysis and feature extraction on the original inference request during the request phase, generating request-side feature vectors and a first-level candidate set of tasks to the model; and, in the post-decision mode, to review and score the multi-path candidate outputs generated by the inference execution module. The resource status awareness module is used to collect resource monitoring data from each edge node and model instance, construct resource status vectors, calculate risk metrics on the resource side, and transmit them to the mode selection and routing decision module. The mode selection and routing decision module is used to determine the decision mode based on the request side feature vector, first-level candidate set, resource status vector and risk metric, combined with policy parameters through two-level screening and adaptation calculation, and to send the corresponding routing plan to the inference execution module; and, in the post-decision mode, to call the review score to make a post-decision decision to select the optimal output and transmit it to the result output and recording module. The decision mode includes a pre-decision mode and a post-decision mode, and the corresponding routing plans are a single routing plan and a parallel / cascaded routing plan, respectively. The inference execution module is used to perform inference on the original inference request according to the routing plan and record the running statistics of each inference service path in real time. In the pre-decision mode, the single-path inference result and its operational statistics are transmitted to the result output and recording module. In the post-decision mode, the generated multi-path candidate output and operational statistics are fed back to the mode selection and routing decision module. The results output and recording module is used to construct a unified response from the received data and feed it back to the upstream business system, and to log the entire process of model inference service decision and update the strategy parameters through the feedback link.

2. The model reasoning service decision system based on intent engine according to claim 1, characterized in that, The mode selection and routing decision module includes a policy parameter learning submodule. The strategy parameter learning submodule is used to estimate and calibrate the strategy parameters offline based on the system's historical operation logs, and update the strategy parameters by minimizing empirical risks.

3. A model reasoning service decision method based on an intent engine, implemented based on the model reasoning service decision system according to any one of claims 1 to 2, characterized in that, Includes the following steps: S100: Receive the user's original inference request and its associated SLO project field through the inference request input module, and transmit them to the intent engine module and the inference execution module respectively. S200. In the intent engine module, during the request phase, intent analysis and feature extraction are performed on the original inference request to generate a request-side feature vector. And based on the adaptation function of task type and model capability label, construct the candidate mapping relationship from task to model, and then construct the first-level candidate set of functional adaptation; S300: Collect resource monitoring data of each edge node and model instance through the resource status perception module, and then construct resource status vector and calculate risk measurement on the resource side; S400. In the mode selection and routing decision module, the decision mode is determined by combining the request side feature vector, first-level candidate set, resource status vector and risk metric with the strategy parameters through two-level screening and adaptation calculation, and the corresponding routing plan is sent to the inference execution module. S500: The routing plan is executed through the inference execution module. In the pre-decision mode, the single-path inference result and its running statistics are transmitted to the result output and recording module. In the post-decision mode, the generated multi-path candidate output and running statistics are fed back to the mode selection and routing decision module. The decision mode includes a pre-decision mode and a post-decision mode, and the corresponding routing plans are a single routing plan and a parallel / cascaded routing plan, respectively. S600 In the result output and recording module, the single-path inference result of the pre-judgment mode or the optimal output selected by the post-judgment decision in the post-judgment mode and its judgment metadata are constructed into a unified response and fed back to the upstream business system. The entire process of model inference service judgment is logged and the strategy parameters are updated.

4. The model reasoning service decision method based on intent engine according to claim 3, characterized in that, In step S200, the request-side feature vector generated by intent analysis and feature extraction of the original inference request is represented as follows: ;in, Indicates the task type. Indicates the input mode. This indicates the latency budget in the SLO project field. This indicates the reliability preference in the SLO project field. Indicates the normalized request size. The uncertainty of intent is expressed as follows: In the formula, Indicates traversing variables, Indicates the original reasoning request, Represents the task type space, Indicates a request regarding the original reasoning. The conditional probability, These represent the original reasoning requests. The text length, number of images, and audio duration, Represents information entropy; In S200, the first-level candidate set for functional adaptation is represented as follows: ;in, Representing model instances For task types Functionality compatibility Indicates the pre-screening threshold. Represents a collection of model instances.

5. The model reasoning service decision method based on intent engine according to claim 3, characterized in that, In step S300, the resource state vector is a set of state sub-vectors for each model instance, where any model instance... State subvectors Represented as: In the formula, Indicates CPU / GPU utilization. Indicates memory / video memory usage. Indicates the current length of the inference request queue. Indicates available network bandwidth. This indicates the predicted response latency for the current load. Indicators representing service health status , This represents the average historical processing time at the instance level. Indicates the queuing delay coefficient. Indicates the historical correction factor; Risk measurement on the resource side Represented as: In the formula, This represents the activation function. Indicates the safe load threshold. Represents a collection of model instances.

6. The model reasoning service decision method based on intent engine according to claim 3, characterized in that, S400 includes the following steps: S401. Determine if the first-level candidate set for function adaptation is empty; if yes, trigger the degradation strategy and return feedback information that the request cannot be processed to the requesting end; if no, proceed to S402. S402. Based on the first-level candidate set, the resource state vector is combined to perform a second-level resource feasibility screening to obtain a subset of candidate instances that meet the local operating conditions as the resource constraint input space. S403. Based on the resource constraint input space points, calculate the fit of the decision mode according to the request side feature vector, resource state vector and risk metric, and select the decision mode; the decision mode includes pre-decision mode and post-decision mode. S404. When the decision mode is the pre-decision mode, the first-level candidate set is used as the candidate set. Combined with the resource state vector, a single-path routing plan is generated according to the policy parameters and sent to the inference execution module. When the decision mode is the post-decision mode, a subset of candidate instances is used as the candidate set that satisfies the resource constraints. Parallel or cascaded routing plans are generated according to the policy parameters and sent to the inference execution module. The system receives multiple candidate outputs and resource consumption data from the inference execution module when executing the routing plan. For each candidate output, it calls the review score calculated by the intent engine module and constructs a cost item by combining the resource consumption data. After execution, it makes a decision to select the optimal output and transmits it to the result output and recording module.

7. The model reasoning service decision method based on intent engine according to claim 6, characterized in that, In step S403, the formulas for calculating the fit between the pre-decision mode and the post-decision mode are as follows: The pattern decision rules are as follows: In the formula, Indicates the pre-decision mode Adaptability Post-judgment mode Adaptability Indicates the uncertainty of intent. This represents the feasibility score of the candidate model within the SLO latency budget for the current inference request. A risk metric representing a resource state vector. This indicates the reliability preference in the SLO project field. , and These represent the weighting coefficients for intent uncertainty, reliability preference, and resource risk measurement under the predictive model, respectively. , and These represent the weighting coefficients for intent uncertainty, reliability preference, and resource risk measurement under the post-judgment model, respectively.

8. The model reasoning service decision method based on intent engine according to claim 6, characterized in that, In step S404, when the decision mode is the pre-decision mode, the first-level candidate set is used as the candidate set, and the resource state vector is combined to meet the delay budget. Under the constraints, a single target model instance is selected, and execution parameters are configured to form a single-path routing plan, which is then sent to the inference execution module; whereby, the single target model instance... for: In the formula, This represents the resource overhead under the premise of satisfying latency constraints. This represents the end-to-end delay estimate; When the decision mode is post-decision mode, the candidate output with the highest score calculated by the post-decision decision is selected as the optimal output; the calculation formula for the post-decision decision is as follows: In the formula, Indicates the first The post-decision decision score of the path candidate output. The request-side feature vector calculated by the intent engine module With the Path candidate output Semantic consistency score, The representation intent engine module calculates the first Path candidate output Coverage score, Represents the resource state vector The cost item, This indicates the optimal output. Indicates the index of the optimal output.

9. The model reasoning service decision method based on intent engine according to claim 6, characterized in that, In step S404, the strategy parameters are updated using an empirical risk minimization approach, and the update formula is as follows: In the formula, This indicates the updated policy parameters. This indicates the policy parameters before the update. This indicates the latency budget in the SLO project field. Indicates in the strategy parameters Under its influence, targeting the first Service performance evaluation values ​​for each sample This indicates the number in the historical operation log. The request-side feature vector of each sample, This indicates the number in the historical operation log. The resource state vector of each sample, Represents the regularization coefficient. This represents the regularization term for the policy parameters, used to constrain the complexity of the policy parameters to prevent overfitting. This represents the loss function, used to quantify the performance and latency budget of service execution. The cost of the deviation between them.

Images