Inference task request processing methods, apparatus, computer equipment, storage media, and computer program products
By combining complexity assessment of inference task requests with network quality information, the target inference node is dynamically selected and a compressed context is generated on the edge side. This solves the problem that edge devices have difficulty balancing quality and efficiency when processing inference tasks. It enables simple tasks to be processed quickly locally and complex tasks to be fully processed on remote nodes, thereby improving overall processing efficiency.
Patent Information
- Application Number
- CN202611398543.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-09-09
- Publication Date
- 2026-11-17
AI Technical Summary
In existing technologies, edge devices struggle to balance inference quality and processing efficiency when handling inference tasks. This leads to problems such as simple tasks being unnecessarily sent to the cloud, increasing communication overhead, and complex tasks experiencing a decline in inference quality due to insufficient computing power.
By evaluating the complexity of inference task requests and dynamically determining the target inference node based on network quality information, and generating a compressed context on the edge, simple tasks are processed quickly locally on the edge, while complex tasks are fully processed on remote nodes, reducing the overhead of repeated inference.
It effectively solves the problem of balancing reasoning quality and processing efficiency when handling reasoning tasks. Simple tasks are processed quickly locally, while complex tasks are fully processed on remote nodes, reducing communication latency and computational overhead, and improving overall processing efficiency.
Smart Images

Figure CN123123353A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of computer technology, and in particular to a reasoning task request processing method, apparatus, computer equipment, storage medium, and computer program product. Background Technology
[0002] With the development of large language models, lightweight language models are being deployed on edge devices in scenarios such as industrial production, intelligent manufacturing, and smart homes. While performing inference locally on edge devices can reduce data transmission, their computing power and model size are limited, which may lead to insufficient inference quality when handling complex inference tasks. Therefore, it is necessary to leverage cloud computing power for inference.
[0003] Currently, edge devices typically employ relatively fixed processing strategies for all inference tasks, such as uploading tasks uniformly to the cloud for processing or statically routing tasks based solely on task type. However, this fixed processing strategy leads to simple tasks potentially being unnecessarily sent to the cloud, generating additional communication overhead and response latency, while complex tasks may be processed locally on the edge, resulting in degraded inference quality due to insufficient computing power on the edge. Furthermore, when tasks are processed by the cloud, the edge device needs to send the task request and related context to the cloud, resulting in a large amount of communication data, further increasing communication overhead and response latency.
[0004] Therefore, existing technologies suffer from the technical problem of balancing reasoning quality and processing efficiency when handling reasoning tasks. Summary of the Invention
[0005] Based on this, the purpose of this application is to at least solve one of the above-mentioned technical defects, especially the technical defect in the prior art that it is difficult to balance reasoning quality and processing efficiency when processing reasoning tasks. This application provides a reasoning task request processing method, apparatus, computer equipment, storage medium and computer program product.
[0006] In a first aspect, this application provides an inference task request processing method, applied to an edge inference node, wherein the edge inference node is communicatively connected to at least one remote inference node; the method includes: Obtain the inference task request, evaluate the complexity of the inference task request, and obtain a complexity score; If the complexity score is greater than or equal to the preset first threshold, then obtain the network quality information between the end-side inference node and each remote inference node; Based on network quality information and complexity scores, the target inference node is determined from the edge inference node and each remote inference node; When the target inference node is a remote inference node, the initial inference result on the edge side in response to the inference task request is obtained, and based on the inference task request and the initial inference result on the edge side, an edge-side compressed context is generated and sent to the remote inference node; the remote inference node is used to perform inference based on the edge-side compressed context. Receive the inference results returned by the remote inference node as the response to the inference task request.
[0007] In an exemplary embodiment, the complexity of a reasoning task request is evaluated to obtain a complexity score, including: The intent of the inference task request is identified, the intent type is determined, and the initial complexity score is determined based on the intent type. Determine the number of entities involved in the reasoning task request, the number of reasoning steps, the depth of context dependency, and the matching confidence; the matching confidence represents the degree of matching between the reasoning task request and the knowledge entries in the local knowledge base on the client side. The initial complexity score is adjusted based on the number of entities, the number of inference steps, the depth of context dependencies, and the matching confidence to obtain the final complexity score.
[0008] In an exemplary embodiment, remote inference nodes include edge inference nodes and cloud inference nodes; network quality information includes network quality levels from edge inference nodes to edge inference nodes and network quality levels from edge inference nodes to cloud inference nodes; determining a target inference node from edge inference nodes and each remote inference node based on network quality information and complexity scores includes: Based on the complexity range to which the complexity score belongs, and the network quality level from the edge inference node to the edge inference node and the network quality level from the edge inference node to the cloud inference node, the routing strategy for the inference task request is determined in the pre-built routing strategy matrix. Based on the routing strategy, the target inference node is determined among the end-side inference nodes, edge inference nodes, and cloud inference nodes.
[0009] In an exemplary embodiment, based on the inference task request and the initial inference result on the device side, a device-side compressed context is generated, including: Extract the core semantic summary of the reasoning task request, extract key conclusions and uncertainties from the preliminary reasoning results on the edge, and identify the entities involved in the reasoning task request and generate reference identifiers for the entities. When the remote inference node has cached the complete description of the entity, an edge-side compressed context is generated based on the core semantic summary, key conclusions, uncertainties, and entity reference identifiers. When the remote inference node does not cache the complete description of the entity, an edge-side compressed context is generated based on the core semantic summary, key conclusions, uncertainties, entity reference identifiers, and the complete description of the entity.
[0010] Secondly, this application provides a method for processing inference task requests, applied to edge inference nodes, wherein the edge inference nodes are communicatively connected to end-side inference nodes and cloud inference nodes; the method includes: The receiver sends the end-side compressed context sent by the end-side inference node; the end-side compressed context consists of the core semantic summary of the inference task request, the key conclusions in the end-side preliminary inference results, the uncertainties in the end-side preliminary inference results, and the entity information involved in the inference task request; the entity information is the reference identifier or complete description of the entity; Inference is performed based on the edge-side compressed context to obtain edge inference results; If the edge inference result does not represent any unresolved uncertainties, the edge inference result is sent to the edge inference node as a response to the inference task request. If there are unresolved uncertainties in the edge inference results, an edge compressed context is generated based on the edge compressed context and the edge inference results and sent to the cloud inference node; Receive the inference results returned by the cloud inference node and send them to the edge inference node as a response to the inference task request.
[0011] In an exemplary embodiment, an edge compression context is generated based on the edge compression context and the edge inference result, including: Extract confirmed facts, excluded possibilities, and unresolved uncertainties from the edge reasoning results; Based on the confirmed facts, excluded possibilities, unresolved uncertainties in the edge reasoning results, as well as the core semantic summary, key conclusions, and uncertainties in the edge compressed context, an edge compressed context is generated.
[0012] Thirdly, this application provides an inference task request processing apparatus, applied to an edge inference node, wherein the edge inference node is communicatively connected to at least one remote inference node; the apparatus includes: The edge evaluation module is used to obtain inference task requests, evaluate the complexity of inference task requests, and obtain a complexity score. The edge acquisition module is used to acquire network quality information between the edge inference node and each remote inference node if the complexity score is greater than or equal to a preset first threshold. The endpoint determination module is used to determine the target inference node from the endpoint inference node and each remote inference node based on network quality information and complexity score; The endpoint sending module is used to obtain the endpoint preliminary inference result in response to the inference task request when the target inference node is a remote inference node, and to generate an endpoint compressed context based on the inference task request and the endpoint preliminary inference result and send it to the remote inference node; the remote inference node is used to perform inference based on the endpoint compressed context. The edge receiving module is used to receive the inference results returned by the remote inference node as a response to the inference task request.
[0013] Fourthly, this application provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the above-described method.
[0014] Fifthly, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the above-described method.
[0015] Sixthly, this application provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the above-described method.
[0016] As can be seen from the above technical solutions, the embodiments of this application have the following advantages: The inference task request processing method, apparatus, computer equipment, storage medium, and computer program product provided in this application first assess the complexity of the inference task request and then dynamically determine the target inference node based on network quality information. This allows simple tasks to be processed quickly locally on the endpoint, while complex tasks are fully processed on remote inference nodes. This avoids the high latency and high cost of sending all tasks to remote nodes, and also avoids insufficient inference quality caused by processing all tasks on the endpoint. Furthermore, when it is necessary to send to a remote node, an endpoint-side compressed context is generated to compress and transmit the preliminary inference results and core task information already completed on the endpoint. This reduces the overhead of repetitive inference on the remote node, allowing it to focus on key issues that cannot be resolved on the endpoint. This improves overall processing efficiency while ensuring inference quality, effectively solving the technical problem of balancing inference quality and processing efficiency when processing inference tasks. Attached Figure Description
[0017] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0018] Figure 1This application provides an application environment for a reasoning task request processing method in an embodiment of this application. Figure 2 A flowchart illustrating a reasoning task request processing method provided in an embodiment of this application; Figure 3 A flowchart illustrating a task complexity evaluation method provided in an embodiment of this application; Figure 4 A flowchart illustrating another reasoning task request processing method provided in an embodiment of this application; Figure 5 A flowchart illustrating a method for generating and passing cross-level compression contexts provided in an embodiment of this application; Figure 6 A schematic diagram illustrating a closed-loop process for continuous learning of routing strategies provided in an embodiment of this application; Figure 7 This application provides a schematic diagram of the overall process architecture of a reasoning task request processing method. Figure 8 A schematic diagram of the structure of a reasoning task request processing device provided in an embodiment of this application; Figure 9 A schematic diagram of another inference task request processing device provided in an embodiment of this application; Figure 10 This is a schematic diagram of the internal structure of a computer device provided in an embodiment of this application. Detailed Implementation
[0019] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0020] The inference task request processing method provided in this application embodiment can be applied to, for example, Figure 1In the application environment shown, the edge inference node 102, edge inference node 104, and cloud inference node 106 are connected via network communication. The edge inference node 102 is used to obtain inference task requests, evaluate the complexity of the inference task requests to obtain a complexity score; if the complexity score is greater than or equal to a preset first threshold, the edge inference node 102 obtains the network quality information between the edge inference node and each remote inference node, and determines the target inference node from the edge inference node and each remote inference node based on the network quality information and the complexity score; when the target inference node is a remote inference node, the edge inference node 102 obtains the edge preliminary inference result for the inference task request, generates an edge compressed context based on the inference task request and the edge preliminary inference result and sends it to the remote inference node (edge inference node 104 or cloud inference node 106), and receives the inference result returned by the remote inference node (edge inference node 104 or cloud inference node 106) as the response result. Among them, the edge inference node 102 can be, but is not limited to, an x60 AI controller, embedded control device, industrial gateway, smart home central control device, smart vehicle terminal, edge computing box, etc. The edge inference node 104 can be, but is not limited to, an edge server, industrial computer, edge computing node, edge computing gateway, etc. deployed in a park or factory. The cloud inference node 106 can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services.
[0021] In some embodiments of this application, edge inference nodes and cloud inference nodes are deployed with inference models of different parameter sizes and processing capabilities, forming a three-tiered capability hierarchy. Edge inference nodes deploy lightweight models with fewer than one billion parameters, which are quantized to integers and run on local neural network acceleration units. The end-to-end inference latency is 30 to 100 milliseconds, suitable for simple tasks such as state queries, single-device control, and fixed-template responses. The context window is approximately 2,000 to 4,000 words, and multi-step inference capability is limited to one or two steps. Edge inference nodes deploy medium-sized models with tens of billions of parameters, deployed on edge servers in parks or factories. Edge devices access these nodes via LANs or dedicated lines, with typical latency of 200 to 500 milliseconds. These are suitable for cross-device linkage, moderately complex state inference, multi-turn dialogue understanding, and preliminary fault diagnosis. The context window is approximately 32,000 words, supporting three to eight steps of chained inference. Large-scale models with hundreds of billions to trillions of parameters are deployed on cloud-based inference nodes, provided through public or private cloud services. Typical latency is 500 to 3,000 milliseconds, suitable for complex cross-system causal reasoning, comprehensive judgment of broad knowledge, complex natural language generation, and strong logic programming. The context window is typically larger than 128,000 words, supporting complex inference chains of dozens of steps. The three levels of nodes form a complementary capability gradient, with latency, cost, and inference capabilities ranging from low to high.
[0022] In one exemplary embodiment, Figure 2 This is a flowchart illustrating a reasoning task request processing method provided in an embodiment of this application, as shown below. Figure 2 As shown, a method for processing inference task requests is provided, which is applied to... Figure 1 Taking the end-side inference node 102 as an example, the explanation includes the following steps S202 to S210. Wherein: Step S202: Obtain the inference task request and evaluate the complexity of the inference task request to obtain a complexity score.
[0023] Among them, edge inference nodes refer to device nodes deployed on the local site with limited computing resources and local inference capabilities, such as the x60 AI controller, on which a lightweight edge model runs for preliminary processing of inference tasks.
[0024] Among them, remote inference nodes refer to computing nodes that are connected to edge inference nodes via a network and have stronger inference capabilities than edge inference nodes. For example, edge servers deployed in parks or factories, or cloud inference services deployed in public or private clouds.
[0025] Among them, inference task requests refer to task inputs proposed by users or upper-layer applications that need to be processed by the inference model, such as device control instructions in natural language form, status query requests, fault diagnosis requests, etc. The request may include task description text, context information and priority markers.
[0026] Complexity assessment refers to the process of quantitatively analyzing the difficulty of a reasoning task request. A complexity score is calculated by integrating features from multiple dimensions to determine which level of reasoning node is suitable to handle the task.
[0027] The complexity score is a quantitative value of the complexity assessment output, used to characterize the difficulty of processing the reasoning task request. The higher the score, the more complex the task and the higher the requirement for reasoning ability.
[0028] Optionally, after receiving an inference task request, the edge inference node first performs a complexity assessment on the request and obtains a complexity score.
[0029] Step S204: If the complexity score is greater than or equal to a preset first threshold, then obtain the network quality information between the end-side inference node and each remote inference node.
[0030] The preset first threshold is a critical value for the complexity score used to distinguish between simple and non-simple tasks, such as 4 points. When the complexity score is less than this threshold, the task can be directly processed by the edge inference node; when the complexity score is greater than or equal to this threshold, it is necessary to further combine network quality information to determine the target inference node.
[0031] Among them, network quality information refers to information used to characterize the communication link status between the end-side inference node and each remote inference node, such as network latency, bandwidth, packet loss rate or comprehensive network quality level, which is used to help determine whether the remote inference node is reachable.
[0032] Optionally, if the complexity score is lower than a preset first threshold, it indicates that the task is relatively simple and can be directly inferred by the local lightweight model from the edge inference node without querying network quality information. If the complexity score is greater than or equal to the preset first threshold, it indicates that the task may exceed the capabilities of the edge, and it is necessary to obtain network quality information between the edge inference node and each remote inference node to determine whether the remote nodes are available.
[0033] In some embodiments of this application, the edge inference node acquires network quality information with each remote inference node, including round-trip latency, available bandwidth, and packet loss rate from the edge to the edge inference node, and round-trip latency, available bandwidth, and packet loss rate from the edge to the cloud inference node. Based on these indicators, network quality is divided into four levels: a good level indicates low latency, sufficient bandwidth, and no packet loss; a degraded level indicates slightly increased latency or limited bandwidth but still possible communication; a poor level indicates significantly increased latency or increased packet loss rate, and unstable communication; and a disconnected level indicates that a valid connection cannot be established. The network quality level can be obtained through periodic probing or real-time measurement and serves as a key input in subsequent routing decisions.
[0034] Step S206: Based on network quality information and complexity scores, determine the target inference node from the end-side inference nodes and each remote inference node.
[0035] The target inference node refers to the inference node that is finally selected to handle the current inference task request after complexity evaluation and network quality judgment. It can be an edge inference node or any remote inference node.
[0036] Optionally, the target inference node can be determined according to a preset routing strategy based on network quality information and complexity score.
[0037] Step S208: When the target inference node is a remote inference node, the preliminary inference result on the end side in response to the inference task request is obtained, and an end-side compressed context is generated and sent to the remote inference node based on the inference task request and the preliminary inference result on the end side; the remote inference node is used to perform inference based on the end-side compressed context.
[0038] Among them, the preliminary inference result on the edge side refers to the partial conclusions obtained by the edge inference node after quickly predicting the inference task request using the local lightweight model before determining that it needs to be sent to remote processing. It may include confirmed facts, preliminary judgments and uncertainties.
[0039] Among them, the edge-side compressed context refers to the context information generated by the edge-side inference node after summarizing and compressing the original task description and the edge-side preliminary inference results when sending inference tasks to remote inference nodes. This smaller context information is used to reduce the amount of data transmitted and preserve semantic continuity.
[0040] Optionally, when the target inference node is an edge inference node, inference is performed directly locally and the result is returned. When the target inference node is a remote inference node, the edge inference node first uses the local model to quickly predict the task request to obtain the edge preliminary inference result, then extracts the core semantics of the task and the preliminary conclusion to generate an edge compressed context, and sends the compressed context to the remote inference node, which then performs complete inference based on the context.
[0041] Step S210: Receive the inference result returned by the remote inference node as a response to the inference task request.
[0042] The reasoning result refers to the final answer or execution suggestion generated by the target reasoning node in response to the reasoning task request.
[0043] The response result refers to the final output returned by the end-side inference node to the task initiator, that is, the inference result returned by the remote inference node.
[0044] Optionally, finally, the end-side inference node receives the inference result returned by the remote inference node and returns it as a response result.
[0045] In the aforementioned inference task request processing method, the complexity of the inference task request is first assessed, and then the target inference node is dynamically determined based on network quality information. This allows simple tasks to be processed quickly locally on the endpoint, while complex tasks are fully processed on remote inference nodes. This avoids the high latency and high cost of sending all tasks to remote nodes, and also avoids insufficient inference quality caused by processing all tasks on the endpoint. Furthermore, when it is necessary to send tasks to remote nodes, an endpoint-side compressed context is generated to compress and transmit the preliminary inference results and core task information already completed on the endpoint. This reduces the overhead of repeated inference on remote nodes, allowing them to focus on key issues that cannot be resolved on the endpoint. This improves overall processing efficiency while ensuring inference quality, effectively solving the technical problem of balancing inference quality and processing efficiency when processing inference tasks.
[0046] In an exemplary embodiment, the complexity of a reasoning task request is evaluated to obtain a complexity score, including: identifying the intent of the reasoning task request, determining the intent type, and determining an initial complexity score based on the intent type; determining the number of entities involved in the reasoning task request, the number of reasoning steps, the depth of context dependency, and the matching confidence; the matching confidence characterizes the degree of matching between the reasoning task request and knowledge entries in the local knowledge base on the client side; and correcting the initial complexity score based on the number of entities, the number of reasoning steps, the depth of context dependency, and the matching confidence to obtain a complexity score.
[0047] Intent recognition refers to the process of using a classification model to determine the semantic type of a reasoning task request. A lightweight intent classifier can be deployed on the device side to classify requests into intent types such as status query, single device control, multi-device linkage, fault diagnosis, cross-system reasoning, and open-ended question answering.
[0048] Among them, intent type refers to the category label output by intent recognition, and different types of intents correspond to different basic complexities.
[0049] The initial complexity score refers to the base score determined based on the intent type, which serves as the starting point for complexity assessment.
[0050] For example, status query type gets 1 point, single device control type gets 2 points, multi-device linkage type gets 5 points, fault diagnosis type gets 7 points, cross-system reasoning type gets 9 points, and open-ended question answering type gets 8 points.
[0051] The number of entities refers to the number of independent device or system entities involved in the inference task request, such as workshops, equipment, and sensors mentioned in the request, which can be obtained through entity recognition and matching statistics with the local device registry.
[0052] The number of reasoning steps refers to the length of the reasoning chain estimated based on logical connectors, conditional clauses, or causal words in the task request. Keywords such as "if," "because," "therefore," "compare," and "synthesize" indicate that the more logical reasoning words appear, the more reasoning steps are required.
[0053] Context dependency depth refers to the degree to which the current task request depends on the number of rounds of historical dialogue or historical state, such as no historical dependency, dependency on the most recent 1 to 2 rounds, dependency on a longer history, or need to cross the history of a session.
[0054] Among them, the matching confidence refers to the degree of matching between the reasoning task request and the knowledge entries in the local knowledge base on the client side. It can be calculated by vector similarity. The higher the matching confidence, the higher the coverage of the local knowledge base on the client side for the task, and the lower the difficulty of task processing.
[0055] The local knowledge base on the edge refers to a collection of domain knowledge stored locally on the edge inference node, such as equipment manuals, common fault databases, and standard operating procedure databases. It is stored in vectorized form to support fast local matching.
[0056] The "correction" refers to the process of adjusting the initial complexity score positively or negatively based on the number of entities, the number of inference steps, the depth of contextual dependencies, and the matching confidence, so that the final complexity score more accurately reflects the actual processing difficulty of the task.
[0057] The complexity score refers to the final quantified value obtained after correction, which is used for subsequent routing decisions.
[0058] Optionally, refer to Figure 3 When the edge inference node evaluates the complexity of an inference task request, it first calls a lightweight intent classifier to semantically classify the request text, determine its intent type, and map it to an initial complexity score. Then, it uses entity recognition and matching with the local registry to count the number of unique entities involved in the request, adding a correction score based on the number range. Simultaneously, it scans for logical reasoning keywords in the request text, estimates the required inference steps, and adds a correction score. Next, it determines the contextual dependency depth based on the historical dependencies of the current session and adds a correction score. Finally, it encodes the request into a vector and performs similarity matching with the local knowledge base, adjusting the complexity score based on the matching score; a higher matching degree results in a lower score. Combining all these steps yields the final complexity score.
[0059] In addition, this application also provides a network degradation exemption check mechanism, that is, for tasks marked with CRITICAL priority (such as security alarm related inference), regardless of the complexity score, when the network is unavailable, L1 processing is forced and the alarm mark "This judgment is made based on the limited capabilities of the end side and manual review and confirmation are recommended" is immediately triggered, without entering the delay queue to wait, and priority is given to ensuring the timeliness of security response.
[0060] In this embodiment, the inference task request is comprehensively and quantitatively evaluated from multiple dimensions, including intent type, number of entities, number of inference steps, depth of contextual dependencies, and matching confidence. This allows the complexity score to reflect the actual processing difficulty of the task from different perspectives, such as semantic type, object scope, inference depth, historical dependencies, and local knowledge coverage, avoiding misjudgments of task complexity based on a single feature. Based on this multi-dimensional evaluation, subsequent routing decisions can more accurately determine whether the task should be processed locally on the endpoint or sent to a remote node. This ensures that simple tasks do not suffer efficiency losses due to excessive uploading, and complex tasks do not suffer quality degradation due to errors remaining on the endpoint, thus balancing inference quality and processing efficiency overall.
[0061] In an exemplary embodiment, remote inference nodes include edge inference nodes and cloud inference nodes; network quality information includes network quality levels from the edge inference node to the edge inference node and network quality levels from the edge inference node to the cloud inference node; determining the target inference node from the edge inference node and each remote inference node based on the network quality information and complexity score includes: determining a routing strategy for the inference task request in a pre-built routing strategy matrix based on the complexity interval to which the complexity score belongs, and the network quality levels from the edge inference node to the edge inference node and from the edge inference node to the cloud inference node; and determining the target inference node from the edge inference node, edge inference node, and cloud inference node based on the routing strategy.
[0062] Among them, edge inference nodes refer to medium-sized inference nodes deployed on edge servers in parks or factories, such as 14B parameter level models. Their inference capabilities are between the edge and the cloud, and they can be accessed through local area networks or dedicated lines.
[0063] Among them, cloud inference nodes refer to large-scale inference services deployed in public or private clouds, such as models with hundreds of billions to trillion parameters. They have the strongest inference capabilities and the broadest knowledge coverage, and are usually accessed through the public network.
[0064] Among them, network quality level refers to the result of classifying the quality of communication links from end-side inference nodes to edge inference nodes or cloud inference nodes, such as dividing them into four levels: good, degraded, poor, and disconnected.
[0065] The complexity range refers to multiple numerical ranges divided according to the complexity score. For example, the range for simple tasks is 0 to 4 points, the range for medium tasks is 5 to 10 points, and the range for complex tasks is 11 to 20 points. Different ranges correspond to different default routing preferences.
[0066] The routing strategy matrix refers to a pre-built mapping table used to jointly determine the routing strategy based on the complexity range and the network quality level. For example, when the complexity is moderate and the network is good, the route is routed to the edge inference node, and when the complexity is complex and the cloud network is good, the route is routed to the cloud inference node.
[0067] The routing strategy refers to the decision rules used to determine the target inference node based on matrix matching, such as direct local processing, sending to the edge, sending to the cloud, or degraded processing.
[0068] The target inference node refers to the inference node ultimately selected according to the routing strategy to execute the current inference task. It can be an edge inference node or a cloud inference node.
[0069] Optionally, after obtaining the complexity score and network quality information, the edge inference node first determines which complexity range the complexity score falls into. Then, it obtains the network quality levels for the edge inference node and the cloud inference node, respectively. Based on the complexity range and the two network quality levels, it searches for the corresponding routing strategy in a pre-defined routing strategy matrix. For example, when the complexity score is in the simple range, regardless of network quality, it is routed to the edge inference node for local processing; when the complexity is in the medium range and the network quality to the edge is good or degraded, it is routed to the edge inference node; when the complexity is in the complex range and the network quality to the cloud is good, it is routed to the cloud inference node; when the cloud network is poor but the edge network is acceptable, the task that should have been routed to the cloud is downgraded and routed to the edge inference node, and a quality declaration is added to the result; when neither the edge nor the cloud is reachable, it is downgraded to local processing at the edge inference node and the result is marked. Finally, the target inference node is selected according to the determined strategy.
[0070] In this embodiment, by mapping complexity scores to complexity intervals and combining them with two levels of network quality—from the edge to the cloud and from the device side—routing strategies are determined within a pre-built routing strategy matrix, achieving a linkage decision between complexity and network conditions. This mechanism ensures that complex tasks are prioritized for processing by high-capacity nodes while fully considering the impact of actual network conditions on the feasibility and latency of remote inference. It can automatically downgrade routes when network conditions are poor, avoiding long waiting times or task failures due to network unavailability or poor quality. Simultaneously, it effectively utilizes the strong inference capabilities of remote nodes to ensure quality when the network is good. This routing strategy, combining complexity awareness and network awareness, enables the system to select the most suitable processing node when facing different complexities and network conditions, thus balancing inference quality and processing efficiency.
[0071] In some embodiments of this application, the target inference node is determined based on the complexity range to which the complexity score belongs, and the two-level network quality levels from the edge to the cloud and from the endpoint, according to a pre-built routing strategy matrix. Specifically, when the task is of simple complexity, regardless of network quality, it is routed to the endpoint inference node for local processing. When the task is of medium complexity and the edge network quality is good or degraded, it is routed to the edge inference node; if the edge network quality is poor or disconnected, it is degraded to the endpoint inference node for processing, and an explanation is added to the result stating that the local model is used to handle the medium-complexity task due to network limitations. When a task is of high complexity and the cloud network quality is good, it is routed to a cloud inference node. If the cloud network quality is degraded or poor while the edge network quality is good or degraded, it is degraded to an edge inference node for processing. A quality statement is added to the returned results, indicating to the user that a lower-level model is being used due to network limitations, and suggesting that the task be re-initiated after network recovery for a more comprehensive analysis. If both the cloud and edge networks are poor or disconnected, it is degraded to an edge inference node. The edge model performs a conservative response—first providing an approximate answer based on the local knowledge base, explicitly stating the low confidence level, and marking the task as to be reprocessed after network recovery, and adding it to a delay queue. When the network is restored to a reachable state, the system automatically retrieves the task from the delay queue, re-initiates inference according to the original routing level, and pushes the updated results to the user. For tasks marked as urgent priority (e.g., security alarm-related inference), regardless of the complexity score, edge processing is forced when the network is unavailable, with a manual review prompt added, and the task is not added to the delay queue to prioritize response timeliness.
[0072] In an exemplary embodiment, a client-side compressed context is generated based on the inference task request and the preliminary inference result on the client side. This includes: extracting the core semantic summary of the inference task request; extracting key conclusions and uncertainties from the preliminary inference result on the client side; identifying the entities involved in the inference task request and generating reference identifiers for the entities; when the remote inference node has cached the complete description of the entity, the client-side compressed context is generated based on the core semantic summary, key conclusions, uncertainties, and reference identifiers of the entity; when the remote inference node has not cached the complete description of the entity, the client-side compressed context is generated based on the core semantic summary, key conclusions, uncertainties, reference identifiers of the entity, and the complete description of the entity.
[0073] Among them, the core semantic summary refers to a concise description that retains the core meaning of the task objective after summarizing the original text of the reasoning task request. For example, a detailed description can be compressed into "Decision on handling temperature anomalies in Workshop No. 3".
[0074] Among them, key conclusions refer to facts that have been confirmed or partial conclusions that have been completed in the preliminary reasoning results from the end side, such as "the current temperature is 32℃, which is 18 to 28℃ outside the normal range".
[0075] Uncertain matters refer to key points that cannot be determined or require further judgment in the preliminary reasoning results from the end side, such as "the status of the ventilation system is unknown" or "whether it is necessary to link with other workshops".
[0076] In this context, an entity refers to a specific device, system, or object involved in the reasoning task request, such as "Workshop No. 3" or "exhaust fan".
[0077] The entity reference identifier is a short identifier used to uniquely identify an entity, such as an entity ID or number, which is used to replace the full description when the full description of the entity has been cached on the remote inference node to save transmission volume.
[0078] The complete description of an entity refers to its detailed attribute information, such as device type, location, parameter range, etc., which usually involves a large amount of data.
[0079] Among them, the complete description of a cached entity refers to the complete description of the entity that has been stored locally on the remote inference node. It can be retrieved directly through the entity reference identifier without having to transmit the complete description repeatedly in the context.
[0080] In this context, a complete description of an uncached entity refers to an entity whose complete description does not exist locally on the remote inference node. The complete description needs to be carried in the context so that the remote inference node can correctly understand the task.
[0081] Among them, the edge-side compressed context refers to a compressed data structure consisting of core semantic summaries, key conclusions, uncertainties, entity reference identifiers, and optional complete entity descriptions, which is used to transmit reasoning task information between levels.
[0082] Optionally, after determining that the task request needs to be sent to the remote inference node, the edge inference node extracts a summary from the original inference task request to generate a concise description of the task objective. Simultaneously, it extracts confirmed key facts and preliminary conclusions, as well as a list of unresolved uncertainties, from the initial inference results on the edge. For each entity involved in the task request, a corresponding entity reference identifier is generated. Then, it checks whether the remote inference node has cached the complete description of the entity. If it has, only the entity reference identifier is included in the context; if not, the complete description of the entity must also be included. Finally, this information is organized into an edge compressed context, such as structured data containing fields like a task summary, key facts, conclusion summary, entity identifier list, and upgrade reason, and sent to the remote inference node.
[0083] In this embodiment, by extracting the core semantic summary of the inference task request and the key conclusions and uncertainties in the initial inference results from the endpoint, and replacing the shareable complete entity description with entity reference identifiers, the amount of context data sent from the endpoint to the remote node is significantly compressed, reducing network transmission burden and parsing overhead for remote nodes. Simultaneously, the compressed context retains the core task information and the inference results already completed on the endpoint, allowing remote inference nodes to focus directly on the unresolved uncertainties on the endpoint without needing to re-understand the complete background, reducing redundant computation and improving overall processing efficiency. This mechanism effectively reduces the communication and computation costs of cross-node inference while ensuring the semantic integrity of the context, balancing inference quality and processing efficiency.
[0084] In some embodiments of this application, the client-side compressed context adopts a structured format, which includes, in addition to the core semantic summary of the task, key conclusions of the client-side preliminary reasoning, and uncertainties, a session identifier, the current round number, the previous processing level, a timestamp, an entity reference list, and an upgrade reason explanation. The core semantic summary of the task is a high-level summary of the original request (e.g., "decision on handling abnormal temperature in Workshop 3"); key conclusions are facts confirmed by the client (e.g., "current temperature is 32 degrees Celsius, exceeding the normal range of 18 to 28 degrees Celsius"); uncertainties are specific points that the client cannot determine (e.g., "ventilation system status unknown," "whether other workshops need to be linked"); the entity reference list contains the unique identifier and type of each entity, and if the remote node does not cache the complete description of the entity, a complete description field is appended; the upgrade reason explanation explains why remote node intervention is needed (e.g., "decision needs to be made by combining the ventilation system status and the impact of cross-workshop linkage"). The data volume of this compressed context is compressed by approximately 85% to 90% compared to transmitting the complete original dialogue and device status, significantly reducing transmission overhead.
[0085] In some embodiments of this application, each inference node maintains a shared entity description cache. When an edge inference node first sends a compressed context involving an entity to a remote inference node, if the remote node has not cached the complete description of the entity, the context carries the complete description. Upon receiving it, the remote node stores it in its local cache and marks the synchronization time and source level. In subsequent sessions or interactions involving the same entity, the edge only transmits the entity reference identifier, and the remote node retrieves the complete description from its local cache, eliminating the need for repeated transmission. Cache entries can be set with a lifespan or updated on demand. When the entity's state changes, the edge actively triggers a synchronization update. This mechanism further reduces the amount of redundant data in cross-level communication.
[0086] In some embodiments of this application, after receiving the inference result returned by the remote inference node, the edge inference node caches the result summary (including the decision conclusion, the action to be executed, the confidence score, and whether manual review is recommended) locally and associates it with the current session identifier. Subsequent simple follow-up questions in the same session (e.g., "When will the previous suggestion be executed?") can be answered independently by the edge lightweight model based on the cached summary, without needing to be upgraded to the remote node again. Simultaneously, the edge inference node records detailed information for each routing decision, including task identifier, complexity score, two-level network quality level, actual routing level, actual latency, estimated latency and cost if routing to the cloud, actual cloud call costs avoided, and result quality score. The system summarizes this statistical information daily and generates reports for operations and maintenance personnel to evaluate the effectiveness of the routing strategy.
[0087] In one exemplary embodiment, Figure 4 This is a flowchart illustrating a reasoning task request processing method provided in an embodiment of this application, as shown below. Figure 4 As shown, a method for processing inference task requests is provided. Taking the application of this method to an edge inference node 104 as an example, the edge inference node is communicatively connected to the terminal inference node and the cloud inference node; the method includes the following steps S402 to S410. Wherein: Step S402: Receive the end-side compressed context sent by the end-side inference node; the end-side compressed context consists of the core semantic summary of the inference task request, the key conclusions in the end-side preliminary inference results, the uncertainties in the end-side preliminary inference results, and the entity information involved in the inference task request; the entity information is the reference identifier or complete description of the entity.
[0088] Among them, edge inference nodes refer to medium-sized inference nodes deployed on edge servers in parks or factories. They have stronger inference capabilities and broader knowledge coverage than edge inference nodes, but are weaker than cloud inference nodes.
[0089] Among them, edge-side compressed context refers to the compressed context information generated by the edge-side inference node when upgrading the task to the edge inference node, which includes the core semantic summary, key conclusions, uncertainties and entity information of the inference task.
[0090] Optionally, the edge inference node receives the end-side compressed context sent by the end-side.
[0091] Step S404: Perform inference based on the edge-side compressed context to obtain the edge inference result.
[0092] Among them, edge reasoning results refer to the reasoning conclusions or action suggestions output by edge reasoning nodes after performing reasoning based on the edge-side compressed context.
[0093] Optionally, after receiving the compressed context sent by the end side, the edge inference node uses its own medium-sized model to infer the task summary, key conclusions and uncertainties in the context, and attempts to resolve the uncertainties left by the end side.
[0094] Step S406: If the edge inference result indicates that there are no unresolved uncertainties, the edge inference result is sent to the end-side inference node as a response to the inference task request.
[0095] Among them, unresolved uncertainties refer to key unknowns that still exist in the results of marginal reasoning and require stronger reasoning ability or a broader knowledge background to resolve, such as judgments that require data support from the production planning system.
[0096] Optionally, after the inference is completed, if there are no more unsolvable uncertainties in the edge inference result, the edge inference result is sent directly to the end-side inference node.
[0097] Step S408: If there are unresolved uncertainties in the edge inference result representation, generate an edge compressed context based on the edge compressed context and the edge inference result and send it to the cloud inference node.
[0098] Among them, edge compression context refers to the compression context generated by the edge inference node when it needs to be further upgraded to the cloud inference node. It is a compression context that superimposes the edge inference result summary on the edge compression context, and is used to enable the cloud inference node to focus on the remaining unresolved issues.
[0099] Among them, cloud-based inference nodes refer to large-scale inference services deployed in the cloud, which have the strongest multi-step inference capabilities and the broadest knowledge coverage.
[0100] Optionally, after the inference is completed, if there are still unresolved uncertainties in the edge inference results, it indicates that the task may be beyond the edge's capabilities and needs to be further upgraded to the cloud. In this case, the edge inference node generates an edge compressed context based on the edge compressed context and its own inference results. This context contains confirmed facts, excluded possibilities, and remaining uncertainties, and is sent to the cloud inference node.
[0101] Step S410: Receive the inference result returned by the cloud inference node and send it to the edge inference node as a response to the inference task request.
[0102] The response result refers to the final inference result sent by the edge inference node to the end-side inference node.
[0103] Optionally, after cloud processing is completed, the edge inference node receives the final inference result returned by the cloud and forwards it to the edge inference node.
[0104] In the aforementioned inference task request processing method, the edge inference node acts as an intermediate layer between the client and the cloud. After receiving the compressed context from the client, it first attempts to complete the inference based on its own capabilities. Only when there are still unresolved uncertainties will it be further upgraded to the cloud, generating an edge compressed context to transmit the inference progress, so that the cloud does not need to repeat the inference work already completed at the edge. This hierarchical upgrade and compressed transmission mechanism enables moderately complex tasks to be solved efficiently at the edge level, avoiding unnecessary cloud calls; complex tasks can also reduce the amount of cloud input and redundant calculations through preliminary inference at the edge, ensuring the inference quality of complex tasks while shortening the overall processing time, effectively balancing inference quality and processing efficiency.
[0105] In an exemplary embodiment, generating an edge compression context based on the edge compression context and edge inference results includes: extracting confirmed facts, excluded possibilities, and unresolved uncertainties from the edge inference results; and generating the edge compression context based on the confirmed facts, excluded possibilities, unresolved uncertainties from the edge inference results, as well as the core semantic summary, key conclusions, and uncertainties from the edge compression context.
[0106] Among them, confirmed facts refer to factual information that has been verified by the edge reasoning node during the reasoning process, such as "the ventilation system is operating normally".
[0107] Among them, "excluded possibilities" refers to possible situations that have been ruled out by the edge reasoning node through reasoning or evidence, such as "the possibility that the process in Workshop 4 is not affected by airflow disturbance has been ruled out".
[0108] Among them, unresolved uncertainties refer to key points that still exist in the edge reasoning results and require further judgment by cloud reasoning nodes, such as "the sensitivity of the current process stage of Workshop 4 to airflow disturbances requires data support from the production planning system".
[0109] Among them, the core semantic summary refers to the description of the core objective of the reasoning task preserved in the compressed context on the edge.
[0110] Among them, the key conclusions refer to the key conclusions that have been confirmed by the preliminary inference on the end side in the end-side compressed context.
[0111] Uncertainty refers to the unresolved uncertainties in the initial inference of the endpoint recorded in the endpoint compression context.
[0112] Among them, the edge compression context refers to the compression context composed of the aforementioned confirmed facts, excluded possibilities, unresolved uncertainties, and the core semantic summary, key conclusions, and uncertainties in the edge compression context, which is used for cloud inference nodes to continue inference.
[0113] Optionally, when generating an edge compressed context, the edge inference node first extracts confirmed facts, ruled-out possibilities, and unresolved issues that still need to be addressed in the cloud from the edge inference results. Then, it merges this information with the task core summary, key conclusions, and uncertainties from the edge compressed context to form the edge compressed context. This context no longer repeats the complete entity descriptions already transmitted from the edge; it only references entity information in the shared cache through entity reference identifiers. The final generated edge compressed context is then sent to the cloud inference node.
[0114] In this embodiment, confirmed facts, ruled-out possibilities, and unresolved uncertainties are extracted from the edge inference results and merged with the core task information in the edge compressed context to generate an edge compressed context. This allows cloud inference nodes to clearly understand the work completed at both the edge and edge levels, concentrating inference resources on the remaining unresolved key points. This avoids repeatedly verifying confirmed facts and re-ruling ruled-out possibilities, significantly reducing the amount of input data and processing time for cloud inference. At the same time, it ensures that complex inference tasks can obtain high-quality final conclusions in the cloud, thus balancing inference quality and processing efficiency.
[0115] In some embodiments of this application, the system continuously optimizes complexity evaluation weights and routing thresholds based on historical routing decisions and user feedback. Feedback signals include three categories: the first is explicit user correction, where a user is dissatisfied with the answer and requests reprocessing, and the system marks this routing as insufficient level; the second is result consistency verification, where the system periodically compares the judgment results from the edge or end-user with those from the cloud. If the low-level judgment matches the cloud judgment, the system increases confidence in routing this type of task to the low level; otherwise, it decreases confidence; the third is the frequency of follow-up questions, where if a user initiates follow-up questions or re-asks within a short period after a low-level answer, it is considered an implicit signal of insufficient quality. Based on this feedback, the system adjusts the basic complexity weights for each intent type online, using a conservative learning rate, and the adjustment range does not exceed 30% of the initial weight. Adjustments exceeding this range require manual review. Furthermore, the system performs offline analysis weekly, statistically analyzing the actual processing quality distribution across complexity intervals, recalibrating the complexity thresholds, and generating candidate new strategies. The new strategy will first be tested on a small percentage of traffic (e.g., 5%). A / B testing will be used to compare the overall performance of the new and old strategies in terms of latency, cost, and quality. Once the new strategy is confirmed to be superior to the old strategy, the gray-scale ratio will be gradually expanded until a full switch is implemented.
[0116] In some embodiments of this application, when the edge inference node determines that the current task exceeds the processing capacity of the local lightweight model and needs to be upgraded to a higher-level node, the process of generating and transferring the cross-level compressed context is as follows: Figure 5 As shown, it specifically includes: First, the edge inference node performs preliminary processing of the inference task request using its local model, extracting a summary of the completed inference results, including confirmed key facts and some conclusions reached. Simultaneously, the edge inference node identifies uncertainties that cannot be determined or require further verification during the inference process and generates a clear explanation of the escalation reason, informing the remote inference node why intervention is necessary.
[0117] Next, the edge inference node performs reference replacement processing on the device or system entities involved in the task request, that is, it replaces the entity's complete description information with a pre-agreed entity unique identifier. If the remote inference node has not yet cached the entity's complete description locally, the edge node attaches the complete description in the compression context and updates its cache after receiving it; if the remote node has already cached it, only the entity identifier is transmitted, thereby significantly reducing the amount of data transmission.
[0118] Based on the extracted summary, uncertainties, upgrade reasons, and entity reference information, the edge inference node generates a structured compressed context. This compressed context typically contains only 300 to 500 bytes of valid information, while transmitting the complete original dialogue history and device status information usually requires 2,000 to 5,000 bytes, resulting in a compression ratio of approximately 85% to 90%, significantly reducing inter-layer communication overhead.
[0119] After receiving the edge-side compressed context, the remote inference node (e.g., the edge inference node) focuses its inference on the uncertainties remaining on the edge side. If the edge inference node still cannot completely resolve all uncertainties, it uses the same compression mechanism to generate an edge compressed context by combining its own inference progress (including confirmed facts, ruled-out possibilities, and remaining uncertainties) with the core semantic summary in the edge-side compressed context, and then further transmits it to the cloud inference node.
[0120] After receiving the edge compressed context, the cloud-based inference node does not need to re-understand the entire problem background. Instead, it focuses on the remaining key uncertainties for in-depth reasoning based on the reasoning results already completed at both the edge and end-side levels, and outputs the final conclusion.
[0121] Finally, the cloud-based or edge-based inference nodes send the inference results back to the edge inference nodes in a compressed format. The edge inference nodes cache the received result summary locally, including the decision conclusion, the action to be executed, and the confidence score. For subsequent simple follow-up questions in the same session (such as inquiries about the details of previously suggested suggestions), the edge inference nodes can directly answer independently from their local models based on the cached summary information, without triggering the upgrade process again, thus ensuring the semantic continuity and response efficiency of interactions within the same session.
[0122] In some embodiments of this application, the system constructs a continuous learning loop based on historical routing decisions and user feedback, dynamically optimizing the complexity evaluation weights and routing thresholds. (See also...) Figure 6 The specific process is as follows: After each routing decision is executed and the inference result is delivered to the user, the system enters the feedback signal collection phase. Feedback signals come from three channels: First, explicit user correction signals, where the system marks the routing as potentially lacking in a sufficient level when the user is dissatisfied with the returned result and actively requests reprocessing. Second, result consistency verification signals, where the system periodically compares the judgment results from the edge or terminal level with the large cloud model asynchronously. If the lower-level judgment matches the cloud judgment, the system's confidence in routing such tasks to the lower level is strengthened; otherwise, confidence is reduced. Third, follow-up question frequency signals, where if the user initiates follow-up questions or re-asks within a short period after a lower-level model's response, the response is considered to have implicit quality deficiencies and serves as a negative feedback signal.
[0123] Based on the collected feedback signals, the system determines whether the current routing decision is under-routing (i.e., the task is assigned to a level with insufficient capacity, resulting in poor quality) or over-routing (i.e., the task is assigned to a level with excessive capacity, causing unnecessary delays and wasted costs). Based on the determination, the system adjusts the base complexity weights of each intent type online. The adjustment uses a conservative learning rate (e.g., 0.02), and each adjustment does not exceed 30% of the initial weight. Adjustments exceeding this range trigger manual review and confirmation to avoid drastic fluctuations in the routing strategy caused by a single feedback.
[0124] In addition to online fine-tuning, the system performs an offline summary analysis once a week. This analysis statistically analyzes the actual processing quality distribution across different complexity ranges, identifies whether there are any deviations in existing scoring thresholds (such as the boundary between simple and medium, and the boundary between medium and complex), and generates updated threshold configuration schemes to form candidate new strategies.
[0125] After generating candidate strategies, the system first conducts a small-scale, phased trial to verify the new strategy. For example, 5% of the total requests are selected as the experimental group, using the new strategy for routing decisions. The remaining traffic continues to use the old strategy as the control group. A / B testing is used to compare the overall performance of the new and old strategies in terms of latency, cost, and quality. Once it is confirmed that the new strategy is superior to the old one overall, the phased trial is gradually expanded until a full switch is implemented. Through this continuous learning loop combining online fine-tuning and offline optimization, routing accuracy can be continuously improved as the system runs longer.
[0126] To facilitate a holistic understanding of the collaborative relationships among the above steps within the system, Figure 7 An exemplary overall flowchart of a reasoning task request processing method is provided, such as... Figure 7 As shown, the entire inference task request processing method is executed according to the following end-to-end flow: First, the edge inference node receives a task request input. In response to this request, the system initiates two evaluation processes in parallel: First, a real-time task complexity evaluation is performed, which comprehensively considers the intent type of the request, the number of entities involved, the number of required inference steps, the depth of dependence on historical context, and the matching confidence of the edge's local knowledge base, outputting a quantified complexity score. Second, a real-time network quality evaluation is performed, probing the network connection status from the edge inference node to the edge inference node and to the cloud inference node, including round-trip latency, available bandwidth, and packet loss rate, and outputting two levels of network quality.
[0127] Based on complexity scores and two-level network quality ratings, the system queries the routing decision fusion matrix to jointly determine which inference node level the task should be routed to. According to the routing decision results, the task is distributed to the corresponding level in the L1 lightweight edge model, the L2 medium-sized edge model, or the L3 large cloud model for processing. If the task needs to be upgraded from the current level to a higher level, a compressed context passing mechanism is triggered, transmitting the summary of completed inference and remaining uncertainties in a compressed format level by level to ensure semantic continuity of cross-level inference.
[0128] After the target level completes inference, the inference result is sent back to the edge inference node in compressed format. The edge inference node caches the result summary locally for reuse in subsequent simple follow-up questions within the same session, avoiding repeated upgrades. Simultaneously, the system collects feedback signals from this routing decision, including whether the user initiated corrections, the sampling consistency between low-level results and cloud results, and the frequency of subsequent user questions. These signals serve as input for continuous learning of the routing strategy, used to adjust complexity evaluation weights online and optimize routing thresholds offline, forming a complete adaptive closed loop.
[0129] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.
[0130] The following describes the inference task request processing apparatus provided in the embodiments of this application. The inference task request processing apparatus has the same inventive concept as the inference task request processing method described above. The solution to the problem provided by the apparatus is similar to the solution described in the method above. Therefore, the specific limitations of one or more inference task request processing apparatus embodiments provided below can be referred to the limitations of the inference task request processing method above. The inference task request processing apparatus described below and the inference task request processing method described above can be referred to each other, and will not be repeated here.
[0131] In one exemplary embodiment, Figure 8 This is a schematic diagram of the structure of a reasoning task request processing device provided in an embodiment of this application, as shown below. Figure 8 As shown, the inference task request processing device includes: an end-side evaluation module 802, an end-side acquisition module 804, an end-side determination module 806, an end-side sending module 808, and an end-side receiving module 810, wherein: The edge evaluation module 802 is used to obtain inference task requests and evaluate the complexity of inference task requests to obtain a complexity score. The edge acquisition module 804 is used to acquire network quality information between the edge inference node and each remote inference node if the complexity score is greater than or equal to a preset first threshold. The endpoint determination module 806 is used to determine the target inference node from the endpoint inference node and each remote inference node based on network quality information and complexity score; The endpoint sending module 808 is used to obtain the endpoint preliminary inference result for the inference task request when the target inference node is a remote inference node, and generate an endpoint compressed context based on the inference task request and the endpoint preliminary inference result and send it to the remote inference node; the remote inference node is used to perform inference based on the endpoint compressed context. The end-side receiving module 810 is used to receive the inference results returned by the remote inference node as a response to the inference task request.
[0132] In an exemplary embodiment, the edge evaluation module 802 is specifically used to identify the intent of the inference task request, determine the intent type, and determine an initial complexity score based on the intent type; determine the number of entities involved in the inference task request, the number of inference steps, the context dependency depth, and the matching confidence; the matching confidence characterizes the degree of matching between the inference task request and the knowledge entries in the edge local knowledge base; and correct the initial complexity score based on the number of entities, the number of inference steps, the context dependency depth, and the matching confidence to obtain a complexity score.
[0133] In an exemplary embodiment, the remote inference node includes an edge inference node and a cloud inference node; the network quality information includes the network quality level from the end-side inference node to the edge inference node and the network quality level from the end-side inference node to the cloud inference node; the end-side determination module 806 is specifically used to determine the routing strategy for the inference task request in a pre-built routing strategy matrix based on the complexity interval to which the complexity score belongs, and the network quality level from the end-side inference node to the edge inference node and the network quality level from the end-side inference node to the cloud inference node; and based on the routing strategy, determine the target inference node among the end-side inference node, the edge inference node, and the cloud inference node.
[0134] In an exemplary embodiment, the end-side sending module 808 is specifically configured to extract the core semantic summary of the inference task request, extract key conclusions and uncertainties from the end-side preliminary inference results, determine the entities involved in the inference task request, and generate reference identifiers for the entities; when the remote inference node has cached the complete description of the entity, an end-side compressed context is generated based on the core semantic summary, key conclusions, uncertainties, and reference identifiers of the entity; when the remote inference node has not cached the complete description of the entity, an end-side compressed context is generated based on the core semantic summary, key conclusions, uncertainties, reference identifiers of the entity, and the complete description of the entity.
[0135] In one exemplary embodiment, Figure 9 This is a schematic diagram of the structure of a reasoning task request processing device provided in an embodiment of this application, as shown below. Figure 9 As shown, the inference task request processing device includes: an edge receiving module 902, an edge inference module 904, an edge first sending module 906, an edge generation module 908, and an edge second sending module 910, wherein: The edge receiving module 902 is used to receive the end-side compressed context sent by the end-side inference node; the end-side compressed context consists of the core semantic summary of the inference task request, the key conclusions in the end-side preliminary inference results, the uncertainties in the end-side preliminary inference results, and the entity information involved in the inference task request; the entity information is the reference identifier or complete description of the entity. The edge inference module 904 is used to perform inference based on the edge-side compressed context and obtain edge inference results. The edge first sending module 906 is used to send the edge inference result to the end-side inference node as a response to the inference task request if the edge inference result characterizes that there are no unresolved uncertainties. The edge generation module 908 is used to generate an edge compression context based on the edge compression context and the edge inference result when there are unresolved uncertainties in the edge inference result representation and send it to the cloud inference node. The edge second sending module 910 is used to receive the inference results returned by the cloud inference node and send them to the edge inference node as a response to the inference task request.
[0136] In an exemplary embodiment, the edge generation module 908 is used to extract confirmed facts, excluded possibilities, and unresolved uncertainties from the edge reasoning results; and to generate an edge compression context based on the confirmed facts, excluded possibilities, and unresolved uncertainties from the edge reasoning results, as well as the core semantic summary, key conclusions, and uncertainties in the edge-side compressed context.
[0137] In one exemplary embodiment, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any of the inference task request processing methods described above.
[0138] In one exemplary embodiment, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of any of the inference task request processing methods described above.
[0139] In one exemplary embodiment, this application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of any of the inference task request processing methods described in the above embodiments.
[0140] Indicatively, such as Figure 10 As shown, Figure 10 This is a schematic diagram of the internal structure of a computer device 1000 provided in an embodiment of this application. The computer device 1000 can be provided as a server. (Refer to...) Figure 10 The computer device 1000 includes a processing component 1002, which further includes one or more processors, and memory resources represented by memory 1001 for storing instructions, such as application programs, that can be executed by the processing component 1002. The application programs stored in memory 1001 may include one or more modules, each corresponding to a set of instructions. Furthermore, the processing component 1002 is configured to execute instructions to perform the inference task request processing method of any of the above embodiments.
[0141] The computer device 1000 may also include a power supply component 1003 configured to perform power management of the computer device 1000, a wired or wireless network interface 1004 configured to connect the computer device 1000 to a network, and an input / output (I / O) interface 1005. The computer device 1000 may operate on an operating system stored in memory 1001, such as Windows Server™, Mac OS X™, Unix™, Linux™, Free BSD™, or similar.
[0142] Those skilled in the art will understand that Figure 10 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.
[0143] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.
[0144] The various embodiments in this specification are described in a progressive manner. Each embodiment focuses on the differences from other embodiments. The various embodiments can be combined as needed, and the same or similar parts can be referred to each other.
[0145] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A method for processing inference task requests, characterized in that, The method is applied to an edge-side inference node, which is communicatively connected to at least one remote inference node; the method includes: Obtain a reasoning task request and evaluate the complexity of the reasoning task request to obtain a complexity score; If the complexity score is greater than or equal to a preset first threshold, then network quality information between the end-side inference node and each of the remote inference nodes is obtained; Based on the network quality information and the complexity score, the target inference node is determined from the end-side inference node and each of the remote inference nodes; When the target inference node is the remote inference node, the initial inference result on the edge side in response to the inference task request is obtained, and an edge-side compressed context is generated and sent to the remote inference node based on the inference task request and the initial inference result on the edge side; the remote inference node is used to perform inference based on the edge-side compressed context. The inference result returned by the remote inference node is received as the response result to the inference task request.
2. The method according to claim 1, characterized in that, The process of evaluating the complexity of the inference task request to obtain a complexity score includes: The inference task request is subjected to intent recognition to determine the intent type, and an initial complexity score is determined based on the intent type. The number of entities involved in the reasoning task request, the number of reasoning steps, the depth of context dependency, and the matching confidence are determined; the matching confidence represents the degree of matching between the reasoning task request and the knowledge entries in the local knowledge base on the client side. The initial complexity score is corrected based on the number of entities, the number of inference steps, the context dependency depth, and the matching confidence to obtain a final complexity score.
3. The method according to claim 1, characterized in that, The remote inference nodes include edge inference nodes and cloud inference nodes; the network quality information includes the network quality level from the edge inference node to the endpoint inference node and the network quality level from the endpoint inference node to the cloud inference node; The step of determining the target inference node from the edge inference nodes and each of the remote inference nodes based on the network quality information and the complexity score includes: Based on the complexity range to which the complexity score belongs, and the network quality level from the end-side inference node to the edge inference node and the network quality level from the end-side inference node to the cloud inference node, a routing strategy for the inference task request is determined in a pre-built routing strategy matrix. Based on the routing strategy, a target inference node is determined among the end-side inference node, the edge inference node, and the cloud inference node.
4. The method according to claim 1, characterized in that, The step of generating a compressed context on the client side based on the inference task request and the initial inference result on the client side includes: Extract the core semantic summary of the inference task request, extract the key conclusions and uncertainties in the preliminary inference results on the edge, and identify the entities involved in the inference task request and generate reference identifiers for the entities. When the remote inference node has cached the complete description of the entity, an edge-side compressed context is generated based on the core semantic summary, the key conclusions, the uncertainties, and the entity's reference identifier. When the remote inference node does not cache the complete description of the entity, an edge-side compressed context is generated based on the core semantic summary, the key conclusions, the uncertainties, the entity's reference identifier, and the entity's complete description.
5. A method for processing inference task requests, characterized in that, The method is applied to edge inference nodes, which are communicatively connected to end-side inference nodes and cloud-based inference nodes; the method includes: The device receives an end-side compressed context sent by the end-side inference node; the end-side compressed context consists of the core semantic summary of the inference task request, the key conclusions in the end-side preliminary inference results, the uncertainties in the end-side preliminary inference results, and the entity information involved in the inference task request; the entity information is the reference identifier or complete description of the entity. Inference is performed based on the aforementioned end-side compressed context to obtain edge inference results; If the edge inference result indicates that there are no unresolved uncertainties, the edge inference result is sent to the end-side inference node as a response to the inference task request; If the edge inference result indicates the presence of unresolved uncertainties, an edge compression context is generated based on the edge compression context and the edge inference result and sent to the cloud inference node. The inference result returned by the cloud inference node is received and sent to the edge inference node as a response to the inference task request.
6. The method according to claim 5, characterized in that, The step of generating an edge compression context based on the endpoint compression context and the edge inference result includes: Extract confirmed facts, ruled-out possibilities, and unresolved uncertainties from the edge reasoning results; Based on the confirmed facts, excluded possibilities, and unresolved uncertainties in the edge reasoning results, as well as the core semantic summary, key conclusions, and uncertainties in the edge-side compressed context, an edge compressed context is generated.
7. A reasoning task request processing device, characterized in that, An apparatus for use with an edge-side inference node, wherein the edge-side inference node is communicatively connected to at least one remote inference node; the apparatus includes: The edge evaluation module is used to obtain inference task requests and evaluate the complexity of the inference task requests to obtain a complexity score. The edge-side acquisition module is used to acquire network quality information between the edge-side inference node and each of the remote inference nodes if the complexity score is greater than or equal to a preset first threshold. An end-side determination module is used to determine a target inference node from the end-side inference node and each of the remote inference nodes based on the network quality information and the complexity score. The endpoint sending module is used to, when the target inference node is the remote inference node, obtain the endpoint preliminary inference result for the inference task request, and generate an endpoint compressed context based on the inference task request and the endpoint preliminary inference result and send it to the remote inference node; the remote inference node is used to perform inference based on the endpoint compressed context. The end-side receiving module is used to receive the inference result returned by the remote inference node as a response result to the inference task request.
8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.
9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.
10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.
