Method and apparatus for hybrid inference acceleration for operation and maintenance intelligent agent, and medium
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- ELECTRIC POWER RESEARCH INSTITUTE OF STATE GRID JIBEI ELECTRIC POWER CO LTD
- Filing Date
- 2026-05-09
- Publication Date
- 2026-07-24
Smart Images
Figure CN122452804A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of intelligent operation and maintenance technology, and more specifically, to a hybrid reasoning acceleration method, apparatus, device, and medium for operation and maintenance intelligent agents. Background Technology
[0002] With the widespread adoption of cloud computing, microservices, and large-scale distributed systems, the complexity of operational environments has increased dramatically. Tasks such as fault diagnosis, alarm analysis, and change execution are becoming increasingly demanding, making traditional manual operations and maintenance unsustainable. Operational intelligence agents, as AI-driven decision-making systems geared towards automation, understand user intent through natural language and invoke tools to execute operations, becoming a key technology for improving operational efficiency.
[0003] However, operational intelligence agents face significant challenges in inference efficiency during actual deployment. First, the large volume of concurrent alarms during peak periods necessitates the use of large models to handle all requests, resulting in extremely high computational resource consumption and operational costs. Second, the varying difficulty of operational tasks leads to a severe waste of resources when using large models to handle both simple queries and complex root cause analyses.
[0004] Therefore, there is an urgent need for a hybrid reasoning acceleration method for operational intelligent agents that can significantly improve computing resource utilization, reduce operating costs, increase throughput, and reduce latency while ensuring decision quality, thus supporting its application in large-scale production environments. Summary of the Invention
[0005] In view of the above situation, this application provides a hybrid inference acceleration method, apparatus, device and medium for operation and maintenance intelligent agents, which aims to solve the above problems or at least partially solve the above problems.
[0006] Firstly, this application provides a hybrid inference acceleration method for operational intelligent agents, including: Obtain operation and maintenance request data; Based on a pre-set intent recognition model and complexity evaluation model, the intent of the operation and maintenance request data is recognized, and the complexity score corresponding to the operation and maintenance request data is determined. Based on a pre-set routing strategy model, a target inference path is determined according to the intent and complexity score of the operation and maintenance request data, wherein the target inference path is a rule engine path, a small model inference path, or a large model inference path. Based on the target reasoning path, operation and maintenance decision suggestions are generated according to the intent of the operation and maintenance request data.
[0007] For example, after obtaining the operation and maintenance request data, the method further includes: The maintenance request data is segmented into words based on a pre-set word segmentation model; Based on pre-set annotation specifications, the segmented operation and maintenance request data is annotated. Based on a pre-set operation and maintenance dictionary, the labeled operation and maintenance request data is normalized.
[0008] For example, based on a pre-set routing strategy model, a target inference path is determined according to the intent and complexity score of the operation and maintenance request data, wherein the target inference path is a rule engine path, a small model inference path, or a large model inference path, including: When the intent of the maintenance request data is a first type of intent and the complexity score of the maintenance request data is a first preset range, the target reasoning path is determined to be a rule engine path. When the intent of the maintenance request data is the second type of intent and the complexity score of the maintenance request data is within the second preset range, the target inference path is determined to be the small model inference path. When the intent of the maintenance request data is a third type of intent and the complexity score of the maintenance request data is a third preset range, the target inference path is determined to be a large model inference path.
[0009] For example, the target inference path includes multiple weight layers, and the weight precision and activation value precision of each weight layer are set separately; Based on the target inference path, an operation and maintenance decision suggestion is generated according to the intent of the operation and maintenance request data, including: the weight layer includes an output layer, when the confidence of the operation and maintenance decision suggestion output by the output layer is lower than the preset confidence, the activation value precision of the output layer is switched from the preset precision to the first precision, and the first precision is higher than the preset precision.
[0010] For example, when the target inference path is the large model inference path, based on the target inference path, and according to the intent of the operation and maintenance request data, operation and maintenance decision suggestions are generated, including: Calculate the operation and maintenance semantic vector based on the operation and maintenance request data; Based on the aforementioned operation and maintenance semantic vector, similar historical semantic vectors are determined in the semantic vector database; Based on the similar historical semantic vectors, target candidate caches are determined in the target cache layer, wherein the target cache layer is at least one of the system prompt cache layer, the session context cache layer, and the task-specific cache layer; The operation and maintenance decision recommendations are generated based on the target candidate cache.
[0011] For example, based on a pre-set speculative decoding window model, the prediction window length for speculative decoding is adjusted according to the intent of the maintenance request data; Based on a pre-set draft model of the operation and maintenance domain, multiple candidate tokens are generated according to the prediction window length of the speculative decoding and the target candidate cache; Based on a pre-set large model, the operation and maintenance decision suggestions are generated according to multiple candidate tokens.
[0012] For example, based on a pre-set draft model of the operation and maintenance domain, and according to the prediction window length of the speculative decoding and the target candidate cache, multiple candidate tokens are generated, including: Based on a pre-set operation and maintenance knowledge graph, a root cause decision tree structure is constructed according to the target candidate cache; Based on the aforementioned draft model of the operation and maintenance domain, multiple candidate tokens are generated in parallel for each root cause branch in the root cause decision tree structure.
[0013] Secondly, this application provides a hybrid inference acceleration device for operational intelligent agents, comprising: The acquisition module is used to acquire operation and maintenance request data; The identification and scoring module is used to identify the intent of the operation and maintenance request data based on a pre-set intent identification model and complexity evaluation model, and to determine the complexity score corresponding to the operation and maintenance request data. The routing strategy module is used to determine the target inference path based on a pre-set routing strategy model and according to the intent and complexity score of the operation and maintenance request data. The target inference path is a rule engine path, a small model inference path, or a large model inference path. The reasoning module is used to generate operation and maintenance decision suggestions based on the target reasoning path and the intent of the operation and maintenance request data.
[0014] Thirdly, this application provides a computer device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the hybrid inference acceleration method for operation and maintenance agents as described in the first aspect.
[0015] Fourthly, this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the hybrid inference acceleration method for operational intelligent agents as described in the first aspect.
[0016] The above-described technical solutions adopted in the embodiments of this application can achieve the following beneficial effects: This application distinguishes the difficulty of operation and maintenance tasks by performing intent recognition and complexity scoring when processing operation and maintenance requests. It matches inference paths according to the difficulty of operation and maintenance tasks, which reduces the processing request volume of large models. While ensuring the quality of decision-making, it significantly improves the utilization of computing resources, reduces operating costs, increases throughput, and reduces latency, thus improving the application of operation and maintenance intelligence in large-scale production environments. Attached Figure Description
[0017] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings: Figure 1 This is a schematic diagram of an application environment for a hybrid inference acceleration method for operation and maintenance intelligent agents according to an embodiment of the present invention; Figure 2 This is a flowchart illustrating a hybrid inference acceleration method for operation and maintenance intelligent agents according to an embodiment of the present invention; Figure 3 yes Figure 2 A flowchart illustrating a specific implementation of step S4; Figure 4 yes Figure 3 A flowchart illustrating a specific implementation of step S44; Figure 5 This is a schematic diagram of a hybrid inference acceleration device for operation and maintenance intelligent agents according to an embodiment of the present invention; Figure 6 This is a schematic diagram of the structure of a computer device according to an embodiment of the present invention; Figure 7 This is another structural schematic diagram of a computer device according to one embodiment of the present invention. Detailed Implementation
[0018] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0019] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such use can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the term "comprising" and its variations should be interpreted as open-ended terms meaning "including but not limited to."
[0020] The technical solutions provided by the various embodiments of this application are described in detail below with reference to the accompanying drawings.
[0021] As mentioned above, current operational intelligence agents suffer from extremely high computational resource consumption and operational costs during inference processes. To address this issue, this application provides a hybrid inference acceleration method for operational intelligence agents.
[0022] The hybrid inference acceleration method for operation and maintenance agents provided in this embodiment of the invention can be applied to, for example... Figure 1 In this application environment, the device communicates with the server via a network. The server can obtain maintenance request data from the device; based on a pre-set intent recognition model and complexity evaluation model, it performs intent recognition on the maintenance request data and determines the complexity score corresponding to the maintenance request data; based on a pre-set routing strategy model, it determines a target inference path according to the intent and complexity score of the maintenance request data, wherein the target inference path is a rule engine path, a small model inference path, or a large model inference path; based on the target inference path, it generates maintenance decision suggestions according to the intent of the maintenance request data. This application, when processing maintenance requests, differentiates the difficulty of maintenance tasks by performing intent recognition and complexity scoring, and matches inference paths according to the difficulty of maintenance tasks. This reduces the processing request volume of large models, significantly improves computing resource utilization, reduces operating costs, increases throughput, and reduces latency while ensuring decision quality, thus improving the practical application of maintenance intelligence in large-scale production environments.
[0023] The device side can be, but is not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices. The server side can be implemented using a standalone server or a server cluster consisting of multiple servers. The invention will now be described in detail through specific embodiments.
[0024] Please see Figure 2 As shown, Figure 2 A flowchart illustrating a hybrid inference acceleration method for operation and maintenance agents provided in an embodiment of the present invention includes the following steps: S1: Obtain operation and maintenance request data.
[0025] In one embodiment, the method includes: constructing a corpus specifically for the operations and maintenance (O&M) domain. The data used to construct the O&M corpus consists of historical O&M request data, including a historical alarm work order database, O&M operation logs, system configuration documents, and a fault case library.
[0026] In one embodiment, the method further includes: cleaning historical operation and maintenance data, removing noise information, and unifying the format; and constructing annotation specifications based on the structural characteristics of operation and maintenance tasks, and performing fine-grained annotation on key information such as task type, operation object, execution conditions, and expected results corresponding to historical operation and maintenance data.
[0027] In one embodiment, the method further includes: constructing a dedicated operation and maintenance dictionary. The dedicated operation and maintenance dictionary includes a device terminology dictionary, an operation verb dictionary, and an indicator name dictionary, etc., to ensure accurate word segmentation. The method utilizes the word segmentation capabilities of a pre-trained word segmentation model to process text words, while simultaneously introducing an operation and maintenance-specific dictionary for vocabulary normalization. For example, "restart service," "service restart," and "restart service" are standardized into the standard expression "restart service," and entity recognition and structured processing are performed on information such as device number, IP address, and timestamp.
[0028] In one embodiment, after acquiring the operation and maintenance request data, the method further includes preprocessing the operation and maintenance request data. Preprocessing the operation and maintenance request data includes: segmenting the operation and maintenance request data into words based on a pre-set word segmentation model; annotating the segmented operation and maintenance request data based on pre-set annotation specifications; and normalizing the annotated operation and maintenance request data based on a pre-set operation and maintenance-specific dictionary.
[0029] In one embodiment, the word segmentation model can be a pre-trained language model based on Transformer, a sequence labeling model based on deep learning such as THULAC and HanLP, or a hybrid rule-based and statistical word segmentation model, such as the Jieba word segmentation model, an enhanced version for the operations and maintenance domain.
[0030] S2: Based on the pre-set intent recognition model and complexity evaluation model, perform intent recognition on the operation and maintenance request data, and determine the complexity score corresponding to the operation and maintenance request data.
[0031] In one embodiment, the intent recognition model and complexity evaluation model are fine-tuned using a domain-specific corpus for operations and maintenance (O&M) based on a general pre-trained model. This allows the general pre-trained model to better understand O&M terminology and expressions. The intent recognition and complexity evaluation tasks are trained separately using task heads, ultimately outputting a structured intent representation and a complexity score. The structured O&M intent representation includes dimensions such as task type, operation objective, and time constraints. The complexity score is a continuous score within the 0-1 range. Furthermore, during training, the complexity dimensions of the intent recognition model and complexity evaluation model include operational complexity, knowledge complexity, and risk complexity. For example, in operational complexity, "restarting a single server" has low complexity, while "batch restarting cluster servers and verifying service availability" has high complexity; in knowledge complexity, "querying CPU load" has low complexity, while "investigating network packet loss issues in a distributed system" has high complexity; and in risk complexity, "viewing logs" has low complexity, while "modifying core configuration files" has high complexity. The intent recognition model and complexity evaluation model are trained on a pre-set labeled historical operation and maintenance request dataset, which includes manually labeled scores. Specifically, multiple operation and maintenance experts are invited to score the complexity of the operation and maintenance request data in the prepared historical operation and maintenance request dataset, with scores ranging from 0 to 1, where 0 represents the simplest and 1 represents the most complex.
[0032] S3: Based on the pre-set routing strategy model, determine the target inference path according to the intent and complexity score of the operation and maintenance request data, wherein the target inference path is a rule engine path, a small model inference path, or a large model inference path.
[0033] In one embodiment, the routing strategy model resides in the intent-aware routing layer. The routing strategy model is the core scheduling hub of the hybrid inference acceleration method for operations and maintenance agents. It invokes the intent and complexity score of the operations and maintenance request data, and comprehensively determines the optimal inference path as the target inference path. This application's three-level inference routing mechanism achieves precise matching of inference resources through differentiated routing via rule engine paths, small model inference paths, or large model inference paths, reducing average inference latency by more than 60% while ensuring no loss of quality in complex operations and maintenance decisions.
[0034] In one embodiment, the target inference path is a rule engine path, a small model inference path, or a large model inference path.
[0035] In one embodiment, the intent of the operation and maintenance request data is divided into several types according to the task type, as shown in Table 1.
[0036] The inference path selection strategy of this application is shown in Table 1 for different types of operation and maintenance tasks with varying complexity.
[0037] Table 1 Reasoning Path Selection Strategy
[0038] Specifically, when the intent of the maintenance request data is a first type of intent and the complexity score of the maintenance request data is within a first preset range, the target inference path is determined to be a rule engine path. Further, the first type of intent includes status query and indicator monitoring types. The first preset range is a complexity score less than 0.2. Further, the first type of intent can also be the basis for the complexity score. There is consistency between the first type of intent and the first preset range.
[0039] Specifically, when the intent of the maintenance request data is a second type of intent and the complexity score of the maintenance request data is within a second preset range, the target inference path is determined to be a small model inference path. Further, the second type of intent includes indicator query, alarm analysis, and configuration suggestion types. The second preset range is a complexity score greater than or equal to 0.2 and less than 0.6. Furthermore, the second type of intent can also be the basis for the complexity score. The second type of intent and the second preset range are consistent.
[0040] Specifically, when the intent of the maintenance request data is a third type of intent and the complexity score of the maintenance request data is within a third preset range, the target inference path is determined to be a large model inference path. Further, the third type of intent includes root cause analysis and change planning. The third preset range is a complexity score greater than or equal to 0.6 and less than or equal to 1. Furthermore, the third type of intent can also be the basis for the complexity score. There is consistency between the third type of intent and the third preset range.
[0041] Specifically, (1) Rule engine path: For high-frequency, standardized simple operation and maintenance queries (such as service status query, indicator threshold check, etc.), operation and maintenance requests with a complexity score of less than 0.2 are directly routed to the lightweight rule engine for processing, and the response latency is controlled within 50ms, completely avoiding the overhead of LLM inference. (2) Small model inference path: For medium-complexity operation and maintenance tasks (such as alarm classification, configuration suggestion generation, etc.), operation and maintenance requests with a complexity score in the range of 0.2-0.6 are routed to a lightweight dedicated small model (less than 7B parameters). Combined with the quantitative inference engine, the inference latency is controlled within 500ms, meeting the real-time requirements of most operation and maintenance scenarios. (3) Large model inference path: For high-complexity operation and maintenance tasks (such as complex fault root cause analysis, multi-step operation planning, etc.), requests with a complexity score higher than 0.6 are routed to the large model. At the same time, speculative decoding and KV caching acceleration mechanisms are enabled to maximize the inference speed while ensuring the quality of inference.
[0042] In one embodiment, the method further includes: acquiring the running load status; when the running load status is high load, prioritizing the maintenance request data of the same target inference path, and performing inference based on the priority ranking result. This achieves dynamic load balancing adjustment of the maintenance agent. Specifically, the quantification standard for high load status is determined by a comprehensive multi-dimensional index, including the system's real-time monitoring of GPU memory utilization (threshold set above 85%), request queue length (exceeding 1.5 times the current processing capacity), average inference latency (exceeding 1.3 times the normal baseline), and the number of available tokens in the token bucket algorithm (below 20% of the preset capacity). When any key dimension triggers a preset threshold or the weighted comprehensive load score of multiple dimensions exceeds 0.7 (normalized to the 0-1 range), it is determined to be a high load state. Specifically, when prioritizing maintenance request data in the small model inference path, the small model routing ratio is automatically increased, and the ranking is based on complexity scores. For example, among requests in the original 0.2-0.6 complexity range, edge maintenance requests with scores of 0.5-0.6 are preferentially switched to the small model path. Priority queue management is implemented for maintenance requests in the large model inference path to ensure timely response to critical maintenance decisions. For example, priorities are assigned based on task urgency and business impact, with urgent root cause analysis requests being processed first, and non-critical analysis tasks being downgraded or delayed. These quantified conditions can be dynamically adjusted through configuration files to adapt to resource differences and maintenance SLA requirements in different deployment environments. The routing ratio in the small model inference path is automatically increased, and priority queue management is implemented for maintenance requests in the large model inference path to ensure timely response to critical maintenance decisions. Through continuous learning of historical target inference path decisions and actual effects, routing strategy parameters are automatically iteratively optimized.
[0043] S4: Based on the target reasoning path, generate operation and maintenance decision suggestions according to the intent of the operation and maintenance request data.
[0044] In one embodiment, the target inference path is a rule engine path, a small model inference path, or a large model inference path. The target inference path includes multiple weight layers, each with its own set weight precision and activation value precision. Further, based on the target inference path, generating operation and maintenance decision suggestions according to the intent of the operation and maintenance request data includes: the weight layer includes an output layer; when the confidence level of the operation and maintenance decision suggestion output by the output layer is lower than a preset confidence level, the activation value precision of the output layer switches from a preset precision to a first precision, where the first precision is higher than the preset precision. This setting is primarily applied when the target inference path is a large model inference path.
[0045] Specifically, a differentiated quantization precision strategy is adopted for the weights and activation values of different layers in the target inference path. For example, the heads in the attention layer that are strongly correlated with key entities in the operation and maintenance domain (device names, indicator values, operation commands, etc.) maintain FP16 precision, while the weight layers (output layers) with less impact on overall operation and maintenance decisions are quantized to INT8 or even INT4, maximizing inference speed improvement while minimizing precision loss. The weight precision of each weight layer is pre-set and differentiated; the specific settings for high-weight layers and low-weight layers are manually adjusted according to needs. The weight precision of each weight layer does not change during the inference process.
[0046] Specifically, the activation value precision adaptively switches during runtime. During inference, the confidence level of the operational decision recommendations output by the output layer is monitored in real time. When a low-confidence output is detected (below the preset confidence level, potentially corresponding to complex operational scenarios), the activation value precision of the output layer automatically reverts to a higher precision (i.e., first precision) for inference, ensuring the reliability of critical operational decisions. For routine query tasks that do not involve high-risk operations, low-precision quantization is maintained to maximize inference throughput. Furthermore, during inference, the confidence score (e.g., the maximum softmax probability) and the distribution of operational keywords (e.g., whether key entities such as device names, operation commands, and fault codes exhibit abnormal sparsity or ambiguity) of the target inference path output are monitored in real time to dynamically adjust the activation value precision. For example, in root cause analysis tasks, if the output of the large model inference path is "restart nginx service," and the confidence level of this output decision is below 0.6, and the output is missing a key IP address or error code, the system determines it to be low-confidence and automatically reverts from INT4 to FP16 precision for re-inference, ensuring the reliability of high-risk operations. For routine query tasks such as "querying CPU utilization" that do not involve any changes, INT4 low-precision inference is maintained even if the confidence level fluctuates, in order to maximize concurrent throughput, and a degraded retry is only triggered when the output parsing fails.
[0047] In one embodiment, considering the frequent toolchain calls by the operations and maintenance agent, the method further includes fusing operators of a preset weight layer. Specifically, operator fusion optimization is performed on the attention layer computation operators and FFN layer computation operators related to tool calls to reduce memory read and write operations. Simultaneously, PagedAttention technology is used to optimize the KV cache memory allocation for multiple concurrent requests, avoiding memory fragmentation-induced GPU memory waste and increasing GPU memory utilization from the typical 60% to over 90%.
[0048] The task-aware dynamic quantization strategy of this application is based on the dynamic precision switching strategy of the operation and maintenance task risk level and real-time confidence feedback. It maximizes the quantization benefits while minimizing the loss of precision. Combined with operator fusion and PagedAttention optimization, it improves the GPU memory utilization to more than 90%, increases the system throughput by more than 3 times, and effectively supports the processing of large-scale concurrent operation and maintenance requests.
[0049] In one embodiment, the large model inference path includes a key-value (KV) cache management model and a domain-enhanced speculative decoding model. In operational scenarios, the computational bottleneck of LLM large model inference mainly lies in the KV cache management for attention computation. This application significantly improves the cache hit rate through an operational domain-adaptive KV cache management model. Speculative decoding is an important technique for improving the decoding speed of large models, but the acceptance rate of general draft models is low (typically below 60%) under the lexical distribution of the operational domain. This application improves the token acceptance rate of draft models in operational scenarios through a domain-enhanced speculative decoding model.
[0050] In one embodiment, such as Figure 3 As shown, when the target inference path is the large model inference path, based on the target inference path, operation and maintenance decision suggestions are generated according to the intent of the operation and maintenance request data, including: S41: Calculate the operation and maintenance semantic vector based on the operation and maintenance request data; S42: Based on the operation and maintenance semantic vector, determine similar historical semantic vectors in the semantic vector database; S43: Based on the similar historical semantic vector, determine the target candidate cache in the target cache layer, wherein the target cache layer is at least one of the system prompt cache layer, the session context cache layer, and the task-specific cache layer; S44: Generate the operation and maintenance decision suggestions based on the target candidate cache.
[0051] Specifically, the KV cache management model includes a semantic vector database, which is constructed by semantically encoding the KV cache based on historical operation and maintenance request data. Unlike caching mechanisms based on precise string matching, the KV cache management model employs semantic vector indexing technology.
[0052] Specifically, semantic vectors are calculated based on the intent of the maintenance request data using a pre-trained semantic vector computation model. The pre-trained semantic vector computation model can be Sentence-BERT, OpenAI Embeddings API, etc., without further limitation. Upon receiving maintenance request data, the semantic vector of the maintenance request data is calculated, and its similarity to historical semantic vectors in the semantic vector database is calculated. When the similarity between the historical semantic vector and the semantic vector of the maintenance request data exceeds a threshold (default 0.85), the historical semantic vector is determined as a similar historical semantic vector. Based on the similar historical semantic vector, a target candidate cache corresponding to the similar historical semantic vector is determined in the target cache layer, and this target candidate cache is directly reused. Reusing historical key-value caches skips the prefix calculation overhead. For a large number of semantically similar queries in maintenance scenarios (such as "query XX service status" type requests), the cache hit rate can reach over 75%.
[0053] Specifically, the KV cache management model in this application implements an operation and maintenance context-aware cache layering system. The KV cache management model divides the KV cache into three layers: a system prompt cache layer, a session context cache layer, and a task-specific cache layer. The system prompt cache layer, session context cache layer, and task-specific cache layer each employ different caching strategies and eviction mechanisms. This application places different types of operation and maintenance request data into different storage layers, thereby maximizing resource utilization while ensuring performance.
[0054] Specifically, the system prompt cache layer caches static content such as operation and maintenance knowledge and tool definitions, including instructions for using operation and maintenance tools, definitions of professional terms in the operation and maintenance field, troubleshooting steps for common faults, and basic system configuration information. The system prompt cache layer employs a persistent caching strategy to completely eliminate redundant calculation overhead. The session context cache layer caches dynamic content such as dialogue history and user intent, including the user's historical dialogue records, the user's current intent state, and intermediate calculation results. The session context cache layer employs an LRU strategy, dynamically adjusting the cache window based on memory capacity. The task-specific cache layer caches task-level content such as the context and intermediate results of specific operation and maintenance tasks, including the execution steps of a specific operation and maintenance task, its intermediate results, and its dependencies. The caching strategy of the task-specific cache layer provides fine-grained management for specific operation and maintenance task instances.
[0055] In one embodiment, the method further includes: predicting high-frequency operation and maintenance scenarios based on a pre-built operation and maintenance knowledge graph and operation and maintenance request data; calculating predicted semantic vectors based on the high-frequency operation and maintenance scenarios; predicting similar historical semantic vectors in a semantic vector database based on the predicted semantic vectors; and determining predicted candidate caches in the target cache layer based on the predicted similar historical semantic vectors. This application implements knowledge graph-guided cache preheating, predicting high-frequency operation and maintenance scenarios based on the operation and maintenance knowledge graph, and proactively pre-calculating and caching the KV intermediate states of high-frequency operation and maintenance tasks during periods of low system load, thereby achieving predictive cache preheating and further reducing inference latency under peak load.
[0056] Specifically, firstly, the system constructs an operations and maintenance (O&M) knowledge graph using historical O&M logs. Nodes represent O&M entities (such as services, hosts, and alarms), and edges represent co-occurrence or causal frequencies. Graph algorithms calculate the weighted heat of nodes and identify frequent subgraph patterns as high-frequency scenarios. During periods of low system load (e.g., GPU utilization <30%), the scheduler proactively performs pre-filling calculations for the aforementioned prediction tasks, caching the generated key-value pairs (KV) with semantic vectors as indices in a high-speed memory pool (based on GPU memory or CPU RAM+DRAM tiers, commonly managed by Redis or FAISS). When real O&M request data arrives, the semantic similarity of the request data is calculated to match the pre-cached entries. If a match is found, the historical KV cache is directly loaded, skipping redundant calculations and thus reducing first-word latency during peak loads.
[0057] This application's adaptive semantic key-value cache management introduces semantic vector similarity retrieval into key-value cache management in operation and maintenance scenarios. Combined with a hierarchical caching architecture and a warm-up strategy guided by the operation and maintenance knowledge graph, it increases the key-value cache hit rate to over 75%, significantly reduces redundant computation overhead, and is particularly suitable for processing a large number of semantically similar requests in operation and maintenance scenarios.
[0058] In one embodiment, such as Figure 4 The method described herein also includes: S441: Based on a pre-set speculative decoding window model, adjust the prediction window length for speculative decoding according to the intent of the maintenance request data; S442: Based on the pre-set operation and maintenance domain draft model, generate multiple candidate tokens according to the prediction window length of the speculative decoding and the target candidate cache; S443: Based on a pre-set large model, generate the operation and maintenance decision suggestions according to multiple candidate tokens.
[0059] Specifically, the operations and maintenance (O&M) domain draft model is based on a general-purpose small model. It undergoes supervised fine-tuning using an O&M-specific corpus (historical operation logs, alarm texts, operation manuals, etc.). Knowledge distillation technology is used to transfer the output distribution of the large model in O&M tasks to the O&M domain draft model, enabling it to better fit the output probability distribution of O&M scenarios and increasing the token acceptance rate to over 80%. In other words, the teacher model (large model) is trained to learn the knowledge and rules of O&M reasoning. Then, knowledge distillation is used to transfer the O&M reasoning capabilities to the O&M domain draft model, reducing the computational load and time of reasoning while maintaining reasoning quality, thus meeting the real-time requirements of practical applications.
[0060] Specifically, the speculative decoding window model is used for adaptive speculative window control. Based on the task type to which the intent of the maintenance request data belongs, the historical token acceptance rate corresponding to this task type is determined. The prediction window size of speculative decoding is dynamically adjusted based on the task type and the historical token acceptance rate. Further, the maintenance mode is determined by the task type to which the intent of the maintenance request data belongs and the historical token acceptance rate. When the maintenance mode is stable, the prediction window length of speculative decoding is adjusted to a first length; when the maintenance mode is high uncertainty, the prediction window length of speculative decoding is adjusted to a second length. When the prediction window length of speculative decoding is the first length, 8-12 tokens are received; when the prediction window length of speculative decoding is the second length, 2-4 tokens are received.
[0061] Specifically, the stable mode provides standardized operational output with a stable acceptance rate. It calculates the variance and mean of historical token acceptance rates from 10 historical draft models of similar task types in real-time. If the variance of the historical token acceptance rate exceeds a preset variance threshold and the mean of the historical token acceptance rate is higher than a first preset mean threshold, it is considered a stable mode. In this case, the prediction window size for speculative decoding is increased to 8-12 tokens. Specifically, the preset variance threshold can be set to 0.05, and the first preset mean threshold can be set to 0.75. The stable mode corresponds to strongly formatted outputs such as operation commands and alarm level descriptions.
[0062] Specifically, the high uncertainty mode is characterized by highly creative and personalized analysis outputs. This is determined by statistically analyzing the average historical token acceptance rate of three similar task types in the operations and maintenance domain draft model. If the average historical token acceptance rate is lower than a second preset average threshold, it is classified as a high uncertainty mode. Specifically, the second preset average threshold can be set to 0.5. High uncertainty modes can also be determined based on the detected intent in the operations and maintenance request data showing a higher proportion of unstructured natural language than a first preset proportion and a lower entity repetition rate than a second preset proportion. For example, the first preset proportion can be set to 70%, and the second preset proportion to 50%. When a high uncertainty mode is identified, the prediction window size for speculative decoding is reduced to 2-4 tokens. All thresholds are pre-calibrated based on the operations and maintenance labeled dataset and support online dynamic adjustment.
[0063] In one embodiment, such as Figure 5 As shown, step S442 generates multiple candidate tokens based on a pre-set operation and maintenance domain draft model, according to the prediction window length of the speculative decoding and the target candidate cache. It also includes: constructing a root cause decision tree structure based on a pre-set operation and maintenance knowledge graph and the target candidate cache; and generating multiple candidate tokens for each root cause branch in the root cause decision tree structure in parallel based on the operation and maintenance domain draft model.
[0064] Specifically, constructing the root cause decision tree structure includes: finding matching problem nodes in the operations and maintenance knowledge graph based on the semantic vector of the target candidate cache; retrieving all root causes and their probability distributions of the problem node from the operations and maintenance knowledge graph; arranging all root causes according to their probability distribution from high to low to form branches of the decision tree; and labeling each branch with historical cases or rules from the target candidate cache. The resulting root cause decision tree reflects both the professional logic of the operations and maintenance domain and the actual situation of the current scenario, providing a clear analytical framework for the subsequent generation of candidate tokens. This application implements a tree-structured speculative decoding extension. This application targets multi-branch decision-making scenarios in operations and maintenance, implementing parallel speculative decoding of a tree structure. The draft model in the operations and maintenance domain simultaneously generates multiple candidate paths, and the large model verifies all branches in parallel, selecting the optimal result, significantly improving the inference efficiency in multi-candidate scenarios.
[0065] The domain-enhanced speculative decoding framework of this application improves the token acceptance rate from below 60% in general scenarios to above 80% by fine-tuning the operation and maintenance domain and training the operation and maintenance domain draft model through knowledge distillation, combined with adaptive speculative window control and tree-shaped speculative decoding extension, achieving an additional 2-3 times decoding acceleration effect on the large model inference path.
[0066] In one embodiment, the method further includes: performing post-processing verification after generating operation and maintenance decision suggestions, wherein post-processing verification includes format standardization verification, compliance verification, and logical consistency verification. All inference path output results are processed by a unified post-processing verification module: (1) Format standardization: formatting the operation and maintenance decision suggestion text results generated by each inference path into a standard operation and maintenance operation instruction format to ensure interface compatibility with downstream toolchains; (2) Compliance verification: verifying in real time whether the operation and maintenance operations in the generated operation and maintenance decision suggestions violate security specifications and operation constraints, triggering alarms and generating correction suggestions when violations are detected; (3) Logical consistency verification: using the operation and maintenance knowledge graph to perform consistency checks on the entity relationships and operation logic in the operation and maintenance decision suggestions, filtering out potential erroneous operation suggestions.
[0067] In one embodiment, a reasoning process visualization module is built to improve the interpretability of the system. For each reasoning request, the system can display the routing decision basis, cache hit status, speculative decoding acceptance rate, and the confidence distribution of the final reasoning result, enhancing the understanding and trust of operations and maintenance personnel in the system's reasoning process.
[0068] This application addresses the core issues of low inference efficiency, high latency, and high resource consumption in operation and maintenance intelligent agent scenarios. It proposes a hybrid inference acceleration method for operation and maintenance intelligent agents. This method achieves the organic integration of large and small model collaboration, efficient cache reuse, parallel decoding acceleration, and accuracy adaptive quantization through four core models: intent recognition model and complexity evaluation model, routing strategy model, KV cache management model, domain-enhanced speculative decoding model, and dynamic quantization inference engine. While ensuring the accuracy of operation and maintenance decisions, it achieves the goal of reducing end-to-end inference latency by more than 60% and increasing throughput by more than 3 times.
[0069] As can be seen, in the above scheme, when processing operation and maintenance requests, this application distinguishes the difficulty of operation and maintenance tasks by performing intent recognition and complexity scoring, and matches inference paths according to the difficulty of operation and maintenance tasks, thereby reducing the processing request volume of large models. Under the premise of ensuring decision quality, it significantly improves the utilization of computing resources, reduces operating costs, increases throughput, and reduces latency, thus improving the application of operation and maintenance intelligence in large-scale production environments.
[0070] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.
[0071] In one embodiment, a hybrid inference acceleration device for operation and maintenance agents is provided, which corresponds one-to-one with the hybrid inference acceleration method for operation and maintenance agents described in the above embodiments. For example... Figure 5As shown, this hybrid inference acceleration device for operation and maintenance intelligent agents includes an acquisition module 101, an identification and scoring module 102, a routing strategy module 103, and an inference module 104. Detailed descriptions of each functional module are as follows: Module 101 is used to acquire operation and maintenance request data; The identification and scoring module 102 is used to identify the intent of the operation and maintenance request data based on a pre-set intent identification model and complexity evaluation model, and determine the complexity score corresponding to the operation and maintenance request data. The routing strategy module 103 is used to determine the target inference path based on a pre-set routing strategy model and according to the intent and complexity score of the operation and maintenance request data, wherein the target inference path is a rule engine path, a small model inference path, or a large model inference path. The reasoning module 104 is used to generate operation and maintenance decision suggestions based on the target reasoning path and the intent of the operation and maintenance request data.
[0072] Specifically, the acquisition module 101 is also used to segment the maintenance request data into words based on a pre-set word segmentation model; Based on pre-set annotation specifications, the segmented operation and maintenance request data is annotated. Based on a pre-set operation and maintenance dictionary, the labeled operation and maintenance request data is normalized.
[0073] Specifically, the routing strategy module 103 is also used to determine the target inference path as the rule engine path when the intent of the operation and maintenance request data is a first type of intent and the complexity score of the operation and maintenance request data is a first preset range. When the intent of the maintenance request data is the second type of intent and the complexity score of the maintenance request data is within the second preset range, the target inference path is determined to be the small model inference path. When the intent of the maintenance request data is a third type of intent and the complexity score of the maintenance request data is a third preset range, the target inference path is determined to be a large model inference path.
[0074] Specifically, the routing strategy module 103 is also used to obtain the running load status. When the running load status is high load, it prioritizes the operation and maintenance request data of the same target inference path and performs inference based on the priority ranking result.
[0075] Specifically, the inference module 104 also includes a KV cache management module and a domain-enhanced speculative decoding module.
[0076] Specifically, the KV cache management module is used to calculate the operation and maintenance semantic vector based on the operation and maintenance request data; determine similar historical semantic vectors in the semantic vector database based on the operation and maintenance semantic vector; determine target candidate caches in the target cache layer based on the similar historical semantic vectors, wherein the target cache layer is at least one of the system prompt cache layer, the session context cache layer, and the task-specific cache layer; and generate the operation and maintenance decision suggestion based on the target candidate caches.
[0077] Specifically, the KV cache management module is also used to predict high-frequency operation and maintenance scenarios based on a pre-built operation and maintenance knowledge graph and operation and maintenance request data; calculate predicted semantic vectors based on the high-frequency operation and maintenance scenarios; predict similar historical semantic vectors in the semantic vector database based on the predicted semantic vectors; and determine predicted candidate caches in the target cache layer based on the predicted similar historical semantic vectors.
[0078] Specifically, the domain-enhanced speculative decoding module is used to adjust the prediction window length of speculative decoding based on a pre-set speculative decoding window model and the intent of the operation and maintenance request data; based on a pre-set operation and maintenance domain draft model, it generates multiple candidate tokens according to the prediction window length of speculative decoding and the target candidate cache; and based on a pre-set large model, it generates the operation and maintenance decision suggestions based on the multiple candidate tokens.
[0079] Specifically, the domain-enhanced speculative decoding module is also used to construct a root cause decision tree structure based on a pre-set operation and maintenance knowledge graph and the target candidate cache; and to generate multiple candidate tokens for each root cause branch in the root cause decision tree structure in parallel based on the operation and maintenance domain draft model.
[0080] Specifically, the hybrid inference acceleration device for operation and maintenance agents also includes a dynamic quantization inference engine, which is used to set the weight precision and activation value precision of each weight layer respectively; it is also used to switch the activation value precision of the output layer from the preset precision to the first precision when the confidence of the operation and maintenance decision suggestion output by the output layer is lower than the preset confidence, and the first precision is higher than the preset precision.
[0081] Specifically, the hybrid inference acceleration device for operation and maintenance agents also includes a post-processing verification module 105, which is used to perform post-processing verification after generating operation and maintenance decision suggestions. The post-processing verification includes format standardization verification, compliance verification, and logical consistency verification.
[0082] This invention provides a hybrid inference acceleration device for operation and maintenance agents. When processing operation and maintenance requests, this application distinguishes the difficulty of operation and maintenance tasks by performing intent recognition and complexity scoring, and matches inference paths according to the difficulty of operation and maintenance tasks. This reduces the processing request volume of large models, significantly improves the utilization of computing resources, reduces operating costs, increases throughput, and reduces latency while ensuring decision quality, thus improving the application of operation and maintenance agents in large-scale production environments.
[0083] Specific limitations regarding the hybrid inference acceleration device for operational agents can be found in the limitations of the hybrid inference acceleration method for operational agents described above, and will not be repeated here. Each module in the aforementioned hybrid inference acceleration device for operational agents can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in the computer device in hardware form, or stored in the memory of the computer device in software form, so that the processor can call and execute the operations corresponding to each module.
[0084] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 6 As shown, the computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile and / or volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The network interface is used for communication with external devices via a network connection. When the computer program is executed by the processor, it implements the functions or steps of a hybrid inference acceleration method for operational intelligent agents on the server side.
[0085] In one embodiment, a computer device is provided, which may be a device terminal, and its internal structure diagram may be as follows: Figure 7 As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The network interface is used to communicate with an external server via a network connection. When the computer program is executed by the processor, it implements the functions or steps of a hybrid inference acceleration method for operational intelligent agents on the device side.
[0086] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to perform the following steps: Obtain operation and maintenance request data; Based on a pre-set intent recognition model and complexity evaluation model, the intent of the operation and maintenance request data is recognized, and the complexity score corresponding to the operation and maintenance request data is determined. Based on a pre-set routing strategy model, a target inference path is determined according to the intent and complexity score of the operation and maintenance request data, wherein the target inference path is a rule engine path, a small model inference path, or a large model inference path. Based on the target reasoning path, operation and maintenance decision suggestions are generated according to the intent of the operation and maintenance request data.
[0087] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor: Obtain operation and maintenance request data; Based on a pre-set intent recognition model and complexity evaluation model, the intent of the operation and maintenance request data is recognized, and the complexity score corresponding to the operation and maintenance request data is determined. Based on a pre-set routing strategy model, a target inference path is determined according to the intent and complexity score of the operation and maintenance request data, wherein the target inference path is a rule engine path, a small model inference path, or a large model inference path. Based on the target reasoning path, operation and maintenance decision suggestions are generated according to the intent of the operation and maintenance request data.
[0088] It should be noted that the functions or steps that can be implemented by the computer-readable storage medium or computer device described above can be referred to the relevant descriptions on the server side and device side in the foregoing method embodiments. To avoid repetition, they will not be described one by one here.
[0089] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.
[0090] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.
[0091] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.
Claims
1. A hybrid reasoning acceleration method for operation and maintenance intelligent agents, characterized in that, include: Obtain operation and maintenance request data; Based on a pre-set intent recognition model and complexity evaluation model, the intent of the operation and maintenance request data is recognized, and the complexity score corresponding to the operation and maintenance request data is determined. Based on a pre-set routing strategy model, a target inference path is determined according to the intent and complexity score of the operation and maintenance request data, wherein the target inference path is a rule engine path, a small model inference path, or a large model inference path. Based on the target reasoning path, operation and maintenance decision suggestions are generated according to the intent of the operation and maintenance request data.
2. The method according to claim 1, characterized in that, After obtaining the operation and maintenance request data, the method further includes: The maintenance request data is segmented into words based on a pre-set word segmentation model; Based on pre-set annotation specifications, the segmented operation and maintenance request data is annotated. Based on a pre-set operation and maintenance dictionary, the labeled operation and maintenance request data is normalized.
3. The method according to claim 1, characterized in that, Based on a pre-set routing strategy model, a target inference path is determined according to the intent and complexity score of the operation and maintenance request data. This target inference path can be a rule engine path, a small model inference path, or a large model inference path, including: When the intent of the maintenance request data is a first type of intent and the complexity score of the maintenance request data is a first preset range, the target reasoning path is determined to be a rule engine path. When the intent of the maintenance request data is the second type of intent and the complexity score of the maintenance request data is within the second preset range, the target inference path is determined to be the small model inference path. When the intent of the maintenance request data is a third type of intent and the complexity score of the maintenance request data is a third preset range, the target inference path is determined to be a large model inference path.
4. The method according to claim 1, characterized in that, The target inference path includes multiple weight layers, and the weight precision and activation value precision of each weight layer are set separately. Based on the target inference path, an operation and maintenance decision suggestion is generated according to the intent of the operation and maintenance request data, including: the weight layer includes an output layer, when the confidence of the operation and maintenance decision suggestion output by the output layer is lower than the preset confidence, the activation value precision of the output layer is switched from the preset precision to the first precision, and the first precision is higher than the preset precision.
5. The method according to claim 1, characterized in that, When the target inference path is the large model inference path, based on the target inference path, operation and maintenance decision suggestions are generated according to the intent of the operation and maintenance request data, including: Calculate the operation and maintenance semantic vector based on the operation and maintenance request data; Based on the aforementioned operation and maintenance semantic vector, similar historical semantic vectors are determined in the semantic vector database; Based on the similar historical semantic vectors, target candidate caches are determined in the target cache layer, wherein the target cache layer is at least one of the system prompt cache layer, the session context cache layer, and the task-specific cache layer; The operation and maintenance decision recommendations are generated based on the target candidate cache.
6. The method according to claim 5, characterized in that, The method further includes: Based on a pre-set speculative decoding window model, the prediction window length for speculative decoding is adjusted according to the intent of the maintenance request data. Based on a pre-set draft model of the operation and maintenance domain, multiple candidate tokens are generated according to the prediction window length of the speculative decoding and the target candidate cache; Based on a pre-set large model, the operation and maintenance decision suggestions are generated according to multiple candidate tokens.
7. The method according to claim 5, characterized in that, Based on a pre-set draft model of the operations and maintenance domain, and according to the prediction window length of the speculative decoding and the target candidate cache, multiple candidate tokens are generated, including: Based on a pre-set operation and maintenance knowledge graph, a root cause decision tree structure is constructed according to the target candidate cache; Based on the aforementioned draft model of the operation and maintenance domain, multiple candidate tokens are generated in parallel for each root cause branch in the root cause decision tree structure.
8. A hybrid inference acceleration device for operation and maintenance intelligent agents, characterized in that, include: The acquisition module is used to acquire operation and maintenance request data; The identification and scoring module is used to identify the intent of the operation and maintenance request data based on a pre-set intent identification model and complexity evaluation model, and to determine the complexity score corresponding to the operation and maintenance request data. The routing strategy module is used to determine the target inference path based on a pre-set routing strategy model and according to the intent and complexity score of the operation and maintenance request data. The target inference path is a rule engine path, a small model inference path, or a large model inference path. The reasoning module is used to generate operation and maintenance decision suggestions based on the target reasoning path and the intent of the operation and maintenance request data.
9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the hybrid inference acceleration method for operational agents as described in any one of claims 1 to 7.
10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the hybrid inference acceleration method for operational agents as described in any one of claims 1 to 7.