Method and device for token anomaly repair through computing power by intelligent computing cloud platform

Abstract

The application provides a method and device for realizing Token abnormality repair by computing power through an intelligent computing cloud platform, relates to the technical fields of intelligent computing centers, intelligent computing centers, computing power infrastructure and intelligent computing cloud, and the method for realizing Token abnormality repair by computing power through the intelligent computing cloud platform comprises the following steps: S1, acquiring a current output Token of a large model; S2, detecting an abnormal state based on multidimensional state information of the current output Token to determine an abnormal starting point Token from a Token sequence; S3, determining a target rollback point from candidate rollback points based on the abnormal starting point Token, and determining a KV checkpoint associated with the target rollback point; and S4, regenerating a first Token subsequence between the KV checkpoint and the current output Token to perform abnormality repair. The application reduces computing power consumption in the intelligent computing cloud platform while improving the success rate of abnormality repair.

Description

Technical Field

[0001] This invention relates to the fields of intelligent computing centers, smart computing centers, computing power infrastructure, and smart computing cloud technologies, specifically to a method and apparatus for an intelligent computing cloud platform to perform token anomaly repair through computing power. Background Technology

[0002] With the rapid development of artificial intelligence technology, "intelligent computing centers" and "smart computing centers" have emerged.

[0003] An "intelligent computing center" refers to a facility that provides the necessary computing power, data, and algorithms for artificial intelligence applications (such as the development, training, and inference of deep learning models) by utilizing large-scale heterogeneous computing resources, including general-purpose and intelligent computing power. Intelligent computing centers encompass facilities, hardware, and software, and can provide full-stack capabilities from underlying computing power to top-level application enablement.

[0004] "Intelligent computing center" includes, but is not limited to, "intelligent computing center".

[0005] "Intelligent computing center" or artificial intelligence computing center is a type of computing infrastructure that provides computing power services, data services, and algorithm services required for artificial intelligence applications, based on artificial intelligence theory and adopting artificial intelligence computing architecture.

[0006] "Computing power" is the core of "intelligent computing center" and "smart computing center". It is the ability of computer equipment or computing / data center to process parameters. It is the ability of computer hardware and software to work together to execute a certain computing requirement. It is the computing power to achieve the target result output by processing parameter data. It is a new type of productivity that integrates parameter computing power, network carrying capacity and data storage capacity. It mainly provides services to society through computing power infrastructure.

[0007] In intelligent computing cloud platforms, large language models typically generate text using an autoregressive approach, meaning the model predicts the next token based on existing context and the sequence of generated tokens. Therefore, subsequent tokens are highly dependent on previously generated tokens. If an error occurs at a particular token, it will propagate through the context, affecting subsequent tokens. A single localized token error can render the entire output unusable, and retrying after a complete failure consumes a significant amount of tokens, computing power, and request latency on the intelligent computing cloud platform.

[0008] It is evident that since the emergence of intelligent computing centers, the inability to partially retry the output token when a large model outputs an abnormal token has been a pressing problem in this field. Summary of the Invention

[0009] This invention provides a method and apparatus for repairing token anomalies through computing power in an intelligent computing cloud platform, in order to solve the problem that existing technologies cannot effectively improve the security of computing power operation during the process of security protection of computing power operation.

[0010] To solve the above problems, the present invention is implemented as follows: In a first aspect, the present invention provides a method for an intelligent computing cloud platform to perform token anomaly repair through computing power, comprising: Step S1: Obtain the token currently output by the large model deployed in the intelligent computing cloud platform, and update the token sequence based on the currently output token; Step S2: Using computing power, based on the multi-dimensional state information of the currently output Token, perform multi-dimensional abnormal state detection on the currently output Token, and when an abnormality is detected, perform abnormal backtracking on the currently output Token to determine the abnormal starting Token from the Token sequence. Step S3: Based on the abnormal starting point Token, determine the target rollback point from the candidate rollback points corresponding to the Token sequence, and determine the KV checkpoint associated with the target rollback point; Step S4: Regenerate the first Token subsequence from the Token corresponding to the associated KV checkpoint to the currently output Token, and perform anomaly repair on the Token sequence based on the first Token subsequence.

[0011] In some embodiments, step S4 includes: Step S4.1: Determine the exception type of the exception starting point Token, and based on the exception type, determine the target path for Token regeneration from the candidate paths for Token regeneration; wherein, the exception type includes structural exception, schema exception, semantic exception, language drift, repeated loop, code error and security exception; Step S4.2: Based on the target path and the associated KV checkpoints, regenerate the first Token subsequence.

[0012] In some embodiments, step S4.1 includes: Step S4.1.1: Determine the computing power cost of the intelligent computing cloud platform and the historical number of failed token regeneration attempts; Step S4.1.2: Determine multiple candidate paths corresponding to the anomaly type; Step S4.1.3: Construct a path selection function based on the anomaly type, the computing power cost, and the historical failure count; Step S4.1.4: Based on the path selection function, determine the target path from the plurality of candidate paths.

[0013] In some embodiments, step S4.2 includes: Step S4.2.1: Based on the target path, regenerate the second token subsequence from the token corresponding to the associated KV checkpoint to the currently output token; Step S4.2.2: Perform an abnormal state risk assessment on the second Token subsequence to obtain a risk assessment value; Step S4.2.3: In response to the fact that the risk assessment value of the second Token subsequence is less than the risk threshold, the second Token subsequence with the smallest risk assessment value is selected as the first Token subsequence.

[0014] In some embodiments, step S4.2.2 further includes: Step S4.2.2.1: In response to the risk assessment value of the second Token subsequence being greater than or equal to the risk threshold, perform risk verification on the second Token subsequence, and determine the first Token subsequence from the second Token subsequence that has passed the risk verification.

[0015] In some embodiments, step S4.2.2.1 includes: Step S4.2.2.1.1: Obtain the second Token subsequence that failed the risk verification, and use it as the third Token subsequence that failed to regenerate the token; Step S4.2.2.1.2: Calculate the number of failures, remaining token budget, remaining computing power budget, and risk level of the third token subsequence regeneration failure; Step S4.2.2.1.3: Based on the number of failures, remaining token budget, remaining computing power budget, and risk level, construct the computing power upgrade function; Step S4.2.2.1.4: In response to the computing power upgrade function instruction, perform computing power upgrade and execute the preset computing power upgrade operation.

[0016] In some embodiments, step S4 further includes: Step S4.1': Determine the position information of the Token corresponding to the associated KV checkpoint in the Token sequence; Step S4.2': Based on the location information, truncate the Token sequence, retain the fourth Token subsequence before the location information, and delete the fifth Token subsequence starting from the location information; Step S4.3': Combine the first Token subsequence and the fourth Token subsequence in sequence to obtain the repaired Token sequence.

[0017] In some embodiments, the multi-dimensional anomaly detection includes at least two of the following: anomaly risk detection, structure detection, schema and tool call detection, semantic constraint and entity consistency detection, language style drift detection, and duplication and degradation detection.

[0018] In some embodiments, step S2 includes: Step S2.1: Obtain the status information corresponding to at least one abnormal status detection item from the multi-dimensional status information; Step S2.2: Based on the state information, perform anomaly detection on the currently output Token for the abnormal state detection item.

[0019] In some embodiments, step S2.2 includes: Step S2.2.1: In response to detecting an anomaly in the currently output Token, determine the anomaly classification dimension of the currently output Token; wherein, the anomaly classification dimension includes deterministic anomalies and probabilistic anomalies; Step S2.2.2: Based on the anomaly classification dimension, determine the anomaly starting point Token from the Token sequence.

[0020] In some embodiments, step S2.2.2 includes: Step S2.2.2.1: In response to the anomaly classification dimension being the deterministic anomaly, determine the currently output Token as the anomaly origin Token; Step S2.2.2.2: In response to the anomaly classification dimension being the probabilistic anomaly, calculate the anomaly contribution score and comprehensive risk increment for each Token in the Token sequence; Step S2.2.2.3: Based on the abnormal contribution score and comprehensive risk increment of each Token, determine the Token that meets the abnormal starting point Token screening condition, and use it as the abnormal starting point Token.

[0021] In some embodiments, the candidate rollback points include: structural closure points, semantic boundaries, code statement boundaries, tool parameter boundaries, low-risk token boundaries, and key-value checkpoints.

[0022] In some embodiments, step S3 includes: Step S3.1: Based on the preset scoring rules, score each candidate rollback point to obtain the score value corresponding to each candidate rollback point; Step S3.2: Determine the candidate rollback point with the largest score value as the target rollback point.

[0023] In some embodiments, step S3.1 includes: Step S3.1.1: Determine the scoring characteristics of each candidate rollback point; Step S3.1.2: Based on the defined scoring function and the scoring features, score each candidate rollback point.

[0024] In some embodiments, the method further includes: Step S5: Determine the computing power saving information for regenerating the first Token subsequence based on the KV checkpoint.

[0025] In some embodiments, step S5 includes: Step S5.1: Determine the token length, risk verification cost, and local recovery cost for regenerating the first token subsequence based on the KV checkpoint; Step S5.2: Based on the Token length, risk verification cost, and local recovery cost, determine the computing power saving ratio and global saving information, which will be used as the computing power saving information.

[0026] Secondly, the present invention also provides an apparatus for an intelligent computing cloud platform to perform token anomaly repair through computing power, comprising: The acquisition module is used to acquire the token currently output by the large model deployed in the intelligent computing cloud platform, and update the token sequence based on the currently output token; The anomaly detection module is used to detect multi-dimensional anomalies in the currently output Token based on the multi-dimensional state information of the currently output Token using computing power, and when an anomaly is detected, to backtrack the currently output Token to determine the starting Token of the anomaly from the Token sequence. The rollback point determination module is used to determine the target rollback point from the candidate rollback points corresponding to the Token sequence based on the abnormal starting point Token, and to determine the KV checkpoint associated with the target rollback point; The Token regeneration module is used to regenerate a first Token subsequence from the Token corresponding to the associated KV checkpoint to the currently output Token, and to perform anomaly repair on the Token sequence based on the first Token subsequence.

[0027] Thirdly, the present invention also provides an electronic device, including a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein when the computer program is executed by the processor, it implements the steps in the method for Token anomaly repair by computing power of the intelligent computing cloud platform described in the first aspect above.

[0028] Fourthly, the present invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps in the method for Token anomaly repair by computing power in the intelligent computing cloud platform described in the first aspect above.

[0029] Fifthly, the present invention also provides a computer program product, including computer instructions, which, when executed by a processor, implement the steps in the method for Token anomaly repair by computing power in the intelligent computing cloud platform described in the first aspect above.

[0030] In this invention, step S1 involves obtaining the token currently output by the large model deployed in the intelligent computing cloud platform and updating the token sequence based on the currently output token; step S2 involves using computing power to detect multi-dimensional abnormal states of the currently output token based on its multi-dimensional state information, and when an anomaly is detected, performing anomaly backtracking on the currently output token to determine the starting token of the anomaly from the token sequence; step S3 involves determining the target rollback point from the candidate rollback points corresponding to the token sequence based on the starting token of the anomaly, and determining the KV checkpoint associated with the target rollback point; step S4 involves regenerating the first token subsequence from the token corresponding to the associated KV checkpoint to the currently output token, and performing anomaly repair on the token sequence based on the first token subsequence. This allows for simultaneous output and detection of tokens, enabling timely detection of anomalies as they occur, significantly shortening the error propagation chain. By determining the starting token of the anomaly, accurate anomaly localization can be achieved, providing clear boundaries for subsequent repairs. During the token regeneration process, selecting a target path for local token regeneration can reduce invalid tokens and redundant calculations. This saves computing power on the intelligent computing cloud platform while improving the success rate of token regeneration and computing power utilization, thereby further enhancing the success rate of anomaly repair.

[0031] In addition, by detecting and repairing tokens while they are being output, this invention can accelerate the model's inference process without interrupting the model's output, and enable users to obtain continuous output, thus improving the streaming output experience. Attached Figure Description

[0032] To more clearly illustrate the technical solution of the present invention, the accompanying drawings used in the description of the present invention will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0033] Figure 1 This is a flowchart of a method for repairing token anomalies using computing power on an intelligent computing cloud platform, provided by the present invention. Figure 2 This is a structural diagram of a system provided by the present invention that uses computing power to repair token anomalies in an intelligent computing cloud platform. Figure 3 This is a structural diagram of a device for repairing token anomalies through computing power provided by the present invention in an intelligent computing cloud platform; Figure 4 This is a schematic diagram of the structure of an electronic device provided in this invention. Detailed Implementation

[0034] The technical solutions of this invention will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0035] The “computing power” mentioned in this invention refers to: the ability of computer equipment or computing / data center to process information; the ability of computer hardware and software to work together to perform a certain computing requirement; the computing power to achieve the target result output by processing information data; and a new type of productivity that integrates information computing power, network carrying capacity, and data storage capacity, mainly providing services to society through computing power infrastructure.

[0036] The "computational power" (CP) described in this invention refers to the ability of a data center server to process data and output results. It is a comprehensive indicator of a data center's computing power, encompassing general computing power, supercomputing power, and intelligent computing power. The commonly used unit of measurement is floating-point operations per second (FLOPS, 1 EFLOPS = 10^18 FLOPS), with higher values ​​indicating stronger overall computing power. It is estimated that 1 EFLOPS is approximately the computing power output of 5 Tianhe-2A supercomputers, 500,000 mainstream server CPUs, or 2 million mainstream laptops. The calculation formula is: CP = CP 通用 +CP 智能 +CP超级 .

[0037] The "Network Power" (NP) mentioned in this invention refers to the performance of data transmission capability of computing facilities, which includes comprehensive capabilities such as network architecture, network bandwidth, transmission latency, intelligent management and scheduling, and involves network transmission within and between data centers. It is a comprehensive indicator for measuring network transmission scheduling capability.

[0038] The "Storage Power" (SP) described in this invention refers to the comprehensive capabilities of a data center in four aspects: data storage capacity, performance, security and reliability, and green and low-carbon operation. It is a comprehensive indicator for measuring the data storage capacity of a data center, including external storage devices such as storage arrays and internal storage devices within servers. The commonly used unit of measurement for storage capacity is exabytes (EB, 1EB = 2^60 bytes), while the commonly used unit of measurement for performance is the number of read / write operations per second (IOPS / TB). Disaster recovery ratio is an important indicator of security and reliability.

[0039] The "computing infrastructure" mentioned in this invention refers to a new type of information infrastructure that integrates information computing power, network carrying capacity, and data storage capacity, enabling centralized computing, storage, transmission, and application of information.

[0040] The "new information infrastructure" mentioned in this invention refers to network infrastructure such as 5G networks, fiber optic broadband networks, backbone networks, international communication networks, and satellite internet; computing infrastructure such as data centers, general computing centers, intelligent computing centers, and supercomputing centers; and new technology facilities such as artificial intelligence, blockchain, and quantum computing.

[0041] The “computing power” mentioned in this invention includes: general computing power, intelligent computing power, and supercomputing power.

[0042] The "general computing power" mentioned in this invention refers to the computing power provided by servers based on CPU (Central Processing Unit) chips, which is used to support basic general computing such as cloud computing and edge computing.

[0043] The "intelligent computing power" mentioned in this invention refers to: a computing platform deployed on a large scale based on dedicated chips such as GPU (Graphics Processing Unit), FPGA (Field Programmable Gate Array), and ASIC (Application Specific Integrated Circuit) for various artificial intelligence innovative applications, such as natural language processing and machine vision.

[0044] The “supercomputing power” mentioned in this invention refers to the computing power provided by high-performance computing clusters such as supercomputers. It utilizes the centralized computing resources of multiple computer systems working in parallel and uses a dedicated operating system to handle extremely complex or data-intensive problems. It is mainly used for computing in cutting-edge scientific fields, such as planetary simulation, drug molecule design, and gene analysis.

[0045] The "intelligent computing center" described in this invention refers to a facility that, through the use of large-scale heterogeneous computing resources, including general-purpose computing power (CPU) and intelligent computing power (GPU, FPGA, ASIC, etc.), primarily provides the necessary computing power, data, and algorithms for artificial intelligence applications (such as the development, training, and inference of deep learning models). The intelligent computing center encompasses facilities, hardware, and software, and can provide full-stack capabilities from underlying computing power to top-level application enablement.

[0046] The "intelligent computing cloud platform" mentioned in this invention, abbreviated as "intelligent computing cloud", refers to a cloud computing platform that integrates hardware and software resources based on an intelligent computing center.

[0047] The "intelligent computing center" mentioned in this invention includes, but is not limited to, "smart computing center".

[0048] The "intelligent computing center" mentioned in this invention, also known as an artificial intelligence computing center, is a type of computing infrastructure that provides computing power services, data services, and algorithm services required for artificial intelligence applications, based on artificial intelligence theory and adopting an artificial intelligence computing architecture.

[0049] The "computing center" mentioned in this invention refers to a facility that is mainly composed of infrastructure such as wind, thermal, hydro, and electricity, and IT hardware and software equipment, and has computing power, carrying capacity, and storage capacity, including general data centers, intelligent computing centers, supercomputing centers, etc.

[0050] The "supercomputing center" mentioned in this invention refers to a supercomputing data center, which is a data center based on supercomputers or large-scale computing clusters. It can provide large-scale computing, storage and network services and is widely used in aerospace, defense, oil exploration, climate modeling and genome sequencing and other application scenarios.

[0051] The “computing resources” mentioned in this invention refer to the technologies and facilities required for the development of the digital society that have the ability to compute, transmit, store and apply information, including but not limited to computing resources such as CPUs and GPUs, network resources such as switches and routers, storage resources such as storage arrays and distributed storage, security resources such as firewalls and intrusion detection systems, and supporting and guaranteeing resources such as wind, fire, water and electricity.

[0052] The "model" mentioned in this invention includes, but is not limited to, "large language model" and "multimodal large model".

[0053] The "large language model" mentioned in this invention refers to a large-scale language model (LLM), which is a language model with a large number of parameters. It is designed to understand and generate human language, and is trained with a large amount of text data. It can perform a wide range of tasks, including text summarization, translation, and sentiment analysis.

[0054] The “Multimodal Large Models” mentioned in this invention refer to models that combine multimodal information such as text, images, videos, and audio for training, including but not limited to multimodal large language models.

[0055] The “Token” mentioned in this invention refers to the basic unit of measurement when a large language model or multimodal large model processes text, code, image descriptions, document content, or other serialized inputs and outputs.

[0056] It is important to emphasize that different models use different tokenizers. The same text may correspond to different numbers of tokens in different models.

[0057] The "Token Computing Power Ledger" mentioned in this invention refers to a system used in an intelligent computing cloud platform to record in real time the number of tokens output by a large model and the computing power consumption.

[0058] The "anomaly classification dimension" described in this invention refers to a logical perspective or standard for categorizing and classifying all anomalies of a token based on whether their root cause is inevitable. This anomaly classification dimension includes deterministic anomalies and probabilistic anomalies.

[0059] The “computing cost” mentioned in this invention refers to the sum of all costs required to acquire and use the computing power of a GPU or intelligent computing cloud platform.

[0060] Please see Figure 1 , Figure 1 This is a flowchart of a method for repairing token anomalies using computing power on an intelligent computing cloud platform, as provided by the present invention. Figure 1As shown, the method includes: Step S1: Obtain the token currently output by the large model deployed in the intelligent computing cloud platform, and update the token sequence based on the currently output token.

[0061] It should be noted that training and running large models requires massive computing power, and the computing power requirements of large models are dynamic. Intelligent computing cloud platforms can provide massive computing power, as well as computing power that can dynamically change with the needs of large models.

[0062] In this invention, large models include, but are not limited to: large text models, large code models, multimodal large models, text generation models after speech transcription, image description generation models, and any models that are generated stepwise using autoregressive tokens or discrete units.

[0063] In some embodiments, the tokens currently output by the large model deployed in the intelligent computing cloud platform can be obtained by real-time monitoring of the decoder of the large model.

[0064] In some embodiments, the token currently output by the large model can be based on the user's input information. That is, when the user inputs information into the large model, the output of the large model can be monitored in real time to determine the token currently output by the large model.

[0065] For example, you can use a Token stream monitor to monitor the tokens currently being output by a large model in real time.

[0066] It should be noted that the Token Stream Monitor can be implemented either inside a large model decoder, or as a model service gateway, an agent runtime sidecar, a software development kit (SDK) middleware, or a browser-side streaming parser.

[0067] In some embodiments, once the current output token of the large model is determined, the token sequence can be updated based on the current output token. The token sequence is a sequence arranged chronologically for each token, and each token represents a token from the historical output of the large model.

[0068] In other words, when obtaining the current output token, it is also possible to obtain the content of the large model decoder output per token or per token block, the distribution of logit units, sampling parameters, probabilities, attention summaries, hidden state summaries, and decoding timestamps.

[0069] In some embodiments, by obtaining the decoding timestamp of the token currently output by the large model, the token currently output by the large model can be updated to the token sequence based on the decoding timestamp, so that the updated token sequence includes the token currently output by the large model.

[0070] The beneficial technical effect of step S1 is that by obtaining the token currently output by the large model, it is possible to perform token-by-token anomaly detection during the output of information by the large model, thereby reducing the failure rate of structured output, reducing the waste of computing power in model inference on the intelligent computing cloud platform, and saving the computing power of the intelligent computing cloud platform.

[0071] Step S2: Using computing power, the current output Token is subjected to multi-dimensional abnormal state detection based on its multi-dimensional state information. When an abnormality is detected, the current output Token is backtracked to determine the starting Token of the abnormality from the Token sequence.

[0072] It should be noted that during the process of outputting tokens from a large model, the multi-dimensional state information of the tokens can also be maintained to determine the multi-dimensional state information corresponding to each token.

[0073] The multi-dimensional state information includes, but is not limited to: structure stack, structured definition specification (schema) progress, code syntax state, language style state, entity number table, duplicate statistics window, security state, token computing power ledger, key-value (KV) cache page table, and candidate rollback point set, etc.

[0074] Optionally, for the first The token output by each step, and its corresponding multi-dimensional state information. It can be represented as: (1) in, Represents a token sequence. Indicates the structural state. Indicates the schema or tool call status. Indicates a semantically consistent state. Indicates the format status. Indicates the state of language style. This indicates a repetitive degradation state. Indicates a safe state. This represents the KV cache page table. This represents the set of candidate rollback points.

[0075] In some embodiments, the candidate rollback point set includes multiple candidate rollback points, each of which can be used to restore the large model's KV Cache and the output token sequence to a specific "node" state during the previous generation process. A node can be a candidate rollback point.

[0076] In some embodiments, computing power can be used to detect abnormal states of the currently output Token. Since the Token has multi-dimensional state information, abnormal state detection is also the detection of multi-dimensional abnormal states.

[0077] For example, multi-dimensional state information includes structural state, semantic state, etc., so the detection of multi-dimensional abnormal states can be performed on the current output token to detect structural abnormal states and semantic abnormal states respectively.

[0078] In other words, computing power can be used to detect abnormal states for each state of the token simultaneously.

[0079] In some embodiments, a detection order can be set for the detection of abnormal states in different dimensions, so that the abnormal state in one dimension can be detected first by computing power, and then the abnormal state in another dimension can be detected.

[0080] It should be noted that the detection granularity for the multi-dimensional abnormal state detection of this invention can be a single token, a group of tokens, a word, a field, a sentence, a line of code, or a tool parameter boundary. For Chinese scenarios, word boundaries, punctuation boundaries, and entity boundaries can be used in combination.

[0081] In some embodiments, when an anomaly is detected in the currently output token through computing power, it can be determined whether the currently output token is a deterministic anomaly or a probabilistic anomaly. That is, based on whether the currently output token is a deterministic or probabilistic anomaly, anomaly backtracking can be performed on the currently output token to determine the starting token of the anomaly from the token sequence.

[0082] The exception start token can be the currently output token; the exception start token can also be any token in the token sequence preceding the currently output token.

[0083] In some embodiments, if the currently output token is a deterministic anomaly, it means that the currently output token is the root cause of the anomaly; if the currently output token is a probabilistic anomaly, it means that the currently output token may cause the anomaly to occur.

[0084] Taking structural anomalies as an example, if the currently output token is a deterministic anomaly, it means that the token has a clear structural error such as syntax or formatting. If the output token is a probabilistic anomaly, the token is structurally valid, but its semantic logic may be abnormal.

[0085] In other words, if the currently output token is a deterministic anomaly, then the currently output token can be determined as the starting token of the anomaly; if the currently output token is a probabilistic anomaly, then each token in the token sequence can be evaluated to assess the contribution score of each token to the anomaly, and thus the starting token of the anomaly can be determined based on the contribution score of each token to the anomaly.

[0086] For example, the earliest token that contributes the most to the anomaly and is located in the most recent semantic segment can be identified as the starting token of the anomaly.

[0087] The beneficial technical effect of step S2 is that by performing multi-dimensional abnormal state detection on the currently output Token, and performing abnormal backtracking on the currently output Token when an abnormality is detected, the starting point Token of the abnormality can be determined from the Token sequence. This allows for accurate location of the starting point that caused the abnormality, avoiding simply taking the detected abnormal Token as the starting point Token of the abnormality, and improving the accuracy of abnormality location.

[0088] Step S3: Based on the abnormal starting point Token, determine the target rollback point from the candidate rollback points corresponding to the Token sequence, and determine the KV checkpoint associated with the target rollback point.

[0089] In some embodiments, the candidate rollback point corresponding to the token sequence can be the candidate rollback point corresponding to each token in the token sequence. This can be understood as a set of candidate rollback points corresponding to the token sequence, which includes a subset of candidate rollback points corresponding to each token. Each subset of candidate rollback points includes at least one candidate rollback point.

[0090] In some embodiments, at least one candidate rollback point corresponding to the abnormal starting token can be determined from the candidate rollback points corresponding to the token sequence, and the target rollback point of the abnormal starting token can be determined from the at least one candidate rollback point.

[0091] In some embodiments, a target rollback point can be determined from at least one candidate rollback point by scoring the candidate rollback points. The scoring result can be a score value for each candidate rollback point.

[0092] For example, each candidate rollback point can be scored according to a set scoring dimension to obtain a score value for each candidate rollback point. The higher the score value, the more suitable the candidate rollback point is for rollback operation.

[0093] In other words, the candidate rollback point with the highest score can be selected as the target rollback point.

[0094] In some embodiments, the scoring dimensions may include structural security, semantic preservation, computational efficiency savings, key-value recoverability, and risk of re-failure. Each dimension can be scored individually to obtain a sub-value for each dimension, and then the sub-values ​​can be weighted and summed to obtain the score value of the candidate rollback point.

[0095] Optionally, a scoring function can be defined based on the scoring dimension to score candidate rollback points.

[0096] It should be noted that the target rollback point can be determined based on a scoring function, or it can be determined using dynamic programming, reinforcement learning strategies, heuristic rules, risk minimization strategies, or adaptive strategies based on historical request statistics. This invention does not impose specific limitations on the method for determining the target rollback point.

[0097] It's important to note that a rollback point is a logical concept. During a rollback operation, physically usable data for recovery calculations, such as key-value cache page tables, is required. Key-value checkpoints can record data used for recovery calculations; therefore, once the target rollback point is determined, the associated key-value checkpoints can be identified.

[0098] Optionally, the KV checkpoint can record the key and value of the token in the form of a KV cache page table. The token can be recovered based on its key and value.

[0099] For example, an associated key-value checkpoint can be set for each candidate rollback point, generating a relationship between the candidate rollback point and the key-value checkpoint. After the target rollback point is determined, the key-value checkpoint associated with the target rollback point can be determined based on this relationship.

[0100] It should be noted that KV checkpoints can be saved at fixed token intervals, structural boundaries, natural segment boundaries, risk-increased locations, user-specified breakpoints, or GPU page boundaries. Alternatively, only page table references and detector summaries can be saved without copying the complete tensor.

[0101] The beneficial technical effect of step S3 is that by determining the optimal rollback point from the candidate rollback points as the target rollback point corresponding to the abnormal starting token, it is possible to roll back to the position where the abnormality can be eliminated while preserving the generated legal content to the maximum extent. This reduces invalid tokens and redundant calculations, and saves computing power of the intelligent computing cloud platform.

[0102] Step S4: Regenerate the first Token subsequence from the Token corresponding to the associated KV checkpoint to the currently output Token, and perform anomaly repair on the Token sequence based on the first Token subsequence.

[0103] In some embodiments, a key-value checkpoint is a checkpoint at any position in the token sequence. That is, a key-value checkpoint can correspond to a token in the token sequence.

[0104] For example, the tokens in the token sequence are arranged in order as Token1, Token2 and Token3, and their corresponding positions 1, 2 and 3 can be used to set KV checkpoints at any of the positions 1, 2 and 3.

[0105] If a KV checkpoint is set at position 2, the Token corresponding to the KV checkpoint will be Token2.

[0106] In this invention, Token anomaly repair refers to regenerating the abnormal Token. In order to avoid excessive computing power consumption in regenerating the Token, the Token between the associated KV checkpoint and the currently output Token can be regenerated.

[0107] In some embodiments, the token sequence is repaired by determining the token corresponding to the associated KV checkpoint from the token sequence, regenerating the tokens from the token corresponding to the associated KV checkpoint to the currently output token as the first token subsequence, and using the first token subsequence to replace the original tokens in the token sequence.

[0108] It should be noted that replacing the original tokens in the token sequence with the first token subsequence means replacing the tokens between the tokens corresponding to the KV checkpoints associated with the token sequence and the currently output tokens with the first token subsequence.

[0109] For example, the Token sequence includes Token1, Token2, Token3, Token4, Token5 and Token6; if the Token corresponding to the associated KV checkpoint is Token3 and the current output Token is Token6, then the original Tokens are Token3, Token4, Token5 and Token6.

[0110] Regenerate the first token subsequence from the token corresponding to the associated KV checkpoint to the currently output token. If the first token subsequence includes: Token3', Token4', Token5' and Token6', then the token sequence after replacing with the first token subsequence includes Token1, Token2, Token3', Token4', Token5' and Token6'.

[0111] In some embodiments, the token regeneration process can be regarded as an anomaly repair process. Therefore, the anomaly repair path, i.e. the token regeneration path, can be determined, and the tokens between the tokens corresponding to the associated KV checkpoints and the currently output tokens can be regenerated using the token regeneration path.

[0112] In other words, multiple pre-set candidate paths can be obtained, and the target path can be determined from the multiple candidate paths. The target path can then be used to regenerate the tokens between the tokens corresponding to the associated KV checkpoints and the tokens of the current output.

[0113] In some embodiments, there is a correspondence between candidate paths and anomaly types. The anomaly type is related to the dimension of the detected anomalous state. For example, when detecting structural anomalies, if an anomaly is detected, the anomaly type is "structural anomaly."

[0114] In some embodiments, by determining the current anomaly type, at least one candidate path corresponding to the anomaly type can be determined, and then the target path can be determined from at least one candidate path.

[0115] For example, a path selection function can be defined, and the target path can be determined from at least one candidate path based on the path selection function.

[0116] Optionally, the path selection function can solve for the cost of token regeneration using each candidate path and determine the candidate path with the minimum cost as the target path.

[0117] In some embodiments, to improve the success rate of token regeneration, multiple token subsequences can be generated based on the target path, and a first token subsequence can be determined from the multiple token subsequences. Optionally, a risk assessment can be performed on the multiple token subsequences, and the token subsequence with the lowest risk assessment result can be selected as the first token subsequence.

[0118] It should be noted that this invention only regenerates the first token subsequence from the token corresponding to the associated KV checkpoint to the currently output token. The tokens between the first token in the token sequence and the token corresponding to the associated KV checkpoint can be directly recovered based on the KV cache page table recorded by the KV checkpoint. Thus, the recovered token and the first token subsequence can be used as the token sequence after the anomaly is repaired.

[0119] It should be noted that the detection of multi-dimensional abnormal states, the determination of the abnormal starting point Token, the determination of the target rollback point, and the process of Token regeneration in this invention are all based on the computing power of the intelligent computing cloud platform.

[0120] The beneficial technical effect of step S4 is that regenerating the first token subsequence from the token corresponding to the associated KV checkpoint to the currently output token avoids regenerating the entire token sequence, saving computing power consumption of the intelligent computing cloud platform, maximizing computing power reuse, improving computing power utilization, and reducing the operating cost of the intelligent computing cloud platform. By restoring the tokens before the KV checkpoint and regenerating the tokens after the KV checkpoint, the regenerated first token subsequence can conform to the coherence and consistency of the contextual semantics, avoiding inconsistencies in the token sequence after anomaly repair, and improving the repair effect and success rate of anomaly repair. Moreover, by regenerating local tokens, the repair latency can be significantly reduced, the inference process can be accelerated, and the user experience can be improved.

[0121] In this invention, step S1 involves obtaining the token currently output by the large model deployed in the intelligent computing cloud platform and updating the token sequence based on the currently output token; step S2 involves using computing power to detect multi-dimensional abnormal states of the currently output token based on its multi-dimensional state information, and when an anomaly is detected, performing anomaly backtracking on the currently output token to determine the starting token of the anomaly from the token sequence; step S3 involves determining the target rollback point from the candidate rollback points corresponding to the token sequence based on the starting token of the anomaly, and determining the KV checkpoint associated with the target rollback point; step S4 involves regenerating the first token subsequence from the token corresponding to the associated KV checkpoint to the currently output token, and performing anomaly repair on the token sequence based on the first token subsequence. This allows for simultaneous output and detection of tokens, enabling timely detection of anomalies as they occur, significantly shortening the error propagation chain. By determining the starting token of the anomaly, accurate anomaly localization can be achieved, providing clear boundaries for subsequent repairs. During the token regeneration process, selecting a target path for local token regeneration can reduce invalid tokens and redundant calculations. This saves computing power on the intelligent computing cloud platform while increasing the success rate of token regeneration, thereby further improving the success rate of anomaly repair and computing power utilization.

[0122] In addition, by detecting and repairing tokens while they are being output, this invention can accelerate the model's inference process without interrupting the model's output, and enable users to obtain continuous output, thus improving the streaming output experience.

[0123] In some embodiments, step S4 includes: Step S4.1: Determine the exception type of the exception starting point Token, and based on the exception type, determine the target path for Token regeneration from the candidate paths for Token regeneration.

[0124] It should be noted that there is a correspondence between candidate paths and anomaly types. Specifically, the anomaly type is related to the dimension of the detected anomalous state. For example, when detecting structural anomalies, if an anomaly is detected, the anomaly type is "structural anomaly."

[0125] For example, when performing semantic state detection, if an anomaly is detected, the anomaly type is semantic anomaly.

[0126] The exception types include structural exceptions, schema exceptions, semantic exceptions, language drift, repeated loops, code errors, and security exceptions.

[0127] In some embodiments, the exception start point Token is determined when an exception is detected in the current output Token of the large model. Therefore, the exception type of the exception start point Token is the exception type of the currently output Token.

[0128] In some embodiments, the candidate path corresponding to the exception type of the exception starting token can be determined based on the correspondence between exception type and candidate path, so as to determine the target path for token regeneration from the candidate path.

[0129] In some embodiments, the cost of token regeneration using each candidate path can be determined, and the candidate path with the lowest cost can be identified as the target path.

[0130] The beneficial technical effect of step S4.1 is that, based on the exception type of the exception starting token, the target path can be determined, which can realize the token regeneration by determining different target paths according to different exception types, thereby improving the success rate of token regeneration.

[0131] Step S4.2: Based on the target path and associated KV checkpoints, regenerate the first Token subsequence.

[0132] In some embodiments, the KV cache page table of the associated KV checkpoint storage can be obtained to determine the key and value of the Token, and the first Token subsequence can be regenerated based on the key and value of the Token through the target path.

[0133] For example, if the target path is a context-free grammar constraint, the first token subsequence can be regenerated based on the key and value of the token and the context-free grammar constraint.

[0134] In some embodiments, during the process of regenerating the first token subsequence, multiple token subsequences may be regenerated, and the token subsequence with the lowest abnormal risk among the multiple regenerated token subsequences may be determined as the first token subsequence.

[0135] In other words, for the key and value of the token corresponding to an associated KV checkpoint, different random sampling parameters can be used to sample the key and value of the token, so as to regenerate multiple token subsequences based on the target path and different sampling results.

[0136] In some embodiments, determining the token subsequence with the lowest abnormal risk as the first token subsequence can be achieved by performing a risk assessment on multiple regenerated token subsequences, and then determining the token subsequence with the lowest risk assessment value as the first token subsequence based on the risk assessment value.

[0137] It should be noted that the regeneration of the first token subsequence can be accomplished by the original large model, or by a higher-capability model, a low-cost small model, a code-specific model, a Structured Query Language (SQL) specific model, a validation model, a utility function, or a rule template.

[0138] The beneficial technical effect of step S4.2 is that it enables the selection of differentiated token generation paths based on the anomaly type, which is conducive to the accurate allocation of computing resources, saves computing power of the intelligent computing cloud platform, and further improves token regeneration efficiency.

[0139] In some embodiments, step S4.1 includes: Step S4.1.1: Determine the computing power cost of the intelligent computing cloud platform and the number of historical failures in Token regeneration.

[0140] In some embodiments, the selection of a path is related to the computing power of the intelligent computing cloud platform and the failure information of token regeneration based on that path. That is, when determining the target path, paths with too many failed token regeneration attempts or excessive computing power consumption will not be selected.

[0141] It should be noted that the computing power of the intelligent computing cloud platform is a heterogeneous computing system in which multiple types of chips work together. The GPU is the core computing power of the intelligent computing cloud platform, so the computing power cost of the GPU can be statistically analyzed and used as the computing power cost of the intelligent computing cloud platform.

[0142] The cost of computing power refers to the total cost of acquiring and using the computing power of GPUs or intelligent computing cloud platforms. This can include costs related to hardware procurement, infrastructure construction, operation and maintenance, and many other aspects.

[0143] In some embodiments, token regeneration can be monitored in real time, and it can be determined whether each token regeneration is successful, so as to count the historical number of token regeneration failures.

[0144] Among them, Token regeneration failure refers to the abnormal state of the first Token subsequence that is regenerated being too risky, for example, the risk assessment value is greater than a set threshold.

[0145] The beneficial technical effect of step S4.1.1 is that by determining the computing power cost of the intelligent computing cloud platform and counting the historical failures of Token regeneration, it is helpful to select the path with low computing power consumption and high Token regeneration success rate when determining the target path.

[0146] Step S4.1.2: Determine multiple candidate paths corresponding to the anomaly type.

[0147] In some embodiments, multiple candidate paths can be pre-configured for each exception type.

[0148] For example, the multiple candidate paths corresponding to structural anomalies are: enabling stronger syntax mask, context-free grammar constraints, bracket stack compensation, field order template, or low-temperature sampling.

[0149] Several candidate paths for schema exceptions include: locking approved fields and regenerating only the current field or the current object; applying a local mask to enumerations, types, value ranges, and required fields; and repairing the schema template if necessary.

[0150] Multiple candidate paths for abnormal tool call parameters are: rollback at the parameter boundary, call tool metadata or external database to verify field names, units, IDs and time ranges, and then use the verification results as a local regeneration context.

[0151] Multiple candidate paths corresponding to semantic anomalies are: entity tables and numeric tables in fixed security prefixes, adding conflicting entities or units to the prohibited list or constraint hints; and calling the retrieval or business rule engine to confirm the correct entity when necessary.

[0152] The multiple candidate paths corresponding to language drift are: reducing temperature, increasing target language and stylistic constraints, and enabling language vocabulary bias or style verification model reordering.

[0153] The multiple candidate paths corresponding to the repetitive loop are: rollback to before the repetitive segment, increase the repetition penalty, adjust the top-p or temperature, enable contrastive decoding, disable the most recently repeated n-gram and restrict similar sentence patterns. Here, n-gram refers to a continuous sequence of n consecutive items (items, words, characters or tokens) in text or speech.

[0154] The possible paths for code errors are: rollback to the nearest statement boundary, enable parser, compiler, SQL dry-run, variable table or field table validation, and perform pre-execution validation on local candidates.

[0155] Multiple candidate paths for security anomalies include: rolling back to before the risky segment, switching security templates, permission verification, de-identification processing, or manual approval strategies, and refusing to continue generating sensitive suffixes if necessary.

[0156] The beneficial technical effect of step S4.1.2 is that by setting multiple candidate paths for each anomaly type, different target paths can be used to regenerate tokens for different anomaly types, thereby achieving differentiated token regeneration and improving the success rate of token regeneration.

[0157] Step S4.1.3: Construct a path selection function based on the anomaly type, computing cost, and historical failure count.

[0158] It should be noted that the path selection is not fixed and can be determined by a combination of factors, including the anomaly type, computing power cost, and historical failure count. Alternatively, path selection can also be based on risk level, business scenario, remaining token budget, and GPU load.

[0159] In some embodiments, the path selection function can combine anomaly type, computing cost, and historical failure count to make intelligent decisions based on anomaly type, computing cost, and historical failure count, and determine the target path from candidate paths.

[0160] For example, the path selection function can determine the candidate path that strikes the best balance between performance, cost and stability based on the anomaly type, computing power cost and historical failure count, and use it as the target path.

[0161] In some embodiments, the success probability of token regeneration based on any path can be determined based on the number of historical failures and the number of token regenerations, and a path selection function can be constructed based on the exception type, computing cost, and success probability.

[0162] In some embodiments, token regeneration can affect the user's waiting time and can also determine the latency of token regeneration based on any path, so as to construct a path selection function based on the exception type, computing cost, success probability and latency.

[0163] In some embodiments, the risks of token regeneration vary depending on the path taken. It is also possible to determine the future anomaly risk of token regeneration based on any path and to construct a path selection function based on anomaly type, computing cost, success probability and latency, as well as future anomaly risk.

[0164] The beneficial technical effect of step S4.1.3 is that, based on the anomaly type, computing power cost, and historical failure count, a path selection function is constructed, which can realize automated intelligent decision-making to determine the target path with good repair effect and low computing power cost from the candidate paths.

[0165] Step S4.1.4: Determine the target path from multiple candidate paths based on the path selection function.

[0166] Alternatively, the path selection function is as follows: (2) in, Indicates the target path. Indicates the exception type. Indicates the exception type is The corresponding candidate path, This represents the mathematical expectation when using this candidate path. Indicates potential future risks. Indicates computing power cost. Indicates time delay. This indicates the probability of success. , and These represent the weight values ​​for computing power cost, latency, and success probability, respectively.

[0167] In some embodiments, the target path can be determined by substituting multiple candidate paths corresponding to the anomaly type determined in step S4.1.2 into the above formula (2) for path selection.

[0168] The beneficial technical effect of step S4.1.4 is that by determining the target path from multiple candidate paths through the path selection function, the target path can be dynamically selected, avoiding the use of a single path for token regeneration, increasing the adaptability of path selection, and improving the success rate of token regeneration.

[0169] In some embodiments, step S4.2 includes: Step S4.2.1: Based on the target path, regenerate the second token subsequence from the token corresponding to the associated KV checkpoint to the token of the current output.

[0170] In some embodiments, to improve the success rate of token regeneration, multiple second token subsequences can be regenerated based on the target path when regenerating the first token subsequence. The second token subsequences can be considered as candidate token subsequences to determine the first token subsequence.

[0171] The beneficial technical effect of step S4.2.1 is that by regenerating multiple second token subsequences, fault tolerance redundancy and selection space can be provided, which is conducive to improving the fault tolerance of token regeneration and thus improving the repair success rate.

[0172] Step S4.2.2: Perform an abnormal state risk assessment on the second Token subsequence to obtain the risk assessment value.

[0173] In some embodiments, the risk assessment of the abnormal state of the second token subsequence can be determined based on the comprehensive risk of the second token subsequence, the legality of its structure or schema, the semantic conflict distance, and the computing cost.

[0174] Optionally, a risk detector can be used to detect the overall risk of the second token subsequence. The legality of the structure or schema can be detected by performing a legality check on the second token subsequence.

[0175] In some embodiments, the semantic conflict distance is used to quantify the degree of deviation between the second token subsequence and the original token sequence in terms of semantic coherence, logical consistency, and factual compatibility. A larger distance indicates a more severe semantic conflict and a lower quality of repair.

[0176] In some embodiments, a hierarchical quantization approach can be used to determine the semantic conflict distance of the second token subsequence. For example, based on the vector representation corresponding to the second token subsequence, the representation layer distance is calculated, and the probability layer distance and knowledge layer distance of the second token subsequence are determined. Then, the multi-layer distances are weighted and summed to obtain the semantic conflict distance.

[0177] In some embodiments, the computational cost of the second token subsequence can be determined based on the quantity of each token in the second token subsequence. The more tokens generated, the higher the computational cost.

[0178] Optionally, the formula for risk assessment of abnormal states for the second token subsequence is as follows: (3) in, Indicates the second token subsequence The risk assessment value, Indicates comprehensive risk. This represents the weighting value of the overall risk. Indicates the legality of the structure or schema. The weight value representing legality, Indicates semantic conflict distance, The weight values ​​representing the semantic conflict distance, Indicates computing power cost. The weight value represents the cost of computing power. Indicates the confidence level of the large model. The weight value represents the confidence level. This represents the original input to the large model. This indicates a sequence in the token sequence that does not undergo token regeneration. You can set a target rollback point.

[0179] The beneficial technical effect of step S4.2.2 is that by conducting a risk assessment of the abnormal state of the second token subsequence, it is possible to identify the second token subsequence with abnormal risk in advance, so as to screen out the first token subsequence without abnormality, thereby improving the success rate of token anomaly repair.

[0180] Step S4.2.3: In response to the risk assessment value of the second Token subsequence being less than the risk threshold, the second Token subsequence with the smallest risk assessment value is selected as the first Token subsequence.

[0181] In some embodiments, the magnitude of the risk assessment value can indicate the risk level of the second token subsequence, such as whether it is low-risk or high-risk. Specifically, the determination of whether the second token subsequence is low-risk or high-risk can be based on the magnitude of the risk assessment value and a risk threshold.

[0182] Optionally, in response to the risk assessment value of the second token subsequence being less than the risk threshold, the second token subsequence is determined to be low-risk; in response to the risk assessment value of the second token subsequence being greater than or equal to the risk threshold, the second token subsequence is determined to be high-risk.

[0183] In some embodiments, for a low-risk second token subsequence, the second token subsequence with the smallest risk assessment value can be determined as the first token subsequence based on the risk assessment value.

[0184] The beneficial technical effect of step S4.2.3 is that by selecting the second Token subsequence with the smallest risk assessment value as the first Token subsequence, the quality of Token regeneration can be quantified, and the second Token subsequence with good quality can be determined as the first Token subsequence, saving the extra computing power consumption caused by repeated trial and error.

[0185] In some embodiments, step S4.2.2 further includes: Step S4.2.2.1: In response to the risk assessment value of the second Token subsequence being greater than or equal to the risk threshold, perform risk verification on the second Token subsequence and determine the first Token subsequence from the second Token subsequence that has passed the risk verification.

[0186] In some embodiments, in response to the risk assessment value of the second token subsequence being greater than or equal to the risk threshold, and the second token subsequence being determined to be high-risk, a risk verification can be performed on the second token subsequence to re-verify its risk.

[0187] In this invention, risk verification can employ methods such as single-candidate real-time verification, multi-candidate beam search, sampling rearrangement, rule scoring, executor testing, manual approval, or business system replay verification. This invention does not specifically limit the method of risk verification.

[0188] In some embodiments, a second token subsequence that passes the risk verification can be obtained, and based on the risk assessment value of each of the second token subsequences that pass the risk verification, the second token subsequence with the smallest risk assessment value can be determined as the first token subsequence.

[0189] The beneficial technical effect of step S4.2.2.1 is that by performing risk verification on the high-risk second token subsequence, intelligent decision-making can be achieved for the second token subsequence, reducing the misjudgment rate of the second token subsequence.

[0190] In some embodiments, step S4.2.2.1 includes: Step S4.2.2.1.1: Obtain the second Token subsequence that failed the risk verification, and use it as the third Token subsequence that failed to regenerate the token.

[0191] It should be noted that if multiple token regenerations in the same region still fail to pass the risk verification, it indicates that the anomaly may not be a localized issue, but rather a result of conflicts in prompt words, schema, external evidence, model capabilities, or business rules.

[0192] In other words, upgrading computing power can prevent token regeneration failures caused by conflicts with prompt words, schemas, external evidence, model capabilities, or business rules.

[0193] In some embodiments, the second token subsequence that fails the risk verification can be used as the third token subsequence that fails to regenerate, so as to count the number of failures of the third token subsequence in regeneration.

[0194] The beneficial technical effect of step S4.2.2.1.1 is that by determining the third Token subsequence that failed to regenerate the Token, basic information can be provided for performing computing power upgrades, which is conducive to more accurate computing power upgrades based on the information of the current Token regeneration failure.

[0195] Step S4.2.2.1.2: Count the number of failures, remaining token budget, remaining computing power budget and risk level of the third token subsequence regeneration failure.

[0196] In some embodiments, an upgrade can be performed based on the number of failures, the risk level of the current computing power service, and the computing power budget. By statistically analyzing the number of failures in the rebirth of the third token subsequence, the remaining token budget, the remaining computing power budget, and the risk level, it can be determined whether to perform a computing power upgrade.

[0197] For example, a counter can be used to count the number of times the third token subsequence failed to regenerate.

[0198] In some embodiments, the remaining token budget refers to the number of tokens that have not yet been used out of the total number of tokens that are allowed to be generated. That is, the total number of tokens that are currently allowed to be generated and the number of tokens that have been regenerated can be obtained, and the difference between the total number and the number of tokens that have been regenerated can be calculated to determine the remaining token budget.

[0199] In some embodiments, the remaining computing power budget refers to the computing power that has not yet been consumed by the current computing power service. The risk level can be determined based on the risk assessment value of the third token subsequence. For example, if the risk assessment value is greater than the risk threshold, it is determined to be high risk.

[0200] The beneficial technical effect of step S4.2.2.1.2 is that, based on the number of failures, the risk level of the current computing power business, and the computing power budget, the computing power upgrade can be performed when multiple token regenerations in the same area still fail to pass the risk verification. This can improve the effectiveness of the computing power upgrade.

[0201] Step S4.2.2.1.3: Construct a computing power upgrade function based on the number of failures, remaining token budget, remaining computing power budget, and risk level.

[0202] Optionally, the constructed computing power upgrade function is as follows: (4) in, Indicates the number of failures. Indicates the remaining token budget. Indicates the remaining computing power budget. Indicates the risk level. Indicates the threshold number of times. Indicates the token budget threshold. Indicates the computing power budget threshold. This indicates the risk level threshold.

[0203] It should be noted that the computing power upgrade function is a decision trigger function, used to determine whether a computing power upgrade should be performed. When any one of the conditions on the right side of formula (4) is met, A value of 1 triggers the upgrade.

[0204] In other words, a decision is made when the number of failures exceeds a threshold, or the remaining token budget is less than a threshold, or the remaining computing power budget is less than a threshold, or the risk level exceeds a threshold. The value is 1.

[0205] The beneficial technical effect of step S4.2.2.1.3 is that, based on the number of failures, the remaining token budget, the remaining computing power budget, and the risk level, a computing power upgrade function is constructed, which can realize multi-dimensional judgment on whether computing power upgrade is needed and improve the accuracy of the timing of computing power upgrade.

[0206] Step S4.2.2.1.4: Respond to the computing power upgrade function instruction to perform computing power upgrade and execute the preset computing power upgrade operation.

[0207] In some embodiments, by determining whether the computing power upgrade function is 1, if the computing power upgrade function is 1, it can be determined that a computing power upgrade will be performed, and a preset computing power upgrade operation can be obtained.

[0208] In some embodiments, any operation can be selected from preset computing power upgrade operations to perform computing power upgrade.

[0209] Among these, the computing power upgrade operations can include switching to a higher-capacity model, rewriting only abnormal segments using expert models, calling external tools for verification and completion, increasing the local bundle search width, upgrading the low-cost detector to a large model validator, upgrading local repair to paragraph-level rewriting starting from the previous chapter, and upgrading automatic machine repair to manual approval.

[0210] Upgrading computing power can also involve a complete rewrite, rolling back to an earlier section, switching models, increasing detection intensity, calling external tools, asking users for supplementary information, or submitting for manual review.

[0211] The beneficial technical effect of step S4.2.2.1.4 is that by performing the computing power upgrade operation when the computing power upgrade function indicates that the computing power upgrade is to be performed, it is possible to achieve automatic local repair as much as possible in low-risk and low-cost scenarios, and to stop the consumption of invalid tokens in a timely manner in high-risk or multiple failure scenarios, thereby saving the computing power of the intelligent computing cloud platform.

[0212] In some embodiments, step S4 further includes: Step S4.1': Determine the position information of the KV checkpoint corresponding to the Token in the Token sequence.

[0213] It should be noted that the token anomaly repair in this invention can be implemented by regenerating a local token, while other tokens in the token sequence need to be retained.

[0214] For example, for a token corresponding to a key-value checkpoint, tokens preceding it in the token sequence can be considered prefixes, and tokens following it can be considered suffixes. Prefixes can be retained while suffixes are cleared, and token anomaly repair can be achieved by regenerating the suffixes.

[0215] In some embodiments, preserving the prefix requires explicitly specifying the position information of the token corresponding to the KV checkpoint in the token sequence to avoid errors in token preservation. The position information can indicate which token in the token sequence the KV checkpoint corresponds to.

[0216] The beneficial technical effect of step S4.1' is that by determining the position information of the KV checkpoint corresponding to the Token in the Token sequence, it is possible to identify the Tokens that should be retained, avoid retaining abnormal Tokens and causing Token anomaly repair failure, and improve the success rate of anomaly repair.

[0217] Step S4.2': Truncate the Token sequence based on the location information, retain the fourth Token subsequence before the location information, and delete the fifth Token subsequence starting from the location information.

[0218] In some embodiments, during large model decoding, KV checkpoints can record the keys of historical tokens for each layer of attention modules. Sum Assume the large model has Layer, each layer If there are several attention heads, then the KV checkpoint at step t can be represented as: (5) If the entire token sequence is to be regenerated, a large number of key-value caches corresponding to the prefix need to be recalculated. Instead, the token sequence can be truncated based on location information, and the fourth token subsequence before the location information can be retained based on the key-value cache page table corresponding to the key-value checkpoint. This allows for rollback to the target rollback point to restore the fourth token subsequence.

[0219] Based on the KV cache page table corresponding to the KV checkpoint, rollback to the target rollback point. It can be represented as: (6) In some embodiments, the KV cache is managed in units of fixed-size token blocks or pages, and candidate rollback points are aligned with page boundaries or structural boundaries as much as possible. Located within a page, it can preserve the entire preceding page and truncate the last page.

[0220] If a copy-on-write mechanism is used, multiple second token subsequences can share the same prefix KV cache, creating new KV pages only at the branch suffix. This supports local bundle search and multiple candidate verifications without repeatedly copying the huge prefix KV cache.

[0221] In some embodiments, in addition to restoring the KV cache, the following states also need to be restored during rollback: output buffer, structure stack, schema progress, entity number table, language style window, repeating n-gram count, sampling random seed or sampling state, beam candidate state, invoked tool record, revocable streaming segment state, and computing power ledger.

[0222] If some tokens have been streamed to users but have not yet been confirmed, they can be marked as "revocable fragments" through the front-end protocol; if they are irrevocable, the partial repair content can be packaged into a subsequent correction segment, or a delayed confirmation streaming strategy can be adopted in high-reliability scenarios.

[0223] It should be noted that truncating the token sequence based on location information yields a fourth token subsequence preceding the location information and a fifth token subsequence starting from the location information. In other words, the fourth token subsequence is the prefix, and the fifth token subsequence is the suffix.

[0224] In this invention, the fourth Token subsequence needs to be retained, while the fifth Token subsequence needs to be deleted so that the first Token subsequence can be regenerated at the corresponding position of the fifth Token subsequence.

[0225] For a detailed implementation of regenerating the first token subsequence, please refer to any of the above embodiments, which will not be repeated here.

[0226] The beneficial technical effect of step S4.2' is that by retaining the fourth Token subsequence before the location information and deleting the fifth Token subsequence starting from the location information, it is possible to retain the security prefix and only remove the abnormal suffix, thereby reducing invalid tokens and duplicate calculations and saving computing power of the intelligent computing cloud platform.

[0227] Step S4.3': Combine the first Token subsequence and the fourth Token subsequence in sequence to obtain the repaired Token sequence.

[0228] In some embodiments, the regenerated first Token subsequence and the retained fourth Token subsequence can be combined in sequence to achieve Token anomaly repair, so that the repaired Token sequence does not contain the anomaly repaired Token sequence.

[0229] The beneficial technical effect of step S4.3' is that by combining the first Token subsequence and the fourth Token subsequence in sequence to obtain the repaired Token sequence, it is possible to only repair the abnormal suffix and retain the secure prefix, thereby reducing the computing power consumption of Token regeneration and improving the repair efficiency.

[0230] In this invention, the multi-dimensional anomaly detection includes at least two of the following: anomaly risk detection, structure detection, schema and tool call detection, semantic constraint and entity consistency detection, language style drift detection, and duplication and degradation detection.

[0231] In some embodiments, step S2 includes: Step S2.1: Obtain the status information corresponding to at least one abnormal status detection item from the multi-dimensional status information.

[0232] In some embodiments, the multi-dimensional state information includes state information corresponding to each abnormal state detection item. At least one abnormal state detection item's state information can be obtained from the multi-dimensional state information based on the correspondence between the abnormal state detection items and the state information.

[0233] For example, the status information corresponding to structural detection includes the structural stack and schema progress.

[0234] For example, the state information corresponding to semantic constraints and entity consistency detection includes language style state and entity number table.

[0235] The beneficial technical effect of step S2.1 is that obtaining the state information corresponding to at least one abnormal state detection item from the multi-dimensional state information enables each abnormal state detection to accurately use the corresponding state information for detection, thereby improving the accuracy of abnormal state detection.

[0236] Step S2.2: Based on the state information, perform anomaly detection on the current output Token for anomaly detection items.

[0237] For anomaly risk detection, multiple detectors can be used to detect the anomaly risk of the currently output token, such as determining the deterministic anomaly flag of the currently output token. and probabilistic anomaly scores .

[0238] Among them, deterministic anomalies include illegal JSON state machines, impossible-to-close bracket stacks, explicit errors in schema field types, and incorrect placement of SQL reserved words; probabilistic anomalies include language style drift, increased repetition, sudden drops in entity confidence, and abnormal changes in output distribution entropy.

[0239] In some embodiments, a deterministic exception flag is determined for the currently output token. and probabilistic anomaly scores Afterwards, the anomaly risk score of the currently output token can be calculated. The anomaly risk score at step t is calculated. The formula is shown below: (7) in, This represents a set of detectors, including multiple detectors. , Indicates detector weights, This is the deterministic anomalous amplification factor.

[0240] In some embodiments, if a deterministic anomaly exists, the anomaly localization process can be initiated immediately to determine the starting token of the anomaly; if only a probabilistic anomaly exists, a hysteresis threshold can be set, including a high-risk threshold and a low-risk persistence threshold, to avoid frequent rollbacks due to normal fluctuations of a single token.

[0241] For example, the following triggering conditions can be used to determine whether to perform a rollback: (8) in, The high-risk threshold The low-risk sustained threshold To observe the window length, The number of steps required for high-risk activities within the window. This represents the abnormal risk score at step i.

[0242] For structure detection, a structure detector can be used to check whether the currently output token satisfies the partial validity and completeness of the target structured language.

[0243] In this invention, the structure detector can be implemented using a finite state machine, a pushdown automaton, a context-free grammar parser, an incremental parser, a template state machine, a regularization state machine, or a business rule state machine.

[0244] Taking JSON as an example, the corresponding status information for structure detection includes bracket stack, string status, escape status, key-value pair status, array element status, and comma status.

[0245] In some embodiments, structure detection can be achieved by performing state transitions and structure transitions on the currently output token.

[0246] If the current state is The currently output token is The structural transition is as follows: (9) in, express The structural state after the transfer.

[0247] when When, it means This causes the structure to enter an illegal state, such as the presence of an extra comma at the beginning of an object, an unclosed string containing an invalid control character, or continuing to generate field names after the array has ended. In this case, the structure detector can output a deterministic exception, and a deterministic exception flag will be displayed. .

[0248] For non-strict formal languages ​​such as Markdown tables, claim numbers, storyboard numbers, and chapter levels, the structure detector can achieve structure transfer through state templates and numbering transfer rules.

[0249] The state transition can be represented as: (10) in, Representing state The state after the transfer Let d be the state update function for the d-th detector. The currently output token. For the output probability distribution of the large model at step t, or a summary thereof, This can be represented by a hidden state, an attention summary, or a token obtainable from the server. This represents the original input to the large model.

[0250] For schema and tool call detection, a schema and tool call detector can be used. The schema and tool call detector can not only determine whether the currently output token is valid, but also whether the currently output token still has a valid completion path.

[0251] Optionally, let the target schema be The token sequence is Then the predicate can be completed as follows: (11) in, This indicates that the token subsequence can be regenerated, but has not yet been regenerated.

[0252] like If this is the case, it means that the current output token has caused the schema to become unrepairable, and an anomaly location should be triggered.

[0253] For example, if a field requires that the enumeration value can only be "add", "modify", or "delete", and the model generates "updating" and the current field string is already closed, then subsequent tokens cannot make that field a valid enumeration again.

[0254] For example, if the tool calls parameters requiring "amount_yuan" to be an integer in yuan, and the model generates "amount_wan" in the field name and closes the object, then although the JSON syntax is valid, the schema path has deviated.

[0255] For Agent tool calls, the tool call status can be broken down into function name status, parameter object status, parameter value status, required field status, business domain status, and external tool executability status.

[0256] For example, if a flight search tool requires city, date, and number of passengers as parameters, it can call a lightweight place name checker or alias mapping when the city field is completed; check the date format, time zone, and whether it exceeds the bookable range when the date field is completed; and check whether the number is a positive integer when the number of passengers is completed.

[0257] Therefore, errors can be detected when the parameters are partially completed, rather than after the entire tool call output has finished.

[0258] For semantic constraints and entity consistency detection, a semantic constraint detector can be used. A semantic constraint detector can handle constraints that cannot be fully expressed through syntax.

[0259] In this invention, the semantic constraint detector can be implemented using rule engines, domain dictionaries, vector retrieval, small model classifiers, large model validators, knowledge graphs, database queries, external interface verification, or a combination of these methods.

[0260] Optionally, a set of constraints can be extracted from user input, prompts, retrieval evidence, business rules, and generated tokens: (12) Each constraint It can include entity name, entity type, value, unit, time, location, variable name, field name, formula symbol, reference number, language style, or business rule.

[0261] For the semantic fragment newly formed in step t It can calculate the conflict score between the entity and the constraint set, achieving consistency between semantic constraints and entities. For example, if the conflict score is greater than a threshold, an anomaly is detected. Among these, semantic segments... This is the semantic fragment corresponding to the currently output Token.

[0262] In some embodiments, semantic fragments can be calculated. Calculate semantic fragments by considering the entity conflict distance, numerical conflict distance, unit conflict distance, and relation or reference conflict distance for each constraint in the constraint set. The conflict score of each constraint in the constraint set.

[0263] Alternatively, the formula for calculating the conflict score is as follows: (13) in, Indicates conflict score, Indicates the distance of entity conflict. Indicates the distance of numerical collisions. Indicates the unit collision distance. Indicates the distance of a relationship or reference conflict. The weight value represents the distance of entity conflict. The weight value representing the distance of numerical conflicts. The weight value represents the unit conflict distance. The weight value representing the distance of a relationship or reference conflict.

[0264] For example, if a user requests that the monetary unit be "ten thousand yuan," the large model generates "yuan" in the text and combines it with the numerical value to form a definite meaning. Increased; when the user requests to reference only the "customer_id" field from table A, but the large model generates "cust_id" and this field does not exist, Increase; Previously, N was defined as the number of samples; later, when N is interpreted as the number of network layers, Increase.

[0265] It's important to note that semantic constraint and entity consistency detection doesn't involve performing high-cost, large-scale model judgments on every token. Instead, it employs a layered triggering mechanism. By identifying entity boundaries, number boundaries, sentence boundaries, and parameter boundaries, a lightweight classifier, dictionary, vector retrieval, rule engine, or verification model is only invoked when a semantic segment closes or its confidence level changes abruptly. This approach reduces computational consumption while maintaining real-time performance.

[0266] Language style drift detection can be performed using a language style detector. A language style detector can detect whether the currently output token deviates from the target language or style.

[0267] Suppose the required output language distribution is The language style distribution of the currently output tokens within the current sliding window is as follows: The drift can be measured by the divergence distance, which means that language style drift can be detected based on the divergence between the language style distribution within the current sliding window and the required output language distribution.

[0268] Optionally, computational language style drift The formula is shown below: (14) in, The length of the sliding window. It can be calculated from stylistic classifiers, term density, sentence length, honorifics, or legalistic features. This represents a coefficient, which can be a constant. This indicates the divergence calculation. This represents the sequence of tokens within the sliding window.

[0269] Optionally, language style drift can be detected, i.e. If the value is not 0, it indicates that the currently output token has an anomaly of language style drift.

[0270] For duplicate and degradation detection, a duplicate detector can be used. The duplicate detector can simultaneously calculate the n-gram duplicate rate, semantic similarity, and output distribution entropy, and determine whether there is a risk of duplicate or degradation based on the n-gram duplicate rate, semantic similarity, and output distribution entropy.

[0271] Here, an n-gram represents a continuous sequence of n consecutive items (items, words, characters, or tokens) in text or speech. In this invention, an n-gram is a continuous sequence of n tokens.

[0272] In some embodiments, the repetition risk can be calculated based on the n-gram repetition rate, semantic similarity, and output distribution entropy, and when the repetition risk is greater than a set value, repetition and degradation anomalies can be detected.

[0273] Let the length be n The set of n-grams is Then it can be done Each n-gram The number of times an element appears within a window can be used to calculate the repetition rate within the window, based on that number and the n-gram set. .

[0274] Optionally, in-window repetition rate The calculation formula is as follows: (15) in, express Number of times it appears in the window It is an integer.

[0275] In some embodiments, the output distribution entropy can be determined based on the probability distribution of the model output. Let the... t The step model output probability distribution is Then the output distribution entropy The calculation formula is as follows: (16) in, Represents any token, This represents a token sequence. Indicates at time t Predicting the next token is exactly what it is. The probability of.

[0276] It should be noted that, if A sharp increase, and A prolonged and abnormally low level indicates that the large model may be stuck in repetitive output; if An abnormally high probability of semantic instability between tokens may indicate that the large model is in an uncertain drift state.

[0277] If the model has similar probabilities for many tokens, the uncertainty is high and the entropy is large. If the model concentrates its probabilities almost exclusively on a few tokens, the uncertainty is low and the entropy is small. In repetitive scenarios, the model may continuously and very definitively output the same type of token. In this case, the probability distribution becomes very sharp, and the entropy decreases.

[0278] In some embodiments, semantic similarity can be the text similarity between adjacent windows. The text similarity between adjacent windows can be obtained by calculating the similarity between corresponding tokens of adjacent windows, such as cosine similarity.

[0279] In some embodiments, the repetition risk can be calculated based on window repetition rate, semantic similarity, and output distribution entropy. The formula is shown below: (17) in, The weight value representing the repetition rate. This indicates the weight values ​​corresponding to the output distribution entropy. Represents the baseline entropy. This represents the weight value corresponding to the text similarity between adjacent windows. Indicates the similarity of text between adjacent windows. This represents the sequence of tokens within two consecutive sliding windows.

[0280] If two consecutive windows contain almost identical content, it indicates that the model may be repeating the same passage.

[0281] The beneficial technical effect of step S2.2 is that different detection methods can be used to detect anomalies in the current output Token, thereby improving the accuracy of anomaly detection, avoiding errors caused by a single method, and thus promptly detecting anomalies in the current output Token.

[0282] In some embodiments, step S2.2 includes: Step S2.2.1: In response to the detection that the current output Token is abnormal, determine the abnormality classification dimension of the current output Token.

[0283] It should be noted that the anomaly classification dimensions include deterministic anomalies and probabilistic anomalies. The current output token is either a deterministic anomaly or a probabilistic anomaly, and the corresponding anomaly starting token is different.

[0284] In some embodiments, the anomaly classification dimension of the currently output token can be determined based on its anomaly risk score. For example, if the anomaly risk score is greater than a set score threshold, the currently output token is determined to be a deterministic anomaly; if the anomaly risk score is less than or equal to the set score threshold, the currently output token is determined to be a probabilistic anomaly.

[0285] The beneficial technical effect of step S2.2.1 is that by determining the anomaly classification dimension of the currently output Token, different anomaly starting tokens can be determined in the case of deterministic anomalies and probabilistic anomalies, thereby improving the accuracy of anomaly starting token location.

[0286] Step S2.2.2: Determine the starting token of the anomaly from the token sequence based on the anomaly classification dimension.

[0287] In some embodiments, if the currently output Token is a deterministic anomaly, it can be determined that the currently output Token is the starting token of the anomaly; if the currently output Token is a probabilistic anomaly, each Token in the Token sequence can be filtered to determine the starting token of the anomaly.

[0288] For example, the earliest token that shows a significant increase in risk and is located within the most recent semantic segment can be selected as the starting token of the anomaly.

[0289] In some embodiments, the anomaly initiation token can be filtered based on the anomaly contribution score of each token in the token sequence.

[0290] The beneficial technical effect of step S2.2.2 is that by determining different Tokens as the starting point Tokens of anomalies, the accuracy of anomaly location can be improved, which helps to determine the rollback boundary and avoids the recovery of Tokens with anomalies.

[0291] In some embodiments, step S2.2.2 includes: Step S2.2.2.1: In response to the anomaly classification dimension being a deterministic anomaly, determine the currently output Token as the anomaly origin Token.

[0292] In some embodiments, for deterministic anomalies, the anomaly originating token can be the token that caused the illegal transition state of the token, which is the currently output token. The determination formula is as follows: (18) in, Indicates the token representing the starting point of the anomaly. Indicating an illegal transfer status of the Token, the starting Token of the exception is the Token number. The token of the step, that is, the token currently output.

[0293] The beneficial technical effect of step S2.2.2.1 is that by clearly defining the current output Token as the starting token of the anomaly when it is a deterministic anomaly, it is easier to trigger a rollback immediately when an anomaly is detected, thereby reducing the latency of the token.

[0294] Step S2.2.2.2: In response to the anomaly classification dimension being probabilistic anomalies, calculate the anomaly contribution score and overall risk increment for each token in the token sequence.

[0295] In some embodiments, for probabilistic anomalies, an anomaly contribution score and a comprehensive risk increment for each token in the token sequence can be calculated. The comprehensive risk increment can be calculated based on the anomaly risk score. The difference in anomaly risk scores between two adjacent tokens can be used as the comprehensive risk increment. The calculation formula is shown below: (19) in, Indicates the overall increase in risk. Indicates the first i The abnormal risk score of each token. Indicates the first i -1 token's abnormal risk score.

[0296] In some embodiments, the anomaly contribution score of each token in the token sequence can be determined based on the probabilistic anomaly score of each token. Optionally, the probabilistic anomaly score of each token within the backtracking window length in the token sequence can be determined to calculate the anomaly contribution score of each token within the backtracking window length.

[0297] In some embodiments, the detector set can be Each detector Abnormal contribution scores were calculated for all.

[0298] Optionally, calculate the abnormal contribution score. The formula is shown below: (20) in, and Indicates the weight value. Indicates taking the positive part. Indicates the length of the backtracking window. Indicates the first i The probabilistic anomaly score of each token. Indicates the first j The probabilistic anomaly score of each token. Represents the conditional generation probability. It can be represented as given the original input and the generated prefixes Under these conditions, the model predicts The probability value. Indicates the first token to the second token. i A sequence of tokens up to -1 token.

[0299] The beneficial technical effect of step S2.2.2.2 is that when the anomaly classification dimension is probabilistic anomaly, the anomaly contribution score and comprehensive risk increment of each Token in the Token sequence are calculated. Based on the anomaly contribution score and comprehensive risk increment, the starting token of the anomaly can be determined. This can realize the determination of the starting token of the anomaly from multiple dimensions and enhance the accuracy of determining the starting token of the anomaly.

[0300] Step S2.2.2.3: Based on the abnormal contribution score and comprehensive risk increment of each Token, determine the Token that meets the screening conditions for abnormal starting point Token, and use it as the abnormal starting point Token.

[0301] In some embodiments, the abnormal starting point token screening conditions can be determined based on the abnormal contribution score and the comprehensive risk increment, and it can be determined whether each token in the token sequence meets the abnormal starting point token screening conditions, so that the token that meets the abnormal starting point token screening conditions is used as the abnormal starting point token.

[0302] Optionally, the exception origin Token filtering condition can be expressed as: (twenty one) in, Indicates the threshold for abnormal contributions. This indicates the threshold for risk increment.

[0303] It should be noted that if any token itself exhibits obvious abnormal signs and the cumulative risk increases significantly after that token, taking the minimum means selecting the earliest token among those that meet the screening criteria as the starting point of the anomaly, rather than finding the token with the highest risk.

[0304] In other words, the abnormal starting point token selection condition represented by formula (21) is: the earliest token with significantly increased risk and located in the most recent semantic segment.

[0305] For example, if the model starts language drifting from the 100th token, the risk is highest at the 110th token. The token that should actually be rolled back is often the 100th token, not the 110th token.

[0306] For example, for entity-digit conflicts, the abnormal starting point token can be traced back to the starting token of the digit or entity segment; for language drift, the abnormal starting point token can be traced back to the starting point of the window that caused the language classification distribution to cross the threshold; for repeated loops, the abnormal starting point token can be traced back to the position of the first loop closure of the repeated segment.

[0307] The beneficial technical effect of step S2.2.2.3 is that by using the anomaly starting point token screening condition, the token that the anomaly risk appears earliest can be identified as the anomaly starting point token. This is beneficial for setting the rollback point at the source of the anomaly, avoiding the propagation of the anomaly, and thus improving the anomaly repair effect.

[0308] In this invention, candidate rollback points include: structural closure points, semantic boundaries, code statement boundaries, tool parameter boundaries, low-risk token boundaries, and key-value checkpoints.

[0309] For example, structural closure points include the closure of JSON objects or array elements, the closure of extensible markup language (XML) tags, the restoration of the bracket stack to a stable level, and the end of a Markdown table row.

[0310] Semantic boundaries are periods, semicolons, the end of paragraphs, the end of list items, the end of claims, and the end of code comment blocks.

[0311] The boundaries of code statements are the end of a function definition, the end of an indented block, the end of an SQL clause, the balancing of parentheses in an expression, and the end of a statement with a semicolon.

[0312] Tool parameters are defined by the following boundaries: the end of a key-value pair, the end of an array element, the completion of a required field group, or the passing of a partial validation of a parameter object.

[0313] KV checkpoints are saved once for every M tokens, once for each structural boundary, once for each natural segment, or more frequently in high-risk tasks.

[0314] The low-risk token boundary is the token position where the risk of several recent detectors is below the threshold and the language, format, and structure are stable.

[0315] In some embodiments, a candidate rollback point may record a token index, as well as the current structure stack, schema state, entity number table snapshot, duplicate statistics window summary, random sampling state, beam state, KV cache page table reference, and revocable states that have been streamed to the user.

[0316] In some embodiments, step S3 includes: Step S3.1: Based on the preset scoring rules, score each candidate rollback point to obtain the score value corresponding to each candidate rollback point.

[0317] In some embodiments, the preset scoring rules may include a scoring function and scoring features. By determining the scoring features of each candidate rollback point and inputting them into the scoring function for calculation, the scoring value corresponding to each candidate rollback point can be calculated.

[0318] Optionally, the scoring features may include the boundary integrity of the candidate rollback point, the availability of the KV checkpoint, the preservation of semantic integrity, the computational savings of retrying relative to the whole, the risk of failure again when regenerating from that point, and the impact of rollback on streaming experience and latency.

[0319] The beneficial technical effect of step S3.1 is that scoring the candidate rollback points can determine the target rollback point based on the performance of the candidate rollback points, rather than simply selecting the rollback point closest to the anomaly, which helps to save computing power and improve the success rate of rollback.

[0320] Step S3.2: Determine the candidate rollback point with the highest score as the target rollback point.

[0321] In some embodiments, the formula for determining the target rollback point is as follows: (twenty two) in, Indicates the target rollback point. Indicates a candidate rollback point. Represents the set of candidate rollback points. This indicates the rating value.

[0322] The beneficial technical effect of step S3.2 is that by determining the candidate rollback point with the highest score as the target rollback point, it is possible to select a rollback point that saves computing power and has a high rollback success rate, which is conducive to improving the success rate of subsequent anomaly repair.

[0323] In some embodiments, step S3.1 includes: Step S3.1.1: Determine the scoring characteristics of each candidate rollback point.

[0324] In some embodiments, the scoring features for each candidate rollback point may include: boundary integrity, KV checkpoint availability, preservation of semantic integrity, computational savings relative to overall retries, risk of failure again after regeneration from that point, and the impact of rollback on streaming experience and latency.

[0325] Optionally, the boundary integrity of a candidate rollback point can be determined by determining whether the candidate rollback point is on a valid logical or syntactic boundary.

[0326] Optionally, the availability of KV checkpoints for candidate rollback points can be determined by checking whether the historical KV cache corresponding to the candidate rollback point is actually stored in memory.

[0327] Optionally, the semantic integrity of the candidate rollback point can be determined by whether the retained prefix token after truncation from the candidate rollback point is sufficient to convey core user intent and factual information.

[0328] Alternatively, the computational savings of a candidate rollback point relative to the overall retry can be determined by determining the computational savings of selecting any candidate rollback point compared to regenerating the entire sequence.

[0329] Optionally, the risk of a candidate rollback point failing to be regenerated from that point can be determined by predicting the probability of triggering the anomaly again after regeneration from any candidate rollback point.

[0330] Optionally, the impact of rolling back at a candidate rollback point on the streaming experience and latency can be determined by determining the magnitude of the visible latency to the user when selecting a candidate rollback point in a token-by-token output scenario.

[0331] The beneficial technical effect of step S3.1.1 is that by determining the scoring characteristics of each candidate rollback point, the favorable and unfavorable effects of rollback based on each candidate rollback point can be considered from multiple perspectives, so as to achieve a balance between favorable and unfavorable effects, thereby improving the rollback effect of using the target rollback point for rollback.

[0332] Step S3.1.2: Based on the defined scoring function and scoring features, score each candidate rollback point.

[0333] In some embodiments, a scoring function can be defined based on scoring features, and the value corresponding to the scoring feature of each candidate rollback point can be input into the scoring function to score each candidate rollback point. Optionally, the scoring function is as follows: (twenty three) in, Indicates boundary integrity. Indicates the availability of KV cache checkpoints. This indicates that semantic integrity is preserved. This represents the computational power saving compared to the overall retry rate. This indicates the risk of failure again if the process is restarted from this point. This indicates the impact of rollback on streaming experience and latency.

[0334] The weight values ​​represent the boundary integrity. The weight value representing the availability of KV checkpoints. The weight value representing the preservation of semantic integrity The weight value representing the computational power saved relative to the overall retries. This represents the weight value indicating the risk of regenerating the risk of failure again from that point. The weighted value represents the impact of rollback on streaming experience and latency.

[0335] The beneficial technical effect of step S3.1.2 is that scoring each candidate rollback point from multiple perspectives can automatically calculate the optimal balance between protecting computing power budget and ensuring user experience, thereby improving the effect of using the target rollback point for rollback and helping to improve the success rate of subsequent repairs.

[0336] In some embodiments, the method further includes: Step S5: Determine the computing power saving information for regenerating the first token subsequence based on KV checkpoints.

[0337] It should be noted that during each token regeneration process, a token computing power ledger can be maintained, recording information such as the number of generated tokens, the number of saved key-value checkpoints, the number of abnormal tokens, the number of rolled-back tokens, the number of locally regenerated tokens, the verification model consumption, the tool call consumption, and the overall retry hypothesis cost. Furthermore, after regenerating the first token subsequence, based on the above information, computing power savings can be calculated, thereby clarifying the computing power saved by using the intelligent computing cloud platform of this invention to achieve token anomaly repair through computing power.

[0338] For example, the ratio of the savings between locally regenerated tokens and overall retries can be used as information on computing power savings.

[0339] For example, global savings information can be obtained by using a local regeneration token method on a computing intelligent cloud platform, and this information can be used as computing power saving information.

[0340] The beneficial technical effect of step S5 is that by saving information through unified computing power, the computing power saved for the intelligent computing cloud platform can be visualized.

[0341] In some embodiments, step S5 includes: Step S5.1: Determine the token length, risk verification cost, and local recovery cost for regenerating the first token subsequence based on KV checkpoints.

[0342] In some embodiments, the token length of the first token subsequence is regenerated based on the KV checkpoint. This can be the number of tokens in the first token subsequence. The token length is determined by counting the number of tokens in the first token subsequence.

[0343] In some embodiments, the cost of risk verification is related to the amount of resources required to generate a token. That is, the computing power consumed in the risk verification process can be uniformly converted into the computing power required to generate a token, based on its floating-point operation cost. In other words, the cost of risk verification equals the computing power consumed in generating a token.

[0344] In some embodiments, local recovery cost refers to the fixed overhead from rollback to the target rollback point. This may include the cost of KV cache state switching and memory loading, the cost of resetting the generation context, and the cost of maintaining cache consistency.

[0345] The beneficial technical effect of step S5.1 is that by clarifying the computing power consumption and recovery cost of local token regeneration based on KV checkpoints, the computing power saving information of local token regeneration can be accurately calculated, thereby improving the accuracy of computing power saving information.

[0346] Step S5.2: Based on the token length, risk verification cost, and local recovery cost, determine the computing power saving ratio and global saving information as computing power saving information.

[0347] In some embodiments, the token length, risk verification cost, and local recovery cost can be summed to determine the cost information of local token regeneration and obtain the cost information of overall retry, so as to calculate the computing power saving ratio based on the cost information of local token regeneration and the cost information of overall retry.

[0348] Optionally, the formula for calculating the computing power saving ratio is as follows: (twenty four) in, Indicates the percentage of computing power saved. Indicates the token length. This indicates the cost of risk verification. Indicates the cost of partial restoration. This indicates the cost information for using a complete retry.

[0349] In some embodiments, global resource saving information can be computational resource saving for multiple requests. That is, a large model can receive multiple input information sent by multiple users at the same time. The input information is a request. For each request, the large model may output an abnormal token. In this case, local token regeneration is required to realize token anomaly repair.

[0350] For multiple requests, the savings information for each request can be calculated through local token regeneration, and the savings information of each request can be summed to obtain the global savings information. (25) in, This indicates global savings information. Indicates a request set. Indicates a request, This represents the overall retry cost. This indicates the cost of partial restoration.

[0351] The beneficial technical effect of step S5.2 is that by saving information through unified computing power, it is possible to visually demonstrate the computing power saved by the local token regeneration for the intelligent computing cloud platform.

[0352] Based on the above embodiments, the present invention can also propose a system for repairing token anomalies through computing power in an intelligent computing cloud platform, so as to realize the method of repairing token anomalies through computing power in the intelligent computing cloud platform of the present invention.

[0353] The intelligent computing cloud platform's system for token anomaly repair via computing power includes: a token stream access module, a multi-dimensional state maintenance module, a structure detector, a schema and tool call detector, a semantic constraint detector, a language style detector, a duplication and degradation detector, a code and SQL detector, a security detector, an anomaly location module, a rollback point selection module, a KV checkpoint and recovery module, a local regeneration path module, and a computing power budget and upgrade module.

[0354] Figure 2 This is a structural diagram of a system provided by the present invention that uses computing power to repair token anomalies in an intelligent computing cloud platform.

[0355] The Token stream access module is used to receive the content, logit distribution, sampling parameters, probability, attention summary, hidden state summary, and decoding timestamp output by the large model decoder per token or per token block.

[0356] Multidimensional state maintenance module: used to maintain the current output structure stack, schema progress, code syntax state, language style state, entity number table, duplicate statistics window, security policy state, token computing power ledger, KV cache page table and candidate rollback point set.

[0357] Structure detector: Used to detect whether illegal transfers occur in structural states such as JSON, XML, Markdown tables, brackets, quotation marks, list numbers, chapter levels, and claim reference relationships.

[0358] Schema and tool call detector: Used to detect whether tool name, function parameters, field types, required fields, enumeration values, array length, object level, parameter unit, parameter value range, and partial output can still be completed into a valid schema.

[0359] Semantic constraint detector: Used to generate results based on user input, system prompts, business rules, retrieval evidence, and preceding text, and maintain consistency constraints such as entities, numbers, units, time, location, variable names, table names, field names, and claim numbers.

[0360] Language style detector: Used to detect whether the output language, tone, professionalism, domain terminology, paragraph style and target text style deviate from the preset requirements, such as a Chinese answer suddenly becoming English, or a patent document suddenly becoming colloquial.

[0361] Repetition and Degradation Detector: Used to detect repeated n-grams, cyclic sentences, entropy collapse, long-term high-similarity token fragments, meaningless padding, and the risk of model output falling into repetitive patterns.

[0362] Code and SQL detector: Used to detect code brackets, indentation, variable definitions and usage, function signatures, SQL table fields, JOIN relationships, WHERE conditions, and statement executability boundaries.

[0363] Security detectors are used to detect whether the output deviates from security policies, compliance requirements, permission boundaries, and rules for handling sensitive information.

[0364] Anomaly localization module: used to determine the origin or fragment of anomaly token based on deterministic illegal transfer, sudden increase in risk score, detector contribution, context backtracking, and local repairability.

[0365] Rollback point selection module: used to select the optimal rollback location from candidate boundaries, taking into account structural integrity, semantic loss, rollback distance, KV cache availability, computing power saving, user streaming experience and historical failure count.

[0366] KV Checkpoint and Recovery Module: Used to save KVcache checkpoints by Token block, semantic boundary or structural boundary during generation, and to restore the KV cache, sampling state, detector state and output buffer corresponding to the security prefix during rollback.

[0367] Local Regeneration Path Module: Used to select paths such as low-temperature resampling, strong constraint decoding, local beamsearch, schema compensation template, tool verification, validation model rearrangement, local rewriting of high-capability models, or overall upgrade based on the anomaly type.

[0368] Computing power budget and upgrade module: used to record the cost of invalid tokens, partial recovery cost, candidate verification cost and overall retry cost, and to upgrade computing power or processing flow when multiple anomalies occur in the same area.

[0369] In other words, the Token stream access module can execute step S1 in this invention to obtain the Token currently output by the large model deployed in the intelligent computing cloud platform; the multi-dimensional state maintenance module, structure detector, schema and tool call detector, semantic constraint detector, language style detector, repetition and degradation detector, code and SQL detector, security detector, and anomaly localization module can execute step S2 in this invention to detect multi-dimensional anomalies in the currently output Token based on the multi-dimensional state information of the currently output Token using computing power, and when an anomaly is detected, to backtrack the currently output Token to determine the starting Token of the anomaly from the Token sequence.

[0370] The rollback point selection module, KV checkpoint and recovery module can execute step S3 in this invention to determine the target rollback point from the candidate rollback points corresponding to the Token sequence based on the abnormal starting point Token, and determine the KV checkpoint associated with the target rollback point; the local regeneration path module can execute step S4 in this invention to regenerate the first Token subsequence from the Token corresponding to the associated KV checkpoint to the currently output Token, and perform abnormal repair on the Token sequence based on the first Token subsequence.

[0371] The computing power budgeting and upgrade module can execute step S5 in this invention to determine computing power saving information for regenerating the first token subsequence based on KV checkpoints.

[0372] It should be noted that, within the intelligent computing cloud platform, the system for token anomaly repair implemented through computing power can be deployed as a large model inference gateway, model service sidecar, decoder plugin, or agent runtime middleware. For self-hosted models, the token stream monitor can directly access the decoding loop, performing detection immediately after each sampling step, and can directly control the KV cache page table, sampling parameters, and local branches. For third-party models or black-box application programming interfaces (APIs), the system can perform text-level detection at the streaming output layer and approximate partial recovery through a process of "stopping generation—truncating security prefixes—appending repair hints—partial continuation writing."

[0373] In the GPU inference framework, key-value checkpoints can collaborate with paged key-value management, prefix caching, sequential batch processing, and speculative decoding. Instead of copying the entire key-value tensor, the system can record page table snapshots at structural boundaries or every M tokens; when a rollback occurs, only the page corresponding to the abnormal suffix needs to be released, and the safe prefix page is reused. For multiple local candidates, the system can share the same prefix page table and use copy-on-write to create suffix pages for different candidates, thereby reducing the memory overhead of local bundle search.

[0374] Among them, local candidates and different candidates refer to the second Token subsequence in this invention.

[0375] To avoid the detector becoming a bottleneck, the system can employ a layered detection approach. Low-cost detectors such as structural state machines, schema local verification, and n-gram duplication statistics are executed synchronously on the CPU or inference thread; medium-cost detectors such as language style, semantic conflict, and security policies can be executed asynchronously by boundary or window; and high-cost verification models are triggered only when the risk increases, suffix selection occurs, or multiple failures occur. Through this architecture, the system can achieve both token-level fast response and control over detection computational overhead.

[0376] For example, the intelligent computing cloud platform can perform token-level rollback and regeneration of JSON structured output through a system that uses computing power to repair token anomalies.

[0377] In an enterprise data extraction scenario, a large model needs to extract the contract number, party A, party B, amount, currency, signing date, and payment node from the contract text uploaded by the user, and output an object that conforms to a preset JSON Schema.

[0378] The system can activate the structure detector and schema detector as soon as the model outputs the first left curly brace. For each generated token, the structure detector updates the bracket stack and string state; the schema detector updates the current field path, field type, and required field set. For example, if the model has already generated the following fragment "Amount field name, colon, string start", the system knows that the current field should be a numeric type. If the model subsequently generates Chinese units and closes the string, the schema detector determines that the field cannot be completed as a numeric type, thus triggering an exception at the field value closure.

[0379] The system determines the origin of the anomaly based on the anomaly localization module, starting with the token value of the specified field, rather than the beginning of the entire JSON object. The nearest safe point in the candidate rollback point set is either before the colon in the amount field or at the closing position of the previous key-value pair. If the contract number, Party A, and Party B fields preceding the amount field have passed structural and semantic checks, the system retains these secure prefixes and their KV cache, rolling back only the amount field suffix. Subsequently, the local regeneration path module enables strong schema constraints and unit specification hints, requiring the amount field to output a numerical value, the currency field to output an enumerated value, and the unit description to be placed in another field. After local generation is complete, the system uses a schema detector and an entity number consistency detector to verify whether the amount value matches the original contract text. If it passes, it continues generating subsequent fields.

[0380] In this embodiment, if the complete JSON length is 800 tokens, and the exception occurs at token 520, rolling back to token 500, and local regeneration and verification consuming 80 equivalent tokens, then compared to the overall retry of approximately 800 tokens, the computing power saving ratio can be calculated by substituting into the above formula (24): 90% (26) For example, the intelligent computing cloud platform can also perform partial recovery of abnormal Agent tool call parameters by using computing power to repair Token anomalies.

[0381] In the Agent's business process, the large model needs to call the "Query Inventory" tool. Tool parameters include product ID, warehouse code, quantity, and delivery date. The system requires that the output be a tool call object, and the product ID must exist in the product master data table, the warehouse code must be within the current user's permissions, and the delivery date must not be earlier than the current date.

[0382] The tool invokes a detector to perform local validation immediately upon closing a parameter field. When the model generates and closes the Product ID field, the system calls a lightweight master data index to confirm the Product ID's validity and locks it to the entity table. When the model generates the Warehouse Code field, the permission validator detects that the code does not belong to the current user's scope and outputs a deterministic business anomaly. The anomaly localization module locates the anomaly starting at the Warehouse Code value (Token), and the security rollback point selection module rolls back to before the start of that key-value pair, retaining the function name, Product ID, and quantity parameters.

[0383] During partial regeneration, the system adds the list of optional warehouse codes or alias mapping results to the local context, enables enumeration masking or candidate rearrangement, and only allows the model to select from valid warehouse codes. If the partial regeneration passes the permission verification, the delivery date field is generated. If the same warehouse field fails three times consecutively, the computing power upgrade module can switch to deterministic tool filling, that is, the rule engine directly fills in the default warehouse within the permission scope, or the tool can be called and submitted for manual confirmation.

[0384] This example illustrates that the system does not simply require the model to "recall the tool," but rather rolls back and repairs at the parameter-level boundaries, thereby avoiding the repeated rewriting of the entire tool call object and preceding inference chain.

[0385] For example, the intelligent computing cloud platform can also perform statement boundary rollback in code generation by using computing power to repair token anomalies.

[0386] In the code generation scenario, the large model generates Python functions based on user requirements. The model has correctly generated the function signature, parameter validation, and data preprocessing in the first half, but it misses a parenthesis when generating the loop body and continues to generate multiple lines of code.

[0387] The code detector maintains a bracket stack, indentation stack, variable definition table, and statement boundaries at the end of each line. When the bracket stack at the end of a line indicates an error that cannot be reasonably completed by subsequent indentation, or the interpreter's incremental parser returns a clear syntax error, the system triggers an exception. The exception location module locates the exception starting point Token in the expression of this line, and the safe rollback point selection module selects the previous complete statement as the rollback point, rather than rolling back the entire function.

[0388] The local regeneration path module enables code syntax constraints, variable name locking, and low-temperature sampling. The system passes the defined variable table to the local regeneration context, prohibiting the generation of undefined variables or changes to function signatures. For multiple generated suffixes, the system can perform lightweight compilation and unit test snippets separately, selecting the suffix that passes syntax checks and best matches the original requirements. Since the key-value cache for function signatures and parameter validation is preserved, local repair only requires regeneration around the loop body.

[0389] This example illustrates that the system is applicable not only to JSON, but also to generation tasks such as code and SQL that require both structural and semantic constraints.

[0390] For example, the intelligent computing cloud platform can also correct field names and table relationships in SQL generation by using computing power to repair token anomalies.

[0391] In a Natural Language to SQL (NLP) scenario, a user queries, "Calculate the order amount for each customer over the past 30 days and output the top 10 orders in descending order of amount." The database contains tables named `orders`, `customers`, and `order_items`. When the model generates the SQL, if a non-existent field "customer_name_text" is incorrectly generated in the `SELECT` clause, subsequent `GROUP BY` and `ORDER BY` clauses may continue to reference the incorrect field.

[0392] The SQL detector maintains database schema, table aliases, field sets, and JOIN relationships. When the model completes a field name token fragment and encounters a comma or space boundary, the detector checks if the field exists in the currently visible table set. If the field does not exist, but a similar field "customer_name" exists, the semantic constraint detector outputs a high-risk error, and the schema detector outputs a field non-executable exception. The system locates the origin of the exception to the erroneous field name starting token, rolls back to before the field name, and adds valid field suffixes and table alias constraints during local regeneration.

[0393] If the generated suffix passes the field existence check, the system continues to monitor the JOIN condition. If the date field in the subsequent WHERE condition has an incorrect unit, the system rolls back only that condition segment. This breaks down the SQL generation process into multiple verifiable local segments, reducing the final execution failure rate.

[0394] For example, the intelligent computing cloud platform can also perform number and parameter verification in multimodal storyboards by using computing power to repair token anomalies.

[0395] In scenarios involving the generation of text-based videos or advertising creatives, the large model needs to output storyboard scripts. Each storyboard includes a shot number, shot type, duration, scene description, narration, subtitles, and generation parameters. If the model jumps to the 7th shot after the 5th shot, or changes the duration unit from "seconds" to "frames," the subsequent video generation system may not be able to parse it correctly.

[0396] The structure detector maintains shot numbers and field order, the entity number detector maintains duration units and total duration constraints, and the schema detector maintains mandatory fields for each shot. If a shot number jumps, the system rolls back to the end position of the previous shot; if the duration field unit is incorrect, the system only rolls back the value of that field; if the language in the scene description changes from Chinese to English, the language style detector rolls back to the beginning of the description sentence and regenerates it. The suffix after partial regeneration can also be checked by the video parameter validator to see if the total duration meets the user's requirements.

[0397] This example illustrates the system's suitability for a multimodal generation pipeline, particularly for complex tasks that combine text generation, image generation, video generation, and tool orchestration on a computing platform.

[0398] For example, the intelligent computing cloud platform can also optimize computing power in cloud-based batch inference by using computing power to repair token anomalies.

[0399] In the intelligent computing cloud platform, a single customer calls upon a large model daily to generate millions of structured engineering orders. The inference gateway maintains a token-level computing power ledger and key-value checkpoints for each request.

[0400] When a request encounters an anomaly, the system prioritizes partial recovery. The platform aggregates the token savings, partial repair success rate, and detector overhead for each type of anomaly, dynamically adjusting the checkpoint frequency and detector weights. For example, for JSON extraction tasks, where structural anomalies are frequent and partial recovery success rates are high, the platform can increase the density of structural boundary checkpoints; for tasks with frequent semantic conflicts, the platform can increase the frequency of entity number detector calls; and for low-value short text tasks, the platform can reduce detection granularity to save detection costs.

[0401] This example illustrates that the system is not only a one-time generation error correction method, but can also form a cloud platform-level token cost governance capability. The platform can provide users with value-added capabilities such as "reliable structured output service", "partial token recovery service", "generation quality audit service", and "invalid token saving report".

[0402] Please see Figure 3 , Figure 3 This is a structural diagram of a device for repairing token anomalies through computing power provided by the present invention, such as... Figure 3 As shown, the device 300 for repairing token anomalies through computing power in an intelligent computing cloud platform includes: The acquisition module 301 is used to acquire the token currently output by the large model deployed in the intelligent computing cloud platform, and update the token sequence based on the currently output token; The anomaly detection module 302 is used to detect multi-dimensional anomalies in the currently output Token based on the multi-dimensional state information of the currently output Token using computing power, and when an anomaly is detected, to backtrack the currently output Token to determine the starting Token of the anomaly from the Token sequence. The rollback point determination module 303 is used to determine the target rollback point from the candidate rollback points corresponding to the Token sequence based on the abnormal starting point Token, and to determine the KV checkpoint associated with the target rollback point; The Token regeneration module 304 is used to regenerate a first Token subsequence from the Token corresponding to the associated KV checkpoint to the currently output Token, and to perform anomaly repair on the Token sequence based on the first Token subsequence.

[0403] In some embodiments, the Token regeneration module 304 is further configured to: Determine the anomaly type of the starting token, and based on the anomaly type, determine the target path for token regeneration from the candidate paths for token regeneration; wherein, the anomaly type includes structural anomaly, schema anomaly, semantic anomaly, language drift, repeated loop, code error, and security anomaly; Based on the target path and the associated KV checkpoint, the first Token subsequence is regenerated.

[0404] In some embodiments, the Token regeneration module 304 is further configured to: Determine the computing power cost of the intelligent computing cloud platform, as well as the historical number of failed token regeneration attempts; Determine multiple candidate paths corresponding to the anomaly type; A path selection function is constructed based on the anomaly type, the computing cost, and the historical failure count. The target path is determined from the plurality of candidate paths based on the path selection function.

[0405] In some embodiments, the Token regeneration module 304 is further configured to: Based on the target path, a second token subsequence is regenerated from the token corresponding to the associated KV checkpoint to the token currently output. A risk assessment of abnormal states is performed on the second Token subsequence to obtain a risk assessment value; In response to the fact that the risk assessment value of the second Token subsequence is less than the risk threshold, the second Token subsequence with the smallest risk assessment value is selected as the first Token subsequence.

[0406] In some embodiments, the Token regeneration module 304 is further configured to: In response to the risk assessment value of the second token subsequence being greater than or equal to the risk threshold, the second token subsequence is subjected to risk verification, and the first token subsequence is determined from the second token subsequence that passes the risk verification.

[0407] In some embodiments, the Token regeneration module 304 is further configured to: Obtain the second Token subsequence that fails the risk verification, and use it as the third Token subsequence that fails to regenerate the token; The number of failures, remaining token budget, remaining computing power budget, and risk level of the third token subsequence regeneration failure are statistically analyzed. Based on the number of failures, remaining token budget, remaining computing power budget, and risk level, a computing power upgrade function is constructed. In response to the computing power upgrade function instruction, a preset computing power upgrade operation is performed.

[0408] In some embodiments, the Token regeneration module 304 is further configured to: Determine the position information of the token corresponding to the associated KV checkpoint in the token sequence; Based on the location information, the token sequence is truncated, retaining the fourth token subsequence before the location information and deleting the fifth token subsequence starting from the location information; The first Token subsequence and the fourth Token subsequence are combined in sequence to obtain the repaired Token sequence.

[0409] In some embodiments, the multi-dimensional anomaly detection includes at least two of the following: anomaly risk detection, structure detection, schema and tool call detection, semantic constraint and entity consistency detection, language style drift detection, and duplication and degradation detection.

[0410] In some embodiments, the anomaly detection module 302 is further configured to: Obtain the status information corresponding to at least one abnormal status detection item from the multi-dimensional status information; Based on the state information, perform anomaly detection on the currently output Token using the anomaly detection item.

[0411] In some embodiments, the anomaly detection module 302 is further configured to: In response to detecting an anomaly in the currently output Token, an anomaly classification dimension for the currently output Token is determined; wherein, the anomaly classification dimension includes deterministic anomalies and probabilistic anomalies; Based on the anomaly classification dimension, the anomaly starting point Token is determined from the Token sequence.

[0412] In some embodiments, the anomaly detection module 302 is further configured to: In response to the anomaly classification dimension being the deterministic anomaly, the currently output Token is determined as the anomaly origin Token; In response to the anomaly classification dimension being the probabilistic anomaly, the anomaly contribution score and overall risk increment of each token in the token sequence are calculated; Based on the anomaly contribution score and overall risk increment of each Token, the Token that meets the screening criteria for the anomaly starting point Token is determined as the anomaly starting point Token.

[0413] In some embodiments, the candidate rollback points include: structural closure points, semantic boundaries, code statement boundaries, tool parameter boundaries, low-risk token boundaries, and key-value checkpoints.

[0414] In some embodiments, the rollback point determination module 303 is further configured to: Based on the preset scoring rules, each candidate rollback point is scored to obtain the score value corresponding to each candidate rollback point; The candidate rollback point with the highest score is determined as the target rollback point.

[0415] In some embodiments, the rollback point determination module 303 is further configured to: Determine the scoring characteristics for each candidate rollback point; Each candidate rollback point is scored based on the defined scoring function and the scoring features.

[0416] In some embodiments, the device 300 for the intelligent computing cloud platform to perform token anomaly repair via computing power further includes: The determination module is used to determine the computing power saving information for regenerating the first Token subsequence based on the KV checkpoint.

[0417] In some embodiments, the determining module is further configured to: Determine the token length, risk verification cost, and local recovery cost for regenerating the first token subsequence based on the KV checkpoint; Based on the token length, risk verification cost, and local recovery cost, the computing power saving ratio and global saving information are determined and used as the computing power saving information.

[0418] The device for repairing token anomalies through computing power provided by the present invention is capable of implementing the various processes of the above-mentioned method for repairing token anomalies through computing power in the intelligent computing cloud platform. The technical features are one-to-one and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0419] It should be noted that the device for repairing token anomalies through computing power in the intelligent computing cloud platform of this invention can be a device, or it can be a component, integrated circuit, or chip in an electronic device.

[0420] The present invention also provides an electronic device, see below. Figure 4 , Figure 4 This is a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. The electronic device includes a memory 401, a processor 402, and a program or instructions stored in the memory 401 that run on the memory. When the program or instructions are executed by the processor 402, they can achieve the following: Figure 1 The corresponding intelligent computing cloud platform achieves the same beneficial effect through computing power in any step of the method embodiment for Token anomaly repair, and will not be elaborated here.

[0421] The processor 402 can be a CPU, ASIC, FPGA or GPU.

[0422] Those skilled in the art will understand that all or part of the steps of the above-described method embodiment for implementing token anomaly repair through computing power on the intelligent computing cloud platform can be accomplished by hardware related to program instructions, and the program can be stored in a readable medium.

[0423] The present invention also provides a readable storage medium on which a computer program is stored, and which, when executed by a processor, can perform the above-described functions. Figure 1 The corresponding intelligent computing cloud platform can achieve any step in the method embodiment for token anomaly repair through computing power, and can achieve the same technical effect. To avoid repetition, it will not be described again here. The storage medium mentioned is such as read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0424] The present invention also provides a computer program product, including computer instructions that, when executed by a processor, implement the above-described... Figure 1 The corresponding intelligent computing cloud platform implements each process of the token anomaly repair method embodiment through computing power and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0425] The terms "first," "second," etc., used in this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to these processes, methods, products, or apparatuses. Additionally, the use of "and / or" in this application indicates at least one of the connected objects, such as A and / or B and / or C, representing seven possibilities: A alone, B alone, C alone, both A and B present, both B and C present, both A and C present, and A, B, and C present.

[0426] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0427] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, air conditioner, or second terminal device, etc.) to execute the methods of the various embodiments of this application.

[0428] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. A method for restoring token anomalies using computing power on an intelligent computing cloud platform, characterized in that, The method includes: Step S1: Obtain the token currently output by the large model deployed in the intelligent computing cloud platform, and update the token sequence based on the currently output token; Step S2: Using computing power, based on the multi-dimensional state information of the currently output Token, perform multi-dimensional abnormal state detection on the currently output Token, and when an abnormality is detected, perform abnormal backtracking on the currently output Token to determine the abnormal starting Token from the Token sequence. Step S3: Based on the abnormal starting point Token, determine the target rollback point from the candidate rollback points corresponding to the Token sequence, and determine the KV checkpoint associated with the target rollback point; Step S4: Regenerate the first Token subsequence from the Token corresponding to the associated KV checkpoint to the currently output Token, and perform anomaly repair on the Token sequence based on the first Token subsequence.

2. The method according to claim 1, characterized in that, Step S4 includes: Step S4.1: Determine the exception type of the exception starting point Token, and based on the exception type, determine the target path for Token regeneration from the candidate paths for Token regeneration; wherein, the exception type includes structural exception, schema exception, semantic exception, language drift, repeated loop, code error and security exception; Step S4.2: Based on the target path and the associated KV checkpoints, regenerate the first Token subsequence.

3. The method according to claim 2, characterized in that, Step S4.1 includes: Step S4.1.1: Determine the computing power cost of the intelligent computing cloud platform and the historical number of failed token regeneration attempts; Step S4.1.2: Determine multiple candidate paths corresponding to the anomaly type; Step S4.1.3: Construct a path selection function based on the anomaly type, the computing power cost, and the historical failure count; Step S4.1.4: Based on the path selection function, determine the target path from the plurality of candidate paths.

4. The method according to claim 2, characterized in that, Step S4.2 includes: Step S4.2.1: Based on the target path, regenerate the second token subsequence from the token corresponding to the associated KV checkpoint to the currently output token; Step S4.2.2: Perform an abnormal state risk assessment on the second Token subsequence to obtain a risk assessment value; Step S4.2.3: In response to the fact that the risk assessment value of the second Token subsequence is less than the risk threshold, the second Token subsequence with the smallest risk assessment value is selected as the first Token subsequence.

5. The method according to claim 4, characterized in that, Step S4.2.2 further includes: Step S4.2.2.1: In response to the risk assessment value of the second Token subsequence being greater than or equal to the risk threshold, perform risk verification on the second Token subsequence, and determine the first Token subsequence from the second Token subsequence that has passed the risk verification.

6. The method according to claim 5, characterized in that, Step S4.2.2.1 includes: Step S4.2.2.1.1: Obtain the second Token subsequence that failed the risk verification, and use it as the third Token subsequence that failed to regenerate the token; Step S4.2.2.1.2: Calculate the number of failures, remaining token budget, remaining computing power budget, and risk level of the third token subsequence regeneration failure; Step S4.2.2.1.3: Based on the number of failures, remaining token budget, remaining computing power budget, and risk level, construct the computing power upgrade function; Step S4.2.2.1.4: In response to the computing power upgrade function instruction, perform computing power upgrade and execute the preset computing power upgrade operation.

7. The method according to claim 1, characterized in that, Step S4 further includes: Step S4.1': Determine the position information of the Token corresponding to the associated KV checkpoint in the Token sequence; Step S4.2': Based on the location information, truncate the Token sequence, retain the fourth Token subsequence before the location information, and delete the fifth Token subsequence starting from the location information; Step S4.3': Combine the first Token subsequence and the fourth Token subsequence in sequence to obtain the repaired Token sequence.

8. The method according to any one of claims 1-7, characterized in that, The multi-dimensional anomaly detection includes at least two of the following: anomaly risk detection, structure detection, schema and tool call detection, semantic constraint and entity consistency detection, language style drift detection, and duplication and degradation detection.

9. The method according to claim 8, characterized in that, Step S2 includes: Step S2.1: Obtain the status information corresponding to at least one abnormal status detection item from the multi-dimensional status information; Step S2.2: Based on the state information, perform anomaly detection on the currently output Token for the abnormal state detection item.

10. The method according to claim 9, characterized in that, Step S2.2 includes: Step S2.2.1: In response to detecting an anomaly in the currently output Token, determine the anomaly classification dimension of the currently output Token; wherein, the anomaly classification dimension includes deterministic anomalies and probabilistic anomalies; Step S2.2.2: Based on the anomaly classification dimension, determine the anomaly starting point Token from the Token sequence.

11. The method according to claim 10, characterized in that, Step S2.2.2 includes: Step S2.2.2.1: In response to the anomaly classification dimension being the deterministic anomaly, determine the currently output Token as the anomaly origin Token; Step S2.2.2.2: In response to the anomaly classification dimension being the probabilistic anomaly, calculate the anomaly contribution score and comprehensive risk increment for each Token in the Token sequence; Step S2.2.2.3: Based on the abnormal contribution score and comprehensive risk increment of each Token, determine the Token that meets the abnormal starting point Token screening condition, and use it as the abnormal starting point Token.

12. The method according to any one of claims 1-7, characterized in that, The candidate rollback points include: structural closure points, semantic boundaries, code statement boundaries, tool parameter boundaries, low-risk token boundaries, and key-value checkpoints.

13. The method according to claim 12, characterized in that, Step S3 includes: Step S3.1: Based on the preset scoring rules, score each candidate rollback point to obtain the score value corresponding to each candidate rollback point; Step S3.2: Determine the candidate rollback point with the largest score value as the target rollback point.

14. The method according to claim 13, characterized in that, Step S3.1 includes: Step S3.1.1: Determine the scoring characteristics of each candidate rollback point; Step S3.1.2: Based on the defined scoring function and the scoring features, score each candidate rollback point.

15. The method according to any one of claims 1-7, characterized in that, The method further includes: Step S5: Determine the computing power saving information for regenerating the first Token subsequence based on the KV checkpoint.

16. The method according to claim 15, characterized in that, Step S5 includes: Step S5.1: Determine the token length, risk verification cost, and local recovery cost for regenerating the first token subsequence based on the KV checkpoint; Step S5.2: Based on the Token length, risk verification cost, and local recovery cost, determine the computing power saving ratio and global saving information, which will be used as the computing power saving information.

17. A device for repairing token anomalies using computing power on an intelligent computing cloud platform, characterized in that, The device includes: The acquisition module is used to acquire the token currently output by the large model deployed in the intelligent computing cloud platform, and update the token sequence based on the currently output token; The anomaly detection module is used to detect multi-dimensional anomalies in the currently output Token based on the multi-dimensional state information of the currently output Token using computing power, and when an anomaly is detected, to backtrack the currently output Token to determine the starting Token of the anomaly from the Token sequence. The rollback point determination module is used to determine the target rollback point from the candidate rollback points corresponding to the Token sequence based on the abnormal starting point Token, and to determine the KV checkpoint associated with the target rollback point; The Token regeneration module is used to regenerate a first Token subsequence from the Token corresponding to the associated KV checkpoint to the currently output Token, and to perform anomaly repair on the Token sequence based on the first Token subsequence.

18. An electronic device, characterized in that, include: A processor, a memory, and a program stored in the memory and executable on the processor, wherein when the program is executed by the processor, the program implements the steps of the method for Token anomaly repair by computing power of the intelligent computing cloud platform as described in any one of claims 1 to 16.

19. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the method for Token anomaly repair by computing power in the intelligent computing cloud platform as described in any one of claims 1 to 16.

20. A computer program product, characterized in that, The method includes computer instructions that, when executed by a processor, implement the steps of the method for restoring token anomalies through computing power using the intelligent computing cloud platform as described in any one of claims 1 to 16.

Images