Reinforcement learning method and system based on state compression and unlabeled rewards

Abstract

The present application discloses a reinforcement learning method, system, and electronic device based on state compression and unlabeled rewards. The method comprises: receiving user prompts output by a model, converting the user prompts into a structured state vector, wherein the state vector includes a task goal, a function call sequence, and an environmental feedback result; mapping the function call sequence into a complex plane vector, and automatically generating an unlabeled reward value based on the complex plane vector; based on the unlabeled reward value, using a low-rank adaptive network in a model training engine to fine-tune the parameters of the model, and performing batch training using a token gradient optimization strategy; identifying and extracting function call instructions from the text generated by the model, converting the function call instructions into tool API calls, and injecting the execution results of the tool API calls into the next round of model input in real time. The solution of the present application improves decision-making quality, optimizes resource efficiency, reduces implementation costs, and achieves scalable application scenarios.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a reinforcement learning method, system, and electronic device based on state compression and unlabeled rewards. Background Art

[0002] In the fields of deep learning and natural language processing, the construction of large language models has always been a hot topic of research. Large models can achieve more accurate and intelligent task processing by learning patterns and knowledge from large amounts of data.

[0003] While traditional reinforcement learning methods based on proximal policy optimization (PPO) can optimize short sequence generation, their token-level reward mechanism causes the model to become trapped in local optimization when applied to O1-level LLMs (large language models with long-context reasoning capabilities), failing to globally coordinate multi-step function call decisions. Experiments show that human-level agreement is less than 30%. While the Transformer Reinforcement Learning (TRL) framework integrates model and reinforcement learning, it requires the computation of gradients over long sequences, resulting in GPU resource consumption exceeding double that of pre-training. Furthermore, the memory requirements for aligning a 30-billion-parameter model far exceed the limits of conventional hardware. Existing reward models rely on manually labeled "environment feedback-function call" paired data, which is expensive and lacks coverage for the long-tail combinations found in dynamic scenarios. Furthermore, non-O1-level models (with context lengths less than 4k tokens) lack reasoning capabilities and are unable to generate coherent multi-step function call sequences, resulting in a correct call probability of less than 10% in complex tasks. These issues severely hinder the efficient deployment and scalability of autonomous AI. Summary of the Invention

[0004] In order to solve the problems in the prior art such as token-by-token optimization causing the model to focus too much on local details and ignore global logic, high video memory usage, low computational efficiency, high manual labeling costs and insufficient scene coverage, and insufficient context length of non-O1 level models leading to logical breaks, the present application provides the following technical solutions.

[0005] The first aspect of the present application provides a reinforcement learning method based on state compression and unlabeled rewards, comprising:

[0006] receiving user prompts output by the model, and converting the user prompts into a structured state vector, wherein the state vector includes a task goal, a function call sequence, and an environment feedback result;

[0007] Mapping the function call sequence into a complex plane vector, automatically generating an unlabeled reward value based on the complex plane vector, and back-propagating the unlabeled reward value to all tokens in the function call sequence;

[0008] Based on the unlabeled reward value, a low-rank adaptive network is used in the model training engine to fine-tune the parameters of the model, and batch training is performed using a token gradient optimization strategy;

[0009] Identify and extract function call instructions from the text generated by the model, convert the function call instructions into tool API calls, and inject the execution results of the tool API calls into the next round of model input in real time.

[0010] Preferably, converting the user prompt into a structured state vector further comprises:

[0011] Parse the function call, extract compliant JSON instructions from the token sequence, filter redundant tokens, automatically delete intermediate tokens irrelevant to the function call, and retain only the core operation nodes.

[0012] Preferably, mapping the function call sequence into a complex plane vector, automatically generating an unlabeled reward value based on the complex plane vector, and back-propagating the unlabeled reward value to all tokens in the function call sequence further comprises:

[0013] The current function call sequence is formed into a trajectory τ, and the trajectory τ is mapped to a vector ψ(τ) on the complex plane based on the similarity between the trajectory τ and the positive sample library and the negative sample library; the positive sample library pre-stores the trajectories that have passed the expert review, and the negative sample library pre-stores the trajectories that have not passed the expert review;

[0014] Calculate the reward value r based on the vector ψ(τ) on the complex plane:

[0015] r=1-|ψ(τ)|, where |ψ(τ)| represents the modulus of the vector ψ(τ).

[0016] Preferably, the step of fine-tuning the parameters of the model using a low-rank adaptive network in the model training engine further includes:

[0017] Add low-rank decomposition matrices A and B to the query, key, and value projection layers in the model's attention mechanism, and convert the complete weight matrix Decomposed into the product of two low-rank matrices W+ΔW=W+BA, where the matrix matrix And the rank r<<min(d,k); d is the number of rows of the complete weight matrix W, k is the number of columns of the complete weight matrix W. During the training process, the original weight W remains frozen and only the low-rank matrices A and B are updated.

[0018] Preferably, the step of identifying and extracting function call instructions from the text generated by the model, converting the function call instructions into tool API calls, and injecting the execution results of the tool API calls into the next round of model input in real time further comprises:

[0019] Use regular expressions combined with JSON parsing to extract standardized function call instructions from the text generated by the model;

[0020] Converting standardized function call instructions into calls to tool APIs, including network diagnostic tools, computing resource monitoring tools, data transfer tools, and file operation tools;

[0021] Abstract the execution results of tool API calls into Boolean feedback, indicating whether the operation is successful or the test result is normal;

[0022] The Boolean feedback results are integrated into the system state and prepare for the next round of model input, recording the executed function calls and their feedback, and controlling the context length through a sliding window mechanism.

[0023] The second aspect of the present application provides a reinforcement learning system based on state compression and unlabeled rewards, comprising:

[0024] A state compression module is configured to receive user prompts output by the model and convert the user prompts into a structured state vector, wherein the state vector includes a task goal, a function call sequence, and an environment feedback result;

[0025] an unlabeled reward module, configured to map the function call sequence into a complex plane vector, automatically generate an unlabeled reward value based on the complex plane vector, and back-propagate the unlabeled reward value to all tokens in the function call sequence;

[0026] A model training engine, configured to fine-tune parameters of a pre-trained language model using a low-rank adaptive network based on the unlabeled reward value, and perform batch training using a token gradient optimization strategy;

[0027] The environment interaction module is used to identify and extract function call instructions from the text generated by the model, convert the function call instructions into tool API calls, and inject the execution results of the tool API calls into the next round of model input in real time.

[0028] A third aspect of the present application provides an electronic device, comprising a processor and a memory, wherein the memory stores a plurality of instructions, and the processor is configured to read the instructions and execute the method provided in the first aspect above.

[0029] A fourth aspect of the present application provides a computer-readable storage medium, which stores a plurality of instructions. The plurality of instructions can be read by a processor and execute the method provided in the first aspect above.

[0030] The beneficial effects of this application are as follows: through function call-level state compression and gradient broadcast mechanisms, the model coordinates multi-step decisions from a global perspective, enhancing the match between intelligent agent behavior and human expectations. The model is able to generate coherent multi-step function call sequences in complex tasks. The unlabeled reward model enables the system to maintain efficient decision-making capabilities when faced with unseen abnormal scenarios. The lightweight training architecture reduces resource requirements and breaks through the hardware limitations of large model deployment. The gradient calculation optimization strategy avoids backpropagation of all tokens in long sequences, shortening the model iteration cycle. The state compression mechanism converts fine-grained token-level state into high-level function call-level state. Combined with the dynamic context management strategy, it enables the system to process longer interaction sequences under limited memory conditions. The quantum dynamics reward model automatically constructs a library of positive and negative samples and dynamically calculates reward values ​​by analyzing the similarity of historical function call trajectories, eliminating the reliance on manually labeled data. Automatic adaptation to environmental changes reduces manual intervention and maintenance requirements, improving diagnostic accuracy, task automation rate, and automatic repair rate. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 1 is a component diagram of the reinforcement learning system described in this application.

[0032] Figure 2 Flowchart of the reinforcement learning method based on state compression and unlabeled rewards according to the present application. DETAILED DESCRIPTION

[0033] In order to better understand the above technical solution, the above technical solution will be described in detail below with reference to the accompanying drawings and specific implementation methods.

[0034] The method provided in the present application can be implemented in the following terminal environment, which may include one or more of the following components: a processor, a memory and a display screen. Among them, at least one instruction is stored in the memory, and the instruction is loaded and executed by the processor to implement the reinforcement learning method based on state compression and unlabeled rewards described in the following embodiments. The processor may include one or more processing cores. The processor uses various interfaces and lines to connect the various parts within the entire terminal, and executes various functions of the terminal and processes data by running or executing instructions, programs, code sets or instruction sets stored in the memory, and calling data stored in the memory. The memory may include random access memory (RAM) or read-only memory (ROM). The memory can be used to store instructions, programs, codes, code sets or instructions. The display screen is used to display the user interface of each application.

[0035] In addition, those skilled in the art will appreciate that the structure of the terminal described above does not limit the terminal. The terminal may include more or fewer components, or a combination of certain components, or a different arrangement of components. For example, the terminal may also include a radio frequency circuit, an input unit, a sensor, an audio circuit, a power supply, and other components, which will not be described in detail here.

[0036] Example 1

[0037] In response to the shortcomings of existing technologies, this application reconstructs the state space of reinforcement learning and compresses the original fine-grained token-level state into a higher-level function call-level state. The model uses a complete function call (an instruction sequence consisting of approximately 1,000 tokens) as the basic optimization unit, and evenly propagates the overall reward to related tokens through a gradient broadcast mechanism, allowing the model to coordinate multi-step decisions from a global perspective. The model no longer focuses on a single parameter describing a token, but instead focuses on generating a complete and compliant function call sequence, thereby significantly improving human consistency.

[0038] This application adopts a dual lightweight design. First, by freezing the base model parameters and only updating the low-rank adaptation layer (LoRA), the number of trainable parameters is significantly reduced. Second, when calculating gradients, the policy update is focused only on the initial output token, avoiding backpropagation for all tokens in a long sequence. For example, when training a 30B parameter model, only 25% of the GPU resources of traditional methods are required to start, and the training speed is increased by 50%, breaking through the hardware limitations of large model deployment.

[0039] This application introduces a self-supervised reward mechanism. By analyzing the similarity of historical function call trajectories, a library of positive and negative samples is automatically constructed, and the trajectories are mapped to the complex plane to quantify their consistency with human goals. In its implementation, the model dynamically calculates reward values ​​based on the distance between the current output and the historical trajectories, without the need for any manual labeling. For example, in the task of diagnosing network faults, it automatically determines whether the current function call sequence is close to the expert operation pattern, thereby generating reward values ​​in real time and adapting to dynamic environmental changes.

[0040] This application further adopts a structured state representation and environmental feedback iteration mechanism. By explicitly recording function call history and environmental feedback, a state vector containing task goals, executed operations, and results is constructed, forcing the retention of key decision information.

[0041] like Figure 1 As shown, the system architecture of this application consists of four core modules, which work together to achieve efficient alignment and decision optimization of autonomous AI.

[0042] The state compression module converts the traditional token-level fine-grained state into an abstract function-call-level state. Its input is user prompts (e.g., "Compress the file and send an email") and historical function call records, and its output is a structured state vector (containing the task goal, a list of executed functions, and environmental feedback). The quantum dynamics unlabeled reward module receives the output of the state compression module and maps the current function call sequence to the complex plane by analyzing similar cases in the historical trajectory library. It then dynamically generates reward values ​​based on modulus calculations (e.g., the closer the trajectory is to the expert mode, the higher the reward), enabling self-supervised training without manual labeling. The lightweight training engine integrates a low-rank adaptation policy network and a gradient compression mechanism. It only updates parameters for the initial output tokens while freezing the base model weights, significantly reducing video memory and computing resource consumption. For example, when training a 30B model, GPU requirements are reduced by 75%. The environment interaction module parses the JSON instructions generated by the model and calls external tools (such as file compression interface), injecting the execution results (Boolean feedback) into the next round of model input in real time, forming a "decision-execution-feedback" closed loop.

[0043] After user prompts are abstracted into high-level states by the state compression module, the model generates a sequence of function calls. The reward module assigns rewards based on trajectory consistency, and the training engine optimizes the strategy based on these rewards. The environment interaction module executes actions and returns feedback, updating the state and entering the next iteration. Through its modular design, the system achieves the collaborative goals of optimizing long-sequence decisions, improving resource efficiency, and eliminating annotation dependencies.

[0044] like Figure 1As shown, the embodiment of the present application provides a reinforcement learning method based on state compression and unlabeled rewards, which specifically includes:

[0045] Step 101: Receive user prompts output by the model, and convert the user prompts into a structured state vector, where the state vector includes a task goal, a function call sequence, and an environment feedback result.

[0046] The state compression module converts fine-grained token-level state into high-level function-call-level state to support multi-step decision coherence for autonomous AI. The state compression module takes as input the original user prompt (e.g., "Check current network bandwidth and perform a stress test") and historical interaction data, and outputs a structured state vector containing the task goal, a list of executed functions, and corresponding environmental feedback.

[0047] That is, we can define a state space that transforms the input user prompts into a structured state vector:

[0048] s j =<task objective, called function list, environment feedback history>.

[0049] For example, when the user prompt is "zip all RLlecture prefix files and send them to the professor", it is converted into a structured vector:

[0050] <'Send RLlecture prefix file to professor', function list: Extract, Send, environment feedback history: Extract successful, Send successful>,

[0051] In practice, the form is <'Send RLlecture prefix file to professor', Extract, Success, Send, Success>.

[0052] In the specific implementation, the state compression module completes state abstraction through the following steps:

[0053] First, the function call is parsed and the legal JSON instructions (such as " <networkp2pbw>" indicates detecting point-to-point bandwidth), ignoring the descriptive text in the middle;

[0054] Secondly, redundant tokens are filtered, and intermediate tokens that are not related to function calls (such as parameter explanation text) are automatically deleted, leaving only the core operation nodes.

[0055] Taking the network bandwidth detection task as an example, when the model first generates " <networkp2pbw>After the function is called and executed, the environment feedback "True" indicates that the bandwidth is normal. The state compression module converts the original output sequence (such as "The user needs me to detect the problem of network bandwidth, I called the function <networkp2pbw>After running, it is found that the return result of the function is correct") is compressed into a structured state vector <"Detect network bandwidth", <networkp2pbw>, True>, as the input for the next round of decision making. If other functions need to be called later (such as " <networkallreduce>”), the module further updates the status to <"Detect network bandwidth", <networkp2pbw>,True, <networkallreduce>,False>, ensuring that the model generates coherent decisions based on the complete operation history. This design solves the problem of logical discontinuity in long sequence scenarios by forcibly retaining key decision nodes.

[0056] Only the tokens at the beginning of the generated function call are updated with policy gradients; the latest 10 function call records are retained through a sliding window mechanism to prevent memory overflow.

[0057] In an optional embodiment, for extremely complex multi-step decision-making tasks, a hierarchical reinforcement learning architecture can be used to decompose the decision-making process into high-level strategies (used for task decomposition and planning, determining overall goals and phased tasks) and low-level strategies (implementing specific function calls). This hierarchical architecture can more effectively handle long-term dependencies and improve the system's performance in complex tasks.

[0058] Step 102: Map the function call sequence into a complex plane vector, automatically generate an unlabeled reward value based on the complex plane vector, and back-propagate the unlabeled reward value to all tokens in the function call sequence.

[0059] The reward calculation module generates unlabeled rewards based on the principles of quantum dynamics. It receives the function call sequence output by the state compression module and automatically generates reward values ​​that reflect human preferences through complex plane mapping and distance calculation, without relying on manually labeled data.

[0060] A gradient broadcast mechanism is used to assign a uniform reward value to the complete function call sequence (for example, the reward value is +1 when an email is successfully sent); the reward value is evenly back-propagated to all relevant tokens in the sequence to avoid overfitting of a single token.

[0061] According to a specific embodiment, the reward calculation module may include the following three components:

[0062] The experience trace library is used to store historical function call sequences and their execution results, which is divided into positive sample library (ε T ) and negative sample library (ε F The positive sample library stores high-quality trajectories that have passed expert review (e.g., a function call sequence that successfully resolved a network failure). The negative sample library stores trajectories that have failed expert review, such as those with logical inconsistencies or execution failures. Both positive and negative sample libraries are continuously updated through automatic analysis of environmental feedback.

[0063] The complex plane mapper maps the current function call sequence trajectory τ to a vector ψ(τ) on the complex plane. The mapping rule is as follows: the positive sample trajectory is mapped to the origin (coordinate (0, 0)), and the negative sample trajectory is mapped to the first quadrant of the unit circle. Its coordinates can be expressed as (cosθ, sinθ), where θ is determined by the trajectory characteristics. Based on the similarity between the current function call trajectory τ and the positive and negative samples, it is mapped to a point ψ(τ) = (x, y) on the complex plane.

[0064] A quantum expectation value calculator is used to calculate the reward value r based on a complex plane vector. The specific formula is: r = 1 - |ψ(τ)|, where |ψ(τ)| represents the modulus of the vector ψ(τ).

[0065] When the model generates a new function call sequence, the reward calculation module first calculates the similarity between the sequence and each trajectory in the sample library. In a preferred embodiment, the similarity between trajectories τ1 and τ2 is calculated using the edit distance EditSim(τ1, τ2). The function EditSim calculates the edit distance based on the function call name and parameters. Specifically, let a =<a,b,c> ,b=<x,y,z> It is part of two trajectories. Only when a=x,b=y,c ≤=z,dist+=1,dist starts to accumulate from 0 until the dist after the final termination is the distance EditSim between the two trajectories.

[0066] Based on the calculated similarity, the position of the current trajectory on the complex plane is determined and the final reward value is calculated. For example, in a network fault diagnosis scenario, if the function call sequence generated by the model is " <networkp2pbw> → <networkallreducebw> → <gpucomputefp>If the sequence is highly similar to the successfully diagnosed cases in the sample library, its complex plane mapping position is close to the origin and obtains a higher reward value (close to 1); conversely, if the sequence contains logical contradictions (such as first detecting that the network is normal, but then performing a network repair operation), it is mapped to a position far away from the origin and obtains a lower reward value.

[0067] The advantage of the reward mechanism is that it completely eliminates the reliance on manually labeled data, can automatically evaluate the quality of current decisions based on historical experience, and continuously improves itself as the system runs and adapts to the dynamically changing environment.

[0068] In an optional embodiment, a hybrid reward model can be constructed by introducing a small amount of manually annotated data on top of the unlabeled reward model. Specifically, the system can employ a two-stage training strategy. First, large-scale pre-training is performed using the unlabeled reward model, followed by fine-tuning using a small amount of high-quality manually annotated data. This approach can further improve the accuracy and stability of the reward model while maintaining low annotation costs, making it particularly suitable for high-risk decision-making scenarios.

[0069] For complex tasks requiring diverse expertise, a multi-agent collaborative framework can be expanded. Multiple specialized agents (such as network diagnostic experts, data processing experts, and security audit experts) work together, each responsible for decision-making in a specific area. A central coordination mechanism integrates suggestions from all parties to better handle complex cross-domain issues.

[0070] Step 103: Based on the unlabeled reward value, a low-rank adaptive network is used in the model training engine to fine-tune the parameters of the model, and batch training is performed using a token gradient optimization strategy.

[0071] The lightweight training engine integrates a Low-Rank Adaptation (LoRA) policy network with a gradient compression mechanism to achieve resource-efficient model training. Through parameter optimization strategies, it significantly reduces computational and storage overhead, enabling large-scale models (e.g., 30B parameters) to be trained on limited hardware.

[0072] The lightweight training engine consists of three core components: a low-rank adaptive policy network, a gradient compression processor, and a batch optimizer.

[0073] The low-rank adaptive strategy network uses LoRA technology to fine-tune the parameters of the pre-trained language model. In the specific implementation, the query, key, and value projection layers in the attention mechanism of the model are added with low-rank decomposition matrices A and B, and the complete weight matrix that originally needs to be updated is replaced by Decomposed into the product of two low-rank matrices W+ΔW=W+BA, where the matrix matrix And the rank r<<min(d,k). d is the number of rows of the complete weight matrix W, and k is the number of columns of the complete weight matrix W. Represents the domain of real numbers. During training, the original weights W remain frozen, and only the low-rank matrices A and B are updated, reducing the number of trainable parameters from the original d×k to (d+k)×r, a parameter reduction of up to 99%.

[0074] The traditional PPO algorithm needs to calculate the complete gradient for each token in the entire sequence, resulting in a memory requirement that is proportional to the length of the sequence. After the reward value is calculated, the present application is handed over to a separate trainer to calculate the gradient and other training parts, and the model parameters are updated according to the gradient of the PPO algorithm. The gradient compression processor is used to meet the special needs of long sequence decision scenarios. It adopts the gradient calculation optimization strategy of the present application, namely the "initial token priority" strategy, which only retains the complete gradient information of the initial part of the sequence (for example, the first 100 tokens), and the gradients of subsequent tokens are sparsely processed or directly set to zero, significantly reducing the computational complexity of backpropagation.

[0075] The batch optimizer implements an efficient batch training strategy, supporting the parallel processing of multiple trajectories within a single iteration. By dynamically increasing batch size and using high-speed NVLink as a backend communication solution, it can simultaneously process 8-32 trajectories on a hardware configuration of eight NVIDIA H800 GPUs, and synchronize gradients across GPUs, further improving training throughput.

[0076] During training, the lightweight training engine uses the following parameters: LoRA rank = 64, scaling factor = 128, target layer: query (Q), key (K), and value (V) projection layer of the attention mechanism in the Transformer architecture; gradient clipping threshold: 2.0; learning rate: 2e-5, cosine annealing scheduling, and batch size dynamically adjusted based on GPU memory, with a maximum supported batch size of 32. The freezing strategy completely freezes the base model parameters and only updates the adaptation layer parameters.

[0077] The core advantage of this engine is that it significantly reduces training resource requirements, enabling reinforcement learning training of a 30B parameter model with as few as four NVIDIA H800 GPUs, saving 75% of hardware resources compared to traditional methods. Furthermore, through gradient compression and batch optimization, training speed is increased by over 50%, significantly shortening model iteration cycles.

[0078] Step 104: Identify and extract function call instructions from the text generated by the model, convert the function call instructions into tool API calls, and inject the execution results of the tool API calls into the next round of model input in real time.

[0079] The environment interaction module parses the function call instructions generated by the model, executes the corresponding operations, and feeds the results back to the model, forming a complete "decision-execution-feedback" closed loop. This module is a key bridge connecting the AI ​​decision-making system with the external environment, ensuring that the model can continuously optimize its decision-making strategy based on real-world feedback.

[0080] The environment interaction module contains a function call parser, a tool interface adapter, a feedback formatter, and a status update manager.

[0081] The function call parser is used to extract standardized function call instructions from the text generated by the model. In a specific embodiment, the function call parser uses a combination of regular expressions and JSON parsing to identify function calls in a specific format (such as " <networkp2pbw>" or "{function:'compress_files', args:{source:' / data', target:'archive.zip'}}"). The parser supports multiple function call formats and handles incomplete or malformed parameters to improve system robustness.

[0082] The tool interface adapter is used to convert standardized function calls into actual tool API calls. The system predefines various tool sets, including:

[0083] Network diagnostic tools, such as NetworkP2pBw (point-to-point bandwidth detection) and NetworkAllreduceBw (full protocol communication performance detection);

[0084] Computing resource monitoring tools, such as GPUMemoryBw (GPU memory bandwidth detection) and GPUComputeFP ​​(GPU floating-point computing performance detection);

[0085] Data transmission tools, such as HostD2hBw (device to host transmission rate detection), HostH2dBw (host to device transmission rate detection), etc.

[0086] File operation tools, used to implement basic file operations such as compression, decompression, search, and movement;

[0087] The tool interface adapter is responsible for parameter validation, permission checking, and exception handling, ensuring that function calls are executed safely and efficiently. When an anomaly is detected (such as insufficient resources, insufficient permissions, or network outages), a standardized error response (such as "False: Insufficient Permissions") is generated and detailed error information is recorded. For recoverable errors, automatic retries or alternative solutions are attempted. For unrecoverable errors, the current operation is terminated and the model is provided with a clear reason for the failure, guiding it to adjust its decision-making strategy.

[0088] The feedback formatter is used to convert the tool execution results of the tool interface adapter into a standardized feedback format. To simplify the model learning process, complex execution results are abstracted into Boolean (True / False) feedback, indicating whether the operation was successful or the test result was normal. For example, when performing a NetworkP2pBw test, True is returned if the bandwidth is normal, and False is returned if it is abnormal. For scenarios that require detailed information, a brief description (such as "bandwidth is below the threshold of 20%") is added to the Boolean value, but the format is kept consistent to facilitate model understanding and learning.

[0089] The state update manager integrates execution results into the system state and prepares the next round of model input. To cope with long interaction sequences, the manager maintains a structured state history, recording executed function calls and their responses. A sliding window mechanism is used to control the context length to prevent it from exceeding the model's processing capacity. When the context length approaches a preset limit (e.g., 4k tokens), a sliding window is used to prioritize recent interactions and key decision points, removing redundant descriptive text to ensure the model always has access to the most relevant historical information.

[0090] In a further embodiment, based on the output function call sequence, explanatory text is further generated to explain the decision rationale and expected results. This increases user trust in the system and facilitates expert review and intervention, making it suitable for high-risk decision-making scenarios. Explainability can be implemented through technologies such as causal inference models or attention mechanism visualization.

[0091] The following is an example of an environment interaction process.

[0092] When the model is generated <networkp2pbw>When a function is called, the function call parser extracts the instruction and passes it to the adapter; the tool interface adapter calls the corresponding network diagnostic tool to perform point-to-point bandwidth detection; after the detection is completed, the feedback formatter returns the result (such as "True" indicates that the bandwidth is normal) to the system; the status update manager adds the result to the status history and prepares for the next round of input "User request to detect network problems, executed <networkp2pbw>The test result is normal (True)", which allows the model to continue to decide the next step.

[0093] Through standardized interactive interfaces and feedback mechanisms, this module enables the model to effectively learn environmental dynamics and make coherent and reasonable multi-step decisions in practical applications, significantly improving the system's performance in complex tasks.

[0094] The overall mechanism of environmental feedback and status update is as follows:

[0095] Function Execution and Feedback Injection: The function execution engine of the environment interaction module securely and efficiently executes model-generated function calls. The execution engine utilizes sandbox isolation technology to ensure that potentially dangerous operations (such as file deletion and network configuration modifications) are performed within a controlled environment. During execution, detailed operation logs are recorded, including execution time, resource consumption, and intermediate state changes, facilitating subsequent analysis and debugging. To simplify the model learning process, all execution results are uniformly converted into Boolean feedback. The conversion rules are as follows: For detection functions (such as NetworkP2pBw): If the detection result is within the normal range, True is returned; otherwise, False is returned. For operation functions (such as compressing files): If the operation is completed successfully, True is returned; otherwise, False is returned. For query functions (such as searching for files): If a match is found, True is returned; otherwise, False is returned. This simplified feedback mechanism significantly reduces the complexity of model learning, allowing it to focus on decision-making logic rather than result parsing.

[0096] Using triple structure<P,F,R> Represents the current state, where P is the user's original prompt, F={f1,f2,...,f n } is the executed function call sequence, R={r1,r2,...,r n } is the corresponding execution result. This structured representation ensures the integrity and consistency of state information.

[0097] Different retention weights are assigned to different types of information to ensure that critical information (such as error feedback and user instruction clarification) is retained in the context first. For information that needs to be retained but is long, semantic compression is performed to retain the core meaning while reducing token consumption.

[0098] To prevent inconsistencies in state information across multiple rounds of interaction, a state verification mechanism is implemented. After each round of interaction, the current state is verified to be consistent with the historical record. If an inconsistency is detected (e.g., the model "forgets" previous incorrect feedback), the correct state information is automatically corrected and forced to be injected, ensuring that decisions are based on accurate historical context.

[0099] In an optional embodiment, in addition to basic textual status, the system also supports multimodal status representation for processing and storing non-textual feedback such as images and audio. For example, in a network topology analysis scenario, a network structure diagram can be included as part of the status to assist the model in understanding the network environment; in a system monitoring scenario, performance curves can be incorporated into the status representation to help the model identify abnormal patterns.

[0100] In terms of training parameters and hardware configuration, according to a preferred embodiment, the hyperparameters can be set as follows: discount factor (γ) = 0.8, clipping threshold (Clip Epsilon) = 0.2; entropy regularization coefficient (β) = 0.1, value loss weight (α) = 0.1.

[0101] The hardware deployment is as follows: the minimum configuration is one server equipped with four NVIDIA H800 GPUs; it supports hybrid parallel training across 8 nodes and 64 GPUs.

[0102] Example 2

[0103] Network fault diagnosis scenario application examples:

[0104] In large-scale distributed training clusters, network failures are a common problem that impacts system performance. Traditional diagnostic methods often rely on manual experience and struggle to cope with complex and changing failure modes. This system achieves automated and efficient network fault diagnosis through a multi-step function call sequence.

[0105] The diagnostic process is as follows:

[0106] Initial state: The user submits a diagnostic request to check for network performance degradation in the training cluster.

[0107] First round of decision-making: calling <networkp2pbw>"Detect point-to-point bandwidth and get feedback True (normal).

[0108] Status update: <"Checking network performance", {" <networkp2pbw>'}, {True}>

[0109] Second round of decision-making: calling <networkallreducebw>"Testing full protocol communication performance, the feedback is False (abnormal).

[0110] Status update: <"Checking network performance", {" <networkp2pbw> ”，" <networkallreducebw>”},{True,False}>.

[0111] The third round of decision-making: calling <busallreducebw>"Test the communication efficiency within a single server and get the feedback True (normal).

[0112] Status update: <"Checking network performance", {" <networkp2pbw> ”，" <networkallreducebw> ”，" <busallreducebw>”}, {True, False, True}>.

[0113] Diagnostic conclusion: The problem is determined to be in the cross-server full protocol communication link, rather than in the internal communication of a single machine or point-to-point communication.

[0114] During this process, the QD-Reward module dynamically calculates a reward by comparing the current diagnostic trajectory with expert diagnostic patterns in the sample library. For example, if the system correctly checks the communication efficiency within a single server after detecting a full-protocol communication anomaly (rather than repeatedly checking components that have already been confirmed to be functioning), this decision is highly consistent with the expert pattern and results in a higher reward of 0.85. Conversely, if the system fails to further narrow the problem after discovering the anomaly, it receives a lower reward of 0.3.

[0115] On a test set containing 100 typical network failure scenarios, the system achieved a diagnostic accuracy of 92% without requiring any manually labeled data for training.

[0116] Example 3

[0117] Application examples of system operation and maintenance automation scenarios:

[0118] In the operation and maintenance management of complex IT infrastructure, traditional methods often rely on expert experience and predefined rules, which are difficult to cope with dynamically changing system states and emerging failure modes. This system can automatically diagnose and repair system problems by learning from historical operation and maintenance operations. The operation and maintenance process is as follows:

[0119] Initial status: Monitoring detects "abnormal increase in response time of server A".

[0120] First round of decision-making: calling <systemmetricscheck>Check basic metrics (CPU, memory, disk) and get feedback: "CPU usage is normal, memory usage is 90%, and disk usage is 60%."

[0121] Second round of decision-making: calling <processanalysis>Analyze the processes that use high memory and get the following feedback: "The database cache process uses 70% of memory."

[0122] The third round of decision-making: calling <dbcacheoptimize>"Optimize database cache configuration and get feedback True (success).

[0123] Verification result: The server response time returns to normal.

[0124] During this process, the QD-Reward module compares current operation traces with historical success cases, assigning high rewards to effective diagnosis and repair steps, guiding the model to learn optimal O&M strategies. In particular, the system identified from historical data that the pattern of "high memory usage + database service" is often associated with cache configuration issues, allowing it to directly pinpoint the root cause of the problem.

[0125] In a test environment containing 200 common system failures, this system achieved an 85% automatic repair rate, reducing the average fault resolution time from 45 minutes with traditional methods to 12 minutes, significantly reducing operation and maintenance costs and system downtime.

[0126] It can be seen that the reinforcement learning method based on state compression and unlabeled rewards of the present invention provides an innovative solution for the alignment of intelligent AI agents. Compared with the existing technology, it has the following advantages:

[0127] 1. Improve decision-making quality:

[0128] Through function-call-level state compression and gradient broadcasting, the model coordinates multi-step decisions from a global perspective, increasing human consistency from 30% with traditional methods to over 85%, significantly enhancing the alignment between agent behavior and human expectations. Structured state representation and an iterative mechanism for environmental feedback enforce the retention of key decision information, enabling the model to generate coherent multi-step function call sequences in complex tasks, increasing the probability of correct calls from 10% with traditional methods to 78%. The unlabeled reward model automatically identifies and evaluates emerging decision patterns, enabling the system to maintain efficient decision-making in the face of unseen, unusual scenarios, improving adaptability by 40%.

[0129] 2. Optimize resource efficiency:

[0130] By freezing the base model parameters and updating only the low-rank adaptation layer, the lightweight training architecture reduces GPU resource requirements by 75%. This enables training of a 30B parameter model on four NVIDIA H800 GPUs, breaking through the hardware limitations of large model deployment. A gradient calculation optimization strategy focuses solely on updating the policy for the initial output token, avoiding backpropagation for all tokens in long sequences. This increases training speed by 50% and significantly shortens model iteration cycles. A state compression mechanism converts fine-grained token-level state to high-level function call-level state. Combined with a dynamic context management strategy, this enables the system to process longer interaction sequences within limited memory, increasing context length by three times.

[0131] 3. Reduce implementation costs:

[0132] The quantum dynamics reward model automatically builds a library of positive and negative examples and dynamically calculates reward values ​​by analyzing the similarity of historical function call traces. This eliminates the need for manually labeled data and reduces data preparation costs by over 90%. The lightweight training architecture reduces hardware requirements, making reinforcement learning affordable for small and medium-sized organizations, expanding the accessibility and application of the technology. The system continuously learns and optimizes during operation, automatically adapting to environmental changes, reducing the need for manual intervention and maintenance, and lowering long-term operation and maintenance costs by 60%.

[0133] 4. Make application scenarios scalable:

[0134] In a test set consisting of 100 typical network failure scenarios, the system achieved a 92% diagnostic accuracy rate, reducing average diagnostic time by 65%, without requiring any manually labeled data for training. In a test of 50 common data processing scenarios, the system achieved an 88% task automation rate, improving average execution efficiency by 40%, and was able to adaptively handle unprecedented file formats and processing requirements. In a test environment encompassing 200 common system failures, the system achieved an 85% automatic repair rate, reducing average fault resolution time from 45 minutes using traditional methods to 12 minutes, significantly reducing operational costs and system downtime.

[0135] In summary, the technical solution proposed in this application comprehensively solves the core challenges faced by O1-level intelligent AI alignment through an innovative state compression reinforcement learning method and an unlabeled reward model, and has achieved remarkable results in improving decision-making quality, optimizing resource efficiency, reducing implementation costs, and expanding application scenarios. It provides a feasible path for the large-scale deployment and application of intelligent AI.

[0136] In an optional embodiment, the state compression strategy can be expanded to an adaptive state compression mechanism. This mechanism dynamically adjusts the state compression granularity and strategy based on task complexity, context length, and available computing resources. For example, it can retain more detailed information when resources are abundant, while adopting a more aggressive compression strategy when resources are limited. This adaptive mechanism achieves a better balance between efficiency and accuracy.

[0137] To address data privacy and security concerns, this solution can be further extended to a federated learning framework. Each participant trains a model locally and shares model updates (rather than the original data). A central server then integrates these updates to form a global model. This approach is particularly suitable for scenarios involving sensitive data (such as medical records and financial transactions), as it can leverage distributed data to improve model performance while protecting data privacy.

[0138] Example 4

[0139] Another aspect of the present application includes a functional module architecture consistent with the aforementioned method flow, that is, the embodiment of the present application further provides a reinforcement learning system based on state compression and unlabeled rewards, including:

[0140] A state compression module is configured to receive user prompts output by the model and convert the user prompts into a structured state vector, wherein the state vector includes a task goal, a function call sequence, and an environment feedback result;

[0141] an unlabeled reward module, configured to map the function call sequence into a complex plane vector, automatically generate an unlabeled reward value based on the complex plane vector, and back-propagate the unlabeled reward value to all tokens in the function call sequence;

[0142] A model training engine, configured to fine-tune parameters of a pre-trained language model using a low-rank adaptive network based on the unlabeled reward value, and perform batch training using a token gradient optimization strategy;

[0143] The environment interaction module is used to identify and extract function call instructions from the text generated by the model, convert the function call instructions into tool API calls, and inject the execution results of the tool API calls into the next round of model input in real time.

[0144] The system can be implemented by the reinforcement learning method based on state compression and unlabeled rewards provided in the above-mentioned embodiment 1. The specific implementation method can be found in the description of embodiment 1 and will not be repeated here.

[0145] The present application also provides a memory storing a plurality of instructions, wherein the instructions are used to implement the reinforcement learning method based on state compression and unlabeled rewards as described in Example 1.

[0146] The present application also provides an electronic device, including a processor and a memory connected to the processor, wherein the memory stores multiple instructions, which can be loaded and executed by the processor to enable the processor to execute the reinforcement learning method based on state compression and unlabeled rewards as described in Example 1.

[0147] Although preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they are aware of the basic inventive concepts. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of this application. Obviously, those skilled in the art may make various changes and modifications to this application without departing from the spirit and scope of this application. Thus, if such changes and modifications fall within the scope of the claims of this application and their equivalents, then this application is intended to include such changes and modifications.< / dbcacheoptimize> < / processanalysis> < / systemmetricscheck> < / busallreducebw> < / networkallreducebw> < / networkp2pbw> < / busallreducebw> < / networkallreducebw> < / networkp2pbw> < / networkallreducebw> < / gpucomputefp> < / networkallreducebw> < / networkp2pbw> < / networkallreduce> < / networkallreduce>

Claims

1. A reinforcement learning method based on state compression and unlabeled rewards, characterized in that: include: receiving user prompts output by the model, and converting the user prompts into a structured state vector, wherein the state vector includes a task goal, a function call sequence, and an environment feedback result; Mapping the function call sequence into a complex plane vector, automatically generating an unlabeled reward value based on the complex plane vector, and back-propagating the unlabeled reward value to all tokens in the function call sequence; Based on the unlabeled reward value, a low-rank adaptive network is used in the model training engine to fine-tune the parameters of the model, and batch training is performed using a token gradient optimization strategy; Identify and extract function call instructions from the text generated by the model, convert the function call instructions into tool API calls, and inject the execution results of the tool API calls into the next round of model input in real time.

2. The reinforcement learning method based on state compression and unlabeled rewards according to claim 1, characterized in that The converting the user prompt into a structured state vector further comprises: Parse the function call, extract compliant JSON instructions from the token sequence, filter redundant tokens, automatically delete intermediate tokens irrelevant to the function call, and retain only the core operation nodes.

3. The reinforcement learning method based on state compression and unlabeled rewards according to claim 1, characterized in that Mapping the function call sequence into a complex plane vector, automatically generating an unlabeled reward value based on the complex plane vector, and back-propagating the unlabeled reward value to all tokens in the function call sequence further includes: The current function call sequence is formed into a trajectory τ, and the trajectory τ is mapped to a vector ψ(τ) on the complex plane based on the similarity between the trajectory τ and the positive sample library and the negative sample library; the positive sample library pre-stores the trajectories that have passed the expert review, and the negative sample library pre-stores the trajectories that have not passed the expert review; Calculate the reward value r based on the vector ψ(τ) on the complex plane: r=1-|ψ(τ)|, where |ψ(τ)| represents the modulus of the vector ψ(τ).

4. The reinforcement learning method based on state compression and unlabeled rewards according to claim 1, characterized in that The method of fine-tuning the parameters of the model using a low-rank adaptive network in the model training engine further includes: Add low-rank decomposition matrices A and B to the query, key, and value projection layers in the model's attention mechanism, and convert the complete weight matrix Decomposed into the product of two low-rank matrices W+ΔW=W+BA, where the matrix matrix And the rank r<<min(d,k); d is the number of rows of the complete weight matrix W, k is the number of columns of the complete weight matrix W. During the training process, the original weight W remains frozen and only the low-rank matrices A and B are updated.

5. The reinforcement learning method based on state compression and unlabeled rewards according to claim 1, characterized in that The method further includes: identifying and extracting function call instructions from the text generated by the model, converting the function call instructions into tool API calls, and injecting the execution results of the tool API calls into the next round of model input in real time. Use regular expressions combined with JSON parsing to extract standardized function call instructions from the text generated by the model; Converting standardized function call instructions into calls to tool APIs, including network diagnostic tools, computing resource monitoring tools, data transfer tools, and file operation tools; Abstract the execution results of tool API calls into Boolean feedback, indicating whether the operation is successful or the test result is normal; The Boolean feedback results are integrated into the system state and prepare for the next round of model input, recording the executed function calls and their feedback, and controlling the context length through a sliding window mechanism.

6. A reinforcement learning system based on state compression and unlabeled rewards, characterized in that include: A state compression module is configured to receive user prompts output by the model and convert the user prompts into a structured state vector, wherein the state vector includes a task goal, a function call sequence, and an environment feedback result; an unlabeled reward module, configured to map the function call sequence into a complex plane vector, automatically generate an unlabeled reward value based on the complex plane vector, and back-propagate the unlabeled reward value to all tokens in the function call sequence; A model training engine, configured to fine-tune parameters of a pre-trained language model using a low-rank adaptive network based on the unlabeled reward value, and perform batch training using a token gradient optimization strategy; The environment interaction module is used to identify and extract function call instructions from the text generated by the model, convert the function call instructions into tool API calls, and inject the execution results of the tool API calls into the next round of model input in real time.

7. The reinforcement learning system based on state compression and unlabeled rewards according to claim 6, characterized in that The state compression module is further configured to: Parse the function call, extract compliant JSON instructions from the token sequence, filter redundant tokens, automatically delete intermediate tokens irrelevant to the function call, and retain only the core operation nodes.

8. The reinforcement learning system based on state compression and unlabeled rewards according to claim 6, characterized in that The unlabeled reward module is further used to: The current function call sequence is formed into a trajectory τ, and the trajectory τ is mapped to a vector ψ(τ) on the complex plane based on the similarity between the trajectory τ and the positive sample library and the negative sample library; the positive sample library pre-stores the trajectories that have passed the expert review, and the negative sample library pre-stores the trajectories that have not passed the expert review; Calculate the reward value r based on the vector ψ(τ) on the complex plane: r=1-|ψ(τ)|, where |ψ(τ)| represents the modulus of the vector ψ(τ).

9. The reinforcement learning system based on state compression and unlabeled rewards according to claim 6, characterized in that The model training engine is further used to: Add low-rank decomposition matrices A and B to the query, key, and value projection layers in the model's attention mechanism, and convert the complete weight matrix Decomposed into the product of two low-rank matrices W+ΔW=W+BA, where the matrix matrix And the rank r<<min(d,k); d is the number of rows of the complete weight matrix W, k is the number of columns of the complete weight matrix W. During the training process, the original weight W remains frozen and only the low-rank matrices A and B are updated.

10. The reinforcement learning system based on state compression and unlabeled rewards according to claim 6, characterized in that The environment interaction module is further used to: Use regular expressions combined with JSON parsing to extract standardized function call instructions from the text generated by the model; Converting standardized function call instructions into calls to tool APIs, including network diagnostic tools, computing resource monitoring tools, data transfer tools, and file operation tools; Abstract the execution results of tool API calls into Boolean feedback, indicating whether the operation is successful or the test result is normal; The Boolean feedback results are integrated into the system state and prepare for the next round of model input, recording the executed function calls and their feedback, and controlling the context length through a sliding window mechanism.

11. An electronic device, characterized in that: The invention comprises a processor and a memory, wherein the memory stores a plurality of instructions, and the processor is used to read the instructions and execute the reinforcement learning method based on state compression and unlabeled reward according to any one of claims 1 to 5.

12. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a plurality of instructions, and the plurality of instructions can be read by a processor to execute the reinforcement learning method based on state compression and unlabeled reward according to any one of claims 1 to 5.

Images