Strategy optimization method and device based on interaction track, equipment and medium

Abstract

The invention relates to the technical field of artificial intelligence, can be applied to business scenes of robot strategy training, financial science and technology, medical treatment and health and the like, and discloses a strategy optimization method and device based on an interaction track, equipment and a medium. The reinforcement learning agent interacts with the environment to generate an action and state sequence and record an interaction track, an optimal action strategy is generated based on a track optimization strategy, an original empirical data set is further generated, and a pre-training model is optimized through supervised learning to obtain a target strategy model. According to the method, a task-related high-quality trajectory is generated through reinforcement learning agent and environment interaction, original empirical data is extracted on this basis, and a pre-training strategy model is optimized in combination with a supervised learning mechanism, so that the sample utilization efficiency is effectively improved, and the generalization ability and execution robustness of the model under a multi-task condition are enhanced.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a strategy optimization method, device, equipment and storage medium based on interaction trajectory. Background Art

[0002] In the field of robot policy training, the current mainstream approach typically relies on large-scale data pre-training, supplemented by human operation demonstrations for policy fine-tuning. However, human demonstration data generally has uneven quality and is expensive to collect. Especially in complex tasks involving multi-step control, delicate operations, and continuous sensory feedback, the accuracy and stability of demonstration data are difficult to guarantee. This makes it difficult for the policies generated during model training to achieve reliable generalization in high-precision scenarios. In particular, in situations where robot execution requires stable and high-frequency interactions, the model's adaptability to scene switching is heavily dependent on the comprehensiveness and structure of the training data, resulting in insufficient robustness during the deployment phase.

[0003] In the fintech sector, intelligent robots are increasingly playing a key role in risk assessment, compliance verification, and process automation. These tasks typically require policy models to possess multimodal understanding capabilities for structured forms, semi-structured text, and operational instructions, and to maintain consistent decision-making accuracy in high-concurrency scenarios. While traditional reinforcement learning methods are somewhat effective for fixed transaction processes, when the task is expanded to multiple process objects or involves cross-system data calls, their strategies often experience a decline in generalization or significant interference with prior model structures. Furthermore, directly optimizing reinforcement learning on pre-trained models can easily lead to unstable policy gradients, which in turn affects the retention and transfer of existing financial knowledge.

[0004] In the healthcare sector, robotics technology is widely used in scenarios such as rehabilitation assistance, image guidance, and drug delivery. These tasks not only require extremely high policy accuracy, but also guarantee policy generalization across patients and scenarios. Due to the complex physiological constraints and high-risk control involved in medical scenarios, the reliability of human demonstration data is difficult to guarantee, and large-scale collection carries ethical and safety costs. Traditional reinforcement learning methods often involve trial-and-error behavior in the initial training phase, which is unacceptable in the medical field. Furthermore, even pre-training models in simulated environments cannot accurately account for policy drift caused by changes in real patient conditions, resulting in limited system stability.

[0005] In summary, existing technologies have significant shortcomings in how to build a policy training mechanism that has both task execution accuracy and cross-task generalization capabilities. In particular, key links such as quality control of data sources, stability assurance during policy optimization, and improving adaptability in cross-domain task applications need to be improved. Summary of the Invention

[0006] The main purpose of the present invention is to provide a strategy optimization method, device, equipment and storage medium based on interaction trajectories, aiming to solve the technical problem that the existing technology cannot efficiently combine reinforcement learning interaction data and supervised learning mechanisms for strategy optimization, resulting in low sample efficiency and insufficient cross-task generalization ability, making it difficult to improve the robustness and adaptability of the strategy.

[0007] To achieve the above objectives, the present invention provides a strategy optimization method based on interaction trajectories, comprising:

[0008] Based on the target task, the initial state distribution and state transition probability function of the environment are defined, and the policy network parameters of the reinforcement learning agent are initialized to generate the initial policy parameters.

[0009] interacting with the environment through the reinforcement learning agent and generating an action sequence based on the initial policy parameters;

[0010] According to the initial state distribution and the state transition probability function, the state sequence of the environment is updated, the action sequence, the state sequence and the corresponding reward data are recorded, and an interaction trajectory is generated according to the recorded sequence and data;

[0011] Determine a cumulative expected reward function based on the action sequence, state sequence, and reward data in the interaction trajectory, and update the initial strategy parameters by maximizing the gradient of the cumulative expected reward function to generate an optimal action strategy;

[0012] Executing the optimal action strategy to generate an original experience data set including a state sequence and an action sequence of the environment;

[0013] The pre-trained strategy model is supervised and learned based on the original experience data set, the parameters of the strategy model are optimized, and a target strategy model is generated.

[0014] Furthermore, to achieve the above-mentioned purpose, the present invention provides a strategy optimization device based on interactive trajectories, comprising:

[0015] The environment modeling module is used to define the initial state distribution and state transition probability function of the environment based on the target task, initialize the policy network parameters of the reinforcement learning agent, and generate the initial policy parameters;

[0016] an interactive decision module, configured to interact with the environment through the reinforcement learning agent and generate an action sequence based on the initial strategy parameters;

[0017] a trajectory generation module, configured to update the state sequence of the environment according to the initial state distribution and the state transition probability function, record the action sequence, state sequence and corresponding reward data, and generate an interaction trajectory according to the recorded sequence and data;

[0018] a strategy optimization module, configured to determine a cumulative expected reward function based on the action sequence, state sequence, and reward data in the interaction trajectory, and to update the initial strategy parameters by maximizing the gradient of the cumulative expected reward function to generate an optimal action strategy;

[0019] A strategy execution module, configured to execute the optimal action strategy and generate an original experience data set including a state sequence and an action sequence of the environment;

[0020] The model optimization module is used to perform supervised learning on the pre-trained strategy model based on the original experience data set, optimize the parameters of the strategy model, and generate a target strategy model.

[0021] Furthermore, to achieve the above-mentioned purpose, the present invention also provides a determination machine device, which includes a memory, a processor, and a policy optimization program based on an interactive trajectory stored in the memory and runnable on the processor. When the policy optimization program based on an interactive trajectory is executed by the processor, the steps of the policy optimization method based on the interactive trajectory as described above are implemented.

[0022] Furthermore, to achieve the above-mentioned purpose, the present invention also provides a machine-readable storage medium, on which a strategy optimization program based on interaction trajectories is stored. When the strategy optimization program based on interaction trajectories is executed by a processor, the steps of the strategy optimization method based on interaction trajectories as described above are implemented.

[0023] Beneficial effects: The present invention relates to the field of artificial intelligence technology and can be applied to business scenarios such as robot strategy training, financial technology, and medical health. It discloses a strategy optimization method, device, equipment, and medium based on interactive trajectories, including: defining the initial state distribution and state transition probability function of the environment, and initializing the policy network parameters; reinforcing the learning agent to interact with the environment to generate an action sequence; updating the state sequence according to the state distribution and transition function and recording the interaction trajectory; optimizing the initial strategy parameters based on the data in the interaction trajectory to generate the optimal action strategy; executing the optimal action strategy to generate the original experience data set; and performing supervised learning on the pre-trained strategy model based on the original experience data set, optimizing the model parameters, and obtaining the target strategy model. The present invention generates high-quality task-related trajectories through the interaction between the reinforcement learning agent and the environment, and extracts the original experience data on this basis, and optimizes the pre-trained strategy model in combination with the supervised learning mechanism, effectively improving the sample utilization efficiency, while enhancing the generalization ability and execution robustness of the model under multi-task conditions, solving the problems of low sample efficiency and weak strategy transfer ability in the prior art. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The present invention will be further described below with reference to the accompanying drawings and embodiments, in which:

[0025] Figure 1 Schematic diagram of an application environment of a strategy optimization method based on interaction trajectories according to an embodiment of the present invention;

[0026] Figure 2 This is a flow chart of an embodiment of a strategy optimization method based on interaction trajectories according to the present invention;

[0027] Figure 3 Schematic diagram of functional modules of a preferred embodiment of the interactive trajectory-based strategy optimization device of the present invention;

[0028] Figure 4 A schematic structural diagram of a determination device according to an embodiment of the present invention;

[0029] Figure 5 FIG. 2 is another structural diagram of a determination device in an embodiment of the present invention. DETAILED DESCRIPTION

[0030] It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0031] The strategy optimization method based on interactive trajectory provided by the embodiment of the present invention can be applied in Figure 1 In an application environment, the user terminal communicates with the server terminal via a network. The server terminal can define the initial state distribution and state transition probability function of the environment through the user terminal and initialize the policy network parameters; the reinforcement learning agent interacts with the environment to generate an action sequence; the state sequence is updated according to the state distribution and transition function and the interaction trajectory is recorded; the initial policy parameters are optimized based on the data in the interaction trajectory to generate the optimal action strategy; the optimal action strategy is executed to generate an original experience data set; the pre-trained policy model is supervised and learned based on the original experience data set to optimize the model parameters and obtain the target policy model. The present invention generates high-quality task-related trajectories by reinforcing the interaction between the learning agent and the environment, extracts original experience data based on this, and optimizes the pre-trained policy model in combination with a supervised learning mechanism, effectively improving sample utilization efficiency, while enhancing the model's generalization ability and execution robustness under multi-task conditions, and solving the problems of low sample efficiency and weak policy transfer ability in the prior art. The user terminal can be, but is not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices. The server terminal can be implemented as an independent server or a server cluster consisting of multiple servers. The present invention is described in detail below through specific embodiments.

[0032] See also Figure 2 , Figure 2This is a flow chart of an embodiment of a strategy optimization method based on interactive trajectories provided by the present invention. It should be noted that although a logical order is shown in the flow chart, in some cases, the steps shown or described may be performed in a different order than that shown here.

[0033] like Figure 2 As shown, the strategy optimization method based on interaction trajectory proposed in the present invention includes the following steps:

[0034] S10, based on the target task, defines the initial state distribution and state transition probability function of the environment, initializes the policy network parameters of the reinforcement learning agent, and generates initial policy parameters;

[0035] In this embodiment, before actually building a strategy model for task execution, it is first necessary to clarify the relationship between the external environment involved in the target task and the task itself. This process requires first analyzing the corresponding physical properties, state change boundaries and behavioral logic based on the spatiotemporal conditions, operational constraints, state response mechanisms and other contents in the task objectives. For example, in tasks involving multi-degree-of-freedom motion planning, visual feedback execution or high-precision action decision-making, environmental information usually includes static parameters such as geometric dimensions, material properties, boundary configurations, and dynamic parameters such as object movement mode, feedback delay or disturbance model. This information jointly determines the value range of the state space and the law of behavioral evolution.

[0036] The generation of the initial state distribution relies on the abstraction and statistical modeling of the aforementioned environmental parameters. The initial state distribution can be defined as a probability density function in the state space, describing the possible states of the system and their probability weights before receiving any action input. State distributions are usually constructed using Gaussian mixture models, Bayesian inference mechanisms, or sampling-based methods. For example, a joint distribution model can be constructed for the position and joint angles of a robot arm, or a joint conditional probability representation of the patient's initial physiological state can be defined in medical operations. The choice of distribution should take into account both modeling accuracy and computational overhead, and must be able to cover the typical state change region before task execution.

[0037] The state transition probability function is used to describe the dynamic law of state change over time under the action of an action. The construction of the transfer function is generally based on the task modeling language or state-action log data analysis. It can be defined in the form of a Markov decision process (MDP) or a partially observable Markov decision process (POMDP), mapping the probability distribution from the current state and input action to the next state. In specific implementation, the state transition function can be established through a simulator, a dynamic model library, or an estimator based on historical data. It is suitable for tasks that model complex action results or the influence of multi-factor disturbances, such as multi-arm collaboration, fine positioning and other application scenarios.

[0038] The initialization of the policy network parameters depends on the matching configuration of the task label set and the state-action structure. The input layer dimension of the policy network needs to be consistent with the state observation dimension, including position, velocity, image features, vital signs, etc.; the output layer dimension needs to be consistent with the action space structure, such as discrete action numbers or multi-dimensional control signals for continuous actions. During the initial construction, the policy network can use modules such as deformable convolution, graph neural network or attention mechanism to adapt to different data types, and set the parameter initialization strategy without introducing task bias. Initialization methods usually include Xavier initialization, He initialization and other methods. Parameter settings based on statistical priors can also be introduced to ensure that the initial policy has the ability to explore in the state space.

[0039] Generating initial policy parameters completes the entire process, from environment modeling, state distribution definition, transition function construction, policy structure configuration, to parameter initialization, setting the stage for the policy model to participate in subsequent interactive learning. This process does not directly rely on human demonstrations or large-scale pre-trained models, but instead, through task analysis and model mapping, provides a reasonable search starting point and boundary control for subsequent training.

[0040] In industrial robotics tasks, a state observation space containing temperature distribution and surface roughness can be generated based on the target process parameters, such as the welding starting point, workpiece material properties, and heat conduction patterns. Finite element simulation can then be used to establish state transition patterns. The policy network inputs can include the current trajectory point position, welding current, and surface temperature feedback. The output layer maps trajectory offset values ​​or tool posture adjustment parameters, initialized with a Gaussian distribution weight with gradient constraints.

[0041] In medical surgical robotic tasks, a state space distribution can be constructed based on preoperative imaging. The initial state includes instrument position and patient tissue deformation parameters, and the state transition function is established through tissue mechanics simulation or medical image change modeling. The policy network structure adopts a multimodal perception model that integrates visual and force information. Its input layer includes image encoding and tactile vectors, and the output layer is small position corrections and path deflection angles. The policy parameters can be initialized using structure-sensitive initialization, enabling high-resolution policy response capabilities in key organ regions.

[0042] In financial tasks, such as optimizing risk management strategies based on reinforcement learning, the initial state distribution can be modeled using historical market data, including market volatility, macroeconomic indicators, and user behavior. The state transition function is constructed based on the market response model. The input layer of the strategy network is configured with multiple asynchronous data sequence channels, and the output is a multi-dimensional asset adjustment recommendation or risk control instruction encoding. Initialization uses batch normalization and small random perturbations to maintain initial stability and generalization of the strategy.

[0043] This embodiment defines the state distribution and state transition probability function at the initial stage of task modeling, and initializes the policy network parameters based on the task label and environmental constraints. This ensures that the policy model has reasonable behavioral boundaries and state response ranges from the outset, thereby reducing sample waste caused by random search during interactive learning. Furthermore, configuring the policy network with a state-action matching structure enhances the policy's initial adaptability to the task structure, effectively improving the convergence speed and policy stability of subsequent training.

[0044] S20, interacting with the environment through the reinforcement learning agent and generating an action sequence based on the initial strategy parameters;

[0045] In this embodiment, once the policy network parameters are initialized, an interaction mechanism is established between the reinforcement learning agent and the environment, gradually generating action sequences corresponding to the dynamic evolution of the state space. The essence of interactive behavior is that the agent, based on its perceived current state, generates action signals based on its current policy at each time step, thereby acting on the environment, pushing the state to the next time point, and receiving new state observations as the basis for the next round of decision-making. Throughout this process, the policy parameters remain at their initial settings and do not undergo adaptive updates, ensuring that the generated action sequences are derived from the full range of behavioral structures that the current policy can express.

[0046] Each interaction cycle consists of four logical units: First, the state observation information for the current time step is extracted from the environment module. This state can be a multidimensional vector structure containing physical position, velocity, image encoding, sensor readings, vital signs, market indicators, and other content, depending on the task's modeling and dimensionality definition of the observation; Next, the current state observation is input into the policy network structure defined by the initial policy parameters. The policy network forward propagates the current input state without gradient propagation to generate the corresponding action probability distribution. The action space can be discrete space (such as a set of motion instructions) or continuous space (such as force value, angle, displacement, etc.), and the output structure must be consistent with the definition of the terminal output layer of the policy network.

[0047] Action instructions are then sampled from the action distribution output by the policy network. This sampling can be based on maximum probability selection (a greedy strategy), random sampling (to enhance exploration), or a softmax selection mechanism with a temperature parameter. Different sampling methods correspond to different exploration-exploitation strategy balances. The sampling result is the action execution instruction for the current time step, which is sent to the environment actuator or simulation module. The environment module updates its internal state based on the action input and generates a new round of state observations for subsequent interactions.

[0048] At each time step, the reinforcement learning agent records the current state observation, the generated action execution instructions, and the feedback value returned by the environment (usually rewards or state changes), and accumulates them as the temporal components of the action sequence. During this phase, the policy is not learned and adjusted, so the resulting action sequence reflects the behavioral trajectory under the current policy parameters and is stable, consistent, and reproducible with policy characteristics.

[0049] Example description: In a robotics task, a reinforcement learning agent is deployed in an industrial collaborative robot arm control system for behavior acquisition of the initial action strategy. Without the need to update parameters, the agent observes the current position, speed, and external working conditions (such as sensor signals or human-computer interaction input) of the end effector based on the initialized policy parameters, and generates real-time control instructions, such as joint angle adjustment and grasping path point selection. During the execution process, the robot interacts with the simulation or real environment to generate action records for multiple time steps. This sequence fully reflects the response characteristics of the initial strategy under different states, providing basic samples for subsequent data optimization. This processing method can obtain operation trajectories with good coverage in batches without fine-tuning the policy parameters, which is helpful for steady-state task modeling and training sample screening.

[0050] In healthcare scenarios, intelligent agents are used to assist in preoperative planning. Their policy initialization phase is based on a parameter structure derived from training with a large number of standard cases. While interacting with a virtual human model, the agent generates instructions for instrument advancement or posture adjustment based on state information such as anatomical images, instrument position, and tissue feedback. These policy-generated action sequences directly influence the feedback from the virtual surgical environment without updating model parameters. Ultimately, a complete operational path is generated, which is used to evaluate the adaptability of the current policy under specific anatomical structures and to form standardized samples for generalization and optimization of the policy model.

[0051] In fintech, agents are deployed within quantitative strategy backtesting systems to generate behavioral trajectory data for fixed strategies. While maintaining constant strategy parameters, the system simulates state observations of the market environment (such as price fluctuations, trading volume, and financial indicators). Based on the current state, the agent generates forward trading instructions (such as buy, sell, or position maintenance) through the strategy network. After executing this sequence of actions, the system records the state changes and behavioral feedback at each time step, generating data trajectories that can be used for subsequent model training. This approach is particularly suitable for generating initial sample sets with stable style labels. This approach reduces the uncertainty of strategy behavior and helps improve the consistency of the expression of training samples.

[0052] This embodiment uses fixed-policy interactions based on initial policy parameters to rapidly evaluate behavioral performance under the current policy and collect stable and controllable policy output sequences. This approach avoids update perturbations and facilitates the subsequent construction of high-confidence datasets for policy evaluation and data distillation. Compared to the interactive learning-while-learning policy optimization process, the action sequences generated in this phase have clear policy origins and controllable behavior patterns, improving data consistency and policy interpretability.

[0053] S30, updating the state sequence of the environment according to the initial state distribution and the state transition probability function, recording the action sequence, state sequence and corresponding reward data, and generating an interaction trajectory according to the recorded sequence and data;

[0054] In this embodiment, the update process of the environmental state extracts the starting state from the initial state distribution, which is represented by the probability density function of the state space. It is usually implemented through a probabilistic sampling method. In the simulation system, the corresponding state vector can be generated based on the scene random seed. The initial state not only involves physical parameters such as position and velocity, but also includes environmental perception conditions, object characteristics or historical residual variables. The execution of state transition depends on a defined state transition probability function, which describes the probability mechanism of how the environmental state changes over time under the current state and action input conditions. In a specific implementation, the function can be described in the form of a tabular model, a parameterized function, a neural network or a model based on a dynamic system.

[0055] The generation of state sequences follows a time-progression logic. At each time step, the system invokes the state transition model based on the previous state and the action execution instructions for that time step, generating the current state until the set trajectory termination condition is reached. After each time step, the system records the current action, the current state, and the reward value associated with that action. Reward data is calculated using a pre-set evaluation function, which calculates the contribution of the current state and action combination to the task objective. This function can be expressed as a scalar value, an interval score, or a normalized value based on the behavioral objective.

[0056] The final interaction trajectory is an ordered structure consisting of three parts: action sequence, state sequence, and corresponding reward value, arranged in chronological order. Each trajectory represents a continuous feedback chain of a complete interaction process and constitutes the core source of training samples.

[0057] In structural task control, a trajectory simulation engine can be used to construct a state transition function, whose input is the previous state and action command, and whose output is a feasible sample of the next state. In high-dimensional continuous state spaces, a conditional variational autoencoder can be used to generate state transition samples, so that state generation retains conditional dependencies while also having generalized expression. In control systems with high action sampling frequencies, a fixed frame rate can be set to advance state evolution and ensure sampling consistency. The reward function can use a target distance function, task completion rate, or policy stability as the source of the reward signal. In implementation, it can be constructed using a piecewise linear function, an empirical regression model, or a manual calibration module. All interaction trajectories are encapsulated as recording units in a standard format, such as JSON, Protobuf, or a tensor array structure, to facilitate batch reading by subsequent training modules.

[0058] Example: In a robot path guidance task, the system samples the initial state of the robotic arm from an initial pose distribution. The policy-generated gripper motion instructions are used to drive the arm. During this process, the dynamics model simulates gripper position changes and force feedback, records the state change trajectory, and generates rewards based on the proximity to the target point. The resulting trajectory data can reflect the execution mode and effectiveness of the initial policy under different starting conditions.

[0059] In healthcare scenarios, the agent's initial state is a multimodal state vector composed of preoperative images and historical operation variables. The environment transition function is driven by a virtual anatomical model, such as a vascular navigation path or a simulated puncture pathway. Each state update is derived from the previous frame's position and the current command output. Rewards are evaluated based on proximity to the target organ and puncture risk factors. Trajectories are used to assess policy stability and safety margins.

[0060] In FinTech scenarios, the initial state is the market state vector at a specific point in time, encompassing multiple asset metrics and risk factors. Actions are sequences of trading behaviors, with state transitions simulated by a market environment simulator based on post-execution market responses. Rewards are provided in the form of returns or risk-adjusted returns, and the resulting trajectory is used for subsequent behavioral analysis and trading strategy refinement.

[0061] This embodiment tightly couples the initialized policy output actions with the state evolution process. Without modifying the policy structure, it automatically captures the policy's response pattern in the real environment, and endows the sample with behavioral preference characteristics through clear reward feedback. This generates trajectory data with stable structure and clear behavior, providing high-quality labeled data for subsequent policy improvement and supervised learning, thereby improving the structural effectiveness and expression consistency of the training samples.

[0062] S40, determining a cumulative expected reward function based on the action sequence, state sequence, and reward data in the interaction trajectory, and updating the initial strategy parameters by maximizing the gradient of the cumulative expected reward function to generate an optimal action strategy;

[0063] In this embodiment, calculating the cumulative expected return based on the interaction trajectory involves extracting the immediate reward at each time step from the time series data and applying a time-decay weight based on a discount factor to it, thereby obtaining an evaluation of the long-term benefit of the behavior under specific policy parameters. Time decay processing of reward data is typically implemented through recursion or sliding window accumulation.

[0064] Constructing the cumulative expected reward function requires introducing the action probability output of the policy network, forming an objective function with policy parameters as variables. In discrete action space, the policy output can be interpreted as a probability distribution over the action set, and the log-likelihood loss objective function is constructed by the logarithmic relationship between the action selections in the interaction trajectory and their probabilities. In continuous space, this objective function can be a maximum likelihood estimator under a Gaussian distribution or an explicit loss expression can be established through policy gradient methods such as REINFORCE.

[0065] Policy parameters are updated by maximizing the gradient of the aforementioned objective function. This requires calculating the derivatives of the policy parameters with respect to the objective function and using the backpropagation algorithm to find the direction of gain in parameter space. This update is typically performed iteratively using an adaptive optimizer (such as Adam or RMSProp), which dynamically adjusts the learning rate based on the gradient history, ensuring smoother parameter adjustments and faster convergence. To prevent the policy from becoming stuck in a deterministic path early in training, a policy entropy term is introduced in the objective function as a regularization factor to maintain diverse action selection.

[0066] Ultimately, after multiple rounds of trajectory sampling, cumulative reward estimation, and policy gradient optimization iterations, a policy parameter set with higher expected returns and greater robustness is formed, which serves as the optimal action strategy for subsequent decision-making or policy execution stages.

[0067] In discrete control tasks, the policy network output layer uses a softmax structure, representing the probability of selecting each action. Policy optimization is achieved by maximizing the log-likelihood of the corresponding probabilities of the actual actions selected in the interaction trajectory. Monte Carlo methods are used to estimate the expected reward for each state-action pair. A discount factor is introduced during training to adjust for the impact of future returns, and an entropy regularization term is added to the objective function to maintain policy stochasticity.

[0068] In high-dimensional continuous control tasks, the policy network outputs a parameterized distribution of actions, such as a Gaussian distribution consisting of a mean and standard deviation. Action execution instructions are generated through sampling. Policy gradients are calculated using variational inference or path sampling algorithms, and policy parameters are updated through backpropagation. Convergence criteria are set during iterative execution, such as when the loss function decreases below a preset threshold or when the policy return stability reaches a desired range, to terminate training and obtain the current optimal policy.

[0069] In terms of the parameter update mechanism, a fixed batch size sampling trajectory can be used and policy updates can be performed in batches. Alternatively, the Critic in the Actor-Critic structure can be used to evaluate the value function under the current policy, assist in gradient signal estimation, and improve training efficiency and stability.

[0070] Example: In the field of robotics, when applied to flexible assembly control tasks, the policy network outputs the target motion direction and velocity of the robot's end effector. The reward function is set based on the grasping success rate and error tolerance. After sampling multiple trajectories, each parameter of the policy network is gradient updated. The system can quickly learn the optimal grasping behavior strategy and improve the success rate of the strategy under complex position deviation conditions.

[0071] In healthcare scenarios, policy optimization can be used to guide decisions about navigation-assisted surgical paths. By evaluating path feasibility and risk indicators through reward functions, the team can gain the ability to plan paths through complex anatomical structures through repeated trajectory sampling. The optimized policy parameters achieve higher task completion accuracy and risk avoidance, making them suitable for personalized navigation needs.

[0072] In the fintech sector, strategy models control the buying and selling rhythm and portfolio adjustment frequency of automated trading systems. By using a cumulative return function to reinforce low-risk, high-return strategies, and incorporating an entropy term to suppress speculative behavior that overfits to historical trends, the resulting strategies achieve more stable returns across multiple asset classes and possess cross-market generalization capabilities.

[0073] This embodiment performs goal-oriented optimization based on interaction trajectories in the policy network parameter space, effectively utilizing the actual reward signals in the trajectories to reversely drive the policy convergence direction, constructing an action output mapping that is most sensitive to task feedback, thereby improving the policy's convergence speed and expected reward level. Incorporating action probability distribution information and entropy regularization factors into the gradient update process effectively alleviates the policy's premature maturity problem and enhances the policy's behavioral exploration capabilities and stability. The resulting policy parameter set has strong task adaptability and generalization capabilities, laying a performance foundation for subsequent data generation and supervised transfer learning.

[0074] S50, executing the optimal action strategy to generate an original experience data set including a state sequence and an action sequence of the environment;

[0075] In this embodiment, after policy training is completed and the optimal action policy with updated parameters is obtained, the policy is loaded into the execution module to guide action selection during the actual task. The optimal action policy can be represented as a mapping from state to action space, inputting and outputting corresponding action execution instructions based on environmental state observations. The process of executing this policy involves continuously generating actions based on state feedback in a real or simulated environment, enabling the robot to complete a task process.

[0076] To collect empirical data, the initial state of the environment must first be set. This initial state can be sampled based on a defined state distribution, encompassing typical task starting points or representative problem states. Subsequently, at each time step, the policy module reads the current environmental state observations and outputs the action instructions for that time step based on the optimal action strategy. This instruction drives the actuator modules of the robot or control system through the execution interface, completing the action and causing the environmental state to change.

[0077] After each time step, the generated action instructions and the corresponding state observation results are stored as a set of experience pairs to form a single-step experience data unit. The entire interaction process continues in chronological order until the task termination signal is triggered, such as reaching the target state or meeting the task success conditions. All single-step experience data units are sequentially arranged and integrated in the time dimension to form an experience data set containing state sequences and action sequences. To ensure that this dataset can be used for subsequent model training or transfer learning, it is necessary to perform unified data structure conversion and format standardization operations on it, such as converting it to a time series tensor format and annotating the task category or state meaning of each sequence.

[0078] A policy stability control mechanism can be introduced during the policy execution phase. When the continuous state change is less than a set threshold, a state refresh operation is performed to prevent the policy from becoming stagnant. In the environment configuration, the initial state can be periodically randomized through programmatic interfaces or physical reset commands to ensure that the generated data covers more scenario states. In terms of the execution control interface, an action queue cache module can be integrated to improve the efficiency of continuous execution, and a state sampling synchronization operation can be performed after each time step to ensure a one-to-one correspondence between state observations and action executions.

[0079] When recording experience data, a key-value store or structured table format can be used to store single-step data units. Each item includes the current state, the executed action, the action execution timestamp, and the environmental feedback state information. In the task termination condition judgment logic, various strategies can be used to flexibly adapt to the task characteristics based on the state reaching the end zone, the cumulative reward reaching a threshold, or the maximum time step limit. During the experience data formatting process, for multi-task scenarios, a task label field must be added to enable the subsequent supervision model to distinguish the learning context.

[0080] Example: In a robotic assembly task, a learned optimal motion strategy is used to control a robot to grasp multiple workpieces and complete the assembly process sequentially. During this process, each motion command controls the robot's trajectory or end-effector state, while each environmental state records the robot's current position, gripper opening and closing status, and workpiece feedback signals. This continuous motion and state information is collected and converted into a structured time series dataset, which is used to train a new strategy model to adapt to different process rhythms.

[0081] In healthcare, optimal policies are used to control virtual assistance systems in simulated virtual patient environments in response to changes in the patient's physiological state, such as ventilator ventilation mode and medication dosage adjustments. Each state sequence represents a patient physiological parameter curve, and each action represents a treatment decision. The resulting empirical data can be used to train a broader health policy model.

[0082] In FinTech scenarios, the strategy execution module simulates asset trading decisions based on historical market conditions, generating action sequences (such as buy, hold, sell) and state sequences (such as market indicators and asset portfolio distribution). The collected empirical data is used to train intelligent asset allocation strategy models for a wider range of market environments.

[0083] This embodiment obtains high-quality, highly compatible empirical data by executing the complete state sequence and action sequence during the current optimal strategy acquisition task. This data not only fully reflects the strategy's response pattern to the task state, but also retains the feedback results of environmental changes on the strategy execution behavior, and has strong strategy adaptability and training value. Compared with random exploration or predefined action generation methods, this method can more efficiently generate data samples that can be used for supervised training, and can naturally integrate task execution preferences, state distribution characteristics, and strategy selection mechanisms, significantly improving the convergence speed and transfer generalization capabilities of subsequent model learning.

[0084] S60, performing supervised learning on the pre-trained policy model according to the original experience data set, optimizing the parameters of the policy model, and generating a target policy model.

[0085] In this embodiment, after obtaining the raw empirical data set collected by executing the optimal action policy, supervised learning is performed on the pre-trained policy model based on this data to improve the accuracy and generalization of the policy's capabilities. The raw empirical data set contains chronologically organized state and action sequences, providing high-quality state-action pairing information and can directly serve as a data source for supervised learning.

[0086] The supervised learning process constructs a loss function based on state inputs and action labels, adjusting model parameters by minimizing the difference between the policy model's predicted action probability distribution and the actual actions in the empirical data. When the policy output is a probability distribution, the loss function typically uses the log-likelihood function, optimizing parameters by maximizing the model's probability response to the empirical actions. The pre-trained policy model serves as the learning agent in this process and can be structured as a fully connected neural network, a convolutional network, or an attention mechanism.

[0087] The optimizer controls the direction and rate of parameter updates during the optimization process in supervised learning. Hyperparameters such as the learning rate, regularization strength, and batch size can be adjusted based on the amount of empirical data and the complexity of the policy model. To prevent overfitting and the loss of old knowledge, the parameter update process can incorporate policy entropy adjustment, knowledge distillation loss fusion, or progressive freezing of some network layers to balance the ability to learn new tasks while retaining the original policy.

[0088] Finally, after several rounds of supervised training, the policy model completes parameter optimization and obtains a target policy model that can better adapt to the current task and related tasks.

[0089] The policy model can be fully trained using data from all time steps in the empirical dataset, or weighted training can be performed on data from key state points through importance sampling to improve the model's sensitivity to key task nodes. In terms of training methods, full supervision can be used, where each state input corresponds to a clear action label; or semi-supervised training can be used, where some outputs of the policy model are guided to align with the action distribution while retaining the model's ability to adaptively explore the space.

[0090] During parameter optimization, optimizers such as Adam, SGD, or RMSprop can be used. Momentum or gradient clipping can be introduced to improve training stability and policy continuity. Supervised learning can be divided into multiple stages, each corresponding to a training cycle. The model can adjust the learning rates of different layers at different stages to achieve localized parameter refinement.

[0091] When the data set source involves multiple operation objects or multi-task scenarios, task identification input can be added for different objects, or a multi-head strategy network can be used to fit the state-action mapping relationship of different tasks respectively, thereby achieving the synergy of strategy sharing and difference modeling in a unified model.

[0092] Example: In a robot visual navigation task, a visual policy model is trained using a previously collected raw experience dataset containing motion instructions corresponding to camera input. Through supervised learning, the model learns how to control the robot to turn or move forward under different visual inputs, improving the robot's response stability in multi-scenario navigation tasks.

[0093] In medical and health scenarios, the intelligent medical record management system collects status records and treatment recommendation actions generated by the execution strategy in real operations, and trains a policy model through these state-action pairs to make it more accurate and standardized, supporting subsequent reuse in different disease treatment processes.

[0094] In finance, market conditions and asset adjustments recorded in raw empirical data sets are used to supervise and train strategy models. Through the optimization process, the model learns trading opportunities and asset allocation patterns based on these supervisory signals. Ultimately, the model can be deployed as a trading recommendation system within a multi-strategy investment management platform, improving system stability and decision consistency across multiple market conditions.

[0095] This embodiment significantly improves the quality of the model's policy output for target tasks by using high-quality, structured raw empirical data for supervised training of the policy model. Compared to traditional reinforcement learning methods that require repeated trial-and-error sampling strategies, this approach offers higher training efficiency and lower task risk, while also enabling rapid adaptation to specific tasks while retaining existing knowledge. This training method effectively leverages existing interactive results, boasts strong sample utilization and cross-task transfer capabilities, and provides a stable and controllable optimization path for the policy model.

[0096] The present invention relates to the field of artificial intelligence technology and can be applied to business scenarios such as robot strategy training, financial technology, and medical health. It discloses a strategy optimization method, device, equipment, and medium based on interactive trajectories, including: defining the initial state distribution and state transition probability function of the environment, and initializing the policy network parameters; reinforcing the learning agent to interact with the environment to generate an action sequence; updating the state sequence according to the state distribution and transition function and recording the interaction trajectory; optimizing the initial strategy parameters based on the data in the interaction trajectory to generate the optimal action strategy; executing the optimal action strategy to generate an original experience data set; and supervised learning of the pre-trained strategy model based on the original experience data set, optimizing the model parameters, and obtaining the target strategy model. The present invention generates high-quality task-related trajectories by reinforcing the learning agent to interact with the environment, and extracts original experience data on this basis. It optimizes the pre-trained strategy model in combination with a supervised learning mechanism, effectively improving the sample utilization efficiency, while enhancing the generalization ability and execution robustness of the model under multi-task conditions, solving the problems of low sample efficiency and weak strategy transfer ability in the prior art.

[0097] In one embodiment, the above step S10 includes:

[0098] S101, analyzing the physical scene and task structure information of the target task and generating a task label set;

[0099] S102, configuring a policy network structure of the reinforcement learning agent based on the task label set;

[0100] S103, constructing a set of environmental parameters consistent with the physical scene of the target task;

[0101] S104, generating an initial state distribution of the environment based on the environmental parameter set;

[0102] S105, defining a mapping relationship between state variables and action variables according to the task tag set and the environment parameter set, and constructing a state transition probability function based on the mapping relationship;

[0103] S106: Initialize the policy network parameters of the reinforcement learning agent based on the policy network structure and the state transition probability function to generate initial policy parameters.

[0104] In this embodiment, parsing the physical scene and task structure information of the target task belongs to the task modeling stage in the model modeling process. The physical scene of the task may include the geometric boundaries of the operating space, the constraint area, the physical properties of the target area, the dynamic state characteristics of the interactive object, etc., such as the position of the workbench in the industrial arm control task, the size and material of the clamped object. In terms of task structure information, it can cover the decomposition logic of the task objectives, operation sequence constraints, reward setting principles or safety boundary rules. After parsing, this information is converted into a task label set, which is usually a set of structured parameters used to express the expected content and constraints of state input and action output, such as whether angular velocity input is included, whether discrete or continuous action is required, etc. The label set can be a vector structure, a nested structure or a key-value mapping table.

[0105] Configuring the policy network structure of a reinforcement learning agent based on a set of task labels involves parameterizing the policy network architecture. In network design, the dimensions of the input layer must correspond one-to-one with each observable variable in the state space, such as position, velocity, and torque. The dimensions of the output layer must match the structure of the action space. If the policy output is a Gaussian distribution parameter, the output must include two channels: mean and variance. The policy network can adopt structures such as feedforward neural networks, convolutional neural networks, and transformer encoders, depending on the requirements for state dynamics and local perception capabilities in the task labels.

[0106] Constructing a set of environmental parameters consistent with the physical scenario of the target task enables the subsequent interactive environment to dynamically evolve within the constraints of the task. This set of environmental parameters typically includes physical layer parameters (such as gravity acceleration, friction coefficient, and connection stiffness), sensor model parameters (such as observation resolution, sampling frequency, and noise model), and dynamic constraints (such as maximum torque limits, joint limits, and system damping). These parameters are used to set the initial configuration of the environment simulator or real platform when constructing the environment simulation.

[0107] To generate the initial state distribution of the environment based on a set of environmental parameters, a probability density function (PDF) must be defined to describe the set of possible states the system can be in at the start of the reinforcement learning interaction. In practical implementations, Gaussian mixture models, Beta distributions, or empirical distribution estimation methods can be used to model the initial state distribution. This distribution ensures sufficient variability and coverage of the system state during each training session while remaining within the reasonable range for the task. Mathematically, this function is defined as a density function P0(s) on the state space S, which is used to initialize the starting state of the training trajectory.

[0108] Defining the mapping relationship between state variables and action variables based on a set of task tags and a set of environmental parameters is a prerequisite for achieving dynamic control capabilities within a strategy. State variables can include observable sensor outputs, historical trajectory information, or embedded environmental tags; action variables can be control signals, motor inputs, voltage commands, and so on. This state-action mapping relationship can be defined based on system modeling methods from control theory, such as first-order state-space models and second-order differential models. Alternatively, it can be directly embedded into the state transition calculation path by integrating with the interface logic of the environmental simulation platform.

[0109] The reinforcement learning agent's policy network parameters are initialized based on the policy network structure and the state transition probability function, ultimately obtaining the initial policy parameters. Initialization typically uses methods such as Gaussian distribution sampling, small-scale uniform distribution, and pre-trained weight transfer to assign values ​​to the neural network parameters. The state transition probability function provides a basis for modeling the next state in reinforcement learning. It can take the form of an analytical expression, a simulation inference function, or a modeler derived from data fitting. The final policy network and state model combine to form a complete agent structure, providing initial capabilities for interactive learning.

[0110] This embodiment systematically maps the structural information, physical properties, and interaction conditions of the target task into a set of parameters that can be used for network configuration and environmental modeling, so that the reinforcement learning agent has the ability to accurately characterize the dynamic characteristics of the task at the initial stage. The consistency of the policy network structure and the task label ensures the adaptation of the input and output channels, avoiding subsequent training failures or slow convergence problems caused by structural mismatch; the linkage definition of the state transition model and environmental parameters improves the authenticity and generalization of the environmental simulation, laying the foundation for subsequent efficient policy learning. The overall construction process realizes a structural closed loop from task information to the learning model, making the initial training goal-oriented and structurally interpretable, improving sample efficiency and reducing the uncertainty of policy convergence.

[0111] In one embodiment, the above step S20 includes:

[0112] S201, during the interaction between the reinforcement learning agent and the environment, obtaining the environment state observation at the current time step;

[0113] S202, generating an action distribution based on the initial strategy parameters and the environmental state observation at the current time step;

[0114] S203, sampling and generating an action execution instruction for the current time step from the action distribution;

[0115] S204, sending the action execution instruction to the robot execution module to drive the state transition of the environment;

[0116] S205, recording the action execution instructions of the current time step and the corresponding environmental state observations to form the action sequence data of the current time step;

[0117] S206 , accumulating the action sequence data of multiple time steps in the interaction process in order of time steps to generate a complete action sequence.

[0118] In this embodiment, the interaction process between the reinforcement learning agent and the environment is the basic process in the policy iterative learning. At each time step, state information needs to be perceived from the environment. The acquisition of environmental state observations can be collected through sensor readings, simulator state snapshots or external measurement systems. The state information can be a continuous variable vector (such as position, velocity, acceleration) or a discrete classification (such as the current task stage, semantic label). State observations are usually encoded as tensors in structure and input into the policy network, and may be normalized or embedded in the coding preprocessing.

[0119] The initial values ​​of the policy parameters come from the policy network structure during the initialization phase. Generating an action distribution based on these initial policy parameters and the current state observation involves inferring the probability distribution function of all possible actions in the current state through the policy network. For example, this outputs a softmax probability vector in discrete action space and the mean and variance parameters of a Gaussian distribution in continuous action space. This action distribution models the agent's behavioral preferences for the current state.

[0120] Sampling action execution instructions from the action distribution is a key step in converting probabilistic models into concrete control signals. Continuous actions can be generated using Bernoulli sampling, polynomial sampling, or reparameterization techniques. The sampling process may be based on the exploration level of the fixed temperature parameter adjustment strategy. In practice, threshold clipping may also be introduced to ensure the safety of the generated actions.

[0121] Action execution commands are sent via a communication interface to the robot's execution module, which includes subsystems such as the motion controller, force control interface, and motor drive. The execution module converts the action commands into low-level execution signals and drives physical transitions in the environment, such as movement of the end-arm or acceleration and steering of an autonomous vehicle. These state transitions are governed by the dynamics of the environment, and are typically nonlinear and time-varying.

[0122] After an action is executed, the action execution instruction and the new state observation at the current time step need to be recorded simultaneously to form a complete interactive data pair, which constitutes the action sequence data for the current time step. The action sequence data not only includes the executed action itself, but also records the difference in state before and after execution, as well as optional additional information such as timestamps or system resource consumption, which can be used for subsequent policy analysis and evaluation.

[0123] To form a complete training sample, the action sequence data generated during all interactions must be accumulated in time-step order to generate a complete action sequence. This sequence represents the trajectory of the agent's response to the environment during an interaction. It is typically encoded as a set of time-indexed state-action pairs, which may include sequences of varying lengths. This complete action sequence can be used to construct reward evaluation functions, determine policy performance, or train behavioral cloning networks.

[0124] This embodiment ensures that the agent's behavior at each time step can be tracked, analyzed, and monitored by constructing a continuous closed loop of state observation, policy output, action sampling, instruction execution, and data recording, thereby improving the integrity and effectiveness of data collection. The use of action distribution modeling rather than direct action output contributes to the controllability and exploration capabilities of the policy and facilitates subsequent policy uncertainty analysis and distribution alignment operations. The interaction data at each step not only reflects the behavioral preferences of the current policy but also provides high-quality, traceable training samples for subsequent policy optimization, thereby improving the stability of policy training and sample utilization efficiency.

[0125] In one embodiment, the above step S30 includes:

[0126] S301, generating an initial environment state by sampling from the initial state distribution;

[0127] S302, generating an action distribution for the first time step based on the initial environment state and initial strategy parameters;

[0128] S303, sampling from the action distribution to generate an action execution instruction for the first time step;

[0129] S304, determining a probability distribution of a next state based on the initial environment state, the action execution instruction, and the state transition probability function, and updating the environment state using a dynamic strategy described by the state transition probability function;

[0130] S305, sampling and generating a new environment state from the probability distribution of the next state, completing a single state transfer;

[0131] S306, determining the instantaneous reward value for the current time step according to a preset reward function;

[0132] S307, recording the initial environment state, action execution instructions, and corresponding instant reward values, and generating an interaction data unit for the first time step;

[0133] S308, repeatedly execute the steps of generating action distribution and sampling action execution instructions, updating environment status, determining immediate reward values, and recording interaction data units of the current time step until the target task is terminated, and integrate the interaction data units of all time steps in chronological order to generate an interaction trajectory containing a complete action sequence, state sequence, and reward data.

[0134] In this embodiment, during the interaction trajectory establishment process, the starting point of the interaction sequence is first determined by sampling from the state distribution defined during the environment initialization phase. The initial environment state is typically represented as a probability density function, which can be a Gaussian distribution, a uniform distribution, or a task-specific empirical distribution. It describes the robot's possible state space position at the beginning of the task, such as the initial pose of the end-of-arm or the initial camera observation frame.

[0135] In the initial state, the action distribution in the current state is calculated based on the initial policy parameters and state observations. The action distribution is output by the policy network, and its form depends on the structure of the action space. In discrete action spaces, this distribution is a class probability, while in continuous action spaces, it is a parameterized probability density model, such as a multi-dimensional Gaussian distribution.

[0136] From this action distribution, a specific action execution instruction is sampled, representing the behavior the agent should take in the current state. For example, in a robotic arm control task, this instruction might correspond to the target displacement of the end effector or the joint angular velocity vector. The sampling process typically introduces a policy temperature parameter to adjust the degree of exploration, and re-parameterization techniques are used when necessary to support gradient propagation.

[0137] The action execution instruction and the current state input state transition probability function are used to determine the probability distribution of the next state. The state transition function is a conditional probability mapping that characterizes how the environment state will be updated under a given state and action. This function can encode the system dynamics (such as Newtonian mechanics and collision modeling) through a simulator or be represented by an empirical modeling network to generate the state distribution after the transition. This distribution reflects the non-deterministic nature of the environment and is particularly important in medical robot operation or financial data environment modeling.

[0138] A new environment state is then sampled from the distribution of this next state, completing the state update for the current time step. This state represents the evolution of the system's controlled state and is used for subsequent reward calculations and policy updates. This operation can be performed using an environmental simulation platform such as Gazebo, MuJoCo, or a financial trading simulation engine, supporting nonlinear and multimodal state dynamics.

[0139] Based on the state-action pair, a preset reward function is used to calculate an immediate reward value. The reward function construction must take into account the task objectives, such as grasping accuracy, path smoothness, time efficiency, or cost savings. The reward value is often a scalar output, but it can also be a structured label set to support multi-objective optimization.

[0140] The initial state, action command, and corresponding reward value are combined to form an interaction data unit for a single time step. Each data unit records a set of state inputs, policy outputs, and environment feedback, and is the basic building block for constructing a trajectory.

[0141] Next, the process repeats the steps of action distribution generation, sampling, state update, reward calculation, and recording until a task termination condition is triggered, such as reaching the target state, exceeding the maximum time step, or satisfying the end signal. Finally, all time step data units are concatenated in chronological order to generate a complete interaction trajectory consisting of a sequence of states, actions, and rewards. This trajectory can be represented as a sequence of triples, which can be used for subsequent reinforcement learning policy optimization or data distillation.

[0142] This embodiment constructs a recursive closed loop of state sampling, policy response, state transition, reward calculation, and data recording, so that each time step has a complete mapping of state-action-reward, thereby constructing high-quality trajectory data that can be used for learning. This trajectory has a clear causal structure, which is conducive to gradient evaluation, reward attribution, and goal-oriented optimization of the policy, while maintaining consistency in the initial policy and environment modeling. The fully recorded action sequence, state sequence, and reward sequence can be reused in supervision processes such as data augmentation, behavior cloning, and multi-task learning, effectively improving sample utilization and policy transfer capabilities.

[0143] In one embodiment, the above step S40 includes:

[0144] S401, multiplying the reward data of each time step in the interaction trajectory by the corresponding discount factor power to generate a discounted cumulative reward value for each time step;

[0145] S402, constructing an objective function including a policy entropy regularization term based on the discounted cumulative reward value, initial policy parameters, action sequence, and state sequence;

[0146] S403, determining the partial derivative of the policy parameter with respect to the discounted cumulative reward value, and generating the gradient direction of the objective function through a policy gradient module;

[0147] S404, based on the gradient direction, updating the initial policy parameters by an adaptive learning rate optimizer, and iteratively performing parameter updates until a convergence condition is met;

[0148] S405: Generate an optimal action strategy based on the updated initial strategy parameters.

[0149] In this embodiment, when using interactive trajectories for strategy optimization, it is first necessary to process the reward data in the trajectory. For each time step in the trajectory, the instantaneous reward is multiplied by the power of the discount factor corresponding to the time step to calculate the discounted cumulative reward value of the time step. The discount factor is usually a real number between 0 and 1, which is used to adjust the weight relationship between the current reward and the future reward. This form of calculation emphasizes the contribution of current behavior to future long-term benefits and is the core link of stable strategy training in reinforcement learning. This operation not only enhances the temporal consistency of the reward signal, but also suppresses the policy disturbance caused by high variance returns in the long term.

[0150] To enhance the comprehensive assessment of long-term rewards during policy optimization, after acquiring interaction trajectories containing action sequences, state sequences, and reward data, the system first multiplies the immediate reward data at each time step by a decreasing discount weight to generate a discounted cumulative reward value corresponding to each time point. This reward value measures the potential future reward performance under different behavioral decision paths and constitutes the primary evaluation signal for policy learning. Furthermore, to further construct the optimization goal, the policy network parameters and the state and action information collected from the interaction trajectories are combined to form an optimization objective function. This objective function not only reflects the long-term expected reward of the policy during reinforcement learning but also incorporates an entropy regularization component to maintain a certain level of uncertainty in the policy output space and mitigate the lack of policy diversity caused by premature convergence. These two components are combined according to the predefined weights to form a weighted structure, allowing the objective function to simultaneously maximize reward and control the breadth of the policy search space.

[0151] After generating the objective function, the policy network's parameters are further evaluated for their influence on that objective function, calculating the partial derivatives between the parameters and the objective function. Based on this guidance, the policy gradient module determines the direction of parameter updates, and, combined with an adaptive adjustment mechanism, dynamically sets the learning rate to iteratively update the policy network. When parameter updates meet stability requirements, an optimal action policy is formed that offers high reward potential within the current environment and task structure and is less susceptible to local optima.

[0152] Subsequently, the initial policy parameters are updated using an adaptive learning rate optimizer based on the resulting gradient direction. This optimizer, which can use methods such as Adam, RMSProp, or SGD, adaptively adjusts the update amplitude in each dimension to achieve a more balanced convergence rate for the policy network across different dimensions of parameter space. This process is iteratively executed until the change in the loss function falls below a set threshold or the maximum number of iterations is reached, marking training convergence.

[0153] Finally, the updated policy parameters are reconfigured to the policy network to form an optimized optimal action policy. This policy has better reward expectations, robustness, and response efficiency in the original interactive environment, and can replace the initial policy for subsequent data generation, distillation training, or deployment applications.

[0154] This embodiment constructs an objective function based on the action sequence, state sequence and reward data in the interaction trajectory, and performs policy gradient updates on the weighted structure containing the entropy regularization term. It can improve the sensitivity to long-term rewards and the ability to explore the policy search space while maintaining the stability of the policy output. This process not only improves the convergence speed of the policy network in the training phase, but also enhances the generalization ability of the policy to the optimal action distribution under different environmental states. Furthermore, the introduction of an adaptive learning rate optimization mechanism can dynamically adjust the parameter update amplitude according to the gradient fluctuation to avoid overfitting local trajectory data and causing policy degradation. The optimal action strategy finally obtained has stronger reliability and efficiency in multi-task migration, distribution generalization and data distillation application scenarios, and significantly improves the stability and deployability of the strategy in high-dimensional dynamic environments.

[0155] In one embodiment, the above step S50 includes:

[0156] S501, loading the optimal action strategy into the strategy execution module of the reinforcement learning agent;

[0157] S502, generating an optimal action execution instruction for the current time step based on the current environment state observation and the optimal action strategy by the strategy execution module;

[0158] S503, driving the robot execution module to execute the optimal action execution instruction through the strategy execution module, updating the environment state and recording the environment state observation of the current time step;

[0159] S504, storing the optimal action execution instructions and the corresponding environmental state observations in time step order to generate a single-step experience data unit;

[0160] S505, repeatedly performing the operations of generating the optimal action execution instruction, updating the environment state, and generating the single-step experience data unit until the target task is terminated, integrating all the single-step experience data units in chronological order to generate an integrated experience data unit;

[0161] S506 , performing format standardization processing on the integrated experience data units to generate an original experience data set containing a complete state sequence and action sequence.

[0162] In this embodiment, in a reinforcement learning system, using the parameters generated by the strategy for playback in a real or simulated environment is a common means of strategy verification and experience collection. In order to apply the optimal action strategy to the data generation process, the strategy parameters must first be loaded into an execution unit with reasoning capabilities. As a strategy execution module, the execution unit has the ability to extract features from the input state information, map it, and output strategy actions. The loading operation not only includes the initialization of the strategy parameters, but also involves the synchronous configuration of the network structure and input dimensions to ensure that the input state dimensions in the subsequent processing link are consistent with the model presets.

[0163] Based on the loaded policy execution module, the system collects state observations from the environment at each time step, feeds them into the policy network for inference, and generates the optimal action execution instructions for the current time step. These action instructions are based on the maximum probability or sample value in the policy distribution and are converted by a parser into control instructions recognizable by the actuator, such as position movement, grasping posture, and path point planning. These generated instructions are then passed to the execution interface of the physical robot or simulation platform, which, through real-time control, drives the underlying execution module to respond to the current environmental state.

[0164] After each action is executed, the environment updates its current state based on the action's results. This state may be represented by visual observations, sensor readings, positional parameters, and so on. This updated state observation is combined with the action execution instructions at a time step to form a single-step experience data unit. This data structure typically includes identification fields such as state input, action output, and execution time index to facilitate subsequent retrieval, training, and visualization.

[0165] This process continues iteratively until the task is complete. During the iterations, the single-step data units generated at each time step are appended to the data cache. After the task is completed, the system structurally integrates all single-step data in time step order to form a sequence of original experience trajectories. This sequence constitutes a complete record of a valid task execution, providing replayability and training usability.

[0166] To ensure data compatibility and structural consistency in subsequent use, the integrated data must be formatted and standardized. Standardization operations can include field alignment, data normalization, data type conversion, and redundant field cleaning. The output is a standardized raw empirical dataset that meets the requirements of supervised learning or distillation training interfaces.

[0167] This embodiment can efficiently collect real interaction data driven by the optimized strategy by loading the optimal action strategy into the strategy execution module and executing it in the environment, avoiding relying on invalid trajectories caused by random exploration, thereby significantly improving the quality and goal orientation of the data. During the collection process, the linkage mechanism of action execution and state observation can form a complete time series sample, enhancing the ability to characterize complex behavioral processes. Through standardization, the original trajectories in different task instances are unified into a structurally consistent empirical data set, which is conducive to batch training in the subsequent supervised learning stage, and improves data utilization and migration generalization capabilities. At the same time, the process can be repeatedly run without additional human intervention, and is suitable for generating large-scale stable interaction sample libraries under multiple scenarios and multiple strategy versions, improving the efficiency of data distillation and the robustness of strategy training.

[0168] In one embodiment, the above step S60 includes:

[0169] S601, training independent reinforcement learning agents for multiple operation objects other than the current operation object in the target task, and generating corresponding independent experience data sets;

[0170] S602, merging the original experience dataset and multiple independent experience datasets of other operation objects to form a balanced dataset;

[0171] S603 , performing supervised learning on the pre-trained policy model using the balanced data set, optimizing the parameters of the policy model by minimizing the log-likelihood loss of the action probability distribution, and generating a target policy model.

[0172] In this embodiment, in the supervised learning stage, in order to make the pre-trained policy model more generalizable, it is first necessary to obtain input and output samples containing diverse behavior patterns. In addition to the original experience data set generated by the current operation object, in order to cover different operation scenarios and target structures, multiple other operation objects are introduced as experience acquisition sources. Each operation object refers to a robot task instance with an independent task structure or control constraint, such as different types of robotic arms, different load conditions or different execution path requirements. For each object, an independent reinforcement learning agent is constructed, and independent training is performed based on its task parameters, control interface and reward feedback mechanism, and interaction data that can be used for offline supervised learning is output. This process can be automatically generated using heterogeneous environment simulation, distributed training framework or preset policy templates, and ultimately forms multiple independent experience data sets with consistent structure but diverse behaviors.

[0173] To construct a set of training samples that simultaneously covers the behavioral characteristics of the current task and the diversity of other task behaviors, the original experience dataset is merged with the independent experience datasets of all other subjects to construct a balanced dataset. This balancing process can reweight, oversample, or undersample samples based on category frequency, ensuring that the influence of each type of data on the training process is nearly equal in the final training set. In addition, to avoid data conflicts and feature shifts, the merging process must ensure that the state space dimensions are unified, the action representations are consistent, and the data from different sources are identified and encoded.

[0174] When using a pre-trained policy model for supervised learning using a balanced dataset, the output of the current policy model is compared to the action instructions recorded in the empirical data. The log-likelihood loss is calculated between the action distributions to measure the deviation between the policy output and high-quality behavior. This log-likelihood loss is suitable for scenarios where the policy model outputs a multi-category discrete action distribution and can also be extended to fit Gaussian distribution parameters for continuous action spaces. During training, the model parameters are updated through backpropagation, allowing it to adapt to a wider range of behavioral patterns based on multi-source experience while maintaining recognition capabilities for the original task.

[0175] The optimized policy model, known as the target policy model, combines pre-trained parameters with behavioral abstractions from multiple tasks, resulting in stronger generalization and transferability across multiple scenarios. This model can be deployed on various robotic platforms and can also serve as an initialization policy for further reinforcement learning fine-tuning.

[0176] For example, first, one or more specialized reinforcement learning (RL) agents are trained independently for a specific robotics task or a key sub-part of a task. These agents learn the optimal behavior policy π(a t |s t ), the goal is to maximize the cumulative expected return objective function:

[0177]

[0178] Where J(π): The objective function of strategy π, which represents the expected cumulative reward. π: The policy, which maps the state to the probability distribution of the selected action. π(a t |s t ): Strategy π in state s t Next select action a t The probability of . E[·]: mathematical expectation. s0: initial state. ρ0: distribution of initial state s0. a t : The action performed at time step t. s t : The state at time step t. s t+1 : Perform action a at time step t tThen transfer to the next state. P(s t+1 |s t ,a t ):State transition probability, indicating that in state s t Execute action a t Then, transfer to state s t+1 probability. The cumulative sum from time step t = 0 to infinity. γ: discount factor, ranging from [0, 1], used to measure the importance of future rewards relative to current rewards. γ t is the discount factor raised to the power of time step t. R(s t ,a t ):reward function, indicating the reward function in state s t Execute action a t Instant rewards after.

[0179] In this way, RL can autonomously learn action sequences that are more optimized and consistent than human demonstrations, especially in tasks requiring extremely high accuracy. This stage of training can be performed using the HIL-SERL framework to achieve sample-efficient and high-performance real-world manipulation skill learning.

[0180] By executing the converged RL policy, a large amount of experience data is generated, including information such as state, action, and reward, to form a balanced and diverse high-quality fine-tuning dataset. For tasks involving multiple operating objects, such as connector insertion tasks, an RL policy is trained separately for each operating object, and the data generated by these policies are merged into a balanced dataset. In addition, in certain long-term tasks, it is possible to choose to apply the RL policy to data collection only for the key subtask segments that require the highest accuracy, while the rest can use human demonstration data. This not only simplifies the complexity of RL training, but also improves the expressiveness and diversity of general policies, while avoiding wasting too many resources on unnecessary long-term tasks.

[0181] Finally, the collected high-quality RL experience dataset D(s t ,a t ) is used to fine-tune a pre-trained large-scale general-purpose robotics policy model (such as Octo). The fine-tuning process uses supervised learning, with the goal of making the general policy imitate the actions produced by the RL expert policy. The loss function is usually the log-likelihood loss:

[0182]

[0183] in, Loss function, used to measure the policy π θThe goal is to minimize the consistency between the expert data D. θ: general policy π θ Parameters. E[·]: mathematical expectation. (s t ,a t ): state-action pairs sampled from the experience dataset D. D: high-quality RL experience dataset, containing (s t ,a t ) trajectory. θ (a t |s t ): The general policy with parameter θ is in state s t Next output action a t probability.

[0184] Fine-tuning with RL-generated data can significantly improve the performance and success rate of general policies in precision manipulation tasks. Furthermore, due to the generalization capabilities of the underlying model, the final policy achieves professional controller-level performance while maintaining flexibility, and even demonstrates better generalization capabilities than the original RL policy in some aspects.

[0185] Example: In a multi-task industrial operation environment, a general robot control policy needs to be trained for object handling, part assembly, and button manipulation. The tasks involve recognizing and processing multiple objects (e.g., objects of varying sizes and materials), requiring the policy model to transfer and generalize well across target tasks. First, for a specific target task (e.g., a button press), the system analyzes the physical structure of the target scene, including key elements such as the button's spatial location, spring force feedback, and press depth. This generates a set of task labels, encompassing key action constraints and force control constraints in the interaction. Subsequently, the policy network architecture is configured based on the task label set: the input layer's dimensions correspond to the observation space (e.g., end-effector pose, camera image, force sensor data), and the output layer corresponds to the robot's motion control space (e.g., six-degree-of-freedom motion commands, gripping force instructions, etc.). Based on physical parameters (e.g., friction coefficient, stiffness, gravity parameters) and sensor models from the simulator or actual controller, a set of environmental parameters is constructed to simulate the task scenario. This is followed by the generation of an initial state distribution (e.g., the robot's starting pose distribution) and a state transition probability function (a conditional probability model describing the change in the end-effector state after executing an action). After loading the reinforcement learning agent, the system begins interacting with the environment. At each time step, the robot generates an action distribution based on current observations (such as visual images, end-user position, and joint angles) through the policy network. It then samples specific action execution instructions (such as moving the end-user down 3 cm or pressing with a force of 2 Newtons). These instructions are sent to the robot's execution module via the controller, driving it to perform real or simulated actions. After execution, the new state observations are recorded, and an immediate reward is calculated based on the quality of the action execution (such as whether a button was successfully pressed). The state, action, and reward triples are recorded and integrated in chronological order to form a complete interaction trajectory. Next, the reward for each time step in the interaction trajectory is multiplied by a predefined discount factor (for example, a decreasing factor of 0.98) to calculate the discounted cumulative reward for each time step. An objective function is constructed based on this reward and the sequence of states and actions collected in the trajectory. This function also includes a policy entropy regularization term to maintain the policy's exploration capability. This objective function measures the deviation between the policy network's output distribution and high-quality behavioral data. Subsequently, the policy gradient module calculates the partial derivatives of the policy parameters, representing the trend of the reward function with respect to the parameters. Based on this, the policy parameters are iteratively optimized over multiple rounds using an adaptive learning rate optimizer (such as Adam or RMSProp). Training is considered converged when the loss trend stabilizes or reaches the set number of iterations, resulting in an optimal action policy that performs better for the target task. The resulting optimal action policy is then loaded back into the policy execution module, deployed in the robot control system, and the task is re-executed to generate higher-quality operation trajectories.These trajectories consist of execution instructions and state observations, organized into single-step experience data units at time steps. These units are then integrated into a new raw experience dataset, and the data format is standardized (e.g., unifying observation dimensions and action encoding). To further enhance the policy's versatility and generalization capabilities, multiple reinforcement learning agents are trained on other manipulation tasks (e.g., moving different objects, assembling different components, and performing different action sequences), generating independent experience datasets. These datasets are then combined with the raw experience dataset from the current task to form a multi-task balanced dataset. This dataset is then used to train a pre-trained general policy model for supervised learning. The policy model parameters are optimized by minimizing the log-likelihood loss between action probability distributions, enabling the model to more accurately map the multi-task states to optimal actions. Ultimately, this target policy model can be directly deployed into the robot control system, exhibiting enhanced transferability and robustness to new, unseen tasks or changing conditions. It enables rapid adaptation and reliable execution across multiple object classes and various manipulation objectives, effectively reducing the burden of repetitive single-task training and improving the efficiency of training data utilization.

[0186] In the task of identifying fraudulent financial transactions, the system requires training an intelligent agent capable of cross-account and multi-channel behavior recognition to automatically decide whether to intercept transactions. First, based on the structural characteristics of the transaction process and anti-fraud rules, a state space is defined, including dimensions such as transaction time, transaction amount, and account behavior score, and an action space, including options such as release, delayed review, and interception. State transition rules are then constructed, and a reinforcement learning agent policy is initialized. Subsequently, through interaction with real or simulated trading environments, state observations are obtained under different trading scenarios. Based on the current policy, actions are generated and executed, such as releasing or intercepting a transaction. The system records the state, action, and reward signal corresponding to each step, indicating whether fraud was successfully identified, to construct a complete interaction trajectory. Based on this interaction trajectory, the discounted cumulative reward corresponding to each transaction behavior is calculated and combined with the action state to construct an objective function. Entropy regularization is applied to maintain the policy's exploratory nature. Gradient calculation and iterative optimization of policy parameters are then performed to ultimately obtain a more optimal action policy for the current financial environment. The optimized policy is then deployed on a financial simulation platform to interact with a large amount of trading data, generating a raw empirical dataset consisting of real state and action sequences. We further introduce independent datasets generated by trading agent models across multiple regions and different financial product lines and merge them with the original agent dataset to form a balanced dataset with more balanced sample distribution and more accurate labels. Based on this balanced dataset, we perform supervised learning on the pre-trained unified financial risk control strategy model. The model parameters are updated by minimizing the log-likelihood loss function of the action probability distribution, forming a highly generalizable risk control strategy model that can adapt to multiple scenarios and multiple account structures.

[0187] During chronic disease follow-up in healthcare services, patient medication adherence behavior is modeled to enhance the personalized intervention recommendation delivery system. Task labels, such as the patient's disease course, rate of change of physiological indicators, and past medication responses, are first extracted to define a state space. Actions include prompting medication, contacting a doctor, and postponing recommendations. A state transition model and policy network are then constructed. Through interaction with a simulated follow-up environment, patient response feedback states are collected and corresponding actions are generated. After each interaction, the system provides a reward signal based on whether the patient's behavior improves, with positive rewards for improved indicators and penalties for worsening indicators. This interaction data is used to generate complete trajectory information. The objective function is generated by combining reward and state-action, and an entropy regularization term is incorporated to construct the optimization objective. Parameters are updated using policy gradients to obtain a preliminary optimized policy. This policy is then repeatedly executed in simulated or real patient scenarios, generating new high-quality behavioral records and constructing a raw empirical dataset. To improve the policy's generalization across diverse disease types and age groups, multiple reinforcement learning agents are trained from multiple hospital or community health management systems, such as for hypertension, diabetes, the elderly, and the young, and their behavioral data is extracted. These independent datasets are merged with the original empirical dataset to construct a more comprehensive balanced dataset. Ultimately, this balanced data is used to conduct supervised learning on the pre-trained health intervention strategy model, enhancing its accuracy in recommending interventions across different disease cycles and patient responses.

[0188] This embodiment introduces multiple other operation objects in addition to the current operation object and trains the reinforcement learning agent to generate independent experience data respectively, which can significantly expand the behavioral coverage of the training data so that the model is not limited to specific tasks or specific action sequences. By merging and constructing a balanced data set, the problem of single-task data bias is effectively alleviated, and the model's ability to understand multi-task collaborative strategies is improved. In supervised training, the log-likelihood loss is used as the optimization target, and the historical optimal behavior distribution can be fully utilized as a label to guide strategy convergence, thereby improving training stability and sample efficiency. The target strategy model finally optimized has stronger robustness, generalization ability and deployment adaptability when facing tasks under different structures, scenarios or constraints, which helps to achieve the goal of strategy generalization and reduce the cost of manual strategy fine-tuning.

[0189] In one embodiment, a strategy optimization device based on interaction trajectory is provided, and the strategy optimization device based on interaction trajectory corresponds to the strategy optimization method based on interaction trajectory in the above embodiment. Figure 3 , Figure 3 This is a functional module diagram of a preferred embodiment of the interactive trajectory-based strategy optimization device of the present invention. It includes an environment modeling module 10, an interactive decision module 20, a trajectory generation module 30, a strategy optimization module 40, a strategy execution module 50, and a model optimization module 60. Each functional module is described in detail below:

[0190] The environment modeling module 10 is used to define the initial state distribution and state transition probability function of the environment based on the target task, initialize the policy network parameters of the reinforcement learning agent, and generate initial policy parameters;

[0191] an interactive decision module 20, configured to interact with the environment through the reinforcement learning agent and generate an action sequence based on the initial strategy parameters;

[0192] a trajectory generation module 30 for updating the state sequence of the environment according to the initial state distribution and the state transition probability function, recording the action sequence, state sequence and corresponding reward data, and generating an interaction trajectory according to the recorded sequence and data;

[0193] a strategy optimization module 40 for determining a cumulative expected reward function based on the action sequence, state sequence, and reward data in the interaction trajectory, and updating the initial strategy parameters by maximizing the gradient of the cumulative expected reward function to generate an optimal action strategy;

[0194] A strategy execution module 50 is used to execute the optimal action strategy and generate an original experience data set including a state sequence and an action sequence of the environment;

[0195] The model optimization module 60 is used to perform supervised learning on the pre-trained policy model based on the original experience data set, optimize the parameters of the policy model, and generate a target policy model.

[0196] In one embodiment, the environment modeling module 10 is specifically configured to:

[0197] Analyze the physical scene and task structure information of the target task and generate a set of task labels;

[0198] Configuring a policy network structure of the reinforcement learning agent based on the task label set;

[0199] Construct a set of environmental parameters that are consistent with the physical scenario of the target task;

[0200] generating an initial state distribution of the environment based on the set of environmental parameters;

[0201] Defining a mapping relationship between state variables and action variables according to the task tag set and the environment parameter set, and constructing a state transition probability function based on the mapping relationship;

[0202] Based on the policy network structure and the state transition probability function, the policy network parameters of the reinforcement learning agent are initialized to generate initial policy parameters.

[0203] In one embodiment, the interactive decision module 20 is specifically configured to:

[0204] During the interaction between the reinforcement learning agent and the environment, obtain the environment state observation at the current time step;

[0205] generating an action distribution based on the initial policy parameters and the environment state observation at the current time step;

[0206] Sampling from the action distribution to generate an action execution instruction for the current time step;

[0207] Sending the action execution instruction to the robot execution module to drive the state transition of the environment;

[0208] Record the action execution instructions of the current time step and the corresponding environmental state observations to form the action sequence data of the current time step;

[0209] The action sequence data of multiple time steps in the interaction process are accumulated in time step order to generate a complete action sequence.

[0210] In one embodiment, the trajectory generation module 30 is specifically configured to:

[0211] Sampling an initial environment state from the initial state distribution;

[0212] generating an action distribution for the first time step based on the initial environment state and initial policy parameters;

[0213] Sampling from the action distribution to generate an action execution instruction for the first time step;

[0214] Determine the probability distribution of the next state based on the initial environment state, the action execution instruction and the state transition probability function, and update the environment state using the dynamic strategy described by the state transition probability function;

[0215] Sampling a new environment state from the probability distribution of the next state to complete a single state transfer;

[0216] Determine the instantaneous reward value of the current time step according to the preset reward function;

[0217] Recording the initial environment state, action execution instructions, and corresponding immediate reward values ​​to generate an interaction data unit for the first time step;

[0218] Repeat the steps of generating action distribution, sampling action execution instructions, updating environment state, determining immediate reward value, and recording interaction data units at the current time step until the target task is terminated. The interaction data units of all time steps are integrated in chronological order to generate an interaction trajectory containing a complete action sequence, state sequence, and reward data.

[0219] In one embodiment, the policy optimization module 40 is specifically configured to:

[0220] Multiplying the reward data of each time step in the interaction trajectory by the corresponding discount factor power to generate a discounted cumulative reward value for each time step;

[0221] Constructing an objective function including a policy entropy regularization term based on the discounted cumulative reward value, the initial policy parameters, the action sequence, and the state sequence;

[0222] Determine the partial derivatives of the policy parameters with respect to the discounted cumulative reward value, and generate the gradient direction of the objective function through the policy gradient module;

[0223] Based on the gradient direction, the initial policy parameters are updated by an adaptive learning rate optimizer, and the parameter updates are iteratively performed until the convergence conditions are met;

[0224] Generate the optimal action strategy based on the updated initial strategy parameters.

[0225] In one embodiment, the policy execution module 50 is specifically configured to:

[0226] Loading the optimal action strategy into the strategy execution module of the reinforcement learning agent;

[0227] Generate an optimal action execution instruction for the current time step based on the current environment state observation and the optimal action strategy through the strategy execution module;

[0228] The strategy execution module drives the robot execution module to execute the optimal action execution instruction, update the environment state and record the environment state observation of the current time step;

[0229] Store the optimal action execution instructions and the corresponding environmental state observations in time step order to generate single-step experience data units;

[0230] Repeat the operations of generating the optimal action execution instruction, updating the environment state, and generating the single-step experience data unit until the target task is terminated, and integrate all the single-step experience data units in chronological order to generate an integrated experience data unit;

[0231] The format of the integrated experience data units is standardized to generate an original experience dataset containing a complete state sequence and action sequence.

[0232] In one embodiment, the model optimization module 60 is specifically configured to:

[0233] Training independent reinforcement learning agents for multiple operation objects other than the current operation object in the target task to generate corresponding independent experience data sets;

[0234] Merging the original experience dataset with a plurality of independent experience datasets of other operation objects to form a balanced dataset;

[0235] The pre-trained policy model is supervised by learning the balanced data set, and the parameters of the policy model are optimized by minimizing the log-likelihood loss of the action probability distribution to generate a target policy model.

[0236] In one embodiment, a determination device is provided. The determination device may be a server, and its internal structure diagram may be as follows: Figure 4 As shown. The determination machine device includes a processor, a memory, a network interface and a database connected via a system bus. Among them, the processor of the determination machine device is used to provide determination and control capabilities. The memory of the determination machine device includes a non-volatile and / or volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a determination machine program and a database. The internal memory provides an environment for the operation of the operating system and the determination machine program in the non-volatile storage medium. The network interface of the determination machine device is used to communicate with an external user terminal through a network connection. When the determination machine program is executed by the processor, it realizes the functions or steps on the server side of a strategy optimization method based on interactive trajectories.

[0237] In one embodiment, a determination device is provided. The determination device may be a user terminal, and its internal structure diagram may be as follows: Figure 5 As shown. The determination machine device includes a processor, memory, network interface, display screen and input device connected via a system bus. Among them, the processor of the determination machine device is used to provide determination and control capabilities. The memory of the determination machine device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a determination machine program. The internal memory provides an environment for the operation of the operating system and the determination machine program in the non-volatile storage medium. The network interface of the determination machine device is used to communicate with an external server through a network connection. When the determination machine program is executed by the processor, it realizes the functions or steps on the user side of a strategy optimization method based on interactive trajectories.

[0238] In one embodiment, a determination machine device is provided, including a memory, a processor, and a determination machine program stored in the memory and executable on the processor. When the processor executes the determination machine program, the following steps are implemented:

[0239] Based on the target task, the initial state distribution and state transition probability function of the environment are defined, and the policy network parameters of the reinforcement learning agent are initialized to generate the initial policy parameters.

[0240] interacting with the environment through the reinforcement learning agent and generating an action sequence based on the initial policy parameters;

[0241] According to the initial state distribution and the state transition probability function, the state sequence of the environment is updated, the action sequence, the state sequence and the corresponding reward data are recorded, and an interaction trajectory is generated according to the recorded sequence and data;

[0242] Determine a cumulative expected reward function based on the action sequence, state sequence, and reward data in the interaction trajectory, and update the initial strategy parameters by maximizing the gradient of the cumulative expected reward function to generate an optimal action strategy;

[0243] Executing the optimal action strategy to generate an original experience data set including a state sequence and an action sequence of the environment;

[0244] The pre-trained strategy model is supervised and learned based on the original experience data set, the parameters of the strategy model are optimized, and a target strategy model is generated.

[0245] In one embodiment, a determination machine readable storage medium is provided, on which a determination machine program is stored. When the determination machine program is executed by a processor, the following steps are implemented:

[0246] Based on the target task, the initial state distribution and state transition probability function of the environment are defined, and the policy network parameters of the reinforcement learning agent are initialized to generate the initial policy parameters.

[0247] interacting with the environment through the reinforcement learning agent and generating an action sequence based on the initial policy parameters;

[0248] According to the initial state distribution and the state transition probability function, the state sequence of the environment is updated, the action sequence, the state sequence and the corresponding reward data are recorded, and an interaction trajectory is generated according to the recorded sequence and data;

[0249] Determine a cumulative expected reward function based on the action sequence, state sequence, and reward data in the interaction trajectory, and update the initial strategy parameters by maximizing the gradient of the cumulative expected reward function to generate an optimal action strategy;

[0250] Executing the optimal action strategy to generate an original experience data set including a state sequence and an action sequence of the environment;

[0251] The pre-trained strategy model is supervised and learned based on the original experience data set, the parameters of the strategy model are optimized, and a target strategy model is generated.

[0252] It should be noted that the above functions or steps that can be implemented by the machine-readable storage medium or the machine device can be referred to the relevant descriptions on the server side and the user side in the aforementioned method embodiment. To avoid repetition, they will not be described one by one here.

[0253] Those skilled in the art will understand that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a determination machine program, and the determination machine program can be stored in a non-volatile determination machine-readable storage medium. When the determination machine program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), Synchronous Link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0254] Those skilled in the art will clearly understand that for the sake of convenience and brevity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0255] It should be noted that if any software tools or components other than those of the Company appear in the embodiments of this application, they are merely for illustration and do not represent actual use. The above embodiments are intended only to illustrate the technical solutions of the present invention, not to limit them. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or replace some of the technical features therein with equivalents. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included in the scope of protection of the present invention.

Claims

1. A strategy optimization method based on interactive trajectory, characterized in that: The following steps are involved: Based on the target task, the initial state distribution and state transition probability function of the environment are defined, and the policy network parameters of the reinforcement learning agent are initialized to generate the initial policy parameters. interacting with the environment through the reinforcement learning agent and generating an action sequence based on the initial policy parameters; According to the initial state distribution and the state transition probability function, the state sequence of the environment is updated, the action sequence, the state sequence and the corresponding reward data are recorded, and an interaction trajectory is generated according to the recorded sequence and data; Determine a cumulative expected reward function based on the action sequence, state sequence, and reward data in the interaction trajectory, and update the initial strategy parameters by maximizing the gradient of the cumulative expected reward function to generate an optimal action strategy; Executing the optimal action strategy to generate an original experience data set including a state sequence and an action sequence of the environment; The pre-trained strategy model is supervised and learned based on the original experience data set, the parameters of the strategy model are optimized, and a target strategy model is generated.

2. The interactive trajectory-based strategy optimization method according to claim 1, characterized in that: Based on the target task, define the initial state distribution and state transition probability function of the environment, initialize the policy network parameters of the reinforcement learning agent, and generate the initial policy parameters, including: Analyze the physical scene and task structure information of the target task and generate a set of task labels; Configuring a policy network structure of the reinforcement learning agent based on the task label set; Construct a set of environmental parameters that are consistent with the physical scenario of the target task; generating an initial state distribution of the environment based on the set of environmental parameters; Defining a mapping relationship between state variables and action variables according to the task tag set and the environment parameter set, and constructing a state transition probability function based on the mapping relationship; Based on the policy network structure and the state transition probability function, the policy network parameters of the reinforcement learning agent are initialized to generate initial policy parameters.

3. The interactive trajectory-based strategy optimization method according to claim 1, characterized in that: Interacting with the environment through the reinforcement learning agent and generating an action sequence based on the initial strategy parameters, including: During the interaction between the reinforcement learning agent and the environment, obtain the environment state observation at the current time step; generating an action distribution based on the initial policy parameters and the environment state observation at the current time step; Sampling from the action distribution to generate an action execution instruction for the current time step; Sending the action execution instruction to the robot execution module to drive the state transition of the environment; Record the action execution instructions of the current time step and the corresponding environmental state observations to form the action sequence data of the current time step; The action sequence data of multiple time steps in the interaction process are accumulated in time step order to generate a complete action sequence.

4. The interactive trajectory-based strategy optimization method according to claim 1, wherein: According to the initial state distribution and the state transition probability function, the state sequence of the environment is updated, the action sequence, state sequence and corresponding reward data are recorded, and an interaction trajectory is generated according to the recorded sequence and data, including: Sampling an initial environment state from the initial state distribution; generating an action distribution for the first time step based on the initial environment state and initial policy parameters; Sampling from the action distribution to generate an action execution instruction for the first time step; Determine the probability distribution of the next state based on the initial environment state, the action execution instruction and the state transition probability function, and update the environment state using the dynamic strategy described by the state transition probability function; Sampling a new environment state from the probability distribution of the next state to complete a single state transfer; Determine the instantaneous reward value of the current time step according to the preset reward function; Recording the initial environment state, action execution instructions, and corresponding immediate reward values ​​to generate an interaction data unit for the first time step; Repeat the steps of generating action distribution, sampling action execution instructions, updating environment state, determining immediate reward value, and recording interaction data units at the current time step until the target task is terminated. The interaction data units of all time steps are integrated in chronological order to generate an interaction trajectory containing a complete action sequence, state sequence, and reward data.

5. The interactive trajectory-based strategy optimization method according to claim 1, characterized in that: Based on the action sequence, state sequence and reward data in the interaction trajectory, a cumulative expected reward function is determined, and the initial strategy parameters are updated by maximizing the gradient of the cumulative expected reward function to generate an optimal action strategy, including: Multiplying the reward data of each time step in the interaction trajectory by the corresponding discount factor power to generate a discounted cumulative reward value for each time step; Constructing an objective function including a policy entropy regularization term based on the discounted cumulative reward value, the initial policy parameters, the action sequence, and the state sequence; Determine the partial derivatives of the policy parameters with respect to the discounted cumulative reward value, and generate the gradient direction of the objective function through the policy gradient module; Based on the gradient direction, the initial policy parameters are updated by an adaptive learning rate optimizer, and the parameter updates are iteratively performed until the convergence conditions are met; Generate the optimal action strategy based on the updated initial strategy parameters.

6. The interactive trajectory-based strategy optimization method according to claim 1, characterized in that: Execute the optimal action strategy to generate an original experience dataset containing the state sequence and action sequence of the environment, including: Loading the optimal action strategy into the strategy execution module of the reinforcement learning agent; Generate an optimal action execution instruction for the current time step based on the current environment state observation and the optimal action strategy through the strategy execution module; The strategy execution module drives the robot execution module to execute the optimal action execution instruction, update the environment state and record the environment state observation of the current time step; Store the optimal action execution instructions and the corresponding environmental state observations in time step order to generate single-step experience data units; Repeat the operations of generating the optimal action execution instruction, updating the environment state, and generating the single-step experience data unit until the target task is terminated, and integrate all the single-step experience data units in chronological order to generate an integrated experience data unit; The format of the integrated experience data units is standardized to generate an original experience dataset containing a complete state sequence and action sequence.

7. The interactive trajectory-based strategy optimization method according to claim 1, wherein: Performing supervised learning on the pre-trained policy model according to the original experience data set, optimizing the parameters of the policy model, and generating a target policy model, including: Training independent reinforcement learning agents for multiple operation objects other than the current operation object in the target task to generate corresponding independent experience data sets; Merging the original experience dataset with a plurality of independent experience datasets of other operation objects to form a balanced dataset; The pre-trained policy model is supervised by learning the balanced data set, and the parameters of the policy model are optimized by minimizing the log-likelihood loss of the action probability distribution to generate a target policy model.

8. A strategy optimization device based on interactive trajectory, characterized in that: The interactive trajectory-based strategy optimization device includes: The environment modeling module is used to define the initial state distribution and state transition probability function of the environment based on the target task, initialize the policy network parameters of the reinforcement learning agent, and generate the initial policy parameters; an interactive decision module, configured to interact with the environment through the reinforcement learning agent and generate an action sequence based on the initial strategy parameters; a trajectory generation module, configured to update the state sequence of the environment according to the initial state distribution and the state transition probability function, record the action sequence, state sequence and corresponding reward data, and generate an interaction trajectory according to the recorded sequence and data; a strategy optimization module, configured to determine a cumulative expected reward function based on the action sequence, state sequence, and reward data in the interaction trajectory, and to update the initial strategy parameters by maximizing the gradient of the cumulative expected reward function to generate an optimal action strategy; A strategy execution module, configured to execute the optimal action strategy and generate an original experience data set including a state sequence and an action sequence of the environment; The model optimization module is used to perform supervised learning on the pre-trained strategy model based on the original experience data set, optimize the parameters of the strategy model, and generate a target strategy model.

9. A determination device, characterized in that: The determination machine device includes a memory, a processor, and an interaction trajectory-based policy optimization program stored in the memory and capable of running on the processor. When the interaction trajectory-based policy optimization program is executed by the processor, the steps of the interaction trajectory-based policy optimization method as described in any one of claims 1 to 7 are implemented.

10. A machine-readable storage medium, characterized in that: The storage medium stores a strategy optimization program based on an interaction trajectory. When the strategy optimization program based on an interaction trajectory is executed by a processor, the steps of the strategy optimization method based on an interaction trajectory as claimed in any one of claims 1 to 7 are implemented.

Images