A reinforcement learning method and system based on structural causal decoupling and a storage medium

Abstract

The application relates to the technical field of reinforcement learning, and discloses a reinforcement learning method and system based on structural causal decoupling and a storage medium, which comprises the following steps: receiving environment observation data, processing the environment observation data through a parameterized encoder to generate a latent state; inputting the latent state and an action variable into a double-channel prediction network, generating an environment spontaneous evolution state through an environment channel, generating an intervention state through an intervention channel, and then generating a next-time prediction latent state; calculating an intrinsic reward of motility based on the intervention state; constructing a joint loss function by using a state prediction residual, conducting gradient parameters corresponding to the residual to the intervention channel for network updating in gradient back propagation; calling the double-channel prediction network to perform Monte Carlo tree search, and selecting a final planning action by fusing the intrinsic reward of motility. The application realizes feature-level decoupling of environment evolution and action intervention, eliminates background noise interference, and improves the reliability of action planning.

Description

Technical Field

[0001] This invention relates to the field of reinforcement learning technology, specifically to a reinforcement learning method, system, and storage medium based on structural causal decoupling. Background Technology

[0002] In reinforcement learning techniques, agents typically construct dynamic environmental models to infer states and plan actions. Existing dynamic prediction models usually treat the environmental evolution process as a whole coupled system, directly inputting the state observation data and action variables of the current time step into a single network structure to predict the environmental state at the next moment. The above processing structure cannot distinguish between the inherent spontaneous evolution of the environment and the causal intervention caused by the agent's actions.

[0003] Due to the lack of structural-level causal decoupling, when there is random background noise or uncontrollable dynamic objects in the physical environment, existing prediction models will confuse the state changes caused by uncontrollable factors with the state changes directly caused by action variables. In exploration tasks lacking external environmental rewards, agents typically use state prediction errors as intrinsic reward signals. The above confusion phenomenon will cause abnormal prediction residuals caused by environmental background noise, resulting in the calculated intrinsic reward containing interference values ​​not caused by actions. Interference values ​​will induce agents to spend exploration steps on continuously observing random background noise, resulting in low efficiency of action exploration.

[0004] Furthermore, when combining Monte Carlo tree search with long-term action planning, existing prediction networks cannot isolate the effects of random environmental changes. The state deduction process directly introduces prediction bias, and as the deduction time step increases, the single-step prediction bias accumulates, leading to computational distortion in the estimated action value. This value distortion ultimately causes the planned action selected by the planning algorithm to deviate from the optimal decision, reducing the reliability of the reinforcement learning system's output control commands. Therefore, this invention proposes a reinforcement learning method, system, and storage medium based on structural causal decoupling to address the shortcomings of existing technologies. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention provides a reinforcement learning method, system, and storage medium based on structural causal decoupling. It solves the problems of existing reinforcement learning prediction models coupling the spontaneous evolution of the environment with action intervention, which makes the calculation of intrinsic rewards susceptible to interference from random background noise in the environment, thus reducing the efficiency of action exploration. It also addresses the problems of low reliability of action planning decisions caused by the inability to isolate random changes in the environment during multi-step state deduction.

[0006] To achieve the above objectives, the present invention provides the following technical solution: The first aspect of the present invention provides a reinforcement learning method based on structural causal decoupling, comprising:

[0007] Receive environmental observation data and process the environmental observation data through a parametric encoder to generate latent states;

[0008] The latent state and action are input into a dual-channel prediction network. The environment channel in the dual-channel prediction network generates an environmental spontaneous evolution state independent of the action, and the intervention channel in the dual-channel prediction network generates an intervention state triggered by the action. The predicted latent state for the next moment is generated based on the latent state, the environmental spontaneous evolution state, and the intervention state.

[0009] The intrinsic reward of initiative is calculated based on the intervention state output of the intervention channel;

[0010] Acquire historical interaction trajectory data, use the predicted latent state at the next moment and the historical interaction trajectory data to calculate the dynamic prediction residual, construct a joint loss function that includes dynamic prediction loss, and during the gradient backpropagation process of the joint loss function, transmit the gradient parameters corresponding to the dynamic prediction residual caused by the action to the intervention channel to update the dual-channel prediction network.

[0011] The updated dual-channel prediction network is invoked to perform Monte Carlo tree search, and the search tree is updated by incorporating the intrinsic reward of initiative. Based on the search tree, the final planned action is selected and applied to the environment.

[0012] This invention constructs a dual-channel prediction network that separates the environmental evolution from the action intervention process. The environmental channel introduces an information bottleneck to block the input path of action variables, while the intervention channel sets a zero-action anchoring constraint. During the backpropagation process of joint training, the network structure constraint is used to transmit the gradient parameters corresponding to the state prediction residuals caused by actions to the intervention channel, achieving feature-level causal decoupling. The intrinsic reward of initiative is calculated based on the independently output intervention state, eliminating the interference of random background noise in the environment. Combined with Monte Carlo tree search, the intrinsic reward of initiative is integrated into the node prediction value to optimize the action planning process.

[0013] Preferably, the generation of latent states through processing environmental observation data using a parametric encoder specifically includes:

[0014] The parameterized encoder is invoked to map environmental observation data to a low-dimensional latent space, generating latent states for the corresponding time steps;

[0015] When environmental observation data is set as two-dimensional image data, the parameterized encoder is implemented as a convolutional neural network or a visual converter model.

[0016] When environmental observation data is set as multi-axis sensor numerical data, the parameterized encoder is implemented as a multilayer perceptron network.

[0017] Preferably, the environmental channel is equipped with an information bottleneck restriction, which restricts the computational connection path of the physical disconnection action transmitted to the environmental channel;

[0018] The dual-channel prediction network only feeds the current latent state tensor into the input port of the environment channel, and does not perform the splicing and fusion operation of the latent state tensor and the action tensor within the environment channel.

[0019] Preferably, the intervention channel sets a zero-action anchoring constraint at the network computing layer. The zero-action anchoring constraint stipulates that when the input action is set as a no-operation instruction, the output of the intervention channel is restricted to a zero vector.

[0020] The generation of the next-moment prediction latent state based on latent state, spontaneous environmental evolution state, and intervention state specifically includes:

[0021] The latent state, the spontaneously evolving state of the environment, and the intervention state are added together to generate the predicted latent state for the next time step.

[0022] Preferably, the intrinsic reward for initiative calculated based on the intervention state output of the intervention channel specifically includes:

[0023] The obtained intervention states are used as intervention state vectors, and the 2-norm of the intervention state vectors is calculated.

[0024] Obtain the adaptive moving average normalization parameter, which includes the moving average of the intervention magnitudes observed under similar conditions and the corresponding standard deviation;

[0025] The bias value is calculated based on the L2 norm, moving average, and standard deviation of the intervention state vector;

[0026] The hyperparameters are superimposed onto the bias values, and the activation function is called to perform a nonlinear mapping on the superposition result, generating the final intrinsic reward of initiative. The calculation process is independent of the output of the environmental channel.

[0027] Preferably, when performing the step of updating the dual-channel prediction network:

[0028] When sampling historical interaction trajectory data from the experience replay pool and updating the network, the mean and variance of the intervention amplitude of the current training batch data are statistically calculated and combined with the sliding parameters of the historical records, and then weighted and accumulated according to the preset momentum decay rate to update, while simultaneously maintaining the adaptive moving average normalization parameters.

[0029] Preferably, the calculation of dynamic prediction residuals and the construction of a joint loss function using the predicted latent state at the next time step and historical interaction trajectory data specifically includes:

[0030] A multi-step time unfolding mechanism is executed along the time axis. Starting from the initial time step state of the sampled historical interaction trajectory data, the dual-channel prediction network is continuously called to perform potential state deduction for multiple subsequent consecutive time steps to obtain the calculation results generated at each deduction time step.

[0031] The calculation results generated at each simulation time step are compared with the search statistics target values ​​saved in the historical interaction trajectory data to generate multiple loss components, including strategy loss, value prediction loss, external reward prediction loss and dynamic prediction loss.

[0032] The strategy loss, value prediction loss, external reward prediction loss, and dynamic prediction loss are linearly summed according to preset weighting coefficients to form a joint loss function.

[0033] Preferably, the search tree is updated by integrating intrinsic rewards of initiative, and the final planned action is selected based on the search tree to act on the environment, specifically including:

[0034] During the node inference stage, the feature state vectors corresponding to the edge leaf nodes and the action vectors of decision selection are input into the dual-channel prediction network, which outputs the next prediction state and the prediction of the external environment reward.

[0035] The predicted external environment reward, the intrinsic motivation reward, and the state prediction value corresponding to the next predicted state are linearly superimposed and aggregated to calculate the enhanced cumulative value. The search tree is then updated by performing a backpropagation operation along the traversal path to the initial root node.

[0036] Read the action access count records of all candidate actions under the initial root node, and extract the candidate action with the largest absolute value of the action access count record as the final planned action to be applied to the environment.

[0037] A second aspect of the present invention also provides a reinforcement learning system based on structural causal decoupling, comprising:

[0038] The state coding module is used to receive environmental observation data and process the environmental observation data through a parametric encoder to generate latent states;

[0039] The causal prediction module is used to input latent states and actions into a dual-channel prediction network. It generates spontaneous environmental evolution states independent of actions through the environment channel in the dual-channel prediction network, and generates intervention states triggered by actions through the intervention channel in the dual-channel prediction network. Based on the latent state, spontaneous environmental evolution states, and intervention states, it generates the predicted latent state for the next moment.

[0040] The reward calculation module is used to calculate the intrinsic reward of initiative based on the intervention state output by the intervention channel;

[0041] The joint training module is used to acquire historical interaction trajectory data, calculate the dynamic prediction residual using the predicted latent state at the next time step and the historical interaction trajectory data, construct a joint loss function that includes dynamic prediction loss, and during the gradient backpropagation process of the joint loss function, the gradient parameters corresponding to the dynamic prediction residual caused by the action are transmitted to the intervention channel to update the dual-channel prediction network.

[0042] The action planning module is used to call the updated dual-channel prediction network to perform Monte Carlo tree search, integrate the intrinsic reward of initiative to update the search tree, and select the final planned action to act on the environment based on the search tree.

[0043] A third aspect of the present invention also provides a storage medium storing a computer program, which, when executed by a processor, implements the reinforcement learning method based on structural causal decoupling provided in the first aspect of the present invention, as described in any of the preceding claims.

[0044] This invention provides a reinforcement learning method, system, and storage medium based on structural causal decoupling. It offers the following advantages:

[0045] 1. This invention constructs a dual-channel prediction network that includes an environment channel and an intervention channel. In the environment channel, an information bottleneck is set to block the input of action variables into the computational connection path. In the intervention channel, a zero-action anchoring constraint is set. During the backpropagation of the gradient of the joint loss function, the dual-channel prediction network transmits the gradient parameters corresponding to the state prediction residuals caused by action variables to the intervention channel. This achieves feature-level decoupling between spontaneous environmental evolution and causal intervention of actions without the need for manual data annotation, thereby improving the accuracy of the dual-channel prediction network in performing state inference calculations.

[0046] 2. This invention calculates the intrinsic reward of initiative based on the intervention state output by the intervention channel. The calculation process of the intrinsic reward of initiative completely excludes the spontaneous evolution state of the environment output by the environment channel, avoids the mixing of random background noise of the environment when calculating the reward value, and ensures that the exploration drive extracted by the agent in the task environment lacking external environmental rewards comes from the physical intervention effect of actual actions. This prevents the agent from getting caught in an ineffective exploration process targeting environmental background noise and improves the action exploration efficiency of reinforcement learning methods.

[0047] 3. When calling the dual-channel prediction network to perform Monte Carlo tree search, this invention linearly superimposes and aggregates the predicted external environment reward, the intrinsic motivation reward, and the state prediction value to calculate the enhanced cumulative value. The enhanced cumulative value is then used for backpropagation to update the search tree. The independently decoupled intrinsic motivation reward is integrated into the online action evaluation step, optimizing the selection logic of the search tree node expansion direction. This alleviates the planning failure phenomenon caused by the accumulation of errors in the single-step prediction residual in the multi-step time expansion mechanism, and improves the reliability of the final planned action output by the action planning module. Attached Figure Description

[0048] Figure 1 This is a schematic diagram of the system architecture of the present invention;

[0049] Figure 2 This is a schematic diagram of the overall method flow of the present invention;

[0050] Figure 3 This is a schematic diagram of the computational derivation steps of the dual-channel prediction network of the present invention;

[0051] Figure 4 This is a schematic diagram of the intrinsic reward calculation steps of the present invention;

[0052] Figure 5 This is a schematic diagram showing the performance comparison of multiple algorithms based on the Atari 100k benchmark of the present invention;

[0053] Figure 6 This is a schematic diagram of the ablation experiment trend of the dual-channel causal decoupling architecture of the present invention. Detailed Implementation

[0054] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0055] Please see Figure 1 This invention provides a reinforcement learning system based on structural causal decoupling, including a state encoding module, a causal prediction module, a reward calculation module, a joint training module, and an action planning module.

[0056] The state encoding module receives data generated by the interaction between the agent and the environment. The state encoding module calls the parameterized encoder to map the environmental observation data to a low-dimensional latent space and generate the latent state corresponding to the time step.

[0057] The causal prediction module receives the latent state output by the state encoding module and the action output by the agent. The causal prediction module is divided into an environment channel and an intervention channel. The environment channel uses the information bottleneck structure to physically cut off the path of the action variable into the environment channel. The environment channel only receives the latent state and outputs the spontaneous evolution state of the environment independent of the action variable. The intervention channel receives both the latent state and the action output by the agent. The intervention channel forces the setting of zero action anchoring constraints at the network computing level. Based on the zero action anchoring constraints, the intervention channel outputs the intervention state directly caused by the action variable. The causal prediction module adds the latent state, the spontaneous evolution state of the environment, and the intervention state to generate the predicted latent state for the next time step.

[0058] The reward calculation module receives the intervention state vector output by the intervention channel, calculates the L2 norm of the intervention state vector and obtains the adaptive moving average normalization parameter during the training process, and calculates the intrinsic reward of initiative based on the L2 norm of the intervention state vector and the adaptive moving average normalization parameter. The calculation process excludes the participation of the environmental channel output results.

[0059] The joint training module acquires historical interaction trajectory data from the experience replay pool. Based on the multi-step time unfolding mechanism, the joint training module calculates the dynamic prediction residual and constructs a joint loss function that includes the dynamic prediction loss. During the gradient backpropagation process of minimizing the joint loss function, due to the information bottleneck limitation of the environmental channel, the joint training module transmits the gradient parameters corresponding to the state prediction residual caused by the action variables to the intervention channel, thus completing the update of the causal prediction module.

[0060] The action planning module calls the updated causal prediction module to perform a Monte Carlo tree search. During the tree expansion phase, the action planning module inputs the virtual state and optional actions corresponding to the current node, obtains the virtual next-moment state deduced by the causal prediction module, and obtains the intrinsic reward of agency calculated by the reward calculation module based on the optional actions. During the backpropagation phase of the search tree value, the action planning module adds the intrinsic reward of agency to the node evaluation value to calculate the enhanced node value. The action planning module backpropagates the enhanced node value along the search tree path to the root node. After completing the tree search traversal, it extracts the access frequency distribution of the optional branch actions under the root node and selects the branch action with the largest access frequency as the final planned action.

[0061] See attached document Figure 2 This invention provides a reinforcement learning method based on a structural causal decoupling world model, comprising the following steps:

[0062] S100 receives environmental observation data, calls the parameterized encoder to map the environmental observation data to a low-dimensional latent space, and generates the latent state corresponding to the time step;

[0063] S200 uses a dual-channel prediction network that inputs latent state and action. It generates an environment spontaneous evolution state independent of action by physically cutting off the input path of action variables in the environment channel. It generates an intervention state directly caused by action by applying an intervention channel with zero action anchoring constraints. The latent state, the environment spontaneous evolution state and the intervention state are added together to obtain the predicted latent state for the next time step.

[0064] S300: Obtain the intervention state vector output by the intervention channel and calculate the L2 norm of the intervention state vector. Combined with the pre-obtained adaptive moving average normalization parameter, calculate the intrinsic reward of initiative independently of the output of the environmental channel.

[0065] S400 acquires historical interaction trajectory data to calculate dynamic prediction residuals, constructs a joint loss function including dynamic prediction loss, and during gradient backpropagation, based on the information bottleneck limitation of the environmental channel, transmits the gradient parameters of the state prediction residuals caused by the action to the intervention channel for network update.

[0066] S500 calls the updated dual-channel prediction network to perform Monte Carlo tree search. During the node deduction phase, it obtains the intrinsic reward of agency, adds the intrinsic reward of agency to the node evaluation value to calculate the value of the enhanced node, updates the search tree by backpropagation based on the value of the enhanced node, and selects the final planned action to act on the environment based on the access frequency distribution of the branch actions under the root node.

[0067] Reference Figure 2 The execution process for receiving environmental observation data in step S100, calling the parameterized encoder to map the environmental observation data to a low-dimensional latent space, and generating the latent state for the corresponding time step specifically includes the following sub-steps:

[0068] S101, Obtain environmental observation data for the current time step under a partially observable Markov decision process; the agent is at time step... To continuously interact with the environment, the agent is limited by the conditions of partially observable Markov decision processes. It cannot obtain global real state data of the environment. Instead, the agent receives local high-dimensional observation data from the environment. In specific physical application scenarios, environmental observation data includes various lower-level implementation forms, such as two-dimensional image data containing pixel matrices, three-dimensional LiDAR point cloud data containing spatial coordinate information, and multi-axis sensor numerical data containing robotic arm joint angle values.

[0069] S102, Construct a parametric encoder for feature extraction. The system initializes the parametric encoder network as a state feature extraction module. The parametric encoder has a set of network weight parameters that can be optimized by gradients. In order to fully support feature extraction of environmental observation data of different modalities, the network topology of the parametric encoder is matched according to the lower-level implementation form of the environmental observation data. When the environmental observation data is set as two-dimensional image data, the parametric encoder is specifically implemented as a convolutional neural network or a vision converter model. When the environmental observation data is set as multi-axis sensor numerical data, the parametric encoder is specifically implemented as a multilayer perceptron network. For the specific network layer composition and neuron connection relationship of the convolutional neural network, vision converter model and multilayer perceptron network, those skilled in the art can refer to conventional manual documents for deployment.

[0070] S103, through the parameterized encoder, outputs the low-dimensional latent state corresponding to the time step. In the actual operation principle of reinforcement learning, directly using high-dimensional local observation data for state inference and long-term action planning will lead to the curse of dimensionality and a large consumption of computational resources. Therefore, the parameterized encoder uses the acquired environmental observation data as the network input tensor to perform forward propagation calculation, aiming to remove redundant background information in the original local observation data. Through the above forward propagation calculation, the parameterized encoder maps and compresses the data in the high-dimensional observation space into a continuous low-dimensional latent space, thereby outputting a low-dimensional latent state representing the core features of the current environment. During the system operation, the agent at time step... Receive observation data via encoder Mapped to latent state ,in, This represents the local high-dimensional observation data input to the encoder. This indicates the encoder that performs the mapping operation. This represents the set of learnable network weight parameters within the parametric encoder. The initial values ​​are randomly generated by the system and iteratively updated through backpropagation during subsequent end-to-end joint training phases, representing the latent state. It is a low-dimensional vector form containing real values, and the dimension of the low-dimensional latent state is smaller than the original dimension of the environmental observation data. The low-dimensional latent state serves as a feature anchor point on the discrete time series and as the basic input data for subsequent causal dynamic evolution model to perform feature-level decoupling and state inference calculation.

[0071] See attached document Figure 3The execution process for step S200, which involves using a dual-channel prediction network for latent state and action input, generates an environment spontaneous evolution state independent of the action by physically severing the input path of the action variable. It also generates an intervention state directly triggered by the action by applying an intervention channel with zero action anchoring constraints. Finally, it adds the latent state, the environment spontaneous evolution state, and the intervention state to obtain the predicted latent state for the next time step. This process specifically includes the following sub-steps:

[0072] S201 constructs a mathematical model for latent state transitions that conforms to an additive causal generation process in a low-dimensional latent space. Traditional world models treat environmental dynamics as a single black box. The single black box prediction architecture cannot distinguish the causal relationship between spontaneous environmental changes and action interventions. To break the single black box prediction architecture, the system constructs a dual-channel prediction network. From the perspective of the objective laws of the physical world, the state of the environment in the next moment can usually be approximated as the result of the superposition of the inertial evolution of the environmental objects themselves and the intervention force applied by the external intelligent agent. The system assumes that the generation of latent states follows an additive causal generation process. The system decomposes the state evolution into an environmentally autonomous evolution term independent of actions and a causal intervention term directly caused by actions. The state transition equation of the additive causal generation process is as follows:

[0073] ;

[0074] in, Indicates time step The latent state; Indicates time step The latent state; This represents the environment-autoformed terms output by the environment channel, independent of the action. This represents the causal intervention items directly triggered by the action, as output by the intervention channel. This indicates that the agent is at time step The output action; This represents random noise in the environment. Its value is usually set to follow a multivariate Gaussian distribution with a mean of zero, and is used to absorb random fluctuations in the dynamic environment that cannot be determined.

[0075] S202, an environment channel is set up in the dual-channel prediction network. The environment channel is used to capture the inherent spontaneous temporal evolution sequence of the environment. Since the spontaneous motion of environmental objects usually depends on historical motion trajectories, the lower-level network features of the environment channel are specifically implemented as recurrent neural networks. The recurrent neural network specifically includes gated recurrent unit networks or long short-term memory networks. For the specific internal hierarchical structure and gating logic of the gated recurrent unit network or long short-term memory network, those skilled in the art can refer to conventional neural network manuals for deployment. The environment channel sets information bottleneck restrictions at the network physical architecture level. The information bottleneck restrictions physically cut off the computational connection path for action variables to enter the environment channel. In the specific network computation graph construction, the system only feeds the current latent state tensor into the input port of the recurrent neural network. The system does not perform the splicing and fusion operation of latent state tensor and action tensor in the environment channel. The above-mentioned underlying physical isolation measures ensure that the output of the environment channel strictly meets the conditional independence restriction. The conditional independence restriction formula is as follows:

[0076] );

[0077] in, The symbol for conditional independence; In probability theory, the given conditional notation indicates that the above formula represents the condition given a latent state. Under these conditions, the output results and actions of the environmental channel They are independent of each other.

[0078] S203, an intervention channel is set in the dual-channel prediction network. The intervention channel is specifically designed to fit the instantaneous direct causal consequences of state-action pairs. The lower-level network features of the intervention channel are specifically implemented as a multilayer perceptron to establish a nonlinear mapping relationship between latent states, action variables, and state changes. The input of the intervention channel simultaneously receives the current latent state and the action output by the agent. The system enforces a zero-action anchoring constraint at the network computation level of the intervention channel. The zero-action anchoring constraint, at the loss calculation level, mandates that when the input action is set to a no-operation instruction, specifically, in discrete action space tasks, the no-operation instruction is implemented as a preset no-operation action index; in continuous action space tasks, the no-operation instruction is implemented as a zero-value vector. The output of the intervention channel is strictly limited to a zero vector. The zero-action anchoring constraint serves as a necessary mathematical condition for blocking non-causal backdoor paths and achieving causal recognition. The system satisfies this constraint by adding an additional penalty term to the loss function of the joint training, calculating the L2 norm of the intervention channel output vector when the input is a no-operation and forcing it to approach zero, or by directly multiplying the indicator function of the action vector at the network output layer. The formula for the zero-action anchoring constraint is as follows:

[0079] ;

[0080] in, Indicates a no-operation action; The zero-action anchoring constraint, which represents the identity symbol, ensures that the intervention channel outputs a non-zero causal intervention term only when an effective action occurs.

[0081] S204, the next time step is predicted and the latent state is calculated. The dual-channel prediction network operates on the environment channel to process the current latent state to generate an environment spontaneous evolution state independent of the action. Simultaneously, the dual-channel prediction network operates on the intervention channel to process the current latent state and the action to generate an intervention state directly caused by the action. When performing forward prediction calculation, the dual-channel prediction network adds the current latent state, the environment spontaneous evolution state, and the intervention state to generate the predicted latent state for the next time step. The above addition operation strictly maps the state transition equation of the additive causal generation process, realizing the causal decoupling and dynamic prediction deduction at the state feature level in pure observation data without manual annotation.

[0082] See attached document Figure 4 The execution process of obtaining the intervention state vector output by the intervention channel in step S300 and calculating the L2 norm of the intervention state vector, combined with the pre-acquired adaptive moving average normalization parameter, and calculating the intrinsic reward of initiative independently of the environmental channel output, specifically includes the following sub-steps:

[0083] S301, Determine the information source for extracting intrinsic rewards for agency. When performing action exploration in a task environment lacking external rewards, the agent relies on intrinsic rewards to guide the exploration direction. According to the physical operation logic of the real environment, the spontaneous evolution state of the environment output by the environment channel will objectively contain random background noise. If the overall state prediction error is taken as the intrinsic reward, the random background noise of the environment will produce abnormal prediction error values, thereby inducing the agent to fall into an ineffective exploration state of continuously observing background noise. In order to eliminate the interference of random background noise of the environment, the information source is strictly limited to the output result of the intervention channel.

[0084] S302, Calculate the L2 norm of the intervention channel output and obtain the time step. The intervention state vector output by the lower intervention channel is subjected to a L2 norm calculation operation. The specific implementation of the L2 norm calculation operation is to solve for the arithmetic square root of the sum of squares of each element in the intervention state vector. For the specific matrix algebra derivation process of the underlying L2 norm calculation, those skilled in the art can refer to conventional linear algebra manuals for deployment. The L2 norm of the intervention state vector is used in a physical sense to quantitatively measure the magnitude of the absolute influence of the agent's actions on the environment.

[0085] S303. Obtain the adaptive moving average normalization parameter for numerical scaling. During long-sequence reinforcement learning training, the numerical scale of the intervention amplitude fluctuates significantly due to the continuous changes in the agent's exploration state. To stabilize the reward signal, the adaptive moving average normalization parameter is maintained and updated synchronously when sampling in the experience replay pool and updating the network. The adaptive moving average normalization parameter includes the moving average of the intervention amplitude observed by the model in similar states and the corresponding standard deviation. In practice, the mean and variance of the intervention amplitude statistics of the current training batch data are used, combined with the moving average parameter of the historical records, and weighted and accumulated according to the preset momentum decay rate. The momentum decay rate is usually set to a constant value between 0.99 and 0.999 in actual deployment to obtain a smooth statistic. The adaptive moving average normalization parameter is used to normalize the L2 norm calculated at the current time step to prevent the explosion of the reward numerical scale in the later stage of model training and to ensure the stability of the convergence of the dynamic programming algorithm.

[0086] S304: Based on the extracted information source and numerical scaling parameters, the intrinsic reward of initiative is calculated. The bias value is calculated based on the L2 norm, moving average, and standard deviation of the intervention state vector. A shifting hyperparameter is superimposed on the bias value, and an activation function is called to perform a nonlinear mapping on the superposition result, generating the final intrinsic reward of initiative. The above calculation process completely excludes the participation of environmental channel output results, assigning reward values ​​only to novel state changes caused by the agent's actions. The mathematical operation of the intrinsic reward of initiative follows the formula below:

[0087] ;

[0088] in, Indicates time step The calculated intrinsic reward of initiative; This represents the L2 norm of the intervention channel output vector, which is the second norm of the intervention state vector; This represents the moving average value in the normalized parameters of the adaptive moving average. This represents the standard deviation in the adaptive moving average normalization parameter; This represents the set of network weight parameters corresponding to the dual-channel dynamic model. This represents the translation hyperparameter, which is set to a constant value. The translation hyperparameter is used to adjust the basic threshold for determining intrinsic rewards. In actual test environment deployments, the translation hyperparameter is usually set to an empirical value of 0.1 to provide basic exploration motivation.

[0089] See attached document Figure 2For step S400, the dynamic prediction residual is calculated based on a multi-step time expansion mechanism, and a joint loss function is constructed. Due to the information bottleneck limitation of the environmental channel, the gradient parameters corresponding to the state prediction residual caused by the action variables are transmitted to the intervention channel to complete the update. The execution process specifically includes the following sub-steps:

[0090] S401, extract trajectory data and determine the set of model parameters to be optimized. During the continuous interaction with the environment, the agent generates state observation sequences, action commands, and reward values. The above interaction data is stored in the experience replay pool. During the training phase, historical trajectory data fragments are randomly sampled from the experience replay pool as training batch data, and the set of model parameters participating in end-to-end joint training is determined. The set of model parameters includes encoder network parameters for feature mapping, dual-channel dynamic model network parameters for state inference, and prediction head network parameters for value evaluation. The prediction head network specifically includes a policy prediction branch and a value prediction branch, which are used to receive latent states and output the corresponding action policy probability distribution and state prediction value, respectively.

[0091] S402 executes a multi-step time unfolding mechanism along the time axis. In model-based reinforcement learning, if only single-step state prediction is performed, small prediction errors will be amplified in the long sequence of action deduction, forming compound error accumulation, which will lead to the failure of long-term planning. Therefore, in order to capture the long-term time dependencies of environmental dynamics and alleviate error accumulation to improve the accuracy of long-term planning, the model performs joint optimization through a multi-step time unfolding mechanism. The specific execution logic of the multi-step time unfolding mechanism is as follows: the model starts from the initial time step state of the sampled historical trajectory data, continuously calls the dual-channel prediction network to perform potential state deduction for multiple subsequent consecutive time steps, and synchronously outputs the corresponding policy distribution and estimated value. The multi-step time unfolding mechanism collects the calculation results generated by each deduction time step in a unified manner for subsequent calculation of accumulated errors. The total number of unfolded time steps is usually set to a fixed constant sequence length according to the temporal span of the specific task.

[0092] S403, construct a joint loss function for end-to-end optimization. The predicted values ​​output by the multi-step time unrolling mechanism are compared with the search statistics (specifically, the actual external reward, the action access distribution generated after Monte Carlo tree search, and the final reward value) stored in the experience replay pool to generate multiple loss components. These components include policy loss, value prediction loss, external reward prediction loss, and dynamics prediction loss. The dynamics prediction loss forces the superimposed latent state of the dual-channel prediction network to approximate the true next-time latent state. The lower-level calculation methods for dynamics prediction loss, value prediction loss, and external reward prediction loss are implemented using mean squared error calculation, while the lower-level calculation method for policy loss is implemented using cross-entropy loss calculation or relative entropy calculation. By linearly summing the multiple loss components according to preset weight coefficients, a joint loss function is formed. The mathematical operation of the joint loss function follows the formula below:

[0093] ;

[0094] in, Represents the joint loss function of the overall model; This represents the set of model parameters, which specifically includes the set of learnable network weight parameters within the parametric encoder. The set of network weight parameters corresponding to the dual-channel dynamic model and prediction head network parameters ; Indicates the experience replay pool Trajectory data sampled in the middle Find the expected value; This indicates the total number of time steps in the multi-step time unfolding, which typically ranges from 10 to 50 steps in actual deployments. Indicates the indexed time step in the time unrolling process; Indicates index time step The resulting strategic losses; Indicates index time step The resulting loss in value prediction; Indicates index time step The resulting external reward prediction loss; Indicates index time step The resulting dynamic prediction loss; The weighted hyperparameters representing the loss in value prediction; The weighted hyperparameters representing the loss in predicting external rewards; The weight hyperparameters represent the dynamic prediction loss. These weight hyperparameters are used to balance the influence of each gradient descent direction and are usually set to an empirical constant close to 1.0 according to task requirements.

[0095] S404 executes gradient backpropagation and implicitly achieves feature-level causal decoupling. After the joint loss function is constructed, the gradient optimizer is triggered to update the network parameters. For the backpropagation differentiation and parameter iteration calculation of the gradient optimizer, those skilled in the art can refer to conventional deep learning framework documents for deployment. The above content is well-known technology in this field and will not be elaborated here. In the process of minimizing the dynamic prediction loss, the optimization mechanism is physically limited by the bottleneck of environmental channel information. Since action variables cannot be input into the environmental channel, the gradient optimizer cannot correct the state prediction residual caused by the action by adjusting the network parameters inside the environmental channel. In order to achieve the convergence of minimizing the joint loss function, the gradient optimizer is objectively forced to allocate all gradient parameters related to the state prediction residual caused by the action to the intervention channel for network weight update. The above-mentioned asymmetric limitation of the underlying network physical architecture enables the dual-channel network to automatically complete the feature-level decoupling of independent environmental evolution and action causal intervention in pure observation data without relying on additional manual annotation.

[0096] See attached document Figure 2 The execution process of the Monte Carlo tree search planning for causal perception based on the fusion of initiative rewards in step S500 specifically includes the following sub-steps:

[0097] S501 constructs the initial root node of the Monte Carlo tree. During the online action planning phase, it receives environmental observation data from the real physical environment at the current time step. Since directly performing tree search in the high-dimensional observation space would cause a combinatorial explosion problem in computational dimensions, it calls the trained parametric encoder to map and compress the environmental observation data into a state vector in a low-dimensional latent space. This state vector serves as the initial root node state for the Monte Carlo tree search process. The Monte Carlo tree consists of multiple nodes and directed edges connecting different nodes. Nodes represent virtual state features in the low-dimensional latent space, and directed edges represent virtual action instructions selected by the agent in a specific state. Each node maintains a record of the number of action accesses and statistical parameters of the estimated action value.

[0098] S502 performs node expansion operations based on the upper bound confidence tree logic. During the internal simulation traversal of the search algorithm, to find a balance between unknown high-reward actions and known high-reward actions, it starts from the initial root node state and traverses downwards along the Monte Carlo tree topology to find edge leaf nodes to be expanded. During this topology traversal, the prior probability of the policy distribution output by the predictor network guides the search direction selection. The traversal path is determined by maximizing the upper bound confidence tree score. The upper bound confidence tree score comprehensively evaluates the current estimated action value and the exploration value of unknown actions. Specifically, it combines the estimated value of actions with the exploration incentive term based on prior probability and visit count decay, prioritizing action branches with high prior probability and fewer current visits for expansion. At the underlying numerical calculation level, the specific magnitude of the exploration incentive term is proportional to the prior probability of the policy distribution and inversely proportional to the number of times the action has been visited.

[0099] S503 invokes the decoupled dynamic model to perform state inference and activity reward acquisition. When the traversal path reaches an edge leaf node, it extracts the feature state vector and action vector corresponding to the edge leaf node. The feature state vector and action vector are then input into the end-to-end trained dual-channel prediction network. Traditional black-box prediction networks are prone to inference illusions when encountering random environmental changes, leading to planning failure. However, the dual-channel prediction network, based on the aforementioned decoupled architecture, performs forward inference calculations, which can accurately isolate the influence of spontaneous environmental evolution and action intervention, and output the next predicted state. The mathematical operation process of the next predicted state follows the additive causal generation formula:

[0100] ;

[0101] in, This indicates the next predicted state obtained from the decoupled dynamic model derivation; This represents the feature state vector corresponding to the edge leaf node; This represents the action vector for making a decision at the edge leaf node; This represents the spontaneous evolution state component of the environment, which is independently inferred by the environment channel in a dual-channel prediction network. This represents the causal intervention state component directly triggered by the action vector, calculated and output by the intervention channel in the dual-channel prediction network. Simultaneously, the dual-channel prediction network outputs the prediction external environment reward corresponding to the action vector. Furthermore, the intervention feature vector output by the intervention channel is obtained. Using the extracted information source and numerical scaling parameters, the intrinsic reward of initiative is calculated.

[0102] S504 executes value enhancement calculation and search tree backpropagation update. For the expanded edge leaf nodes, the value evaluation network in the prediction head module is called. The value evaluation network receives the next predicted state and outputs the state prediction value corresponding to the next predicted state. In order to seamlessly inject the driving force of long-sequence causal exploration into the online action evaluation system, the search algorithm linearly superimposes and aggregates the predicted external environment reward, the intrinsic motivation reward, and the state prediction value to calculate the enhanced cumulative value. The mathematical operation of the enhanced cumulative value follows the formula below:

[0103] ;

[0104] in, This represents the enhanced cumulative value obtained from the calculation of the fusion initiative reward; This represents the predicted external environment reward output by the dual-channel prediction network. The weight hyperparameter represents the intrinsic reward of initiative. In order to ensure the final convergence stability of the dynamic programming algorithm, the weight hyperparameter is set to a relatively high empirical value in the early stage of training. This empirical value is usually set to 0.5, and gradually annealed to zero in the later stage of training as the number of iterations increases. This represents the intrinsic reward of initiative calculated based on the characteristics of pure intervention effects; The discount factor hyperparameter represents the discount rate that controls the future value discounting ratio. It usually ranges from 0.9 to 0.99 and is used to balance the proportion of immediate rewards to long-term accumulated value. This represents the estimated state value output by the value assessment network at the edge leaf nodes.

[0105] After calculating the enhanced cumulative value, a backpropagation operation is performed along the traversal path towards the initial root node. The backpropagation operation uses the enhanced cumulative value to update the action visit count records and estimated action value statistics parameters of all nodes traversed on the traversal path layer by layer.

[0106] S505 extracts statistical distribution features and outputs physical execution instructions. Within an independent decision time step, it executes multiple iterations of the above simulation unfolding, state deduction, value enhancement, and backpropagation calculation processes. After the iteration calculation process is completed, the initial root node accumulates sufficient prior statistical information of actions. It reads the action access count records of all candidate actions under the initial root node. The more action access counts a candidate action has, the more likely it is to have not only a higher expected return in the conventional environment, but also a significant and real physical environment intervention influence. It eliminates the interference of random background noise in the environment, directly extracts the candidate action with the largest absolute value of the action access count record, and issues the candidate action as an actual physical action instruction to the real physical environment for execution, completing the causal perception closed-loop planning of a single control cycle.

[0107] Specific application examples:

[0108] To verify the real-world performance of the reinforcement learning method based on structural causal decoupling described in this invention in complex hybrid dynamic environments, this embodiment selects the standard Atari 100k benchmark environment for specific engineering deployment and extracts the time steps of a single decision inference. We use real tensor data to perform calculations and verification of the underlying formulas.

[0109] Physical network parameter settings:

[0110] The system initializes a dual-channel prediction network. The lower-level network of the environment channel is implemented as a single-layer gated recurrent unit (GRU) network with a hidden layer neuron dimension of 256. The lower-level network of the intervention channel is implemented as a multilayer perceptron (MLP) containing 33 hidden layers with a neuron width of 256 in each layer, and the ReLU activation function is used between each hidden layer.

[0111] Numerical verification of state deduction and decoupled computation:

[0112] To intuitively illustrate the computational logic, the high-dimensional latent state is simplified and represented as a 44-dimensional continuous vector in the embodiment, at time step... The current low-dimensional feature state vector extracted by the state coding module is The action chosen by the agent at the current node is the one-hot encoded vector of the right-moving instruction. .

[0113] Zero-action anchoring and causal intervention calculation:

[0114] Verify the zero-action anchoring constraint formula for the intervention channel: When inputting a no-operation command At that time, the system forces the network output to be a vector of all zeros. When a valid action is input At that time, the environmental spontaneous evolution state components independently inferred by the environmental channel are output via forward propagation as follows: The causal intervention state component directly triggered by the action vector, calculated and output by the intervention channel, is: .

[0115] State prediction calculation:

[0116] During the node expansion phase of the Monte Carlo tree, the state transition equation for the additive causal generation process is specifically calculated as follows: ;

[0117] Substituting the specific numerical values ​​in the example, we perform vector addition:

[0118] ;

[0119] Deduction results Numerical decoupling and superposition of the inertial evolution of the environment and the effects of active intervention by intelligent agents have been achieved.

[0120] Calculation of intrinsic reward for initiative:

[0121] Extract the output vector of the intervention channel and calculate its L2 norm (2-norm):

[0122] ;

[0123] Extract the adaptive moving average normalization parameters and set the moving average value. Standard deviation The translation hyperparameters are preset to: The specific calculation of the intrinsic reward formula for initiative is as follows: ;

[0124] Substitute the values ​​above: ;

[0125] In this calculation, the random background noise variation in the environment is represented by... If completely excluded, the resulting 0.9569 represents the pure exploration reward unaffected by environmental noise.

[0126] Backpropagation calculation of search tree value:

[0127] Suppose a dual-channel prediction network outputs a prediction of the external environment reward. State prediction value Discount factor hyperparameter The hyperparameter of intrinsic reward weight of initiative The enhanced cumulative value formula is calculated as follows: ;

[0128] Substitute the values: ;

[0129] The algorithm propagates the cumulative enhancement value of 10.47845 back along the search path to the root node, thus completing the online action planning.

[0130] To demonstrate the superiority of the above causal decoupling model in real-world physical environments, combined with the appendix... Figure 5 With appendix Figure 6 The experimental data were used to verify the effectiveness:

[0131] Performance surpassing existing state-of-the-art (SOTA) models: Comparative verification

[0132] See attached document Figure 5(Line chart comparing the performance of multiple algorithms based on the Atari 100k benchmark) During the 100×103 (i.e., 100k) frame training period with interaction with the environment, the proposed method (CDWM) exhibits a faster convergence speed. At the end of training, the normalized average human score of the proposed method reached a peak of 2.81%. In comparison, the existing technology EZ-V2 (long dashed line in the figure) which does not employ causal decoupling scored 2.69%, and the existing technology BBF scored 2.26%. Experimental data prove that the proposed method improves sample utilization and final decision performance by introducing an active reward mechanism.

[0133] Ablation Study of Dual-Channel Decoupled Architecture:

[0134] See attached document Figure 6 (Dual-channel causal decoupling architecture ablation experiment): The system performs network channel stripping tests on control-driven tasks (taking Pong game as an example) and environment-driven tasks (Seaquest game as an example):

[0135] For the complete dual-channel model: the system scored 13 points in the Pong task and 2464 points in the Seaquest task, which serve as performance benchmarks.

[0136] When the environmental channel was removed: In the Pong task (where the environment has almost no spontaneous evolution), the score remained at 13 points, proving that the control features were not affected; however, in the Seaquest task, the score dropped to 589 points because the model could not predict the spontaneous movement of the submarine and the fish school, resulting in severe hallucinations in the inference.

[0137] When the intervention channel is removed: In the Pong task, the score drops to 44 points due to the loss of the ability to infer the causal relationship of actions; in the Seaquest task, although the autonomous evolutionary features of the environment exist, the agent is unable to assess the consequences of its own actions, and the score also decreases to 720 points.

[0138] Appendix Figure 6 The data proves that the dual-channel dynamic model of the present invention objectively achieves physical isolation between the spontaneous evolution of the environment and the causal intervention of actions. The absence of any single channel will lead to the loss of specific dynamic characteristics, thus proving the necessity and scientific nature of the combined dual-channel architecture and zero-action anchoring constraint of the present invention.

Claims

1. A reinforcement learning method based on structural causal decoupling, characterized in that, include: Receive environmental observation data and process the environmental observation data through a parametric encoder to generate latent states; The latent state and action are input into a dual-channel prediction network. An environmental spontaneous evolution state independent of the action is generated through the environment channel of the dual-channel prediction network. An intervention state triggered by the action is generated through the intervention channel of the dual-channel prediction network. The predicted latent state for the next time step is generated based on the latent state, the environmental spontaneous evolution state, and the intervention state. The intrinsic reward of initiative is calculated based on the intervention state output by the intervention channel; Historical interaction trajectory data is acquired, and the dynamic prediction residual is calculated using the predicted latent state at the next moment and the historical interaction trajectory data. A joint loss function containing dynamic prediction loss is constructed. During the gradient backpropagation process of the joint loss function, the gradient parameter corresponding to the dynamic prediction residual caused by the action is transmitted to the intervention channel to update the dual-channel prediction network. The updated dual-channel prediction network is invoked to perform Monte Carlo tree search, the search tree is updated by fusing the intrinsic reward of the agency, and the final planned action is selected based on the search tree and applied to the environment.

2. The reinforcement learning method based on structural causal decoupling according to claim 1, characterized in that, The process of generating latent states by processing the environmental observation data through a parametric encoder specifically includes: The parameterized encoder is invoked to map the environmental observation data to a low-dimensional latent space, generating the latent state corresponding to the time step; When the environmental observation data is set as two-dimensional image data, the parameterized encoder is implemented as a convolutional neural network or a visual converter model; When the environmental observation data is set as multi-axis sensor numerical data, the parameterized encoder is implemented as a multilayer perceptron network.

3. The reinforcement learning method based on structural causal decoupling according to claim 1, characterized in that, The environment channel is equipped with an information bottleneck restriction, which physically cuts off the computational connection path for the action to be transmitted to the environment channel; The dual-channel prediction network only feeds the tensor corresponding to the current latent state into the input port of the environment channel, and does not perform the splicing and fusion operation of the latent state tensor and the action tensor within the environment channel.

4. The reinforcement learning method based on structural causal decoupling according to claim 1, characterized in that, The intervention channel enforces a zero-action anchoring constraint at the network computing layer. The zero-action anchoring constraint stipulates that when the input action is set to a no-operation instruction, the output of the intervention channel is limited to a zero vector. Generating the next-moment predicted latent state based on the latent state, the spontaneous evolution state of the environment, and the intervention state specifically includes: The latent state, the spontaneous evolution state of the environment, and the intervention state are added together to generate the predicted latent state for the next time step.

5. The reinforcement learning method based on structural causal decoupling according to claim 1, characterized in that, The calculation of intrinsic motivational reward based on the intervention state output by the intervention channel specifically includes: The obtained intervention state is used as the intervention state vector, and the L2 norm of the intervention state vector is calculated. Obtain the adaptive moving average normalization parameter, which includes the moving average of the intervention magnitudes observed under similar conditions and the corresponding standard deviation; The bias value is calculated based on the L2 norm of the intervention state vector, the moving average, and the standard deviation; The hyperparameters are superimposed on the bias value, and the activation function is called to perform a nonlinear mapping on the superposition result to generate the final intrinsic reward of initiative. The calculation process is independent of the output result of the environmental channel.

6. The reinforcement learning method based on structural causal decoupling according to claim 5, characterized in that, When performing the step of updating the dual-channel prediction network: When sampling the historical interaction trajectory data from the experience replay pool and updating the network, the mean and variance of the intervention amplitude of the current training batch data are statistically calculated and updated by combining the sliding parameters of the historical records with the preset momentum decay rate, and the adaptive moving average normalization parameters are maintained simultaneously.

7. The reinforcement learning method based on structural causal decoupling according to claim 1, characterized in that, The calculation of dynamic prediction residuals and the construction of a joint loss function using the predicted latent state at the next time step and the historical interaction trajectory data specifically include: A multi-step time unfolding mechanism is executed along the time axis. Starting from the initial time step state of the sampled historical interaction trajectory data, the dual-channel prediction network is continuously invoked to perform potential state deduction for multiple subsequent consecutive time steps to obtain the calculation results generated by each deduction time step. The calculation results generated at each simulation time step are compared with the search statistics target values ​​stored in the historical interaction trajectory data to generate multiple loss components, including strategy loss, value prediction loss, external reward prediction loss and dynamic prediction loss. The strategy loss, the value prediction loss, the external reward prediction loss, and the dynamics prediction loss are linearly summed according to preset weighting coefficients to form a joint loss function.

8. The reinforcement learning method based on structural causal decoupling according to claim 1, characterized in that, The search tree is updated by integrating the intrinsic reward of the initiative, and the final planned action is selected based on the search tree to act on the environment, specifically including: During the node inference phase, the feature state vectors corresponding to the edge leaf nodes and the action vectors of decision selection are input into the dual-channel prediction network, which outputs the next prediction state and the predicted external environment reward. The predicted external environment reward, the intrinsic motivation reward, and the state prediction value corresponding to the next predicted state are linearly superimposed and aggregated to calculate the enhanced cumulative value. The search tree is then updated by performing a backpropagation operation along the traversal path to the initial root node. Read the action access count records of all candidate actions under the initial root node, and extract the candidate action with the largest absolute value of the action access count record as the final planned action to be applied to the environment.

9. A reinforcement learning system based on structural causal decoupling, characterized in that, A reinforcement learning method based on structural causal decoupling as described in any one of claims 1-8, comprising: The causal prediction module is used to input latent states and actions into a dual-channel prediction network, generate an environmental spontaneous evolution state independent of the action through the environment channel in the dual-channel prediction network, generate an intervention state triggered by the action through the intervention channel in the dual-channel prediction network, and generate a predicted latent state for the next time step based on the latent state, the environmental spontaneous evolution state, and the intervention state. The reward calculation module is used to calculate the intrinsic reward of initiative based on the intervention state output by the intervention channel; A joint training module is used to acquire historical interaction trajectory data, calculate the dynamic prediction residual using the predicted latent state at the next moment and the historical interaction trajectory data, construct a joint loss function including dynamic prediction loss, and during the gradient backpropagation process of the joint loss function, the gradient parameters corresponding to the dynamic prediction residual caused by the action are transmitted to the intervention channel to update the dual-channel prediction network. The action planning module is used to call the updated dual-channel prediction network to perform Monte Carlo tree search, integrate the intrinsic reward of the agency to update the search tree, and select the final planned action to act on the environment based on the search tree.

10. A storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements a reinforcement learning method based on structural causal decoupling as described in any one of claims 1-8.

Images