A value-driven cross-domain strategy generalization method and system

Abstract

The present invention generally relates to the field of reinforcement learning technology and proposes a value-driven cross-domain policy generalization method and system. The method includes: providing source domain training data and target domain training data; filtering the source domain training data based on value-driven methods; and inputting the filtered source domain training data and target domain training data into an intelligent agent for training. While the source domain itself does not need to be changed, the algorithm of the present invention no longer focuses on the dynamic consistency of the data, but instead filters the source domain data based on the cross-domain consistency of value. This can effectively improve the generalization ability of strategies in reinforcement learning and thus effectively save computing resources.

Description

Technical Field

[0001] The present invention generally relates to the field of reinforcement learning technology. Specifically, the present invention relates to a value-driven cross-domain policy generalization method and system. Background Art

[0002] Existing active intelligence algorithms can leverage vast amounts of data to solve complex real-life tasks. However, these agents are limited to specific environments and lack the ability to generalize across diverse environments. While humans can accelerate their learning by leveraging prior knowledge of other tasks or other agents, current reinforcement learning algorithms often struggle to generalize across highly diverse tasks. Therefore, improving the generalization capabilities of strategies is a key issue in the current development of reinforcement learning.

[0003] In the field of reinforcement learning, existing algorithms typically train optimal control strategies for a fixed task. However, when the environment or the task itself changes, the existing control strategies often fail to adapt. For example, when using reinforcement learning algorithms to train a walking control strategy for a quadruped robot, the old strategy often fails to control the robot in the face of environmental changes such as fluctuations in the robot's voltage or changes in the external environment.

[0004] To address the cross-domain adaptation problem of reinforcement learning strategies, an agent needs to be given relatively unrestricted source domain data and limited target domain data to train a high-level strategy for the target domain. This requires algorithms to address the dynamic differences between the source and target domains. Existing approaches include:

[0005] A dynamics adaptation method based on a parameterizable simulator. When the source domain in which the agent resides is parameterized or dynamically adjustable, this method explicitly adjusts the dynamic parameters of the source domain simulator based on data from the source and target domains, making it sufficiently close to the dynamic rules of the target domain. Finally, the policy trained in the adjusted source domain can be directly deployed to the target domain. However, this method relies on a parameterized simulator, and the physical properties of the simulators corresponding to many complex tasks are not adjustable. When the corresponding target domain is the real world, the dynamic differences between the target domain and the source domain may be due to various reasons (such as changes in the robot's external environment). At this time, if the source domain simulator cannot simulate the corresponding changes, this method will not be able to make the dynamics of the source domain close to the dynamics of the target domain, and thus will not be able to train a generalizable policy.

[0006] This method is a dynamic adaptation method based on action conversion. It fits the corresponding inverse dynamics model and dynamics model based on the data of the source and target domains. When the intelligent agent interacts and trains with the source domain, the actions executed by the policy are corrected based on the trained dynamics model, so that the corrected actions in the target domain and the original actions in the source domain maintain dynamic consistency, ultimately achieving cross-domain adaptation of the policy. However, this method requires fitting a complex dynamics model. However, for complex robotic systems, the dynamics equations are very complex. The neural network requires a large number of samples for training to fit the dynamics equations, and a large number of samples need to be obtained by interacting with the environment. Under the constraints of limited environmental interaction, it is often impossible to obtain sufficient samples to fit the dynamics model.

[0007] A dynamic adaptive method based on reward compensation, which dynamically compensates the reward value of the data collected in the source domain by training a domain classifier. Specifically, when the corresponding data is close to the dynamic rules of the target domain, the algorithm provides the agent with an additional positive reward. When the data does not conform to the dynamic rules of the target domain, the algorithm provides the agent with a negative reward as a penalty. The value of the corresponding reward compensation is determined by the likelihood estimate of the domain classifier. However, this method requires the assumption that there is at least one high-level strategy in the target domain and that this strategy also performs well in the source domain. When the source and target domains do not meet the above assumptions, the method will penalize all source domain data based on the domain classifier, thereby hindering the strategy's exploration of the target domain and causing the algorithm to fail to converge to a high-level strategy. Summary of the Invention

[0008] To at least partially address the above-mentioned problems in the prior art, the present invention proposes a value-driven cross-domain policy generalization method, comprising the following steps:

[0009] Provide source domain training data and target domain training data:

[0010] Filter source domain training data, including:

[0011] Provide state-action data pair (s, a) to determine the next state s′ of the source domain src , and determine the state set of the target domain at the next moment through the target domain dynamics model

[0012] The next state s′ of the source domain is calculated through the value function network src And the state set of the target domain at the next moment Perform value evaluation to generate the source domain state value Q src And the target domain state value set as well as

[0013] Compare the generated source domain state value Q src And the target domain state value set Filter out source domain training data whose state value gap is less than a threshold ∈; and

[0014] The filtered source domain training data and target domain training data are input into the agent for training.

[0015] In one embodiment of the present invention, the value-driven cross-domain policy generalization method further includes training a target domain dynamics model, including:

[0016] Determine target domain training data, where the target domain training data includes a state transition tuple (s, a, r, s′), where s represents the state, a represents the action, r represents the reward, and s′ represents the state at the next moment;

[0017] Input (s, a) to the target domain dynamics model, and output a Gaussian distribution of the prediction (r, s′) from the target domain dynamics model; and

[0018] The target domain dynamics model is trained using the maximum likelihood estimation with the true (r, s′) as the label.

[0019] In one embodiment of the present invention, the value-driven cross-domain policy generalization method further includes training a value function network, wherein the value function is learned using a speaker-critic algorithm and iterated using a Bellman operator;

[0020] The experience stored in the offline dataset of the value function network is a set of state transition tuples (s, a, r, s′), where s represents the state, a represents the action, r represents the reward, and s′ represents the state at the next moment. The Bellman operator sets the learning objective of the value function Q(s, a) as follows: Where a′ represents the greedy action at the next moment; the Bellman loss L is expressed as follows L = (Q(s, a) - y) 2 , and the value function can be trained by minimizing the loss function.

[0021] In one embodiment of the present invention, it is provided that the generated source domain state value Q is compared src And the target domain state value set The following steps are involved:

[0022] Based on the target domain state value set Construct the Gaussian distribution of the target domain state value, which is expressed as follows:

[0023] as well as

[0024] Determine the source domain state value Q srcThe likelihood estimate in the target domain state value Gaussian distribution is used to quantify the cross-domain value gap of the state-action data pair (s, a), which is expressed as follows:

[0025]

[0026] The larger the likelihood estimate, the smaller the cross-domain value gap.

[0027] The present invention also proposes a value-driven cross-domain policy generalization system, which is characterized by comprising:

[0028] a data providing module, configured to provide source domain training data and target domain training data;

[0029] A data filtering module is configured to perform the following actions:

[0030] Provide state-action data pair (s, a) to determine the next state s′ of the source domain src ,and

[0031] Determine the state set of the target domain at the next moment through the target domain dynamics model

[0032] The next state s′ of the source domain is calculated through the value function network src And the state set of the target domain at the next moment Perform value evaluation to generate the source domain state value Q src And the target domain state value set as well as

[0033] Compare the generated source domain state value Qsrc and the target domain state value set Filter out source domain training data whose state value gap is less than a threshold ∈; and

[0034] The data training module is configured to input the filtered source domain training data and target domain training data into the intelligent agent for training.

[0035] In one embodiment of the present invention, the value-driven cross-domain policy generalization system further includes a target domain dynamics model learning module, which is configured to perform the following actions:

[0036] Determine target domain training data, where the target domain training data includes a state transition tuple (s, a, r, s′), where s represents the state, a represents the action, r represents the reward, and s′ represents the state at the next moment;

[0037] Input (s, a) to the target domain dynamics model, and output a Gaussian distribution of the prediction (r, s′) from the target domain dynamics model; and

[0038] The target domain dynamics model is trained using the maximum likelihood estimation with the true (r, s′) as the label.

[0039] In one embodiment of the present invention, the value-driven cross-domain policy generalization system further includes a value function network learning module, which is configured to train the value function network, wherein the value function is learned using the orator-critic algorithm and iterated through the Bellman operator;

[0040] The experience stored in the offline dataset of the value function network is a set of state transition tuples (s, a, r, s′), where s represents the state, a represents the action, r represents the reward, and s′ represents the state at the next moment. The Bellman operator sets the learning objective of the value function Q(s, a) as follows: Where a′ represents the greedy action at the next moment; the Bellman loss L is expressed as follows L = (Q(s, a) - y) 2 , and the value function can be trained by minimizing the loss function.

[0041] The present invention also provides a computer-readable storage medium having a computer program stored thereon, which performs the steps of the method when executed by a processor.

[0042] The present invention further provides a computer system comprising:

[0043] a processor configured to execute machine-executable instructions; and

[0044] A memory having machine-executable instructions stored thereon, which, when executed by a processor, perform the steps according to the method.

[0045] The present invention has at least the following beneficial effects: The present invention proposes a value-driven cross-domain policy generalization method, which does not require obtaining a parameterized simulator, does not require precise fitting of a dynamic model, and does not require dynamics-independent policy assumptions. While the source domain itself does not need to change, the algorithm no longer focuses on the dynamic consistency of the data, but instead filters the source domain data based on the cross-domain consistency of value. Specifically, this method predicts the next-moment state for the corresponding target domain generated for any source domain data, and compares the value similarity between the next-moment state of the target domain and the next-moment state of the original source domain under the same value function. Based on the value consistency, source domain data with a higher degree of value similarity can be screened for subsequent training, thereby enhancing the cross-domain generalization ability of the intelligent agent to a changing environment. This method can effectively improve the generalization ability of strategies in reinforcement learning, and can effectively save computing resources compared to the existing technology. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] To further illustrate the advantages and features of various embodiments of the present invention, a more detailed description of various embodiments of the present invention will be presented with reference to the accompanying drawings. It will be understood that these drawings depict only typical embodiments of the present invention and are not to be considered as limiting the scope of the present invention. In the drawings, for clarity, identical or corresponding components will be represented by the same or similar reference numerals.

[0047] Figure 1 A schematic diagram of a computer system implementing the system and / or method according to the present invention is shown.

[0048] Figure 2 A flow chart of a value-driven cross-domain policy generalization method in an embodiment of the present invention is shown.

[0049] Figure 3 A schematic diagram of a value-driven cross-domain policy generalization system in an embodiment of the present invention is shown.

[0050] Figure 4 A schematic diagram comparing the training results of this method and the prior art is shown. DETAILED DESCRIPTION

[0051] It should be noted that the components in the drawings may be shown exaggeratedly for the sake of illustration and are not necessarily correct to scale. In the drawings, identical or functionally identical components are provided with the same reference numerals.

[0052] In the present invention, unless otherwise specified, the phrases "disposed on," "disposed above," and "disposed above" do not exclude the presence of intermediate components. Furthermore, "disposed on or above" merely indicates the relative positional relationship between two components and, in certain circumstances, such as after reversing the product orientation, can be converted to "disposed below or below," and vice versa.

[0053] In the present invention, each embodiment is only intended to illustrate the aspects of the present invention and should not be construed as limiting.

[0054] In the present invention, unless otherwise specified, the quantifiers "a" and "an" do not exclude the presence of multiple elements.

[0055] It should also be noted that in the embodiments of the present invention, for the sake of clarity and simplicity, only a portion of the parts or components may be shown, but those skilled in the art will understand that, under the teachings of the present invention, the required parts or components can be added according to the needs of the specific scenario. In addition, unless otherwise stated, the features of different embodiments of the present invention can be combined with each other. For example, a feature in the second embodiment can be used to replace a corresponding or functionally identical or similar feature in the first embodiment, and the resulting embodiment also falls within the scope of disclosure or description of this application.

[0056] It should also be noted that, within the scope of the present invention, terms such as "same," "equal," and "equal to" do not imply absolute equality of values, but rather allow for a certain reasonable error. In other words, such terms also encompass "substantially the same," "substantially equal," and "substantially equal." Similarly, in the present invention, terms such as "perpendicular to" and "parallel to" indicating direction also encompass the meaning of "substantially perpendicular to" and "substantially parallel to."

[0057] In addition, the numbering of the steps of the methods of the present invention does not limit the order in which the steps are to be performed. Unless otherwise specified, the steps of the methods may be performed in different orders.

[0058] The present invention will be further described below with reference to the accompanying drawings in conjunction with specific embodiments.

[0059] Figure 1 The computer system 100 for implementing the system and / or method according to the present invention is shown. Unless otherwise specified, the method and / or system according to the present invention may be Figure 1 The present invention may be implemented in the computer system 100 shown, or the present invention may be implemented in a distributed manner in multiple computer systems 100 according to the present invention via a network, such as a local area network or the Internet. The computer system 100 of the present invention may include various types of computer systems, such as handheld devices, laptop computers, personal digital assistants (PDAs), multi-processor systems, microprocessor-based or programmable consumer electronic devices, network PCs, minicomputers, mainframe computers, network servers, tablet computers, etc.

[0060] like Figure 1 As shown, the computer system 100 includes a processor 111, a system bus 101, a system memory 102, a video adapter 105, an audio adapter 107, a hard drive interface 109, an optical drive interface 113, a network interface 114, and a universal serial bus (USB) interface 112. The system bus 101 can be any of several types of bus structures, such as a memory bus or memory controller, a peripheral bus, and a local bus using various types of bus architectures. The system bus 101 is used for communication between various bus devices. In addition to Figure 1 In addition to the bus devices or interfaces shown in , other bus devices or interfaces are also conceivable. System memory 102 includes read-only memory (ROM) 103 and random access memory (RAM) 104, wherein ROM 103 can, for example, store basic input / output system (BIOS) data for implementing basic routines for information transmission at startup, and RAM 104 is used to provide the system with a running memory with faster access speed. Computer system 100 also includes a hard disk drive 109 for reading and writing to hard disk 110, an optical drive interface 113 for reading and writing to optical media such as CD-ROM, etc. Hard disk 110 can, for example, store an operating system and application programs. The drive and its associated computer-readable medium provide non-volatile storage of computer-readable instructions, data structures, program modules and other data for computer system 100. Computer system 100 can also include a video adapter 105 for image processing and / or image output, which is used to connect to output devices such as display 106. The computer system 100 may further include an audio adapter 107 for audio processing and / or audio output, which is used to connect to output devices such as speakers 108. In addition, the computer system 100 may further include a network interface 114 for network connection, wherein the network interface 114 may be connected to the Internet 116 via a network device such as a router 115, wherein the connection may be wired or wireless. In addition, the computer system 100 may further include a universal serial bus (USB) interface 112 for connecting to peripheral devices, wherein the peripheral devices include, for example, a keyboard 117, a mouse 118, and other peripheral devices such as a microphone, a camera, etc.

[0061] When the present invention Figure 1 When implemented on the computer system 100, there is no need to obtain a parameterized simulator, there is no need to accurately fit the dynamics model, and there is no need for dynamics-independent policy assumptions. While the source domain itself does not need to change, the algorithm no longer focuses on the dynamic consistency of the data, but instead filters the source domain data based on the cross-domain consistency of value. Specifically, this method generates a corresponding target domain prediction for any source domain data at the next moment, and compares the value similarity between the target domain state at the next moment and the original source domain state at the next moment under the same value function. Based on this value consistency, source domain data with a higher degree of value similarity can be filtered for subsequent training. This method can effectively improve the generalization ability of strategies in reinforcement learning, and thus can effectively save computing resources.

[0062] Furthermore, the various embodiments may be provided as a computer program product that may include one or more machine-readable media having machine-executable instructions stored thereon, which, when executed by one or more machines, such as computers, computer networks, or other electronic devices, may cause the one or more machines to perform operations according to the various embodiments of the present invention. The machine-readable medium may include, but is not limited to, a floppy disk, an optical disk, a CD-ROM (Compact Disk Read Only Memory) and a magneto-optical disk, a ROM (Read Only Memory), a RAM (Random Access Memory), an EPROM (Erasable Programmable Read Only Memory), an EEPROM (Electrically Erasable Programmable Read Only Memory), a magnetic or optical card, a flash memory, or other types of media / machine-readable media suitable for storing machine-executable instructions.

[0063] Furthermore, various embodiments may be downloaded as a computer program product, wherein the program may be transmitted from a remote computer (e.g., a server) to a requesting computer (e.g., a client) via a communication link (e.g., a modem and / or a network connection) using one or more data signals implemented and / or modulated by a carrier wave or other propagation medium. Thus, a machine-readable medium as used herein may, but is not required to, include such a carrier wave.

[0064] In the present invention, each module of the system according to the present invention can be implemented using software, hardware, firmware or a combination thereof. When the module is implemented using software, the function of the module can be implemented by a computer program flow, for example, the module can be implemented by a code segment (such as a code segment in a language such as C, C++) stored in a storage device (such as a hard disk, a memory, etc.), wherein when the code segment is executed by a processor, the corresponding function of the module can be implemented. When the module is implemented using hardware, the function of the module can be implemented by setting a corresponding hardware structure, for example, by hardware programming a programmable device such as a field programmable gate array (FPGA) to implement the function of the module, or by designing an application-specific integrated circuit (ASIC) including electronic devices such as a plurality of transistors, resistors and capacitors to implement the function of the module. When the module is implemented using firmware, the function of the module can be written into a read-only memory such as an EPROM or EEPROM of the device in the form of program code, and when the program code is executed by the processor, the corresponding function of the module can be implemented. In addition, certain functions of the module may need to be implemented by separate hardware or through collaboration with the hardware, for example, the detection function is implemented by corresponding sensors (such as proximity sensors, accelerometers, gyroscopes, etc.), the signal transmission function is implemented by corresponding communication devices (such as Bluetooth devices, infrared communication devices, baseband communication devices, Wi-Fi communication devices, etc.), the output function is implemented by corresponding output devices (such as displays, speakers, etc.), and so on.

[0065] Figure 2FIG. 1 shows a flow chart of a value-driven cross-domain policy generalization method in an embodiment of the present invention, as shown in FIG. Figure 2 As shown, the method includes:

[0066] Step 201: Provide source domain training data and target domain training data.

[0067] Step 202: Provide state-action data pair (s, a) to determine the next state s′ of the source domain src , and determine the state set of the target domain at the next moment through the target domain dynamics model

[0068] Step 203: The next state s′ of the source domain is calculated through the value function network. src And the state set of the target domain at the next moment Perform value evaluation to generate the source domain state value Q src And the target domain state value set

[0069] Step 204: Compare the generated source domain state value Q src And the target domain state value set Filter out source domain training data whose state value gap is less than the threshold ∈.

[0070] Step 205: Input the filtered source domain training data and target domain training data into the intelligent agent for training.

[0071] The present invention proposes a value-driven data screening method to achieve cross-dynamic adaptation of strategies by selectively sharing source domain data. Specifically, the algorithm uses the trained source domain dynamics model to obtain paired next-time-step states for the same state-action pair, and then the algorithm quantifies the value of the next-time-step state in the above paired data based on the value function network in training. When the gap between the target domain state value and the source domain state value in the above paired data is small enough, the algorithm will use the source domain data for subsequent training of the strategy network and the value function network. Compared with the existing method based on single-step dynamics difference, the value function-driven data filtering method proposed in the present invention has a better target domain strategy performance guarantee.

[0072] Figure 3 FIG. 1 shows a schematic diagram of a value-driven cross-domain policy generalization system in an embodiment of the present invention. Figure 3 As shown in Figure 1, given the training data of the source domain and the target domain, in order to prevent the use of all source domain data from affecting the performance of the strategy in the target domain, this method selects some source domain data that is beneficial to strategy training based on value-driven data screening. The value-driven data screening system includes:

[0073] The target domain dynamics model learning module uses supervised learning to train the target domain dynamics model. Given the target domain data (that is, a set of state transition tuples (s, a, r, s′)), which includes the state s, action a, reward r, and the next moment state s′. The input of the target domain dynamics model is (s, a), and the output is a Gaussian distribution of predicted (r, s′). The target domain dynamics model is trained using the maximum likelihood estimation with the true (r, s′) as the label. In addition, the dynamics model can be trained using an integrated network, which includes multiple independent dynamics models, each with different initialization parameters and the same network architecture.

[0074] The value function network learning module uses the Actor-Critic algorithm in reinforcement learning to learn the value function through the critic model and iterates through the Bellman operator. The experience stored in the offline dataset is a set of state transition tuples (s, a, r, s′), each of which contains state s, action a, reward r, and next state s′. The Bellman operator sets the learning goal of the value function Q(s, a) as Where a′ is the greedy action at the next moment. The Bellman loss is defined as L = (Q(s, a) - y) 2 A single value function can be trained by minimizing the loss function. The original value function network can be expanded to include two independent networks, each with the same network structure but different initialization parameters. When using the Bellman operator, a value function network with a relatively small value target can be used to suppress the problem of overestimation of the value function during training.

[0075] Since in the process of filtering source domain data based on value drive, given a state-action data pair (s, a), only the corresponding source domain state s′ at the next moment is available. src , but there is no corresponding target domain state at the next moment, so the target domain state prediction module needs to predict the target domain state at the next moment based on the target domain dynamics model. Based on the integrated dynamics model, the same number of target domain state sets at the next moment can be predicted This is to prevent biased state predictions caused by prediction errors of a single dynamic model itself.

[0076] Obtain the next state s' of the source domain src And the state set of the target domain at the next moment Finally, the cross-domain state value comparison module can use the above-trained value function network to evaluate the value of these states and finally obtain the corresponding source domain state value Q src and the target domain state value set In order to compare cross-domain state values, we can first construct the target domain state value Gaussian distribution based on the target domain state value set, which can be expressed as follows:

[0077]

[0078] Then get the source domain state value Q src Likelihood estimate in this distribution To quantify the cross-domain value gap of the state-action data pair (s, a), the larger the likelihood estimate, the smaller the cross-domain value gap;

[0079] Finally, the cross-domain state value gap is compared with the pre-set threshold ∈. When the gap is smaller than the threshold, data sharing is performed. Specifically, the source domain data tuple (s, a, r, s′ src ) for subsequent Actor-Critic algorithm training.

[0080] The performance of strategies trained using value-driven data screening can be evaluated in dynamic offset scenarios across a variety of continuous robotic control tasks. For example, four robotic simulation environments can be provided, each offering two forms of dynamic offset (kinematic offset and morphological offset). The robot simulation environments include bipedal and quadruped robots, enabling a comprehensive evaluation of the application value and practical effects of this method. Furthermore, this method can transfer control strategies from simulation environments to real robotic systems.

[0081] According to this method, multiple robot-related tasks are trained on the real physical simulator MuJoCo, and the training results are compared with those of the existing technology. The comparison results are as follows: Figure 4 As shown, the broken line 401 represents the patented method, the broken line 402 represents the best method currently, and the broken line 403 represents other related methods. As can be seen from the comparison, the effect achieved by the present method is better than that of other related technical methods currently.

[0082] Although various embodiments of the present invention have been described above, it should be understood that they are presented by way of example only and not limitation. It will be apparent to those skilled in the relevant art that various combinations, modifications, and variations may be made thereto without departing from the spirit and scope of the present invention. Therefore, the breadth and scope of the present invention disclosed herein should not be limited by the exemplary embodiments disclosed above, but should be defined solely in accordance with the appended claims and their equivalents.

Claims

1. A value-driven cross-domain policy generalization method, characterized by: The following steps are involved: Provide source domain training data and target domain training data: Filter source domain training data, including: Provide state-action data pair (s, a) to determine the next state s′ of the source domain src , and determine the state set of the target domain at the next moment through the target domain dynamics model The next state s′ of the source domain is calculated through the value function network src And the state set of the target domain at the next moment Perform value evaluation to generate the source domain state value Q src And the target domain state value set as well as Compare the generated source domain state value Q src And the target domain state value set Filter out source domain training data whose state value gap is less than a threshold ∈; and The filtered source domain training data and target domain training data are input into the agent for training.

2. The value-driven cross-domain policy generalization method according to claim 1 is characterized in that: It also includes training the target domain dynamics model, which includes: Determine target domain training data, where the target domain training data includes a state transition tuple (s, a, r, s′), where s represents the state, a represents the action, r represents the reward, and s′ represents the state at the next moment; Input (s, a) to the target domain dynamics model, and output a Gaussian distribution of the prediction (r, s′) from the target domain dynamics model; and The target domain dynamics model is trained using the maximum likelihood estimation with the true (r, s′) as the label.

3. The value-driven cross-domain policy generalization method according to claim 1 is characterized in that: It also includes training the value function network, where the value function is learned using the orator-critic algorithm and iterated through the Bellman operator; The experience stored in the offline dataset of the value function network is a set of state transition tuples (s, a, r, s′), where s represents the state, a represents the action, r represents the reward, and s′ represents the state at the next moment. The Bellman operator sets the learning objective of the value function Q(s, a) as follows: Where a′ represents the greedy action at the next moment; the Bellman loss L is expressed as follows L = (Q(s, a) - y) 2 , and the value function can be trained by minimizing the loss function.

4. The value-driven cross-domain policy generalization method according to claim 1 is characterized in that: Compare the generated source domain state value Q src And the target domain state value set The following steps are involved: Based on the target domain state value set Construct the Gaussian distribution of the target domain state value, which is expressed as follows: as well as Determine the source domain state value Q src The likelihood estimate in the target domain state value Gaussian distribution is used to quantify the cross-domain value gap of the state-action data pair (s, a), which is expressed as follows: The larger the likelihood estimate, the smaller the cross-domain value gap.

5. A value-driven cross-domain policy generalization system, characterized by: include: a data providing module, configured to provide source domain training data and target domain training data; A data filtering module is configured to perform the following actions: Provide state-action data pair (s, a) to determine the next state s′ of the source domain src , and determine the state set of the target domain at the next moment through the target domain dynamics model The next state s′ of the source domain is calculated through the value function network src And the state set of the target domain at the next moment Perform value evaluation to generate the source domain state value Q src And the target domain state value set as well as Compare the generated source domain state value Q src And the target domain state value set Filter out source domain training data whose state value gap is less than the threshold ∈; as well as The data training module is configured to input the filtered source domain training data and target domain training data into the intelligent agent for training.

6. The value-driven cross-domain policy generalization system according to claim 5, characterized in that: Also included is a target domain dynamics model learning module, which is configured to perform the following actions: Determine target domain training data, where the target domain training data includes a state transition tuple (s, a, r, s′), where s represents the state, a represents the action, r represents the reward, and s′ represents the state at the next moment; Input (s, a) to the target domain dynamics model, and output a Gaussian distribution of the prediction (r, s′) from the target domain dynamics model; and The target domain dynamics model is trained using the maximum likelihood estimation with the true (r, s′) as the label.

7. The value-driven cross-domain policy generalization system according to claim 5, characterized in that: Also included is a value function network learning module configured to train the value function network, wherein the value function is learned using an orator-critic algorithm and iterated through a Bellman operator; The experience stored in the offline dataset of the value function network is a set of state transition tuples (s, a, r, s′), where s represents the state, a represents the action, r represents the reward, and s′ represents the state at the next moment. The Bellman operator sets the learning objective of the value function Q(s, a) as follows: Where a′ represents the greedy action at the next moment; the Bellman loss L is expressed as follows L = (Q(s, a) - y) 2 , and the value function can be trained by minimizing the loss function.

8. A computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the computer program performs the steps of the method according to any one of claims 1 to 4.

9. A computer system comprising: a processor configured to execute machine-executable instructions; as well as A memory having machine executable instructions stored thereon, wherein the machine executable instructions, when executed by a processor, perform the steps of the method according to any one of claims 1 to 4.

Images