Decision-making method and device universal for multiple scenes

Abstract

The invention provides a decision-making method and device universal for multiple scenes, and is applied to the technical field of intelligent decision-making. The method comprises the following steps: acquiring an observed value and an action value of an agent at the current moment; inputting the observation value and the action value into a task state prompter module to obtain a task state prompting sequence; inputting the task state prompt sequence into a generalized dynamics module to obtain an observation value of the agent at the next moment; wherein the generalized dynamics module is a circulating state space model based on a converter. Due to the fact that the problem that dynamics is difficult to fit in different scenes can be solved, the learning return rate of the multi-scene offline reinforcement learning method can be increased, and the multi-scene universal requirement of the decision-making method is met.

Description

Technical Field

[0001] The present invention relates to the field of intelligent decision-making technology, and in particular to a decision-making method and device applicable to multiple scenarios. Background Art

[0002] Model-based reinforcement learning methods show potential advantages in offline settings. In multi-scenario offline reinforcement learning, it is necessary to use multi-scenario offline datasets to learn the general ability of the agent to adapt to different environments. However, the learning return rate of multi-scenario offline reinforcement learning methods in the existing technology is low. Summary of the invention

[0003] The present invention provides a decision-making method and device that are universal to multiple scenarios, so as to solve the problem of making an intelligent agent adapt to different environments.

[0004] The present invention provides a decision-making method universal for multiple scenarios, comprising: obtaining the observation value and action value of an intelligent agent at the current moment; inputting the observation value and the action value into a task state prompt module to obtain a task state prompt sequence; inputting the task state prompt sequence into a generalized dynamics module to obtain the observation value of the intelligent agent at the next moment; wherein the generalized dynamics module is a cyclic state space model based on a transformer.

[0005] According to a decision-making method that is universal for multiple scenarios provided by the present invention, after inputting the observation value and the action value into a task status prompt module to obtain a task status prompt sequence, the method further includes: inputting the task status prompt sequence and the observation value at the next moment into a reward module to obtain a current reward.

[0006] According to a decision-making method universal for multiple scenarios provided by the present invention, the generalized dynamics module includes a deterministic state model, a random state model and an observation decoding model.

[0007] According to a decision-making method that is universal for multiple scenarios provided by the present invention, the task state prompt sequence is input into a generalized dynamics module to obtain the observation value of the agent at the next moment, including: inputting the task state prompt sequence into the deterministic state model to obtain a deterministic state; inputting the deterministic state into the random state model to obtain a random state; inputting the deterministic state and the random state into the observation decoding model to obtain the observation value of the agent at the next moment.

[0008] The present invention also provides a decision-making device that is universal for multiple scenarios, comprising the following modules: an acquisition module and a processing module; the acquisition module is used to acquire the observation value and action value of the intelligent agent at the current moment; the processing module is used to input the observation value and the action value into the task status prompt module to obtain a task status prompt sequence; the task status prompt sequence is input into the generalized dynamics module to obtain the observation value of the intelligent agent at the next moment; wherein the generalized dynamics module is a cyclic state space model based on a transformer.

[0009] According to a decision-making device universal for multiple scenarios provided by the present invention, the processing module is used to input the task status prompt sequence and the observation value at the next moment into the reward module to obtain the current reward.

[0010] According to a decision-making device universal for multiple scenarios provided by the present invention, the generalized dynamics module includes a deterministic state model, a random state model and an observation decoding model.

[0011] According to a decision-making device universal for multiple scenarios provided by the present invention, the processing module is used to input the task state prompt sequence into the deterministic state model to obtain a deterministic state; input the deterministic state into the random state model to obtain a random state; input the deterministic state and the random state into the observation decoding model to obtain the observation value of the intelligent agent at the next moment.

[0012] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, it implements a decision-making method applicable to multiple scenarios as described in any one of the above.

[0013] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a decision-making method applicable to multiple scenarios as described in any one of the above.

[0014] The present invention also provides a computer program product, comprising a computer program, which, when executed by a processor, implements any of the above-mentioned decision-making methods applicable to multiple scenarios.

[0015] The decision-making method and device for multiple scenarios provided by the present invention can obtain the observation value and action value of the agent at the current moment; input the observation value and the action value into the task state prompt module to obtain the task state prompt sequence; input the task state prompt sequence into the generalized dynamics module to obtain the observation value of the agent at the next moment; wherein the generalized dynamics module is a cyclic state space model based on a transformer. Through this scheme, the task state prompt sequence can be input into the generalized dynamics module to obtain the observation value of the agent at the next moment. Since the generalized dynamics module can solve the problem that dynamics is difficult to fit in different scenarios, the learning return rate of the multi-scenario offline reinforcement learning method can be improved, thereby realizing the multi-scenario universal requirements of the decision-making method. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0017] Figure 1 This is one of the flowcharts of the decision-making method universal to multiple scenarios provided by the present invention; Figure 2 This is the second flowchart of the decision-making method universal for multiple scenarios provided by the present invention; Figure 3 This is the third flowchart of the decision-making method universal for multiple scenarios provided by the present invention; Figure 4 It is a schematic diagram of the structure of a decision-making device universal for multiple scenarios provided by the present invention; Figure 5 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0018] In order to make the purpose, technical solutions and advantages of this application clearer, the technical solutions in this application will be clearly and completely described below in conjunction with the drawings in this application. Obviously, the described embodiments are part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0019] It should be noted that, in the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0020] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0021] In order to facilitate the clear description of the technical solutions of the embodiments of the present application, in the embodiments of the present application, the words "first", "second", etc. are used to distinguish the same items or similar items with basically the same functions and effects. Those skilled in the art can understand that the words "first", "second", etc. are not limiting the quantity and execution order.

[0022] The embodiments of the present application describe some exemplary embodiments for the purpose of explanation. It should be understood that the present application can be implemented in other ways that are not specifically shown in the drawings.

[0023] Model-based reinforcement learning methods show potential advantages in offline settings. Benefiting from the flexibility of modifying the reward function and the stability of the optimization process, model-based reinforcement learning methods can perform well in multi-task problems. Therefore, it is feasible to use model-based methods to solve offline reinforcement learning problems in multiple scenarios.

[0024] This application uses a learned model as a substitute for the real environment in the offline policy evaluation and learning process. To effectively play the role of the world requires the model to be able to fit the dynamic knowledge of different types of environments. Inspired by the success of multi-task prompt methods in vision and language, this application decided to incorporate the inductive bias of this architecture into the learning process of the world model. Task-prompt can encode specific task information for sequence prediction models to eliminate ambiguity between different tasks.

[0025] This application proposes a task-cued generalizable world model (TGW) framework that combines cues and model-based methods to solve offline reinforcement learning problems in multiple scenarios. TGW is used to learn from various datasets. The paradigm of learning a generalized world model. This application assumes an environment representing a single task. It is a generalized world model A subset of .

[0026] TGW will take into account different transfers and rewards Impact on the final task. Figure 1 As shown, TGW consists of three modules, namely, task state prompter module, generalized dynamics module and reward module. These three modules are flexible and can be configured according to actual requirements.

[0027] The function of TGW is the same as that of the real task environment, interacting with the intelligent agent based on the current observation value. and action value As input, take the current reward and the state value at the next moment as output.

[0028] Since policy optimization is based on the world model, accurate and reliable world models can be generated under the policy. The return should be as close as possible to the return in the real environment. . Therefore, the ultimate goal of TGW is to minimize the following expression.

[0029] .

[0030] like Figure 2 As shown, the embodiment of the present application provides a decision method applicable to multiple scenarios, and the decision method applicable to multiple scenarios can be applied to a decision device applicable to multiple scenarios. The decision method applicable to multiple scenarios can include S101-S103: S101. A decision-making device applicable to multiple scenarios obtains the observation value and action value of the intelligent agent at the current moment.

[0031] The decision-making device that is universal in multiple scenarios can obtain the observation value of the agent at the current moment and action value .

[0032] S102, the decision-making device applicable to multiple scenarios inputs the observation value and the action value into a task status prompt module to obtain a task status prompt sequence.

[0033] S103, a decision-making device common to multiple scenarios inputs the task state prompt sequence into a generalized dynamics module to obtain the observation value of the agent at the next moment.

[0034] The generalized dynamics module is a converter-based cyclic state-space model.

[0035] Optionally, the generalized dynamics module includes a deterministic state model, a random state model and an observation decoding model. The decision-making device universal to multiple scenarios can input the task state prompt sequence into the deterministic state model to obtain a deterministic state; input the deterministic state into the random state model to obtain a random state; input the deterministic state and the random state into the observation decoding model to obtain the observation value of the agent at the next moment.

[0036] Alternatively, if Figure 1 As shown, after the observation value and the action value are input into the task status prompt module to obtain the task status prompt sequence, the decision-making device applicable to multiple scenarios can input the task status prompt sequence and the observation value at the next moment into the reward module to obtain the current reward.

[0037] The decision-making method for multiple scenarios provided in the embodiment of the present application can be executed by a decision-making device for multiple scenarios, or a control module for decision-making for multiple scenarios in the decision-making device for multiple scenarios. In the embodiment of the present application, the decision-making device for multiple scenarios provided in the embodiment of the present application is described by taking the decision-making method for multiple scenarios executed by the decision-making device for multiple scenarios as an example, so as to realize the multi-scenario universal requirements of the decision-making method.

[0038] Specifically, this application uses the sequence decision trajectory of offline data D As a task status reminder This application uses a multi-layer perceptron neural network (MLP) to transform the original observations Mapping to latent space , and then tokenize the data. Figure 3 As shown, this application prompts the task status of task k at time t The definitions are as follows: ; in, .

[0039] 2M is the task prompt The number of is generally equal to the length of the context of the sequence prediction. At the beginning of the evaluation, Task-specific datasets Then, based on the first-in, first-out mechanism It is gradually replaced by historical information of the interaction between the intelligent agent and the real environment.

[0040] In order to learn a general dynamics model with better generalization ability and more accuracy, this application proposes the TransRSSM network structure, which combines the advantages of the transformer and the recursive state space model (RSSM). The transformer is a model that is very suitable for establishing sequence prediction relationships in abstract space because it has a very outstanding ability in extracting contextual relationships. In RSSM, this application assumes that the latent space of the dynamics contains deterministic states and randomness state ,This approach has been shown to be very effective in PlaNet experiments, dividing the states into deterministic states and stochastic states will improve the robustness of the world model prediction.

[0041] The input of TransRSSM is the tokenized latent state , which contains the current deterministic state , the output is the original observation , will be sent to the agent as input. Initial determination state It is a probability distribution First, the application uses the transformer to predict the deterministic state in the next step , task status prompt As input, the deterministic state of the next step will be generated by the transformer model based on the contextual information of the task prompt.

[0042] This application assumes that the random state satisfies the normal distribution, which can be obtained by Then this application uses MLP to predict the mean and variance of the random state, which can be expressed as Finally, the next observed state that the agent can receive will be determined by the observation model predict.

[0043] Therefore, the TransRSSM model of this application consists of the following three components: Deterministic state model: ; Random state model: ; Observation decoding model: .

[0044] This application also introduces uncertainty estimation of the state in actual deployment training, and uses this uncertainty estimation as an internal reward for policy learning, so that policy learning can avoid states where the world model predicts inaccurately during exploration. The ideal way to learn a dynamics model is to have enough data to cover the entire space. However, this is unrealistic, especially in the offline reinforcement learning setting, which is bound to be affected by distribution drift. Therefore, this application estimates the uncertainty of the generalized dynamics model as an internal reward for policy optimization. Estimating the uncertainty of functions is an important research area in machine learning, which is also useful for learning world models in reinforcement learning. This application uses an ensemble method to train models with different initial parameters. N dynamic models of . We then use the variance of the internal deterministic state as an estimate of uncertainty. We define the internal reward: For external rewards, this application will use MLP for learning, which can be expressed as: The final reward expression is: .in is a hyperparameter that represents the agent’s risk tolerance level. The better the behavior strategy performs in the offline dataset, the less exploration should be done outside the data.

[0045] During the training process, the length of the input sequence Token is 2M. and There is a supervision label for each. The model parameters are , its predicted probability distribution can be expressed as: .

[0046] Because this application is modeling the dynamics function, it does not calculate the loss of the behavior strategy in the sequence token. Obviously, the more diverse the actions in the same state, the more accurate the dynamics fit. Finally, for batch processing Next For each task, the training loss can be expressed as:

[0047] In the embodiments of the present application, a new task setting, named multi-scenario offline reinforcement learning, is proposed, which aims to learn a high-reward strategy based on a fixed data set across scenes; a task-cued general world model (TGW) framework suitable for various environments is proposed, which can be effectively generalized to visible and invisible tasks; TransRSSM is proposed, which combines the advantages of transformers and recursive state space models to fit the dynamics function.

[0048] The above mainly introduces the solution provided by the embodiment of the present application from the perspective of the method. In order to realize the above functions, it includes hardware structures and / or software modules corresponding to the execution of each function. Those skilled in the art should easily realize that, in combination with the units and algorithm steps of each example described in the embodiment disclosed herein, the embodiment of the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present application.

[0049] It should be noted that the device in the embodiment of the present application includes a virtual device and a physical device. The virtual device can be a decision-making device that is universal in multiple scenarios, and the physical device can include an electronic device, a computer storage medium, and a computer program product.

[0050] It should be noted that the embodiment of the present application can divide the functional modules of the decision-making device common to multiple scenarios according to the above method example. For example, each functional module can be divided corresponding to each function, or two or more functions can be integrated into one processing module. The above integrated module can be implemented in the form of hardware or in the form of software functional modules. Optionally, the division of modules in the embodiment of the present application is schematic and is only a logical function division. There may be other division methods in actual implementation.

[0051] like Figure 4 As shown, the embodiment of the present application provides a decision-making device 400 that is universal for multiple scenarios. The decision-making device 400 that is universal for multiple scenarios includes: an acquisition module 401 and a processing module 402. The acquisition module 401 can be used to obtain the observation value and action value of the agent at the current moment; the processing module 402 is used to input the observation value and the action value into the task state prompt module to obtain the task state prompt sequence; the task state prompt sequence is input into the generalized dynamics module to obtain the observation value of the agent at the next moment; wherein the generalized dynamics module is a cyclic state space model based on a transformer.

[0052] Optionally, the processing module 402 may also be used to input the task status prompt sequence and the observation value at the next moment into a reward module to obtain a current reward.

[0053] Optionally, the generalized dynamics module includes a deterministic state model, a random state model and an observation decoding model.

[0054] Optionally, the above-mentioned processing module 402 is used to input the task state prompt sequence into the deterministic state model to obtain a deterministic state; input the deterministic state into the random state model to obtain a random state; input the deterministic state and the random state into the observation decoding model to obtain the observation value of the agent at the next moment.

[0055] In an embodiment of the present application, the task state prompt sequence can be input into a generalized dynamics module to obtain the observation value of the agent at the next moment. Since the generalized dynamics module can solve the problem that dynamics are difficult to fit in different scenarios, the learning return rate of the multi-scenario offline reinforcement learning method can be improved, thereby realizing the multi-scenario general requirements of the decision-making method.

[0056] Figure 5 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 5 As shown, the electronic device may include: a processor 510, a communication interface 520, a memory 530 and a communication bus 540, wherein the processor 510, the communication interface 520 and the memory 530 communicate with each other through the communication bus 540. The processor 510 may call the logic instructions in the memory 530 to execute a decision-making method common to multiple scenarios, the method comprising: obtaining the observation value and action value of the agent at the current moment; inputting the observation value and the action value into the task state prompt module to obtain the task state prompt sequence; inputting the task state prompt sequence into the generalized dynamics module to obtain the observation value of the agent at the next moment; wherein the generalized dynamics module is a cyclic state space model based on a transformer.

[0057] In addition, the logic instructions in the above-mentioned memory 530 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0058] On the other hand, the present invention also provides a computer program product, which includes a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the decision-making method that is universal for multiple scenarios provided by the above-mentioned methods, and the method includes: obtaining the observation value and action value of the intelligent agent at the current moment; inputting the observation value and the action value into a task status prompt module to obtain a task status prompt sequence; inputting the task status prompt sequence into a generalized dynamics module to obtain the observation value of the intelligent agent at the next moment; wherein the generalized dynamics module is a cyclic state space model based on a transformer.

[0059] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the multi-scenario general decision-making method provided by the above-mentioned methods, the method comprising: obtaining the observation value and action value of the agent at the current moment; inputting the observation value and the action value into a task status prompt module to obtain a task status prompt sequence; inputting the task status prompt sequence into a generalized dynamics module to obtain the observation value of the agent at the next moment; wherein the generalized dynamics module is a transformer-based cyclic state space model.

[0060] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0061] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0062] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A decision-making method applicable to multiple scenarios, characterized in that: include: Get the observation value and action value of the agent at the current moment; Inputting the observation value and the action value into a task state prompt module to obtain a task state prompt sequence; Inputting the task state prompt sequence into a generalized dynamics module to obtain the observation value of the agent at the next moment; The generalized dynamics module is a converter-based cyclic state-space model.

2. The multi-scenario general decision-making method according to claim 1 is characterized in that: After inputting the observation value and the action value into a task status prompt module to obtain a task status prompt sequence, the method further includes: The task status prompt sequence and the observation value at the next moment are input into the reward module to obtain the current reward.

3. The multi-scenario decision-making method according to claim 1, characterized in that: The generalized dynamics module includes a deterministic state model, a random state model and an observation decoding model.

4. The multi-scenario general decision-making method according to claim 3 is characterized in that: The step of inputting the task state prompt sequence into a generalized dynamics module to obtain the observation value of the agent at the next moment includes: Inputting the task state prompt sequence into the deterministic state model to obtain a deterministic state; Inputting the deterministic state into the random state model to obtain a random state; The deterministic state and the random state are input into the observation decoding model to obtain the observation value of the agent at the next moment.

5. A decision-making device applicable to multiple scenarios, characterized in that: include: Acquisition module and processing module; The acquisition module is used to obtain the observation value and action value of the agent at the current moment; The processing module is used to input the observation value and the action value into the task state prompt module to obtain a task state prompt sequence; input the task state prompt sequence into the generalized dynamics module to obtain the observation value of the agent at the next moment; The generalized dynamics module is a converter-based cyclic state-space model.

6. The multi-scenario decision-making device according to claim 5, characterized in that: The processing module is used to input the task status prompt sequence and the observation value at the next moment into the reward module to obtain the current reward.

7. The multi-scenario decision-making device according to claim 5, characterized in that: The generalized dynamics module includes a deterministic state model, a random state model and an observation decoding model.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, it implements the decision-making method applicable to multiple scenarios as described in any one of claims 1 to 4.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the decision-making method applicable to multiple scenarios as claimed in any one of claims 1 to 4 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the decision-making method applicable to multiple scenarios as claimed in any one of claims 1 to 4 is implemented.

Images