World model disturbance method, device and equipment and storage medium

Abstract

The invention discloses a world model disturbance method and device, equipment and a storage medium. The method comprises the steps of obtaining a task scene state and a task execution action of a current execution task; the task scene state and the task execution action are input into a target disturbance model obtained through pre-training for state disturbance prediction, the target disturbance model is composed of a target world model and a random residual error model set which are the same in model dimension, and the target world model and the random residual error model set are selected from the target world model and the random residual error model set. The random residual model set comprises at least one neural network model; according to the output of the target world model, the target task scene state of the current execution task at the next moment is predicted, and the coverage rate of world model simulation data is increased in the mode that the world model outputs disturbance, so that a reinforcement learning strategy trained in the world model can have robustness for distributed data.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a world model perturbation method, device, equipment and storage medium. Background Art

[0002] World models are a method used in the fields of artificial intelligence and machine learning to describe and understand an environment. Based on the current state and past observations, a world model can predict future states. By simulating possible future states, world models help intelligent agents (such as robots or virtual agents) make better decisions, allowing them to try different actions in their internal model and choose those that maximize rewards or reach their goals.

[0003] Model-based reinforcement learning is a specific application of world models, which accelerates the process of agent reinforcement learning by leveraging world models. In this approach, the agent not only gains experience by interacting with the actual environment, but also simulates future states and results through the world model, thereby performing a lot of trial and error in such a virtual world model.

[0004] Due to the limited training data and the fact that the training data cannot cover all possible situations, the world model can often only learn accurately in some areas. On untrained data, the world model can often only rely on its generalization ability to "guess". We call "guessed" data "out-of-distribution data". The performance of the world model on out-of-distribution data is not guaranteed and is likely to be wrong. Therefore, the strategy obtained through reinforcement learning in the world model cannot make accurate decisions on out-of-distribution data. Summary of the invention

[0005] The present invention provides a world model perturbation method, device, equipment and storage medium, which increases the coverage of world model simulation data by outputting perturbations of the world model, so that the reinforcement learning strategy trained in the world model can be robust to out-of-distribution data.

[0006] According to one aspect of the present invention, a method for perturbing a world model is provided. The method comprises:

[0007] Get the task scene status and task execution action of the current task;

[0008] Inputting the task scene state and the task execution action into a pre-trained target disturbance model to perform state disturbance prediction, wherein the target disturbance model is composed of a target world model and a random residual model set of the same model dimension, and the random residual model set includes at least one neural network model;

[0009] According to the output of the target world model, the target task scene state of the currently executed task at the next moment is predicted.

[0010] According to another aspect of the present invention, a world model perturbation device is provided. The device comprises:

[0011] The task information acquisition module is used to obtain the task scene status and task execution action of the current task;

[0012] A task state disturbance module, used for inputting the task scene state and the task execution action into a pre-trained target disturbance model to perform state disturbance prediction, wherein the target disturbance model is composed of a target world model and a random residual model set of the same model dimension, and the random residual model set includes at least one neural network model;

[0013] The scene state prediction module is used to predict the target task scene state of the currently executed task at the next moment according to the output of the target world model.

[0014] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0015] at least one processor; and

[0016] a memory communicatively connected to the at least one processor; wherein,

[0017] The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can perform the world model perturbation method described in any embodiment of the present invention.

[0018] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the world model perturbation method described in any embodiment of the present invention when executed.

[0019] The technical solution of the embodiment of the present invention obtains the task scene state and task execution action of the currently executed task. The task scene state and the task execution action are input into the pre-trained target disturbance model to perform state disturbance prediction, wherein the target disturbance model is composed of a target world model and a random residual model set of the same model dimension, and the random residual model set includes at least one neural network model. According to the output of the target world model, the target task scene state of the currently executed task at the next moment is predicted. By performing parameter perturbations in the parameter space of the world model, the coverage of the world model simulation data is improved, so that the intelligent agent can see a large amount of data when performing reinforcement learning training in the world model, improve its effect on out-of-distribution data, and ultimately ensure the effect in the real physical environment.

[0020] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0022] Figure 1 is a flow chart of a world model perturbation method provided according to Embodiment 1 of the present invention;

[0023] Figure 2 is a flow chart of a world model perturbation method provided according to Embodiment 2 of the present invention;

[0024] Figure 3 is a schematic diagram of a target disturbance model provided according to Embodiment 2 of the present invention;

[0025] Figure 4 is a structural diagram of a world model perturbation device provided according to Embodiment 3 of the present invention;

[0026] Figure 5 It is a structural schematic diagram of an electronic device for implementing the world model perturbation method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0027] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0028] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0029] Embodiment 1

[0030] Figure 1 A flowchart of a world model perturbation method is provided for the first embodiment of the present invention. This embodiment is applicable to the case of improving the coverage of world model simulation data. The method can be performed by a world model perturbation device. The world model perturbation device can be implemented in the form of hardware and / or software. The world model perturbation device can be configured in an electronic device. Figure 1 As shown, the method includes:

[0031] S101, obtaining the task scene status and task execution action of the currently executed task.

[0032] The currently executed task can refer to any execution task that requires state prediction. The task scene state refers to the specific situation of the scene in which the task is currently executed. The task scene state can contain multiple types of information, depending on the application field. For example, in a game environment, the task scene state may include the character's position, speed, health, item holdings, etc.; in a robot control task, the state may include the robot's joint angles, speed, acceleration, and sensor readings, etc.

[0033] Task execution actions refer to the specific actions taken to perform the current task, that is, the behavior selected and executed by the agent according to its strategy. Task execution actions can be continuous (such as controlling the angle change of a robot arm) or discrete (such as choosing the direction of movement in a game).

[0034] Exemplarily, when the currently executed task is a maze navigation task, the task scene state can be used to describe the position of the agent in the maze and possible other related information. For example, in a maze composed of grids, each grid can be represented by a pair of coordinates (x, y). Therefore, the task scene state can be simply represented by the coordinates (x, y) of the current position of the agent, the visible part of the maze map, the direction of the agent, whether to carry collection items, and the remaining time of the task. Task execution actions can be actions that the agent can take. For example, in a maze navigation task, the task execution action is usually the direction of movement (such as moving up, moving down, moving left, and moving right, etc.). In some cases, the task execution action may also include staying still, collecting items, and placing items.

[0035] S102: Input the task scenario state and the task execution action into a pre-trained target disturbance model to perform state disturbance prediction.

[0036] It should be noted that the target perturbation model can be composed of a target world model and a random residual model set. The target world model and the random residual model should be models with the same dimensions, such as the target world model and the random residual model both have a 3-layer fully connected neural network.

[0037] The random residual model set includes at least one neural network model. The neural network model in the random residual model set is usually a simple and randomly initialized neural network, which can be used to generate perturbations without training. The neural network model in the random residual model set may include but is not limited to a fully connected network, a convolutional neural network, a recurrent neural network, a variational autoencoder, and a generative adversarial network.

[0038] The target world model can be obtained by collecting a batch of real environment data of state-action-next-moment state, and optimizing the world model using the MSE loss function and the Adam optimizer.

[0039] Specifically, the task scene state and the task execution action are used as prediction conditions and input into a pre-trained target disturbance model, so that the target disturbance model performs state disturbance prediction.

[0040] Exemplarily, the process of predicting state disturbance in the target disturbance model is as follows:

[0041] Based on the task scenario state, the task execution action and the target world model, a predicted task scenario state is obtained; based on the task scenario state, the task execution action and the random residual model set, an incremental task scenario state is determined; the predicted task scenario state and the incremental task scenario state are combined and processed to realize the state disturbance prediction of the target disturbance model.

[0042] The predicted task scene state refers to the task scene state predicted by the target world model at the next moment. The incremental task scene state may refer to the disturbance amount provided by the random residual model set.

[0043] That is to say, the task scene state and the task execution action are input into the target world model for scene state prediction, and the predicted task scene state can be obtained. The task scene state and the task execution action are input into the random residual model set for scene state prediction, and the incremental task scene state can be obtained. The predicted task scene state and the incremental task scene state are merged to obtain the final state, so as to realize the state disturbance prediction of the target disturbance model.

[0044] It should be noted that the combined processing of the predicted task scenario state and the incremental task scenario state can be the addition processing of the predicted task scenario state and the incremental task scenario state, or the predicted task scenario state and the incremental task scenario state can be respectively multiplied by their respective corresponding preset coefficients and the product results can be added.

[0045] Taking the current task as the maze navigation task, the starting state of the task scene state may be (1,1), which means that the agent is initially located in the lower left corner of the maze. If the agent chooses to move to the right, the task execution action is to move to the right. The target world model predicts the next state to be (2,1) based on the current state (1,1) and the action "move right". The neural network model in the random residual model set may give a small offset, such as (-1,0), which means that the agent may deviate slightly from the expected path. The output of the target perturbation model is (2,1)+(-1,0)=(1,1). Then the state at the next moment is updated to (1,1).

[0046] It should be noted that the examples given here are idealized, and in actual applications, the target disturbance model may give more complex outputs. In addition, the specific representation of the task scene state and the task execution action depends on the design of the maze environment and the complexity of the task. The examples are only detailed descriptions and are not limitations of the present invention.

[0047] Exemplarily, based on the task scenario state, the task execution action and the random residual model set, an incremental task scenario state is determined, including: based on a preset screening method, obtaining a target neural network model from the random residual model set; inputting the task scenario state and the task execution action into the target neural network model for incremental calculation to obtain an incremental task scenario state.

[0048] The preset screening method can be set according to actual needs, such as setting the screening method according to actual experience, or using a screening method for tasks of the same type. The present invention does not limit this. Preferably, the present invention adopts a random screening method, and the number of screenings is single. That is, a neural network model is randomly selected from a set of random residual models, and the neural network model is determined as the target neural network model. The task scene state and the task execution action are input into the target neural network model for incremental calculation, and the incremental task scene state is obtained according to the output of the target neural network model, thereby improving the randomness of the increment, and then all possible situations are simulated through the target perturbation model, in the hope of covering the real situation, so that the strategy can also see the real situation when training in the world model, and improve the robustness of the strategy on out-of-distribution data.

[0049] S103: predicting the target task scenario state of the currently executed task at the next moment according to the output of the target disturbance model.

[0050] The target task scenario state may refer to a target disturbance model performing disturbance prediction to obtain a disturbance scenario state.

[0051] Specifically, the output of the target perturbation model is received, and the output result is determined as the target task scene state of the current task at the next moment. After obtaining the target task scene state, the performance of the agent strategy is measured by defining appropriate evaluation indicators, including the performance on in-distribution data and out-of-distribution data. At the same time, the performance of the agent strategy under different perturbation intensities and the generalization ability on different tasks can also be tested.

[0052] Exemplarily, the expression for predicting the target task scenario state of the currently executed task at the next moment is as follows:

[0053] s′=f(s,a;θ)+clip(g(s,a;φ i ),-c,c);

[0054] i~uniform({1,2,...,N});

[0055] Where f(s,a;θ) refers to the target world model, g(s,a;φ i) refers to the random residual model set, N refers to the number of neural network models in the random residual model set, i refers to the i-th neural network model in the random residual model set, s refers to the task scene state, a refers to the task execution action, θ refers to the model parameters of the target world model, φ i refers to the model parameters of the random residual model set, c refers to the random perturbation range, and the clip() function is used to limit the output range of the neural network model.

[0056] Uniform means uniformly randomly selecting an element from the set in the brackets, that is, randomly selecting a neural network model from the set of random residual models. Clip() means clipping the value of the first item in the brackets so that the value is within the range of the second and third items in the brackets. The clip() function is specifically used to limit the range of the residual model output to ensure that the disturbance is not too large. The c value determines the amplitude of the disturbance and needs to be adjusted according to the actual application scenario to achieve the best disturbance effect without destroying the stability of the model, ensuring that the disturbance does not cause the state prediction to deviate too much from the normal range, thereby helping to maintain the stability of the model.

[0057] The technical solution of the embodiment of the present invention obtains the task scene state and task execution action of the currently executed task. The task scene state and the task execution action are input into the pre-trained target disturbance model to perform state disturbance prediction, wherein the target disturbance model is composed of a target world model and a random residual model set of the same model dimension, and the random residual model set includes at least one neural network model. According to the output of the target world model, the target task scene state of the currently executed task at the next moment is predicted. By performing parameter perturbations in the parameter space of the world model, the coverage of the world model simulation data is improved, so that the intelligent agent can see a large amount of data when performing reinforcement learning training in the world model, improve its effect on out-of-distribution data, and ultimately ensure the effect in the real physical environment.

[0058] Embodiment 2

[0059] Figure 2 This is a flow chart of a world model perturbation method provided in the second embodiment of the present invention. This embodiment, based on the above embodiments, concretizes the training process of the target world model. Figure 2 As shown, the method includes:

[0060] S201. Obtain sample offline data and sample task scenario states corresponding to the sample offline data.

[0061] The sample offline data at least includes historical task scene status and historical task execution actions.

[0062] Specifically, a large amount of state-action-new state-reward quadruple data needs to be collected. This can be achieved by letting the agent interact with the environment randomly or according to a certain strategy. The data set should be large and diverse enough to cover various possible states of the environment. The collected data is cleaned and preprocessed, including but not limited to removing outliers, standardizing numerical features, etc., so that sample offline data and sample task scene states can be obtained.

[0063] S202: Input the sample offline data into a preset training model to perform task scene state prediction, and obtain an output task scene state based on the output of the preset training model.

[0064] Among them, the preset training model includes at least one of a regression model, a deep neural model, a generative adversarial network or a variational autoencoder.

[0065] Specifically, the selected preset training model is trained using sample offline data, and the training goal is to enable the preset training model to accurately predict the state at the next moment and obtain the output task scenario state.

[0066] For example, for a regression model, mean square error can be used as the loss function. For a probability model, cross entropy loss or other appropriate probability metrics can be used as the loss function.

[0067] S203, determining a training error based on the output task scenario state and the sample task scenario state, and back-propagating the training error to the preset training model to adjust network parameters in the preset training model.

[0068] Specifically, the output task scenario state is compared with the sample task scenario state to determine the training error, and the training error is back-propagated to the preset network model to adjust the network parameters in the preset network model until the preset convergence conditions are met.

[0069] S204: When a preset convergence condition is met, determine that the training of the preset training model is completed and obtain a target world model.

[0070] Specifically, the preset convergence condition may include when the number of iterations reaches a preset number or the training error converges.

[0071] When the preset convergence condition is met, it is determined that the training of the preset network model is completed, and at this time, the preset network model after the training can be used as the target world model.

[0072] S205: Combining the target world model and the random residual model set to construct a target disturbance model.

[0073] Figure 3It is a schematic diagram of the target disturbance model provided by the present invention. Figure 3 As shown in the figure, the target world model and the random residual model set are combined in parallel. The task scene state and task execution action are input into the target world model and the random residual model set respectively. The target world model outputs the predicted task scene state at the next moment, while the random residual model set randomly selects a neural network model and outputs an incremental task scene state with the same dimension as the predicted task scene state through the neural network model. Finally, the outputs of the two models are added to obtain the target task scene state with disturbance at the next moment.

[0074] S206: Obtain the task scenario status and task execution action of the currently executed task.

[0075] S207: Input the task scenario state and the task execution action into a pre-trained target disturbance model to perform state disturbance prediction.

[0076] S208. Predicting the target task scenario state of the currently executed task at the next moment according to the output of the target disturbance model.

[0077] The technical solution of the present invention uses sample offline data and sample task scene states to train the target world model to avoid the reinforcement learning agent from performing high-cost trial and error directly in the real physical world, and can ensure the effectiveness of the agent in the real physical environment.

[0078] Embodiment 3

[0079] Figure 4 FIG. 1 is a schematic diagram of the structure of a world model perturbation device provided in Embodiment 3 of the present invention. Figure 4 As shown, the device comprises:

[0080] The task information acquisition module 301 is used to obtain the task scene status and task execution action of the currently executed task;

[0081] A task state disturbance module 302 is used to input the task scene state and the task execution action into a pre-trained target disturbance model to perform state disturbance prediction, wherein the target disturbance model is composed of a target world model and a random residual model set of the same model dimension, and the random residual model set includes at least one neural network model;

[0082] The scene state prediction module 303 is used to predict the target task scene state of the currently executed task at the next moment according to the output of the target disturbance model.

[0083] The technical solution of the embodiment of the present invention obtains the task scene state and task execution action of the currently executed task. The task scene state and the task execution action are input into the pre-trained target disturbance model to perform state disturbance prediction, wherein the target disturbance model is composed of a target world model and a random residual model set of the same model dimension, and the random residual model set includes at least one neural network model. According to the output of the target world model, the target task scene state of the currently executed task at the next moment is predicted. By performing parameter perturbations in the parameter space of the world model, the coverage of the world model simulation data is improved, so that the intelligent agent can see a large amount of data when performing reinforcement learning training in the world model, improve its effect on out-of-distribution data, and ultimately ensure the effect in the real physical environment.

[0084] Optionally, the task state disturbance module 302 includes:

[0085] A first scene state prediction unit, configured to obtain a predicted task scene state based on the task scene state, the task execution action and the target world model;

[0086] A second scene state prediction unit, configured to determine an incremental task scene state based on the task scene state, the task execution action and the random residual model set;

[0087] The task state disturbance unit is used to combine the predicted task scenario state and the incremental task scenario state to realize the state disturbance prediction of the target disturbance model.

[0088] Optionally, the second scene state prediction unit is specifically configured to:

[0089] Based on a preset screening method, a target neural network model is screened from the random residual model set;

[0090] The task scenario state and the task execution action are input into the target neural network model for incremental calculation to obtain an incremental task scenario state.

[0091] Optionally, the task state disturbance unit is specifically used to:

[0092] The predicted task scenario state and the incremental task scenario state are added together, and the addition result is determined as the target task scenario state.

[0093] Optionally, the scene state prediction module 303 is specifically used to:

[0094] s′=f(s,a;θ)+clip(g(s,a;φ i ),-c,c);

[0095] i~uniform({1,2,...,N});

[0096] Where f(s,a;θ) refers to the target world model, g(s,a;φ i ) refers to the random residual model set, N refers to the number of neural network models in the random residual model set, i refers to the i-th neural network model in the random residual model set, s refers to the task scene state, a refers to the task execution action, θ refers to the model parameters of the target world model, φ i refers to the model parameters of the random residual model set, c refers to the random perturbation range, and the clip() function is used to limit the output range of the neural network model.

[0097] Optionally, the device further comprises:

[0098] A world model training module is used to obtain sample offline data and sample task scenario states corresponding to the sample offline data, wherein the sample offline data at least includes historical task scenario states and historical task execution actions; input the sample offline data into a preset training model to predict the task scenario state, and obtain the output task scenario state based on the output of the preset training model; determine the training error based on the output task scenario state and the sample task scenario state, and back-propagate the training error to the preset training model to adjust the network parameters in the preset training model; when the preset convergence conditions are met, determine that the training of the preset training model is completed and obtain the target world model.

[0099] Optionally, the preset training model includes at least one of a regression model, a deep neural model, a generative adversarial network or a variational autoencoder.

[0100] The world model perturbation device provided in the embodiment of the present invention can execute the world model perturbation method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0101] Embodiment 4

[0102] Figure 5A schematic diagram of the structure of an electronic device 10 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0103] like Figure 5 As shown, the electronic device 10 includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, ROM 12 and RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0104] A number of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0105] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The processor 11 executes the various methods and processes described above, such as the method of world model perturbation.

[0106] In some embodiments, the method world model perturbation may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the method world model perturbation described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to perform the method world model perturbation in any other suitable manner (e.g., by means of firmware).

[0107] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0108] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0109] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0110] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0111] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0112] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0113] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0114] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for perturbing a world model, characterized in that: include: Get the task scene status and task execution action of the current task; Inputting the task scene state and the task execution action into a pre-trained target disturbance model to perform state disturbance prediction, wherein the target disturbance model is composed of a target world model and a random residual model set of the same model dimension, and the random residual model set includes at least one neural network model; According to the output of the target disturbance model, the target task scene state of the currently executed task at the next moment is predicted.

2. The method according to claim 1, characterized in that: The step of inputting the task scene state and the task execution action into a pre-trained target disturbance model to perform state disturbance prediction includes: Based on the task scene state, the task execution action and the target world model, obtaining a predicted task scene state; Determining an incremental task scenario state based on the task scenario state, the task execution action and the random residual model set; The predicted task scenario state and the incremental task scenario state are combined and processed to implement state disturbance prediction by the target disturbance model.

3. The method according to claim 2, characterized in that The step of determining the incremental task scene state based on the task scene state, the task execution action and the random residual model set includes: Based on a preset screening method, a target neural network model is screened from the random residual model set; The task scenario state and the task execution action are input into the target neural network model for incremental calculation to obtain an incremental task scenario state.

4. The method according to claim 2, characterized in that: The predicting, based on the output of the target world model, the target task scene state of the currently executed task at the next moment includes: The predicted task scenario state and the incremental task scenario state are added together, and the addition result is determined as the target task scenario state.

5. The method according to claim 1, characterized in that The predicting, based on the output of the target world model, the target task scene state of the currently executed task at the next moment includes: s′=f(s,a;θ)+clip(g(s,a;φ i ),-c,c); i~uniform({1,2,...,N}); Where f(s,a;θ) refers to the target world model, g(s,a;φ i ) refers to the random residual model set, N refers to the number of neural network models in the random residual model set, i refers to the i-th neural network model in the random residual model set, uniform({1,2,...,N}) represents uniform random sampling from 1 to N, s refers to the task scene state, a refers to the task execution action, θ refers to the model parameters of the target world model, φ i refers to the model parameters of the random residual model set, c refers to the random perturbation range, and the clip() function is used to limit the output range of the neural network model.

6. The method according to claim 1, characterized in that The training process of the target world model includes: Acquire sample offline data and sample task scenario states corresponding to the sample offline data, wherein the sample offline data at least includes historical task scenario states and historical task execution actions; Inputting the sample offline data into a preset training model to predict the task scene state, and obtaining an output task scene state based on the output of the preset training model; Determine a training error based on the output task scenario state and the sample task scenario state, and back-propagate the training error to the preset training model to adjust network parameters in the preset training model; When the preset convergence condition is met, it is determined that the training of the preset training model is completed and the target world model is obtained.

7. The method according to claim 6, characterized in that The preset training model includes at least one of a regression model, a deep neural model, a generative adversarial network or a variational autoencoder.

8. A world model perturbation device, characterized in that: include: The task information acquisition module is used to obtain the task scene status and task execution action of the current task; A task state disturbance module, used for inputting the task scene state and the task execution action into a pre-trained target disturbance model to perform state disturbance prediction, wherein the target disturbance model is composed of a target world model and a random residual model set of the same model dimension, and the random residual model set includes at least one neural network model; The scene state prediction module is used to predict the target task scene state of the currently executed task at the next moment according to the output of the target disturbance model.

9. An electronic device, characterized in that: The electronic device comprises: At least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the world model perturbation method described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the world model perturbation method according to any one of claims 1 to 7 when executed.

Images