World model enhancement method and device based on context prompt

Abstract

The invention provides a world model enhancement method and device based on context prompt, and belongs to the technical field of deep reinforcement learning. The method comprises: obtaining an image description statement according to current image input; obtaining a vector representation input by a current image through a convolutional neural network, obtaining a vector representation of an image description statement by using a language model, and splicing the image vector representation and the statement vector representation to obtain a new vector representation; the new vector representation is used for realizing modeling of a world model by using a recurrent neural network, and the world model performs action decision on the basis of historical actions and environment information contexts according to current vector input, so that the occurrence frequency of a trajectory rollback phenomenon during intelligent agent exploration is effectively reduced. When the environment changes, the intelligent agent can be adjusted in time according to the context information, and the adaptability and flexibility of the intelligent agent are improved. According to the method, the possibility of wrong decision making of the intelligent agent is effectively reduced, and the background task completion number of the intelligent agent within the limited step number is effectively increased.

Description

Technical Field

[0001] The present invention relates to the technical field of deep reinforcement learning, and in particular, to a method and device for enhancing a world model based on context cues. Background Art

[0002] In traditional world model modeling, the behavior of an agent is usually modeled by abstracting the dynamic changes of the environment. This modeling method consists of three parts: a visual module, a memory module, and an action module. It assumes that the environment is a static and independent entity, and the actions and decisions of the agent are based only on the current environmental state, without considering its historical trajectory or potential future state changes. Although this modeling method can effectively capture the regularity of the environment and the action strategy adoption pattern, it ignores the dynamic participation of the agent in the environment and the importance of its behavior trajectory. Specifically, the agent does not record its own action path and lacks long-term feedback on its previous actions. This leads to the problem that when facing a modeled scene, the agent may make repeated and ineffective decision-making errors. For example, the agent may take an action opposite to the previous step in an attempt to return to the previous state, but in fact, this is meaningless and may lead to an ineffective loop and inefficiency of behavior. This phenomenon is usually referred to as trajectory regression, which reflects the agent's lack of effective memory and understanding of its decision-making history. Therefore, how to improve the trajectory regression phenomenon of the agent is still worthy of exploration. Summary of the Invention

[0003] In view of this, embodiments of the present invention provide a method and device for enhancing a world model based on context cues to solve the technical problem of possible trajectory regression phenomena of world model agents during the exploration process.

[0004] The technical solution adopted by the present invention is as follows: In a first aspect, the present invention provides a method for enhancing a world model based on context cues, including: Obtaining the background task description statement of the simulation environment, the environmental observation image, and the historical action instruction information of the agent, and using a multimodal large language model to extract the image description statement context information from the environmental observation image; wherein, the historical action instruction information is concatenated at the end of the image description statement context information; Using a convolutional neural network to extract visual features from the environmental observation image to obtain an image feature vector, and performing text feature extraction on the background task description statement and the image description statement context information to obtain a background task feature vector and a context text feature vector respectively. At the same time, the background task feature vector and the context text feature vector are concatenated into a first text feature vector, and the first text feature vector is fused with the image feature vector to obtain a fusion feature vector; Input the historical action instruction information and the fused feature vector of the agent into a recurrent neural network for deterministic state update, and model the current simulation environment based on the deterministic state update result to obtain a world model; Based on the world model, the agent uses the actor-critic algorithm for imagination learning, and determines the current optimal action sequence of the agent according to the imagination learning result; execute actions in the simulation environment according to the current optimal action sequence, and at the same time obtain the action feedback, environment state update information, and updated agent visual image given by the simulation environment after the execution of the action, and judge whether the background task in the simulation environment is successfully completed after the execution of the action. If the background task is not successfully completed after the execution of the action, make an action decision based on the updated agent visual image, adjust the current optimal action sequence according to the action decision result, and execute the next action; if the background task is successfully completed after the execution of the action, the simulation environment returns a reward with a value of 1 at the same time.

[0005] Further, the using a convolutional neural network to extract visual features from the environmental observation image to obtain an image feature vector, and extracting text features from the background task description statement and the context information of the image description statement to obtain a background task feature vector and a context text feature vector respectively, includes: Use a convolutional neural network with a stride of 1 and a convolutional kernel size of 2 to extract visual features from the environmental observation image to obtain a 512-dimensional image feature vector; Input the background task description statement and the context information of the image description statement into the Sentence-Transformer tool for text feature extraction to obtain a 512-dimensional context text feature vector and a background task feature vector .

[0006] Further, the splicing the background task feature vector and the context text feature vector into a first text feature vector, and fusing the first text feature vector and the image feature vector to obtain a fused feature vector, includes: Splice the 512-dimensional context text feature vector and the background task feature vector to obtain a first text feature vector. The splicing process is shown in the following formula: where, represents the first text feature vector, represents the vector representation of the current background task description statement, represents the vector representation of the context information of the image description statement, represents the vector splicing operation; The first text feature vector is mapped through a multi-layer perceptron to obtain a 512-dimensional second text feature vector. The 512-dimensional second text feature vector and the 512-dimensional image feature vector are feature concatenated and fused to obtain a fused feature vector.

[0007] Further, the historical action instruction information of the agent and the fused feature vector are input into a recurrent neural network for deterministic state update, and the current simulation environment is modeled according to the deterministic state update result to obtain a world model, including: Receiving the historical action information of the agent and the fused feature vector as initial inputs, and passing them into a gated recurrent unit (GRU) of a recurrent neural network for deterministic state update. The update function of the GRU of the recurrent neural network is expressed as: where, is a mapping function for representing the updated state; represents the deterministic state, represents the latent state before update; represents the action information input this time; represents the stochastic state, and the stochastic state is represented as a Gaussian distribution; After the deterministic state update is completed, the deterministic state update result is input into the Prior network and the Posterior network respectively; among them, the Prior network predicts the state distribution based on the historical information and models the current simulation environment using a Gaussian distribution to obtain a world model; the Posterior network combines the fused feature vector and calculates the posterior distribution through the encoder output and state update; The updated deterministic state and the stochastic state are combined to obtain a complete state representation. The state representation can more comprehensively represent the state of the environment. Based on this state representation, parameters such as future states and observations can be predicted, and combining the stochastic state can improve the accuracy and robustness of the prediction.

[0008] Further, based on the world model, the agent performs imagination learning using the actor-critic algorithm and determines the current optimal action sequence of the agent according to the imagination learning result, including: Based on the actor-critic algorithm, a multi-layer perceptron is used to construct an actor network and a critic network. The actor network and the critic network are shown as follows: Actor network: Critic network: where, represents The probability of taking a certain action under the current conditions at a moment, represents the potential state representation of historical information, represents the observation that needs to be learned, represents the policy, represents the mapping function for estimating the reward value under given conditions; Set the reward function for the actor network to learn : where, is the current reward, is the discount factor, represents whether to continue learning, represents the observation that needs to be learned at the +1 moment, represents the potential state representation of historical information at the +1 moment, represents the learning reward at the +1 moment, is used to control the smoothness of the discount and , the critic uses categorical cross-entropy loss for discrete regression on two hot-spot coding targets; The actor network, based on the world model, selects actions according to the complete state representation for imagination learning. At the same time, the critic network evaluates the action rewards of the actions of the actor network's imagination learning through the reward function and feeds back the action reward evaluation results to the actor network, so that the actor network adjusts the actions of imagination learning according to the action reward evaluation results, maximizing the action rewards of the actor network's imagination learning; In each round of imagination learning process of the actor network, when the number of imagination learning times T = 15 or when the prediction of the world model is that learning is completed, the imagination learning ends, and the current best action sequence of the agent is determined according to the actions in the imagination learning process of the actor network.

[0009] Furthermore, the world model is provided with a feature learning loss function and a future prediction loss function; the feature learning loss function and the future prediction loss function are used to minimize the learning loss and future prediction loss for the training of the world model; where, the feature learning loss function is the sum of the image loss , the language loss , the reward loss , the end prediction loss and the regularization loss . The specific loss function of the feature learning loss function is shown in the following formula: Image loss function: Language loss function: Reward loss function: End prediction loss function: Regularization term loss function: where, is the categorical cross-entropy loss function; is the binary cross-entropy loss function; represents the update that stops the gradient calculation; , is a hyperparameter; represents regularization; represents function; represents the observation to be learned; represents the prediction for at time; represents the actual image observation at time; represents the image observation prediction at time; represents the description statement at time, represents the description statement prediction at time; represents the actual reward obtained at time, represents the reward prediction obtained at time; represents the encoding method that allows two different positions to be activated simultaneously; represents the predicted value of whether the event continues; Future prediction loss function Specifically, it is shown as the following formula: where, represents the observation to be learned; , is a hyperparameter; represents the prediction for at time.

[0010] Furthermore, the critic network and the actor network are respectively provided with a critic network loss function and an actor network loss function , specifically shown as the following formula: where, represents the categorical cross-entropy loss function; represents the stop gradient update; represents the policy; represents entropy; is the regularized sparsity of entropy; is the normalization factor, , which is used to calculate a dynamic range for normalizing return values greater than 1, so as to maintain the stability of the actor network loss during training; represents the estimated value of the current value function at time is the estimated cumulative value at time

[0011] In a second aspect, the present invention provides a world model enhancement device based on context prompts, including: A preprocessing module, configured to obtain the background task description statement of the simulation environment, the environmental observation image, and the historical action instruction information of the intelligent agent, and use a multimodal large language model to extract the image description statement context information from the environmental observation image; wherein, the historical action instruction information is concatenated at the end of the image description statement context information; A vision module, configured to use a convolutional neural network to perform visual feature extraction on the environmental observation image to obtain an image feature vector, and perform text feature extraction on the background task description statement and the image description statement context information to obtain a background task feature vector and a context text feature vector respectively, and at the same time concatenate the background task feature vector and the context text feature vector into a first text feature vector, and fuse the first text feature vector with the image feature vector to obtain a fused feature vector; A memory module, configured to input the historical action instruction information of the intelligent agent and the fused feature vector into a recurrent neural network for deterministic state update, and model the current simulation environment according to the deterministic state update result to obtain a world model; An action module, configured to enable the intelligent agent to perform imagination learning based on the world model using the actor-critic algorithm, and determine the current best action sequence of the intelligent agent according to the imagination learning result; execute actions in the simulation environment according to the current best action sequence, and at the same time obtain the action feedback, environmental state update information, and updated visual image of the intelligent agent given by the simulation environment after the execution of the action, and judge whether the background task in the simulation environment is successfully completed after the execution of the action. If the background task is not successfully completed after the execution of the action, then make an action decision based on the updated visual image of the intelligent agent, adjust the current best action sequence according to the action decision result, and execute the next action; if the background task is successfully completed after the execution of the action, the simulation environment simultaneously returns a reward with a value of 1.

[0012] In summary, the beneficial effects of the present invention are as follows: A method for enhancing a world model based on context cues provided by the present invention obtains an image description statement according to a current image input; obtains a vector representation of the current image input through a convolutional neural network, obtains a vector representation of the image description statement by using a language model, splices the graph vector representation and the statement vector representation to obtain a new vector representation; uses a recurrent neural network to implement modeling of the world model for the new vector representation, and the world model makes action decisions based on historical actions and environmental information context according to the current vector input, effectively reducing the frequency of occurrence of the trajectory backtracking phenomenon during the exploration of the intelligent agent. When the environment changes, the intelligent agent can adjust in a timely manner according to the context information, enhancing the adaptability and flexibility of the intelligent agent; when facing a new scenario, the context information provides additional knowledge for the world model, helping the intelligent agent quickly understand the current environmental characteristics and task requirements to make more reasonable decisions in the new environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following will briefly introduce the drawings required to be used in the embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained according to these drawings, and all of these are within the protection scope of the present invention.

[0014] Figure 1 It is a flowchart of a method for enhancing a world model based on context cues of the present invention; Figure 2 It is a schematic diagram of the feature vector fusion process of the present invention; Figure 3 It is a schematic diagram of the world model modeling process of the present invention; Figure 4 It is an aerial view of the HomeGrid simulation environment of the present invention; Figure 5 It is a schematic diagram of the work flow of the intelligent agent executing tasks of the present invention; Figure 6 It is a schematic diagram of the work of the intelligent agent touching an obstacle in the task execution process of the present invention; Figure 7 It is a schematic diagram of the working principle of a device for enhancing a world model based on context cues of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0015] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. It should be noted that in this document, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. If there is no conflict, the various features in the present invention and its embodiments can be combined with each other, and all are within the protection scope of the present invention.

[0016] The detailed implementation process of the present invention is shown in the following embodiments.

[0017] Embodiment 1: Refer to Figure 1 as shown Figure 1 is a flowchart of a method for enhancing a world model based on context cues of the present invention. Refer to Figure 1 as shown, the method for enhancing a world model based on context cues of the present invention includes: S1: Obtain the background task description statement of the simulation environment, the environmental observation image, and the historical action instruction information of the agent, and use a multimodal large language model to extract the image description statement context information from the environmental observation image; wherein, the historical action instruction information is concatenated at the end of the image description statement context information; S2: Use a convolutional neural network to extract visual features from the environmental observation image to obtain an image feature vector, and perform text feature extraction on the background task description statement and the image description statement context information to obtain a background task feature vector and a context text feature vector respectively. At the same time, concatenate the background task feature vector and the context text feature vector into a first text feature vector, and fuse the first text feature vector with the image feature vector to obtain a fused feature vector; S3: Input the historical action instruction information of the agent and the fused feature vector into a recurrent neural network for deterministic state update, and model the current simulation environment according to the deterministic state update result to obtain a world model; S4: Based on the world model, the agent uses the actor-critic algorithm for imagination learning, and determines the current optimal action sequence of the agent according to the imagination learning result; execute actions in the simulation environment according to the current optimal action sequence, and at the same time obtain the action feedback, environmental state update information, and updated agent visual image given by the simulation environment after the action is executed. Determine whether the background task in the simulation environment is successfully completed after the action is executed. If the background task is not successfully completed after the action is executed, make an action decision based on the updated agent visual image, adjust the current optimal action sequence according to the action decision result, and execute the next action; if the background task is successfully completed after the action is executed, the simulation environment returns a reward with a value of 1 at the same time.

[0018] Furthermore, the specific implementation process of step S1 in the embodiments of the present invention includes: S11: Receive an environmental observation image of 64*64*3 provided by the simulation environment , the background task description statement, and the historical action instruction information of the agent; S12: The task statement of the simulation environment mainly includes three parts: the background task description statement, the image description statement, and the action instruction information. The background task description statement is given by the simulation environment, and the image description statement obtains the context information of the fixed-format image description statement through the multi-modal large language model. Specifically, the environmental observation image is divided into 9 sub-images of 3*3 size, and each sub-image is respectively identified for items, and the overall image description statement is obtained by combining the item positions; the previous historical action instruction information is added at the end of the statement of the context information of the image description statement, indicating that the current image observation is the result of the interaction between this action and the simulation environment, taking the causal relationship into consideration.

[0019] Among them, an example of the image description statement extracted in the embodiments of the present invention is as follows: The item in location 1 is [item] and you are probably in [room]. …… The location 5 is your location and you are probably in [room]. …… The item in location 9 is [item] and you are probably in [room]. Specifically, as shown in Figure 4 , the simulation environment in the embodiments of the present invention is specifically the HomeGrid simulation environment. HomeGrid is a multi-task grid environment containing different objects, containers, and rooms. The agent will not only receive task instructions in the form of language but also receive language prompts. These prompts provide prior knowledge about the dynamics of the world, information about the state of the world, and also provide corrective feedback to the agent. In addition, the agent can also obtain the same information through autonomous interaction with the environment. The agent can observe the 3*3 grid space observation information centered on itself.

[0020] Among them, the action space of the agent in the HomeGrid environment is shown in Table 1 below: Table 1 Action space table of the agent in the HomeGrid environment In addition, in the HomeGrid environment, at each time step, an item has a 0.05 probability of moving to a new location, and at the same time, a new object is generated at a random location with a probability of 0.1 multiplied by the remaining number of item types.

[0021] In the HomeGrid environment, language information is provided to the agent in a step-by-step manner, that is, only one language token is provided at each time step. This step-by-step information transmission method ensures that the agent can continuously receive and process language instructions during the process of actions and environmental state changes.

[0022] Furthermore, referring to Figure 2 As shown, in step S2 of the embodiment of the present invention, a convolutional neural network is used to extract visual features from the environmental observation image to obtain an image feature vector, and text features are extracted from the background task description statement and the context information of the image description statement to obtain a background task feature vector and a context text feature vector respectively, including: A convolutional neural network (CNN) with a stride of 1 and a convolutional kernel size of 2 is used to extract visual features from the environmental observation image to obtain a 512-dimensional image feature vector (i.e., the visual feature vector). Specifically, the convolutional neural network (CNN) can perform operations such as convolution, activation, and pooling stacking to extract visual features.

[0023] The background task description statement and the context information of the image description statement are respectively input into the Sentence-Transformer tool for text feature extraction to obtain a 512-dimensional text feature vector, that is, the context text feature vector and the background task feature vector . When the Sentence-Transformer tool extracts the text feature vector, text feature extraction is performed through methods such as word embedding, position encoding, and sequence modeling. In addition, a long short-term memory network (LSTM) can be used to replace the Sentence-Transformer tool for text feature extraction.

[0024] Among them, when extracting visual features and text features, the environmental observation image with a size of 64*64*3 (i.e., Figure 2 the original image in) and the statement text information (such as instructions, conversations) can be preprocessed to improve the efficiency of subsequent feature extraction. Visual preprocessing of the image can adopt processing methods such as normalization and size adjustment, and preprocessing of the statement text information can be performed by methods such as word segmentation, embedding, and encoding.

[0025] Furthermore, referring to Figure 2As shown, in step S2 of the embodiment of the present invention, based on the multimodal fusion mechanism, the background task feature vector and the context text feature vector are concatenated to form a first text feature vector, and the first text feature vector is fused with the image feature vector to obtain a fused feature vector, including: The 512-dimensional context text feature vector and the background task feature vector are vector-concatenated to obtain a first text feature vector. The concatenation process is shown in the following formula: Wherein, represents the first text feature vector, represents the vector representation of the current background task description statement (i.e., the background task feature vector), represents the vector representation of the context information of the image description statement, represents the concatenation operation of vectors; The first text feature vector is mapped through a multi-layer perceptron to obtain a 512-dimensional second text feature vector. The 512-dimensional second text feature vector and the 512-dimensional image feature vector are feature-concatenated and fused to obtain a 1024-dimensional fused feature vector.

[0026] After obtaining the 1024-dimensional fused feature vector, the fused feature vector can be input into the modeled simulation environment to perform corresponding actions, and in the process of the action observation feedback loop, the specific execution order is: the action affects the environment, new observations, and continuously updates the state.

[0027] Furthermore, as shown in Figure 3 in step S3 of the embodiment of the present invention, the historical action instruction information of the agent and the fused feature vector are input into a recurrent neural network for deterministic state update, and the current simulation environment is modeled according to the deterministic state update result to obtain a world model, including: Receive the historical action information of the agent ( action ), and the fused feature vector ( embed ) as the initial input, and input it into the recurrent neural network GRU (i.e., the gated recurrent unit) for deterministic state update. The update function of the recurrent neural network GRU is expressed as: Wherein, is a mapping function used to represent the updated state; represents the deterministic state, represents the latent state before update; represents the action information input this time; represents the random state, and the random state is represented as a Gaussian distribution.

[0028] After the deterministic state update is completed, the deterministic state update results are input into the Prior network and Posterior network respectively; among them, the Prior network is based on historical information Predicted state distribution , and use Gaussian distribution to model the current simulation environment to obtain the world model; the Posterior network combines the fusion feature vector and calculates the posterior distribution through the encoder output and state update ; The core role of the Prior network and Posterior network is to capture the uncertainty in the system and to achieve the model's prediction of future states through probability distribution rather than deterministic prediction. deter ) and random state ( stoch ) are combined to form a complete state representation. The state representation can more comprehensively represent the state of the environment. Based on this state representation, future state and observation parameters can be predicted. Combining random states can improve the accuracy and robustness of predictions.

[0029] Furthermore, the intelligent agent of the embodiment of the present invention learns in imagination through the actor-critic algorithm to maximize the target expected reward. The learning of action strategy is not directly constrained by language, but only serves as a prompt. This setting can make full use of the multimodal features learned by the world model. The main purpose of the actor network is to learn the distribution space of actions, while the critic network guides the learning of the actor by estimating the expected discounted sum of rewards under a certain sequence.

[0030] Therefore, in step S4 of the embodiment of the present invention, the agent uses the actor-critic algorithm to perform imagination learning based on the world model, and determines the current optimal action sequence of the agent according to the imagination learning result, which specifically includes: Based on the actor-critic algorithm, a multilayer perceptron (MLP) is used to construct the actor network and the critic network. The actor network and the critic network are shown in the following formula: Actor Network: Critics Network: in, express The probability of taking a certain action under the current conditions at a certain moment, Represents the potential state representation of historical information, represents the observations to be learned, Indicates strategy, A mapping function that represents the estimated reward value under given conditions; Set the reward function for the actor network to learn : where is the current reward is the discount factor indicates whether to continue learning indicates the observation that needs to be learned at time t+1 indicates the latent state representation of the historical information at time t+1 indicates the learning reward at time t+1 is used to control the smoothness of the discount and , the critic uses categorical cross-entropy loss for discrete regression on two hot encoding targets; Based on the world model, the actor network selects actions according to the complete state representation for imagination learning. At the same time, the critic network evaluates the actions of the actor network's imagination learning through the reward function and feeds back the action reward evaluation results to the actor network, so that the actor network adjusts the actions of imagination learning according to the action reward evaluation results, maximizing the action rewards of the actor network's imagination learning; In each round of imagination learning process of the actor network, when the number of imagination learning times T=15 or when the prediction of the world model is that learning is completed (i.e., =1), the imagination learning ends, and the current best action sequence of the agent is determined according to the actions in the actor network's imagination learning process.

[0031] Among them, the actor-critic algorithm is a reinforcement learning algorithm based on value function and policy function. Among them, the actor network is responsible for learning the policy function and selecting actions according to the current state; the critic network is responsible for evaluating the quality of the policy and giving corresponding reward signals. The goal of the actor-critic algorithm is to continuously optimize the policy function so that the agent can obtain the maximum cumulative reward. The actor-critic algorithm can be used to train a robot agent so that it can learn the optimal action policy through interaction with the environment.

[0032] Furthermore, in the embodiments of the present invention, the world model is provided with a feature learning loss function and a future prediction loss function; the feature learning loss function and the future prediction loss function are used to minimize the learning loss and future prediction loss of the world model training. Among them, the feature learning loss function is the image loss , the language loss , the reward loss , the end prediction loss Sum of the regularization loss The specific loss function of the feature learning loss function is shown as follows: Image loss function: Language loss function: Reward loss function: End prediction loss function: Regularization term loss function: Wherein, Is the categorical cross-entropy loss function; Is the binary cross-entropy loss function; Indicates the update to stop gradient calculation; , is a hyperparameter; Indicates regularization; Indicates Function; Indicates the observation to be learned; Indicates for The prediction at time; Indicates The actual image observation at time; Indicates The image observation prediction at time; Indicates The description statement at time, Indicates The description statement prediction at time; Indicates The actual reward obtained at time, Indicates The reward prediction obtained at time; Indicates the encoding method that allows two different positions to be activated simultaneously; Indicates the predicted value of whether the event continues. In addition, Not only contains the information of the current observation, but also contains additional information for predicting the reward and whether the task continues.

[0033] In the embodiment of the present invention, the agent passes through The observation and potential recurrent state at time - 1 are combined to generate the prediction for Time. Therefore, the future prediction loss function of the world model Is specifically shown as follows: Wherein, Indicates the observation to be learned; , is a hyperparameter; Indicates for Prediction of time

[0034] Furthermore, in the embodiments of the present invention, the critic network and the actor network are respectively provided with a critic network loss function and an actor network loss function , as shown in the following formula: where represents the categorical cross-entropy loss function; represents stopping gradient update; represents the policy; represents the entropy; is the regularization sparsity of the entropy; is a normalization factor , used to calculate a dynamic range for normalizing return values greater than 1, so as to maintain the stability of the actor network loss during training; represents the estimated value of the current value function at time is the estimated cumulative value at time.

[0035] Based on historical actions and environmental information context, the embodiments of the present invention make action decisions, effectively reducing the frequency of trajectory backtracking during agent exploration. When the environment changes, the agent can adjust in a timely manner according to the context information, improving the adaptability and flexibility of the agent; when facing a new scenario, the context information provides additional knowledge for the world model, helping the agent quickly understand the current environmental characteristics and task requirements to make more reasonable decisions in the new environment; at the same time, the context prompt provides the task goal, which can help the agent adjust the strategy and strengthen the agent's action planning ability.

[0036] Embodiment 2: Referring to Figure 4 shown, based on the above Embodiment 1, the embodiments of the present invention give an aerial view of the HomeGrid simulation environment. Before the agent executes the task, it is necessary to set the parameters of the simulation environment. Refer to Tables 2, 3, and 4 shown.

[0037] Table 2 Floor types corresponding to room types in the simulation environment Table 3 List of item types Table 4 Setting table of statement prompt types and statement contents The parameters include the room type, its corresponding floor, the types of interactive items, the types of statement prompts, and the statement content, etc. Figure 4 It shows areas such as the living room, dining room, and kitchen in the HomeGrid simulation environment, as well as the furnishings of items in each area. To distinguish the area boundaries and items, Figure 4 different areas and items in it are distinguished by filling different degrees of gray scale. Based on the above Figure 4 HomeGrid simulation environment, referring to Figure 5 as shown, the embodiments of the present invention give a working schematic diagram of the agent executing the task of putting a tissue into the trash can. After setting the parameters of the simulation environment, the process of the agent in the embodiments of the present invention executing the task of putting a tissue into the trash can is specifically as follows: 1. The trash can is found when the number of steps is 0; 2. The tissue is found when the number of steps is 7; 3. The tissue is picked up when the number of steps is 8 (successfully pick up the tissue, the white part); 4. Find the way back to the trash can; 5. Arrive at the trash can when the number of steps is 15; 6. The tissue is successfully put into the trash can, and the reward is reward + 1 (the tissue is successfully placed, and there is no white part in the background).

[0038] In addition, referring to Figure 6 as shown, based on the parameters of the simulation environment in the above Figure 4 the embodiments of the present invention also give a schematic diagram of the agent touching an obstacle in the process of executing the task of picking up fruits. The specific execution process is as follows: 1. The fruit is found when the number of steps is 0, and the previous step action is set to none at this time; 2. The fruit is picked up when the number of steps is 2, and the previous step action is set to pick up at this time; 3. Look for the table, and the previous step action is up at this time; 4. Touch the obstacle, reward - 0.5, and the previous step action is right at this time.

[0039] From the process of the agent executing tasks described above, it can be seen that the agent in the embodiment of the present invention takes the joint representation of the image modality and the image description statement as input, explores the path to complete the background task in the simulation environment according to the current input, and returns the cumulative reward value in 100 steps of actions; after each action, the simulation environment returns the image modality input in the current agent's field of view, and the multimodal large model obtains the image description statement according to the current image input; then, the vector representation of the current image input is obtained through the convolutional neural network, the vector representation of the image description statement is obtained by using the language model, the graph vector representation and the statement vector representation are spliced to obtain a new vector representation; the new vector representation is used to implement the modeling of the world model by the recurrent neural network, and the world model makes action decisions according to the current vector input; the agent executes actions to interact with the simulation environment, and at the same time realizes the change of the image modality in the agent's field of view, and also judges whether the background task is completed and whether to return the task reward. The present invention effectively reduces the possibility of incorrect decisions by the agent and effectively improves the number of background tasks completed by the agent within the limited number of steps.

[0040] Embodiment 3: Based on the method in Embodiment 1, the embodiment of the present invention further provides a world model enhancement device based on context prompts, including: A preprocessing module, configured to obtain the background task description statement of the simulation environment, the environmental observation image, and the historical action instruction information of the agent, and extract the image description statement context information from the environmental observation image by using a multimodal large language model; wherein, the historical action instruction information is spliced at the end of the statement of the image description statement context information; A vision module, configured to use a convolutional neural network to perform visual feature extraction on the environmental observation image to obtain an image feature vector, perform text feature extraction on the background task description statement and the image description statement context information respectively to obtain a background task feature vector and a context text feature vector, and at the same time splice the background task feature vector and the context text feature vector into a first text feature vector, and fuse the first text feature vector with the image feature vector to obtain a fused feature vector; A memory module, configured to input the historical action instruction information of the agent and the fused feature vector into a recurrent neural network for deterministic state update, and model the current simulation environment according to the deterministic state update result to obtain a world model; An action module, which is used for the agent to perform imagination learning based on the world model using the actor-critic algorithm, and determine the current optimal action sequence of the agent according to the imagination learning result; execute actions in the simulation environment according to the current optimal action sequence, and at the same time obtain the action feedback, environment state update information and updated agent visual image given by the simulation environment after the actions are executed, and judge whether the background task in the simulation environment is successfully completed after the actions are executed. If the background task is not successfully completed after the actions are executed, make action decisions based on the updated agent visual image, adjust the current optimal action sequence according to the action decision result, and execute the next action; if the background task is successfully completed after the actions are executed, the simulation environment returns a reward with a value of 1 at the same time.

[0041] Specifically, referring to Figure 7 the schematic diagram of the working principle of the device shown, the specific working logic (i.e., working principle) of the world model enhancement device based on context cues in the embodiments of the present invention is as follows: In the preprocessing module and the visual module, the simulation environment returns the current visible 3*3 image of the agent, and uses the multimodal large model to align the image modality to the language modality to obtain the image description statement, and jointly uses the image modality and the image description statement as the input of the world model; In the memory module, according to the joint input of the image modality and the image description statement, the recurrent neural network models the current simulation environment by processing the input, and the modeled model is the world model; In the action module, the agent performs imagination learning using the actor-critic algorithm based on the world model to find a current optimal action sequence; The agent executes the decision-making actions in the simulation environment, and the simulation environment gives action feedback and environment state updates, and returns the updated agent visual image; if the background task is successfully completed after the actions are executed, the simulation environment returns a reward with a value of 1 at the same time.

[0042] Figure 7 In, V represents the visual module, and the visual module provides input for the memory module; M represents the memory module, and the memory module models the world model according to the input; C represents the action module, and the action module completes the action decision-making based on the world model modeling.

[0043] Among them, the visual module is responsible for compressing the seen image into a representation ; the memory module updates the current internal state through the learning of , and integrates historical information and predicts future states; the action module generates the current action decision according to the current image representation and the internal memory state , finally perform action interaction to change the environmental image and iterate the process.

[0044] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements on some or all of the technical features; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for enhancing a world model based on context cues, characterized in that, Including: Obtain the background task description statement of the simulation environment, the environmental observation image, and the historical action instruction information of the agent, and use a multimodal large language model to extract image description statement context information from the environmental observation image; wherein, the historical action instruction information is concatenated at the end of the image description statement context information. Use a convolutional neural network to perform visual feature extraction on the environmental observation image to obtain an image feature vector, and perform text feature extraction on the background task description statement and the image description statement context information to obtain a background task feature vector and a context text feature vector respectively. At the same time, concatenate the background task feature vector and the context text feature vector into a first text feature vector, and fuse the first text feature vector with the image feature vector to obtain a fused feature vector. Input the historical action instruction information of the agent and the fused feature vector into a recurrent neural network for deterministic state update, and model the current simulation environment according to the deterministic state update result to obtain a world model. Based on the world model, the agent uses the actor-critic algorithm for imagination learning, and determines the current optimal action sequence of the agent according to the imagination learning result; execute actions in the simulation environment according to the current optimal action sequence, and at the same time obtain the action feedback, environmental state update information, and updated agent visual image given by the simulation environment after the execution of the action, and judge whether the background task in the simulation environment is successfully completed after the execution of the action. If the background task is not successfully completed after the execution of the action, make an action decision based on the updated agent visual image, adjust the current optimal action sequence according to the action decision result, and execute the next action; if the background task is successfully completed after the execution of the action, the simulation environment returns a reward with a value of 1 at the same time.

2. The method for enhancing a world model based on context cues according to claim 1, wherein The using a convolutional neural network to perform visual feature extraction on the environmental observation image to obtain an image feature vector, and performing text feature extraction on the background task description statement and the image description statement context information to obtain a background task feature vector and a context text feature vector respectively, includes: Use a convolutional neural network with a stride of 1 and a convolution kernel size of 2 to perform visual feature extraction on the environmental observation image to obtain a 512-dimensional image feature vector. Input the context information of the background task description statement and the image description statement into the Sentence-Transformer tool respectively for text feature extraction to obtain a 512-dimensional context text feature vector and the background task feature vector .

3. The method for enhancing a world model based on context cues according to claim 1, wherein The concatenating the background task feature vector and the context text feature vector into a first text feature vector, and fusing the first text feature vector with the image feature vector to obtain a fused feature vector, includes: A 512-dimensional context text feature vector and a background task feature vector are vector concatenated to obtain a first text feature vector, and the concatenation process is shown as follows: Among them, represents the first text feature vector, represents the vector representation of the current background task description statement, represents the vector representation of the context information of the image description statement, represents the concatenation operation of vectors; Perform mapping processing on the first text feature vector through a multi-layer perceptron to obtain a 512-dimensional second text feature vector, and perform feature concatenation and fusion on the 512-dimensional second text feature vector and the 512-dimensional image feature vector to obtain a fused feature vector.

4. The method for enhancing a world model based on context cues according to claim 1, wherein The inputting the historical action instruction information of the agent and the fused feature vector into a recurrent neural network for deterministic state update, and modeling the current simulation environment according to the deterministic state update result to obtain a world model, includes: Receive the historical action information and the fused feature vector of the receiving agent as the initial input, and pass them into the gated recurrent unit (GRU) of the recurrent neural network for deterministic state update. The update function of the GRU of the recurrent neural network is expressed as: Among them, is a mapping function used to represent the update status; represents the deterministic state, represents the potential state before update; represents the action information of this input; represents the random state, and the random state is represented as a Gaussian distribution; After the deterministic state update is completed, the deterministic state update results are respectively input into the Prior network and the Posterior network; among them, the Prior network is based on historical information predicts the state distribution , and models the current simulation environment using a Gaussian distribution to obtain a world model; the Posterior network combines the fused feature vectors and calculates the posterior distribution through the encoder output and state update ; Combine the updated deterministic state with the stochastic state to obtain the complete state representation.

5. The method for enhancing a world model based on context cues according to claim 1, wherein Based on the world model, the agent uses the actor-critic algorithm for imagination learning and determines the current optimal action sequence of the agent according to the imagination learning result, including: Based on the actor-critic algorithm, use a multi-layer perceptron to construct an actor network and a critic network. The actor network and the critic network are shown as follows: Actor Network: Critic Network: Among them, represents the probability of taking a certain action at a moment under the current conditions, represents the potential state representation of historical information, represents the observation to be learned, represents the policy, represents the mapping function for estimating the reward value under given conditions; Set the reward function for the actor network to learn : Among them, is the current reward, is the discount factor, indicates whether to continue learning, represents the observation to be learned at the +1 moment, represents the potential state representation of the historical information at the +1 moment, represents the learning reward at the +1 moment, is used to control the smoothness of the discount and , and the critic uses categorical cross-entropy loss for discrete regression on two hot encoding targets; The actor network, based on the world model, selects actions according to the complete state representation for imagination learning. At the same time, the critic network evaluates the action rewards of the actions of the actor network's imagination learning through a reward function and feeds back the action reward evaluation results to the actor network, so that the actor network adjusts the actions of the imagination learning according to the action reward evaluation results, maximizing the action rewards of the actor network's imagination learning; In each round of imagination learning process of the actor network, when the number of imagination learning times T = 15 or when the prediction of the world model is that the learning is completed, the imagination learning ends, and the current optimal action sequence of the agent is determined according to the actions in the imagination learning process of the actor network.

6. The method for enhancing a world model based on context cues according to claim 1, wherein The world model is set with a feature learning loss function and a future prediction loss function; the feature learning loss function and the future prediction loss function are used to minimize the learning loss and the future prediction loss of the world model training; Among them, The feature learning loss function is an image loss , a language loss , a reward loss , an end prediction loss and a regularization loss The specific loss function of the feature learning loss function is shown in the following formula: Image loss function: Language loss function: Reward loss function: End prediction loss function: Regular term loss function: Among them, is the categorical cross-entropy loss function; is the binary cross-entropy loss function; represents an update that stops gradient calculation; , which is a hyperparameter; represents regularization; represents function; represents the observation to be learned; represents the prediction at time represents the actual image observation at time represents the prediction of the image observation at time represents the description statement at time represents the prediction of the description statement at time represents the actual reward obtained at time represents the prediction of the reward obtained at time represents an encoding method that allows two different positions to be activated simultaneously; represents the predicted value of whether the event continues; Future prediction loss function As shown in the following formula: Among them, represents the observation to be learned; , is a hyperparameter; represents the prediction at moment.

7. The method for enhancing a world model based on context cues according to claim 5, wherein The critic network and the actor network are respectively provided with a critic network loss function and an actor network loss function , which are specifically shown as follows: Among them, represents the categorical cross-entropy loss function; represents stopping the gradient update; represents the policy; represents the entropy; is the regularization sparsity of the entropy; is the normalization factor, , which is used to calculate a dynamic range for normalizing return values greater than 1, so as to maintain the stability of the actor network loss during training; represents the estimated value of the current value function at time is the estimated cumulative value at time 8. An apparatus for enhancing a world model based on context cues, characterized in that, including: A preprocessing module for obtaining the background task description statement of the simulation environment, the environmental observation image, and the historical action instruction information of the agent, and using a multi-modal large language model to extract the image description statement context information from the environmental observation image; wherein, the historical action instruction information is concatenated at the end of the image description statement context information; A vision module for using a convolutional neural network to extract visual features from the environmental observation image to obtain an image feature vector, and extracting text features from the background task description statement and the image description statement context information to obtain a background task feature vector and a context text feature vector respectively. At the same time, the background task feature vector and the context text feature vector are concatenated into a first text feature vector, and the first text feature vector is fused with the image feature vector to obtain a fused feature vector; A memory module for inputting the historical action instruction information and the fused feature vector of the agent into a recurrent neural network for deterministic state update, and modeling the current simulation environment according to the deterministic state update result to obtain a world model; An action module, which is used for the agent to perform imagination learning based on the world model using the actor-critic algorithm, and determine the current optimal action sequence of the agent according to the results of imagination learning; execute actions in the simulation environment according to the current optimal action sequence, and at the same time obtain the action feedback, environment state update information and updated agent visual image given by the simulation environment after the execution of the actions, and judge whether the background task in the simulation environment is successfully completed after the execution of the actions. If the background task is not successfully completed after the execution of the actions, then make action decisions based on the updated agent visual image, adjust the current optimal action sequence according to the action decision results, and execute the next action; if the background task is successfully completed after the execution of the actions, the simulation environment returns a reward with a value of 1 at the same time.

Images