A world model enhancement method and device based on contextual prompts

Abstract

The present invention provides a world model enhancement method and device based on contextual prompts, which belongs to the field of deep reinforcement learning technology. The method obtains an image description sentence based on the current image input; obtains a vector representation of the current image input through a convolutional neural network, obtains a vector representation of the image description sentence using a language model, splices the graph vector representation and the sentence vector representation to obtain a new vector representation; uses a recurrent neural network to model the world model with the new vector representation, and the world model makes action decisions based on the context of historical actions and environmental information according to the current vector input, effectively reducing the frequency of trajectory regression during the exploration of the intelligent agent. When the environment changes, the intelligent agent can make timely adjustments based on the contextual information, thereby improving the adaptability and flexibility of the intelligent agent. The present invention effectively reduces the possibility of the intelligent agent making wrong decisions and effectively increases the number of background tasks completed by the intelligent agent within a limited number of steps.

Description

Technical Field

[0001] The present invention relates to the field of deep reinforcement learning technology, and in particular to a world model enhancement method and device based on contextual prompts. Background Art

[0002] In traditional world modeling, the behavior of an agent is typically modeled by abstracting the dynamics of the environment. This modeling approach, consisting of a vision module, a memory module, and an action module, assumes the environment is a static, independent entity. The agent's actions and decisions are based solely on the current state of the environment, without considering its historical trajectory or potential future state changes. While this modeling approach effectively captures the regularity of the environment and the patterns of action strategies adopted, it overlooks the agent's dynamic involvement in the environment and the importance of its behavioral trajectory. Specifically, the agent does not record its own action paths and lacks long-term feedback on its past actions. This leads to the possibility of repeated and ineffective decision-making errors when faced with a previously modeled scenario. For example, the agent may take an action opposite to the previous one, attempting to return to a previous state. This is actually meaningless and may lead to ineffective cycles and inefficient behavior. This phenomenon, commonly known as trajectory regression, reflects the agent's lack of effective memory and understanding of its decision history. Therefore, how to improve the trajectory regression phenomenon of agents remains a research topic. Summary of the Invention

[0003] In view of this, an embodiment of the present invention provides a world model enhancement method and device based on contextual prompts to solve the technical problem of trajectory regression that may occur during the exploration process of a world model agent.

[0004] The technical solution adopted in the present invention is:

[0005] In a first aspect, the present invention provides a world model enhancement method based on contextual prompts, comprising:

[0006] Obtaining a background task description statement of the simulation environment, an environment observation image, and historical action instruction information of the intelligent agent, and extracting image description statement context information from the environment observation image using a multimodal large language model; wherein the historical action instruction information is spliced ​​at the end of the sentence of the image description statement context information;

[0007] Using a convolutional neural network to extract visual features from the environmental observation image to obtain an image feature vector, and performing text feature extraction on the context information of the background task description sentence and the image description sentence to obtain a background task feature vector and a context text feature vector, respectively, and concatenating the background task feature vector and the context text feature vector into a first text feature vector, and fusing the first text feature vector with the image feature vector to obtain a fused feature vector;

[0008] The agent's historical action command information and fused feature vector are input into the recurrent neural network for deterministic state update, and the current simulation environment is modeled based on the deterministic state update result to obtain the world model;

[0009] Based on the world model, the intelligent agent uses the actor-critic algorithm to perform imaginative learning and determines the current optimal action sequence of the intelligent agent based on the results of imaginative learning; it performs actions in the simulation environment based on the current optimal action sequence, and obtains action feedback, environment state update information and updated intelligent agent visual image given by the simulation environment after the action is executed, and judges whether the background task in the simulation environment is successfully completed after the action is executed. If the background task is not successfully completed after the action is executed, the intelligent agent makes an action decision based on the updated intelligent agent visual image, adjusts the current optimal action sequence based on the action decision result, and executes the next action; if the background task is successfully completed after the action is executed, the simulation environment also returns a reward of 1.

[0010] Furthermore, the use of a convolutional neural network to extract visual features from the environmental observation image to obtain an image feature vector, and performing text feature extraction on the background task description statement and the image description statement context information to obtain a background task feature vector and a context text feature vector, respectively, includes:

[0011] A convolutional neural network with a step size of 1 and a convolution kernel size of 2 is used to extract visual features from the environmental observation image to obtain a 512-dimensional image feature vector;

[0012] The context information of the background task description sentence and the image description sentence are respectively input into the Sentence-Transformer tool for text feature extraction to obtain a 512-dimensional context text feature vector and background task feature vector .

[0013] Furthermore, the step of concatenating the background task feature vector and the context text feature vector into a first text feature vector and fusing the first text feature vector with the image feature vector to obtain a fused feature vector includes:

[0014] The 512-dimensional context text feature vector and background task feature vector Perform vector splicing to obtain the first text feature vector. The splicing process is shown in the following formula:

[0015]

[0016] in, represents the first text feature vector, Represents the vector representation of the current background task description statement, A vector representation representing the contextual information of the image description sentence, Represents the concatenation operation of vectors;

[0017] The first text feature vector is mapped by a multi-layer perceptron to obtain a 512-dimensional second text feature vector, and the 512-dimensional second text feature vector is concatenated and fused with the 512-dimensional image feature vector to obtain a fused feature vector.

[0018] Furthermore, the agent's historical action instruction information and fused feature vector are input into a recurrent neural network for deterministic state update, and the current simulation environment is modeled according to the deterministic state update result to obtain a world model, including:

[0019] The historical action information and fusion feature vector of the intelligent agent are received as initial input and passed into the recurrent neural network GRU for deterministic state update. The update function of the recurrent neural network GRU is expressed as:

[0020]

[0021] in, Is a mapping function used to represent the update state; Represents a deterministic state, Represents the potential state before the update; Represents the action information of this input; Represents a random state, which is represented by a Gaussian distribution;

[0022] After the deterministic state update is completed, the deterministic state update results are input into the Prior network and the Posterior network respectively; among them, the Prior network is based on historical information Predicted state distribution , and use Gaussian distribution to model the current simulation environment to obtain the world model; the Posterior network combines the fusion feature vector and calculates the posterior distribution through the encoder output and state update ;

[0023] The updated deterministic state is combined with the stochastic state to obtain a complete state representation. This state representation can more comprehensively represent the state of the environment. Based on this state representation, we can predict parameters such as future states and observations. Incorporating the stochastic state can improve the accuracy and robustness of the prediction.

[0024] Furthermore, the agent performs imaginative learning based on the world model using an actor-critic algorithm, and determines the agent's current optimal action sequence based on the imaginative learning results, including:

[0025] Based on the actor-critic algorithm, a multi-layer perceptron is used to construct the actor network and the critic network. The actor network and the critic network are shown as follows:

[0026] Actor Network:

[0027] Critics Network:

[0028] in, express The probability of taking a certain action under the current conditions at any moment, Represents the potential state representation of historical information, represents the observations to be learned, Indicates strategy, A mapping function that represents the estimated reward value under given conditions;

[0029] Setting the reward function for actor network learning :

[0030]

[0031] in, For the current reward, is the discount factor, Indicate whether to continue learning. express +1 moment to learn observations, express The potential state representation of historical information at time +1, express +1 moment of learning reward, is used to control the smoothing of the discount and ,The critic performs discrete regression on two-hot encoded targets using categorical cross-entropy loss;

[0032] Based on the world model, the actor network selects actions for imaginative learning according to the complete state representation. At the same time, the critic network evaluates the action rewards of the actor network's imaginative learning through the reward function and feeds back the action reward evaluation results to the actor network so that the actor network can adjust the imaginative learning actions according to the action reward evaluation results to maximize the action rewards of the actor network's imaginative learning.

[0033] In each round of imaginary learning of the actor network, when the number of imaginary learning times T=15 or when the world model predicts that learning is complete, the imaginary learning ends, and the current optimal action sequence of the agent is determined based on the actions of the actor network during the imaginary learning process.

[0034] Furthermore, the world model is provided with a feature learning loss function and a future prediction loss function; the feature learning loss function and the future prediction loss function are used to minimize the learning loss and the future prediction loss of the world model training; wherein,

[0035] The feature learning loss function is the image loss , language loss , reward loss , end prediction loss and regularization loss The specific loss function of the feature learning loss function is as follows:

[0036] Image loss function:

[0037] Language loss function:

[0038] Reward loss function:

[0039] End prediction loss function:

[0040] Regularization loss function:

[0041] in, is the classification cross entropy loss function; is the binary cross entropy loss function; Indicates stopping the update of gradient calculation; , is a hyperparameter; represents regularization; express function; Indicates observations that need to be learned; Express prediction of the moment; express Actual image observation at the moment; express Image observation prediction at a given moment; express Descriptive statement of the moment, express Prediction of descriptive statements at the moment; express Get rewards every moment. express Get reward predictions at all times; Indicates a coding method that allows two different locations to be activated simultaneously; Indicates the predicted value of whether the event will continue;

[0042] Future prediction loss function The specific formula is as follows:

[0043]

[0044] in, Indicates observations that need to be learned; , is a hyperparameter; Express Predictions of the moment.

[0045] Furthermore, the critic network and the actor network are respectively provided with a critic network loss function and the actor network loss function , as shown in the following formula:

[0046]

[0047]

[0048] in, represents the classification cross entropy loss function; Indicates stopping gradient update; express strategy; represents entropy; is the regularized sparsity of entropy; is the normalization factor, , used to calculate a dynamic range for normalizing reward values ​​greater than 1, thereby maintaining the stability of the actor network loss during training; express The estimated value of the current value function at the moment, for The estimated cumulative value of the moment.

[0049] In a second aspect, the present invention provides a world model enhancement device based on contextual prompts, comprising:

[0050] A preprocessing module is configured to obtain background task description statements, environment observation images, and historical action instruction information of the agent in the simulation environment, and to extract image description statement context information from the environment observation images using a multimodal large language model; wherein the historical action instruction information is spliced ​​at the end of the image description statement context information;

[0051] a visual module, configured to extract visual features from the environmental observation image using a convolutional neural network to obtain an image feature vector, and to extract text features from the context information of the background task description statement and the image description statement to obtain a background task feature vector and a context text feature vector, respectively; concatenate the background task feature vector and the context text feature vector into a first text feature vector, and fuse the first text feature vector with the image feature vector to obtain a fused feature vector;

[0052] The memory module is used to input the agent's historical action command information and fused feature vectors into the recurrent neural network for deterministic state update, and to model the current simulation environment based on the deterministic state update results to obtain a world model;

[0053] The action module is used for the intelligent agent to perform imaginative learning based on the world model using the actor-critic algorithm, and determine the current optimal action sequence of the intelligent agent based on the results of imaginative learning; perform actions in the simulation environment based on the current optimal action sequence, and at the same time obtain action feedback, environment state update information and updated intelligent agent visual image given by the simulation environment after the action is executed, and judge whether the background task in the simulation environment is successfully completed after the action is executed. If the background task is not successfully completed after the action is executed, the action decision is made based on the updated intelligent agent visual image, the current optimal action sequence is adjusted according to the action decision result, and the next action is executed; if the background task is successfully completed after the action is executed, the simulation environment also returns a reward of 1.

[0054] In summary, the beneficial effects of the present invention are as follows:

[0055] The present invention provides a world model enhancement method based on contextual cues. The method obtains an image description sentence based on the current image input; obtains a vector representation of the current image input through a convolutional neural network, obtains a vector representation of the image description sentence using a language model, and concatenates the image vector representation with the sentence vector representation to obtain a new vector representation; the new vector representation is used to model the world model using a recurrent neural network. The world model makes action decisions based on the context of historical actions and environmental information according to the current vector input, effectively reducing the frequency of trajectory regression during the agent's exploration. When the environment changes, the agent can make timely adjustments based on the contextual information, enhancing the agent's adaptability and flexibility. When faced with new scenarios, the contextual information provides additional knowledge to the world model, helping the agent quickly understand the current environmental characteristics and task requirements, so as to make more reasonable decisions in the new environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following is a brief introduction to the drawings required for use in the embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work, and these are all within the scope of protection of the present invention.

[0057] Figure 1 This is a flow chart of a world model enhancement method based on contextual prompts according to the present invention;

[0058] Figure 2 Schematic diagram of the feature vector fusion process of the present invention;

[0059] Figure 3 A schematic diagram of the world modeling process of the present invention;

[0060] Figure 4 An aerial view of the HomeGrid simulation environment of the present invention;

[0061] Figure 5 A schematic diagram of the workflow of the intelligent agent of the present invention performing tasks;

[0062] Figure 6 A schematic diagram of the working process of the intelligent agent of the present invention touching an obstacle during the task execution process;

[0063] Figure 7 The figure is a schematic diagram of the working principle of a world model enhancement device based on contextual prompts of the present invention. DETAILED DESCRIPTION

[0064] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. It should be noted that, in this article, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. If there is no conflict, the various features of the present invention and the embodiments can be combined with each other and are all within the scope of protection of the present invention.

[0065] The detailed implementation process of the present invention is shown in the following examples.

[0066] Example 1: Reference Figure 1 As shown, Figure 1 This is a flow chart of a world model enhancement method based on contextual prompts of the present invention. Figure 1 As shown, the world model enhancement method based on contextual prompts of the present invention includes:

[0067] S1: Obtaining a background task description statement of the simulation environment, an environment observation image, and historical action instruction information of the intelligent agent, and extracting image description statement context information from the environment observation image using a multimodal large language model; wherein the historical action instruction information is spliced ​​at the end of the sentence of the image description statement context information;

[0068] S2: Using a convolutional neural network to extract visual features from the environmental observation image to obtain an image feature vector, and performing text feature extraction on the context information of the background task description sentence and the image description sentence to obtain a background task feature vector and a context text feature vector, respectively. Simultaneously, the background task feature vector and the context text feature vector are concatenated into a first text feature vector, and the first text feature vector is fused with the image feature vector to obtain a fused feature vector.

[0069] S3: Input the agent's historical action command information and fused feature vector into the recurrent neural network for deterministic state update, and model the current simulation environment based on the deterministic state update result to obtain the world model;

[0070] S4: Based on the world model, the intelligent agent uses the actor-critic algorithm to perform imaginative learning, and determines the current optimal action sequence of the intelligent agent based on the imaginative learning results; performs actions in the simulation environment based on the current optimal action sequence, and obtains action feedback, environment state update information and updated intelligent agent visual image given by the simulation environment after the action is executed, and determines whether the background task in the simulation environment is successfully completed after the action is executed. If the background task is not successfully completed after the action is executed, the intelligent agent makes an action decision based on the updated intelligent agent visual image, adjusts the current optimal action sequence based on the action decision result, and executes the next action; if the background task is successfully completed after the action is executed, the simulation environment simultaneously returns a reward of 1.

[0071] Furthermore, the specific implementation process of step S1 of the embodiment of the present invention includes:

[0072] S11: Receive the 64*64*3 environmental observation image provided by the simulation environment , background task description statements and historical action instruction information of the agent;

[0073] S12: The task statement of the simulation environment mainly consists of three parts: background task description statement, image description statement, and action instruction information. The background task description statement is given by the simulation environment, while the image description statement obtains fixed-format image description statement context information through a multimodal large language model. Specifically, the environmental observation image is divided into 9 sub-images of 3*3 size, and objects are identified in each sub-image separately. The overall image description statement is obtained by combining the object position. The historical action instruction information of the previous step is added to the end of the sentence of the image description context information, indicating that the current image observation is the result of the interaction between the action and the simulation environment, taking causal relationships into consideration.

[0074] The following are examples of image description sentences extracted by the embodiment of the present invention:

[0075] The item in location 1 is [item] and you are probably in [room].

[0076] …

[0077] The location 5 is your location and you are probably in [room].

[0078] …

[0079] The item in location 9 is [item] and you are probably in [room].

[0080] Specifically, refer to Figure 4 As shown, the simulation environment of this embodiment of the present invention is specifically the HomeGrid simulation environment. HomeGrid is a multi-task grid environment containing various objects, containers, and rooms. The agent receives not only verbal task instructions but also verbal prompts. These prompts provide prior knowledge about world dynamics and information about the world state, as well as corrective feedback for the agent. Furthermore, the agent can obtain the same information through autonomous interaction with the environment, observing a 3x3 grid space centered on itself.

[0081] Among them, the action space of the intelligent agent in the HomeGrid environment is shown in Table 1 below:

[0082] Table 1 Action space of intelligent agents in HomeGrid environment

[0083]

[0084] In addition, in the HomeGrid environment, at each time step, an item has a probability of 0.05 to move to a new location, and a new object is generated at a random location with a probability of 0.1 multiplied by the remaining item types.

[0085] In the HomeGrid environment, language information is provided to the agent in a step-by-step manner, with only one language token provided at each time step. This step-by-step information delivery ensures that the agent can continuously receive and process language instructions as its actions and environment states change.

[0086] Further, refer to Figure 2 As shown, in step S2 of the embodiment of the present invention, a convolutional neural network is used to extract visual features of the environment observation image to obtain an image feature vector, and text feature extraction is performed on the background task description sentence and the image description sentence context information to obtain a background task feature vector and a context text feature vector, respectively, including:

[0087] A convolutional neural network (CNN) with a stride of 1 and a convolution kernel size of 2 is used to extract visual features from the environmental observation image, obtaining a 512-dimensional image feature vector (i.e., a visual feature vector). Specifically, the convolutional neural network (CNN) may employ operations such as convolution, activation, and pooling stacking to extract visual features.

[0088] The context information of the background task description sentence and the image description sentence are respectively input into the Sentence-Transformer tool for text feature extraction to obtain a 512-dimensional text feature vector, namely the context text feature vector and background task feature vector The Sentence-Transformer tool extracts text feature vectors through word embedding, positional encoding, and sequence modeling. Alternatively, the Long Short-Term Memory (LSTM) network can be used to replace the Sentence-Transformer tool for text feature extraction.

[0089] Among them, when extracting visual features and text features, the environment observation image with a size of 64*64*3 (i.e. Figure 2 Preprocessing of raw images (e.g., instructions and dialogue) and sentence text (e.g., instructions and dialogue) improves the efficiency of subsequent feature extraction. Image visual preprocessing can include normalization and resizing, while sentence text preprocessing can include word segmentation, embedding, and encoding.

[0090] Further, refer to Figure 2 As shown, in step S2 of the embodiment of the present invention, based on the multimodal fusion mechanism, the background task feature vector and the context text feature vector are concatenated into a first text feature vector, and the first text feature vector is fused with the image feature vector to obtain a fused feature vector, including:

[0091] The 512-dimensional context text feature vector and background task feature vector Perform vector splicing to obtain the first text feature vector. The splicing process is shown in the following formula:

[0092]

[0093] in, represents the first text feature vector, The vector representation representing the current background task description statement (i.e., the background task feature vector), A vector representation representing the contextual information of the image description sentence, Represents the concatenation operation of vectors;

[0094] The first text feature vector is mapped by a multi-layer perceptron to obtain a 512-dimensional second text feature vector, and the 512-dimensional second text feature vector is concatenated and fused with the 512-dimensional image feature vector to obtain a 1024-dimensional fused feature vector.

[0095] After obtaining the 1024-dimensional fused feature vector, the fused feature vector can be input into the modeled simulation environment to perform the corresponding action, as well as in the action-observation feedback loop. The specific execution order is: action affects the environment, new observation, and continuous update of the status.

[0096] Further, refer to Figure 3 As shown, in step S3 of the embodiment of the present invention, the historical action instruction information and the fused feature vector of the intelligent agent are input into the recurrent neural network for deterministic state update, and the current simulation environment is modeled according to the deterministic state update result to obtain a world model, including:

[0097] Receive the agent's historical action information ( action ) and the fused feature vector ( embed ) as the initial input and passed into the recurrent neural network GRU (i.e., gated recurrent unit) for deterministic state update. The update function of the recurrent neural network GRU is expressed as:

[0098]

[0099] in, Is a mapping function used to represent the update state; Represents a deterministic state, Represents the potential state before the update; Represents the action information of this input; Represents a random state, which is represented by a Gaussian distribution.

[0100] After the deterministic state update is completed, the deterministic state update results are input into the Prior network and the Posterior network respectively; among them, the Prior network is based on historical information Predicted state distribution , and use Gaussian distribution to model the current simulation environment to obtain the world model; the Posterior network combines the fusion feature vector and calculates the posterior distribution through the encoder output and state update ;

[0101] The core role of the Prior network and the Posterior network is to capture the uncertainty in the system and to achieve the model's prediction of future states through probability distribution rather than deterministic prediction. Ultimately, the updated deterministic state ( deter ) and random state ( stoch ) are combined to form a complete state representation. This state representation can more comprehensively represent the state of the environment. Based on this state representation, we can predict parameters such as future states and observations. Incorporating random states can improve the accuracy and robustness of predictions.

[0102] Furthermore, the intelligent agent in this embodiment of the present invention learns in imagination through an actor-critic algorithm to maximize the target expected reward. The learning of action strategies is not directly constrained by language, but only serves as a prompt. This setup fully utilizes the multimodal features learned by the world model. The actor network's primary goal is to learn the distribution space of actions, while the critic network guides the actor's learning by estimating the expected discounted sum of rewards under a certain sequence.

[0103] Therefore, in step S4 of the embodiment of the present invention, the agent uses the actor-critic algorithm to perform imaginative learning based on the world model, and determines the current optimal action sequence of the agent based on the imaginative learning results, specifically including:

[0104] Based on the actor-critic algorithm, a multilayer perceptron (MLP) is used to construct the actor network and the critic network. The actor network and the critic network are shown in the following formula:

[0105] Actor Network:

[0106] Critics Network:

[0107] in, express The probability of taking a certain action under the current conditions at any moment, Represents the potential state representation of historical information, represents the observations to be learned, Indicates strategy, A mapping function that represents the estimated reward value under given conditions;

[0108] Setting the reward function for actor network learning :

[0109]

[0110] in, For the current reward, is the discount factor, Indicate whether to continue learning. express +1 moment to learn observations, express The potential state representation of historical information at time +1, express +1 moment of learning reward, is used to control the smoothing of the discount and ,The critic performs discrete regression on two-hot encoded targets using categorical cross-entropy loss;

[0111] Based on the world model, the actor network selects actions for imaginative learning according to the complete state representation. At the same time, the critic network evaluates the action rewards of the actor network's imaginative learning through the reward function and feeds back the action reward evaluation results to the actor network so that the actor network can adjust the imaginative learning actions according to the action reward evaluation results to maximize the action rewards of the actor network's imaginative learning.

[0112] In each round of imaginary learning of the actor network, when the number of imaginary learning times T = 15 or when the world model predicts that learning is complete (i.e. =1), the imaginary learning ends, and the current optimal action sequence of the agent is determined based on the actions of the actor network during the imaginary learning process.

[0113] The actor-critic algorithm is a reinforcement learning algorithm based on a value function and a policy function. The actor network (Actor) is responsible for learning the policy function and selecting actions based on the current state; the critic network (Critic) is responsible for evaluating the policy and providing a corresponding reward signal. The goal of the actor-critic algorithm is to continuously optimize the policy function so that the agent can maximize the cumulative reward. The actor-critic algorithm can be used to train robotic agents, enabling them to learn optimal action strategies through interaction with the environment.

[0114] Furthermore, in an embodiment of the present invention, the world model is provided with a feature learning loss function and a future prediction loss function; the feature learning loss function and the future prediction loss function are used to minimize the learning loss and the future prediction loss of the world model training.

[0115] The feature learning loss function is the image loss , language loss , reward loss , end prediction loss and regularization loss The specific loss function of the feature learning loss function is as follows:

[0116] Image loss function:

[0117] Language loss function:

[0118] Reward loss function:

[0119] End prediction loss function:

[0120] Regularization loss function:

[0121] in, is the classification cross entropy loss function; is the binary cross entropy loss function; Indicates stopping the update of gradient calculation; , is a hyperparameter; represents regularization; express function; Indicates observations that need to be learned; Express prediction of the moment; express Actual image observation at the moment; express Image observation prediction at a given moment; express Descriptive statement of the moment, express Prediction of descriptive statements at the moment; express Get rewards every moment. express Get reward predictions at all times; Indicates a coding method that allows two different locations to be activated simultaneously; Indicates the predicted value of whether the event will continue. In addition, It not only contains information about the current observation, but also contains additional information for predicting rewards and whether the task should continue.

[0122] In the embodiment of the present invention, the agent -1 moment observation and potential cycle state, combined to produce Therefore, the future prediction loss function of the world model is The specific formula is as follows:

[0123]

[0124] in, Indicates observations that need to be learned; , is a hyperparameter; Express Predictions of the moment.

[0125] Furthermore, in the embodiment of the present invention, the critic network and the actor network are respectively provided with a critic network loss function and the actor network loss function , as shown in the following formula:

[0126]

[0127]

[0128] in, represents the classification cross entropy loss function; Indicates stopping gradient update; express strategy; represents entropy; is the regularized sparsity of entropy; is the normalization factor, , used to calculate a dynamic range for normalizing reward values ​​greater than 1, thereby maintaining the stability of the actor network loss during training; express The estimated value of the current value function at the moment, for The estimated cumulative value of the moment.

[0129] The embodiments of the present invention make action decisions based on the context of historical actions and environmental information, effectively reducing the frequency of trajectory regression during the agent's exploration. When the environment changes, the agent can make timely adjustments based on contextual information, improving its adaptability and flexibility. When faced with new scenarios, contextual information provides additional knowledge to the world model, helping the agent quickly understand the current environmental characteristics and task requirements, allowing it to make more reasonable decisions in the new environment. Furthermore, contextual prompts provide task objectives, helping the agent adjust its strategy and strengthen its action planning capabilities.

[0130] Example 2: Reference Figure 4 As shown, this embodiment of the present invention, based on the above-mentioned embodiment 1, provides an overhead view of the HomeGrid simulation environment. Before executing a task, the agent needs to set the parameters of the simulation environment. See Tables 2, 3, and 4.

[0131] Table 2 Corresponding floor tables of room types in the simulation environment

[0132]

[0133] Table 3 List of item types

[0134]

[0135] Table 4 Statement prompt type and statement content setting table

[0136]

[0137] Parameters include room type and its corresponding floor, type of interactive items, statement prompt type and statement content, etc. Figure 4The living room, dining room, kitchen and other areas in the HomeGrid simulation environment are displayed, as well as the items displayed in each area. In order to distinguish the area boundaries and items, Figure 4 Different areas and objects are distinguished by filling different degrees of gray. Figure 4 HomeGrid simulation environment, refer to Figure 5 As shown in FIG, an embodiment of the present invention provides a schematic diagram of the operation of an intelligent agent performing the task of putting paper towels into a trash can. After setting the parameters of the simulation environment, the process of the intelligent agent performing the task of putting paper towels into a trash can in the embodiment of the present invention is as follows:

[0138] 1. When the number of steps is 0, a trash can is found;

[0139] 2. When the step count reaches 7, a tissue is found;

[0140] 3. Pick up the paper towel when the step number is 8 (successfully pick up the paper towel and the white part);

[0141] 4. Find your way back to the trash can;

[0142] 5. Reach the trash can when the step count is 15;

[0143] 6. Successfully put the tissue into the trash can and receive reward +1 (the tissue is successfully placed and there is no white part in the background).

[0144] In addition, refer to Figure 6 As shown, based on the above Figure 4 The embodiment of the present invention also provides a schematic diagram of the agent touching obstacles in the process of performing the task of picking up fruits. The specific execution process is as follows:

[0145] 1. When the step number is 0, fruit is found, and the previous action is set to none;

[0146] 2. When the step number is 2, pick up the fruit. At this time, the previous step action is set to pick up;

[0147] 3. Find the table, the up step action is up;

[0148] 4. If the obstacle is touched, the reward is -0.5, and the next step is right.

[0149] It can be seen from the above process of the agent performing the task that the agent of the embodiment of the present invention takes the joint representation of the image modality and the image description sentence as input, explores the path to complete the background task in the simulation environment based on the current input, and returns the cumulative reward value in 100 steps of action; after each action, the simulation environment returns the image modality input in the current agent's field of view, and the multimodal large model obtains the image description sentence based on the current image input; then the vector representation of the current image input is obtained through the convolutional neural network, and the vector representation of the image description sentence is obtained using the language model, and the graph vector representation and the sentence vector representation are spliced ​​to obtain a new vector representation; the new vector representation is used to model the world model using a recurrent neural network, and the world model implements action decision-making based on the current vector input; the agent performs actions to interact with the simulation environment, while realizing the change of the image modality in the agent's field of view, and also determines whether the background task is completed and whether to return the task reward. The present invention effectively reduces the possibility of the agent making wrong decisions and effectively increases the number of background tasks completed by the agent within a limited number of steps.

[0150] Example 3: Based on the method in Example 1, this embodiment of the present invention further provides a world model enhancement device based on contextual prompts, comprising:

[0151] A preprocessing module is configured to obtain background task description statements, environment observation images, and historical action instruction information of the agent in the simulation environment, and to extract image description statement context information from the environment observation images using a multimodal large language model; wherein the historical action instruction information is spliced ​​at the end of the image description statement context information;

[0152] a visual module, configured to extract visual features from the environmental observation image using a convolutional neural network to obtain an image feature vector, and to extract text features from the context information of the background task description statement and the image description statement to obtain a background task feature vector and a context text feature vector, respectively; concatenate the background task feature vector and the context text feature vector into a first text feature vector, and fuse the first text feature vector with the image feature vector to obtain a fused feature vector;

[0153] The memory module is used to input the agent's historical action command information and fused feature vectors into the recurrent neural network for deterministic state update, and to model the current simulation environment based on the deterministic state update results to obtain a world model;

[0154] The action module is used for the intelligent agent to perform imaginative learning based on the world model using the actor-critic algorithm, and determine the current optimal action sequence of the intelligent agent based on the results of imaginative learning; perform actions in the simulation environment based on the current optimal action sequence, and at the same time obtain action feedback, environment state update information and updated intelligent agent visual image given by the simulation environment after the action is executed, and judge whether the background task in the simulation environment is successfully completed after the action is executed. If the background task is not successfully completed after the action is executed, the action decision is made based on the updated intelligent agent visual image, the current optimal action sequence is adjusted according to the action decision result, and the next action is executed; if the background task is successfully completed after the action is executed, the simulation environment also returns a reward of 1.

[0155] Specifically, refer to Figure 7 The working principle diagram of the device shown in the figure is as follows. The specific working logic (i.e., working principle) of the world model enhancement device based on contextual prompts in an embodiment of the present invention is as follows:

[0156] In the preprocessing module and the vision module, the simulation environment returns the 3*3 image currently visible to the agent. The multimodal large model is used to align the image modality with the language modality to obtain the image description sentence. The image modality and the image description sentence are combined as the input of the world model.

[0157] In the memory module, based on the joint input of the image modality and the image description sentence, the recurrent neural network models the current simulation environment by processing the input. The modeled model is the world model.

[0158] In the action module, the agent uses the actor-critic algorithm to learn through imagination based on the world model and find the current optimal action sequence;

[0159] The agent performs decision-making actions in the simulation environment. The simulation environment provides action feedback and updates the environment state, returning the updated visual image of the agent. If the action successfully completes the background task after execution, the simulation environment also returns a reward of 1.

[0160] Figure 7 In the figure, V represents the visual module, which provides input to the memory module; M represents the memory module, which models the world model based on the input; C represents the action module, which completes the action decision based on the world model modeling.

[0161] Among them, the visual module is responsible for compressing the image seen into a representation ; The memory module is Learning, update the current internal state , and integrate historical information and predict future states; the action module represents the current image and internal memory state Generate current action decision ,Finally, action interaction is performed to change the environment image and iterate the process.

[0162] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A world model enhancement method based on contextual cues, characterized in that: include: Obtaining a background task description statement of the simulation environment, an environment observation image, and historical action instruction information of the intelligent agent, and extracting image description statement context information from the environment observation image using a multimodal large language model; wherein the historical action instruction information is spliced ​​at the end of the sentence of the image description statement context information; Using a convolutional neural network to extract visual features from the environmental observation image to obtain an image feature vector, and performing text feature extraction on the context information of the background task description sentence and the image description sentence to obtain a background task feature vector and a context text feature vector, respectively, and concatenating the background task feature vector and the context text feature vector into a first text feature vector, and fusing the first text feature vector with the image feature vector to obtain a fused feature vector; The agent's historical action command information and fused feature vector are input into the recurrent neural network for deterministic state update, and the current simulation environment is modeled based on the deterministic state update result to obtain the world model; Based on the world model, the intelligent agent uses the actor-critic algorithm to perform imaginative learning and determines the current optimal action sequence of the intelligent agent based on the results of imaginative learning; it performs actions in the simulation environment based on the current optimal action sequence, and obtains action feedback, environment state update information and updated intelligent agent visual image given by the simulation environment after the action is executed, and judges whether the background task in the simulation environment is successfully completed after the action is executed. If the background task is not successfully completed after the action is executed, the intelligent agent makes an action decision based on the updated intelligent agent visual image, adjusts the current optimal action sequence based on the action decision result, and executes the next action; if the background task is successfully completed after the action is executed, the simulation environment also returns a reward of 1.

2. The world model enhancement method based on contextual prompts according to claim 1, characterized in that: The method of using a convolutional neural network to extract visual features from the environment observation image to obtain an image feature vector, and extracting text features from the background task description sentence and the image description sentence context information to obtain a background task feature vector and a context text feature vector, respectively, includes: A convolutional neural network with a step size of 1 and a convolution kernel size of 2 is used to extract visual features from the environmental observation image to obtain a 512-dimensional image feature vector; The context information of the background task description sentence and the image description sentence are respectively input into the Sentence-Transformer tool for text feature extraction to obtain a 512-dimensional context text feature vector and background task feature vector .

3. The world model enhancement method based on contextual prompts according to claim 1, characterized in that: The step of concatenating the background task feature vector and the context text feature vector into a first text feature vector, and fusing the first text feature vector with the image feature vector to obtain a fused feature vector includes: The 512-dimensional context text feature vector and background task feature vector Perform vector splicing to obtain the first text feature vector. The splicing process is shown in the following formula: in, represents the first text feature vector, Represents the vector representation of the current background task description statement, A vector representation representing the contextual information of the image description sentence, Represents the concatenation operation of vectors; The first text feature vector is mapped by a multi-layer perceptron to obtain a 512-dimensional second text feature vector, and the 512-dimensional second text feature vector is concatenated and fused with the 512-dimensional image feature vector to obtain a fused feature vector.

4. The world model enhancement method based on contextual prompts according to claim 1, characterized in that: The agent's historical action command information and fused feature vector are input into the recurrent neural network for deterministic state update, and the current simulation environment is modeled according to the deterministic state update result to obtain a world model, including: The historical action information and fusion feature vector of the intelligent agent are received as initial input and passed into the recurrent neural network GRU for deterministic state update. The update function of the recurrent neural network GRU is expressed as: in, Is a mapping function used to represent the update state; Represents a deterministic state, Represents the potential state before the update; Represents the action information of this input; Represents a random state, which is represented by a Gaussian distribution; After the deterministic state update is completed, the deterministic state update results are input into the Prior network and the Posterior network respectively; among them, the Prior network is based on historical information Predicted state distribution , and use Gaussian distribution to model the current simulation environment to obtain the world model; the Posterior network combines the fusion feature vector and calculates the posterior distribution through the encoder output and state update ; The updated deterministic state is combined with the random state to obtain the complete state representation.

5. The world model enhancement method based on contextual prompts according to claim 1, characterized in that: The agent uses the actor-critic algorithm to perform imaginative learning based on the world model and determines the agent's current optimal action sequence based on the imaginative learning results, including: Based on the actor-critic algorithm, a multi-layer perceptron is used to construct the actor network and the critic network. The actor network and the critic network are shown as follows: Actor Network: Critics Network: in, express The probability of taking a certain action under the current conditions at any moment, Represents the potential state representation of historical information, represents the observations to be learned, Indicates strategy, A mapping function that represents the estimated reward value under given conditions; Setting up the reward function for actor network learning : in, For the current reward, is the discount factor, Indicate whether to continue learning. express +1 moment to learn observations, express The potential state representation of historical information at time +1, express +1 moment of learning reward, is used to control the smoothing of the discount and ,The critic performs discrete regression on two-hot encoded targets using categorical cross-entropy loss; Based on the world model, the actor network selects actions for imaginative learning according to the complete state representation. At the same time, the critic network evaluates the action rewards of the actor network's imaginative learning through the reward function and feeds back the action reward evaluation results to the actor network so that the actor network can adjust the imaginative learning actions according to the action reward evaluation results to maximize the action rewards of the actor network's imaginative learning. In each round of imaginary learning of the actor network, when the number of imaginary learning times T=15 or when the world model predicts that learning is complete, the imaginary learning ends, and the current optimal action sequence of the agent is determined based on the actions of the actor network during the imaginary learning process.

6. The world model enhancement method based on contextual prompts according to claim 1, characterized in that: The world model is provided with a feature learning loss function and a future prediction loss function; the feature learning loss function and the future prediction loss function are used to minimize the learning loss and the future prediction loss of the world model training; in, The feature learning loss function is the image loss , language loss , reward loss , end prediction loss and regularization loss The specific loss function of the feature learning loss function is as follows: Image loss function: Language loss function: Reward loss function: End prediction loss function: Regularization loss function: in, is the classification cross entropy loss function; is the binary cross entropy loss function; Indicates stopping the update of gradient calculation; , is a hyperparameter; represents regularization; express function; Indicates observations that need to be learned; Express predictions of the moment; express Actual image observation at the moment; express Image observation prediction at a given moment; express Descriptive statement of the moment, express Prediction of descriptive statements at the moment; express Get rewards every moment. express Get reward predictions at all times; Indicates a coding method that allows two different locations to be activated simultaneously; Indicates the predicted value of whether the event will continue; Future prediction loss function The specific formula is as follows: in, Indicates observations that need to be learned; , is a hyperparameter; Express Predictions of the moment.

7. The world model enhancement method based on contextual prompts according to claim 5, characterized in that: The critic network and actor network are respectively set with the critic network loss function and the actor network loss function , as shown in the following formula: in, represents the classification cross entropy loss function; Indicates stopping gradient update; express strategy; represents entropy; is the regularized sparsity of entropy; is the normalization factor, , used to calculate a dynamic range for normalizing reward values ​​greater than 1, thereby maintaining the stability of the actor network loss during training; express The estimated value of the current value function at the moment, for The estimated cumulative value of the moment.

8. A world model enhancement device based on contextual prompts, characterized in that: include: A preprocessing module is configured to obtain background task description statements, environment observation images, and historical action instruction information of the agent in the simulation environment, and to extract image description statement context information from the environment observation images using a multimodal large language model; wherein the historical action instruction information is spliced ​​at the end of the image description statement context information; a visual module, configured to extract visual features from the environmental observation image using a convolutional neural network to obtain an image feature vector, and to extract text features from the context information of the background task description statement and the image description statement to obtain a background task feature vector and a context text feature vector, respectively; concatenate the background task feature vector and the context text feature vector into a first text feature vector, and fuse the first text feature vector with the image feature vector to obtain a fused feature vector; The memory module is used to input the agent's historical action command information and fused feature vectors into the recurrent neural network for deterministic state update, and to model the current simulation environment based on the deterministic state update results to obtain a world model; The action module is used for the intelligent agent to perform imaginative learning based on the world model using the actor-critic algorithm, and determine the current optimal action sequence of the intelligent agent based on the results of imaginative learning; perform actions in the simulation environment based on the current optimal action sequence, and at the same time obtain action feedback, environment state update information and updated intelligent agent visual image given by the simulation environment after the action is executed, and judge whether the background task in the simulation environment is successfully completed after the action is executed. If the background task is not successfully completed after the action is executed, the action decision is made based on the updated intelligent agent visual image, the current optimal action sequence is adjusted according to the action decision result, and the next action is executed; if the background task is successfully completed after the action is executed, the simulation environment also returns a reward of 1.

Images