Task planning method of embodied intelligent world model based on visual reasoning

Abstract

This invention proposes a task planning method based on an embodied intelligent world model using visual reasoning. This method combines the language processing capabilities of a large language model with advanced visual perception technology, enabling the intelligent agent to pre-reasonize the environmental state and simulate possible actions through the world model with limited environmental visual information. This allows for effective decision-making and rapid adaptation to constantly changing environmental states in complex environments, thereby significantly improving the agent's task execution efficiency and task completion success rate in complex scenarios.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, specifically to a task planning method that combines visual reasoning and a large language model. Background Technology

[0002] Large Language Models (LLMs) are deep learning models trained on massive amounts of text data, capable of generating natural language text or understanding the meaning of language text. LLMs can handle various natural language tasks, such as text classification, question answering, and dialogue, and are an important pathway to artificial intelligence. Currently, LLMs employ a Transformer architecture and pre-training objectives (such as Language Modeling) similar to small models, differing only in increased model size, training data, and computational resources. The principle behind LLMs is based on deep learning, utilizing vast amounts of data and computational resources to train neural network models with numerous parameters. By continuously adjusting the model parameters, the model achieves optimal performance across various tasks.

[0003] Large language models, due to their superior natural language understanding and reasoning capabilities, are widely used in the field of embodied intelligence to achieve more natural human-computer interaction and task planning. However, existing methods suffer from a lack of direct perception and reasoning capabilities regarding the physical world, leading to difficulties for agents in making effective decisions and adapting to dynamic changes in complex environments. This is because large language models lack an internal world model to predict world states (such as environmental states and intermediate variable values) and simulate long-term outcomes of actions. To address this issue, this invention proposes a method for embodied intelligence world models based on visual reasoning. This method combines the language processing capabilities of large language models with advanced visual perception technology, enabling agents to pre-reasone about environmental states and simulate possible actions using a world model with limited visual environmental information. This allows for effective decision-making and rapid adaptation to constantly changing environmental states in complex environments, significantly improving the agent's task execution efficiency and success rate in complex scenarios. Existing technologies urgently need improvement to address these issues. Summary of the Invention

[0004] The main objective of this invention is to propose a method for an embodied intelligent world model based on visual reasoning. This method aims to address the shortcomings of existing approaches, such as a lack of direct perception and reasoning ability regarding the physical world. These shortcomings hinder intelligent agents from making effective decisions and adapting to dynamic changes in complex environments. The invention offers advantages such as improving the decision-making ability and adaptability of intelligent agents in complex environments, and significantly enhancing task execution efficiency and success rate.

[0005] According to a first embodiment of the present invention, a task planning method based on an embodied intelligent world model of visual reasoning is provided.

[0006] A task planning method based on a visual reasoning-based embodied intelligent world model, comprising the following steps:

[0007] (1) Receive task instructions and acquire visual images of the current environment; the task instructions are mapped to the embedding space through text encoding to obtain the task instruction embedding e text Visual images are mapped to the same embedding space as the task instructions through image encoding to obtain image feature embeddings e. img ;

[0008] (2) Embed the task instructions into e text With image feature embedding e img When pieced together, we get [e] text ,e img ]; will [e text ,e img Input to a large language model, which embeds image features into e img and task instructions embedded e text Perform association to generate text action commands and image feature vectors e voken ;

[0009] (3) Image feature vector e voken The conditional feature vector h is generated through the feature mapping module. voken ;

[0010] (4) Using the diffusion model based on the conditional eigenvector h voken Generate an environmental state image;

[0011] (5) Use the text action instructions generated in step (2) as the new input of the task instructions in step (1), and use the environmental state image generated in step (4) as the new input of the visual image in step (1) to iterate until all states of the task are generated.

[0012] In this invention, the text encoding in step (1) includes segmenting the task instructions into words to form a word set and mapping it to an embedding space. Step (1) specifically involves:

[0013] The task instruction X is segmented into words, and the segmented task instruction is represented as X={x1,…,x...} m}, where x represents the characters after word segmentation of the task instruction, and m represents the number of characters after word segmentation of the task instruction; X is processed into a word set according to the pre-made vocabulary. The processed word set is mapped to the embedding space R,e text ∈Rd Where: d represents the dimension of the embedding space R, that is, the dimension of the vector mapping the processed word set to the embedding space. e text The processed word set is mapped to the representation in the embedding space, i.e., task instruction embedding.

[0014] In this invention, the pre-made vocabulary uses the Llama2 pre-made vocabulary. The task instruction X is input into the wordpiece module. The BPE algorithm in wordpiece first divides the task instruction into individual characters, then counts the frequency of single characters or combinations of adjacent characters in the pre-made vocabulary within the data collection unit. Each time, the single character or combination of adjacent characters with the highest frequency is saved as a word until the loop ends, thereby generating a preprocessed word set. .

[0015] In this invention, mapping the visual image to the same embedding space as the task instructions through image encoding specifically involves dividing the current environment's visual image Y into n non-overlapping image blocks of size P×P, i.e., Y={y1,…,y...}. n The image encoder module uses linear embedding to map image blocks to the same embedding space R,e as the task instructions. img ∈R n×P×P×c Where n represents the number of image patches, P×P represents the size of each image patch, and C represents the number of channels in the image patch. img This is a representation of an image patch mapped to the embedding space, i.e., image feature embedding.

[0016] As a preferred method, the multi-scale feature extraction network (FPN) is used to segment the image Y.

[0017] In this invention, the image encoder module consists of a ViT module, a Q-Former module, and a multilayer perceptron. Image blocks are embedded into the ViT module. The ViT module uses a self-attention mechanism to capture the global relationships between regions within the image blocks, outputting a feature vector containing global semantic information of the image. This feature vector is then fed into the Q-Former module. The Q-Former module selects visual features relevant to the task instructions from the global feature vector extracted by the ViT module using a set of learnable query vectors. The output of the Q-Former module is then input into the multilayer perceptron, mapping it to the same embedding space R as the task instructions.

[0018] Preferably, the large language model mentioned in step (2) is the open-source large model Llama2. The association mentioned in step (2) specifically involves Llama2 embedding image features into e through a self-attention mechanism and hierarchical structure. img and task instructions embedded e textAssociativity with the input [e] text ,e img [Perform multi-layered understanding to generate text action commands and image feature vectors e] voken .

[0019] Preferably, the feature mapping module in step (3) consists of an MLP and a transformer. Step (3) specifically involves: mapping the image feature vector e generated in step (2) to... voken The MLP is transformed into a dimension that conforms to the input dimension of the transformer, and then the transformer is used for encoding and decoding to generate the conditional feature vector h. voken .

[0020] Preferably, the transformer consists of a 6-layer encoder and a 6-layer decoder.

[0021] Preferably, step (3) specifically involves: processing the image feature vector e generated in step (2) voken The MLP is transformed into a dimension that conforms to the input dimension of a transformer. Then, a transformer performs encoding and decoding, and a learnable decoder feature sequence q is used to generate a conditional feature vector h. voken .

[0022] In this invention, step (4) specifically involves: adding random Gaussian noise forward to the visual image Y and applying conditional feature vector h. voken The process of removing random Gaussian noise generates an environmental state image.

[0023] The step of adding random Gaussian noise forward specifically involves adding noise to the image Y during the forward Markov process of the standard diffusion model, thereby obtaining a noisy image.

[0024] The reverse removal of random Gaussian noise specifically involves: in the conditional feature vector h voken Under the guidance of [the relevant authority], a neural network decoder is used to iteratively remove random noise from the noisy image to obtain an environmental state image.

[0025] In this invention, generating all states of a completed task in step (5) means that the similarity between the two generated environmental state images is 95% or higher, preferably 98% or higher. Alternatively, generating all states of a completed task in step (5) means that the number of iterations is greater than or equal to 5, preferably greater than or equal to 10.

[0026] According to a second embodiment provided by the present invention, the use of the method described in the first embodiment is provided.

[0027] The method described in the first implementation scheme can be used in autonomous driving scenarios, smart homes, smart manufacturing scenarios, smart robots, or VR games.

[0028] In this invention, a task instruction refers to an operational command or order issued by a person operating the intelligent agent to the intelligent agent. The visual image of the current environment refers to a photograph taken by the intelligent agent's own camera component, showing its location and environment. The intelligent agent itself has the function of taking pictures and receiving instructions. Based on the task instruction and the visual image of the current environment, the intelligent agent processes the data using the task planning method provided by this invention to obtain a task planning map combining the task instruction and the current environment. This task planning map is then input to the intelligent agent's execution mechanism, and the agent performs the task according to the processed task planning map.

[0029] Throughout the development of this technical field, enabling intelligent agents to make effective decisions and adapt to dynamic changes in complex environments has been a crucial research direction. However, existing technologies have significant limitations in this area, primarily due to a lack of direct perception and reasoning capabilities regarding the physical world. This makes it difficult for intelligent agents to make effective decisions and adapt to dynamic changes in complex environments. To address these issues, this invention proposes a task planning method based on an embodied intelligent world model using visual reasoning.

[0030] This invention proposes a method for embodied intelligent world models based on visual reasoning. This method combines the language processing capabilities of a large language model with advanced visual perception technology, enabling intelligent agents to pre-reasone about environmental states and simulate possible actions using a world model with limited visual information. This allows for effective decision-making and rapid adaptation to constantly changing environmental states in complex environments, significantly improving the agent's task execution efficiency and success rate in complex scenarios.

[0031] This invention receives task instructions and visual images, encodes them into the same embedding space, and uses a large language model to associate image features with task instructions, generating text action instructions and image feature vectors. Then, an environmental state image is generated through a feature mapping module and a diffusion model, and this image is used as new input for iterative processing until the task is completed. By encoding task instructions and visual images into the same embedding space and using a large language model for association, task instructions can be effectively combined with environmental visual information to generate accurate action instructions and environmental state images. The use of the feature mapping module and diffusion model enables the system to generate accurate environmental state images in dynamically changing environments, thereby helping the agent make effective decisions and adapt in complex environments.

[0032] For example, in autonomous driving scenarios, the agent needs to make decisions based on visual information of the road environment and driving task instructions. By encoding the driving task instructions and visual images of the road environment into the same embedding space and associating them through a large language model, accurate driving action instructions and road state images can be generated. Through the road state images generated by the feature mapping module and the diffusion model, the agent can make effective decisions and adapt to dynamically changing road environments.

[0033] Compared to existing technologies, this invention, by encoding task instructions and visual images into the same embedding space and using a large language model for association, can more accurately combine task instructions with environmental visual information, generating more accurate action instructions and environmental state images. The use of a feature mapping module and a diffusion model enables the system to generate accurate environmental state images in dynamically changing environments, thereby significantly improving the agent's task execution efficiency and success rate in complex scenarios.

[0034] Furthermore, this application proposes to segment task instructions into words, forming a word set, and then map it to an embedding space. Specifically, the task instructions are segmented into words, with the segmented instructions represented as multiple characters, and processed into a word set according to a pre-prepared vocabulary. Then, the word set is mapped to the embedding space R to generate the task instruction embedding. This technique, by converting task instructions into embedding vectors, enables task instructions to be processed and associated with visual image features within the same embedding space, thereby solving the problem of how to map task instructions to the embedding space.

[0035] In this invention, existing word segmentation algorithms can be used to segment the task instructions, such as rule-based word segmentation algorithms, statistical model word segmentation algorithms, or deep learning word segmentation algorithms. The pre-built vocabulary can be based on existing language models, such as the pre-built vocabulary of Llama2. The mapping to the embedding space can be implemented using a trained embedding model, such as Word2Vec, GloVe, or BERT.

[0036] In this invention, the word segmentation method involves segmenting the task instructions to obtain their characters. Any existing word segmentation model and method can be used, such as Hidden Markov Models (HMMs) or Conditional Random Fields (CRFs).

[0037] In this invention, wordpiece is a mature existing technology. This invention uses the wordpiece module to segment task instructions into words, and then processes the segmented task instructions to obtain a word set.

[0038] During implementation, Llama2's pre-built vocabulary can be optimized in several ways. For example, increasing the vocabulary size of the pre-built vocabulary can improve the accuracy and efficiency of word segmentation. Furthermore, the segmentation performance can be optimized by adjusting parameters of the BPE algorithm, such as the number of iterations and statistical frequency. Additionally, it can be combined with other word segmentation algorithms, such as WordPiece or SentencePiece, to improve the accuracy and robustness of word segmentation.

[0039] In this invention, by dividing the visual image into multiple non-overlapping image patches and mapping them to an embedding space using linear embedding, it is ensured that image features can be processed in the same space as task instructions. The use of a multi-scale feature extraction network (FPN) further enhances the image feature extraction effect, making the mapping process more accurate and efficient.

[0040] Preferably, the image encoder module can be implemented in various ways, such as using a convolutional neural network (CNN) for image patch feature extraction and embedding, or using a visual transformer (ViT) for global feature capture.

[0041] In this invention, the image encoder module is a built-in module of the open-source large-scale model Llama2, consisting of a ViT module, a Q-Former module, and a multilayer perceptron. The ViT module captures the global relationships between regions in an image patch through a self-attention mechanism, enabling the extraction of global semantic information from the image. The Q-Former module, through a set of learnable query vectors, selects visual features relevant to the task instructions from the global feature vectors extracted by ViT, thereby improving the correlation between image features and task instructions. The multilayer perceptron maps the output of the Q-Former to the same embedding space as the task instructions, allowing image features to be processed and analyzed within the same embedding space. This technical solution not only solves the problem of how to encode visual images and map them to the same embedding space as the task instructions but also improves the accuracy and efficiency of large language models in processing image features and task instructions.

[0042] In this invention, the open-source large model Llama2 is the open-source large model officially released by Meta AI.

[0043] Llama2 associates image feature embeddings (eimg) and task instruction embeddings (etext) through a self-attention mechanism and hierarchical structure. Specifically, Llama2 first encodes the input image features and task instructions using a self-attention mechanism to generate preliminary embedding representations. Then, it further processes these embedding representations through a hierarchical structure, extracting higher-level features layer by layer, ultimately generating text action instructions and image feature vectors (evoken). This multi-layered understanding approach allows Llama2 to better capture complex relationships and context in the input information, thereby improving the agent's perception and reasoning capabilities in the physical environment. By employing Llama2, this application achieves enhanced direct perception and reasoning capabilities of the physical world in large language models. Llama2's self-attention mechanism and hierarchical structure enable it to effectively associate image features and task instructions, thereby generating accurate text action instructions and image feature vectors. This method not only improves the agent's perception of the environment but also enhances its reasoning and decision-making capabilities in complex scenarios. Therefore, this application provides an effective technical means to address the problem of insufficient direct perception and reasoning capabilities of the physical world in large language models.

[0044] In this invention, the feature mapping module consists of an MLP and a transformer. Through their cooperation, they transform image feature vectors into conditional feature vectors, thereby playing a role in generating environmental state images. Using this scheme, the image feature vector `evoken` is first transformed by the MLP into dimensions consistent with the transformer input, and then encoded and decoded by the transformer to generate the conditional feature vector `hvoken`. This method solves the technical problem of how to transform image feature vectors into conditional feature vectors to generate environmental state images, ensuring that the image feature vectors can be effectively used in the subsequent environmental state image generation process.

[0045] Specifically, MLPs can employ a multilayer perceptron structure, using multiple linear transformations and activation functions to convert the input feature vector into dimensions consistent with transformer inputs. The transformer's encoder encodes the input feature vector using a self-attention mechanism and a feedforward neural network, generating an intermediate representation; the decoder then decodes this intermediate representation using a similar mechanism, generating the final conditional feature vector h. voken .

[0046] In this invention, the visual image Y undergoes a process of adding random Gaussian noise forward and removing the random Gaussian noise. The conditional feature vector h... vokenIt plays a guiding and controlling role in the process of generating environmental state images. The combination of noise addition in the forward process and noise removal in the backward process enables the final generated environmental state image to accurately reflect the environmental state described by the conditional feature vector hvoken, thus solving the technical problem of how to generate environmental state images from the conditional feature vector hvoken.

[0047] Preferably, the forward addition of random Gaussian noise can employ a standard forward Markov process. By controlling and guiding the noise, the image after noise addition can better preserve the information of the original image. The reverse removal of random Gaussian noise can be achieved through multiple iterations, gradually removing the noise and ultimately generating a clear environmental image. Furthermore, different neural network decoder structures can be used to improve the denoising effect and efficiency.

[0048] In this invention, the diffusion models include a forward diffusion process and a reverse generation process. The forward diffusion process involves gradually adding noise to the data, transforming it into pure noise. This process is a Markov chain, adding a small amount of Gaussian noise at each step. Starting from the true data distribution, noise is gradually added over multiple time steps, ultimately making the data distribution approximate a standard Gaussian distribution. The reverse generation process learns to gradually remove noise from the data, recovering the original data. This process is also a Markov chain, but in the opposite direction, gradually removing noise. Starting from standard Gaussian noise, through the learned reverse diffusion process, noise is gradually removed, generating new data consistent with the true data distribution.

[0049] In this invention, by setting standards for image similarity or iteration count, the accuracy of the generated environmental state images and the reliability of task completion can be effectively evaluated and guaranteed, thereby improving the efficiency and success rate of intelligent agents in performing tasks in complex environments.

[0050] Preferably, the similarity calculation of the two images follows existing techniques, and this invention employs the Structural Similarity Index (SSIM). Other image similarity algorithms, such as Mean Squared Error (MSE) or Peak Signal-to-Noise Ratio (PSNR), can also be used. The number of iterations can be adjusted according to the complexity of the task and dynamic changes in the environment to ensure the task is completed within a reasonable timeframe. Furthermore, an adaptive algorithm can be introduced to dynamically adjust the number of iterations based on the real-time status of the task execution, further improving the efficiency and accuracy of task completion.

[0051] The method provided by this invention combines the language processing capabilities of a large language model with advanced visual perception technology, enabling intelligent agents to pre-infer environmental states and simulate possible actions using a world model with limited environmental visual information. This improves the efficiency and success rate of task execution in complex scenarios. In autonomous driving scenarios, intelligent agents can acquire visual images of the vehicle's surroundings and combine them with the understanding of driving commands through a large language model to perform effective path planning and obstacle avoidance. In the smart home field, intelligent agents can perceive the home environment visually, understand user commands, and achieve automatic control of smart appliances and home security monitoring. In smart manufacturing scenarios, intelligent agents can monitor production lines visually and optimize equipment scheduling and production processes based on task commands. In the field of intelligent robots, intelligent agents can perceive environmental changes visually and perform autonomous navigation and task execution based on task commands. In the field of VR games, intelligent agents can perceive the game environment visually and control characters and complete tasks based on task commands.

[0052] The method provided by this invention significantly improves the decision-making and adaptability of intelligent agents in complex environments by applying a task planning method based on the embodied intelligent world model of visual reasoning to multiple real-world scenarios, and solves the technical problem in the prior art that intelligent agents have difficulty making effective decisions and adapting to dynamic changes in complex environments.

[0053] Compared with the prior art, the technical solution provided by the present invention has the following beneficial technical effects:

[0054] 1. By combining the language processing capabilities of a large language model with advanced visual perception technology, the intelligent agent can pre-infer the environmental state and simulate possible actions through a world model with limited environmental visual information. This has the advantages of improving the agent's decision-making ability and adaptability in complex environments, and significantly improving task execution efficiency and success rate.

[0055] 2. It adopts innovative image Voken technology to achieve deep fusion of text and image features.

[0056] 3. A visual environment state generation method based on a diffusion model is introduced, which can simulate and predict environmental changes.

[0057] 4. An iterative task planning method is adopted to enable the agent to adapt to the dynamically changing environment.

[0058] 5. By constructing a world model, the efficiency and success rate of intelligent agents in performing tasks in complex scenarios are improved. Attached Figure Description

[0059] Figure 1 This is a schematic diagram of an embodied intelligent world model method for visual reasoning according to the present invention;

[0060] Figure 2 This is a schematic diagram illustrating the training of the embodied intelligent world model of the present invention;

[0061] Figure 3 This is a schematic diagram of the embodied intelligent world model of the present invention. Detailed Implementation

[0062] The technical solutions of this application will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. The components of this application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of this application.

[0063] All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application. Example 1

[0064] A task planning method based on a visual reasoning-based embodied intelligent world model, comprising the following steps:

[0065] (1) Receive task instructions and acquire visual images of the current environment; the task instructions are mapped to the embedding space through text encoding to obtain the task instruction embedding e text Visual images are mapped to the same embedding space as the task instructions through image encoding to obtain image feature embeddings e. img ;

[0066] (2) Embed the task instructions into e text With image feature embedding e img When pieced together, we get [e] text ,e img ]; will [e text ,e img Input to a large language model, which embeds image features into e img and task instructions embedded e text Perform association to generate text action commands and image feature vectors e voken ;

[0067] (3) Image feature vector e voken The conditional feature vector h is generated through the feature mapping module. voken ;

[0068] (4) Using the diffusion model based on the conditional eigenvector h vokenGenerate an environmental state image;

[0069] (5) Use the text action instructions generated in step (2) as the new input of the task instructions in step (1), and use the environmental state image generated in step (4) as the new input of the visual image in step (1) to iterate until all states of the task are generated. Example 2

[0070] Repeat Example 1, except that the text encoding in step (1) includes segmenting the task instructions into words to form a word set and mapping it to an embedding space; specifically:

[0071] The task instruction X is segmented into words, and the segmented task instruction is represented as X={x1,…,x...} m}, where x represents the characters after word segmentation of the task instruction, and m represents the number of characters after word segmentation of the task instruction; X is processed into a word set according to the pre-made vocabulary. The processed word set is mapped to the embedding space R,e text ∈R d Where: d represents the dimension of the embedding space R, that is, the dimension of the vector that maps the processed word set to the embedding space.

[0072] The pre-made vocabulary uses the Llama2 pre-made vocabulary. The task instruction X is input into the wordpiece module. The BPE algorithm in wordpiece first divides the task instruction into individual characters, then counts the frequency of single characters or combinations of adjacent characters in the pre-made vocabulary. Each time, the single character or combination of adjacent characters with the highest frequency is saved as a word. This process continues until the loop ends, thus generating a preprocessed word set. . Example 3

[0073] Repeat Example 2, except that the visual image is mapped to the same embedding space as the task instruction through image encoding. Specifically, a multi-scale feature extraction network (FPN) is used to divide the current environment's visual image Y into n non-overlapping image blocks of size P×P, i.e., Y={y1,…,y...}. n The image encoder module uses linear embedding to map image blocks to the same embedding space R,e as the task instructions. img ∈R n×P×P×c Where n represents the number of image blocks, P×P represents the size of each image block, and C represents the number of channels in the image block;

[0074] The image encoder module consists of a ViT module, a Q-Former module, and a multilayer perceptron. Image patches are embedded into the ViT module, which uses a self-attention mechanism to capture the global relationships between regions within the image patches, outputting a feature vector containing global semantic information of the image. This feature vector is then fed into the Q-Former module, which selects visual features relevant to the task instructions from the global feature vector extracted by the ViT module using a set of learnable query vectors. Finally, the output of the Q-Former is input into the multilayer perceptron, mapping it to the same embedding space R as the task instructions. Example 4

[0075] Repeat Example 3, except that the large language model mentioned in step (2) is the open-source large model Llama2; the association mentioned in step (2) specifically refers to: Llama2 embedding image features into e through a self-attention mechanism and hierarchical structure. img and task instructions embedded e text Associativity with the input [e] text ,e img [Perform multi-layered understanding to generate text action commands and image feature vectors e] voken .

[0076] The image e voken It involves special character embedding, therefore the Llama2 word table is expanded using W. i ∈V∪V img This aligns the text with the image Voken, generating intermediate values ​​for the conditional feature vector. Its loss function can be formalized as follows:

[0077] .

[0078] The feature mapping module in step (3) consists of an MLP and a transformer. Specifically, step (3) involves mapping the image feature vector e generated in step (2) to the MLP and the transformer. voken The MLP is transformed into a dimension that conforms to the input dimension of the transformer, and then the transformer is used for encoding and decoding to generate the conditional feature vector h. voken The transformer consists of a 6-layer encoder and a 6-layer decoder.

[0079] The features generated by the mapping are formalized as follows:

[0080] .

[0081] Example 5

[0082] Repeat Example 4, except that step (4) is specifically: perform a forward random Gaussian noise addition and removal process on the visual image Y to generate an environmental state image;

[0083] The forward addition of random Gaussian noise specifically involves: during the standard forward Markov process of the diffusion model, random Gaussian noise is gradually injected into the image Y until the random Gaussian noise is incorporated into an isotropic Gaussian distribution, thus obtaining the noisy image.

[0084] The reverse removal of random Gaussian noise specifically involves: in the conditional feature vector h voken Under the guidance of [previous method], a neural network decoder is used to iteratively remove random noise from the noisy image to obtain an environmental state image. Then, the reverse Gaussian noise removal process is initiated, utilizing conditional feature vectors and a neural network decoder, i.e. The process involves gradually removing Gaussian noise through fitting until every distribution in the image is restored.

[0085] In this embodiment, the forward process is achieved by moving to Z... t Add random Gaussian noise to generate Z t+1 The process of generating Z t+1 It depends solely on Z t Therefore, the forward process is a Markov process, and it does not contain any trainable parameters. The noise level is determined by the variance β. t The ∈ (0,1) determines the forward process. The forward process can be formalized as:

[0086] ,

[0087] Z is obtained through the reparameter re-parameter technique. t ,in , If t≤0, then:

[0088] ;

[0089] Then the Markov process can be formalized as:

[0090] .

[0091] For the reverse process of removing random Gaussian noise, if the given condition Z is known... t Can Z be determined t-1 The probability distribution, i.e. Given this, we can iteratively sample random noise to generate an image or sentence. If It is unknown, but it can be used. To approximate this, considering that the noise added at each step is relatively small, we assume... It follows a Gaussian distribution and can be modeled using neural networks. Therefore, the inverse conditional denoising process can be formalized as:

[0092] ;

[0093] in: and These are the parameters for which the mean and variance need to be predicted, respectively.

[0094] Given the posterior conditional probability distribution during the forward process The ultimate goal is to minimize the model. The mean squared error loss between the posterior conditional probability distribution q and the KL divergence. The loss function for conditional denoising is formalized as follows:

[0095] .

[0096] Therefore, the h generated in step S3 voken The image Y enters the standard forward Markov process of the diffusion model, and then enters the standard forward noise addition process, adding relatively small noise at each step. until the final state Z is obtained. t .

[0097] The reverse process specifically involves starting from the final state Z. t Conditional vector h voken Under the guidance of [unclear], the process of recovering Y by iteratively removing random noise uses a neural network. To fit.

[0098] In this embodiment, the neural network decoder Using pre-trained The denoising loss function is formalized as follows:

[0099] .

[0100] The U_Net network architecture consists of an encoder (downsampling path) and a decoder (upsampling path), comprising 4 encoder layers, 1 bottleneck layer, and 4 decoder layers.

[0101] Specifically, in this embodiment, the random sampling from the standard Gaussian distribution... and h generated by steps (2) and (3) with the task instructions and image Y. voken Enter the information in step (4) Decode the code to generate reasoning actions for solving the task.

[0102] Example 6

[0103] Repeat Example 5, except that in step (5), random sampling is performed from a standard Gaussian distribution. and h generated by steps (2) and (3) with the task instructions and image Y. voken Enter the information in step (4) Decode the code to generate reasoning actions for solving the task.

[0104] The generation of all states for completing the task refers to the similarity of the environmental state images generated before and after the task being completed reaching 95% or more. Example 7

[0105] Repeat Example 5, except that the generation of all states of the completed task in step (5) means: the number of iterations is 5, and it ends automatically. Example 8

[0106] A task planning method for autonomous driving scenarios, wherein the task planning method adopts the technical solution described in Example 6. Example 9

[0107] A task planning method for a smart home scenario, wherein the task planning method adopts the technical solution described in Example 6. Example 10

[0108] A task planning method for intelligent manufacturing scenarios, which adopts the technical solution described in Example 6. Example 11

[0109] A task planning method for an intelligent robot, wherein the task planning method adopts the technical solution described in Example 6. Example 12

[0110] A task planning method for VR games, which adopts the technical solution described in Example 6.

Claims

1. A task planning method based on a visual reasoning-based embodied intelligent world model, the method comprising the following steps: (1) Receive task instructions and acquire visual images of the current environment; The task instruction is mapped to an embedding space by text encoding to obtain a task instruction embedding e text The visual image is mapped to the same embedding space as the task instruction by image encoding to obtain an image feature embedding e img ; (2) Embed the task instructions into e text With image feature embedding e img When pieced together, we get [e] text ,e img ]; [e text ,e img Input to a large language model, which embeds image features into e img and task instructions embedded e text Perform association to generate text action commands and image feature vectors e voken The large language model mentioned is the open-source large model Llama2; the association specifically refers to: Llama2 embedding image features into e through a self-attention mechanism and hierarchical structure. img and task instructions embedded e text Associativity with the input [e] text ,e img [Perform multi-layered understanding to generate text action commands and image feature vectors e] voken ; (3) Image feature vector e voken The conditional feature vector h is generated through the feature mapping module. voken ; (4) Using the diffusion model based on the conditional eigenvector h voken Generate an environmental state image; (5) Use the text action instructions generated in step (2) as the new input of the task instructions in step (1), and use the environmental state image generated in step (4) as the new input of the visual image in step (1) to iterate until all states of the task are generated.

2. The task planning method according to claim 1, characterized in that: The text encoding in step (1) includes segmenting the task instructions into words to form a word set and mapping it to the embedding space; specifically: The task instruction X is segmented into words, and the segmented task instruction is represented as X={x1,…,x...} m }, where x represents the characters after word segmentation of the task instruction, and m represents the number of characters after word segmentation of the task instruction; X is processed into a word set according to the pre-made vocabulary. The processed word set is mapped to the embedding space R,e text ∈R d Where: d represents the dimension of the embedding space R, that is, the dimension of the vector that maps the processed word set to the embedding space.

3. The task planning method according to claim 2, characterized in that: The pre-made vocabulary uses the Llama2 pre-made vocabulary. The task instruction X is input into the wordpiece module. The BPE algorithm in wordpiece first divides the task instruction into individual characters, then counts the frequency of single characters or combinations of adjacent characters in the pre-made vocabulary. Each time, the single character or combination of adjacent characters with the highest frequency is saved as a word. This process continues until the loop ends, thus generating a preprocessed word set. .

4. The task planning method according to claim 1, characterized in that: The visual image is mapped to the same embedding space as the task instructions through image encoding. Specifically, the visual image Y of the current environment is divided into n non-overlapping image blocks of size P×P, i.e., Y={y1,…,y...}. n The image encoder module uses linear embedding to map image blocks to the same embedding space R,e as the task instructions. img ∈R n×P×P×c Where n represents the number of image blocks, P×P represents the size of each image block, and c represents the number of channels in the image block.

5. The task planning method according to claim 4, characterized in that: The image Y is segmented using a multi-scale feature extraction network (FPN).

6. The task planning method according to claim 5, characterized in that: The image encoder module consists of a ViT module, a Q-Former module, and a multilayer perceptron. Image patches are embedded into the ViT module, which uses a self-attention mechanism to capture the global relationships between regions within the image patches, outputting a feature vector containing global semantic information of the image. This feature vector is then fed into the Q-Former module, which selects visual features relevant to the task instructions from the global feature vector extracted by the ViT module using a set of learnable query vectors. Finally, the output of the Q-Former is input into the multilayer perceptron, mapping it to the same embedding space R as the task instructions.

7. The task planning method according to claim 1, characterized in that: The feature mapping module in step (3) consists of an MLP and a transformer. Specifically, step (3) involves mapping the image feature vector e generated in step (2) to the MLP and the transformer. voken The MLP is transformed into a dimension that conforms to the input dimension of the transformer, and then the transformer is used for encoding and decoding to generate the conditional feature vector h. voken .

8. The task planning method according to claim 7, characterized in that: The transformer consists of a 6-layer encoder and a 6-layer decoder.

9. The task planning method according to claim 1, characterized in that: Step (4) specifically involves: adding random Gaussian noise to the visual image Y in the forward pass and adding noise based on the conditional feature vector h. voken Under the guidance of [the relevant authority], a process of removing random Gaussian noise is performed to generate an environmental state image; The step of adding random Gaussian noise forward specifically involves adding noise to the image Y during the forward Markov process of the standard diffusion model Y to obtain a noisy image. The removal of random Gaussian noise specifically involves: in the conditional feature vector h voken Under the guidance of [the relevant authority], a neural network decoder is used to iteratively remove random noise from the noisy image to obtain an environmental state image.

10. The task planning method according to claim 1, characterized in that: In step (5), generating all states of the completed task means that the similarity between the two generated environmental state images reaches more than 95%.

11. The task planning method according to claim 10, characterized in that: In step (5), generating all states of the completed task means that the similarity between the two generated environmental state images reaches more than 98%.

12. The task planning method according to claim 1, characterized in that: In step (5), generating all states of the completed task means that the number of iterations is greater than or equal to 5.

13. The task planning method according to claim 1, characterized in that: In step (5), generating all states of the completed task means that the number of iterations is greater than or equal to 10.

14. The task planning method according to any one of claims 1-9, wherein the method is applied to autonomous driving scenarios, smart homes, smart manufacturing scenarios, smart robots, or VR games.

Images