Task planning method of intelligent world model with body based on visual reasoning

Abstract

The invention provides a task planning method for an intelligent world model with a body based on visual reasoning, and the method combines the language processing capability of a large language model and an advanced visual perception technology, so that an intelligent agent can pre-reasone an environment state and simulate possible actions through the world model under the condition of obtaining a small amount of environment visual information; therefore, effective decision making and rapid adaptation to constantly changing environment states are realized in a complex environment, and the task execution efficiency and the task completion success rate of the intelligent agent in a complex scene are remarkably improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a task planning method combining visual reasoning and a large language model. Background Art

[0002] Large language models (LLMs) are deep learning models trained using large amounts of text data. They can generate natural language text or understand the meaning of text. Large language models can handle a variety of natural language tasks, such as text classification, question-answering, and conversational conversation, and are a key path to artificial intelligence. Currently, large language models use a similar Transformer architecture and pre-training objectives (such as language modeling) as small models. They differ from small models in terms of increased model size, training data, and computing resources. Large language models are based on deep learning, which utilizes vast amounts of data and computing resources to train neural network models with numerous parameters. By continuously adjusting model parameters, the model achieves optimal performance across a variety of tasks.

[0003] Large language models, due to their superior natural language understanding and reasoning capabilities, are widely used in the field of embodied intelligence to enable more natural human-computer interaction and task planning. However, existing approaches suffer from a lack of direct perception of the physical world and insufficient reasoning capabilities, making it difficult for agents to make effective decisions and adapt to dynamic changes in complex environments. This is because large language models lack an internal world model to predict world states (such as environmental states and intermediate variable values) and the long-term consequences of simulated actions. To address this issue, this paper proposes a method for embodied intelligence world models based on visual reasoning. This method combines the language processing capabilities of large language models with advanced visual perception technologies, enabling agents to use world models to predict environmental states and simulate possible actions with minimal visual information. This allows them to make effective decisions and quickly adapt to changing environmental conditions in complex environments. This significantly improves the efficiency and success rate of task execution in complex scenarios. To address these issues, the existing technology urgently needs improvement. Summary of the Invention

[0004] The main purpose of this paper is to propose a method for embodied intelligent world models based on visual reasoning. This approach aims to address the technical issues faced by existing methods, such as a lack of direct perception of the physical world and insufficient reasoning capabilities, which make it difficult for intelligent agents to make effective decisions and adapt to dynamic changes in complex environments. This approach has the advantages of improving the decision-making ability and adaptability of intelligent agents in complex environments, significantly enhancing the efficiency and success rate of task execution.

[0005] According to a first embodiment provided by the present invention, a task planning method of an embodied intelligent world model based on visual reasoning is provided.

[0006] A task planning method based on an embodied intelligent world model based on visual reasoning, the method comprising the following steps: (1) Receive task instructions and obtain the visual image of the current environment; map the task instructions to the embedding space through text encoding to obtain the task instruction embedding e text ; The visual image is mapped to the same embedding space as the task instruction through image encoding to obtain the image feature embedding e img ; (2) Embed task instructions into e text Embedded with image features img Put together, we get [e text ,e img ]; [e text ,e img ] Input the large language model, which embeds the image features into e img and task instructions embedded in e text Associate and generate text action instructions and image feature vector e voken ; (3) Image feature vector e voken Generate the conditional feature vector h through feature mapping module mapping voken ; (4) Using the diffusion model according to the conditional eigenvector h voken Generate an image of the environment state; (5) The text action instructions generated in step (2) are used as the new input of the task instructions in step (1), and the environment state image generated in step (4) is used as the new input of the visual image in step (1), and the process is repeated until all states for completing the task are generated.

[0007] In the present invention, the text encoding in step (1) includes segmenting the task instructions into words, forming a word set, and mapping it to the embedding space. Step (1) is specifically as follows: The task instruction X is segmented, and the task instruction after segmentation is represented as X={x1,…,x m}, where x represents the characters after the task instruction is segmented, and m represents the number of characters after the task instruction is segmented; X is processed into a word set according to the pre-made word list , map the processed word set to the embedding space R,e text ∈R d ; Where: d represents the dimension of the embedding space R, that is, the dimension of the vector mapped from the processed word set to the embedding space. textThe processed word set is mapped to the representation of the embedding space, that is, the task instruction embedding.

[0008] In the present invention, the prefabricated vocabulary adopts the prefabricated vocabulary of Llama2. That is, the task instruction X is input into the wordpiece module. The BPE algorithm in wordpiece first divides the task instruction into characters one by one, and then counts the number of times a single character or a combination of adjacent characters in a data unit appears in the prefabricated vocabulary. Each time, the single character or adjacent character combination with the largest number of occurrences is saved as a word until the loop is completed, thereby generating a preprocessed word set. .

[0009] In the present invention, the visual image is mapped to the same embedding space as the task instruction through image coding. Specifically, the visual image Y of the current environment is divided into n non-overlapping image blocks of size P×P, that is, Y={y1,…,y n The image encoder module uses linear embedding to map the image block to the same embedding space R,e as the task instruction. img ∈R n×P×P×c ; Where n represents the number of image blocks, P×P represents the size of each image block, and C represents the number of channels of the image block. img The representation of the image block mapped to the embedding space is called image feature embedding.

[0010] Preferably, a multi-scale feature extraction network FPN is used to divide the image Y.

[0011] In this paper, the image encoder module consists of a ViT module, a Q-Former module, and a multilayer perceptron. The block-by-block image embedding input ViT uses a self-attention mechanism to capture the global relationships between regions within the image block and outputs a feature vector containing the global semantic information of the image. This feature vector is passed as input to the Q-Former module. The Q-Former selects visual features relevant to the task instruction from the global feature vector extracted by ViT using a set of learnable query vectors. The Q-Former output is then input into the multilayer perceptron, which maps it to the same embedding space R as the task instruction.

[0012] As a preference, the large language model in step (2) is the open source large model Llama2. The association in step (2) is specifically as follows: Llama2 embeds image features into e through self-attention mechanism and hierarchical structure. img and task instructions embedded in e text Relationship is associated, and the input [e text ,e img ] to perform multi-layer understanding and generate text action instructions and image feature vectors evoken .

[0013] Preferably, the feature mapping module in step (3) is composed of MLP and transformer. Step (3) is specifically as follows: the image feature vector e generated in step (2) is transformed into voken It is converted into the dimension that matches the transformer input through MLP, and then encoded and decoded through transformer to generate the conditional feature vector h voken .

[0014] Preferably, the transformer consists of a 6-layer encoder and a 6-layer decoder.

[0015] As a preferred embodiment, step (3) is specifically as follows: the image feature vector e generated in step (2) is converted into voken It is converted into the dimension that matches the transformer input through MLP, and then encoded and decoded through a transformer and a learnable decoder feature sequence q to generate the conditional feature vector h voken .

[0016] In the present invention, step (4) is specifically as follows: adding random Gaussian noise to the visual image Y and performing the forward pass according to the conditional feature vector h voken The process of removing random Gaussian noise to generate an image of the environment state.

[0017] The forward addition of random Gaussian noise specifically includes: subjecting the image Y to a standard forward Markov process of a diffusion model, adding noise to the image Y, and obtaining a noisy image.

[0018] The reverse removal of random Gaussian noise is specifically as follows: voken Under the guidance of , a neural network decoder is used to iteratively remove random noise from the noisy image to obtain the environment state image.

[0019] In the present invention, the generation of all states of the completed task in step (5) means that the similarity between the two generated environmental state images reaches 95% or more, preferably 98% or more. Alternatively, the generation of all states of the completed task in step (5) means that the number of iterations is greater than or equal to 5, preferably greater than or equal to 10.

[0020] According to a second embodiment provided by the present invention, use of the method according to the first embodiment is provided.

[0021] The method described in the first embodiment is used in autonomous driving scenarios, smart homes, smart manufacturing scenarios, smart robots or VR gaming fields.

[0022] In the present invention, a task instruction refers to an operational instruction or command issued by an operator to an intelligent agent. A visual image of the current environment refers to a photograph of the agent's location and surroundings captured by the agent's own camera. The agent itself possesses the capabilities of capturing and receiving instructions. Based on the task instruction and the visual image of the current environment, the agent processes the task instruction using the task planning method provided by the present invention, obtaining a task planning diagram that combines the task instruction and the current environment. This task planning diagram is then input into the agent's actuator, which then performs the instructed task according to the resulting task planning diagram.

[0023] Throughout the development of this technology, intelligent agents' ability to effectively make decisions and adapt to dynamic changes in complex environments has been a key research direction. However, existing technologies in this area have significant limitations, primarily due to a lack of direct perception of the physical world and insufficient reasoning capabilities. This makes it difficult for intelligent agents to effectively make decisions and adapt to dynamic changes in complex environments. To address these issues, this paper proposes a task planning method based on an embodied intelligent world model based on visual reasoning.

[0024] This paper proposes a method for embodied intelligent world models based on visual reasoning. This method combines the language processing capabilities of a large language model with advanced visual perception technology. This method enables intelligent agents to use a world model to pre-infer environmental states and simulate possible actions with minimal visual information. This allows them to make effective decisions and quickly adapt to changing environmental conditions in complex environments, significantly improving the efficiency and success rate of task execution in complex scenarios.

[0025] The present invention receives task instructions and visual images, encodes them into the same embedding space respectively, and associates image features and task instructions through a large language model to generate text action instructions and image feature vectors. Then, an environmental state image is generated through a feature mapping module and a diffusion model, and it is used as a new input for iteration until the task is completed. By encoding task instructions and visual images into the same embedding space and associating them using a large language model, the task instructions can be effectively combined with the environmental visual information to generate accurate action instructions and environmental state images. The use of the feature mapping module and the diffusion model enables the system to generate accurate environmental state images in a dynamically changing environment, thereby helping the intelligent agent to make effective decisions and adapt in complex environments.

[0026] For example, in autonomous driving scenarios, intelligent agents must make decisions based on visual information about the road environment and driving task instructions. By encoding the driving task instructions and the visual image of the road environment into the same embedding space and associating them through a large language model, accurate driving maneuver instructions and road state images can be generated. The road state images generated by the feature mapping module and diffusion model enable the intelligent agent to make effective decisions and adapt to dynamically changing road environments.

[0027] Compared to existing technologies, this invention more accurately combines task instructions with environmental visual information by encoding them into the same embedding space and linking them using a large language model. This allows for more precise action instructions and environmental state images to be generated. The use of a feature mapping module and diffusion model enables the system to generate accurate environmental state images in dynamically changing environments, significantly improving the efficiency and success rate of task execution by intelligent agents in complex scenarios.

[0028] Furthermore, the present application also proposes to segment the task instructions to form a word set, and map it to the embedding space. Specifically, the task instructions are segmented, and the segmented task instructions are represented as multiple characters, and processed into a word set according to a pre-made vocabulary. Then, the word set is mapped to the embedding space R to generate the task instruction embedding. This technical feature solves the problem of how to map the task instructions to the embedding space by converting the task instructions into an embedding vector, so that the task instructions can be processed and associated with the visual image features in the same embedding space.

[0029] In the present invention, existing word segmentation algorithms can be used to segment task instructions, such as rule-based word segmentation algorithms, statistical model word segmentation algorithms, or deep learning word segmentation algorithms. The pre-built word list can be based on an existing language model, such as the pre-built word list of Llama2. The mapping process to the embedding space can be implemented using a trained embedding model, such as Word2Vec, GloVe, or BERT.

[0030] In the present invention, the word segmentation method is to segment the task instruction to obtain the characters of the task instruction, and any word segmentation model and method in the prior art can be used, such as Hidden Markov Model (HMM) and Conditional Random Field (CRF).

[0031] In the present invention, wordpiece is an existing mature technology. The present invention uses the wordpiece module to segment the task instructions, and then processes the segmented task instructions to obtain a word set.

[0032] During implementation, Llama2's pre-built word lists can be optimized in a variety of ways. For example, increasing the size of the pre-built word list can improve word segmentation accuracy and efficiency. Furthermore, the BPE algorithm parameters, such as the number of iterations and statistical frequency, can be adjusted to optimize word segmentation performance. Furthermore, other word segmentation algorithms, such as WordPiece or SentencePiece, can be combined to improve word segmentation accuracy and robustness.

[0033] In this paper, by dividing the visual image into multiple non-overlapping image blocks and mapping them into an embedding space using linear embedding, we ensure that image features and task instructions can be processed in the same space. The use of a multi-scale feature extraction network (FPN) further enhances the extraction of image features, making the mapping process more accurate and efficient.

[0034] Preferably, the image encoder module can be implemented in a variety of ways, for example, using a convolutional neural network (CNN) for image block feature extraction and embedding, or using a visual transformer (ViT) for global feature capture.

[0035] In the present invention, the image encoder module is a module that the open source large model Llama2 itself has, and is composed of a ViT module, a Q-Former module, and a multi-layer perceptron. The ViT module captures the global relationship between each region in the image block through the self-attention mechanism, thereby realizing the extraction of global semantic information of the image. The Q-Former module can select visual features related to the task instructions from the global feature vector extracted by ViT through a set of learnable query vectors, thereby improving the correlation between the image features and the task instructions. The multi-layer perceptron maps the output of the Q-Former to the same embedding space as the task instructions, so that the image features can be processed and analyzed in the same embedding space as the task instructions. This technical solution not only solves the problem of how to encode and map visual images to the same embedding space as the task instructions, but also improves the accuracy and efficiency of the large language model in processing image features and task instructions.

[0036] In the present invention, the open source big model Llama2 is the open source big model officially released by Meta AI.

[0037] Llama2 uses a self-attention mechanism and a hierarchical structure to associate image features with eimg and task instructions with etext. Specifically, Llama2 first encodes the input image features and task instructions through a self-attention mechanism to generate a preliminary embedding representation. These embedded representations are then further processed through a hierarchical structure, extracting higher-level features layer by layer, ultimately generating text action instructions and image feature vectors called evoken. This multi-layered understanding approach enables Llama2 to better capture the complex relationships and context in the input information, thereby improving the agent's perception and reasoning capabilities in the physical environment. By adopting Llama2, the present application is able to achieve an improvement in the direct perception and reasoning capabilities of the physical world in a large language model. Llama2's self-attention mechanism and hierarchical structure enable it to effectively associate image features with task instructions, thereby generating accurate text action instructions and image feature vectors. This approach not only improves the agent's ability to perceive the environment, but also enhances its reasoning and decision-making capabilities in complex scenarios. As a result, the present application provides an effective technical means to address the problem of large language models' lack of direct perception and reasoning capabilities of the physical world.

[0038] In this invention, the feature mapping module, composed of an MLP and a transformer, works together to convert image feature vectors into conditional feature vectors, thereby playing a role in the process of generating environmental state images. Through this scheme, the image feature vector evoken is first converted into a dimension that matches the transformer input through the MLP, and then encoded and decoded through the transformer to generate the conditional feature vector hvoken. This method solves the technical problem of how to convert image feature vectors into conditional feature vectors to generate environmental state images, ensuring that the image feature vectors can be effectively used in the subsequent environmental state image generation process.

[0039] Specifically, MLP can use a multi-layer perceptron structure to transform the input feature vector into a dimension that conforms to the transformer input through multi-layer linear transformation and activation function. The encoder part of the transformer encodes the input feature vector through a self-attention mechanism and a feedforward neural network to generate an intermediate representation; the decoder part decodes the intermediate representation through a similar mechanism to generate the final conditional feature vector h voken .

[0040] In the present invention, the visual image Y is subjected to the process of adding random Gaussian noise and removing random Gaussian noise. vokenIt plays a guiding and controlling role in the process of generating the environmental state image. The combination of denoising in the forward process and denoising in the backward process enables the final generated environmental state image to accurately reflect the environmental state described by the conditional eigenvector hvoken, thus solving the technical problem of how to generate the environmental state image through the conditional eigenvector hvoken.

[0041] Preferably, the forward random Gaussian noise addition process can employ a standard forward Markov process. By controlling and directing the noise, the noisy image can better preserve the original image's information. The reverse random Gaussian noise removal process can be performed through multiple iterations, gradually removing the noise and ultimately generating a clear image of the environmental state. Furthermore, different neural network decoder structures can be employed to improve the denoising effect and efficiency.

[0042] In the present invention, the diffusion models include a forward diffusion process and an inverse generation process. The forward diffusion process involves gradually adding noise to the data to convert it into pure noise. This process is a Markov chain, with a small amount of Gaussian noise added at each step. Starting from the true data distribution, noise is gradually added over multiple time steps, ultimately making the data distribution close to a standard Gaussian distribution. The inverse generation process involves learning to gradually remove noise from the noise to restore the original data. This process is also a Markov chain, but in the opposite direction, with gradual denoising. Starting from standard Gaussian noise, through a learned inverse diffusion process, noise is gradually removed to generate new data consistent with the true data distribution.

[0043] In the present invention, by setting the standard of image similarity or number of iterations, the accuracy of the generated environmental state image and the reliability of task completion can be effectively evaluated and guaranteed, thereby improving the efficiency and success rate of the intelligent agent's task execution in a complex environment.

[0044] Preferably, the similarity between two images is calculated using the existing technology, and the present invention uses the Structural Similarity (SSIM) index. Other image similarity algorithms, such as the Mean Squared Error (MSE) or Peak Signal-to-Noise Ratio (PSNR), may also be used. The number of iterations can be adjusted based on the complexity of the task and the dynamic changes in the environment to ensure that the task is completed within a reasonable time. In addition, an adaptive algorithm can be introduced to dynamically adjust the number of iterations based on the real-time execution of the task, further improving the efficiency and accuracy of task completion.

[0045] The method provided by the present invention combines the language processing capabilities of a large language model with advanced visual perception technology, enabling intelligent agents to pre-infer environmental states and simulate possible actions through a world model with a small amount of environmental visual information, thereby improving the efficiency of task execution and the success rate of task completion of intelligent agents in complex scenarios. In autonomous driving scenarios, intelligent agents can obtain visual images of the vehicle's surroundings and combine them with the large language model's understanding of driving instructions to effectively plan paths and avoid obstacles. In the field of smart homes, intelligent agents can use visual perception of the home environment and understand user instructions to achieve automatic control of smart appliances and home security monitoring. In intelligent manufacturing scenarios, intelligent agents can use visual monitoring of production lines and combine task instructions to schedule equipment and optimize production processes. In the field of intelligent robotics, intelligent agents can use visual perception of environmental changes and combine task instructions to perform autonomous navigation and task execution. In the field of VR gaming, intelligent agents can use visual perception of the game environment and combine task instructions to control characters and complete tasks.

[0046] The method provided by the present invention significantly improves the decision-making and adaptability of intelligent agents in complex environments by applying the task planning method of the embodied intelligent world model based on visual reasoning to multiple real-world scenarios, and solves the technical problem in the existing technology that intelligent agents have difficulty in making effective decisions and adapting to dynamic changes in complex environments.

[0047] Compared with the prior art, the technical solution provided by the present invention has the following beneficial technical effects: 1. By combining the language processing capabilities of a large language model with advanced visual perception technology, the intelligent agent can use the world model to pre-infer the environmental state and simulate possible actions with a small amount of environmental visual information. This has the advantages of improving the decision-making ability and adaptability of the intelligent agent in complex environments, and significantly improving the efficiency and success rate of task execution.

[0048] 2. It uses innovative image Voken technology to achieve deep integration of text and image features.

[0049] 3. A visual environment state generation method based on the diffusion model is introduced, which can simulate and predict environmental changes.

[0050] 4. Adopt an iterative task planning method to enable the intelligent agent to adapt to the dynamically changing environment.

[0051] 5. By constructing a world model, the efficiency and success rate of intelligent agents in complex scenarios are improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 A schematic diagram of an embodied intelligent world model method for visual reasoning according to the present invention; Figure 2 This is a schematic diagram of the training of the embodied intelligent world model of the present invention; Figure 3 This is a schematic diagram of the structure of the embodied intelligent world model of the present invention. DETAILED DESCRIPTION

[0053] The technical solutions in this application will be clearly and completely described below in conjunction with the drawings in this application. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. The components of this application generally described and shown in the drawings herein can be arranged and designed in a variety of different configurations. Therefore, the following detailed description of the embodiments of the application provided in the drawings is not intended to limit the scope of the claimed application, but merely represents selected embodiments of the application.

[0054] Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making any creative work shall fall within the scope of protection of the present application. Example 1

[0055] A task planning method based on an embodied intelligent world model based on visual reasoning, the method comprising the following steps: (1) Receive task instructions and obtain the visual image of the current environment; map the task instructions to the embedding space through text encoding to obtain the task instruction embedding e text ; The visual image is mapped to the same embedding space as the task instruction through image encoding to obtain the image feature embedding e img ; (2) Embed task instructions into e text Embedded with image features img Put together, we get [e text ,e img ]; [e text ,e img ] Input the large language model, which embeds the image features into e img and task instructions embedded in e text Associate and generate text action instructions and image feature vector e voken ; (3) Image feature vector e voken Generate the conditional feature vector h through feature mapping module mapping voken ; (4) Using the diffusion model according to the conditional eigenvector h voken Generate an image of the environment state; (5) The text action instructions generated in step (2) are used as the new input of the task instructions in step (1), and the environment state image generated in step (4) is used as the new input of the visual image in step (1), and the process is repeated until all states for completing the task are generated. Example 2

[0056] Repeat Example 1, except that the text encoding in step (1) includes segmenting the task instructions into words to form a word set and mapping it to the embedding space; specifically: The task instruction X is segmented, and the task instruction after segmentation is represented as X={x1,…,x m}, where x represents the characters after the task instruction is segmented, and m represents the number of characters after the task instruction is segmented; X is processed into a word set according to the pre-made word list , map the processed word set to the embedding space R,e text ∈R d ; Where: d represents the dimension of the embedding space R, that is, the dimension of the vector mapped from the processed word set to the embedding space.

[0057] The prefabricated word list adopts the prefabricated word list of Llama2. That is, the task instruction X is input into the wordpiece module. The BPE algorithm in wordpiece first divides the task instruction into characters one by one, and then counts the number of times a single character or a combination of adjacent characters in a statistical unit appears in the prefabricated word list. Each time, the single character or adjacent character combination with the most occurrences is saved as a word until the loop is completed, thereby generating a preprocessed word set. . Example 3

[0058] Repeat Example 2, except that the visual image is mapped to the same embedding space as the task instruction through image coding. Specifically, the multi-scale feature extraction network FPN is used to divide the visual image Y of the current environment into n non-overlapping image blocks of size P×P, that is, Y={y1,…,y n The image encoder module uses linear embedding to map the image block to the same embedding space R,e as the task instruction. img ∈R n×P×P×c ; Where n represents the number of image blocks, P×P represents the size of each image block, and C represents the number of image block channels; The image encoder module consists of a ViT module, a Q-Former module, and a multi-layer perceptron. The block-by-block image embedding input is ViT, which uses a self-attention mechanism to capture the global relationship between regions in the image block and outputs a feature vector containing the global semantic information of the image. The feature vector output by ViT is passed as input to the Q-Former module. Q-Former selects visual features related to the task instructions from the global feature vector extracted by ViT through a set of learnable query vectors. The output of Q-Former is input into the multi-layer perceptron and mapped to the same embedding space R as the task instruction. Example 4

[0059] Repeat Example 3, except that the large language model in step (2) is the open source large model Llama2; the association in step (2) is specifically: Llama2 embeds image features into e through the self-attention mechanism and hierarchical structure img and task instructions embedded in e text Relationship is associated, and the input [e text ,e img ] to perform multi-layer understanding and generate text action instructions and image feature vectors e voken .

[0060] The image e voken It is a special character embedding, so the word table of Llama2 is expanded by W i ∈V∪V img , aligning the text with the image Voken so as to generate the intermediate value of the conditional feature vector. Its loss function can be formalized as follows: .

[0061] The feature mapping module in step (3) is composed of MLP and transformer. Step (3) is specifically as follows: the image feature vector e generated in step (2) is transformed into voken It is converted into the dimension that matches the transformer input through MLP, and then encoded and decoded through transformer to generate the conditional feature vector h voken ; The transformer consists of a 6-layer encoder and a 6-layer decoder.

[0062] The features generated by the mapping are formalized as follows: .

[0063] Example 5 Repeat Example 4, except that step (4) is specifically as follows: forward adding random Gaussian noise and removing random Gaussian noise to the visual image Y to generate an environment state image; The forward addition of random Gaussian noise is specifically as follows: the image Y is put into the standard forward Markov process of the diffusion model, and random Gaussian noise is gradually injected into the image Y until the random Gaussian noise is added to the isotropic Gaussian distribution to obtain the noisy image The reverse removal of random Gaussian noise is specifically as follows: voken Under the guidance of , the neural network decoder is used to iteratively remove random noise from the noisy image to obtain the environmental state image. Enter the reverse Gaussian noise removal process, using the conditional feature vector and the neural network decoder, that is, , fitting each distribution in the process of gradually removing Gaussian noise until the image is restored.

[0064] In this embodiment, the forward process is to Z t Add random Gaussian noise to generate Z t+1 The process of generating Z t+1 Depends only on Z t , so the forward process is a Markov process, and the forward process does not contain any trainable parameters, and the noise level is determined by the variance β t ∈(0,1)determines. The forward process can be formalized as: , Get Z by reparameterization technique t ,in , , t≤0, then: ; The Markov process can be formalized as: .

[0065] For the reverse process of removing random Gaussian noise, if the given condition Z is known t Can determine Z t-1 The probability distribution of If , then we can iteratively sample random noise to generate an image or sentence. is unknown, you can use Considering that the noise added in each step is relatively small, we assume It obeys Gaussian distribution and can be modeled using neural networks. Therefore, the reverse process of conditional denoising can be formalized as: ; in: and are the parameters whose mean and variance need to be predicted respectively.

[0066] Given the posterior conditional probability distribution in the forward process , the ultimate goal is to minimize the model The mean square error loss between the KL divergence of the posterior conditional probability distribution q. The loss function of its conditional denoising is formalized as follows: .

[0067] Therefore, h generated in step S3 voken The image Y enters the standard forward Markov process of the diffusion model, and then enters the standard forward noise process, adding relatively small noise at each step , until the final state Z is obtained t .

[0068] The reverse process is specifically from the final state Z t , conditional vector h voken Under the guidance of the process of recovering Y by iteratively removing random noise, a neural network is used To fit.

[0069] In this embodiment, the neural network decoder , using pre-trained The denoising loss function is formalized as follows: .

[0070] The U_Net network architecture includes an encoder (downsampling path) and a decoder (upsampling path), which consists of a 4-layer encoder, a 1-layer bottleneck layer, and a 4-layer decoder.

[0071] Specifically, in this embodiment, the random sampling from the standard Gaussian distribution , and h generated by the task instruction and image Y through steps (2) and (3) voken , enter the value in step (4) Decode and generate reasoning actions to solve the task.

[0072] Example 6 Repeat Example 5, except that in step (5), randomly sample from the standard Gaussian distribution , and h generated by the task instruction and image Y through steps (2) and (3) voken , enter the value in step (4) Decode and generate reasoning actions to solve the task.

[0073] The generation of all states of the completed task means that the similarity between the two generated environmental state images reaches more than 95%. Example 7

[0074] Repeat Example 5, except that in step (5), all states of generating completed tasks refer to: the number of iterations is 5, and then the task ends automatically. Example 8

[0075] A task planning method for an autonomous driving scenario, wherein the task planning method adopts the technical solution described in Example 6. Example 9

[0076] A task planning method for a smart home scenario, which adopts the technical solution described in Example 6. Example 10

[0077] A task planning method for an intelligent manufacturing scenario, wherein the task planning method adopts the technical solution described in Example 6. Example 11

[0078] A task planning method for an intelligent robot, wherein the task planning method adopts the technical solution described in Example 6. Example 12

[0079] A task planning method for a VR game, wherein the task planning method adopts the technical solution described in Example 6.

Claims

1. A task planning method based on an embodied intelligent world model based on visual reasoning, the method comprising the following steps: (1) Receive task instructions and obtain visual images of the current environment; The task instruction is mapped to the embedding space through text encoding to obtain the task instruction embedding e text ; The visual image is mapped to the same embedding space as the task instruction through image encoding to obtain the image feature embedding e img ; (2) Embedding task instructions into e text Embedded with image features img Put together, we get [e text ,e img ]; text ,e img ] Input the large language model, which embeds the image features into e img and task instructions embedded in e text Associating and generating text action instructions and image feature vectors e voken ; (3) Image feature vector e voken Generate conditional feature vector h through feature mapping module mapping voken ; (4) Using the diffusion model according to the conditional feature vector h voken Generate an environment status image; (5) Use the text action instructions generated in step (2) as the new input of the task instructions in step (1), and use the environment state image generated in step (4) as the new input of the visual image in step (1), and iterate until all states for completing the task are generated.

2. The task planning method according to claim 1, characterized in that: The text encoding in step (1) includes segmenting the task instructions into words to form a word set and mapping it to the embedding space; specifically: The task instruction X is segmented and the task instruction after segmentation is expressed as X={x1,…,x m }, where x represents the characters after the task instruction is segmented, and m represents the number of characters after the task instruction is segmented; X is processed into a word set according to the prefabricated word list , map the processed word set to the embedding space R,e text ∈R d ; Where: d represents the dimension of the embedding space R, that is, the dimension of the vector mapped from the processed word set to the embedding space.

3. The task planning method according to claim 2, characterized in that: The prefabricated word list adopts the prefabricated word list of Llama2, that is, the task instruction X is input into the wordpiece module, and the BPE algorithm in the wordpiece first divides the task instruction into characters one by one, and then counts the number of times a single character or a combination of adjacent characters in a statistical unit appears in the prefabricated word list, and each time the single character or the combination of adjacent characters with the largest number of occurrences is saved as a word until the cycle is completed, thereby generating a preprocessed word set .

4. The task planning method according to claim 1, characterized in that: The visual image is mapped to the same embedding space as the task instruction through image encoding. Specifically, the visual image Y of the current environment is divided into n non-overlapping image blocks of size P×P, that is, Y={y1,…,y n }; The image encoder module uses linear embedding to map image blocks to the same embedding space R as the task instruction, e img ∈R n×P×P×c ; Where n represents the number of image blocks, P×P represents the size of each image block, and C represents the number of channels of the image block; Preferably, a multi-scale feature extraction network FPN is used to divide the image Y.

5. The task planning method according to claim 4, characterized in that: The image encoder module consists of a ViT module, a Q-Former module and a multi-layer perceptron; the divided image blocks are embedded into the input ViT, which uses a self-attention mechanism to capture the global relationship between the regions in the image block and outputs a feature vector containing the global semantic information of the image; the feature vector output by ViT is passed as input to the Q-Former module, which selects visual features related to the task instructions from the global feature vector extracted by ViT through a set of learnable query vectors; the output of Q-Former is input into the multi-layer perceptron to map it to the same embedding space R as the task instruction.

6. The task planning method according to any one of claims 1 to 5, characterized in that: The large language model in step (2) is the open source large model Llama2; the association in step (2) is specifically as follows: Llama2 embeds image features into e through the self-attention mechanism and hierarchical structure. img and task instructions embedded in e text Relationship is associated with the input [e text ,e img ] to perform multi-layer understanding and generate text action instructions and image feature vectors e voken .

7. The task planning method according to claim 1, characterized in that: The feature mapping module in step (3) is composed of MLP and transformer. Step (3) is specifically as follows: the image feature vector e generated in step (2) is transformed into voken The MLP is used to transform the shape into the dimension that matches the transformer input, and then the transformer is used to encode and decode the shape to generate the conditional feature vector h. voken ; Preferably, the transformer consists of a 6-layer encoder and a 6-layer decoder.

8. The task planning method according to claim 1, characterized in that: Step (4) is as follows: add random Gaussian noise to the visual image Y and add the conditional feature vector h voken Under the guidance of , the random Gaussian noise process is removed to generate the environment state image; The forward addition of random Gaussian noise is specifically as follows: the image Y is subjected to a standard forward Markov process of a diffusion model, and the image Y is subjected to noise addition to obtain a noisy image; The reverse removal of random Gaussian noise is specifically as follows: voken Under the guidance of , a neural network decoder is used to iteratively remove random noise from the noisy image to obtain the environment state image.

9. The task planning method according to claim 1, characterized in that: The generation of all states of the completed task in step (5) means that the similarity between the two generated environmental state images reaches 95% or more, preferably 98% or more; or the number of iterations is greater than or equal to 5 times, preferably greater than or equal to 10 times.

10. The task planning method according to any one of claims 1 to 9, wherein the method is used in autonomous driving scenarios, smart homes, smart manufacturing scenarios, smart robots or VR game fields.

Images