Scene consistency image instruction generation method based on task decomposition

Abstract

The present application relates to a task decomposition-based scene consistency image instruction generation method, and the generation method comprises the following steps: step S1, image analysis, object recognition, position and state recognition are performed; step S2, instruction understanding is performed, and set key information is extracted; step S3, feature extraction of a pre-scene image in a latent space is realized through a VAE encoder; step S4, a task decomposition generation step sequence is generated, and step-by-step description is performed; step S5, based on the step sequence, an instruction weight matrix with front and rear correlations is generated; the instruction weight matrix with front and rear correlations is designed based on the step sequence of step S4; step S6, based on the image reference of the pre-scene and the instruction weight, a visual instruction of the current scene is generated; the present application is reasonable in design, compact in structure and convenient to use.

Description

Technical Field

[0001] This invention relates to a method for generating scene-consistent image instructions based on task decomposition, applicable to smart home scenarios. Background Technology

[0002] In computer processing, it is generally necessary to segment the collected natural scene images at the pixel level to separate the different regions or objects, so as to deeply understand and recognize the image content.

[0003] In image segmentation within a smart home scenario, the floor and interior walls are used to define basic data elements. By processing and segmenting the collected data, the algorithm can be better applied, effectively improving recognition accuracy and processing speed, thereby meeting the needs of real-time applications.

[0004] However, existing technologies have many problems in the fields of multimodal interaction and image generation. For example, they lack intuitiveness: traditional video generation methods may not provide clear enough step guidance, and users may find it difficult to identify each key step from a continuous video stream; another problem is the incoherence of generated image content: when generating images for a multi-step scene, existing technologies often fail to maintain the consistency of content between images, resulting in visually and semantically incoherent image sequences that affect the user experience.

[0005] To address these issues, this invention proposes solutions to the following three main technical problems in existing technologies: First, there is a need to improve intuitiveness: In smart home scenarios, users require clear and intuitive visual guidance to perform complex tasks. Existing video generation methods may not provide sufficiently clear step-by-step guidance, making it difficult for users to identify each key step from a continuous video stream. This invention aims to provide more intuitive visual guidance by generating a consistent image sequence that breaks down a series of key steps. Second, there is a need to enhance content coherence: When generating images for multi-step scenarios, existing technologies often fail to maintain content consistency between images, resulting in visually and semantically incoherent image sequences that negatively impact user experience. This invention enhances content coherence by ensuring visual and semantic consistency in the image sequence. Summary of the Invention

[0006] Existing technologies in smart home scenarios rely on traditional video generation and image processing techniques. These methods often fail to provide sufficiently clear step-by-step guidance or maintain content consistency between images when generating multimodal interactions and image sequences. This makes it difficult for users to identify each key step from a continuous video stream when performing tasks, and the generated image sequences are visually and semantically incoherent, impacting the user experience.

[0007] In general, the technical problem this invention aims to solve is to provide a scene-consistent image instruction generation method based on task decomposition. By providing a series of clear, coherent, and highly interactive image sequences, it addresses key issues in existing technologies for multimodal interaction and image generation in smart home scenarios. The goal is to provide a more intuitive, coherent, and highly interactive solution for predicting smart home user behavior.

[0008] To solve the above problems, the technical solution adopted by the present invention is as follows:

[0009] A method for generating scene consistency image instructions based on task decomposition, the method comprising the following steps;

[0010] Step S1: Perform image analysis, object recognition, and position and state recognition;

[0011] Step S2: Perform instruction comprehension and extract key setting information;

[0012] Step S3: Feature extraction of the preceding scene image in the latent space is achieved through the VAE encoder;

[0013] Step S4: Decompose the task into a step sequence and describe each step.

[0014] Step S5: Based on the step sequence, generate an instruction weight matrix with sequential correlation; using the step sequence of step S4, design the instruction weight matrix with sequential correlation:

[0015] Step S6: Generate visual instructions for the current scene based on the image reference and instruction weights of the preceding scene.

[0016] When generating visual instructions for the current scene, the features of the preceding images and the associated prompts are referenced. The visual instructions for the current scene are generated by the combined effect of the features of the preceding images and the associated prompts.

[0017] As a further improvement to the above technical solution:

[0018] In step S1, firstly, a smart home scene is set up; then, data is collected, a large model is built, and the model's capabilities are applied or invoked; secondly, the user uploads image information to the multimodal large model, and the large model analyzes and processes the currently uploaded image information to obtain data information.

[0019] The extraction of data information involves extracting the feature information of the image, which then serves as the basis for generating image instructions.

[0020] In step S1, the data information is encoded by the VAE encoder and then extracted as the feature vector of the image in the latent space.

[0021] In step S2, the following steps are performed;

[0022] S2.1, Issue user command:

[0023] S2.2, Obtain system prompt words;

[0024] S2.3, Extract key information:

[0025] S2.4, The model is confirmed and feedback is given to the user.

[0026] In step S3, for the data information uploaded in step S1, firstly, after initialization, the preceding scene image is used as a reference image to input into the large model to generate a new visual instruction image, thereby generating a scene-consistent image, which serves as a visual instruction to guide the user's actual operation.

[0027] Then, by feeding the entire preceding image as a reference image into the VAE encoder, the consistency of the overall image is maintained;

[0028] Secondly, when generating the visual instruction image of the current scene from the reference instruction prompt in step S2, the preceding scene image exists in the latent space as a feature vector through the VAE encoder. The preceding scene image is denoised by a sampler. The similarity between the newly generated image and the reference image is adjusted within the range of 0.01 to 1.

[0029] In step S4, a step sequence is generated based on the steps obtained in step S2.

[0030] In step S5, firstly, based on the reference image uploaded by the user in step S1, when generating the first visual instruction image, the weight parameters in the first row of the matrix are used to add correlation to the instruction, and the prompt words are obtained after processing.

[0031] Then, when generating the second visual instruction image, the weight parameters in the second row of the matrix are used to add correlation to the instructions in the visual instruction image, and the prompt words are obtained after processing.

[0032] Similarly, when generating the Nth visual instruction image, the weight parameter of the corresponding Nth row in the matrix is ​​used to add correlation to the instruction, and the prompt word is obtained after processing.

[0033] The method employs a variational autoencoder with system prompts to generate prompt words.

[0034] This invention improves the design of an instruction weight matrix based on task distribution decomposition, which ensures the consistency and correlation of images before and after the generation of visual instructions for scene images; and achieves generation control of the consistency of image correlation within the latent space based on noise processing of the latent space.

[0035] This invention provides more intuitive visual guidance by generating a consistent image sequence that breaks down a series of key steps. It effectively captures subtle changes and long-term dependencies in user behavior, improving prediction accuracy and predicting the user's next action throughout the entire behavioral sequence. This allows for a better understanding and prediction of complex patterns in user behavior. Furthermore, by ensuring visual and semantic consistency in the image sequence, it enhances content coherence, improves data processing efficiency, and increases the model's ability to handle large-scale datasets.

[0036] In summary, by addressing key issues in the prior art, this invention offers the following advantages: It improves the intuitiveness and accuracy of task execution, enabling users to complete complex tasks more effectively. It enhances user experience and interactivity, providing smoother and more natural visual guidance. It improves data processing efficiency and model scalability, making the technology applicable to a wider range of application scenarios. Attached Figure Description

[0037] Figure 1 This is a schematic diagram of the original smart home scene visual instruction generation scheme.

[0038] Figure 2 This is a schematic diagram of the improved processing flow of the present invention.

[0039] Figure 3 This is an improved flowchart of the present invention.

[0040] Figure 4 This is a schematic diagram of image analysis according to the present invention.

[0041] Figure 5 The image of this invention is encoded into a latent space noise map by VAE.

[0042] Figure 6 This is a schematic diagram of the visual instructions of the present invention.

[0043] Figure 7 This is a schematic diagram of the instruction weight matrix of the present invention.

[0044] Figure 8 This is a schematic diagram of the visual instructions for generating the current scene based on the preceding scene and instructions of the present invention. Detailed Implementation

[0045] Example 1, as Figure 1The previous solution acquired images and voice commands from the user scenario and performed multimodal large-scale model analysis on these images and text. Image information analysis identified objects, locations, and states, which were then used for visual feedback, such as image annotation or animation video generation. Key command information was extracted from text commands to guide user operation steps or provide operational suggestions, and even enable direct interaction with home appliances.

[0046] Taking cooking pumpkin porridge in the kitchen as an example,

[0047] [Input]: Visual input: The user uploads a photo of a kitchen, which includes items needed to cook porridge, such as a stove, pots and pans, and pumpkin.

[0048] Text instructions: Users give instructions, such as "Please cook pumpkin porridge on the stove in the kitchen and set the time to 30 minutes".

[0049] [Processing Procedure]: Image Analysis: The multimodal model first analyzes the uploaded kitchen photos to identify the position and state of objects such as the stove, pots and pans.

[0050] Instruction understanding: The model understands the user's text instructions and extracts the two key pieces of information: "cook pumpkin porridge" and "30 minutes".

[0051] Task planning: Based on the identified objects and understood instructions, the model plans the steps for cooking pumpkin porridge, such as first cutting the pumpkin into chunks, then putting it in a pot with water, and finally cooking it on the stove.

[0052] [Output]: Step description: The LLaVA model generates a series of step descriptions, such as "cut the pumpkin into chunks", "put the cut pumpkin into the pot", "add an appropriate amount of water", "put the pot on the stove", "set the stove temperature and start timing for 30 minutes".

[0053] Visual feedback: The model can also generate visual feedback, such as marking the location and status of the pumpkin porridge cooking on uploaded kitchen photos, or generating an animation of the porridge cooking process.

[0054] If certain items are missing in the kitchen scene, the model can provide suggestions based on empirical data, such as "You need to buy pumpkin and pots to cook porridge".

[0055] Example 2, as Figure 2-8 The process of this embodiment is described as follows;

[0056] Step S1: Perform image analysis, object recognition, and position and state recognition;

[0057] First, a smart home scenario is set up. Then, data is collected, using methods such as voice capture, camera images, or video capture. Next, the user uploads image information to a multimodal large model, which analyzes and processes the uploaded image information to obtain data, such as identifying the objects in the image, their coordinates, and the relationships between them. This data extraction essentially extracts the image's feature information, which serves as the foundation for generating subsequent image commands.

[0058] In this process, this information is encoded by a VAE encoder and extracted as feature vectors in the latent space. To better illustrate these image features, we provide the following... Figure 4 .

[0059] Step S2 involves understanding the instructions and extracting key information; this process can be illustrated with a specific example:

[0060] S2.1, issue the user instruction: "I want to cook pumpkin porridge. I need to prepare pumpkin, glutinous rice and water, and then follow the specific ratio and steps."

[0061] S2.2, Obtain system prompt words;

[0062] For example, the system prompt might say: Extract the ingredients, proportions, and cooking steps from the instructions.

[0063] S2.3, Extract key information:

[0064] For example, key information: ingredients: pumpkin, glutinous rice, water.

[0065] Ratio: Requires a specific ratio (the system can provide a default suggestion if the user does not provide a specific ratio). Cooking steps: Follow specific steps (the system can provide a default suggestion if the user does not provide specific steps).

[0066] S2.4, Model confirmation and feedback to the user:

[0067] For example, the model confirms and provides feedback to the user: "Okay, I will guide you on how to cook pumpkin porridge. You will need to prepare pumpkin, glutinous rice, and water. Usually, the ratio of pumpkin to glutinous rice is about 1:1, and the amount of water is adjusted according to the hardness of the pumpkin. The cooking steps are as follows: 1) Peel and cut the pumpkin into chunks, put them in a pot, and add water; 2) The pumpkin chunks in the pot begin to soften; 3) Cook until the pumpkin porridge is thick."

[0068] Step S3: Feature extraction of the preceding scene image in the latent space is achieved through the VAE encoder;

[0069] First, each time a new visual instruction image is generated, the preceding scene image is used as a reference image and input into the model to generate a scene-consistent image as a visual instruction to guide the user's actual operation.

[0070] Then, this is achieved by feeding the entire preceding image as a reference image into the VAE encoder, which maximizes the consistency of the overall image. For example... Figure 5 .

[0071] Secondly, when generating the visual instruction image for the current scene based on the reference instruction prompt, the preceding scene image exists in the latent space as a feature vector through a VAE encoder. This invention performs noise reduction processing on this preceding scene image using a sampler. The example parameter given here is 0.8, and this value can be adjusted within the range of (0.01-1). Taking 0.8 as an example, it can be understood that the similarity between the newly generated image and the reference image is 80%. Figure 6 .

[0072] Step S4: Decompose the task into a step sequence and describe each step.

[0073] Through step S2, for example, the initial steps of cooking pumpkin porridge can be broken down into the following three-step sequence:

[0074] First, peel and cut the pumpkin into chunks, put it in a pot, and add water;

[0075] Then, the pumpkin chunks in the pot began to soften;

[0076] Next, cook until the pumpkin porridge is thick.

[0077] Step S5: Based on the step sequence, generate an instruction weight matrix with sequential correlation.

[0078] Taking step S4 as an example, the following instruction weight matrix is ​​designed to be associated with each step: Figure 7 .

[0079] In step S5, the present invention explains the weight matrix of this instruction:

[0080] First, based on the reference image uploaded by the user in step S1, when generating the first visual instruction image, the weight parameters in the first row of the matrix are used to add relevance to the instruction. The processed prompt words are shown below:

[0081] For example (peel and cut the pumpkin into chunks, put it in a pot, and add water: 1), (the pumpkin chunks in the pot begin to soften: 0.6), (cook until the pumpkin porridge is thick: 0.6);

[0082] Then, when generating the second visual instruction image, the weight parameters in the second row of the matrix are used to add correlation to the instructions in the visual instruction image. The processed prompts are shown in the example below, where the order of the prompts has changed because the earlier the prompts are, the greater their influence on the image generation.

[0083] (The pumpkin chunks in the pot begin to soften: 1), (Peel and cut the pumpkin into chunks, put them in the pot, and add water: 0.6), (Cook until the pumpkin porridge is thick: 0.6);

[0084] Similarly, when generating the third visual instruction image, the weight parameters in the third row of the matrix are used to add relevance to the instruction. An example of the processed prompt words is shown below:

[0085] (Cook until the pumpkin porridge is thick: 1), (The pumpkin chunks in the pot begin to soften: 0.6), (Peel and cut the pumpkin into chunks, put them in the pot, and add water: 0.6);

[0086] Step S6: Generate visual instructions for the current scene based on the image reference and instruction weights of the preceding scene. When generating visual instructions for the current scene, the features of the preceding image and the associated prompt words are referenced; these two parts work together to generate the visual instructions for the current scene. For example... Figure 8 .

[0087] From Figure 8 In the process, we see that based on the reference image uploaded by the user and the input instructions, and then based on the steps of the task breakdown, visual instructions corresponding to each step are generated.

[0088] Based on the above steps S1-S6, a scenario-consistent image instruction generation scheme based on task decomposition can be realized.

[0089] This invention achieves key step decomposition and consistent image sequence generation: By breaking down complex tasks into key steps and generating a consistent image sequence for each step, this invention provides intuitive visual guidance, thereby enhancing user experience and task execution accuracy. This invention generates an instruction weight matrix with sequential relationships to obtain step-by-step instructions for relational processing; this invention generates visual instructions for the current scene based on image references of preceding scenes and instruction weights.

[0090] This invention achieves visual and semantic coherence of image sequences: This invention ensures that the generated image sequences maintain visual and semantic coherence, which not only improves the quality of the image sequences, but also enhances the accuracy of user behavior prediction and the efficiency of multimodal interaction.

[0091] This invention improves the intuitiveness and accuracy of task execution: by breaking down complex tasks into key steps and generating a consistent image sequence for each step, users can more intuitively understand each step, thereby improving the accuracy and efficiency of task execution. This solves the problem in existing technologies where users find it difficult to identify key steps from continuous video streams.

[0092] This invention enhances user experience and interactivity by ensuring that the generated image sequences remain visually and semantically coherent. Users experience a smoother, more natural interaction when performing tasks because the image sequences provide clear visual guidance and coherent semantic information. This solves the problem of inconsistent image generation content in existing technologies, thus improving the user experience.

[0093] This invention improves data processing efficiency and model scalability: By optimizing data processing and feature extraction techniques, this invention enhances the model's ability to handle large-scale datasets. The model can process and generate image sequences more quickly and efficiently while maintaining high-quality output. This solves the problems of low data processing efficiency and poor model generalization ability in existing technologies, making this invention applicable to a wider range of application scenarios, such as smart homes, education, and healthcare.

[0094] Introduction to basic terminology: Variational Autoencoder (VAE)

[0095] A variational autoencoder (VAE) is a deep generative model that generates new data samples by learning a latent representation of the input data. In a VAE, the latent space refers to the low-dimensional representation learned by the model that captures the main features and variations of the data. This latent space can be viewed as a compressed representation of the data, allowing the model to introduce randomness when generating new samples, thus producing diversity and novelty.

[0096] VAE consists of two main parts: an encoder and a decoder;

[0097] Encoder: Maps input data to the latent space and produces the mean and variance of the latent variables;

[0098] Decoder: Generates reconstructed data from latent variables;

[0099] VAE control image generation steps

[0100] First, the VAE's encoder extracts information from the front-end input and performs feature extraction on it. Figure 2 The extracted information includes a person's facial expressions, gender, beard, hair color, and other features.

[0101] Then, this feature information is adjusted using a sampler;

[0102] Finally, the feature information adjusted by the sampler is passed through the VAE encoder to generate an image consistent with the feature information.

[0103] System Prompt

[0104] In large-scale language models, system prompts are a key technique for guiding the model's content generation. A well-designed system prompt can effectively control the model's output, ensuring it conforms to specific settings or requirements. This technique has wide applications in content generation, dialogue systems, and text summarization.

[0105] The impact of prompt weights and arrangement on image generation

[0106] In the field of image generation, the weighting and order of prompts have a significant impact on the generated image. The weighting of prompts determines the importance of each feature in the image generation process. Prompts with higher weights are given more attention by the model and thus become more prominent in the generated image.

[0107] The order of the prompts also represents different weights. In some image generation models, such as Stable Diffusion and Flux1, the order of the prompts affects how the model understands and combines these features to generate an image. Generally, the earlier the prompt appears, the higher its weight, meaning it has a greater impact on the generated image.

[0108] The present invention has been described in detail for the purpose of making the disclosure clearer, and the prior art will not be listed in detail.

[0109] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. It is obvious to those skilled in the art that multiple technical solutions of the present invention can be combined. These modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention. All technical contents not described in detail in the present invention are well-known technologies.

Claims

1.A method for generating scene-consistent image instructions based on task decomposition, characterized in that: The generation method includes the following steps; Step S1: Perform image analysis, object recognition, and position and state recognition; Step S2: Perform instruction comprehension and extract key setting information; Step S3: Feature extraction of the preceding scene image in the latent space is achieved through the VAE encoder; Step S4: Decompose the task into a step sequence and describe each step. Step S5: Based on the step sequence, generate an instruction weight matrix with sequential correlation; using the step sequence of step S4, design the instruction weight matrix with sequential correlation: Step S6: Generate visual instructions for the current scene based on the image reference and instruction weight of the preceding scene. When generating visual instructions for the current scene, the features of the preceding image and the prompt words after association processing are referenced. The visual instructions for the current scene are generated by the combined effect of the features of the preceding image and the prompt words after association processing. 2.The task decomposition based scene-consistent image instruction generation method of claim 1, characterized in that: In step S1, firstly, a smart home scene is set up; Then, data is collected; secondly, the user uploads image information to the multimodal large model, which analyzes and processes the uploaded image information to obtain data information. The extraction of data information involves extracting the feature information of the image, which then serves as the basis for generating image instructions. 3.The task decomposition based scene-consistent image instruction generation method of claim 1, characterized in that: In step S1, the data information is encoded by the VAE encoder and then extracted as the feature vector of the image in the latent space. 4.The task decomposition based scene-consistent image instruction generation method of claim 1, characterized in that: In step S2, the following steps are performed; S2.1, Issue user command: S2.2, Obtain system prompt words; S2.3, Extract key information: S2.4, The model is confirmed and feedback is given to the user. 5.The task decomposition based scene-consistent image instruction generation method of claim 1, wherein: In step S3, for the data information uploaded in step S1, firstly, after initialization, the preceding scene image is used as a reference image to input into the large model to generate a new visual instruction image, thereby generating a scene-consistent image, which serves as a visual instruction to guide the user's actual operation. Then, by feeding the entire preceding image as a reference image into the VAE encoder, the consistency of the overall image is maintained; Secondly, when generating the visual instruction image of the current scene from the reference instruction prompt in step S2, the preceding scene image exists in the latent space as a feature vector through the VAE encoder. The preceding scene image is denoised by a sampler. The similarity between the newly generated image and the reference image is adjusted within the range of 0.01 to 1. 6.The task decomposition based scene-consistent image instruction generation method of claim 1, wherein: In step S4, a step sequence is generated based on the steps obtained in step S2. 7.The task decomposition based scene-consistent image instruction generation method of claim 1, wherein: In step S5, firstly, based on the reference image uploaded by the user in step S1, when generating the first visual instruction image, the weight parameters in the first row of the matrix are used to add correlation to the instruction, and the prompt words are obtained after processing. Then, when generating the second visual instruction image, the weight parameters in the second row of the matrix are used to add correlation to the instructions in the visual instruction image, and the prompt words are obtained after processing. Similarly, when generating the Nth visual instruction image, the weight parameter of the corresponding Nth row in the matrix is ​​used to add correlation to the instruction, and the prompt word is obtained after processing. 8.The task decomposition based scene-consistent image instruction generation method of claim 1, wherein: The method employs a variational autoencoder with system prompts to generate prompt words.

Images