Robot large model-oriented operation scene alignment generation method and system and medium

Abstract

The invention relates to a robot large model-oriented operation scene alignment generation method and system and a medium. The problems that in the prior art, videos aligned with real scenes cannot be generated, details related to physical interaction are lacked, and the teaching effect and the reliability of follow-up planning are affected are solved. According to the scheme of the invention, the additional scene background information is enhanced by using the infrared image, so that the fine-grained structure information in the scene is obtained, and a more comprehensive video generation context prompt is obtained. And then, the enhanced prompts are accurately converted into high-quality video contents by using a diffusion model, so that the generated video is aligned with a real scene, and the teaching effect of the robot and the reliability of subsequent planning are enhanced.

Description

Technical Field

[0001] The present disclosure relates to the field of robot teaching and planning, and in particular to a method, system and medium for aligning and generating operation scenes for a large robot model. Background Art

[0002] Large robot models such as GR-2 can generate robot operation videos for teaching, and have strong generalization ability and multi-task versatility. However, in complex general scenes, the generated videos are often difficult to accurately align with the real scene. Summary of the invention

[0003] The purpose of this disclosure is to propose a method for aligning and generating working scenes for a large robot model, so as to overcome the problem that the existing technology cannot generate videos aligned with real scenes, lacks details related to physical interactions, and affects the reliability of teaching effects and subsequent planning. The specific technical solution is as follows.

[0004] In a first aspect, the present disclosure proposes a method for aligning and generating a work scene for a large robot model, the method comprising the following steps: the robot simultaneously acquires a visible image and an infrared image of the same work scene; based on the visible image, a visible video with noise is acquired, and then a visible video latent feature is acquired; based on the infrared image, an infrared video with noise is acquired, and then an infrared video latent feature is acquired; the visible video latent feature and the infrared latent feature are spliced ​​to obtain a mixed video latent feature; based on the visible image and the infrared image, object structure segmentation information in the work scene and description information about the work scene are acquired; based on the segmentation information, spatial condition features are acquired; based on the description information, text features are acquired, and the spatial condition features and the text features are used as multimodal conditions; under the guidance of the multimodal conditions, the mixed video latent features are denoised to obtain denoised mixed video latent features; based on the denoised mixed video latent features, the inverse operation of splicing is used to obtain denoised visible video latent features and denoised infrared video latent features, and then visible video and infrared video for robot work motion planning are obtained.

[0005] In one implementation of the above technical solution, the spatial condition feature is constructed by the following formula: Where: m i is the i-th segmentation area, t i is the category of the i-th segmented region, T5 is the text-to-text Transformer model, and N is the total number of segmented regions.

[0006] In one implementation of the above technical solution, the segmented area is obtained by constructing a panoramic segmentation model based on infrared and visible color images using a Mask2Former model, and the category corresponding to the segmented area is generated using a multimodal large language model prompt.

[0007] In one implementation of the above technical solution, the description information about the work scene is generated under prompting using a multimodal large language model, and the generated content includes an overview, characteristics, several details and environmental description of the work scene.

[0008] In one implementation of the above technical solution, the visible video latent features are obtained by encoding based on the visible image and initialization noise using a first encoder, and the infrared video latent features are obtained by encoding based on the infrared image and initialization noise using a second encoder.

[0009] In one embodiment of the above technical solution, the visible video and infrared video used for robot operation motion planning, the acquisition step includes: using a first decoder to decode the denoised visible video latent features to obtain a visible video; using a second decoder to decode the denoised infrared video latent features to obtain an infrared video.

[0010] In one implementation of the above technical solution, under the guidance of multimodal conditions, a preset diffusion model is used to denoise the latent features of the mixed video, and in the diffusion model, a cross-attention mechanism is used to process the multimodal conditions.

[0011] In one implementation of the above technical solution, the diffusion model is a visual Transformer model. In each Block of the visual Transformer, for the features processed by the attention mechanism, a first cross-attention mechanism is first used to establish an association fusion between the feature and the spatial condition feature to obtain the first feature, and then a second cross-attention mechanism is used to establish an association fusion between the first feature and the text feature to obtain the second feature.

[0012] In a second aspect, the present disclosure proposes a computer-readable storage medium storing a computer program that can be loaded by a processor and execute any of the above methods.

[0013] In a third aspect, the present disclosure proposes a robot, comprising a first camera, a second camera, and a video generation module, wherein the video generation module comprises a condition unit, a mixed video latent feature unit, a denoising unit, and a video unit; the first camera is used to obtain a visible image of a robot operation scene, and the second camera is used to obtain an infrared image of the same robot operation scene; the mixed video latent feature unit is configured to obtain a visible video with noise based on the visible image, thereby obtaining a visible video latent feature, obtain an infrared video with noise based on the infrared image, thereby obtaining an infrared video latent feature, and splice the visible video latent feature and the infrared latent feature to obtain a mixed video latent feature; the condition unit, The system is configured to obtain object structure segmentation information and description information about the work scene in the work scene based on the visible image and the infrared image, obtain spatial condition features based on the segmentation information, obtain text features based on the description information, and use the spatial condition features and the text features as multimodal conditions; the denoising unit is configured to denoise the mixed video latent features under the guidance of the multimodal conditions to obtain denoised mixed video latent features; the video unit is configured to obtain denoised visible video latent features and denoised infrared video latent features based on the denoised mixed video latent features by using the inverse operation of splicing, and then obtain visible video and infrared video for robot work motion planning.

[0014] The beneficial technical effect of the disclosed technical solution is: infrared images are used to enhance scene information to obtain fine-grained structural information in the scene, thereby obtaining more comprehensive contextual prompts for video generation. Subsequently, by designing an efficient visual Transformer fine-tuning method, these enhanced prompts are accurately converted into high-quality video content, so that the generated video is aligned with the real scene, enhancing the robot teaching effect and the reliability of subsequent planning. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0016] Figure 1 , schematic diagram of multimodal initial scene description.

[0017] Figure 2 , schematic diagram of multimodal video generation.

[0018] Figure 1In the figure, 100 is a panoramic segmentation, 200 is a dense description generation, a is a visible image, b is a region segmentation by object structure, c is an infrared image, d is a dense text annotation, 310 is a predicted segmentation region category, and 320 is a dense description annotation; Figure 2 Among them, 510 is the initialized video conditioned on the visible light image, 520 is the initialized video conditioned on the infrared image, 530 is the mixed video latent feature, 540 is the denoised mixed video latent feature, 550 is the visible light video, 560 is the infrared video, 600 is the diffusion model, 610 is the first encoder, 620 is the second encoder, 630 is the first decoder, 640 is the second decoder, 650 represents repeated iterative denoising operations, 311 is the spatial conditional feature, 321 is the textual feature, and 300 is the multimodal condition. DETAILED DESCRIPTION

[0019] Existing technologies mostly rely on image and text conditions for video generation, which cannot accurately capture the fine-grained structural information in the scene, especially the details related to physical interaction, so video generation is often based on the current view image as the alignment condition. In complex real-world scenes, if only the current view image is used, it is difficult to describe the complex physical properties of the scene, and the generated video will be difficult to accurately align with the real scene, affecting the teaching effect and the reliability of subsequent planning.

[0020] Based on this, this case proposes a scene alignment method based on multimodal infrared enhanced description to reduce the deviation between the generated video and the actual scene. This method is of great significance in the field of robot teaching and planning, and can predict the physical interaction behavior of objects in the simulated scene, thereby optimizing subsequent action planning.

[0021] The following will be combined with the accompanying drawings to clearly and completely describe how to implement the technical solution of this case. Obviously, the described implementation method is only a part of the implementation method of this case, not all of the implementation methods. Based on the implementation method in this case, all other implementation methods obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0022] 1. Initial description of the scene

[0023] Compared with complex real-world scenes, it is difficult to describe the complex physical properties of the scene using only the current view image. Currently, video generation is often based on the current view image as an alignment condition. Considering the complementarity of infrared image and visible image information, and the fact that the operation scene for large robot models is easy to collect infrared image semi-open scenes, infrared and visible images are used in combination to perform the initial scene description. The initial scene description serves as additional scene background information, providing the context for video generation and controlling video generation.

[0024] The visible image is preferably a color image (see Figure 1 (a in the figure). Figure 1 The c in Figure 1 Infrared image of the color image in a.

[0025] See also Figure 1 In b, a panoramic segmentation model based on infrared and visible color images is constructed based on the Mask2Former model, and a plurality of object structure segmentation regions are obtained through panoramic segmentation (100). In the segmentation, the infrared image can enhance the scene information, thereby facilitating the acquisition of fine-grained structure information in the scene.

[0026] Constructing segmented image I based on object structure segmentation region c The fine-grained object segmentation information M = [(m i , t i )] i , where m i ∈R W×H×1 is the i-th segmentation area, and t i is the corresponding category description. W×H×1 It is a real number field with spatial dimensions of W×H×1. W is the width of the image, H is the height of the image, and the number 1 is the number of category channels. The category description is generated by the multimodal large language model MLLM prediction:

[0027] ti ＝ MLLM(m i , prompt i )

[0028] where prompt i = "Briefly describe this area" is a description generation prompt. Figure 1 Taking the visible image in FIG. 1 as an example, the predicted segmented region categories (310) with the assistance of the infrared image are keyboard, metal cover and table.

[0029] The description information about the operation scene is taken as a dense description, denoted as T, and I c is the visible image, I ir is the visible image I c The corresponding infrared image:

[0030] T=MLLM(I c , I ir ,prompt ir = "Briefly describe the area")

[0031] Among them, prompt ir Provide a detailed description of the figure, prompt ir="Overview, features, details 1, details 2, environmental description" is a prompt for generating dense description. Figure 1 The d in the figure is a dense text annotation, and the dense description generation (200) and dense description annotation (320) are shown in Table 1. It should be noted that detailed descriptions can be added as needed, and are not limited to the above-mentioned details 1 and 2, but more detailed descriptions will increase the difficulty of generating MLLM.

[0032] Table 1

[0033]

[0034] After completing the multimodal initial scene description process, fine-grained distribution information and multi-dimensional description information can be obtained. The multi-dimensional description information consists of category description and dense description. The specific content of dense description varies from person to person. Summary is an overview of the content displayed on the color image. Features are descriptions of the most prominent elements. Details 1 and Details 2 are feature descriptions based on different granularity sizes. For example, Details 1 describes large, relatively detailed features, while Details 2 describes more detailed features. Environmental description is a description of background scene features.

[0035] (II) Fine-tuning

[0036] A video generation model is constructed using a neural network. In one model implementation, the video generation model includes a first video encoder (610), a second video encoder (620), a diffusion model (600), a first decoder (630), and a second decoder (640), see Figure 2 shown.

[0037] When the model is initialized, the visible light image in the initial description of the scene is used as a condition to obtain an initialization video conditioned on the visible light image (510), and the infrared image in the initial description of the scene is used as a condition to obtain an initialization video conditioned on the infrared image (520).

[0038] The first video encoder is configured to obtain visible video latent features based on a given initial visible image and random Gaussian noise. The second video encoder is configured to obtain infrared video latent features based on a given initial infrared image and random Gaussian noise. The visible video latent features and the infrared latent features are concatenated to obtain mixed video latent features (530). The mixed video latent features are composed of image features and noise features.

[0039] The mixed video latent features are denoised using a preset diffusion model (600). The mixed video latent features are used as input, and the generation of denoised mixed video latent features (540) is guided and controlled by adding multimodal conditions (300). The denoising operation (650) is repeated and the denoised mixed video latent features are used as input of the diffusion model until the operation stop condition is met to obtain the final mixed video latent features. The number of operations is set. The above operation process is a denoising process. In each denoising operation, the diffusion model is integrated into the internal latent features using a cross-attention method, thereby controlling the changes in the internal latent features during the denoising process.

[0040] Based on the final mixed video latent features, the inverse operation of splicing can be performed to obtain denoised visible video latent features and denoised infrared video latent features. Based on the denoised visible light video latent features, the configured first video decoder is used to obtain the generated visible light video (550). Based on the denoised infrared video latent features, the configured second video decoder is used to obtain the generated infrared video (560).

[0041] The multimodal conditions involved in the above process include text features (321) and spatial condition features (311), which are used as prompts for fine-tuning. The text features are extracted based on dense description text using model T5. T5 is a text-to-text Transformer model (Text-to-Text Transformer).

[0042] The spatial condition feature is constructed using model T5 based on the visible image in the initial description of the scene in the following way:

[0043]

[0044] Among them, T5(t i )∈R 1×1×E Describe the category t i The features of M are used to generate fine-grained spatial conditional features M cond ∈R W ×H×E ∑ is the accumulation symbol. N is the total number of segmented regions. M cond Helps control further enhancing the alignment effect.

[0045] The above diffusion model can use visual Transformer to accurately convert these enhanced cues into high-quality video content by designing an efficient visual Transformer fine-tuning method, so that the generated video is aligned with the real scene, enhancing the robot teaching effect and the reliability of subsequent planning.

[0046] Exemplarily, the designed visual transformer has 30 blocks. In each block, for the feature processed by the attention mechanism, the first cross attention mechanism is used to establish the association fusion between the feature and the spatial condition feature to obtain the first feature, and then the second cross attention mechanism is used to establish the association fusion between the first feature and the text feature to obtain the second feature.

[0047] Specifically, the features processed by the attention mechanism in each Block are recorded as x', and the first cross attention mechanism is processed as follows:

[0048] x1=attention(x',down2d(M cond ), t)

[0049] Where t is the current iteration number. down2d will M cond Downsample to 4×4 size, and control the internal deep features x′ through standard attention fusion spatial fine-grained conditions. Table 2 gives the down2d module downsampling network parameters.

[0050] Then, the second cross attention mechanism is used to process as follows:

[0051] x2=attention(x1,T5(T),t)

[0052] Table 2

[0053]

[0054] The training method of the above model is a commonly used training model method. After the model is trained, an infrared camera can be deployed on the robot to obtain a complementary modality with the visible light image. The text description obtained by the visible image and infrared image can be used to guide the prediction of the physical interaction behavior of objects in the operation scene for robot operation action planning.

[0055] In summary, this case uses a multimodal large language model to improve the expression ability of fine-grained spatial features, and uses infrared modality to supplement the deficiencies of visible light modality to construct a multi-level scene background description that integrates multimodal information. In addition, the visual Transformer model is used for fine-tuning to more accurately align the real scene and generate realistic robot operation videos, thereby effectively supporting robot automatic planning and teaching.

[0056] The visual Transformer in this case can be replaced by a Unet neural network.

[0057] The terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, features defined as "first" and "second" may explicitly or implicitly include one or more of the features.

[0058] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method of the present invention can be implemented by means of software plus necessary general hardware, and of course can also be implemented by special hardware including special integrated circuits, special CPUs, special memories, special components, etc. In general, all functions performed by computer programs can be easily implemented by corresponding hardware, and the specific hardware structures used to implement the same function can also be various, such as analog circuits, digital circuits or special circuits, etc. However, for the present disclosure, software program implementation is a better implementation method in most cases.

[0059] Through the description of the above implementation methods, the technicians in the relevant field can clearly understand that a robot can be realized based on the method of this case. Exemplarily, the robot includes a first camera, a second camera, and a video generation module, and the video generation module includes a condition unit, a mixed video latent feature unit, a denoising unit, and a video unit. The first camera is used to obtain a visible image of the robot's working scene, and the second camera is used to obtain an infrared image of the same robot's working scene. The mixed video latent feature unit is configured to obtain a visible video with noise based on the visible image, and then obtain a visible video latent feature, obtain an infrared video with noise based on the infrared image, and then obtain an infrared video latent feature, and splice the visible video latent feature and the infrared latent feature to obtain a mixed video latent feature. The condition unit is configured to obtain object structure segmentation information and description information about the working scene in the working scene based on the visible image and the infrared image, obtain spatial condition features based on the segmentation information, obtain text features based on the description information, and use the spatial condition features and the text features as multimodal conditions. The denoising unit is configured to denoise the mixed video latent features under the guidance of multimodal conditions to obtain denoised mixed video latent features. The video unit is configured to obtain denoised visible video latent features and denoised infrared video latent features based on the denoised mixed video latent features by using an inverse operation of splicing, thereby obtaining visible video and infrared video for robot operation action planning.

[0060] The robot can use infrared images to enhance additional scene background information, use infrared modalities to supplement the deficiencies of visible light modalities, obtain fine-grained structural information in the scene, and generate more comprehensive video contexts for constructing multimodal conditions. By simultaneously initializing videos conditioned on visible light images and initializing videos conditioned on infrared images, visible light videos and infrared videos that align with real scenes and conform to actual robot operations are generated under the guidance of multimodal conditions, providing effective support for robot automatic planning and teaching and improving reliability. Among them, the visual Transformer model is fine-tuned to accurately convert prompts in multimodal conditions into high-quality video content.

[0061] Although the embodiments of the present disclosure are described above in conjunction with the accompanying drawings, the present disclosure is not limited to the above-mentioned specific embodiments and application fields, and the above-mentioned specific embodiments are merely illustrative and instructive, rather than restrictive. A person of ordinary skill in the art can also make many forms under the guidance of this specification and without departing from the scope of protection of the claims of the present disclosure, all of which belong to the scope of protection of the present disclosure.

Claims

1. A method for aligning and generating a work scene for a robot large model, characterized in that: The method comprises the following steps: The robot acquires visible and infrared images of the same operating scene simultaneously; Obtaining a visible video with noise based on the visible image, thereby obtaining a visible video latent feature; obtaining an infrared video with noise based on the infrared image, thereby obtaining an infrared video latent feature; and splicing the visible video latent feature and the infrared latent feature to obtain a mixed video latent feature; Based on the visible image and the infrared image, obtaining object structure segmentation information in the operation scene and description information about the operation scene, obtaining spatial condition features based on the segmentation information, obtaining text features based on the description information, and using the spatial condition features and the text features as multimodal conditions; Under the guidance of multimodal conditions, denoising the mixed video latent features to obtain denoised mixed video latent features; Based on the denoised mixed video latent features, the inverse operation of splicing is used to obtain denoised visible video latent features and denoised infrared video latent features, and then obtain visible video and infrared video for robot operation action planning.

2. The method according to claim 1, characterized in that: The spatial condition characteristics are constructed by the following formula: Where: m i is the i-th segmentation area, t i is the category of the i-th segmented region, T5 is the text-to-text Transformer model, and N is the total number of segmented regions.

3. The method according to claim 2, characterized in that The segmented regions are obtained by constructing a panoramic segmentation model based on infrared and visible color images using a Mask2Former model, and the categories corresponding to the segmented regions are generated using a multimodal large language model prompt.

4. The method according to claim 1, characterized in that: The description information about the operation scene is generated under prompting using a multimodal large language model, and the generated content includes an overview, characteristics, several details and an environmental description of the operation scene.

5. The method according to claim 1, characterized in that: The visible video latent features are obtained by encoding the visible image and initialization noise using a first encoder, and the infrared video latent features are obtained by encoding the infrared image and initialization noise using a second encoder.

6. The method according to claim 1, characterized in that The visible video and infrared video used for robot operation motion planning, the acquisition steps include: The first decoder is used to decode the latent features of the denoised visible video to obtain the visible video; the second decoder is used to decode the latent features of the denoised infrared video to obtain the infrared video.

7. The method according to claim 1, characterized in that Under the guidance of multimodal conditions, a preset diffusion model is used to denoise the latent features of the mixed video. In the diffusion model, a cross-attention mechanism is used to process the multimodal conditions.

8. The method according to claim 7, characterized in that The diffusion model is a visual Transformer model. In each Block of the visual Transformer, for the features processed by the attention mechanism, a first cross-attention mechanism is first used to establish an association fusion between the feature and the spatial condition feature to obtain the first feature, and then a second cross-attention mechanism is used to establish an association fusion between the first feature and the text feature to obtain the second feature.

9. A computer-readable storage medium, characterized in that: A computer program is stored which can be loaded by a processor and execute the method according to any one of claims 1 to 8.

10. A robot, characterized in that: The robot comprises a first camera, a second camera, and a video generation module, wherein the video generation module comprises a condition unit, a mixed video latent feature unit, a denoising unit, and a video unit; The first camera is used to obtain a visible image of the robot's operating scene, and the second camera is used to obtain an infrared image of the same robot's operating scene; The mixed video latent feature unit is configured to obtain a visible video with noise based on the visible image, thereby obtaining a visible video latent feature, obtain an infrared video with noise based on the infrared image, thereby obtaining an infrared video latent feature, and splice the visible video latent feature and the infrared latent feature to obtain a mixed video latent feature; The condition unit is configured to obtain object structure segmentation information and description information about the operation scene in the operation scene based on the visible image and the infrared image, obtain spatial condition features based on the segmentation information, obtain text features based on the description information, and use the spatial condition features and the text features as multimodal conditions; The denoising unit is configured to denoise the mixed video latent features under the guidance of the multimodal conditions to obtain denoised mixed video latent features; The video unit is configured to obtain denoised visible video latent features and denoised infrared video latent features based on the denoised mixed video latent features, using the inverse operation of splicing, and then obtain visible video and infrared video for robot operation action planning.

Images