A hierarchical robot operation strategy generation method, device and equipment

Abstract

The application discloses a layered robot operation strategy generation method, device and equipment, and the generation method comprises the following steps: collecting a hand-eye camera image and an external camera image of a robot, and obtaining a natural language instruction; taking the hand-eye camera image and the external camera image as image observation, and inputting a cyclic consistency variational autoencoder to obtain semantic features and spatial features; generating a new-view-angle hand-eye camera image, and performing multi-view fusion with the image observation to update the semantic features; inputting the semantic features and the natural language instruction into a basic skill discriminator to generate a robot skill category; and inputting the spatial features and the robot skill category into an action generator to generate a robot action. The application relates to the technical fields of robot intelligent control and artificial intelligence, can significantly improve the success rate and reliability of a robot in performing long-time-series and multi-task operations in a complex and changing environment, and has a wide application prospect.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of robot intelligent control and artificial intelligence technology, and particularly relates to a layered robot operation strategy generation method, device and equipment. BACKGROUND

[0002] With the continuous expansion of the application field of robots, multi-task robot operation based on natural language instructions has become an important direction of agent research. In long-time sequence, multi-stage operation tasks, robots not only need to understand task semantics, but also need to accurately perceive environmental changes and respond flexibly. However, there are two main challenges in the current robot strategy generation method:

[0003] First, the existing perception means is usually based on a single perspective, resulting in incomplete understanding of the environment, especially under the conditions of occlusion and local observation, lacking the ability to reason about unobserved areas;

[0004] Second, the existing strategy learning method usually fails to fully distinguish semantic features and spatial geometric features when modeling features, resulting in mutual interference between sub-task discrimination and action generation, affecting the overall generalization ability and execution stability.

[0005] Some research attempts to introduce large-scale generative models to enhance environmental perception, but due to the artifact or distortion problem of generated images, direct use may lead to strategy degradation. At the same time, large generative models have high inference overhead, making it difficult to meet real-time control requirements. Therefore, it has important research value and application prospect to study a new robot operation strategy generation method that combines generative completion, feature decoupling, cross-domain adaptation, and inference optimization. SUMMARY

[0006] In view of the deficiencies in the prior art described above, the present application provides a layered robot operation strategy generation method, device and equipment, which can effectively improve the long-time sequence task execution ability of robots based on language instructions in complex environments.

[0007] To achieve the above purpose, the present application provides a layered robot operation strategy generation method, comprising the following steps:

[0008] Step 1, collect the hand-eye camera image and external camera image of the robot, and obtain the natural language instruction;

[0009] Step 2, taking the hand-eye camera image and external camera image collected at the current time as image observation, and inputting the image observation into a recurrent consistency variational autoencoder for feature decoupling to obtain semantic features and spatial features;

[0010] Step 3, determine whether a new perspective hand-eye camera image needs to be generated:

[0011] If yes, a new perspective hand-eye camera image is generated, and the new perspective hand-eye camera image is multi-perspective fused with the image observation, and after updating the semantic feature, step 4 is performed;

[0012] Otherwise, step 4 is performed.

[0013] Step 4, a robot skill category is generated based on the semantic feature and the natural language instruction, and a robot action is generated based on the spatial feature and the robot skill category.

[0014] Compared with the prior art, the present application has the following beneficial technical effects:

[0015] The present application uses a cyclic consistency variational autoencoder to decouple the features of the image observation, and then optimizes the hierarchical strategy reasoning, thereby improving the sub-skill discrimination accuracy and action generation stability. In addition, the generated new perspective hand-eye camera image enhances the completion of the environment observation, and improves the three-dimensional perception ability of the robot in a complex environment. The success rate and reliability of the robot in performing long-time sequence and multi-task operations in a complex and changing environment can be significantly improved, and the present application has a wide application prospect. BRIEF DESCRIPTION OF DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor on the basis of the drawings shown.

[0017] Figure 1 The flow chart of the hierarchical robot operation strategy generation method in the embodiment of the present application;

[0018] Figure 2 The structural block diagram of the hierarchical robot operation strategy generation device in the embodiment of the present application;

[0019] Figure 3 The structural block diagram of the terminal device in the embodiment of the present application.

[0020] The implementation, functional features and advantages of the present application will be further described with reference to the drawings. DETAILED DESCRIPTION

[0021] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0022] Furthermore, the technical solutions of the various embodiments of the present invention can be combined with each other, but only if they are feasible for those skilled in the art. If the combination of technical solutions is contradictory or cannot be implemented, it should be considered that such combination of technical solutions does not exist and is not within the scope of protection claimed by the present invention.

[0023] Example 1

[0024] like Figure 1 The above describes a hierarchical robot operation strategy generation method disclosed in this embodiment, which mainly includes the following steps:

[0025] Step 1: Based on the camera mounted on the robot and the external fixed camera, acquire the robot's hand-eye camera images and external camera images, and obtain natural language commands;

[0026] Step 2: The hand-eye camera image and the external camera image acquired at the current moment are used as image observations, and the image observations are input into the cyclic consistency variational autoencoder for feature decoupling to obtain semantic features and spatial features;

[0027] Step 3: Determine whether a new perspective hand-eye camera image needs to be generated:

[0028] If so, generate a new perspective hand-eye camera image, and fuse the new perspective hand-eye camera image with the image observation from multiple perspectives. After updating the semantic features, proceed to step 4.

[0029] Otherwise, proceed to step 4;

[0030] Step 4: Generate robot skill categories based on semantic features and natural language instructions, and generate robot actions based on spatial features and robot skill categories.

[0031] In this embodiment, the generation process of semantic features and spatial features is as follows: extract the semantic feature vector and spatial feature vector of the hand-eye camera image, and extract the semantic feature vector and spatial feature vector of the external camera image; concatenate the semantic feature vectors of the hand-eye camera image and the external camera image to obtain semantic features; and concatenate the spatial feature vectors of the hand-eye camera image and the external camera image to obtain spatial features.

[0032] Image observation is denoted as which contains information from visible light sensors (i.e. cameras), such as object color, background texture, scene layout, etc. Assuming the robot needs to learn multiple skills or operate in different tasks on the same environment, a simple 2D convolution or Transformer feature extraction on s can obtain certain semantic and geometric representations, but the internal usually does not explicitly distinguish between "high-level semantic abstraction" and "low-level spatiotemporal details". Once it is necessary to adapt different types of downstream tasks in the policy network, it often happens that some sub-tasks are over-fitted and others are not well generalized. To this end, the embodiment introduces a cycle-consistency constraint through a cycle-consistent variational autoencoder, which implicitly "cleans up" the latent variables and forms complementary semantic and spatial subspaces.

[0033] Let represent the complete latent variable, where the semantic feature focuses on expressing the semantic information of the scene, such as object categories, object distribution relationships in the scene, etc.; and the spatial feature focuses on the geometric details of the object, such as the geometric pose, position coordinates, occlusion relationship or grasping angle, etc. The traditional variational autoencoder only provides a latent vector , ignoring the need for independent modeling of semantics and geometry. The cycle-consistent variational autoencoder in the embodiment further makes the following assumptions where is the joint prior distribution, is the independent prior of the semantic feature, is the independent prior of the spatial feature, and the above assumptions represent that the semantic feature and the spatial feature are independent of each other in the prior distribution. For any image observation , its posterior distribution is also correspondingly split into , where , correspond to the parameters of the semantic encoder and the geometric encoder, respectively.

[0034] The cycle-consistent variational autoencoder in the embodiment, in the training process, while forcing the network to learn useful representations through reconstruction loss, also helps to add cycle-consistency constraints to ensure the correctness and separation of semantics and geometry. Specifically, the training of the cycle-consistent variational autoencoder is realized through a forward process and a backward process. In order to strengthen the decoupling effect under the same distribution of the scene, the operation of exchanging latent variables is adopted, that is, given two image observations , sampled under the same distribution of the scene, respectively encode to obtain and Then, at the decoding end, their semantic features are exchanged to check whether the generated reconstructed image still maintains the correct semantics and spatial geometry. If the two image observations come from the same task scene, but the specific positions of the objects have slight differences, but the high-level semantic distribution relationship between the objects remains unchanged, then the semantic feature information obtained after feature space decoupling should be consistent. After feature exchange, an image reconstruction with the same semantic layout information and a mixture of specific spatial geometric layouts should be obtained. If the model decoupling is sufficient at this time, the re-encoding of the reconstructed image will not result in a large error in the cycle consistency constraint, indicating that the semantic subspace and spatial subspace are indeed effectively isolated in functional partitioning, improving the learning efficiency under multi-skill and multi-scene conditions.

[0035] In practice, the training process of the cycle-consistent variational autoencoder is as follows:

[0036] Acquire two frames of images from the same scene , and image observation in different scenarios ;

[0037] Image observation , , Inputting a cyclic consistent variational autoencoder yields image observations. semantic features Spatial features Image observation semantic features Spatial features and image observation semantic features Spatial features ;

[0038] semantic features Spatial features After combining and decoding, the reconstructed image observation is obtained. and spatial features With semantic features After combining and decoding, the reconstructed image observation is obtained. And based on reconstructed image observation , ,Right now: , ,in, Based on The probability distribution for decoding. Sampled from it, Similarly;

[0039] Due to image observation , from the same task scene, exchange their semantic features The layout relationship in the scene should not be changed, that is, the reconstructed image after the cross-state semantic feature exchange of the image should still belong to the same semantic category, that is, the same object type and scene structure, and the specific position or pose depends on the spatial feature subspace, and thus the forward process loss can be calculated;

[0040] ;

[0041] wherein, is a hyperparameter for balancing the reconstruction error and the prior constraint, is the reconstruction error after cross-reconstruction, is the prior constraint, so that the latent variable distribution in the semantic subspace and the spatial subspace does not deviate too much from the standard normal distribution, thereby preventing the model from concentrating too much information in a single subspace;

[0042] Next, the reverse process loss is calculated using the cycle consistency. A semantic feature is randomly sampled, and the spatial feature is combined with the semantic feature to obtain the reconstructed image observation , and the spatial feature is combined with the semantic feature to obtain the reconstructed image observation ;

[0043] The reconstructed image observations , are input into the cycle consistency variational autoencoder to obtain the semantic feature and the spatial feature of the reconstructed image observation , and the semantic feature and the spatial feature of the reconstructed image observation , and the reverse process loss is calculated based on the semantic feature and is:

[0044] ;

[0045] wherein, p is the decoding process, q is the encoding process; by minimizing the above reverse process loss, the network will be difficult to utilize the information of one subspace to transfer the information of another subspace, thereby forcing the semantic encoder and the geometric encoder to truly focus on the information type corresponding to itself, forming a clear decoupling structure of semantics-geometry; For the encoding process that only leaves semantic features, the reverse loss process only calculates the difference between the reconstructed semantic features;

[0046] Finally, the total loss is calculated based on the forward loss and the reverse loss , and the network parameters of the cycle-consistent variational autoencoder are updated based on the total loss.

[0047] In the visual-based hierarchical skill learning task, the robot needs to rely on visual information for environment perception and task decision. By introducing a pre-trained visual large model to generate images from new perspectives, the robot can obtain more environmental information, thereby improving the accuracy of high-level skill discrimination. However, due to the high computational cost of the pre-trained large model, if new perspective generation is introduced at all times, it will lead to excessive consumption of computational resources, and may also introduce too much redundant information. Therefore, a reasonable triggering strategy needs to be designed to introduce new perspective generation only at key moments, thereby balancing computational efficiency and perception integrity. To this end, this embodiment proposes a new perspective triggering strategy based on inter-frame mutual information, which detects high uncertainty in visual information at the moment, and then introduces a pre-trained large model for completion. That is, in the specific implementation process of step 3, whether to generate a new perspective hand-eye camera image is determined as follows:

[0048] First, the mutual information between the current frame hand-eye camera image and the previous frame hand-eye camera image is calculated .

[0049] Second, the average value and the standard deviation of the previous frame hand-eye camera image are calculated. Determine whether is true, if so, the new perspective hand-eye camera image needs to be generated, otherwise it does not need to be generated, where is a hyperparameter.

[0050] Mutual information is a measure of the correlation between two random variables, which can be used to evaluate the degree of information redundancy between the current frame and the previous frame. Assuming that the joint probability distribution of the current frame and the previous frame is , the marginal probability distributions are and , then the mutual information between the two frames is defined as:

[0051] ;

[0052] The mutual information measure reflects the degree of information sharing between the current frame and the previous frame. If the mutual information between the two frames is high, it means that can obtain information from A lot of information can be obtained at this time, and there is no need to additionally introduce a new view to complete. However, if the mutual information between two frames is low, it indicates that the information is relatively large compared to the previous frame, which may be due to a large change in the robot's view or a large loss of information due to occlusion of the key target area, and in this case, new view generation should be triggered to complete the missing information.

[0053] Since the inter-frame mutual information varies greatly between different task scenarios, the dynamic mutual information threshold method is selected for calculation in this embodiment. For example, review the previous 10 frames of hand-eye camera images, calculate the average value of the previous 10 frames of hand-eye camera images and the standard deviation , and use the hyperparameter to control the threshold value of new view generation, which can be set to . If the mutual information of the current frame is lower than the dynamic mutual information threshold, it is considered that the visual information of the current frame has changed significantly, thereby triggering new view generation.

[0054] In this embodiment, a pre-trained GenWarp model is used to generate a new view of the hand-eye camera image, and the spatial relationship between the robot and the target is combined during the generation process to adaptively select the view, thereby improving the semantic consistency and geometric structure rationality of the new view of the hand-eye camera image in the robot operation process. Specifically:

[0055] Assume that the maximum available view of the target object is at a distance of , when the view , the projection deformation of the object will exceed the acceptable threshold , thereby affecting semantic consistency. Define the structural similarity measure between the generated image and the real view image , and assume that the relationship between the view and the camera-object distance obeys the following function:

[0056] ;

[0057] wherein is a correction factor related to the distance, is a parameter for controlling the influence of view change on structural similarity, is an angle offset. Starting from the perspective projection transformation, as the rotation angle increases, the projection deformation of the target object increases exponentially, so an exponential decay function is used to describe the similarity decline trend. In the case of close distance, the object projection is large, and even if a large view change occurs, the image can still maintain a high similarity, so is approximately constant. In the far distance case, the object projection is small and the background information occupies a large proportion, leading to significant deformation of the image when the viewing angle changes, so should be decreased. Through experimental observation, we can get:

[0058] ;

[0059] where is the distance attenuation coefficient, then the structural similarity function can be expressed as:

[0060] ;

[0061] Set the similarity threshold , the optimal angle of the new view generation is , so that:

[0062] ;

[0063] After calculation, we can get:

[0064] ;

[0065] The formula shows that the optimal angle of the new view generation decreases with the increase of distance , which shows that in the far distance, even a small change in viewing angle can lead to significant semantic distortion, and the azimuth angle needs to be more cautious, while in the near distance, a larger viewing angle change can be allowed. This conclusion provides a constraint criterion for new view generation, so that the generated image has better semantic consistency in the largest possible range.

[0066] In the specific implementation process of step 3, the process of multi-view fusion of the new view hand-eye camera image and the image observation is as follows: first, input the new view hand-eye camera image into the cyclic consistency variational autoencoder to decouple the features, and get the semantic feature vector and spatial feature vector of the new view hand-eye camera image; then, splice the semantic feature vector of the new view hand-eye camera image with the semantic feature obtained in step 2 to get the updated semantic feature. Since the new view image has slight geometric distortion, but the semantic information is accurate, only the high-level semantic features are retained for high-level skill discrimination to avoid the influence of slight distortion on the success rate of robot operation.

[0067] In the hierarchical skill learning of robots, in order to effectively convert high-level task instructions into low-level action sequences, the concept of basic skill is proposed and widely used. Basic skills refer to reusable atomic operation units, such as robot grasping, rotation and translation operations, which constitute the basic elements of complex tasks. In the specific implementation process of step 4, the generation process of robot skill categories is:

[0068] First, the MiniLM language model is used as an encoder to encode natural language instructions into instruction features, wherein the MiniLM language model is distilled from a large-scale Transformer model, the training corpus covers general language datasets such as Wikipedia, and the output is a 384-dimensional vector, which can map natural language sentences of any length to a fixed-length semantic vector space, providing stable and compact language input features for the strategy model;

[0069] Then, the semantic features and instruction features are input into the basic skill discriminator to obtain the probability of each skill category of the robot, wherein the basic skill discriminator specifically uses a conditional classifier, for example, given a semantic feature and a natural language instruction , the probability output of the basic skill discriminator can be defined as:

[0070] ;

[0071] wherein, represents a set of skill categories, is a feature transformation realized by neural network parameters , and the function is used to map the output of the feature transformation to a probability distribution of the skill categories;

[0072] Finally, the skill category with the highest probability is output as the skill category of the robot.

[0073] In the specific implementation process, the generation process of the robot action is:

[0074] First, the spatial features are input into the diffusion generation model to generate more general geometric spatial features through a step-by-step reverse sampling process, wherein the diffusion generation model introduces a geometric spatial feature reconstruction loss of real environment observation samples in the training stage to reduce the difference between action prediction distributions in virtual and real environments and improve deployment reliability.

[0075] Then, the geometric spatial features, instruction features, and robot skill categories are input into the conditional action generator to generate robot actions.

[0076] The diffusion generation model is a generation model based on stochastic differential equations, which realizes machine learning reconstruction from noise to data samples by defining a forward diffusion process and a corresponding inverse diffusion process. Specifically, the DDPM method is used to gradually add Gaussian noise to the spatial features . By gradually increasing the noise, the original geometric features are gradually disturbed and eventually tend to approach a standard Gaussian distribution. When The distribution of the latent space can be considered as a standard Gaussian distribution. The inverse diffusion process then learns a denoising function through the neural network to gradually recover the original distribution, which enables the diffusion model to effectively cover more diverse spatial geometric feature representations beyond the initial geometric feature distribution.

[0077] As a preferred embodiment, domain consistency constraints are introduced during the diffusion training process to improve the coherence and executability of action generation in virtual training environments and actual deployment environments. Under the framework of transfer learning, it is assumed that the source domain and the target domain respectively obey different probability distributions and , where represents the observed image. In the virtual-real transfer task, is determined by the image observation data distribution in the virtual environment, and is determined by the image data collected by the sensors or cameras in the real world. Due to , the performance of the model trained in the source domain directly used in the target domain will usually decrease. Therefore, domain adaptation methods need to be introduced to enable the model to learn a cross-domain shared feature representation, thereby reducing the impact of distribution bias.

[0078] Consider the domain adaptive encoder , where is the feature embedding space. The goal is to find an optimal mapping such that the feature distributions of the source domain and the target domain are aligned in the latent space, i.e., satisfy:

[0079] ;

[0080] To this end, the present embodiment adopts the Maximum Mean Discrepancy (MMD) as a measure of the similarity of feature distributions. MMD calculates the similarity of two distributions by comparing their mean embeddings in the kernel space, which is defined as follows:

[0081] ;

[0082] where is a feature mapping function that maps the original features to a high-dimensional reproducing kernel Hilbert space , so that data from different distributions have comparability in this space. The present embodiment uses a Gaussian kernel function for mapping. If , the MMD value is close to zero, indicating that the feature distributions of the two domains are highly aligned.

[0083] To optimize the domain adaptation capability, a domain adaptive encoder is used to extract feature embeddings , where represents the low-dimensional feature representation extracted from the input image. In the training process, the data of the source domain and the target domain are input respectively , and the MMD loss is minimized. By gradient descent optimization of the domain adaptive encoder, the domain distribution difference can be gradually reduced while maintaining the task performance. In the specific application process, the MMD loss is trained together with the loss function in the training process of the diffusion generation model and the action generator, so as to obtain the transferable robot control strategy. The action generator in the embodiment is obtained by improving the unconditional action generator in the SPIL method, that is, the skill category is introduced as an additional input to construct a conditional skill generator. The essence is to use the skill representation in the latent space

[0084] and the explicit skill category label y to jointly generate the robot action matched with the current task requirement, while using the robot action in the demonstration data as a supervision signal for training. Specifically, the training process of the action generator is consistent with the conditional variational autoencoder paradigm, the input is the action sequence x and the corresponding soft skill label y from the demonstration data, the encoder maps it to the latent space, and the decoder is responsible for the reconstruction of the action sequence. The training objective function is:

[0085]

[0086] where is the category conditional prior distribution, which is used to realize the structured modeling of the latent space; is the prior distribution, and the standard Gaussian distribution is taken in the embodiment; is the prior constraint, is the structured constraint, is the weight coefficient for adjusting the influence of each regularization term. Through the above structured training objective function, the action generator can not only accurately generate the robot action, but also form clear skill category boundaries in the latent space.

[0087] It is worth noting that although each step in the embodiment Figure 1 is displayed in sequence according to the arrow, these steps are not necessarily executed in the order indicated by the arrow. Unless otherwise specified in this article, the execution of these steps has no strict order restriction, and these steps can be executed in other orders. Moreover, Figure 1 ​​​At least one of the steps in the method can include a plurality of sub-steps or a plurality of stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution order of the sub-steps or stages is not necessarily sequential, but can be alternately or alternately executed with at least one of the other steps or the sub-steps or stages of the other steps.

[0088] Embodiment 2

[0089] Based on the hierarchical robot operation strategy generation method in Embodiment 1, this embodiment discloses a hierarchical robot operation strategy generation device, which refers to Figure 2 The hierarchical robot operation strategy generation device includes an image acquisition unit, an instruction acquisition unit, a feature decoupling unit, a new view generation unit, a multi-view fusion unit, a skill discrimination unit, and an action generation unit, specifically:

[0090] The image acquisition unit is configured to acquire the hand-eye camera image and the external camera image of the robot.

[0091] The instruction acquisition unit is configured to acquire the natural language instruction of the robot.

[0092] The feature decoupling unit is configured to input the hand-eye camera image and the external camera image acquired at the current time as image observation, and input the image observation into a recurrent consistency variational autoencoder for feature decoupling to obtain semantic features and spatial features.

[0093] The new view generation unit is configured to generate a new view hand-eye camera image.

[0094] The multi-view fusion unit is configured to perform multi-view fusion on the new view hand-eye camera image and the image observation to update the semantic features.

[0095] The skill discrimination unit is configured to input the semantic features and the natural language instruction into a basic skill discriminator to generate a robot skill category.

[0096] The action generation unit is configured to input the spatial features and the robot skill category into an action generator to generate a robot action.

[0097] In this embodiment, the specific working processes and working principles of the image acquisition unit, the instruction acquisition unit, the feature decoupling unit, the new view generation unit, the multi-view fusion unit, the skill discrimination unit, and the action generation unit are the same as those in Embodiment 1, so they will not be described again in this embodiment. The various unit modules can be realized by software, hardware, and combinations thereof, and the various unit modules can be embedded in or independent of the processor in the computer device in hardware form, or can be stored in the memory in the computer device in software form, so that the processor can call and execute the operations corresponding to the above various unit modules.

[0098] Embodiment 3

[0099] As Figure 3 shown is a terminal device disclosed in the embodiment, comprising a transmitter, a receiver, a memory and a processor. The transmitter is configured to transmit instructions and data, the receiver is configured to receive instructions and data, the memory is configured to store computer execution instructions, and the processor is configured to execute the computer execution instructions stored in the memory to implement the method in Embodiment 1.

[0100] It should be noted that the above-mentioned memory can be independent or integrated with the processor. When the memory is independently arranged, the terminal device further comprises a bus for connecting the memory and the processor.

[0101] The above is only the preferred embodiment of the present application, and does not limit the protection scope of the present application. Any equivalent structural transformation made under the inventive concept of the present application, or direct / indirect application in other related technical fields, is included in the protection scope of the present application.

Claims

1. A hierarchical robot operation policy generation method characterized by comprising: The method comprises the following steps: Step 1, collecting the hand-eye camera image and the external camera image of the robot, and obtaining a natural language instruction; Step 2, taking the hand-eye camera image and the external camera image collected at the current time as image observations, and inputting the image observations into a recurrent consistency variational autoencoder for feature decoupling to obtain semantic features and spatial features, and the training process of the recurrent consistency variational autoencoder is as follows: Acquiring two image observations under the same scene , , and image observations under different scenes ; observing the image , , inputting the image observation into a cycle-consistent variational autoencoder to obtain semantic features and spatial features of the image observation , semantic features and spatial features of the image observation , and semantic features and spatial features of the image observation ; combining the spatial features with the semantic features to obtain reconstructed image observations , and combining the spatial features with the semantic features to obtain reconstructed image observations , and computing a forward pass loss based on the reconstructed image observations , ; randomly sample a semantic feature , decode to obtain a reconstructed image observation , decode to obtain a reconstructed image observation , decode to obtain a reconstructed image observation , decode to obtain a reconstructed image observation , decode to obtain a reconstructed image observation , decode to obtain a reconstructed image observation ; reconstructing image observations , input image observations input cycle-consistent variational autoencoder to obtain reconstructed image observations semantic features and spatial features of the reconstructed image observations semantic features and spatial features based on the semantic features and calculating a total loss based on the forward process loss and the backward process loss, and updating the network parameters of the recurrent consistency variational autoencoder based on the total loss; Step 3, judging whether a new view hand-eye camera image needs to be generated at present, specifically: calculating the mutual information of the current frame hand-eye camera image and the last frame hand-eye camera image ; and calculating the average value of the previous frame hand-eye camera image and the standard deviation , judging whether the following is established , wherein is a hyperparameter: if yes, generating a new-view hand-eye camera image, and performing multi-view fusion on the new-view hand-eye camera image and the image observations, specifically: inputting the new-view hand-eye camera image into the recurrent consistency variational autoencoder for feature decoupling to obtain a semantic feature vector and a spatial feature vector of the new-view hand-eye camera image, splicing the semantic feature vector of the new-view hand-eye camera image with the semantic features obtained in step 2, updating the semantic features, and then performing step 4; otherwise, performing step 4; Step 4, generating a robot skill category based on the semantic features and the natural language instruction, and generating a robot action based on the spatial features and the robot skill category. 2.The layered robot operation policy generation method of claim 1, wherein, In step 2, the generation process of the semantic features and the spatial features is as follows: extracting a semantic feature vector and a spatial feature vector of the hand-eye camera image, and extracting a semantic feature vector and a spatial feature vector of the external camera image; splicing the semantic feature vectors of the hand-eye camera image and the external camera image to obtain the semantic features; splicing the spatial feature vectors of the hand-eye camera image and the external camera image to obtain the spatial features. 3.The layered robot operation policy generation method according to claim 1 or 2, characterized by, In step 3, a pre-trained GenWarp model is used to generate a new-view hand-eye camera image, and the spatial relationship between the robot and the target is combined in the generation process to adaptively select a view, thereby improving the semantic consistency and geometric structure rationality of the new-view hand-eye camera image in the robot operation process. 4.The layered robot operation policy generation method according to claim 1 or 2, characterized by, In step 4, the generation process of the robot skill category is as follows: first, using a MiniLM language model as an encoder to encode the natural language instruction into instruction features; then, inputting the semantic features and the instruction features into a basic skill discriminator to obtain probabilities of each skill category of the robot, wherein the skill categories include grasping, rotating, and translating; finally, outputting the skill category with the highest probability as the robot skill category.

5. The hierarchical robot operation policy generation method according to claim 4, characterized by, In step 4, the generation process of the robot action is as follows: first, inputting the spatial features into a diffusion generative model to generate more general geometric spatial features through a step-by-step reverse sampling process; then, inputting the geometric spatial features, the instruction features, and the robot skill category into a conditioned action generator to generate the robot action.

6. A hierarchical robot operation policy generation apparatus characterized by comprising: The layered robot operation strategy generation device adopts the method of any one of claims 1 to 5, and comprises: an image collection unit configured to collect a hand-eye camera image and an external camera image of a robot; An instruction obtaining unit is configured to obtain a natural language instruction of a robot; A feature decoupling unit is configured to take a hand-eye camera image and an external camera image collected at a current moment as image observation, input the image observation into a loop-consistent variational autoencoder for feature decoupling, and obtain semantic features and spatial features; A new-viewpoint generating unit is configured to generate a new-viewpoint hand-eye camera image; A multi-view fusion unit is configured to perform multi-view fusion on the new-viewpoint hand-eye camera image and the image observation, and update the semantic features; A skill discriminating unit is configured to input the semantic features and the natural language instruction into a basic skill discriminator, and generate a robot skill category; An action generating unit is configured to input the spatial features and the robot skill category into an action generator, and generate a robot action.

7. A terminal device, characterized by, The terminal device is provided with: a memory configured to store a program; a processor configured to execute the program stored in the memory, and when the program is executed, the processor is configured to execute the method in any one of claims 1 to 5.

Images