A structure-preserving robot operation target state generation method and system

Abstract

The application discloses a kind of structural keeping robot operation target state generation method and system, wherein method includes: constructing the diffusion generation process of target state, gradually noisy disturbance is carried out to target state point cloud to form the noisy state representation corresponding to multiple diffusion time steps;Scene point cloud, operation object point cloud, task description information and noisy state representation are input into the target state prediction network trained, and noisy state representation is gradually conditionally denoised prediction, and the noise prediction result under current time step is obtained;Noisy state is updated gradually based on noise prediction result, and target state point cloud is obtained;Target state point cloud is used as the target state of robot operation task.The application keeps the internal geometric structure consistency of object in target state generation process, reduces the structural distortion caused by generation model, improves the stability and reliability of target state to object pose recovery and robot operation planning, and has good engineering application value.

Description

Technical Field

[0001] This invention relates to the field of robot manipulation and three-dimensional perception technology, and in particular to a method and system for generating target states of robot manipulation with structure preservation. Background Technology

[0002] Robotic manipulation tasks typically require determining the target state of an object after manipulation to support subsequent motion planning and execution. In recent years, learning-based methods have increasingly adopted target state prediction as an intermediate representation, decoupling task understanding from low-level control processes by predicting the target configuration of the manipulated object. Existing target state prediction methods mostly use regression or generative models to directly predict the target pose or key points. However, in complex scenarios, due to contact relationships and spatial constraints between objects and the environment, target state prediction is prone to geometric distortion, leading to prediction results that do not meet rigid body constraints, thus affecting the stability of subsequent pose recovery and motion planning. Especially in the generation process based on diffusion models, independently updating each point in the point cloud can easily disrupt the internal geometric relationships of the object, causing deformation or inconsistencies in the generated results, making them difficult to directly use for robot manipulation. Therefore, it is necessary to propose a generation method that can maintain the consistency of the object's geometric structure during target state generation to improve the reliability and executability of target prediction in robotic manipulation tasks. Summary of the Invention

[0003] To address the shortcomings of existing technologies, this invention provides a method and system for generating target states for robot operations while maintaining structure. By introducing geometric consistency constraints during the target state generation process, the generated results maintain the inherent geometric structure of the object while satisfying task semantics and scene constraints, thereby improving the executability and stability of robot operation tasks.

[0004] On the one hand, a method for generating target states for robot operations with structure preservation is provided, including: Obtain the operation scene information of the robot operation task, wherein the operation scene information includes: scene point cloud, operation object point cloud and task description information; The diffusion generation process of the target state is constructed by progressively perturbing the target state point cloud with noise to form noisy state representations corresponding to multiple diffusion time steps; The scene point cloud, the object point cloud, the task description information, and the noisy state representation are input into the trained target state prediction network. The noisy state representation is subjected to progressive conditional denoising prediction to obtain the noise prediction result at the current time step. Based on the noise prediction result at the current time step, the noisy state is progressively updated to obtain the target state point cloud. The target state point cloud is used as the target state of the robot operation task. The target state prediction network uses a loss function during the training phase that is a weighted sum of two parts: the first part is a structural consistency constraint term established based on the operational point cloud; the other part is a noise prediction loss term.

[0005] On the other hand, a structure-preserving robot operation target state generation system is provided, including: The acquisition module is configured to acquire the operation scene information of the robot operation task, wherein the operation scene information includes: scene point cloud, operation object point cloud and task description information; The construction module is configured to: construct the diffusion generation process of the target state, and gradually perturb the target state point cloud with noise to form a noisy state representation corresponding to multiple diffusion time steps; The generation module is configured to: input the scene point cloud, the point cloud of the manipulated object, the task description information, and the noisy state representation into the trained target state prediction network; perform progressive conditional denoising prediction on the noisy state representation to obtain the noise prediction result at the current time step; based on the noise prediction result at the current time step, progressively update the noisy state to obtain the target state point cloud; and use the target state point cloud as the target state of the robot's operation task. The target state prediction network uses a loss function during the training phase that is a weighted sum of two parts: the first part is a structural consistency constraint term established based on the operational point cloud; the other part is a noise prediction loss term.

[0006] Furthermore, an electronic device is also provided, including: Memory, used for non-transitory storage of computer-readable instructions; and Processor, for executing the computer-readable instructions, When the computer-readable instructions are executed by the processor, they perform the method described in the first aspect above.

[0007] In another aspect, a storage medium is also provided for non-transitory storage of computer-readable instructions, wherein when the non-transitory computer-readable instructions are executed by a computer, the method described in the first aspect is performed.

[0008] In another aspect, a computer program product is also provided, including a computer program that, when run on one or more processors, is used to implement the method described in the first aspect above.

[0009] The above technical solution has the following advantages or beneficial effects: This invention provides a structure-preserving method and system for generating target states for robot operations. By introducing structural consistency constraints during target state generation, and limiting changes in the relative spatial relationships within the object's point cloud, the generated result maintains the object's inherent geometric structure while satisfying task semantics and scene constraints. This effectively avoids geometric distortion problems caused by the generation model during prediction, thereby improving the physical rationality and stability of the target state. Based on structure-preserving target point clouds, rigid body registration and pose recovery can be performed more accurately, improving the reliability and success rate of robot motion planning. Furthermore, this invention utilizes scene geometric information, object state information, and task semantic information for conditional generation, enabling target state prediction to fully consider environmental constraints and generate executable operation targets even in complex multi-object interaction scenarios. In addition, the target states generated by this invention can be directly integrated with traditional motion planning methods, achieving effective decoupling between target prediction and robot control processes, exhibiting good generalization ability, engineering applicability, and application promotion value. Attached Figure Description

[0010] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0011] Figure 1 A schematic diagram of the overall framework of a structure-preserving robot operation target state generation method provided in an embodiment of the present invention; Figure 2 This is a schematic diagram of the target state prediction network structure provided in an embodiment of the present invention; Figure 3 A schematic diagram of the hierarchical point cloud sensing coding network structure provided in an embodiment of the present invention; Figure 4 A flowchart illustrating the structure-preserving robot operation target state generation method provided in an embodiment of the present invention; Figure 5 This is a schematic diagram of the target state diffusion generation process provided in an embodiment of the present invention; Figure 6 This is a schematic diagram of the structural consistency constraint mechanism provided in an embodiment of the present invention, used to illustrate how the geometric relationship of the object point cloud is maintained during the target state generation process. Detailed Implementation

[0012] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0013] Example 1 This embodiment provides a method for generating target states of robot operations while maintaining structure; like Figures 1 to 6 As shown, a method for generating target states for structure-preserving robot operations includes: S101: Obtain the operation scene information of the robot operation task, wherein the operation scene information includes: scene point cloud, operation object point cloud and task description information; S102: Construct the diffusion generation process of the target state, and gradually perturb the target state point cloud with noise to form a noisy state representation corresponding to multiple diffusion time steps; S103: Input the scene point cloud, the point cloud of the manipulated object, the task description information, and the noisy state representation into the trained target state prediction network, perform stepwise conditional denoising prediction on the noisy state representation to obtain the noise prediction result at the current time step; based on the noise prediction result at the current time step, update the noisy state stepwise to obtain the target state point cloud; use the target state point cloud as the target state of the robot operation task. The target state prediction network uses a loss function during the training phase that is a weighted sum of two parts: the first part is a structural consistency constraint term established based on the operational point cloud; the other part is a noise prediction loss term.

[0014] Given that robot operation tasks typically rely on accurate prediction of the target state of an object, and existing generation methods are prone to destroying the object's geometry during the prediction process, making it difficult to directly use the generated results for motion planning, this invention proposes a structure-preserving target state generation method.

[0015] This invention constructs a target state diffusion generation process, introducing structural consistency constraints during the generation process to ensure that the target state maintains stable internal geometric relationships while satisfying task semantics and scene constraints. Specifically, firstly, the geometric relationships of the environment are modeled using scene point clouds and manipulated object point clouds, and a conditional generation model is established by combining task description information; then, the target state is gradually restored through a diffusion generation mechanism, and a structure preservation mechanism is introduced during the denoising and updating process to constrain the relative spatial relationships within the point cloud, thereby avoiding geometric distortions generated during the generation process.

[0016] Through the above methods, the present invention can generate target state point clouds that satisfy operational semantics and maintain structural consistency, and can be further converted into object poses to support robot motion planning and operation execution, thereby achieving effective connection between target prediction and robot control process.

[0017] Further, in step S101: acquiring the operation scene information of the robot operation task, the operation scene information including: scene point cloud, operation object point cloud, and task description information, specifically including: The scene point cloud is used to represent various objects and spatial structures in the environment; the operation object point cloud is used to represent the three-dimensional geometric shape of the object to be operated on; and the task description information is used to represent the semantic constraints of the operation.

[0018] In this embodiment, the scene point cloud is acquired by a depth camera or an RGB-D sensor, and the point cloud of the manipulated object is extracted by instance segmentation or point cloud cropping.

[0019] In this embodiment, the task description information is a natural language instruction, such as: "Pick up the kettle and pour water into the teapot".

[0020] First, the robot's observation information of the environment at the initial moment of operation is acquired, and the operation scene input is constructed based on this observation. The observation information is acquired through a vision sensor installed at the robot's end effector, which is either a depth camera or an RGB-D camera, used to collect point cloud data containing three-dimensional spatial structure information. Since the manipulated object may be occluded due to the movement of the robotic arm during robot operation, resulting in incomplete observations at intermediate moments, this embodiment only uses a single observation before the start of operation as input, and predicts the subsequent target state based on this initial observation, thereby modeling the robot operation process as an open-loop generation process based on the initial state.

[0021] Based on the observed data, scene point clouds related to the operation task are segmented from the observed images using an open vocabulary segmentation model (such as SAMv2). This invention defines scene point clouds as the point clouds of objects that make task-specific contact with the manipulated object during the operation task, such as the point cloud of the "teacup" in the task of "lifting the kettle to pour water into the teacup". Simultaneously, point cloud data corresponding to the manipulated object (such as the "kettle" in the aforementioned task) is obtained from the observation data using an open vocabulary segmentation model to construct the object point cloud. This is used to represent the three-dimensional geometry of the object to be manipulated. In this embodiment, both the scene point cloud and the object point cloud are represented in the form of point sets, where each point contains three-dimensional spatial coordinate information.

[0022] In addition, task description information describing the robot's operational objectives is obtained. The task description information is used to characterize the semantic relationship between the operation target and the object. In this embodiment, the task description information is given in natural language and is used to constrain the generation result of the target state to meet the task semantic requirements.

[0023] Finally, the scene point cloud is... Manipulating point clouds of objects and task description information As a unified input, it is used for conditional modeling in the subsequent target state generation process, so that the generated target state can depend on the environmental geometry, the object's own shape, and the task semantic constraints simultaneously.

[0024] The generation of the target state point cloud is modeled as a forward diffusion process and a reverse denoising process based on conditional diffusion, which is used to learn the probability distribution of the target state point cloud under given scene and task conditions.

[0025] Further, S102: Constructing the diffusion generation process of the target state, which involves progressively perturbing the target state point cloud with noise to form noisy state representations corresponding to multiple diffusion time steps, specifically includes: Let the target state point cloud be It represents the target configuration of the object operated by the robot after the operation is completed, through the target state point cloud. A forward diffusion process is constructed by applying progressive noise perturbation; specifically, during the forward diffusion process, Gaussian noise is progressively added to the target state point cloud to obtain noisy state representations at different time steps. Its expression is: ; in, Indicates the diffusion time step. The coefficients are related to the time step. The noise follows a standard Gaussian distribution. As the time step increases, the target state point cloud gradually transitions from a well-defined geometric shape to an approximately randomly distributed state.

[0026] Based on this, a reverse denoising process corresponding to the forward diffusion process is constructed to gradually recover the target state point cloud from the noisy state; specifically, through... Figure 2 The target state prediction network shown learns a conditional denoising function to perform noisy prediction of the noisy state at the current time step: ; in, This represents the noisy point cloud state at the current time step. Representing scene point clouds, This represents the point cloud of the manipulated object. This indicates task description information. Indicates a time step. The denoising function to be learned.

[0027] The denoising function to be learned is used to predict the noise component in a noisy state given scene geometric information, object structure information, and task semantic information. Based on the noise prediction result, the noisy state is updated step by step to obtain the state representation for the next time step. The update process is as follows:

[0028] in, It is a random noise variable.

[0029] By iterating the inverse denoising process, the predicted result of the target state point cloud is gradually recovered from the initial noise state.

[0030] Through the above diffusion generation process, the model can learn the distribution characteristics of the target state under given scene constraints and task semantics, and generate a target state point cloud that meets the environmental geometric relationship and task requirements during the inference stage, providing a foundation for subsequent structure-preserving constraints and target state optimization.

[0031] Understandably, the target state point cloud is subjected to progressive noise perturbation to generate multiple noisy state representations with different noise intensities; a diffusion time step sequence is constructed to gradually transition the target state from a low-noise state to a high-noise state; and a set of state representations corresponding to the time steps is established for state recovery during the training and inference phases.

[0032] Further, in step S103: the scene point cloud, the point cloud of the manipulated object, the task description information, and the noisy state representation are input into the trained target state prediction network, and the noisy state representation is subjected to stepwise conditional denoising prediction to obtain the noise prediction result at the current time step, specifically including: S103-1: Perform hierarchical feature encoding on the scene point cloud to obtain the point cloud features of interactive objects; S103-2: Perform joint feature encoding on the point cloud of the manipulated object and the noisy state representation to obtain the point cloud features of the manipulated object; S103-3: Semantically encode the task description information to obtain task text features; S103-4: Encode the diffusion time step to generate diffusion time step features; S103-5: Concatenate the diffusion time step features, task text features, and interactive object point cloud features to obtain a conditional feature sequence; convert the conditional feature sequence into a key sequence and a value sequence; convert the operation object point cloud features into a query sequence. S103-6: Based on the key sequence, value sequence, and query sequence, a cross-attention mechanism is used to calculate the feature fusion result; S103-7: Perform point-level noise decoding on the feature fusion results to obtain noise prediction results corresponding to each point of the point cloud of the manipulated object.

[0033] The input to the trained target state prediction network includes noisy target point clouds. Manipulating point clouds of objects Scene point cloud Task description information and diffusion time step In this process, the task description information is encoded into a task semantic embedding, the diffusion time step is encoded into a time feature, the point cloud of the manipulated object and the point cloud of the scene are extracted using different encoding networks, and then multimodal conditional modeling is performed through a cross-attention feature fusion network. Finally, a point-level noise decoder outputs the noise prediction results corresponding to each point of the point cloud of the manipulated object.

[0034] Further, step S103-1: performing hierarchical feature encoding on the scene point cloud to obtain interactive object point cloud features, specifically including: S103-11: First, the semantic embedding of the object name in the scene is extracted using the CLIP text encoder. Then, the semantic embedding is copied and expanded to match the number of points in the point cloud of the object in the scene. Finally, it is merged with the point cloud of the object in the scene in the channel dimension to obtain the first intermediate variable. S103-12: Input the first intermediate variable into the hierarchical point cloud perceptual coding network to obtain the second intermediate variable; S103-13: Input the point cloud of the scene objects into the multilayer perceptron to obtain the third intermediate variable; S103-14: Summing the second and third intermediate variables yields the point cloud features of the interactive object.

[0035] The hierarchical point cloud perception coding network includes: The point cloud serialization encoding module, the MLP point cloud embedding module, S sequentially connected feature encoding modules, and the output end are connected in sequence. Each feature encoding module includes a grid pooling compression unit and a random serialization allocation unit connected in sequence; The random serialization allocation unit includes: n block sub-units connected in sequence; Each block subunit includes: a point cloud serialization encoding subunit, a sequence block processing subunit, a sparse convolution subunit, an adder, a block attention subunit, and an MLP feature mapping subunit, which are connected in series; the input of the sparse convolution subunit is also connected to the input of the adder.

[0036] Among them, the point cloud serialization encoding module is used to map an unordered point cloud into a one-dimensional ordered sequence; The point cloud serialization and encoding module is used to convert unordered point cloud data into a one-dimensional sequence representation with a sequential structure. It first performs grid discretization processing on the input point cloud coordinates, then calculates the sequence code of each point based on the space filling curve (such as Z-order or Hilbert curve), and sorts the points or constructs a sequence index mapping according to the code, outputting a serialized point cloud or corresponding index. The point cloud serialization and encoding module utilizes the local neighborhood preservation property of the space filling curve to map the three-dimensional spatial proximity relationship into a one-dimensional sequence relationship, thereby achieving a structured representation of the point cloud while reducing computational complexity.

[0037] The MLP point cloud embedding module includes several sequentially connected linear layers, which are used to map the serialized point cloud to the feature space. The MLP point cloud embedding module is used to map the geometric information of the original point cloud to a high-dimensional feature space. It applies a multilayer perceptron transformation to the coordinates or attribute information of the input point cloud point by point, extracts point-level feature representations through multilayer linear mapping and nonlinear activation functions, and outputs high-dimensional point feature vectors. The MLP point cloud embedding module realizes the preliminary encoding of the local geometric attributes of the point cloud based on the point-by-point feature learning mechanism with shared weights.

[0038] Among them, the grid pooling compression unit is used to compress point cloud features; The grid pooling compression unit is used to perform spatial downsampling and feature compression on point clouds. It first divides the input point cloud into multiple spatial voxel units according to a preset grid size, then aggregates the point features within each voxel (such as max pooling or average pooling), and uses the voxel center or representative point as the output to generate a low-resolution point cloud and its corresponding features. The grid pooling compression unit achieves computational scale compression through spatial discretization and local aggregation, while maintaining the overall structural information.

[0039] The point cloud serialization encoding subunit is used to dynamically reorder the point cloud during feature encoding. It performs serialization encoding again on the spatial positions corresponding to the input point cloud features and generates new sorting indices based on different sequence patterns (such as various space filling curves or their transformations), and outputs the reordered point cloud sequence. The point cloud serialization encoding subunit enhances the model's ability to express spatial relationships and its generalization ability by introducing diverse sequence patterns.

[0040] The sequence block processing subunit is used to divide the serialized point cloud into multiple local subsequences. It divides the input sequence sequentially according to the preset block size, and ensures that the length of each subsequence is consistent through padding operations when necessary, and outputs several point cloud blocks. By dividing the global point cloud into local patches, the sequence block processing subunit enables subsequent calculations to be performed efficiently in the local scope, thereby reducing computational complexity and improving parallelism.

[0041] The sparse convolutional subunit is used to extract local spatial structure features of point clouds. It constructs a sparse neighborhood based on the input point cloud features and their spatial coordinates, and performs weighted convolution operations on the point features within the neighborhood to output an enhanced local feature representation. The sparse convolutional subunit uses sparse data structures to reduce invalid computations and supplements the spatial geometric information representation ability that may be weakened during the serialization process.

[0042] The block-based attention subunit is used to model the relationships between points within local point cloud blocks. It calculates the query vector, key vector, and value vector for each point feature within each block, and then performs weighted fusion of the point features within the block using an attention mechanism, outputting context-enhanced point features. Based on a local self-attention mechanism, the block-based attention subunit achieves local spatial relationship modeling while reducing the computational complexity of global attention. The weighted aggregation of point features within each point cloud block by the block-based attention subunit can be represented as follows:

[0043] in, , , These represent the query matrix, key matrix, and value matrix, respectively. This represents the point features within the block. , , For learnable parameters, For feature dimension, This represents the number of points within the block.

[0044] The MLP feature mapping subunit is used to perform nonlinear mapping and dimensional transformation on the attention output features. It expands, activates and compresses the input features through a multi-layer fully connected network and outputs the updated point feature representation.

[0045] For scene branches, scene conditions are determined by the point cloud of scene objects. Provided. Due to the significant scale differences of interactive objects across different tasks, directly using the same dense point-level encoding method as the manipulated object can easily distract the model's attention during feature fusion, weakening the contextual information truly relevant to the manipulated object. Therefore, for the point cloud of the interactive object, a... Figure 3 The feature extraction is performed using a hierarchical point cloud perceptual coding network based on PointTransformer-v3.

[0046] The hierarchical point cloud perception coding network includes point cloud serialization coding, grid pooling compression, random serialization allocation, 3D position coding, and block self-attention modules. Through point cloud serialization embedding and multi-layer feature compression and fusion, it finally outputs compact and information-rich interactive object point cloud features.

[0047] Specifically, firstly, a point cloud serialization encoding module is used to map the disordered point cloud into a one-dimensional ordered sequence; then, an embedding layer composed of multiple linear layers is used to map the serialized point cloud to the feature space; in subsequent consecutive feature encoding modules, grid pooling is first used to compress the point cloud features, then the serialized point cloud is divided into blocks, and different serialization curves are randomly assigned before multiple consecutive block attention modules to achieve implicit information propagation across blocks; in addition, sparse convolution is introduced before each attention layer to obtain enhanced conditional position encoding, and the position encoding is embedded into the point cloud features through skip connections.

[0048] The serialization curve, in this context, refers to a spatial filling mapping method that maps points in a multidimensional space to one-dimensional sequence indices. It transforms unordered point clouds into data representations with a sequential structure. In practice, the input point cloud coordinates are first quantized into a grid, discretizing the continuous space into integer grid coordinates. Then, the discrete coordinates are encoded based on a serialization curve, such as a Z-order curve or a Hilbert curve, generating a unique one-dimensional sequence index for each point. The serialized point cloud is obtained by sorting these indices or establishing an index mapping. This method utilizes the local neighborhood preservation property of the spatial filling curve, ensuring that spatially close points are also as adjacent as possible in the sequence, thus preserving spatial structure information while avoiding complex neighborhood searches. Its formula is:

[0049] in, Represents the three-dimensional coordinates of the input point. Indicates the grid quantization scale, This represents the discretization operation of rounding down. This represents the serialization curve encoding function. This represents the one-dimensional sequence index of the output.

[0050] After completing the hierarchical point cloud perceptual encoding, the interactive object point cloud containing only spatial coordinates is input into a multilayer perceptron for feature mapping. The mapping result is then added element-wise to the features extracted by the hierarchical point cloud perceptual encoding network, thereby achieving the fusion of spatial geometric information and high-level semantic features to obtain the final interactive background point cloud features. .

[0051] Specifically, after completing the hierarchical point cloud perceptual encoding, the interactive object point cloud containing only spatial coordinates will be... Inputting the data into a multilayer perceptron for point-by-point feature mapping, first, the coordinates of each point... Its geometric feature representation is obtained by encoding through a mapping function composed of several layers of linear transformations and nonlinear activation functions. Meanwhile, the hierarchical point cloud perceptual coding network outputs the high-level semantic features of the corresponding points. The two types of features are then aligned along the channel dimension and fused element-wise to obtain the fused point features. This process achieves joint modeling of the original spatial geometric information and deep semantic representation, and its computation process can be represented as follows:

[0052]

[0053] in, Indicates the first The three-dimensional coordinates of the points This represents the geometric features extracted by the multilayer perceptron. This represents the semantic features output by the hierarchical point cloud perceptual coding network. This represents the point features after fusion. and They represent the first Layer weight matrix and bias terms, Represents a non-linear activation function. This indicates the number of layers in the multilayer perceptron. The above method enables the effective fusion of spatial geometric information and high-level semantic features while maintaining the point-to-point correspondence.

[0054] Further, step S103-2: performing joint feature encoding on the point cloud of the manipulated object and the noisy state representation to obtain the point cloud features of the manipulated object, specifically including: For the operation object branch, the encoding of the operation object point cloud simultaneously extracts the object name semantics, object geometric structure, point-by-point spatial location information, and information on the noisy target state at the current time step: First, the semantic embedding of the name of the manipulated object is extracted using the CLIP text encoder. Then, the semantic embedding is copied and expanded to match the number of points in the point cloud of the manipulated object, and finally merged with the point cloud of the manipulated object in the channel dimension. Subsequently, the merged point cloud features of the manipulated object are modeled using a self-attention Transformer to extract the geometric structure information of the object through feature fusion between points. The self-attention Transformer consists of several Transformer processing blocks connected in sequence. Each Transformer processing block first stabilizes the feature distribution through layer normalization, then fuses context information through a multi-head self-attention module, and finally outputs the linearly transformed point cloud features through a feedforward network. Then, the point cloud of the manipulated object containing only spatial coordinates is input into a multilayer perceptron to expand the feature dimension, and the obtained coordinate features are added point by point to the point cloud features after linear transformation to obtain the first point cloud features of the manipulated object, so as to preserve point-by-point spatial position information while enhancing the context representation capability. Finally, a multilayer perceptron is used to process the noisy target point cloud. The feature is mapped to the same dimension as the point cloud feature of the manipulated object, and then added point by point to the first point cloud feature of the manipulated object to obtain the final point cloud feature of the manipulated object. .

[0055] Further, S103-3: Semantically encoding the task description information to obtain task text features, specifically including: First, review the task description information. Semantic encoding is performed. Since task instructions include manipulated objects, interactive objects, and interaction methods, the CLIP text encoder is used to encode the task description, resulting in a task semantic embedding, which represents the operation target, interactive objects, and semantic relationships of the current task.

[0056] Further, S103-4: Encoding the diffusion time step to generate diffusion time step features, specifically including: diffusion time step Sine-cosine position coding is used for time step representation, which is then mapped through a multilayer perceptron and injected into the target state prediction network. This enables the target state prediction network to explicitly distinguish noise distribution at different diffusion stages and provides temporal conditions for subsequent conditional denoising. For the diffusion time step... The encoded vector Represented as:

[0057]

[0058] in, , As the feature embedding dimension, sine and cosine functions of different frequencies together constitute a multi-scale continuous representation of the time step.

[0059] After obtaining the point cloud features of the manipulated object, the point cloud features of the interactive object, the task text features, and the diffusion time step features, the data enters the feature fusion backbone network. In this embodiment, the feature fusion backbone uses a cross-attention Transformer to achieve noise prediction corresponding to each point in the manipulated object point cloud.

[0060] Further, in step S103-5: the diffusion time step features, task text features, and interactive object point cloud features are concatenated to obtain a conditional feature sequence; the conditional feature sequence is then converted into a key sequence and a value sequence; the operational object point cloud features are converted into a query sequence, specifically including: Diffusion time step characteristics and task text features Point cloud features of interactive objects Concatenate the tokens to form a conditional feature sequence; Then, in each cross-attention Transformer block, the feature distribution is first stabilized by layer normalization, and then the point cloud features of the manipulated object are processed by a linear layer. Convert to query sequence ; The conditional feature sequence, composed of interactive object point cloud features, task text features, and diffusion time step features, is converted into a key sequence. Sum sequence .

[0061] Specifically, the interactive object point cloud features, task text features, and diffusion time step features are combined to form a unified conditional feature. ,in, The number of tokens in the sequence. This refers to the channel dimension of the conditional features. The conditional features are then processed using... and Convert to key sequence Sum sequence ,in and This refers to a linear network layer in the cross-attention model.

[0062] This means that the Key and Value in cross-attention do not come solely from the scene point cloud, but are composed of scene features, text features, and time step features.

[0063] Furthermore, S103-6: Based on the key sequence, value sequence, and query sequence, a cross-attention mechanism is used to calculate the feature fusion result, specifically including: The feature fusion result is calculated using the cross-attention mechanism, enabling the point cloud features of the manipulated object to select the most relevant contextual information from the interactive object point cloud features under the constraints of task semantics and diffusion stage conditions. The cross-attention calculation can be expressed as:

[0064] in, For the dimension of the attention subspace; After feature interaction and fusion through multiple layers of cross-attention blocks, a high-level semantic feature representation that incorporates global contextual information is obtained.

[0065] Further, S103-7: performing point-level noise decoding on the feature fusion result to obtain noise prediction results corresponding to each point of the point cloud of the manipulated object, specifically including: After obtaining the output of the cross-attention feature fusion network, the process proceeds to the point-level noise decoding stage. Specifically, a multilayer perceptron is used to decode the fused high-level semantic features, enabling point-by-point feature reconstruction of the object point cloud under the target state. Furthermore, the diffusion noise corresponding to each point is predicted, resulting in noise prediction results corresponding to each point of the object point cloud. .

[0066] Further, S103: Based on the noise prediction result at the current time step, the noisy state is updated step by step to obtain the target state point cloud, specifically including: The noise prediction result is then input into the backdiffusion update formula to update the noisy target point cloud at the current time step, thereby gradually restoring the target state point cloud.

[0067] in, For the diffusion process A noisy point cloud of objects in a step; , This represents the variance of the Gaussian noise injected into the data during the diffusion process. , This is the result of noise prediction. This is a random noise variable used to increase the randomness of the back diffusion process and improve the expressive power of the model. Indicates the first The random noise intensity that needs to be added during the step-by-step diffusion process is used to ensure that the generation process conforms to the true probability distribution, rather than a deterministic process.

[0068] The noise prediction results are input into the backdiffusion update formula to update the noisy target point cloud at the current time step, thereby gradually recovering the target state point cloud. The target state point cloud is used to represent the object configuration after the robot operation is completed.

[0069] Through the aforementioned conditional modeling and multimodal feature fusion process, the network can simultaneously consider scene geometric constraints, object structural information, and task semantics during the diffusion generation process, thereby achieving accurate prediction of the target state point cloud and providing a foundation for the subsequent application of structural consistency constraints.

[0070] Furthermore, the target state prediction network uses a loss function during the training phase that is a weighted sum of two parts. The first part is a structural consistency constraint term established based on the operational point cloud; the other part is a noise prediction loss term, specifically including: ; ; ; in, These are the weighting coefficients for structural consistency constraints. This represents a structural consistency constraint.

[0071] To avoid geometric distortion caused by point-by-point independent updates during the diffusion generation process, structural consistency constraints are introduced into the point cloud of the manipulated object to describe and maintain the internal geometric relationships of the object.

[0072] Specifically, let the initial point cloud of the manipulated object be... It represents the three-dimensional geometric structure of an object in its initial state; let the predicted target state point cloud be... Since the target state point cloud and the initial point cloud maintain a one-to-one correspondence at the point level, structural consistency constraints can be constructed based on the spatial relationship between point pairs.

[0073] During the constraint construction process, the Euclidean distance between any two points in the initial point cloud of the object is first calculated to characterize the object's intrinsic geometric structure.

[0074] Specifically, for any pair of points Its initial geometric relationship is expressed as In the target state point cloud, the geometric relationship between corresponding point pairs is represented as follows: Based on the above point-to-point relationships, a structural consistency constraint term is constructed to ensure that the generated result remains consistent with the initial structure during the denoising process. Its form is expressed as: ; Through structural consistency constraints This ensures that the relative distances between points in the target state point cloud are maintained as much as possible during the generation process, thereby preserving the overall geometric stability of the object. This structural consistency constraint stems from modeling the rigid or near-rigid body characteristics of the object. In most robotic manipulation tasks, the internal structure of the manipulated object does not undergo significant deformation during movement; therefore, the distances between its points should remain constant or vary within an acceptable range.

[0075] Furthermore, to reduce computational complexity, in practical implementations, the set of point pairs can be subsampled, selecting only some key point pairs or point pairs within a local neighborhood to construct constraint relationships, thereby improving computational efficiency while ensuring structural expressiveness. Simultaneously, this structural consistency constraint can be introduced as an additional loss term during the training phase to guide the network in learning a target state distribution that satisfies geometric consistency.

[0076] By means of the above method, a structural consistency constraint is established between the initial state and the target state of the point cloud of the manipulated object, which provides a basis for applying a structure preservation mechanism in the subsequent diffusion denoising process, thereby effectively suppressing the problems of local point drift and overall structural distortion that occur during the generation process.

[0077] The process of applying structural consistency constraints during the denoising update process and updating the target state while preserving the structure is as follows: In this embodiment, during the back diffusion denoising process, structural consistency constraints are introduced as an additional loss term into the model training process, which together with the noise prediction target of the diffusion model constitute a joint optimization target.

[0078] Specifically, during the training phase, for any diffusion time step First, a noisy target point cloud is constructed based on the forward diffusion process. Subsequently, a conditional denoising prediction network is used to predict the noise, resulting in... Based on this, the noise prediction loss of the diffusion model is constructed as follows:

[0079] At the same time, structural consistency constraints are introduced. This is used to ensure that the geometric structure of the target state point cloud remains consistent. When constructing the structural consistency constraint, to avoid multiple cyclic back-diffusion processes for prediction... The problems of low optimization efficiency and high memory consumption of backpropagation caused by this can be solved by using only one denoising process in the following way. : ; This process can be restored through a single noise reduction step. Although its denoising effect cannot reach that of multiple reverse diffusion denoising processes, it can significantly improve optimization efficiency and reduce memory usage. The ultimate training objective is a weighted combination of the two: ;in, These are the weighting coefficients for structural consistency constraints. Through the above joint optimization, the model can maintain the geometric relationships between points inside the manipulated object while learning the target state distribution, thereby avoiding structural distortions in the generated results.

[0080] The structural consistency constraint is constructed by calculating the spatial geometric relationship between points in the point cloud of the operating object, and is used to constrain the geometric relationship between corresponding points in the target state point cloud; wherein, the spatial geometric relationship includes the distance relationship between point pairs.

[0081] The structural consistency constraint term is weighted and combined with the noise prediction loss of the diffusion model as an additional loss term, and used as a training condition to train the target state prediction network.

[0082] During model training, the estimation results of the target state point cloud are obtained through single-step conditional denoising prediction, which are then used to calculate the structural consistency constraints.

[0083] The target state prediction network weight update for structure preservation includes: calculating the structural bias of the generated point cloud after each denoising iteration; adjusting the generated results based on the structural bias; suppressing the independent drift of local points; and maintaining the overall target state to satisfy rigid or near-rigid body structural characteristics.

[0084] After the diffusion process is complete, a target state point cloud is generated, as follows: In this embodiment, the initial target state point cloud is first obtained by sampling from a Gaussian distribution. The initial state is a point cloud representation of complete noise. The point cloud of the manipulated object is kept consistent in the number of points and is used as the initial input state for the diffusion reverse process.

[0085] Subsequently, using the initial noise point cloud As input, a target state point cloud is gradually generated through a back-diffusion denoising process. When the denoising update process reaches a preset termination condition, the final generated target state point cloud is output as the prediction result. The termination condition is reaching a preset diffusion time step. Once the termination condition is met, the currently generated point cloud state is determined as the target point cloud state. .

[0086] In this embodiment, the target state point cloud Point cloud of the object being input Maintaining a one-to-one correspondence at the point level allows for direct use in subsequent geometric calculations. This target state point cloud represents the target spatial configuration of the object after the robot completes its task. Under structural consistency constraints, it maintains stable internal geometric relationships within the object, while simultaneously satisfying spatial relationship requirements with the environment under constraints of scene features and task semantics.

[0087] Furthermore, the method also includes: S104: Calculate the target pose of the object being operated based on the correspondence between the target state point cloud and the current object point cloud.

[0088] The generated target state point cloud can be post-processed to improve the stability and usability of the results. The post-processing includes, but is not limited to: performing coordinate system transformation and restoration on the point cloud, mapping the point cloud from the normalized coordinate system back to the real-world coordinate system; performing scale restoration on the point cloud to restore the true size of the object; and performing smoothing or denoising processing on the point cloud to reduce local disturbances caused by prediction errors.

[0089] After obtaining the target state point cloud, the current point cloud of the object to be manipulated can be used as a basis. With the target state point cloud The correspondence between these points is used to further calculate the rigid body pose transformation of the manipulated object. Specifically, the rotation matrix is ​​solved by minimizing the Euclidean distance between the two sets of corresponding points. With translation vector Its optimization problem can be expressed as:

[0090] The optimization problem can be solved using the singular value decomposition method to obtain the target pose transformation. :

[0091] The target state point cloud is converted into a pose representation that can be used for robot execution using the above method.

[0092] Finally, the target state point cloud or the corresponding target pose is output to the subsequent modules for robot operation planning and execution, thereby achieving an effective connection from target state prediction to robot control.

[0093] Example 2 This embodiment provides a structure-preserving robot operation target state generation system, including: The acquisition module is configured to acquire the operation scene information of the robot operation task, wherein the operation scene information includes: scene point cloud, operation object point cloud and task description information; The construction module is configured to: construct the diffusion generation process of the target state, and gradually perturb the target state point cloud with noise to form a noisy state representation corresponding to multiple diffusion time steps; The generation module is configured to: input the scene point cloud, the point cloud of the manipulated object, the task description information, and the noisy state representation into the trained target state prediction network; perform progressive conditional denoising prediction on the noisy state representation to obtain the noise prediction result at the current time step; based on the noise prediction result at the current time step, progressively update the noisy state to obtain the target state point cloud; and use the target state point cloud as the target state of the robot's operation task. The target state prediction network uses a loss function during the training phase that is a weighted sum of two parts: the first part is a structural consistency constraint term established based on the operational point cloud; the other part is a noise prediction loss term.

[0094] It should be noted that the acquisition module, construction module, and generation module described above correspond to steps S101 to S103 in Embodiment 1. The examples and application scenarios implemented by these modules and their corresponding steps are the same, but they are not limited to the content disclosed in Embodiment 1. It should also be noted that these modules, as part of the system, can be executed in a computer system such as a set of computer-executable instructions.

[0095] The descriptions of each embodiment in the above embodiments have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0096] The proposed system can be implemented in other ways. For example, the system embodiments described above are merely illustrative, and the division of modules described above is only a logical functional division. In actual implementation, there may be other division methods. For example, multiple modules may be combined or integrated into another system, or some features may be ignored or not executed.

[0097] Example 3 This embodiment also provides an electronic device, including: one or more processors, one or more memories, and one or more computer programs; wherein, the processor is connected to the memory, and the one or more computer programs are stored in the memory. When the electronic device is running, the processor executes the one or more computer programs stored in the memory to cause the electronic device to perform the method described in Embodiment 1.

[0098] It should be understood that in this embodiment, the processor can be a central processing unit (CPU), or it can be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc.

[0099] Memory may include read-only memory and random access memory, and provides instructions and data to the processor. A portion of memory may also include non-volatile random access memory. For example, memory may also store information about the device type.

[0100] In the implementation process, each step of the above method can be completed by the integrated logic circuits in the processor hardware or by software instructions.

[0101] The method in Embodiment 1 can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor. The software modules can reside in readily available storage media in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. This storage medium is located in memory; the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above method. To avoid repetition, a detailed description is not provided here.

[0102] Those skilled in the art will recognize that the units and algorithm steps described in connection with the various examples of this embodiment can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this invention.

[0103] Example 4 This embodiment also provides a computer-readable storage medium for storing computer instructions, which, when executed by a processor, complete the method described in Embodiment 1.

[0104] Example 5 This embodiment also provides a computer program product, including a computer program that, when executed by a processor, implements the method in Embodiment 1.

[0105] The present invention also provides at least one computer program product tangibly stored on a non-transitory computer-readable storage medium. The computer program product includes computer-executable instructions, such as instructions included in program modules, which execute in a device on a target real or virtual processor to perform the processes / methods described above. Typically, program modules include routines, programs, libraries, objects, classes, components, data structures, etc., that perform specific tasks or implement specific abstract data types. In various embodiments, the functionality of program modules can be combined or divided among program modules as needed. The machine-executable instructions for the program modules can execute within a local or distributed device. In a distributed device, the program modules can reside in both local and remote storage media.

[0106] The computer program code used to implement the methods of the present invention may be written in one or more programming languages. This computer program code may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the computer or other programmable data processing device, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a computer, partially on a computer, as a stand-alone software package, partially on a computer and partially on a remote computer, or entirely on a remote computer or server.

[0107] In the context of this invention, computer program code or related data may be carried by any suitable carrier to enable a device, apparatus, or processor to perform the various processes and operations described above. Examples of carriers include signals, computer-readable media, and the like. Examples of signals may include electrical, optical, radio, sound, or other forms of propagation signals, such as carrier waves, infrared signals, etc.

[0108] Those skilled in the art will recognize that the units and algorithm steps described in conjunction with the embodiments herein can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0109] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for generating target states for robot operations with structure preservation, characterized in that, include: Obtain the operation scene information of the robot operation task, wherein the operation scene information includes: scene point cloud, operation object point cloud and task description information; The diffusion generation process of the target state is constructed by progressively perturbing the target state point cloud with noise to form noisy state representations corresponding to multiple diffusion time steps; The scene point cloud, the object point cloud, the task description information, and the noisy state representation are input into the trained target state prediction network. The noisy state representation is subjected to progressive conditional denoising prediction to obtain the noise prediction result at the current time step. Based on the noise prediction result at the current time step, the noisy state is progressively updated to obtain the target state point cloud. The target state point cloud is used as the target state of the robot operation task. The target state prediction network uses a loss function during the training phase that is a weighted sum of two parts: the first part is a structural consistency constraint term established based on the operational point cloud; the other part is a noise prediction loss term.

2. The method for generating a target state for robot operation with structure preservation as described in claim 1, characterized in that, The diffusion generation process of the target state is constructed by progressively perturbing the target state point cloud with noise to form noisy state representations corresponding to multiple diffusion time steps, specifically including: Let the target state point cloud be It represents the target configuration of the object operated by the robot after the operation is completed, through the target state point cloud. A forward diffusion process is constructed by applying progressive noise perturbation; specifically, during the forward diffusion process, Gaussian noise is progressively added to the target state point cloud to obtain noisy state representations at different time steps. Its expression is: ； in, Indicates the diffusion time step. The coefficients are related to the time step. The random noise follows a standard Gaussian distribution; as the time step increases, the target state point cloud gradually transitions from a well-defined geometric shape to an approximately randomly distributed state.

3. The method for generating a target state for robot operation with structure preservation as described in claim 1, characterized in that, The scene point cloud, the object point cloud, the task description information, and the noisy state representation are input into the trained target state prediction network. The noisy state representation is then subjected to stepwise conditional denoising prediction to obtain the noise prediction result at the current time step, specifically including: Hierarchical feature encoding is performed on the scene point cloud to obtain the point cloud features of interactive objects; Joint feature encoding is performed on the point cloud of the manipulated object and the noisy state representation to obtain the point cloud features of the manipulated object; Semantic encoding is performed on the task description information to obtain task text features; Encode the diffusion time step to generate diffusion time step features; The diffusion time step features, task text features, and interactive object point cloud features are concatenated to obtain a conditional feature sequence, which is then converted into a key sequence and a value sequence; the interactive object point cloud features are converted into a query sequence. Based on the key sequence, value sequence, and query sequence, a cross-attention mechanism is used to calculate the feature fusion result. Point-level noise decoding is performed on the feature fusion results to obtain noise prediction results corresponding to each point of the point cloud of the manipulated object.

4. The method for generating a target state for robot operation with structure preservation as described in claim 3, characterized in that, Hierarchical feature encoding is performed on the scene point cloud to obtain the point cloud features of interactive objects, specifically including: First, the semantic embedding of the object names in the scene is extracted using the CLIP text encoder. Then, the semantic embedding is copied and expanded to match the number of points in the point cloud of the objects in the scene. Finally, it is merged with the point cloud of the objects in the scene in the channel dimension to obtain the first intermediate variable. The first intermediate variable is input into the hierarchical point cloud perceptual coding network to obtain the second intermediate variable; The point cloud of objects in the scene is input into a multilayer perceptron to obtain a third intermediate variable; The second and third intermediate variables are summed to obtain the point cloud features of the interactive object.

5. The method for generating a target state for robot operation with structure preservation as described in claim 3, characterized in that, Joint feature encoding is performed on the point cloud of the manipulated object and the noisy state representation to obtain the point cloud features of the manipulated object, specifically including: For the operation object branch, the encoding of the operation object point cloud simultaneously extracts the object name semantics, object geometric structure, point-by-point spatial location information, and information on the noisy target state at the current time step: First, the semantic embedding of the name of the manipulated object is extracted using the CLIP text encoder. Then, the semantic embedding is copied and expanded to match the number of points in the point cloud of the manipulated object, and finally merged with the point cloud of the manipulated object in the channel dimension. Subsequently, the merged point cloud features of the manipulated object are modeled using a self-attention Transformer to extract the geometric structure information of the object through feature fusion between points. The self-attention Transformer consists of several Transformer processing blocks connected in sequence. Each Transformer processing block first stabilizes the feature distribution through layer normalization, then fuses context information through a multi-head self-attention module, and finally outputs the linearly transformed point cloud features through a feedforward network. Then, the point cloud of the manipulated object containing only spatial coordinates is input into a multilayer perceptron to expand the feature dimension, and the obtained coordinate features are added point by point to the point cloud features after linear transformation to obtain the first point cloud features of the manipulated object, so as to preserve point-by-point spatial position information while enhancing the context representation capability. Finally, a multilayer perceptron is used to process the noisy target point cloud. The feature is mapped to the same dimension as the point cloud feature of the manipulated object, and then added point by point to the first point cloud feature of the manipulated object to obtain the final point cloud feature of the manipulated object. .

6. The method for generating a target state for robot operation with structure preservation as described in claim 1, characterized in that, Based on the noise prediction results at the current time step, the noisy state is updated step by step to obtain the target state point cloud, specifically including: The noise prediction result is then input into the backdiffusion update formula to update the noisy target point cloud at the current time step, thereby gradually restoring the target state point cloud. ； in, For the diffusion process A noisy point cloud of objects in a step; , This represents the variance of the Gaussian noise injected into the data during the diffusion process. , This is the result of noise prediction; As a random noise variable, it is used to increase the randomness of the back diffusion process and improve the expressive power of the model; Indicates the first The random noise intensity that needs to be added during the step-by-step diffusion process is used to ensure that the generation process conforms to the true probability distribution, rather than a deterministic process.

7. The method for generating a target state for a structure-preserving robot operation as described in claim 1, characterized in that, The target state prediction network uses a loss function during the training phase that is a weighted sum of two parts. The first part is a structural consistency constraint term established based on the operational point cloud. The other part is the noise prediction loss term, which specifically includes: ； ； ； in, These are the weighting coefficients for structural consistency constraints. This represents a structural consistency constraint.

8. A structure-preserving robot operation target state generation system, characterized in that, include: The acquisition module is configured to acquire the operation scene information of the robot operation task, wherein the operation scene information includes: scene point cloud, operation object point cloud and task description information; The construction module is configured to: construct the diffusion generation process of the target state, and gradually perturb the target state point cloud with noise to form a noisy state representation corresponding to multiple diffusion time steps; The generation module is configured to: input the scene point cloud, the point cloud of the manipulated object, the task description information, and the noisy state representation into the trained target state prediction network; perform progressive conditional denoising prediction on the noisy state representation to obtain the noise prediction result at the current time step; based on the noise prediction result at the current time step, progressively update the noisy state to obtain the target state point cloud; and use the target state point cloud as the target state of the robot's operation task. The target state prediction network uses a loss function during the training phase that is a weighted sum of two parts: the first part is a structural consistency constraint term established based on the operational point cloud; the other part is a noise prediction loss term.

9. An electronic device, characterized in that it comprises: Memory is used to store computer-readable instructions in a non-transitory manner. as well as Processor, for executing the computer-readable instructions, When the computer-readable instructions are executed by the processor, they perform the method described in any one of claims 1-7.

10. A storage medium, characterized in that, Non-transitory storage of computer-readable instructions, wherein when the non-transitory computer-readable instructions are executed by a computer, the method of any one of claims 1-7 is performed.

Images