A robot multi-modal candidate motion trajectory generation method, medium and device

Abstract

This application discloses a method, medium, and device for generating multimodal candidate motion trajectories for a robot. The method includes the following steps: acquiring the hierarchical point cloud structure features of the target object and the state features of the robot, and calculating the spatial relationship features of the robot relative to each level of sub-modules of the target object; performing feature fusion to obtain interaction latent variables representing interaction intentions; using the obtained interaction latent variables as conditions to generate multiple diverse short trajectory latent variables and connecting the trajectories to obtain multiple initial motion trajectories; and applying safety constraints to each initial motion trajectory to obtain multimodal candidate motion trajectories. The medium and device are used to implement the above method. The beneficial effects of this application are: by fusing the spatial relationships between the robot and each level of sub-modules of the target object to generate interaction latent variables, and using these latent variables as conditions, multiple diverse short trajectory latent variables are generated in parallel using inverse denoising, achieving one-time rapid generation of multimodal candidate trajectories.

Description

Technical Field

[0001] This application relates to the field of robotics technology, and in particular to a method, medium, and device for generating multimodal candidate motion trajectories for robots. Background Technology

[0002] Robot trajectory planning is a core technology for enabling robots to autonomously perform object interaction operations; existing mainstream technologies can be divided into two main categories. The first is sampling search methods, such as Rapid Expanding Random Tree (RRT), Optimal Rapid Expanding Random Tree (RRT*), and Probabilistic Path Map (PRM), which construct feasible paths through random sampling. The second is optimization solution methods, such as Covariant Hamiltonian Optimization (CHOMP), Trajectory Optimization (TrajOpt), and Model Predictive Control (MPC), which solve for the optimal trajectory based on the objective function and constraints.

[0003] In interaction scenarios involving hierarchical objects such as drawers, cabinet doors, buttons, and covers, robots need to perform differentiated operations for different sub-modules. Existing technologies have the following drawbacks in practical applications: they only output a single trajectory at a time, lack multi-strategy candidate capabilities, and cannot handle dynamic obstacles and retry requirements for failed operations. They struggle to fully model the spatial relationships between the robot and the target object's hierarchical sub-modules, resulting in insufficient adaptability to complex interaction tasks. Searching and solving for complex obstacles is time-consuming, making it difficult to meet real-time requirements. Furthermore, they lack a high-probability trajectory generation mechanism based on prior historical skills, leading to low trajectory success rate and stability.

[0004] Therefore, there is an urgent need for a technical solution that can quickly model the spatial relationship between robots and target objects and generate high-probability feasible candidate trajectories in batches. Summary of the Invention

[0005] One objective of this application is to provide a method for generating multimodal candidate motion trajectories for robots that can solve at least one of the defects in the aforementioned background art.

[0006] Another object of this application is to provide a computer-readable storage medium capable of implementing a method for generating multimodal candidate motion trajectories for robots that addresses at least one of the deficiencies in the aforementioned background art.

[0007] Another object of this application is to provide an electronic device capable of implementing a method for generating multimodal candidate motion trajectories for robots that addresses at least one of the deficiencies in the aforementioned background art.

[0008] To achieve at least one of the above objectives, one aspect of this application provides a method for generating multimodal candidate motion trajectories for a robot, comprising the following steps: S100: Based on the constructed unified coordinate system, obtain the hierarchical point cloud structure features of the target object and the state features of the robot, and calculate the spatial relationship features of the robot relative to each level of sub-modules of the target object. S200: The hierarchical point cloud structure features of the target object, the state features of the robot, and the spatial relationship features of the calculation are fused to obtain the latent interactive variables that represent the interactive intention. S300: Using the obtained interactive latent variables as conditions, multiple diverse short trajectory latent variables are gradually recovered from multiple sets of noisy short trajectory latent variables with added random noise through reverse denoising; the obtained short trajectory latent variables are expanded into continuous motion sequences and trajectory connections are performed to obtain multiple initial motion trajectories; S400: Apply safety constraints to each initial motion trajectory to obtain candidate motion trajectories for multimodal motion.

[0009] Preferably, in step S100, the hierarchical point cloud structure features of the target object include an overall bounding box and at least one hierarchical sub-module bounding box, each bounding box containing a preset number of three-dimensional points; the robot's state features include initial pose, base center point, and frontal unit vector; the spatial relationship features of the robot relative to each hierarchical sub-module of the target object include distance features, orientation angle, and the target object's operation surface normal vector; wherein, the distance features are calculated using the center point coordinates of each hierarchical sub-module bounding box and the base center point coordinates; the orientation angle is calculated using the direction vector constructed from the center point coordinates of each hierarchical sub-module bounding box and the base center point coordinates and the robot's frontal unit vector.

[0010] Preferably, in step S200, a cross-attention mechanism is used for feature fusion, specifically including the following process: linearly projecting the robot's state features as query Q; fusing the hierarchical point cloud structure features and spatial relationship features of the target object, and linearly projecting the fused features as key K and value V respectively; calculating attention based on the obtained Q, K, and V, and passing the calculated attention through a feedforward neural network, residual connections, and layer normalization to form the final interaction features; pooling the obtained interaction features to obtain fixed-dimensional interaction latent variables.

[0011] Preferably, in step S300, the acquisition of short trajectory latent variables includes the following process: acquiring initial short trajectory latent variables and continuously adding Gaussian noise to the initial short trajectory latent variables to obtain a single set of noisy short trajectory latent variables; obtaining multiple sets of different noisy short trajectory latent variables based on the different randomly added Gaussian noise; constructing an inverse denoising model for noise prediction of each set of noisy short trajectory latent variables, with the goal of minimizing the error between the predicted noise and the actual added Gaussian noise, and training the inverse denoising model; using the interaction latent variable as a condition, using the trained inverse denoising model to perform inverse iterative denoising on multiple sets of different noisy short trajectory latent variables to obtain multiple sets of different short trajectory latent variables.

[0012] Preferably, at any time step t, the latent variable z of the short-track noise after adding Gaussian noise is... traj,t The expression is: ; ; In the formula, α t Represents the noise scheduling parameter, z traj,0 Represents the initial short trajectory latent variables; This represents standard Gaussian noise, s represents the time index, and β represents standard Gaussian noise. s This indicates the noise addition rate parameter.

[0013] Preferably, when iteratively denoising the short-trajectory latent variables of noise, the interaction latent variables used as conditions are injected into the reverse denoising model through cross-attention or feature concatenation, guiding the generation of interaction intentions. Figure 1 The resulting short-trajectory latent variables.

[0014] Preferably, when iteratively denoising the short-track latent variables of noise, the process starts from the short-track latent variable of noise corresponding to the maximum time step and proceeds in reverse iterative denoising towards the minimum time step. The specific expression is as follows: ; ; In the formula, and Let α represent the denoised short-trajectory latent variables of the noise at time steps t and t-1, respectively. t and α s All represent noise scheduling parameters. This represents the cumulative noise scheduling parameter, where 's' represents the time index. σ represents the noise predicted by the inverse denoising model. t denoted by , where z represents the variance of Gaussian noise, and z represents the standard Gaussian noise.

[0015] Preferably, in step S400, the safety constraints include collision constraints, velocity constraints, and curvature smoothing constraints.

[0016] Another aspect of this application provides a computer-readable storage medium storing a computer program; when the computer program is executed by a processor, it implements the above-described method for generating multimodal candidate motion trajectories for robots.

[0017] Another aspect of this application provides an electronic device, including a processor and a memory; the memory is used to store a computer program, and the processor is used to execute the computer program to implement the above-described method for generating multimodal candidate motion trajectories for robots.

[0018] Compared with the prior art, the beneficial effects of this application are as follows: (1) This application generates interactive latent variables by integrating the spatial relationship between the robot and the target object at each level of sub-modules, and uses the latent variables as conditions to generate multiple diverse short trajectory latent variables in parallel using reverse denoising, thereby realizing the one-time rapid generation of multimodal candidate trajectories. Compared with the traditional single trajectory planning method, it significantly improves planning efficiency and multi-strategy response capability, and can effectively support the needs of dynamic obstacle avoidance and retry after operation failure.

[0019] (2) This application establishes the spatial relationship between the robot and the target object at each level of sub-modules, such as relative position, orientation and normal vector, and combines the high probability trajectory distribution generated by reverse denoising. The generated candidate trajectory has higher task adaptability and motion success rate. At the same time, the optimization of safety constraints ensures that the trajectory is collision-free, speed is compliant and motion is smooth, so that the output trajectory can be directly sent to the robot for execution, which significantly improves the operational stability and real-time performance of complex interactive tasks. Attached Figure Description

[0020] Figure 1 This is a schematic diagram of the overall working steps of this application. Detailed Implementation

[0021] The present application will now be further described in conjunction with specific embodiments. It should be noted that, in the description of this specification, the use of terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicates that the specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms should not be construed as necessarily referring to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. In addition, those skilled in the art can combine and integrate the different embodiments or examples described in this specification.

[0022] In the description of this application, it should be noted that the terms "center", "lateral", "longitudinal", "length", "width", "thickness", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", "clockwise", "counterclockwise", etc., which indicate the orientation and positional relationship based on the orientation or positional relationship shown in the accompanying drawings, are only for the convenience of describing this application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and should not be construed as limiting the specific protection scope of this application.

[0023] It should be noted that the terms "first," "second," etc., in the specification and claims of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0024] In this application, unless otherwise expressly specified and limited, the terms "installation," "connection," "joining," and "fixing," etc., should be interpreted broadly. For example, they can refer to a connection, a detachable connection, or an integral part; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication of two components or the interaction between two components. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0025] In this application, unless otherwise expressly specified and limited, "above" or "below" the second feature can include direct contact between the first and second features, or contact between the first and second features through another feature between them. Furthermore, "above," "over," and "on top" of the second feature includes the first feature being directly above or diagonally above the second feature, or simply indicates that the first feature is at a higher horizontal level than the second feature. "Below," "below," and "under" the second feature includes the first feature being directly below or diagonally below the second feature, or simply indicates that the first feature is at a lower horizontal level than the second feature.

[0026] The terms “comprising” and “having”, and any variations thereof, in the specification and claims of this application are intended to cover non-exclusive inclusion, for example, a process, method, system, product, or device that includes a series of steps or units is not necessarily limited to those steps or units that are explicitly listed, but may include other steps or units that are not explicitly listed or that are inherent to such process, method, product, or device.

[0027] One preferred embodiment of this application, such as Figure 1As shown, a method for generating multimodal candidate motion trajectories for a robot includes the following steps: S100: Based on the constructed unified coordinate system, obtain the hierarchical point cloud structure features of the target object and the state features of the robot, and calculate the spatial relationship features of the robot relative to each level sub-module of the target object.

[0028] It should be understood that traditional methods often only use the global point cloud data of the target object, ignoring the spatial prior relationships between the robot and each level of sub-modules of the target object in terms of position, distance, and orientation. This makes it difficult for subsequent action planning to distinguish specific operations. This step calculates the spatial relationship features between the robot and each level of sub-modules of the target object, providing spatial prior data for the subsequent attention mechanism. This enables the model to quickly locate the level of sub-module most relevant to the current robot posture, thereby improving the accuracy of subsequent trajectory generation.

[0029] S200: The hierarchical point cloud structure features of the target object, the state features of the robot, and the spatial relationship features of the calculation are fused to obtain the latent interactive variables that represent the interactive intention.

[0030] It should be understood that the main purpose of this step is to summarize the scattered and heterogeneous feature data from step S100 for use in subsequent trajectory generation guidance. By fusing multi-source information through a cross-attention mechanism, a compact interactive latent variable is obtained, enabling subsequent processes to control the trajectory generation process with extremely low dimensionality and highly abstract semantics.

[0031] S300: Using the obtained interactive latent variables as conditions, reverse the noise reduction process from multiple sets of noisy short trajectory latent variables with added random noise to gradually recover multiple diverse short trajectory latent variables; expand the obtained short trajectory latent variables into a continuous motion sequence and connect the trajectories to obtain multiple initial motion trajectories.

[0032] It should be noted that, compared to traditional trajectory generation methods, this step generates a large number of successful trajectory distributions through reverse denoising, which can reconstruct high-probability short trajectory latent variables from pure noise short trajectory latent variables; and multiple diverse results can be obtained at once by changing the initial added random noise, perfectly meeting the needs of multimodal candidates.

[0033] S400: Apply safety constraints to each initial motion trajectory to obtain candidate motion trajectories for multimodal motion.

[0034] It should be understood that while the trajectory generated by reverse denoising has a high probability of being feasible, it may still theoretically violate physical constraints (such as slightly penetrating obstacles or excessive acceleration). This step corrects the trajectory into a strictly collision-free, smooth, and speed-compliant robot command by applying safety constraints, so that it can be directly issued to the controller for execution without human intervention.

[0035] Understandably, this application generates interactive latent variables by integrating the spatial relationships between the robot and the target object at various levels of sub-modules, and uses these latent variables as conditions to generate multiple diverse short trajectory latent variables in parallel using inverse denoising, thereby achieving one-time rapid generation of multimodal candidate trajectories. Compared with traditional single-trajectory planning methods, it significantly improves planning efficiency and multi-strategy response capabilities, and can effectively support the needs of dynamic obstacle avoidance and retry after operation failure.

[0036] Meanwhile, this application establishes spatial relationships such as the relative positions, orientations, and normal vectors of the robot and the target object's sub-modules at each level. Combined with a high-probability trajectory distribution generated by reverse denoising, the generated candidate trajectories exhibit higher task adaptability and motion success rate. Furthermore, optimization of safety constraints ensures collision-free trajectory execution, speed compliance, and smooth motion, allowing the output trajectory to be directly sent to the robot for execution, significantly improving the operational stability and real-time performance of complex interactive tasks.

[0037] For ease of understanding, each step of the technical solution of this application will be described in detail below.

[0038] In a specific embodiment, when executing step S100, the origin of the constructed unified coordinate system is the scene reference point, the X-axis is the robot's forward direction, the Y-axis is the horizontal and vertical direction, and the Z-axis is the opposite direction of gravity. Under this unified coordinate system, the hierarchical point cloud structure features of the target object, the robot's state features, and the spatial relationship features between the two can be obtained. For ease of understanding, the specific acquisition process of each feature will be described in detail below.

[0039] The hierarchical point cloud structure features of the target object include an overall bounding box and at least one hierarchical sub-module bounding box, each containing a preset number of 3D points. In a specific example, the target object is a cabinet with drawers, and its hierarchical structure includes the overall cabinet, drawers, panels, buttons, slides, and lids; then the hierarchical point cloud structure feature O of the target object can be represented as: O = {B0, B1, B2, B3, B4, B5}; where B0 represents the bounding box of the overall cabinet, B1 represents the bounding box corresponding to the drawer, B2 represents the bounding box corresponding to the panel, B3 represents the bounding box corresponding to the button, B4 represents the bounding box corresponding to the slide, and B5 represents the bounding box corresponding to the lid. For each bounding box, the number of 3D points it contains can be determined according to the actual needs of those skilled in the art; since the bounding box has a cubic structure, in order to ensure the clarity of the bounding box structure, it needs to contain at least 14 3D points, namely 8 corner points and 6 face center points. For any i ∈ {0, 1, 2, 3, 4, 5} bounding box B i It can be represented as: B i ={pi1 p i2 ... p i14}; where p i1 to p i8 p represents the coordinates of the 8 corner points respectively. i9 to p i14 These represent the coordinates of the center points of the six faces.

[0040] It is important to note that the obtained hierarchical point cloud structural features of the target object are unordered point cloud data. To enable the robot to distinguish the relationship between the target object as a whole and its various hierarchical sub-modules, this unordered point cloud data needs to be transformed into structured, semantically meaningful hierarchical features. This allows the robot to clearly distinguish between the whole and its various hierarchical sub-modules, and to capture the geometric connections and spatial dependencies between these sub-modules. This provides a high-level feature representation rich in prior knowledge of the object's structure for subsequent interaction intent modeling. For ease of understanding, a specific example will be used to illustrate this in detail below.

[0041] In a specific example, for each hierarchical submodule B i 14 three-dimensional points {p i1 p i2 ... p i14 A shared multilayer perceptron (SLP) is used to independently encode each 3D point. Since the 3D points corresponding to the hierarchical sub-modules are unordered, the shared SLP ensures insensitivity to the processing order of the 3D points, while significantly reducing the number of parameters and avoiding overfitting. After independent encoding, max pooling is performed on the encoded features of all 3D points corresponding to the hierarchical sub-modules. Max pooling highlights the most significant local features in the geometric structure of each hierarchical sub-module (such as prominent geometric information at corners or face centers), thus enabling the aggregation of the features of 14 3D points into a fixed-length module-level feature vector. Based on the max pooling operation on each hierarchical sub-module, the hierarchical structure feature sequence of the overall target object and the five hierarchical sub-modules can be obtained. Inputting this hierarchical structure feature sequence into a Transformer encoder yields a matrix-structured and semantically meaningful hierarchical point cloud structure feature F. obj .

[0042] Understandably, the Transformer encoder utilizes a self-attention mechanism to capture the interdependencies between different levels of sub-modules. Specifically, these sub-modules are not isolated; for example, there are connection constraints between a drawer panel and the cabinet as a whole, and buttons are typically located in specific areas of the panel. The Transformer encoder can learn high-level structural prior knowledge, such as "the drawer should be pulled out from the front of the cabinet," preventing subsequent interaction modeling from generating operational intentions that violate physical connections (e.g., attempting to pull the drawer out from the side). Simultaneously, the self-attention mechanism can dynamically adjust the feature weights of each level of sub-module, enabling the model to automatically focus on the most relevant level of sub-module when processing different robot poses.

[0043] For the robot's state characteristics F robot This includes the initial pose R0, the base center point r0, and the frontal unit vector. The robot's initial pose can be represented by a set number of three-dimensional points, the specific number of which can be set according to the actual needs of those skilled in the art. In a specific example, the robot's initial pose can be represented by a pose matrix composed of 21 three-dimensional points. The pose matrix can be converted into the robot's pose features through global pooling.

[0044] The spatial relationship characteristics of the robot relative to each sub-module of the target object, including distance feature d. i Orientation angle α i and the normal vector n of the target object's operation surface i Here, the operation surface of the target object refers to the surface of the hierarchical sub-module being operated on that faces the robot when the robot is operating on the hierarchical sub-module of the target object. Distance feature d i The orientation angle α can be calculated using the center point coordinates of the bounding boxes of each sub-module and the center point coordinates of the base. i Direction vectors can be constructed using the center point coordinates of the bounding boxes of each level of sub-modules and the center point coordinates of the base. p i unit vector of the robot's front The result is obtained through calculation. For ease of understanding, it will be represented below using a specific expression.

[0045] Direction vector p i The expression is: p i =c i -r0; where c i This represents the center point of the bounding box corresponding to the i-th level submodule, which can be calculated using the coordinates of the corner points and / or face center points of the bounding box. It's important to know the direction vector. pi The modulus is the distance feature d. i .

[0046] Orientation angle α i The expression is: .

[0047] Based on the parameters calculated above, the spatial relationship characteristics of the robot relative to each level of sub-modules of the target object can be represented by the relationship matrix R, which is expressed as: R = { p i d i α i n i}

[0048] In a specific embodiment, when executing step S200, the fusion of the hierarchical point cloud structure features of the target object, the state features of the robot, and the spatial relationship features between the two can be performed using a cross-attention mechanism. The specific fusion process is as follows: First, the robot's state characteristics F robot After linear projection, it is used as query Q.

[0049] Specifically, the expression for querying Q is: Q=F robot ×W Q In the formula, W Q This represents a learnable linear projection matrix.

[0050] Then, the hierarchical point cloud structure features F of the target object obj The features are fused with the spatial relationship features R, and the fused features are linearly projected to serve as keys K and values ​​V.

[0051] Specifically, for the hierarchical point cloud structure feature F obj The fusion of spatial relationship features R is typically achieved through concatenation or element-wise addition. When using the concatenation method, the fused features KV... fused =Concat(F obj In the formula, Concat(·) represents the join function. Under the element-wise addition method, the fused feature KV... fused =F obj +W r ×R; where W r This represents a learnable linear projection matrix. After obtaining the fused features KV... fused Then, the expressions for the key K and the value V can be obtained as follows: K = KV fused ×W K V = KV fused ×W V In the formula, W K and WV Both represent learnable linear projection matrices.

[0052] Then, based on the obtained Q, K, and V, the attention function Attention(Q, K, V) is calculated. The specific calculation expression is as follows: .

[0053] In the formula, softmax(·) represents the normalization function, and d k Indicates the dimension of key K.

[0054] Then, the calculated attention (Q, K, V) is processed through a feedforward neural network, residual connections, and layer normalization to form the final interaction features; the specific processing steps are well known to those skilled in the art, so they will not be described in detail here.

[0055] Finally, the obtained interaction features are pooled, specifically by global average pooling, to obtain interaction latent variables with fixed dimensions.

[0056] Understandably, pooling interaction features can compress variable-length interaction features into fixed-dimensional vectors, making them easier to input as conditions for subsequent inverse denoising models. Furthermore, pooling forces the model to discard details irrelevant to the interaction intent, such as the fine coordinates of point clouds within each sub-module, retaining only the most essential high-level semantics, such as "pull the drawer" or "press the button," thereby improving generalization ability and computational efficiency.

[0057] In a specific embodiment, the acquisition of short trajectory latent variables in step S300 mainly includes three processes: forward diffusion, training of the reverse denoising model, and reverse denoising. For ease of understanding, each process will be described in detail below.

[0058] During the forward diffusion process, the true short trajectory latent variables can be obtained from the training set data as the initial short trajectory latent variables, and Gaussian noise can be continuously added to the initial short trajectory latent variables to obtain a single set of noisy short trajectory latent variables; based on the different randomly added Gaussian noise, multiple sets of different noisy short trajectory latent variables can be obtained.

[0059] Understandably, the training set data can come from the robot's actual successfully executed interaction trajectories. Specifically, through manual teaching, simulation optimization, or existing mature planners, a large number of successfully executed long-term trajectory sequences can be generated for the same interaction task of the robot. The training set data is then obtained by discretizing and summarizing all the long-term trajectory sequences generated by the robot for different interaction tasks. The forward diffusion process is a fixed Markov chain that can progressively convert real data into pure noise, allowing the model to observe data samples with different levels of noise and thus learn the denoising capabilities required for the inverse process. For ease of understanding, at any time step t, the latent variable z of the noisy short trajectory after adding Gaussian noise is... traj,t It can be represented by an expression, the specific expression being: ; .

[0060] In the formula, α t Represents the noise scheduling parameter, z traj,0 Represents the initial short trajectory latent variables; This represents standard Gaussian noise, s represents the time index, and β represents standard Gaussian noise. s This represents the noise addition rate parameter, used to control the rate at which Gaussian noise is added. As time step t increases, the noise scheduling parameter α... t The value gradually approaches 0, causing the latent variable z of the short-trajectory noise to... traj,t It gradually becomes pure noise.

[0061] During the training of the inverse denoising model, an inverse denoising model can be constructed first to predict noise for each group of noise short trajectory latent variables. Then, the inverse denoising model is trained with the goal of minimizing the error between the predicted noise and the actual added Gaussian noise.

[0062] Understandably, the inverse denoising model can employ a learnable neural network, such as U-Net or Transformer. The input to the inverse denoising model is the latent variable z of the short-trajectory noise. traj,t The time step t and the interactive latent variable z as conditional information interact The output of the inverse denoising model is the predicted noise. Minimizing the error between the predicted noise and the actual added Gaussian noise can be achieved by minimizing the mean square error between the two. That is, the loss function L of the inverse denoising model is the mean square error between the predicted noise and the actual added Gaussian noise. The specific formula for calculating the mean square error is well known to those skilled in the art and will not be elaborated here.

[0063] In the reverse denoising process, the interactive latent variable z interactAs a condition, a pre-trained inverse denoising model is used to perform inverse iterative denoising on multiple sets of different short-trajectory latent variables with noise, resulting in multiple sets of different short-trajectory latent variables z. traj Among them, the latent interactive variable z serves as a condition. interact By injecting cross-attention or feature concatenation into the inverse denoising model, the model is guided to generate interaction intents. Figure 1 The short trajectory latent variable z traj .

[0064] Understandably, the reverse denoising process requires starting from the noisy short-trajectory latent variable corresponding to the maximum time step and iteratively reversing the denoising process towards the minimum time step; that is, starting from pure noise, gradually recovering the clean, noise-free short-trajectory latent variable. The specific reverse iterative denoising process can be represented by the following expression: ; .

[0065] In the formula, and Let α represent the denoised short-trajectory latent variables of the noise at time steps t and t-1, respectively. t and α s All represent noise scheduling parameters. This represents the cumulative noise scheduling parameter, where 's' represents the time index. σ represents the noise predicted by the inverse denoising model. t denoted by , where z represents the variance of Gaussian noise, and z represents the standard Gaussian noise.

[0066] It is important to note the Gaussian noise added during the forward diffusion process. Both the Gaussian noise z used in the reverse denoising process and the Gaussian noise z come from a standard Gaussian distribution, but they are mathematically independent; that is, both are random.

[0067] In one specific embodiment, during step S300, the generated short trajectory latent variables have a low dimensionality and cannot directly drive the robot's movement. Therefore, it is necessary to transform the short trajectory latent variables z... traj Combined with the robot's initial pose R0, the short trajectory latent variables are expanded into a complete long-term trajectory point sequence. For ease of understanding, a detailed description follows.

[0068] Specifically, for the short-trajectory latent variable z traj The expansion can be implemented using a Transformer decoder. The Transformer decoder leverages its sequence modeling capabilities to map latent variables to a high-dimensional robot joint space or Cartesian point space while maintaining temporal coherence. Specifically, it can map the latent variables z of each short trajectory... trajThe initial pose R0 of the robot, along with the initial pose of the robot itself, serves as the input to the Transformer decoder. Since robot kinematics requires continuous velocity (i.e., displacement between trajectory points should not abruptly change), the Transformer decoder can employ an incremental prediction strategy. Starting from the center point of the robot's base, it predicts the displacement increments between adjacent trajectory points time-by-time, thereby generating a complete long-term trajectory sequence containing a preset number of points. This achieves accurate mapping from compact latent variables to a high-resolution motion point cloud. The specific selection of the preset number of points can be set according to the actual needs of those skilled in the art; for example, 1000 points can be selected.

[0069] In a specific embodiment, when executing step S400, the safety constraints mainly include collision constraints, velocity constraints, and curvature smoothing constraints. It is understood that the robot must avoid physical collisions with any obstacles in the environment (including non-operational areas of the target object, workbenches, and other equipment) when performing a task. Although the initial short trajectory latent variables are generated based on successful priors, the real environment may contain dynamic obstacles or model errors, causing trajectory points to fall inside obstacles; therefore, collision constraints are necessary. Since the robot's actuators have maximum speed limits (joint angular velocity or end effector Cartesian velocity), if the displacement between adjacent trajectory points is too large, the corresponding speed will exceed the hardware limit, leading to either increased tracking errors or triggering emergency stop protection. Therefore, incorporating velocity constraints into the optimization ensures that the generated trajectory is within the robot's physical capabilities. Even if the trajectory meets the collision and velocity requirements, there may still be abrupt changes in direction (e.g., a sudden sharp turn in straight motion), which can excite abrupt changes in robot joint acceleration, causing vibration or wear. Therefore, curvature smoothing constraints allow the curvature of the trajectory to change continuously, thereby obtaining a smooth and fluid motion trajectory.

[0070] Specifically, for collision constraints, the minimum Euclidean distance between the robot's position and all obstacles in the environment at each point in the initial motion trajectory can be calculated. This minimum Euclidean distance must be greater than or equal to a preset safe distance threshold. The specific value of the safe distance threshold can be set according to the actual needs of those skilled in the art, for example, 1 cm. For points that violate collision constraints, gradient descent can be used to move the point along the distance gradient direction; or a potential field function can be introduced to incorporate the collision cost into the overall loss function, solving it together with subsequent curvature smoothing constraints.

[0071] For velocity constraints, the average velocity between any two adjacent points in the initial trajectory can be calculated. The calculated average velocity must be less than or equal to a preset maximum velocity, which can be set by those skilled in the art, for example, 0.5 m / s. Points that violate the velocity constraints can be optimized using time resampling (increasing the number of trajectory points and interpolating).

[0072] For curvature smoothing constraints, the sum of the squares of the second-order differences (accelerations) of all adjacent points in the initial trajectory can be minimized as a smoothing cost to ensure continuous trajectory curvature. Alternatively, the smoothing cost and collision cost can be added together as the overall loss function, and a multi-objective optimization of the initial trajectory can be formed under velocity constraints to find the initial trajectory with the minimum overall loss function.

[0073] Another aspect of this application provides a computer-readable storage medium, in a preferred embodiment of which a computer program is stored on the storage medium; when the computer program is executed by a processor, it implements the above-described method for generating multimodal candidate motion trajectories for robots.

[0074] Another aspect of this application provides an electronic device, in one preferred embodiment of which includes a processor and a memory; the memory is used to store a computer program, and the processor is used to execute the computer program to implement the above-described method for generating multimodal candidate motion trajectories for robots.

[0075] The basic principles, main features, and advantages of this application have been described above. Those skilled in the art should understand that this application is not limited to the above embodiments. The embodiments and descriptions in the specification are merely the principles of this application. Various changes and modifications can be made to this application without departing from its spirit and scope, and all such changes and modifications fall within the scope of the claims. The scope of protection claimed by this application is defined by the appended claims and their equivalents.

Claims

1. A method for generating multimodal candidate motion trajectories for a robot, characterized in that, Includes the following steps: S100: Based on the constructed unified coordinate system, obtain the hierarchical point cloud structure features of the target object and the state features of the robot, and calculate the spatial relationship features of the robot relative to each level of sub-modules of the target object. S200: The hierarchical point cloud structure features of the target object, the state features of the robot, and the spatial relationship features of the calculation are fused to obtain the latent interactive variables that represent the interactive intention. S300: Using the obtained interactive latent variables as conditions, multiple diverse short trajectory latent variables are gradually recovered from multiple sets of noisy short trajectory latent variables with added random noise through reverse denoising; the obtained short trajectory latent variables are expanded into continuous motion sequences and trajectory connections are performed to obtain multiple initial motion trajectories; S400: Apply safety constraints to each initial motion trajectory to obtain candidate motion trajectories for multiple modes; In step S200, a cross-attention mechanism is used for feature fusion, which specifically includes the following process: The robot's state features are linearly projected and used as query Q; The hierarchical point cloud structural features and spatial relationship features of the target object are fused, and the fused features are linearly projected and used as keys K and values ​​V. The attention is calculated based on the obtained Q, K, and V, and then the calculated attention is processed through a feedforward neural network, residual connections, and layer normalization to form the final interaction features. The obtained interaction features are pooled to obtain interaction latent variables with fixed dimensions; In step S300, the acquisition of the latent variables for the short trajectory includes the following process: Obtain the initial short trajectory latent variables, and continuously add Gaussian noise to the initial short trajectory latent variables to obtain a single set of noisy short trajectory latent variables; based on the different randomly added Gaussian noise, obtain multiple sets of different noisy short trajectory latent variables. An inverse denoising model is constructed to predict noise for each group of noise short trajectory latent variables. The inverse denoising model is trained with the goal of minimizing the error between the predicted noise and the actual added Gaussian noise. Using interactive latent variables as conditions, a trained inverse denoising model is used to perform inverse iterative denoising on multiple sets of different short-trajectory latent variables with noise, resulting in multiple sets of different short-trajectory latent variables.

2. The method for generating multimodal candidate motion trajectories for robots as described in claim 1, characterized in that, In step S100, the hierarchical point cloud structure features of the target object include an overall bounding box and at least one hierarchical sub-module bounding box, each bounding box containing a preset number of three-dimensional points. The robot's state characteristics include its initial pose, base center point, and frontal unit vector; The spatial relationship characteristics of the robot relative to each level of sub-modules of the target object include distance characteristics, orientation angles, and the normal vector of the target object's operating surface. The distance feature is calculated using the center point coordinates of the bounding boxes of each sub-module and the center point coordinates of the base; the orientation angle is calculated using the direction vector constructed from the center point coordinates of the bounding boxes of each sub-module and the center point coordinates of the base, and the frontal unit vector of the robot.

3. The method for generating multimodal candidate motion trajectories for robots as described in claim 1, characterized in that, At any time step t, the latent variable z of the short-track noise after adding Gaussian noise. traj,t The expression is: ； ； In the formula, α t Represents the noise scheduling parameter, z traj,0 Represents the initial short trajectory latent variables. This represents standard Gaussian noise, s represents the time index, and β represents standard Gaussian noise. s This represents the variance table.

4. The method for generating multimodal candidate motion trajectories for robots as described in claim 1, characterized in that, When iteratively denoising the short-trajectory latent variables of noise, the interaction latent variables as conditions are injected into the reverse denoising model through cross attention or feature concatenation, guiding the generation of short-trajectory latent variables consistent with the current interaction intent.

5. The method for generating multimodal candidate motion trajectories for robots as described in claim 1, characterized in that, When iteratively denoising the short-trajectory latent variables of noise, the process starts from the latent variable of the short-trajectory noise corresponding to the maximum time step and proceeds in reverse iteratively towards the minimum time step. The specific expression is as follows: ； ； In the formula, and Let α represent the denoised short-trajectory latent variables of the noise at time steps t and t-1, respectively. t and α s All represent noise scheduling parameters. This represents the cumulative noise scheduling parameter, where 's' represents the time index. σ represents the noise predicted by the inverse denoising model. t denoted by , where z represents the variance of Gaussian noise, and z represents the standard Gaussian noise.

6. The method for generating multimodal candidate motion trajectories for robots as described in claim 1, characterized in that, In step S400, the safety constraints include collision constraints, velocity constraints, and curvature smoothing constraints.

7. A computer-readable storage medium, characterized in that, The storage medium stores a computer program; when the computer program is executed by a processor, it implements the robot multimodal candidate motion trajectory generation method as described in any one of claims 1-6.

8. An electronic device, characterized in that, It includes a processor and a memory; the memory is used to store a computer program, and the processor is used to execute the computer program to implement the robot multimodal candidate motion trajectory generation method as described in any one of claims 1-6.

Images