Method for generating motion of robot and robot
By scoring the importance of key points and encoding geometric relationships in point cloud data, enhanced point cloud data is generated. Combined with robot state features, this solves the problem of low success rate of robot operation tasks and achieves higher success rate and execution accuracy.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-04-15
- Publication Date
- 2026-07-03
Smart Images
Figure CN122323178A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of robot control technology, and more specifically to a method for generating robot motions and a robot. Background Technology
[0002] In recent years, with the development of artificial intelligence and sensor technology, robots have been increasingly widely used in people's lives. Among them, robot manipulation is a core technology in the field of robotics, which aims to enable robots to understand operational task instructions and perform environmental perception, grasping planning, and action execution in real-world environments to complete the operational tasks specified in the instructions.
[0003] In related technologies, robot operation generally employs a point cloud-based 3D diffusion strategy. Specifically, after acquiring the operation task, the robot uses a depth camera to obtain point cloud data of the work scene, extracts the features of each point in the point cloud through a point cloud encoder, and combines this with the robot's proprioceptive state to generate the motion trajectory of the robot actuators through a diffusion model, and controls the robot to perform the operation task along the aforementioned motion trajectory.
[0004] However, the aforementioned three-dimensional diffusion strategy suffers from a low success rate in operational tasks. Therefore, providing a stable and reliable robot motion generation scheme is a problem that urgently needs to be solved in this field. Summary of the Invention
[0005] The purpose of this application is to address the shortcomings of the existing technology by providing a method for generating robot motions and a robot in general. This method enhances point cloud data by using the importance scores and geometric relationships of each point. Motion commands are then generated based on the enhanced point cloud data and the robot's state characteristics, improving the accuracy of the motion commands and consequently increasing the success rate and execution precision of the robot's tasks.
[0006] To achieve the above objectives, the technical solutions adopted in the embodiments of this application are as follows: In a first aspect, this application provides a method for generating robot actions, the method comprising: In response to the operation task command, the point cloud data of the robot's current working scene and the state characteristics of the robot are acquired; The importance score of each point in the point cloud data is obtained by scoring the key point importance of each point in the point cloud data. The point cloud data is paired geometric relationship encoded to obtain the geometric relationship between two points in the point cloud data; The attention weight of each point is determined based on the importance score of each point in the point cloud data and the geometric relationship between two points in the point cloud data. The point cloud data is weighted and transformed according to the attention weight of each point to generate enhanced point cloud data. Based on the enhanced point cloud data and the robot's state characteristics, the robot generates action instructions corresponding to the operation task instructions to control the robot to perform the corresponding action instructions.
[0007] In some embodiments, determining the attention weight of each point based on the importance score of each point in the point cloud data and the geometric relationship between two points in the point cloud data includes: Based on the importance score of each point in the point cloud data, a key point pair relationship matrix corresponding to the point cloud data is generated; Based on the geometric relationship between two points in the point cloud data, generate a pairwise geometric matrix corresponding to the point cloud data; Based on the keypoint pair relation matrix and the pairwise geometric matrix, the attention weight of each point is determined.
[0008] In some embodiments, determining the attention weight of each point based on the keypoint pair relation matrix and the pairwise geometric matrix includes: Based on the keypoint pair relation matrix and the pairwise geometric matrix, the attention score of each point is determined; Based on the attention scores of each point, the attention weights of each point are determined.
[0009] In some embodiments, generating a key point pair relationship matrix corresponding to the point cloud data based on the importance scores of each point in the point cloud data includes: Traverse all point pairs in the point cloud data, and obtain the importance score of the point pair based on the importance score of the first point and the importance score of the second point in each point pair; wherein each point pair includes a first point and a second point; The key point pair relationship matrix is generated based on the importance scores of all point pairs in the point cloud data.
[0010] In some embodiments, determining the attention weight of each point based on the keypoint pair relation matrix and the pairwise geometric matrix includes: The similarity between two points in all point pairs in the point cloud data is calculated to generate a similarity matrix corresponding to the point cloud data. The spatial proximity deviation between two points in all point pairs of the point cloud data is calculated to generate the spatial proximity deviation matrix corresponding to the point cloud data. Based on the keypoint pair relation matrix, the pairwise geometric matrix, the similarity matrix, and the spatial proximity deviation matrix, the attention weight of each point is determined.
[0011] In some embodiments, calculating the spatial proximity deviation between two points in all point pairs of the point cloud data includes: Calculate the distance between any two points in any point pair in the point cloud data; Based on the distance and a preset coefficient, a spatial proximity bias value is obtained.
[0012] In some embodiments, the step of performing pairwise geometric relationship encoding on the point cloud data to obtain the geometric relationship between two points in the point cloud data includes: Obtain the global augmentation features of the point cloud data; Traverse all point pairs in the point cloud and obtain the feature vector of the first point, the initial feature vector of the second point, and the position difference between the first point and the second point in each point pair; Based on the initial feature vector of the first point, the initial feature vector of the second point, the global augmentation feature, and the position difference of all point pairs in the point cloud data, the geometric relationship encoding value of the point pair is determined.
[0013] In some embodiments, obtaining the global enhanced features of the point cloud data includes: Obtain the global context features of the point cloud data; The global context features are fused with the initial feature vector of each point in the point cloud data to obtain the global enhanced features of the point cloud data.
[0014] In some embodiments, generating motion commands based on the enhanced point cloud data and the robot's state features includes: The robot acquires image data and obtains image features from the image data. The action command is generated based on the enhanced point cloud data, the state features, and the image features.
[0015] In some embodiments, generating the action command based on the enhanced point cloud data, the state features, and the image features includes: The enhanced point cloud data, the state features, and the image features are stitched together to form multimodal conditional features; The multimodal conditional features are input into the diffusion model to generate the action instructions.
[0016] Secondly, another embodiment of this application provides a robot, including: a robot body, and a controller disposed within the robot body, the controller being configured to perform the method described in any of the first aspects above.
[0017] Thirdly, another embodiment of this application provides a controller, including: a processor, a memory, and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the controller is running, the processor communicates with the memory via the bus, and the processor executes the machine-readable instructions to perform the steps of any of the methods described in the first aspect above.
[0018] Fourthly, another embodiment of this application provides a storage medium storing a computer program, which, when executed by a processor, performs the steps of any of the methods described in the first aspect above.
[0019] The beneficial effects of this application are as follows: In the robot motion generation method provided by this application, in response to an operation task instruction, point cloud data of the robot's current working scene and the robot's state features are acquired; key point importance scoring is performed on each point in the point cloud data to obtain an importance score for each point; pairwise geometric relationship encoding is performed on the point cloud data to obtain the geometric relationship between two points in the point cloud data; attention weights for each point are determined based on the importance scores and geometric relationships between the two points in the point cloud data; a weighted transformation is performed on the point cloud data based on the attention weights of each point to generate enhanced point cloud data; and motion instructions corresponding to the operation task instruction are generated based on the enhanced point cloud data and the robot's state features to control the robot to execute the corresponding action operation. This application enhances the point cloud data by using the importance scores and geometric relationships of each point. By generating motion instructions based on the enhanced point cloud data and the robot's state features, the accuracy of the motion instructions is improved, thereby increasing the success rate and execution precision of the robot's operation tasks. Attached Figure Description
[0020] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.
[0021] Figure 1 A flowchart illustrating a robot motion generation method according to an embodiment of this application is shown; Figure 2 A flowchart illustrating the determination of geometric relationships according to an embodiment of this application is shown; Figure 3 A flowchart illustrating the generation of action instructions according to an embodiment of this application is shown; Figure 4 A flowchart illustrating the generation of action instructions is shown in another embodiment of this application; Figure 5 This is a schematic diagram illustrating the motion generation of a robot, provided as an embodiment of this application. Figure 6 This application shows a schematic diagram of the structure of a robot according to an embodiment of the present application; Figure 7 A schematic diagram of the structure of a controller provided in one embodiment of this application is shown. Detailed Implementation
[0022] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. It should be understood that the accompanying drawings in this application are for illustrative and descriptive purposes only and are not intended to limit the scope of protection of this application. Furthermore, it should be understood that the schematic drawings are not drawn to scale. The flowcharts used in this application illustrate operations implemented according to some embodiments of this application. It should be understood that the operations in the flowcharts may not be implemented in sequence, and steps without logical contextual relationships may be reversed or implemented simultaneously. In addition, those skilled in the art, guided by the content of this application, may add one or more other operations to the flowcharts, or remove one or more operations from the flowcharts.
[0023] Furthermore, the described embodiments are merely some, not all, of the embodiments of this application. The components of the embodiments of this application described and illustrated herein can typically be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of the application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.
[0024] It should be noted that the term "comprising" will be used in the embodiments of this application to indicate the presence of the features declared thereafter, but does not exclude the addition of other features.
[0025] The technical solution provided in this application is applicable to task scenarios requiring robot operation, such as home service, industrial manufacturing, and medical rehabilitation. In these scenarios, the robot needs to execute actions corresponding to the user's operation instructions in a real-world environment, such as grasping, placing, and assembling.
[0026] Currently, in related technologies, when a robot performs an operation task, it generally generates a motion trajectory directly based on the characteristics of each point and the robot's state, and controls the robot to perform the operation task according to the generated motion trajectory.
[0027] However, the spatial relationships between points in point cloud data are complex and diverse, and different operational tasks correspond to different point cloud data, with varying critical requirements for different regions within the point cloud data. Therefore, the motion trajectories generated using related technologies are not always accurate, resulting in a low success rate for robots performing operational tasks based on these trajectories.
[0028] Based on this, this application provides a method for generating robot actions. After acquiring point cloud data of the current work scene and the robot's state features, the method performs keypoint importance scoring and pairwise geometric relationship encoding on each point in the point cloud data. Then, based on the importance scores and geometric relationships, a weighted transformation is applied to the point cloud data to generate enhanced point cloud data. This enhanced point cloud feature can simultaneously contain the geometric structure information of the entire point cloud data, as well as keypoint information related to the operation task instructions. The action instructions generated using the enhanced point cloud data combined with the robot's state features are more accurate, thereby improving the success rate of operation task execution.
[0029] It should be noted that the above description of application scenarios is only one of the multiple use scenarios provided in this application. Those skilled in the art should understand that when the robot motion generation method and the robot provided in this application are applied to other use scenarios, their implementation methods and technical effects are similar.
[0030] Figure 1 A flowchart illustrating a robot motion generation method according to an embodiment of this application is shown. Figure 1 As shown, the robot motion generation method provided in this application may include steps 110-160, wherein: Step 110: In response to the operation task command, acquire the point cloud data of the robot's current working scene and the robot's state characteristics.
[0031] The operation task instructions can be specific task descriptions that users input into the robot through voice input interface, text input interface, or other human-computer interaction interface, requiring the robot to perform physical operations. For example, operation task instructions can be "unplug the plug", "pick up the water cup", "hammer the nail", etc.
[0032] Point cloud data is a dataset representing the three-dimensional spatial information of the surface of an object in a robot's working scene, acquired through a depth camera. Point cloud data contains multiple points, each with three-dimensional coordinates (x, y, z), and can be supplemented with color features (r, g, b), normal features, texture features, etc.
[0033] State characteristics are the robot's own proprioceptive information, including but not limited to: the angles of each joint, the pose (position and orientation) of the end effector (gripper, manipulator) in three-dimensional space, and the opening and closing degree of the gripper / the angle of the manipulator's finger joints, etc.
[0034] After receiving the operation task instruction, the robot begins data acquisition. Specifically, the robot uses depth cameras fixedly mounted on its head, robotic arm, or wrist to capture images of the current work scene, obtaining image data containing the target object (e.g., a "plug" or a "cup") and its surrounding environment. The image data can be Red-Green-Blue-D (RGB-D) images or depth images. The robot extracts three-dimensional spatial coordinate information from the acquired image data to generate point cloud data. Simultaneously, the robot acquires its current state characteristics. If the image data captured by the camera does not fully contain the target object, the robot can first perform an adjustment action to adjust the orientation of the depth camera to ensure that the image data taken based on the adjusted orientation can capture the complete target object, or to ensure that the target object is located in the central area of the image data taken based on the adjusted orientation.
[0035] Step 120: Assess the importance of each point in the point cloud data to obtain the importance score of each point in the point cloud data.
[0036] For each point in the point cloud data, a keypoint scoring network is used to score its importance. The importance score is the specific numerical value output by the keypoint importance score; the higher the value, the stronger the relevance of the point to task completion.
[0037] As an example, for a point cloud dataset containing N points, each point k is associated with an initial feature vector. Initial feature vector It encodes attributes such as texture and surface normals. A preset keypoint scoring model is also included. It can be a shallow neural network, used to determine the initial feature vector of point k. Calculate the importance score corresponding to point k. :
[0038] Taking the operation task instruction "unplug" as an example, points in the plug handle area of the point cloud data have a higher importance score because they have specific geometric shape and color features and are highly correlated with the operation task instruction. These points can be considered key points in the point cloud data. Points in the plug cable or background areas of the point cloud data have a lower importance score because they are less correlated with the operation task instruction.
[0039] By assigning importance scores to each point in the point cloud data, adaptive identification and quantitative evaluation of key task points (such as grasping points or points corresponding to contact surfaces) corresponding to operation task instructions are achieved.
[0040] Step 130: Encode the point cloud data in pairs to obtain the geometric relationship between two points in the point cloud data.
[0041] For any pair of points in point cloud data, based on the spatial location information (such as coordinates, distance, and relative orientation) and / or feature information (such as color and normal), a coding value representing the degree of geometric relationship between the two points is calculated using a preset coding method. For example, if the two points in a pair are close to each other and lie on the same plane, a higher geometric relationship coding value will be obtained.
[0042] Encoded values can characterize whether the two points in a point pair belong to the same geometric surface, whether they have a specific spatial configuration (such as edges, corners, parallel or perpendicular relationships), and geometric attributes such as the relative distance and direction between them.
[0043] Taking the operation task instruction "unplug" as an example, two adjacent points in the point cloud data located on the plug handle area have higher geometric relationship encoding values because they are close to each other and located on the same geometric surface. For points located on the plug handle and points in the distant background, their geometric relationship encoding values are lower because they are far apart and lack a direct geometric relationship.
[0044] Step 140: Determine the attention weight of each point based on the importance score of each point in the point cloud data and the geometric relationship between two points in the point cloud data.
[0045] Attention weights are used to represent the importance of each point in the point cloud data in the current operation task.
[0046] The robot identifies key points in point cloud data by using importance scores for each point and understands the structural layout of the point cloud data by analyzing the geometric relationships between point pairs. By combining the importance scores of each point with the geometric relationships between two points, the attention weight of each point in the point cloud data can be determined.
[0047] In some embodiments, keypoints and points around and geometrically related to keypoints are assigned higher attention weights, while other points unrelated to keypoints are assigned lower attention weights. In this way, even in different operational scenarios, the robot's attention can adaptively focus on key regions and their corresponding geometric contexts that are relevant to the current task.
[0048] In some embodiments, step 140 may include: generating a key point pair relation matrix corresponding to the point cloud data based on the importance score of each point in the point cloud data; generating a pairwise geometric matrix corresponding to the point cloud data based on the geometric relationship between two points in the point cloud data; and determining the attention weight of each point based on the key point pair relation matrix and the pairwise geometric matrix.
[0049] An ordered or unordered combination of any two points selected from point cloud data is called a point pair.
[0050] The robot fuses the keypoint pair relationship matrix and the pairwise geometry matrix to obtain a fused attention score matrix. Each element in the fused attention score matrix represents the overall association strength between the two points in a pair. Then, the attention weight for each point is determined based on the fused attention score matrix.
[0051] Among them, the key point relationship matrix and the pairwise geometric matrix can be fused by weighted fusion according to preset weights or by direct addition.
[0052] In some embodiments, determining the attention weight of each point based on the keypoint pair relation matrix and the pair geometry matrix may include: determining the attention score of each point based on the keypoint pair relation matrix and the pair geometry matrix; and determining the attention weight of each point based on the attention score of each point.
[0053] For each point in the point cloud data: the robot sums the attention scores of that point with all other points in the point cloud data to determine the attention score of that point based on the summation result.
[0054] As an example, the attention scores of this point are summed with those of all other points to obtain a sum. This sum is then normalized (or mapped to a preset interval) to determine the normalized (mapped) result as the attention score of this point. Furthermore, the attention weight of each point is obtained based on its attention score.
[0055] Using the above method, if a point has a high importance score, then the attention weight corresponding to that point will be large. Alternatively, if a point does not have a high importance score, but there is a close geometric connection between that point and other key points around it, then the attention weight corresponding to that point will be large.
[0056] Step 150: Perform a weighted transformation on the point cloud data according to the attention weight of each point to generate enhanced point cloud data.
[0057] The original feature vector of each point in the point cloud data is multiplied by its corresponding attention weight to obtain the enhanced point cloud data. This weighted transformation amplifies the features of high-importance points (keypoints) in the enhanced point cloud data, while suppressing the features of low-importance points.
[0058] The original feature vector can include information such as coordinates, color, and normals.
[0059] The enhanced point cloud data not only retains the original feature vector of each point, but also incorporates key point information and geometric structure information relevant to the current task. Compared to the original point cloud features of each point, the enhanced point cloud data has stronger task relevance and structural expressive power.
[0060] Step 160: Generate action instructions corresponding to the operation task instructions based on the enhanced point cloud data and the robot's state characteristics, so as to control the robot to perform the corresponding action instructions.
[0061] Motion commands are control commands generated by the robot and sent to the underlying actuators to drive the robot to perform specific physical operations. Motion commands include, but are not limited to: the target position and orientation of the end effector, the gripping force or opening / closing degree of the gripper / manipulator, and the motion trajectory of each joint of the manipulator.
[0062] The robot performs corresponding operations based on motion instructions, such as moving the robotic arm to the target position, closing the gripper / manipulator to grasp an object, rotating the wrist to flip the object, and moving backward to unplug a plug.
[0063] In summary, the robot motion generation method and robot provided in this application, in response to an operation task instruction, acquire point cloud data of the robot's current working scene and the robot's state features; score the importance of each point in the point cloud data to obtain an importance score for each point; encode pairwise geometric relationships in the point cloud data to obtain the geometric relationship between two points; determine the attention weight of each point based on the importance score and the geometric relationship between two points; perform a weighted transformation on the point cloud data based on the attention weight of each point to generate enhanced point cloud data; and generate motion instructions corresponding to the operation task instruction based on the enhanced point cloud data and the robot's state features to control the robot to execute the corresponding motion instructions. This application enhances the point cloud data by using the importance score and geometric relationship of each point. By generating motion instructions based on the enhanced point cloud data and the robot's state features, the accuracy of the motion instructions is improved, thereby increasing the success rate and execution precision of the robot's operation tasks.
[0064] Figure 2 A flowchart illustrating the determination of geometric relationships according to an embodiment of this application is shown, as follows: Figure 2 As shown, step 130 may include steps 131-133, wherein: Step 131: Obtain global augmentation features of point cloud data.
[0065] Global augmentation features are pre-generated feature representations that incorporate global information from point cloud data.
[0066] In some embodiments, step 131 may include: obtaining global context features of point cloud data; fusing the global context features with the initial feature vector of each point in the point cloud data to obtain global enhanced features of the point cloud data.
[0067] By performing global pooling on the initial feature vectors of all points in the point cloud data, global context features that can characterize the global attributes of the point cloud data are obtained.
[0068] The global pooling operation can be average pooling or max pooling, etc., and this application does not impose any restrictions on it.
[0069] For example, each point in the point cloud data has a C-dimensional initial feature vector. The robot averages the initial feature vectors of all points in the point cloud data in each dimension to obtain a C-dimensional global context feature vector.
[0070] The global enhanced features obtained based on global context features and the initial feature vector of each point contain both global information about the entire point cloud data and retain the local attribute information of each point in the point cloud data.
[0071] For a point cloud dataset containing N points, each point i has an initial feature vector. and three-dimensional coordinates Subsequently, global context features The feature vector of each point is concatenated with the initial feature vector of each point to obtain the global enhanced feature. :
[0072] One approach to concatenation is feature concatenation, which involves copying the C-dimensional global context feature vector N times and concatenating each copy with the initial C-dimensional feature vector of each point in the point cloud data, resulting in a 2C-dimensional global augmented feature for each point. Alternatively, another approach is feature addition, which involves adding the global context feature vector element-wise to the initial feature vector of each point, resulting in a C-dimensional global augmented feature for each point.
[0073] Step 132: Traverse all point pairs in the point cloud and obtain the initial feature vector of the first point, the initial feature vector of the second point, and the positional difference between the first and second points in each point pair.
[0074] The first point and the second point together form a complete point pair. The first point pair and the second point pair are only used to distinguish the two points in the point pair and do not imply any priority in order.
[0075] The positional difference between the first point and the second point is used to characterize the relative positional relationship between the first point and the second point, including information such as distance and direction.
[0076] Step 133: Determine the geometric relationship encoding value of the point pair based on the initial feature vector of the first point, the initial feature vector of the second point, the global augmentation features, and the position difference of all point pairs in the point cloud data.
[0077] The robot can input the initial feature vector of the first point, the initial feature vector of the second point, the global enhancement features, and the position difference of each point pair into the preset geometric relationship encoder f(.), and obtain the geometric relationship encoding value of each point pair output by the preset geometric relationship encoder.
[0078] Furthermore, a geometric matrix is generated based on the geometric relationship encoding values of each point pair. :
[0079] in, Let i be the initial feature vector of the i-th point. Let j be the initial feature vector of the j-th point. Let be the position difference between point i and point j. This is a global enhancement feature.
[0080] In some embodiments, the process of generating a key point pair relationship matrix may include: traversing all point pairs in the point cloud data, obtaining the importance score of the point pair based on the importance score of the first point and the importance score of the second point in each point pair; wherein each point pair includes a first point and a second point; and generating a key point pair relationship matrix based on the importance scores of all point pairs in the point cloud data.
[0081] The importance score of the current point pair is obtained by multiplying the importance scores of the first and second points in the current point pair. Subsequently, a key point pair relationship matrix is obtained based on the importance scores of each point pair in the point cloud data. .
[0082] Key point is the element in the i-th row and j-th column of the matrix. To score the importance of the i-th point Importance score of the j-th point The product obtained by multiplication is:
[0083] In some embodiments, after obtaining the key point pair relationship matrix and the pair geometry matrix, the robot also needs to calculate the feature similarity and spatial proximity deviation between each pair of points in the point cloud data to further enrich the information source of the attention weight.
[0084] Specifically, the similarity between two points in all point pairs in the point cloud data is calculated to generate a similarity matrix corresponding to the point cloud data; the spatial proximity deviation between two points in all point pairs in the point cloud data is calculated to generate a spatial proximity deviation matrix corresponding to the point cloud data; and the attention weight of each point is determined based on the key point pair relationship matrix, the pair geometry matrix, the similarity matrix, and the spatial proximity deviation matrix.
[0085] A similarity matrix is used to characterize the feature similarity between two points in any pair of points in point cloud data.
[0086] The spatial proximity deviation matrix is used to represent the degree of proximity between any two points in a point cloud.
[0087] Based on key point pair relationship matrix Paired geometric matrices Similarity matrix Spatial proximity deviation matrix Determine the fusion attention score for each point. Furthermore, the attention weights of each point are determined based on the fusion attention scores. That is:
[0088] in, , , These are preset constant values used to balance the stability of attention scores.
[0089] In some embodiments, the distance between any two points in a point cloud data pair is calculated; and a spatial proximity bias value is obtained based on the distance and a preset coefficient.
[0090] Wherein, the spatial proximity bias value between the first point i and the second point j in the point pair The calculation method is as follows:
[0091] In other words, the farther apart the two points in a point pair are, the smaller the spatial proximity bias value of that point pair; the closer the two points in a point pair are, the larger the spatial proximity bias value of that point pair. This allows point pairs that are closer to each other to receive higher attention weights in subsequent weighted transformations.
[0092] Figure 3 A flowchart illustrating the generation of action instructions according to an embodiment of this application is shown, as follows: Figure 3 As shown, step 160 may include steps 161 and 162, wherein: Step 161: Obtain image data collected by the robot and acquire image features from the image data.
[0093] Image data can be acquired by the robot using a depth camera, or it can be acquired by the robot using other cameras fixedly mounted on the robot. The image data can be a red-green-blue (RGB) image.
[0094] Feature extraction is performed on image data to obtain image features. Image features can characterize the visual appearance information contained in the work scene, including but not limited to the color, texture, pattern and other details of the target object, which can be used to make up for the lack of visual details in point cloud data.
[0095] Step 162: Generate action instructions based on the enhanced point cloud data, state features, and image features.
[0096] Introducing image features into the process of generating action commands can supplement the enhanced point cloud data with visual detail information, thereby improving the robot's environmental understanding and the accuracy of action decisions.
[0097] In some embodiments, step 162 may include: stitching together the enhanced point cloud data, state features and image features to form multimodal conditional features; and inputting the multimodal conditional features into the diffusion model to generate action instructions.
[0098] The diffusion model can be a generative model. The diffusion model starts with random noise and gradually generates a sequence of actions as the robot's action instructions through multiple iterations of noise reduction.
[0099] Figure 4 A flowchart illustrating the generation of action instructions is shown in another embodiment of this application, as follows: Figure 4 As shown, the inputs of the diffusion model are state features, point cloud features, and image features, and the output is action commands.
[0100] The robot encodes its state information using a state encoder to obtain state features, which are then input into the diffusion model.
[0101] The robot inputs the point cloud features from the point cloud data into the diffusion model.
[0102] The robot encodes the image information using an image encoder to obtain image features, and then inputs these features into a diffusion model.
[0103] The feature extraction process described above can be executed in parallel to improve the robot's processing efficiency.
[0104] Action commands can be the target positions and postures of a series of robot end effectors, or the motion trajectories of the robot's robotic arms, used to drive the robot to complete the execution operations corresponding to the operation task commands.
[0105] During the denoising process, the diffusion model is guided by multimodal conditional features. Since the multimodal conditional features contain global structural relationships and local details, they provide a data foundation for the behavior planning of the diffusion model, thereby ensuring that the generated action instructions not only conform to the geometric constraints of the work scenario, but also match the current state of the robot, achieving accurate operation with context awareness. The robot motion generation method provided in this application, after acquiring point cloud data of the work scene, processes the point cloud data in multiple dimensions to obtain a keypoint pair relation matrix, a pairwise geometric matrix, a similarity matrix, and a spatial proximity deviation matrix. Subsequently, attention weights are determined based on these multi-dimensional matrices to enhance the point cloud data, resulting in enhanced point cloud data. Then, motion commands are determined based on the enhanced point cloud data, state information, and image information. The process of determining attention weights considers the characteristics of point cloud data across multiple dimensions, enabling the attention weights to possess context-aware capabilities. This allows for adaptive focusing on key points related to the operation task within the point cloud data, and the perception of global structural relationships and local details. The motion commands determined based on the enhanced point cloud data have higher accuracy, leading to a higher success rate in task execution.
[0106] Based on the same inventive concept, this application also provides a robot motion generation device corresponding to the robot motion generation method. Since the principle of the device in this application is similar to the robot motion generation method described above in this application, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be described again.
[0107] Figure 5 This is a schematic diagram illustrating the motion generation of a robot according to an embodiment of this application, such as... Figure 5 As shown, the device includes: an acquisition module 201, used to acquire point cloud data of the robot's current working scene and the robot's state characteristics in response to an operation task instruction; The determination module 202 is used to score the importance of each point in the point cloud data to obtain the importance score of each point in the point cloud data; to encode the pairwise geometric relationship of the point cloud data to obtain the geometric relationship between two points in the point cloud data; and to determine the attention weight of each point based on the importance score of each point in the point cloud data and the geometric relationship between two points in the point cloud data. The generation module 203 is used to perform a weighted transformation on the point cloud data according to the attention weight of each point to generate enhanced point cloud data. The control module 204 is used to generate action instructions corresponding to the operation task instructions based on the enhanced point cloud data and the robot's state characteristics, so as to control the robot to execute the operation corresponding to the action instructions.
[0108] In some embodiments, the generation module 203 is specifically used to generate a key point pair relation matrix corresponding to the point cloud data based on the importance score of each point in the point cloud data; and to generate a pairwise geometric matrix corresponding to the point cloud data based on the geometric relationship between two points in the point cloud data. The determination module 202 is specifically used to determine the attention weight of each point based on the key point pair relationship matrix and the pairwise geometric matrix.
[0109] In some embodiments, the determining module 202 is specifically used to determine the attention score of each point based on the key point pair relation matrix and the pairwise geometric matrix; and to determine the attention weight of each point based on the attention score of each point.
[0110] In some embodiments, the determining module 202 is specifically used to traverse all point pairs in the point cloud data and obtain the importance score of the point pair based on the importance score of the first point and the importance score of the second point in each point pair; wherein each point pair includes a first point and a second point; The generation module 203 is specifically used to generate a key point pair relationship matrix based on the importance scores of all point pairs in the point cloud data.
[0111] In some embodiments, the generation module 203 is specifically used to calculate the similarity between two points in all point pairs of point cloud data to generate a similarity matrix corresponding to the point cloud data; and to calculate the spatial proximity deviation between two points in all point pairs of point cloud data to generate a spatial proximity deviation matrix corresponding to the point cloud data. The determination module 202 is specifically used to determine the attention weight of each point based on the key point pair relationship matrix, pairwise geometric matrix, similarity matrix and spatial proximity deviation matrix.
[0112] In some embodiments, the determining module 202 is specifically used to calculate the distance between two points in all point pairs in the point cloud data; and to obtain a spatial proximity bias value based on the distance and a preset coefficient.
[0113] In some embodiments, the acquisition module 201 is specifically used to acquire global enhanced features of point cloud data; traverse all point pairs in the point cloud, and acquire the feature vector of the first point, the initial feature vector of the second point, and the positional difference between the first point and the second point in each point pair; The determination module 202 is specifically used to determine the geometric relationship encoding value of the point pair based on the initial feature vector of the first point, the initial feature vector of the second point, the global enhancement features, and the position difference of all point pairs in the point cloud data.
[0114] In some embodiments, the acquisition module 201 is specifically used to acquire the global context features of point cloud data; The determination module 202 is specifically used to fuse the global context features with the initial feature vector of each point in the point cloud data to obtain the global enhanced features of the point cloud data.
[0115] In some embodiments, the acquisition module 201 is specifically used to acquire image data collected by the robot and to acquire image features of the image data; The generation module 203 is specifically used to generate action instructions based on the enhanced point cloud data, state features, and image features.
[0116] In some embodiments, the determining module 202 is specifically used to stitch together the enhanced point cloud data, state features and image features to form multimodal conditional features; The generation module 203 is specifically used to input multimodal conditional features into the diffusion model and generate action instructions.
[0117] The processing flow of each module in the device and the interaction flow between each module can be referred to the relevant descriptions in the above method embodiments, and will not be detailed here.
[0118] This application also provides a robot. Figure 6 A schematic diagram of the structure of a robot according to an embodiment of this application is shown, as follows: Figure 6 As shown, the robot includes a robot body 410 and a controller 420 disposed within the robot body. The controller 420 is used to execute the steps of the above-described robot motion generation method.
[0119] For example, the robot body 410 can be a humanoid robot or a non-humanoid robot.
[0120] In other embodiments of this application, the controller 420 may also be disposed outside the robot body.
[0121] This application also provides a controller. Figure 7 A schematic diagram of the structure of a controller according to an embodiment of this application is shown, as follows: Figure 7 As shown, the controller includes a processor 421 and a memory 422, and optionally, a bus 423. The memory 422 stores machine-readable instructions executable by the processor 421. When the controller is running, the processor 421 and the memory 422 communicate via the bus 423. When the machine-readable instructions are executed by the processor 421, the steps of the robot motion generation method described above are performed.
[0122] This application also provides a computer-readable storage medium storing a computer program, which, when run by a processor, executes the steps of the robot motion generation method described above.
[0123] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems and devices described above can be referred to the corresponding processes in the method embodiments, and will not be repeated here. In the several embodiments provided in this application, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. Furthermore, multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed mutual coupling or direct coupling or communication connection can be through some communication interfaces; the indirect coupling or communication connection of devices or modules can be electrical, mechanical, or other forms.
[0124] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. If the functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes: USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, optical disks, and other media capable of storing program code.
[0125] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.
Claims
1. A method for generating robot motion, characterized in that, The method includes: In response to the operation task command, the point cloud data of the robot's current working scene and the state characteristics of the robot are acquired; The importance score of each point in the point cloud data is obtained by scoring the key point importance of each point in the point cloud data. The point cloud data is paired geometric relationship encoded to obtain the geometric relationship between two points in the point cloud data; The attention weight of each point is determined based on the importance score of each point in the point cloud data and the geometric relationship between two points in the point cloud data. The point cloud data is weighted and transformed according to the attention weight of each point to generate enhanced point cloud data. Based on the enhanced point cloud data and the robot's state characteristics, the robot generates action instructions corresponding to the operation task instructions to control the robot to perform the corresponding action instructions.
2. The method according to claim 1, characterized in that, The step of determining the attention weight of each point based on the importance score of each point in the point cloud data and the geometric relationship between two points in the point cloud data includes: Based on the importance score of each point in the point cloud data, a key point pair relationship matrix corresponding to the point cloud data is generated; Based on the geometric relationship between two points in the point cloud data, generate a pairwise geometric matrix corresponding to the point cloud data; Based on the keypoint pair relation matrix and the pairwise geometric matrix, the attention weight of each point is determined.
3. The method according to claim 2, characterized in that, The step of determining the attention weight of each point based on the keypoint pair relation matrix and the pairwise geometric matrix includes: Based on the keypoint pair relation matrix and the pairwise geometric matrix, the attention score of each point is determined; Based on the attention scores of each point, the attention weights of each point are determined.
4. The method according to claim 2, characterized in that, The step of generating a key point pair relationship matrix corresponding to the point cloud data based on the importance scores of each point in the point cloud data includes: Traverse all point pairs in the point cloud data, and obtain the importance score of the point pair based on the importance score of the first point and the importance score of the second point in each point pair; wherein each point pair includes a first point and a second point; The key point pair relationship matrix is generated based on the importance scores of all point pairs in the point cloud data.
5. The method according to claim 2, characterized in that, The step of determining the attention weight of each point based on the keypoint pair relation matrix and the pairwise geometric matrix includes: The similarity between two points in all point pairs in the point cloud data is calculated to generate a similarity matrix corresponding to the point cloud data. The spatial proximity deviation between two points in all point pairs of the point cloud data is calculated to generate the spatial proximity deviation matrix corresponding to the point cloud data. Based on the keypoint pair relation matrix, the pairwise geometric matrix, the similarity matrix, and the spatial proximity deviation matrix, the attention weight of each point is determined.
6. The method according to claim 5, characterized in that, The calculation of the spatial proximity deviation between two points in all point pairs of the point cloud data includes: Calculate the distance between any two points in any point pair in the point cloud data; Based on the distance and a preset coefficient, a spatial proximity bias value is obtained.
7. The method according to claim 1, characterized in that, The step of performing pairwise geometric relationship encoding on the point cloud data to obtain the geometric relationship between two points in the point cloud data includes: Obtain the global augmentation features of the point cloud data; Traverse all point pairs in the point cloud and obtain the feature vector of the first point, the initial feature vector of the second point, and the position difference between the first point and the second point in each point pair; Based on the initial feature vector of the first point, the initial feature vector of the second point, the global augmentation feature, and the position difference of all point pairs in the point cloud data, the geometric relationship encoding value of the point pair is determined.
8. The method according to claim 7, characterized in that, The acquisition of global enhanced features of the point cloud data includes: Obtain the global context features of the point cloud data; The global context features are fused with the initial feature vector of each point in the point cloud data to obtain the global enhanced features of the point cloud data.
9. The method according to claim 1, characterized in that, The step of generating action commands based on the enhanced point cloud data and the robot's state features includes: The robot acquires image data and obtains image features from the image data. The action command is generated based on the enhanced point cloud data, the state features, and the image features.
10. The method according to claim 9, characterized in that, The step of generating the action command based on the enhanced point cloud data, the state features, and the image features includes: The enhanced point cloud data, the state features, and the image features are stitched together to form multimodal conditional features; The multimodal conditional features are input into the diffusion model to generate the action instructions.
11. A robot, characterized in that, include: A robot body, and a controller disposed within the robot body, the controller being configured to perform the method described in any one of claims 1-10.