Robot pose determination method and device, equipment, storage medium and program product

Abstract

The invention discloses a robot pose determination method and device, equipment, a storage medium and a program product, and relates to the technical field of machine learning, and the method comprises the steps: obtaining a two-dimensional image, a three-dimensional point cloud and a natural language-based task instruction text of a robot working scene; determining a region of interest corresponding to the to-be-executed task in the two-dimensional image according to the task instruction text, and predicting depth information of the region of interest; acquiring key feature points in the three-dimensional point cloud based on the region of interest and the depth information thereof; mapping each key feature point to a two-dimensional feature space to obtain a three-dimensional position indication vector of each key feature point; and according to the three-dimensional position indication vector, the target pose of the robot end effector is determined. The pose of the end effector of the robot is predicted by combining implicit depth perception and explicit spatial mapping, semantic understanding and spatial perception can be considered, and the pose prediction accuracy is improved, so that the operation performance, robustness and generalization ability of the robot are improved.

Description

Technical Field

[0001] This application relates to the field of machine learning technology, and in particular to a robot pose determination method, apparatus, device, storage medium, and program product. Background Technology

[0002] Visual perception-based dexterous manipulation of robots is a core research direction in the field of robotics. It aims to enable robots to understand unstructured environments through visual sensors and predict the action posture of end effectors when performing tasks.

[0003] Currently, pose prediction for robot end effectors typically relies on isolated two-dimensional images or three-dimensional point cloud data, which makes it difficult to balance semantic understanding and spatial perception. This results in poor pose prediction performance, leading to insufficient robot operation performance, robustness, and generalization ability.

[0004] Specifically, pre-trained representation methods based on 2D images directly utilize features extracted from single or multiple 2D images, leveraging visual models pre-trained on large-scale internet data to obtain strong semantic priors, and predict robot actions through imitation learning and other methods. However, 2D images are essentially projections of the 3D world, severely lacking depth and precise spatial geometric information. This makes it difficult for models to accurately determine the relative position and distance of objects when precise spatial reasoning (such as jacks and stacking) or handling complex occlusion tasks are required. Furthermore, in cluttered scenes, due to the lack of depth and spatial geometric information, models struggle to effectively separate foreground objects from background interference, resulting in insufficient operational robustness. End-to-end policy learning methods based on 3D point clouds directly process geometric data such as 3D point clouds or voxels, learning features and predicting poses and actions through specialized 3D neural networks, theoretically possessing precise spatial perception capabilities. However, due to the severe scarcity of high-quality 3D operational data and the lack of large-scale pre-training foundation, the "semantics" learned by the model are often statistical correlations of geometric shapes rather than high-level, transferable abstract semantic concepts. The lack of task understanding and planning capabilities leads to weak model generalization ability and difficulty in adapting to unseen objects or environments.

[0005] The above content is only used to help understand the technical solution of this application and does not represent an admission that the above content is prior art. Summary of the Invention

[0006] The main objective of this application is to provide a robot pose determination method, apparatus, device, storage medium, and program product, which aims to solve the technical problem that using two-dimensional images or three-dimensional point clouds in isolation makes it difficult to simultaneously take into account semantic understanding and spatial perception, resulting in poor prediction performance of robot pose and insufficient robot operation performance, robustness, and generalization ability.

[0007] To achieve the above objectives, this application proposes a robot pose determination method, which includes: Acquire multimodal data of the robot's working scene; the multimodal data includes two-dimensional images, three-dimensional point clouds, and task instruction text based on natural language, the task instruction text being used to indicate the task to be performed by the robot; Based on the task instruction text, determine the region of interest (ROI) corresponding to the task to be executed in the two-dimensional image, and predict the depth information of the ROI. Based on the depth information and the region of interest, key feature points in the 3D point cloud are obtained; Map each of the key feature points to a two-dimensional feature space to obtain the three-dimensional position indication vector of each of the key feature points; The target pose of the robot end effector is determined based on the three-dimensional position indication vector.

[0008] In one embodiment, the step of mapping each of the key feature points to a two-dimensional feature space to obtain a three-dimensional position indication vector for each of the key feature points includes: Construct a virtual geometry that surrounds each of the key feature points, and establish multiple virtual projection planes with different perspectives on the surface of the virtual geometry; The key feature points are mapped to a two-dimensional feature space through the virtual projection plane to obtain the three-dimensional position indication vector of each key feature point.

[0009] In one embodiment, the step of mapping each of the key feature points to a two-dimensional feature space through the virtual projection plane to obtain the three-dimensional position indication vector of each of the key feature points includes: Establish an index mapping relationship between the three-dimensional space where the three-dimensional point cloud is located and the two-dimensional plane corresponding to the virtual projection plane; Based on the index mapping relationship, calculate the coordinates of the projection points of the key feature points on each of the virtual projection planes; Obtain the two-dimensional position code corresponding to the coordinates of each projection point; The effectiveness weight is calculated based on the geometric relationship between the normal vector of the target feature point in the three-dimensional space and the observation direction of the virtual projection plane; the effectiveness weight characterizes the projection effectiveness. The two-dimensional position codes corresponding to the projection point coordinates of the target feature point on each of the virtual projection planes are weighted and aggregated according to the validity weight to obtain the three-dimensional position indication vector of the target feature point; the target feature point is any one of the key feature points.

[0010] In one embodiment, the step of determining the region of interest (ROI) in the two-dimensional image corresponding to the task to be performed based on the task instruction text, and predicting the depth information of the ROI, includes: The text features of the task instruction text and the visual features of the two-dimensional image are extracted respectively. Calculate the similarity matrix between the visual features and the text features, and generate a task attention heatmap based on the similarity matrix; the task attention heatmap is used to characterize the semantic relevance between each region in the two-dimensional image and the task instruction text. Based on the task-focused heatmap, the region of interest corresponding to the task to be executed in the two-dimensional image is determined; The image patch corresponding to the region of interest is input into the visual base model to obtain the depth information of the region of interest output by the visual base model; The visual foundation model includes a backbone network and a geometric reconstruction decoder. The depth information is obtained by the geometric reconstruction decoder predicting the depth of the patches in the region of interest. The geometric reconstruction decoder is pre-trained based on a hybrid loss function with the model parameters of the backbone network frozen. The hybrid loss function includes a depth reconstruction loss and a feature distillation loss. The depth reconstruction loss characterizes the difference between the predicted depth and the true depth of the geometric reconstruction decoder, and the feature distillation loss is used to constrain the consistency between the current output and the original output of the visual foundation model.

[0011] In one embodiment, the step of determining the region of interest in the two-dimensional image corresponding to the task to be performed based on the task attention heatmap includes: The dynamic threshold is determined based on the numerical distribution of each pixel in the heatmap of the task's focus. A non-uniform masking strategy is adopted to generate a mask matrix based on the dynamic threshold; The two-dimensional image is masked according to the mask matrix to divide the foreground and background regions of the two-dimensional image; the foreground region is the region of interest in the two-dimensional image corresponding to the task to be performed.

[0012] In one embodiment, the step of determining the target pose of the robot end effector based on the three-dimensional position indication vector includes: The key feature points are input into a pre-trained 3D word segmenter, which aggregates the local geometric features of each key feature point and maps the local geometric features into high-dimensional features using a multilayer perceptron. The high-dimensional features and the three-dimensional position indication vector are concatenated and fused to obtain a feature sequence with explicit spatial position information; Obtain the robot's body state; the body state includes the initial pose of the robot's end effector; The feature sequence and the ontology state are input into the visual base model; the visual base model also includes a low-rank adapter and a policy head; the low-rank adapter is obtained by fine-tuning the training while freezing the model parameters of the backbone network; The target pose of the robot end effector, output by the visual base model, is obtained by predicting the output features of the low-rank adapter using the strategy head.

[0013] Furthermore, to achieve the above objectives, this application also proposes a robot pose determination device, which includes: The data acquisition module is used to acquire multimodal data of the robot's working scene; the multimodal data includes two-dimensional images, three-dimensional point clouds, and task instruction text based on natural language, the task instruction text being used to indicate the tasks to be performed by the robot; The depth perception module is used to determine the region of interest corresponding to the task to be executed in the two-dimensional image based on the task instruction text, and to predict the depth information of the region of interest. The downsampling module is used to obtain key feature points in the 3D point cloud based on the depth information and the region of interest. A three-dimensional mapping module is used to map each of the key feature points to a two-dimensional feature space to obtain a three-dimensional position indication vector of each of the key feature points. The pose prediction module is used to determine the target pose of the robot end effector based on the three-dimensional position indication vector.

[0014] In addition, to achieve the above objectives, this application also proposes a robot control device, the device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the robot pose determination method as described above.

[0015] In addition, to achieve the above objectives, this application also proposes a storage medium, which is a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the steps of the robot pose determination method described above.

[0016] In addition, to achieve the above objectives, this application also provides a computer program product, which includes a computer program that, when executed by a processor, implements the steps of the robot pose determination method described above.

[0017] One or more technical solutions proposed in this application have at least the following technical effects: By acquiring multimodal data such as 2D images, 3D point clouds, and natural language-based task instruction text of the robot's working scene, this solution determines the region of interest (ROI) corresponding to the task to be executed in the 2D image based on the task instruction text, and predicts the depth information of the ROI. The task instruction text is used to instruct the robot on the task to be executed. Based on the depth information and the ROI, key feature points in the 3D point cloud are obtained. By mapping each key feature point to a 2D feature space, a 3D position indication vector for each key feature point is obtained. Based on the 3D position indication vectors of the key feature points, the target pose of the robot's end effector is determined. This technical solution fuses 2D images and 3D data, utilizes task instruction text to guide semantic understanding, locates the ROI, and automatically filters background noise, solving problems such as weak semantic understanding and background interference. Furthermore, it achieves implicit geometric understanding of the 2D image through depth estimation, employing a combination of implicit depth perception enhancement and explicit spatial mapping to balance semantic understanding and spatial perception, improving the accuracy of robot pose prediction, thereby enhancing the robot's operational performance, robustness, and generalization ability. Attached Figure Description

[0018] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0019] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a flowchart illustrating an embodiment of the robot pose determination method of this application. Figures 2-3 A schematic diagram illustrating the calculation principle of the effectiveness weights provided in the embodiments of the robot pose determination method of this application; Figure 4 This is a simplified flowchart illustrating the robot pose determination method provided in Embodiment 2 of this application; Figure 5 This is a schematic diagram of the module structure of the robot pose determination device according to an embodiment of this application; Figure 6 This is a schematic diagram of the device structure of the hardware operating environment involved in the robot pose determination method in this application embodiment.

[0021] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0022] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0023] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0024] The main solution of this application embodiment is as follows: Based on multimodal data of the robot's working scene, a dual mechanism of implicit depth reconstruction and explicit spatial mapping is used to balance high-precision 3D spatial perception and strong generalization semantic understanding. The implicit depth reconstruction mechanism forces the model to learn to perceive depth geometry, while the explicit spatial mapping mechanism embeds 3D point cloud data into a 2D feature space losslessly through multi-view projection. This enables the robot to understand the category and semantics of objects as if processing 2D images, and to accurately grasp the position and posture of objects as if processing 3D point clouds, thereby accurately predicting the robot's action pose and improving the robot's performance in high-precision tasks such as precision assembly and stacking.

[0025] In this embodiment, for ease of description, the robot control device will be used as the execution subject in the following description.

[0026] In existing technologies, when predicting the pose of a robot performing a task, two-dimensional images or three-dimensional point cloud data are usually used in isolation, which makes it difficult to take into account both semantic understanding and spatial perception. As a result, it is difficult to accurately predict the pose of the robot, leading to poor operational performance of the robot when performing the task.

[0027] This application provides a solution that combines implicit depth perception with explicit spatial mapping, fusing two-dimensional images and three-dimensional point cloud data. This allows for the prediction of robot pose while taking into account both semantic understanding and spatial perception capabilities, thereby accurately predicting the robot's motion pose, improving the robot's performance when performing tasks, and enabling the robot to complete high-precision operations under the control of the predicted pose.

[0028] It should be noted that the executing entity in this embodiment can be a computing service device with data processing, network communication, and program execution functions, such as a tablet computer, personal computer, or mobile phone, or an electronic device, robot device, robot control device, or controller capable of performing the above functions. The following description uses a robot control device as an example to illustrate this embodiment and the subsequent embodiments.

[0029] Based on this, embodiments of this application provide a robot pose determination method, referring to... Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the robot pose determination method of this application.

[0030] In this embodiment, the robot pose determination method includes steps S10 to S50: S10, acquire multimodal data of the robot's working scene; the multimodal data includes two-dimensional images, three-dimensional point clouds, and task instruction text based on natural language, the task instruction text being used to indicate the task to be performed by the robot; S20, based on the task instruction text, determine the region of interest in the two-dimensional image corresponding to the task to be executed, and predict the depth information of the region of interest; S30, Based on the depth information and the region of interest, obtain the key feature points in the 3D point cloud; S40, map each of the key feature points to a two-dimensional feature space to obtain the three-dimensional position indication vector of each of the key feature points; S50, determine the target pose of the robot end effector based on the three-dimensional position indication vector.

[0031] Acquire multimodal data of the robot's working scene, including 2D images, 3D point clouds, and task instruction text based on natural language. The task instruction text in the multimodal data is used to indicate the tasks to be performed by the robot, which can be object stacking or socket assembly; the 2D images can be RGB images, without specific limitations.

[0032] In one embodiment, multimodal data can be acquired by a data acquisition device mounted on the robot body. For example, two-dimensional images and three-dimensional point clouds can be acquired by a vision sensor. The task instruction text based on natural language can be input by the user, such as "put the red square into the blue box".

[0033] It should be noted that the robot's working environment is its workspace. Multimodal data of the robot's workspace is collected through data acquisition devices installed on the robot itself.

[0034] Furthermore, based on the task instruction text, the region of interest (ROI) corresponding to the task to be executed in the two-dimensional image is determined, and the depth information of the ROI is predicted. In one embodiment, the task instruction text and the two-dimensional image are semantically aligned to locate the region in the two-dimensional image that is directly related to the task instruction text, or that is highly related to the task instruction text; this region is the ROI.

[0035] Based on the region of interest (ROI) and the predicted depth information of the ROI, key feature points in the 3D point cloud are determined. Key feature points in the 3D point cloud are points corresponding to the ROI. In one embodiment, the 3D point cloud is downsampled according to the ROI and its depth information to obtain the key feature points.

[0036] Optionally, the three-dimensional point cloud is downsampled using the farthest point sampling algorithm to retain key geometric feature points in the three-dimensional point cloud, thereby obtaining key feature points that are directly related to or highly related to the task to be executed as indicated by the task instruction text.

[0037] By mapping the key feature points of the 3D point cloud to the 2D feature space, the 3D position indication vector of each key feature point is obtained. Based on the 3D position indication vector, the direction of motion of the robot when performing the task indicated by the task instruction text is determined, thereby determining the target pose of the robot end effector.

[0038] In one embodiment, based on the correspondence between the two-dimensional image represented by the key feature points and the three-dimensional point cloud, a mapping relationship between the three-dimensional space and the two-dimensional feature space is established. According to the established mapping relationship, the key feature points in the three-dimensional point cloud are mapped to the two-dimensional feature space to obtain the three-dimensional position indication vector of each key feature point.

[0039] Based on the region of interest and its depth information, key feature points in the 3D point cloud are determined. On the one hand, this allows for the understanding of the semantics of the 2D image while also taking into account the spatial perception capabilities of the 3D point cloud. On the other hand, by downsampling the 3D point cloud and retaining only the key feature points that are highly relevant to the task to be performed, the amount of data that needs to be processed can be reduced, thereby better meeting the real-time response requirements of the robot when performing the task.

[0040] In this embodiment, multimodal data such as 2D images, 3D point clouds, and natural language-based task instruction text of the robot's working scene are acquired. Based on the task instruction text, the region of interest (ROI) corresponding to the task to be executed in the 2D image is determined, and the depth information of the ROI is predicted. The task instruction text is used to instruct the robot on the task to be executed. Based on the depth information and the ROI, key feature points in the 3D point cloud are acquired. By mapping each key feature point to a 2D feature space, a 3D position indication vector for each key feature point is obtained. Based on the 3D position indication vectors of the key feature points, the target pose of the robot's end effector is determined. This technical solution fuses 2D images and 3D data, utilizes task instruction text to guide semantic understanding, locates the ROI, and automatically filters background noise, solving problems such as weak semantic understanding and background interference. Furthermore, implicit geometric understanding of the 2D image is achieved through depth estimation. A combination of implicit depth perception enhancement and explicit spatial mapping is used to balance semantic understanding and spatial perception, improving the accuracy of robot pose prediction, thereby enhancing the robot's operational performance, robustness, and generalization ability.

[0041] In one feasible implementation, step S20 may include steps S21 to S24: S21, extract the text features of the task instruction text and the visual features of the two-dimensional image respectively; S22, calculate the similarity matrix between the visual features and the text features, and generate a task attention heatmap based on the similarity matrix; the task attention heatmap is used to characterize the semantic relevance between each region in the two-dimensional image and the task instruction text. S23, Based on the task attention heatmap, determine the region of interest in the two-dimensional image corresponding to the task to be executed; S24, input the image patch corresponding to the region of interest into the visual base model to obtain the depth information of the region of interest output by the visual base model; The visual foundation model includes a backbone network and a geometric reconstruction decoder. The depth information is obtained by the geometric reconstruction decoder predicting the depth of the patches in the region of interest. The geometric reconstruction decoder is pre-trained based on a hybrid loss function with the model parameters of the backbone network frozen. The hybrid loss function includes a depth reconstruction loss and a feature distillation loss. The depth reconstruction loss characterizes the difference between the predicted depth and the true depth of the geometric reconstruction decoder, and the feature distillation loss is used to constrain the consistency between the current output and the original output of the geometric reconstruction decoder.

[0042] When determining the region of interest in a 2D image corresponding to the task to be performed, cross-modal semantic alignment of the task instruction text and the 2D image is required.

[0043] Specifically, firstly, the text features of the task instruction text and the visual features of the two-dimensional image are extracted respectively, and the similarity between the text features and the visual features is calculated to obtain the corresponding similarity matrix.

[0044] Then, a task attention heatmap is generated based on the similarity matrix. This heatmap represents the semantic relevance between each region in the 2D image and the task instruction text. Generating a task attention heatmap from task text instructions automatically filters out background noise irrelevant to the task, allowing the model to focus on the "affordance region" related to the task operation. This enables the robot to maintain stable recognition and operation capabilities even in real-world work scenarios with drastic lighting changes, cluttered backgrounds, or occlusions, improving the robot's control stability and robustness.

[0045] Finally, based on the task focus heatmap, the region of interest (ROI) corresponding to the task to be executed indicated by the task instruction text in the two-dimensional image is determined. The patch corresponding to the ROI is then input into the visual base model to obtain the depth information of the ROI output by the visual base model.

[0046] Among them, the visual foundation model is a two-dimensional visual foundation model, such as the ViT (Vision Transformer) model or the CLIP (Contrastive Language-Image Pretraining) model, which is a pre-trained visual model.

[0047] In one embodiment, a pre-trained visual foundation model, such as the CLIP model, is used to perform cross-modal semantic alignment between the task instruction text and the two-dimensional image. The visual features of the two-dimensional image and the textual features of the task instruction text are extracted respectively, and the similarity between the two is calculated to obtain a similarity matrix. A task attention heatmap is generated based on the similarity matrix.

[0048] Furthermore, the depth information of the region of interest (ROI) is obtained by the visual base model predicting the depth of the ROI tiles. This visual base model includes a backbone network and a geometric reconstruction decoder. The geometric reconstruction decoder is pre-trained based on a constructed hybrid loss function with the backbone network's model parameters frozen. This hybrid loss function includes depth reconstruction loss and feature distillation loss. The depth reconstruction loss characterizes the difference between the predicted depth and the true depth of the geometric reconstruction decoder, while the feature distillation loss constrains the consistency between the current output and the original output of the geometric reconstruction decoder, preventing catastrophic forgetting.

[0049] In one embodiment, to enable the two-dimensional vision base model to perceive three-dimensional geometric structures, a task-aware masked autoencoder mechanism is used to extract implicit visual features. Based on this, step S23 further includes steps A11 to A13: A11, determine the dynamic threshold based on the numerical distribution of each pixel in the task-focused heatmap; A12, employing a non-uniform masking strategy, generates a mask matrix based on the dynamic threshold; A13, perform a masking operation on the two-dimensional image according to the mask matrix to divide the foreground region and background region of the two-dimensional image; the foreground region is the region of interest corresponding to the task to be executed in the two-dimensional image.

[0050] The dynamic threshold is determined based on the numerical distribution of each pixel in the task-focused heatmap. This dynamic threshold can be a feature value calculated based on the pixel value of the pixel. This feature value can be one of the median, mean, mode, and preset percentile, without any specific limitation.

[0051] A non-uniform masking strategy is employed, generating a mask matrix based on a threshold. This mask matrix is ​​then used to perform a masking operation on the 2D image, dividing it into foreground and background regions. The non-uniform masking strategy involves preserving foreground regions with a preset first high probability and occluding background regions with a preset second high probability. The unoccluded foreground regions are the regions of interest corresponding to the task command text.

[0052] It should be noted that the first and second highest probabilities can be the same or different; no specific restrictions are imposed on this.

[0053] Furthermore, the unoccluded foreground region tiles, i.e. region of interest tiles, are input into the visual base model. The depth information of the region of interest tiles is predicted by the geometric reconstruction decoder of the visual base model, and implicit visual features are extracted.

[0054] In one feasible implementation, the visual base model encodes the input tiles and concatenates the encoding with a learnable mask token into a geometry reconstruction decoder, which forces the model to reconstruct a depth map of the scene.

[0055] To enable the visual base model to retain general semantic knowledge while learning geometric information, a hybrid loss function for training the geometric reconstruction decoder is constructed as follows: (1) In Formula 1, The depth reconstruction loss measures the difference between the predicted depth and the true depth, and has the following characteristics: (2) Feature distillation loss is used to constrain the consistency between the current output features of the model and the output features of the original frozen model, preventing catastrophic forgetting. (3) In formulas 1-3, For balance coefficient, To predict depth, For true depth, This represents the current output feature of the visual base model. This represents the output features of the original frozen model, i.e., the original output features of the visual base model.

[0056] In one embodiment, key feature points of a 3D point cloud are mapped to a 2D feature space, solving the problem that 2D models cannot directly process 3D point clouds. Furthermore, by establishing a virtual projection space for multi-view projection, lossless mapping of 3D information is achieved. Specifically, step S40 includes steps S41-S42: S41, construct a virtual geometry surrounding each of the key feature points, and establish multiple virtual projection planes with different perspectives on the surface of the virtual geometry; S42, through the virtual projection plane, each of the key feature points is mapped to a two-dimensional feature space to obtain the three-dimensional position indication vector of each of the key feature points.

[0057] A virtual geometry is constructed that surrounds each key feature point, and multiple virtual projection planes with different perspectives are set on the surface of the virtual geometry. Through the established virtual projection planes, each key feature point is projected from multiple perspectives, thereby mapping each key feature point to a two-dimensional feature space and obtaining the three-dimensional position indication vector of each key feature point.

[0058] In one embodiment, step S42 further includes steps B11 to B15: B11, Establish an index mapping relationship between the three-dimensional space where the three-dimensional point cloud is located and the two-dimensional plane corresponding to the virtual projection plane; B12, Based on the index mapping relationship, calculate the coordinates of the projection points of the key feature points on each of the virtual projection planes; B13, obtain the two-dimensional position code corresponding to the coordinates of each projection point; B14, The effectiveness weight is calculated based on the geometric relationship between the normal vector of the target feature point in the three-dimensional space and the observation direction of the virtual projection plane; the effectiveness weight characterizes the projection effectiveness; B15, according to the validity weight, the two-dimensional position codes corresponding to the projection point coordinates of the target feature point on each of the virtual projection planes are weighted and aggregated to obtain the three-dimensional position indication vector of the target feature point; the target feature point is any one of the key feature points.

[0059] After establishing a virtual projection plane on the surface of the virtual geometry surrounding each key feature point, an index mapping relationship is established between the three-dimensional space where the three-dimensional point cloud is located and the two-dimensional plane corresponding to the virtual projection plane. Based on this index mapping relationship, the coordinates of the projection points of each key feature point on each virtual projection plane are calculated.

[0060] Furthermore, the two-dimensional position codes corresponding to the projection coordinates of each key feature point on the virtual projection plane are obtained. Based on the geometric relationship between the normal vector of the target feature point in three-dimensional space and the observation direction of the virtual projection plane, the effectiveness weight of the target feature point is calculated. This effectiveness weight characterizes the projection effectiveness of the target feature point. Based on the calculated effectiveness weight, the two-dimensional position codes corresponding to the projection coordinates of the target feature point on each virtual projection plane are weighted and aggregated to obtain the three-dimensional position vector of the target feature point. Here, the target feature point is any one of the key feature points, and the effectiveness weight is the weight value of the weighted aggregation.

[0061] In one embodiment, a virtual geometry, such as a cube, is constructed that surrounds key feature points. K virtual projection planes are set on its surface. Different virtual projection planes correspond to different projection perspectives. For example, the K virtual projection planes correspond to six different perspectives: front, back, left, right, top, and bottom.

[0062] An index mapping relationship between 3D space and 2D planes is established. For any point in the 3D point cloud, its projection coordinates on each virtual projection plane are calculated. To generate a unified 3D position code for each key feature point, the 2D position codes corresponding to the projection points on each virtual projection plane are first obtained by querying the pre-trained 2D vision basic model, thereby obtaining the 2D position codes of the projection coordinates of each key feature point on the virtual projection plane.

[0063] Then, a weighted aggregation model based on projection effectiveness is used to generate the three-dimensional position indicator vectors of the key feature points. The weighted aggregation model is shown in Equation 4 below: (4) Where K represents the number of virtual projection planes. This is to prevent tiny constants with a denominator of zero. This represents the projection effectiveness weight of the i-th key feature point on the k-th virtual projection plane. This represents the two-dimensional positional encoding of the projection coordinates of the i-th key feature point on the k-th virtual projection plane. This represents the three-dimensional position indicator vector of the i-th key feature point.

[0064] In one embodiment, the validity weight is calculated based on the geometric relationship between the normal vector n of the key feature point and the observation direction v of the virtual projection plane: .

[0065] The calculation principle of validity weight is as follows: Figure 2 and Figure 3 As shown, Figure 2 and Figure 3 Examples are given of different projection validity cases between the i-th key feature point and the k-th virtual projection plane. Specifically, as shown... Figure 2 As shown, under the projected viewpoint, the normal vector of the i-th key feature point is... Observation direction relative to the k-th virtual projection plane The angle between For small included angles less than 90 degrees, then: (Large positive value), in this case, the validity weight of the projection point of the i-th key feature point on the k-th virtual projection plane. This indicates a high weighting, which falls under the category of high-efficiency weighting. Correspondingly, such as... Figure 3 As shown, the normal vector of the i-th key feature point is shown in the side or rear view projection perspective. Observation direction relative to the k-th virtual projection plane The angle between For a large included angle greater than or equal to 90 degrees, then: In this case, the validity weight of the projection point of the i-th key feature point on the k-th virtual projection plane. It has a low or zero weight, which is a case of low / zero effectiveness weight.

[0066] Following the above method, when the surface of a point in a 3D point cloud is directly facing the virtual projection plane, the validity weight, which characterizes the projection validity, is maximized, thus effectively eliminating the influence of viewpoint edge distortion. Therefore, the validity weight of the same key feature point on different virtual projection planes varies depending on the projection viewpoint, and the validity weight of different key feature points on the same virtual projection plane may also differ due to the angle between their normal vector and the observation direction of the virtual projection plane.

[0067] Instead of simple feature stitching or averaging, this method introduces a weighted aggregation model based on projection validity weights (based on the angle between normal vectors or occlusion relationships) to perform weighted aggregation of position codes from multiple views. This solves the problems of information loss and viewpoint distortion when projecting 3D point clouds into 2D features. Furthermore, it can adaptively fuse position information from virtual projection planes from different viewpoints based on the geometric features of the point cloud surface, effectively eliminating blind spots caused by a single viewpoint and significantly improving the accuracy of 3D position information representation. This, in turn, helps to improve the accuracy and precision of robot pose prediction.

[0068] In one feasible implementation, step S50 includes steps S51 to S55: S51, the key feature points are input into a pre-trained three-dimensional word segmenter, the local geometric features of each key feature point are aggregated by the three-dimensional word segmenter, and the local geometric features are mapped into high-dimensional features by a multilayer perceptron; S52, the high-dimensional features and the three-dimensional position indication vector are concatenated and fused to obtain a feature sequence with explicit spatial position information; S53, Obtain the body state of the robot; the body state includes the initial pose of the robot's end effector; S54, the feature sequence and the ontology state are input into the visual base model; the visual base model further includes a low-rank adapter and a policy head; the low-rank adapter is obtained by fine-tuning the training while freezing the model parameters of the backbone network; S55, the target pose of the robot end effector is obtained by predicting the output features of the low-rank adapter through the strategy head.

[0069] Each key feature point is input into a pre-trained 3D word segmenter. The 3D word segmenter aggregates the local geometric features of each key feature point. Then, a multilayer perceptron maps the local geometric features into high-dimensional features, which can be 3D features or features of higher dimensions.

[0070] The high-dimensional features and the three-dimensional position indicator vector are concatenated and fused to obtain a feature sequence with explicit spatial position information.

[0071] The robot's body state is obtained, including the initial pose of the robot's end effector. The fused feature sequence and the robot's body state are input into the visual base model to obtain the target pose of the robot's end effector output by the visual base model.

[0072] The visual base model includes a low-rank adapter and a policy head. The low-rank adapter is fine-tuned by training the backbone network of the visual base model while freezing its model parameters. The fused feature sequence and the robot's ontological state are input into the visual base model. The policy head predicts the output features of the low-rank adapter to obtain the target pose of the robot's end effector, which is output by the visual base model.

[0073] In one embodiment, the robot's body state includes joint angles and angular velocities. The fused feature sequence, along with the robot's body state, is then input into a parameter-frozen two-dimensional vision base model. At this point, only the pre-inserted low-rank adapter (LoRA) parameters are activated in the model for calculation. The output features are passed through a strategy head to directly predict the target pose of the robot's end effector.

[0074] As a feasible implementation method, the target pose executed by the robot's end effector As shown below: (5) The target pose shown in Formula 5 includes: three translation vectors. This corresponds to translational movements in three spatial dimensions; and three rotation vectors. This corresponds to rotational movements in three spatial dimensions; each state vector represents the opening and closing state of the end effector gripper's gripper. .

[0075] In one embodiment, refer to Figure 4 The simplified flowchart of the robot pose determination method shown mainly includes implicit depth perception and explicit 3D space mapping. In the implicit depth perception stage, for the acquired multimodal data of the robot's workspace, a pre-trained image-text alignment model (such as the CLIP model) is used to perform cross-modal semantic alignment on the 2D images and task instruction text. The visual features of the 2D images and the textual features of the task instruction text are extracted respectively, and the similarity between the two is calculated to obtain a similarity matrix. A corresponding task attention heatmap is generated to characterize the relevance of each region in the 2D image to the task to be executed corresponding to the task instruction text.

[0076] Furthermore, such as Figure 4 As shown, a mask matrix is ​​generated based on a task-focused heatmap. Specifically, a dynamic threshold is determined based on the pixel value distribution of the pixels in the task-focused heatmap, and this dynamic threshold is used to divide the two-dimensional image into foreground and background regions. As a feasible implementation, regions composed of pixels with values ​​less than the dynamic threshold are designated as background regions, and regions composed of pixels with values ​​greater than or equal to the dynamic threshold are designated as foreground regions. The task-focused heatmap is then binarized to obtain the mask matrix.

[0077] A non-uniform masking strategy is employed, performing masking operations on the 2D image based on the mask matrix to preserve the foreground region with a high probability and occlude the background region with a high probability. The unoccluded foreground region is the region of interest (ROI) corresponding to the task to be performed. The image patches corresponding to the ROI are input into the visual base model, and the trainable geometric reconstruction decoder in the visual base model is used to reconstruct the depth of the ROI patches to obtain the depth information of the ROI.

[0078] In the implicit depth perception stage, a task attention heatmap is generated through cross-modal semantic alignment to guide mask operations. The visual base model reconstructs the depth of the scene related to the task to be performed through a trainable geometric reconstruction decoder, thereby endowing the two-dimensional visual base model with implicit geometric understanding and spatial perception capabilities.

[0079] In the explicit 3D spatial mapping stage, based on the region of interest and its depth information obtained in the implicit depth perception stage, the 3D point cloud in the multimodal data is downsampled using algorithms such as farthest point sampling. Key feature points related to the task to be performed are retained, and multi-view projection is performed on each key point to explicitly and losslessly map the key feature points of the 3D point cloud into the 2D feature space. Without destroying the semantic space of the 2D vision basic model, lossless inheritance of large-scale general vision knowledge and accurate encoding of 3D geometric information are achieved.

[0080] In the multi-view projection process for each key feature point, a virtual geometry that can surround all key feature points is first created. Multiple virtual projection planes with different viewpoints are then established on the surface of this virtual geometry, forming a three-dimensional virtual projection space. Each key feature point is then projected through these virtual projection planes, mapping it into a two-dimensional feature space to obtain the corresponding three-dimensional position code, i.e., the three-dimensional position indicator vector.

[0081] Furthermore, for multi-view projection, it is first necessary to establish an index mapping relationship between the 3D space or virtual projection space where the 3D point cloud is located and the 2D plane corresponding to the virtual projection plane. Then, for the multi-view projection of any key feature point, on the one hand, based on the established index mapping relationship between the 3D space and the 2D plane, the coordinates of the projection point of the key feature point on each virtual projection plane are calculated, and the 2D position code corresponding to the projection point coordinates is obtained by looking up tables or other methods. On the other hand, according to the geometric relationship between the normal vector of the key feature point in the 3D space and the observation direction of the virtual projection plane, the projection validity weight of the key feature point on the virtual projection plane is calculated. Finally, the calculated validity weight is used as the weight value of the 2D position code, and the 2D position codes corresponding to the projection point coordinates of the key feature point on each virtual projection plane are weighted and aggregated to obtain the 3D position indication vector represented by the 3D position code of the key feature point.

[0082] The 3D position indication vector obtained from multi-view projection is fused with the high-dimensional features output by the 3D word segmenter to obtain a feature sequence with explicit spatial position information. The 3D word segmenter is a trainable lightweight word segmentation model that uses the K-Nearest Neighbors (K-NN) algorithm to aggregate the local geometric features of key feature points and maps each key feature point to a high-dimensional feature through a multilayer perceptron (MLP).

[0083] The fused feature sequence, along with the robot's body state, is input into the visual base model. At this point, only the pre-inserted low-rank adapter parameters are activated for calculation in the model. The output features of the low-rank adapter are passed through a policy head to directly predict the target pose of the robot's end effector.

[0084] In this embodiment, multimodal data of the robot's working scene is fused by combining implicit depth perception and explicit 3D spatial mapping. Specifically, depth reconstruction endows the 2D vision base model with implicit 3D spatial perception capabilities, while multi-view projection explicitly maps key feature points of the 3D point cloud to the 2D feature space. This achieves lossless inheritance of large-scale general visual knowledge and precise encoding of 3D geometric information without destroying the model's semantic space. Thus, when predicting the robot's pose for performing tasks, both semantic understanding and spatial perception capabilities are considered, improving the accuracy and precision of pose prediction. This facilitates precise 3D manipulation of the robot to perform precise tasks such as socket assembly.

[0085] Furthermore, through a cross-modal semantic-guided masking mechanism, attention heatmaps related to the task to be executed are generated based on the task instruction text. This can automatically filter out background noise that is irrelevant to the task, allowing the model to focus on the functional areas related to the task operation. It also uses depth information instead of RGB images as the reconstruction target of the mask, enabling the model, which can only process two-dimensional images, to have the spatial perception ability to reason about three-dimensional spatial structures. This improves the robustness and generalization of the robot in complex working scenarios with varying lighting distances, cluttered backgrounds, or occlusions.

[0086] Furthermore, in multi-view projection, a weighted aggregation model is constructed using projection validity weights. This model computes 2D position codes from multiple perspectives in parallel, along with validity weights based on normal vectors. Through weighted aggregation, a unified 3D position code for the 3D point cloud is obtained. Finally, the 3D position code, incorporating explicit position information and the robot's body state, is input into a frozen backbone network configured with a low-rank adapter, directly predicting the target pose of the end effector when the robot performs its task.

[0087] Compared to using 2D images or 3D point clouds in isolation, there are hybrid methods that combine 2D images with 3D point clouds. For example, rendering 3D point clouds as multi-view images or attempting to upscale 2D features into 3D voxel meshes. The limitations of these hybrid methods lie in the information loss and semantic gap during modality transfer. For instance, when forcibly projecting 3D data into a 2D image, viewpoint occlusion inevitably occurs, leading to the loss of key geometric features. More seriously, feature upscaling often requires retraining positional encoding or significant fine-tuning of the pre-trained model. This disrupts the original semantic space structure of the 2D base model, causing "catastrophic forgetting" when adapting to 3D tasks. The model cannot effectively inherit the general visual knowledge contained in large-scale pre-trained models, making true few-shot transfer learning difficult.

[0088] In this embodiment, a parameter-efficient fine-tuning strategy of freezing the backbone and using a low-rank adapter is adopted for the visual base model. During model training, the massive pre-training parameters of the backbone network are frozen, and only a small number of parameters of the low-rank adapter are fine-tuned. This not only significantly reduces the computational resource overhead during model training but also completely preserves the general feature space of the pre-trained model. This effectively prevents the model from losing its original general capabilities due to forced adaptation to new modalities, as well as the problem of high dependence on demonstration data and low training efficiency in robot operation strategies. Furthermore, through the lossless and efficient mapping mechanism of multi-view projection, the problems of spatial information loss and semantic space misalignment in the "2D-3D" modal conversion can be solved. Mapping the 3D point cloud to the 2D feature space can completely preserve the spatial coordinate information of the 3D point cloud and align it with the original 2D position encoding of the 2D model. This prevents catastrophic forgetting during model fine-tuning and effectively inherits the general knowledge of the large-scale pre-trained model. This allows the robot to quickly adapt to new operating objects and complex environments with only a very small amount of real demonstration data, achieving low-cost and high-efficiency skill transfer.

[0089] It should be noted that the above examples are only for understanding this application and do not constitute a limitation on the robot pose determination method of this application. Any simple transformations based on this technical concept are all within the protection scope of this application.

[0090] This application also provides a robot pose determination device; please refer to [reference needed]. Figure 5 The robot pose determination device includes: The data acquisition module 10 is used to acquire multimodal data of the robot's working scene; the multimodal data includes two-dimensional images, three-dimensional point clouds, and task instruction text based on natural language, the task instruction text being used to indicate the task to be performed by the robot; The depth perception module 20 is used to determine the region of interest corresponding to the task to be executed in the two-dimensional image based on the task instruction text, and to predict the depth information of the region of interest. Downsampling module 30 is used to obtain key feature points in the 3D point cloud based on the depth information and the region of interest; The three-dimensional mapping module 40 is used to map each of the key feature points to a two-dimensional feature space to obtain the three-dimensional position indication vector of each of the key feature points. The pose prediction module 50 is used to determine the target pose of the robot end effector based on the three-dimensional position indication vector.

[0091] In one embodiment, the three-dimensional mapping module 40 is further configured to: Construct a virtual geometry that surrounds each of the key feature points, and establish multiple virtual projection planes with different perspectives on the surface of the virtual geometry; The key feature points are mapped to a two-dimensional feature space through the virtual projection plane to obtain the three-dimensional position indication vector of each key feature point.

[0092] In one embodiment, the three-dimensional mapping module 40 is further configured to: Establish an index mapping relationship between the three-dimensional space where the three-dimensional point cloud is located and the two-dimensional plane corresponding to the virtual projection plane; Based on the index mapping relationship, calculate the coordinates of the projection points of the key feature points on each of the virtual projection planes; Obtain the two-dimensional position code corresponding to the coordinates of each projection point; The effectiveness weight is calculated based on the geometric relationship between the normal vector of the target feature point in the three-dimensional space and the observation direction of the virtual projection plane; the effectiveness weight characterizes the projection effectiveness. The two-dimensional position codes corresponding to the projection point coordinates of the target feature point on each of the virtual projection planes are weighted and aggregated according to the validity weight to obtain the three-dimensional position indication vector of the target feature point; the target feature point is any one of the key feature points.

[0093] In one embodiment, the depth sensing module 20 is further configured to: The text features of the task instruction text and the visual features of the two-dimensional image are extracted respectively. Calculate the similarity matrix between the visual features and the text features, and generate a task attention heatmap based on the similarity matrix; the task attention heatmap is used to characterize the semantic relevance between each region in the two-dimensional image and the task instruction text. Based on the task-focused heatmap, the region of interest corresponding to the task to be executed in the two-dimensional image is determined; The image patch corresponding to the region of interest is input into the visual base model to obtain the depth information of the region of interest output by the visual base model; The visual foundation model includes a backbone network and a geometric reconstruction decoder. The depth information is obtained by the geometric reconstruction decoder predicting the depth of the patches in the region of interest. The geometric reconstruction decoder is pre-trained based on a hybrid loss function with the model parameters of the backbone network frozen. The hybrid loss function includes a depth reconstruction loss and a feature distillation loss. The depth reconstruction loss characterizes the difference between the predicted depth and the true depth of the geometric reconstruction decoder, and the feature distillation loss is used to constrain the consistency between the current output and the original output of the visual foundation model.

[0094] In one embodiment, the depth sensing module 20 is further configured to: The dynamic threshold is determined based on the numerical distribution of each pixel in the heatmap of the task's focus. A non-uniform masking strategy is adopted to generate a mask matrix based on the dynamic threshold; The two-dimensional image is masked according to the mask matrix to divide the foreground and background regions of the two-dimensional image; the foreground region is the region of interest in the two-dimensional image corresponding to the task to be performed.

[0095] In one embodiment, the pose prediction module 50 is further configured to: The key feature points are input into a pre-trained 3D word segmenter, which aggregates the local geometric features of each key feature point and maps the local geometric features into high-dimensional features using a multilayer perceptron. The high-dimensional features and the three-dimensional position indication vector are concatenated and fused to obtain a feature sequence with explicit spatial position information; Obtain the robot's body state; the body state includes the initial pose of the robot's end effector; The feature sequence and the ontology state are input into the visual base model; the visual base model also includes a low-rank adapter and a policy head; the low-rank adapter is obtained by fine-tuning the training while freezing the model parameters of the backbone network; The target pose of the robot end effector, output by the visual base model, is obtained by predicting the output features of the low-rank adapter using the strategy head.

[0096] The robot pose determination device provided in this application, employing the robot pose determination method in the above embodiments, can solve the technical problem that predicting the pose of a robot end effector using isolated two-dimensional images or three-dimensional point cloud data is difficult to balance semantic understanding and spatial perception, resulting in poor prediction performance of robot pose and insufficient robot operation performance, robustness, and generalization ability. Compared with the prior art, the beneficial effects of the robot pose determination device provided in this application are the same as those of the robot pose determination method provided in the above embodiments, and other technical features in the robot pose determination device are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.

[0097] This application provides a robot control device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the robot pose determination method in the above embodiment 1.

[0098] The following is for reference. Figure 6 The diagram illustrates a structural schematic suitable for implementing the robot control device of the embodiments of this application. The robot control device in the embodiments of this application may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Description), PMPs (Portable Media Players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 6 The robot control device shown is merely an example and should not be construed as limiting the functionality and scope of use of the embodiments of this application.

[0099] like Figure 6As shown, the robot control device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage device 1003 into a random access memory (RAM) 1004. The RAM 1004 also stores various programs and data required for the operation of the robot control device. The processing unit 1001, ROM 1002, and RAM 1004 are interconnected via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to the I / O interface 1006: input devices 1007 including, for example, a touchscreen, touchpad, keyboard, mouse, image sensor, microphone, accelerometer, gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage devices 1003 including, for example, magnetic tape, hard disk, etc.; and communication devices 1009. Communication device 1009 allows the robot control device to communicate wirelessly or wiredly with other devices to exchange data. Although the figure shows robot control devices with various systems, it should be understood that implementation or possession of all the systems shown is not required. More or fewer systems may be implemented alternatively.

[0100] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from ROM 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0101] The robot control device provided in this application, employing the robot pose determination method described in the above embodiments, solves the technical problem that predicting the pose of a robot's end effector using isolated two-dimensional images or three-dimensional point cloud data is difficult to balance semantic understanding and spatial perception, resulting in poor robot pose prediction performance and insufficient robot operation performance, robustness, and generalization ability. Compared with the prior art, the beneficial effects of the robot control device provided in this application are the same as those of the robot pose determination method provided in the above embodiments, and other technical features in this robot control device are the same as those disclosed in the previous embodiment method, and will not be repeated here.

[0102] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0103] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0104] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, which are used to execute the robot pose determination method in the above embodiments.

[0105] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems or devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0106] The aforementioned computer-readable storage medium may be included in the robot control device; or it may exist independently and not be assembled into the robot control device.

[0107] The aforementioned computer-readable storage medium carries one or more programs, which, when executed by a robot control device, cause the robot control device to perform the following steps: Acquire multimodal data of the robot's working scene; the multimodal data includes two-dimensional images, three-dimensional point clouds, and task instruction text based on natural language, the task instruction text being used to indicate the task to be performed by the robot; Based on the task instruction text, determine the region of interest (ROI) corresponding to the task to be executed in the two-dimensional image, and predict the depth information of the ROI. Based on the depth information and the region of interest, key feature points in the 3D point cloud are obtained; Map each of the key feature points to a two-dimensional feature space to obtain the three-dimensional position indication vector of each of the key feature points; The target pose of the robot end effector is determined based on the three-dimensional position indication vector.

[0108] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0109] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0110] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0111] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the above-described robot pose determination method. This addresses the technical problem that predicting the pose of a robot's end effector using isolated two-dimensional images or three-dimensional point cloud data is difficult to balance semantic understanding and spatial perception, resulting in poor robot pose prediction performance and insufficient robot operational performance, robustness, and generalization ability. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the robot pose determination method provided in the above embodiments, and will not be repeated here.

[0112] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the robot pose determination method described above.

[0113] The computer program product provided in this application can solve the technical problem that predicting the pose of a robot end effector using isolated two-dimensional images or three-dimensional point cloud data is difficult to balance semantic understanding and spatial perception, resulting in poor robot pose prediction performance and insufficient robot operation performance, robustness, and generalization ability. Compared with the prior art, the beneficial effects of the computer program product provided in this application are the same as those of the robot pose determination method provided in the above embodiments, and will not be repeated here.

[0114] The above description is only a part of the embodiments of this application and does not limit the patent scope of this application. All equivalent structural transformations made under the technical concept of this application and using the contents of the specification and drawings of this application, or direct / indirect applications in other related technical fields, are included in the patent protection scope of this application.

Claims

1. A method for determining robot pose, characterized in that, The robot pose determination method includes: Acquire multimodal data of the robot's working scene; the multimodal data includes two-dimensional images, three-dimensional point clouds, and task instruction text based on natural language, the task instruction text being used to indicate the task to be performed by the robot; Based on the task instruction text, determine the region of interest (ROI) corresponding to the task to be executed in the two-dimensional image, and predict the depth information of the ROI. Based on the depth information and the region of interest, key feature points in the 3D point cloud are obtained; Map each of the key feature points to a two-dimensional feature space to obtain the three-dimensional position indication vector of each of the key feature points; The target pose of the robot end effector is determined based on the three-dimensional position indication vector.

2. The robot pose determination method as described in claim 1, characterized in that, The step of mapping each of the key feature points to a two-dimensional feature space to obtain the three-dimensional position indication vector of each of the key feature points includes: Construct a virtual geometry that surrounds each of the key feature points, and establish multiple virtual projection planes with different perspectives on the surface of the virtual geometry; The key feature points are mapped to a two-dimensional feature space through the virtual projection plane to obtain the three-dimensional position indication vector of each key feature point.

3. The robot pose determination method as described in claim 2, characterized in that, The step of mapping each key feature point to a two-dimensional feature space through the virtual projection plane to obtain the three-dimensional position indication vector of each key feature point includes: Establish an index mapping relationship between the three-dimensional space where the three-dimensional point cloud is located and the two-dimensional plane corresponding to the virtual projection plane; Based on the index mapping relationship, calculate the coordinates of the projection points of the key feature points on each of the virtual projection planes; Obtain the two-dimensional position code corresponding to the coordinates of each projection point; The effectiveness weight is calculated based on the geometric relationship between the normal vector of the target feature point in the three-dimensional space and the observation direction of the virtual projection plane; the effectiveness weight characterizes the projection effectiveness. The two-dimensional position codes corresponding to the projection point coordinates of the target feature point on each of the virtual projection planes are weighted and aggregated according to the validity weight to obtain the three-dimensional position indication vector of the target feature point; the target feature point is any one of the key feature points.

4. The robot pose determination method according to any one of claims 1 to 3, characterized in that, The steps of determining the region of interest (ROI) in the two-dimensional image corresponding to the task to be executed based on the task instruction text, and predicting the depth information of the ROI, include: The text features of the task instruction text and the visual features of the two-dimensional image are extracted respectively. Calculate the similarity matrix between the visual features and the text features, and generate a task attention heatmap based on the similarity matrix; the task attention heatmap is used to characterize the semantic relevance between each region in the two-dimensional image and the task instruction text. Based on the task-focused heatmap, the region of interest corresponding to the task to be executed in the two-dimensional image is determined; The image patch corresponding to the region of interest is input into the visual base model to obtain the depth information of the region of interest output by the visual base model; The visual foundation model includes a backbone network and a geometric reconstruction decoder. The depth information is obtained by the geometric reconstruction decoder predicting the depth of the patches in the region of interest. The geometric reconstruction decoder is pre-trained based on a hybrid loss function with the model parameters of the backbone network frozen. The hybrid loss function includes a depth reconstruction loss and a feature distillation loss. The depth reconstruction loss characterizes the difference between the predicted depth and the true depth of the geometric reconstruction decoder, and the feature distillation loss is used to constrain the consistency between the current output and the original output of the visual foundation model.

5. The robot pose determination method as described in claim 4, characterized in that, The step of determining the region of interest in the two-dimensional image corresponding to the task to be executed based on the task attention heatmap includes: The dynamic threshold is determined based on the numerical distribution of each pixel in the heatmap of the task's focus. A non-uniform masking strategy is adopted to generate a mask matrix based on the dynamic threshold; The two-dimensional image is masked according to the mask matrix to divide the foreground and background regions of the two-dimensional image; the foreground region is the region of interest in the two-dimensional image corresponding to the task to be performed.

6. The robot pose determination method as described in claim 4, characterized in that, The step of determining the target pose of the robot end effector based on the three-dimensional position indication vector includes: The key feature points are input into a pre-trained 3D word segmenter, which aggregates the local geometric features of each key feature point and maps the local geometric features into high-dimensional features using a multilayer perceptron. The high-dimensional features and the three-dimensional position indication vector are concatenated and fused to obtain a feature sequence with explicit spatial position information; Obtain the robot's body state; the body state includes the initial pose of the robot's end effector; The feature sequence and the ontology state are input into the visual base model; the visual base model also includes a low-rank adapter and a policy head; the low-rank adapter is obtained by fine-tuning the training while freezing the model parameters of the backbone network; The target pose of the robot end effector, output by the visual base model, is obtained by predicting the output features of the low-rank adapter using the strategy head.

7. A robot pose determination device, characterized in that, The robot pose determination device includes: The data acquisition module is used to acquire multimodal data of the robot's working scene; the multimodal data includes two-dimensional images, three-dimensional point clouds, and task instruction text based on natural language, the task instruction text being used to indicate the tasks to be performed by the robot; The depth perception module is used to determine the region of interest corresponding to the task to be executed in the two-dimensional image based on the task instruction text, and to predict the depth information of the region of interest. The downsampling module is used to obtain key feature points in the 3D point cloud based on the depth information and the region of interest. A three-dimensional mapping module is used to map each of the key feature points to a two-dimensional feature space to obtain a three-dimensional position indication vector of each of the key feature points. The pose prediction module is used to determine the target pose of the robot end effector based on the three-dimensional position indication vector.

8. A robot control device, characterized in that, The device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the robot pose determination method as described in any one of claims 1 to 6.

9. A storage medium, characterized in that, The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, it implements the steps of the robot pose determination method as described in any one of claims 1 to 6.

10. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the steps of the robot pose determination method as described in any one of claims 1 to 6.

Images