Key point detection method, key point-based target operation method and robot
Through technologies such as 3D voxel representation and cross attention mechanism, the problem of insufficient consistency in key point detection time in the existing technology is solved, and the accuracy and robustness of the operation of target objects in dynamic scenarios is achieved, and it is suitable for complex Robot operation of hinged objects.
Patent Information
- Application Number
- CN202411148973.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-21
- Publication Date
- 2024-12-03
AI Technical Summary
The existing 2D and 3D key point detection methods ignore time consistency, resulting in robots' failure in operation of special objects, especially in dynamically changing and diverse hinged object scenes, which makes it difficult to achieve real-time and robustness.
Using 3D voxel representation, cross attention mechanism, 3D feature handling and implicit reconstruction methods, the key points of the initial and target point clouds can be obtained through deep neural networks and three-dimensional representation architectures, ensuring the spatial and temporal consistency of key points, and using singularity The value decomposition algorithm calculates motion information to control the operation of the robotic arm.
It realizes the accuracy and robustness of target recognition and object operation in dynamic scenes, can effectively capture and track changes in target objects, and is suitable for complex hinged objects and non-rigid objects, improving the accuracy and flexibility of robot operations. sex.
Smart Images

Figure CN119068021A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the technical field of key point detection, and in particular, to a key point detection method, a target operation method based on key point detection, and a robot. Background Art
[0002] With the development of artificial intelligence and computer vision technology, the generalization and intelligence level of robots have gradually improved, making it possible for robots to enter various scenarios such as home, commercial and industrial. There are a large number of special objects in different scenarios, such as refrigerators in home kitchens, whose refrigerator doors have rotating hinge joints; and push-pull devices and drawers in commercial scenarios, which have translational hinge joints. Operating these hinged objects is the basis for realizing the functions of robots, but the diversity of hinged objects, joint uncertainty (such as rotational joints and sliding joints), and the fact that each type may have different ranges and directions of motion make it difficult for robots to identify and operate them.
[0003] Keypoint-based representation has certain advantages in the application of computer vision and robotics, and can be applied to the operation of special objects (such as hinges, non-rigidity, dynamic changes, etc.). However, existing 2D and 3D keypoint detection methods mainly rely on geometric consistency to achieve spatial alignment, while ignoring temporal consistency. Temporal consistency is crucial for the prediction of object motion and the planning of robot actions. Ignoring temporal consistency will bring many problems such as real-time performance, robustness, data fusion, and operation accuracy, and may even cause the robot to fail in operating the target object.
[0004] To address these deficiencies, a “Transporter” method was introduced for 2D data, which reconstructs the target frame from the source frame to merge spatial and temporal information. However, due to the structural differences between 3D point clouds and 2D images, it is not feasible to directly apply the “Transporter” method to 3D point clouds. Therefore, how to achieve a spatiotemporal consistent representation of key points to effectively perform operations on special objects remains an issue to be addressed. Summary of the invention
[0005] On the one hand, the present application provides a key point detection method, including: obtaining an initial point cloud and a target point cloud of a scene; extracting features from the initial point cloud and the target point cloud respectively, and performing voxel grid projection to obtain initial voxel grid features and target voxel grid features; constructing a three-dimensional representation architecture, and obtaining initial key points and target key points based on the initial point cloud and the target point cloud respectively; clearing the features corresponding to the initial key points and the target key points in the initial voxel grid features to form a first voxel grid feature, and transferring the features corresponding to the target key points in the target voxel grid features to the first voxel grid features to form a second voxel grid feature; and reconstructing the shape of the target point cloud based on the second voxel grid features.
[0006] In one embodiment, the method for reconstructing the shape of the target point cloud includes: acquiring query points in three-dimensional space through a sampling strategy and characterizing them as feature vectors of the query points; inputting the initial point cloud features and the feature vectors of the query points into a deep neural network to output a volume field; and extracting the surface of the target point cloud to obtain a geometric representation of the target point cloud.
[0007] In one embodiment, the method of performing voxel grid projection of the extracted features includes: dividing the three-dimensional space into a voxel grid comprising a plurality of voxels; mapping each point in the initial point cloud and the target point cloud to the corresponding voxel; aggregating the features of all points mapped to the voxel to generate a feature representation of the voxel; and projecting the aggregated feature vector to the corresponding position of the voxel grid.
[0008] In one embodiment, the method for obtaining the initial key points and the target key points includes: inputting the initial point cloud and the target point cloud into the three-dimensional representation architecture; using a cross-attention mechanism to calculate the attention weight of each point in the initial point cloud and the target point cloud; weightedly aggregating the features of each point in the initial point cloud and the target point cloud according to the attention weight to obtain aggregated features; using the aggregated features to identify potential key points in the initial point cloud and the target point cloud; and matching the potential key points between the initial point cloud and the target point cloud to obtain the initial key points and the target key points.
[0009] In one embodiment, the deep neural network includes an implicit neural decoder, and the implicit neural decoder maps the feature vector of the query point to the distance of the query point from the target surface to indicate whether the query point belongs to the surface of the target point cloud.
[0010] In one embodiment, the three-dimensional representation architecture is formed by integrating a point feature network, a cross-attention mechanism, and a three-dimensional U-shaped network.
[0011] In one embodiment, in forming the second voxel grid feature, the algorithm based on which the transfer is performed includes: + (o s )=(1-H(o s ))·(1-H(o t ))·φ(o s )+H(o t )·φ(o t ), where φ + (o s ) is the second voxel grid feature, φ(o s ) is the initial voxel grid feature, φ(o t ) is the target voxel grid feature, H(o s ) is the thermal map of the initial key point, H(o t ) is the heat map of the target key points.
[0012] In one embodiment, the method for forming the heat map of the initial key points and the heat map of the target key points includes: generating a corresponding distribution map based on the acquired initial key points and the target key points; learning the variance of the Gaussian distribution of each key point in the initial key point heat map through a neural network to adaptively adjust the Gaussian distribution of each key point in the distribution map; and converting the distribution map into a Gaussian distribution heat map with each key point as the center and a fixed variance.
[0013] In one embodiment, the implicit neural decoder predicts the volume field to satisfy: where Ω is the implicit neural decoder, q e is the encoded feature vector of the query point, is a local feature obtained from the volume of the initial point cloud feature by trilinear interpolation, o s is the initial point cloud, Prob(q|o t ) is the query point q in the target point cloud o t The surface distance of the object, → means if If Prob(q|o t ) also holds true.
[0014] In one embodiment, weighted aggregation of the features of each point in the initial point cloud and the target point cloud according to the attention weight satisfies: f' s =[f s , zs ], f' t =[f t , z t ], where z s is the point feature f of the initial point cloud s The attention weight vector, z t is the point feature f of the target point cloud t The attention weight vector, Yes t The transpose of Yes s The transpose of f' s and f' t are the feature vectors after the fusion of the initial point cloud and the target point cloud, softmax is the activation function, and d is the point feature.
[0015] In one embodiment, the initial point cloud includes a scene point cloud before the hinge object is operated, and the target point cloud includes a scene point cloud after the hinge object is operated.
[0016] On the other hand, the present application provides a target operation method based on key point detection, including: a key point detection method as described in any of the above embodiments; calculating the motion information of the target object based on the initial key points, the target key points and the shape of the reconstructed target point cloud; and controlling the robotic arm to operate the target object according to the motion information.
[0017] In one embodiment, calculating the motion information of the target object includes using a singular value decomposition algorithm to calculate a rotation matrix and a displacement vector between each pair of matching key points, wherein each pair of matching key points is formed by matching the initial key point with the target-shaped key point.
[0018] In one embodiment, the target object includes a hinged object, and the motion information includes at least one of rotation axis information and angle information of the hinged object and a surface normal of a local plane region where the key point is located.
[0019] On the other hand, the present application provides a robot based on key point detection, including: a robotic arm; a visual module, used to obtain an initial point cloud and a target point cloud of a scene; a feature extraction module, used to extract features from the initial point cloud and the target point cloud, and perform voxel grid projection to obtain initial voxel grid features and target voxel grid features; a key point detection module, used to obtain initial key points and target key points; a feature reconstruction module, used to form a second voxel grid feature, and reconstruct the shape of the target point cloud based on the second voxel grid feature; and a control unit, which calculates motion information of the target object based on the initial key points, the target key points and the reconstructed shape of the target point cloud, and controls the robotic arm to operate the target object according to the motion information.
[0020] In one embodiment, the feature reconstruction module includes a clearing and transferring module, and the clearing and transferring module is used to clear the features corresponding to the initial key points and the target key points in the initial voxel grid feature to form a first voxel grid feature, and transfer the features corresponding to the target key points in the target voxel grid feature to the first voxel grid feature to form the second voxel grid feature.
[0021] In one embodiment, the removal and transport module includes a feature transport algorithm, and the feature transport algorithm satisfies: + (o s )=(1-H(o s ))·(1-H(o t ))·φ(o s )+H(o t )·φ(o t ), where φ + (o s ) is the second voxel grid feature, φ(o s ) is the initial voxel grid feature, φ(o t ) is the target voxel grid feature, H(o s ) is the thermal map of the initial key point, H(o t ) is the heat map of the target key points.
[0022] The features and advantages of the embodiments of the present invention will be described in the following description, and in part will become apparent from the description, or will be understood by practicing the present invention. The purposes and other advantages of the embodiments of the present invention can be achieved and obtained by the structures and / or functions specifically pointed out in the description, claims and drawings. BRIEF DESCRIPTION OF THE DRAWINGS
[0023] Other features, purposes and advantages of the present application will become more apparent through the following detailed description of non-limiting embodiments in conjunction with the accompanying drawings. In the accompanying drawings:
[0024] Figure 1 is a flow chart of a key point detection method according to an exemplary embodiment of the present application.
[0025] Figure 2 is a schematic diagram of a key point detection method according to an exemplary embodiment of the present application.
[0026] Figure 3 is a flow chart of a target operation method based on key point detection according to an exemplary embodiment of the present application.
[0027] Figure 4 It is a schematic diagram of a key point guided translation operation of a target object according to an exemplary embodiment of the present application.
[0028] Figure 5 is a schematic diagram of a key point guiding target object rotation operation according to an exemplary embodiment of the present application.
[0029] Figure 6 is a block diagram of a robot based on key point detection according to an exemplary embodiment of the present application.
[0030] Figure 7 It is a schematic diagram of a robot performing simulation operation based on key point detection according to an exemplary embodiment of the present application.
[0031] Figure 8 It is a schematic diagram of a real machine operation of a robot based on key point detection according to an exemplary embodiment of the present application. DETAILED DESCRIPTION
[0032] In order to better understand the present application, a more detailed description will be made of various aspects of the present application with reference to the accompanying drawings. It should be understood that these detailed descriptions are only descriptions of exemplary embodiments of the present application, and are not intended to limit the scope of the present application in any way. Throughout the specification, the same reference numerals refer to the same elements. The expression "and / or" includes any and all combinations of one or more of the associated listed items.
[0033] It should be noted that in this specification, expressions such as "first", "second", etc. are only used to distinguish one feature from another feature, and do not represent any limitation on the features, especially do not represent any order of precedence. In the specification, references to "one embodiment", "embodiment", "example embodiment", "some embodiments", etc. indicate that the described embodiment may include a specific feature, structure or characteristic, but each embodiment may not necessarily include the specific feature, structure or characteristic. In addition, these phrases do not necessarily refer to the same embodiment.
[0034] It should be understood that expressions such as “include,” “including,” “have,” “contain” and / or “comprise” in this specification are open rather than closed expressions, which indicate the existence of stated features, elements and / or parts, but do not exclude the existence of one or more other features, elements, parts and / or combinations thereof.
[0035] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments in this application can be combined with each other. In addition, unless explicitly limited or contradictory to the context, the specific steps included in the method recorded in this application are not necessarily limited to the recorded order, but can be performed in any order or in parallel.
[0036] The features, principles and other aspects of the present application will be described in detail below in conjunction with the accompanying drawings.
[0037] The present application provides a key point detection method that can at least improve or solve the above problems. The present application provides a carrier suitable for 3D data, which adopts 3D voxel representation, cross-attention mechanism, 3D feature transport and implicit reconstruction methods, and applies the above multiple new expressions to 3D target objects, ensuring that the key points are consistent in time series, and achieving the temporal and spatial consistency of the key points, which is conducive to target recognition in dynamic scenes, and can more accurately capture and track the characteristics of the target object that change over time.
[0038] The motion trajectories of various target objects (such as hinge objects) are efficiently represented through sparse key points. The operation trajectories of the target objects can be adaptively generated based on the detected key point matching, and are applicable to both rotational and translational joints.
[0039] By using a hybrid 3D representation architecture that combines the advantages of point-based models and voxel-based models, the geometric and semantic information of objects can be captured more comprehensively. This hybrid representation method improves the accuracy and robustness of key point detection, enabling the model to better understand and process complex 3D data.
[0040] A self-supervised learning framework is adopted, so that the model can be trained without the need for external labeled data, reducing the dependence on large-scale labeled datasets, enabling the model to learn and extract useful feature representations on unlabeled data, improving the flexibility and efficiency of the training process, and being able to operate on a variety of seen and unseen objects.
[0041] In addition, the key point detection method of the present application also adopts a closed-loop control strategy to use the learned key points for precise object manipulation. This strategy adjusts the robot's movements through real-time feedback, improving the accuracy and reliability of the operation, especially when dealing with complex targets such as hinged objects (such as the processing of rotational joints and translational joints). The closed-loop control strategy can ensure that the robot's movements are consistent with the actual motion state of the target object.
[0042] The above advantages enable the key point detection method of this application to effectively deal with hinged objects, non-rigid objects, and dynamic scenes, while achieving a significant improvement in performance in robot vision and manipulation tasks.
[0043] Figure 1 is a flow chart of a key point detection method according to an exemplary embodiment of the present application.
[0044] like Figure 1 As shown, the present application provides a key point detection method 1000 including:
[0045] Step S1100, obtaining an initial point cloud and a target point cloud of a scene;
[0046] Step S1200, extracting features from the initial point cloud and the target point cloud respectively, and performing voxel grid projection to obtain initial voxel grid features and target voxel grid features;
[0047] Step S1300, obtaining initial key points and target key points based on the initial point cloud and the target point cloud respectively;
[0048] Step S1400, clearing features corresponding to the initial key point and the target key point in the initial voxel grid feature to form a first voxel grid feature, and transferring features corresponding to the target key point in the target voxel grid feature to the first voxel grid feature to form a second voxel grid feature;
[0049] Step S1500: reconstructing the shape of the target point cloud based on the second voxel grid feature.
[0050] It should be understood that the steps shown in the key point detection method 1000 based on key point detection are not exclusive, and other steps may be performed before, after, or between any of the steps shown. In addition, some of the steps shown may be performed simultaneously or in different steps. Figure 1 Executed in the order shown. Figure 2 is a schematic diagram of a key point detection method according to an exemplary embodiment of the present application. Figure 1 and Figure 2 The above steps S1100 to S1500 are further described.
[0051] Step S1100, obtaining an initial point cloud and a target point cloud of a scene.
[0052] Combination Figure 1 and Figure 2 As shown, the initial point cloud of the operating environment can be captured by scene scanning, and the scene point cloud after the target object is operated, that is, the target point cloud, can be captured after the target object is operated.
[0053] In some embodiments, a visual module can be used to collect a multi-perspective depth map of an initial scene and a multi-perspective depth map of a target scene after operation of a target object in the scene, and reversely projected into a point cloud of the initial scene and a point cloud of the target scene using camera intrinsic parameters and the depth map of each perspective.
[0054] In some embodiments, the step of obtaining the initial point cloud and the target point cloud further includes converting the multi-view point clouds of the two scenes into the robot coordinate system, thereby obtaining the initial point cloud and the target point cloud, respectively. As an example, the conversion from the scene point cloud to the robot coordinate system is implemented based on the recorded robot end position.
[0055] Exemplarily, the vision module includes any one of a structured light camera, an RGB-D camera, a laser scanning device, a stereo camera, a time-of-flight (ToF) camera, a binocular camera, or a combination of multiple thereof.
[0056] In some embodiments, the target object includes at least one of a hinged object, a non-rigid object, and a dynamically changing object. The hinged object includes an object that uses a hinge mechanism to achieve opening, closing, rotation, translation, or folding functions. A hinge is a mechanical connection component that allows two objects to rotate relative to each other or move linearly around an axis.
[0057] Furthermore, hinged objects include refrigerators (refrigerator doors), microwave ovens, kitchen doors, cabinets, casement windows, sliding windows, drawers, extended leaves of dining tables, folding dining chairs, flip-top kitchen utensils, etc. It should be noted that "hinge" is not limited to the above examples, and any object that needs to realize the opening and closing function by rotation or linear push and pull can contain hinges.
[0058] Furthermore, non-rigid objects include objects that are easily deformed when subjected to external forces, whose shape or volume can change, and such changes are reversible. Furthermore, non-rigid objects include, for example, fruits, vegetables, bread, cheese and other foods, and may also include plastic wrap, garbage bags, rubber frying spoons, rubber gloves, tablecloths and other items.
[0059] Furthermore, a dynamically changing object refers to an object that can form different states at different times. For example, the initial state of a refrigerator is a refrigerator with a closed door, and the target state is a refrigerator with an open door. The initial state and target state of the refrigerator correspond to different times respectively, and the position, opening degree, and opening state of the refrigerator door are different in different states. The refrigerator is a dynamically changing object.
[0060] Step S1200, extracting features from the initial point cloud and the target point cloud respectively, and performing voxel grid projection to obtain initial voxel grid features and target voxel grid features;
[0061] In some embodiments, the three-dimensional space may be divided into a voxel grid including a plurality of voxels (i.e., volume pixels, which represent the smallest unit of data in the three-dimensional space), and each point extracted from the initial point cloud and the target point cloud is mapped to a corresponding voxel. The features of all points mapped to the voxels are then aggregated to generate a feature representation of the voxels, and then the aggregated feature vectors are projected to the corresponding positions of the voxel grid to obtain the initial voxel grid features and the target voxel grid features.
[0062] Exemplarily, a neural network model is used to extract features of the initial point cloud and the target point cloud, and the neural network model is, for example, a PointNet network (point feature network). The PointNet network can directly take the point cloud as input without complex preprocessing or meshing operations in advance, and output the feature vector of each point, thereby greatly simplifying the workflow and improving efficiency. Based on symmetric functions, the network can process any number of points and is insensitive to the order of the points, thereby ensuring the robustness and generalization ability of the model. When PointNet receives point cloud data, a fixed-length feature vector can be generated for each point. The feature vector has a higher dimension, such as 32 dimensions, which is sufficient to capture the complex geometric information and local features in the point cloud.
[0063] In order to further integrate local features to form a comprehensive understanding of the entire scene, the PointNet network also includes a global pooling operation to fuse the feature vectors of all points into a global feature vector. This global feature vector not only contains the overall shape information of the point cloud, but also reflects the structural relationship within the point cloud and provides clues for subsequent classification, segmentation or matching tasks.
[0064] In some embodiments, in order to better adapt to point cloud data of different scales and densities, the PointNet network can be embedded in a more complex architecture, such as multi-scale PointNet or PointNet++. In addition, in order to achieve higher-precision positioning and recognition in three-dimensional space, the scene can be divided into a series of voxel grids, each grid representing a spatial area within a certain range. Exemplarily, the voxel grid size is 64×64×64, that is, the entire scene is divided into 64×64×64 cells, each of which can contain multiple point cloud data points. Through voxel grid processing, not only can the calculation process be accelerated, but also the storage requirements can be effectively managed, making the PointNet network more efficient when processing large-scale point cloud data, which not only simplifies the processing flow of point cloud data, but also improves the accuracy and robustness of three-dimensional space perception.
[0065] Step S1300, obtaining initial key points and target key points based on the initial point cloud and the target point cloud respectively.
[0066] In some embodiments, the initial point cloud and the target point cloud in the scene are input into the three-dimensional representation architecture, and the cross-attention mechanism is used to calculate the attention weight of each point in the initial point cloud and the target point cloud. The cross-attention mechanism dynamically evaluates the importance of each point in the initial point cloud and the target point cloud to the overall feature expression, and then assigns an attention weight to each point. Specifically, by calculating the similarity or correlation between points, the influence of each point in the scene is quantified, and these weights are then used to weighted aggregate the features of each point in the initial point cloud and the target point cloud. The attention-based weighted aggregation strategy not only enhances the feature expression of key points, but also suppresses irrelevant or redundant information, ensuring the accuracy and efficiency of subsequent key point detection.
[0067] Using the above aggregated features, the algorithm will identify potential key points in the initial point cloud and the target point cloud. These key points often correspond to significant features in the scene, such as edges, corners, or texture changes. Once the potential key points are identified, key point matching is performed between the initial point cloud and the target point cloud. This process is used to find corresponding key point pairs in the two sets of point clouds, thereby establishing a spatial association between the starting point clouds. By accurately matching key points, a consistent understanding and positioning of the scene can be achieved even when the viewing angle, lighting or occlusion conditions change.
[0068] In order to further optimize the accuracy and reliability of key point matching, this application introduces a heat map representation based on Gaussian distribution. First, according to the acquired initial key points and target key points, a corresponding distribution map is generated, in which each key point is assigned a Gaussian distribution, and the width (i.e., variance) of the distribution is controlled to reflect the position uncertainty of the key point. The variance of the Gaussian distribution of each key point in the initial key point heat map is learned by a neural network to adaptively adjust the Gaussian distribution of each key point in the distribution map. By reasonably setting the variance parameters, the positioning accuracy and robustness of the key points can be effectively balanced. Subsequently, the distribution map is converted into a Gaussian distribution heat map centered on each key point. This conversion process not only visualizes the spatial distribution of the key points, but also provides more intuitive and detailed information for subsequent point cloud matching, and obtains the heat map of the initial key points and the heat map of the target key points. The heat map can represent the position of the key points in the point cloud, and also contains deep information about the relative importance of the key points, which is conducive to achieving high-precision key point matching.
[0069] By combining the cross-attention mechanism with the Gaussian distribution-based heat map representation, the key point detection and matching scheme effectively improves its ability to understand and parse complex point cloud data, and further improves the positioning and recognition accuracy in three-dimensional space.
[0070] In some implementations, weighted aggregation of the features of each point in the initial point cloud and the target point cloud is performed according to the above attention weights to satisfy:
[0071]
[0072] f' s =[f s , z s ],
[0073]
[0074] f' t =[f t , z t ],
[0075] In the formula, z s is the point feature f of the initial point cloud s The attention weight vector, z t is the point feature f of the target point cloud t The attention weight vector, Yes t The transpose of Yes s The transpose of f' s and f' tis the feature vector after the fusion of the initial point cloud and the target point cloud, softmax is the activation function, and d is the point feature. This weighted aggregation method can extract and transmit key point features, thereby achieving accurate modeling and operation of the target object, and helps to ensure the consistency of features across frames and the understanding of scenes and targets.
[0076] In some embodiments, the three-dimensional representation architecture is a three-dimensional composite architecture formed by the fusion of a point feature network (PointNet network), a cross attention mechanism, and a three-dimensional U-shaped network (3D-UNet network). The initial point cloud and the target point cloud are respectively subjected to feature extraction through their own independent PointNet network encoders, and each point cloud is converted into a 256-dimensional feature vector, which condenses the geometric information of the point and the relative position of the point in the three-dimensional space and its relationship with the surrounding environment. Based on the obtained point cloud features, the attention mechanism is used to calculate the mutual influence between the two point clouds, thereby generating an attention heat map, which not only strengthens the feature expression of the key points in the point cloud, but also further improves the accuracy of key point positioning by aggregating the geometric features of the two point clouds. Based on the attention heat map, points with significant significance (i.e., key points) in the three-dimensional space can be identified. Key points are the main factors that determine the shape, posture or movement of objects.
[0077] The decoder of the PointNet network can be used to upsample the key point features and convert the previously extracted high-dimensional features into dense features. The dimension of the feature is set to 32 dimensions, for example, to retain sufficient information and facilitate subsequent processing. Through upsampling, the features of each key point are refined, providing richer information support for subsequent analysis and matching. Through the joint application of projection and 3D-UNet network, the above-mentioned dense point features are converted into key point saliency volumes, which reflect the distribution of key points in three-dimensional space and the importance of key points relative to the entire scene. The 3D-UNet network has powerful convolution and deconvolution capabilities, and can capture detailed information of key points from different scales and directions, thereby generating more refined and accurate saliency volumes. As an example, the saliency volume can be marginalized along three orthogonal axes to extract 3D key points.
[0078] It should be noted that no matter what kind of target object it is facing, the three-dimensional representation architecture can stably output a fixed number (for example, 6) of key points. This feature greatly simplifies the subsequent processing flow and provides a guarantee for achieving spatiotemporal consistency and predictability.
[0079] Step S1400, clearing the features corresponding to the initial key points and the target key points in the initial voxel grid features to form a first voxel grid feature, and transferring the features corresponding to the target key points in the target voxel grid features to the first voxel grid features to form a second voxel grid feature.
[0080] Voxel grid is a discretized representation of three-dimensional space, which divides the continuous three-dimensional space into a series of cubes of equal volume. Each voxel contains statistical information of the point cloud data in its spatial region, such as the density, average position or color attributes of the points. In the process of key point detection and matching, the initial voxel grid features contain a preliminary description of the entire scene, while the target voxel grid features focus on the local details of the target object. In order to improve the accuracy and efficiency of key point matching, the features corresponding to the identified key points in the initial voxel grid features can be cleared. This operation is conducive to reducing redundant information and avoiding repeated calculation of key point features. At the same time, it can highlight the features of non-critical areas, providing a more comprehensive and balanced perspective for subsequent object recognition and scene understanding.
[0081] As an example, the cleaning process is as follows: first locate the voxel positions corresponding to the initial key points and the target key points, and then remove the feature representations of these voxels from the initial voxel grid features to form a first voxel grid feature. The first voxel grid feature only retains the information of the non-key point area, which is used to understand and construct the complete scene.
[0082] Afterwards, the algorithm transfers the features corresponding to the target key points in the target voxel grid features to the first voxel grid features. This process includes feature alignment and fusion. Specifically, the feature information of the key points in the target voxel grid can be mapped to the corresponding positions of the first voxel grid. Through the feature transfer operation, a second voxel grid feature is formed. The second voxel grid feature not only contains detailed information of non-critical areas, but also integrates the precise features of the target key points, which is conducive to providing rich and comprehensive data support for subsequent tasks such as object recognition, scene understanding and 3D reconstruction.
[0083] It can be understood that the clearing and feature transfer operations described in this application improve the efficiency and accuracy of key point detection and matching by refining the voxel grid features; by removing redundant information and highlighting key point features, it is possible to focus more on key areas in the scene, thereby achieving higher processing speeds and better recognition effects with limited computing resources.
[0084] In some embodiments, in forming the second voxel grid feature, the algorithm based on which the feature is transferred includes:
[0085] φ + (o s )=(1-H(o s ))·(1-H(o t ))·φ(o s )+H(o t )·φ(o t ),
[0086] In the formula, φ + (o s ) is the second voxel grid feature, φ(o s ) is the initial voxel grid feature, φ(o t ) is the target voxel grid feature, H(o s ) is the heat map of the initial key point, H(o t ) is the heat map of the target key points.
[0087] Through the above feature transfer algorithm, the key points in the initial state and the target state are aligned in the feature space, which is helpful for subsequent 3D reconstruction and object manipulation; it "transfers" the information of the target state to the initial state, enhances the feature representation of the initial state, and fuses the information of the two states to provide a more comprehensive perspective for the dynamic understanding of the object. For the feature areas in the initial state that are suppressed by the key point heat map, by adding the features of the target state, the information of such areas can be enhanced and the expressive power of the features can be improved. In addition, in the absence of manual annotation, self-supervised learning and training through feature transfer can reduce the dependence on labeled data. Feature transfer guided by heat maps can selectively update features, avoid comprehensive updates of the entire voxel grid, improve computational efficiency, and more accurately reconstruct the surface of the object in the target state.
[0088] In a closed-loop control system, feature transfer provides an effective mechanism to adjust the robot's operation so that it can reach the target state more accurately. The learned feature transfer strategy can be generalized to new or unseen objects, improving the applicability and flexibility of the method. This feature transfer is the key in the 3D implicit conveyor, which can effectively improve the model's ability to understand the dynamic changes of objects, which is conducive to the realization of subsequent tasks such as robot operation.
[0089] Step S1500: reconstructing the shape of the target point cloud based on the second voxel grid feature.
[0090] In some embodiments, the method for reconstructing the shape of the target point cloud includes: obtaining query points in three-dimensional space through a sampling strategy and characterizing them into feature vectors of the query points; inputting the initial point cloud features and the feature vectors of the query points into a deep neural network to output a volume field; and extracting the surface of the target point cloud to obtain a geometric representation of the target point cloud. This process samples points in three-dimensional space, obtains and fuses the information of these points in the second voxel grid features, and then uses position encoding to enhance the geometric properties of the points, which are finally input into an implicit neural decoder to reconstruct the surface of the target object with high precision.
[0091] Exemplarily, the deep neural network includes an implicit neural decoder that maps a feature vector of a query point to a distance of the point from a target surface to indicate whether the query point belongs to the surface of the target point cloud.
[0092] For example, sampling points in 3D space, a series of points are randomly or regularly selected in the three-dimensional space where the target object is located, denoted as Q = {x1, x2, ..., x n}, these points form the basis for subsequent reconstruction. Get the eigenvalues of these sampling points in the second voxel grid feature. The second voxel grid feature is formed after the key point feature is cleared and moved. It contains detailed information of non-key areas and integrates the precise features of the target key points. By querying the second voxel grid, each sampling point can obtain the eigenvector corresponding to its position.
[0093] The coordinates of the sampled points can be encoded as position information and merged with the voxel grid eigenvalues corresponding to these points. Position encoding is used to enhance the sensitivity of the neural network to the absolute position of the points in the input point cloud data. In three-dimensional space, position encoding can be achieved in many ways, for example, by directly splicing the three-dimensional coordinates of the point into its eigenvector, or by using more complex encoding schemes such as Fourier feature encoding to increase the representation power of the point cloud data. Through this operation, each sampled point not only carries its feature information in the second voxel grid, but also contains a clear indication of its own position, which provides a richer and more accurate input for the subsequent reconstruction process.
[0094] The merged feature vector is input into the implicit neural decoder to reconstruct the surface of the target object. As a model based on deep learning, the implicit neural decoder has an advantage in reconstructing the surface of complex objects by directly processing sparse and irregularly distributed point cloud data. It can learn the implicit representation of the object surface from the point cloud data, that is, a function that can predict whether the input three-dimensional coordinate is inside or outside the object surface, approximating the true shape of the object surface.
[0095] By sampling points in three-dimensional space, acquiring and fusing the second voxel grid features, and combining position encoding with implicit neural decoder, the reconstruction of three-dimensional space and the restoration of object surface are collaboratively achieved.
[0096] In some embodiments, the sampling strategy includes a random sampling strategy, and the number of sampling points is, for example, between 3000 and 10000. The selection of sampling points depends on the actual application requirements. For example, in order to improve the accuracy of reconstructed details, dense sampling near the surface of the object may be given priority, while the sampling density may be appropriately reduced in open areas.
[0097] As an example, the position information can be encoded by a perceptron consisting of 3 fully connected layers, whose dimensions are, for example, (3, 32), (32, 64) and (64, 128), respectively. The implicit neural decoder is represented by a multi-scale hash grid, for example. The output of the implicit neural decoder is a 3D occupancy field, which indicates whether each voxel grid in the 3D space is occupied by the object surface (1) or not occupied (0). The 3D occupancy field is converted into a reconstructed surface of the object in the target state through the Marching Cube algorithm. As an example, in the reconstruction process, the loss function is to minimize the surface of the reconstructed target point cloud and the surface of the collected target point cloud. The reconstruction can use a binary cross entropy function.
[0098] In some embodiments, the deep neural network learns complex patterns and associations from the input point cloud features and query point feature vectors, and then generates a volume field, which is a dense function representation of a three-dimensional space that can reflect the geometry and internal structure of the target object, and can assign an occupancy probability to each point in the three-dimensional space, that is, the possibility that the point belongs to the surface of the target object. Each point in the volume field corresponds to an occupancy probability. The level of occupancy probability reflects the proximity of the point to the surface of the object and its positional relationship inside or outside the object. This probability value provides the basis for subsequent surface extraction. Through dense function representation, not only can the geometry of the target object be accurately reflected, but also its internal structure, such as the position of cavities, channels or internal objects, can be revealed.
[0099] In some embodiments, the deep neural network can learn the spatial relationship between points and the potential geometric structure in the point cloud from the input point cloud features through training. The query point feature vector (i.e., the feature representation of points randomly or regularly selected in three-dimensional space) is also input into the network, combined with the point cloud features, and together constitutes the learning object of the network, thereby converting the point cloud data and the query point feature vector into a form that the deep neural network can understand. The deep neural network learns complex patterns and associations from the input point cloud features and query point feature vectors. Using multiple layers of feature extraction and abstraction, each layer can capture feature information at different levels, from low-level texture features to high-level shape features. Through this step, the network can not only understand the local details of the point cloud data, but also grasp the global structure of the object, providing input for the volume field.
[0100] Exemplarily, the generation of volume fields also includes iterative training and optimization of deep neural networks and fine-tuning of post-processing steps, such as data enhancement, loss function design, and the addition of regularization terms, so as to ensure that the network can learn generalized patterns and associations from limited training data and avoid overfitting.
[0101] As an example, the implicit neural decoder’s prediction of the volume field satisfies:
[0102]
[0103] Where Ω represents the implicit neural decoder, q e is the feature vector of the encoded query point, is the local feature obtained from the volume of the initial point cloud feature by trilinear interpolation, o s is the initial point cloud, Prob(q|o t ) is the query point q in the target point cloud o t The surface distance of the object, → means if If Prob(q|o t ) also holds true.
[0104] The prediction obtains local features from the volume of the initial point cloud features through trilinear interpolation, and extracts local features near the query point from the volume of the initial point cloud features, taking into account the local environmental information of the query point, thereby enhancing the accuracy of the prediction. The initial point cloud carries the original geometric information of the target object. Through in-depth analysis and processing of the data set, the implicit neural decoder can learn the implicit representation of the surface of the object and predict the function of the probability of occupancy of any query point. The output of this function reflects the probability of the query point occupying the surface of the target point cloud. It not only reveals the intrinsic structure of the point cloud data, but also represents the potential connection between the query point and the surface of the object, which is conducive to generating accurate surface distances of objects.
[0105] On the other hand, the present application provides a target operation method based on key point detection.
[0106] Figure 3 is a flow chart of a target operation method based on key point detection according to an exemplary embodiment of the present application. Figure 3 As shown, the target operation method 2000 includes the key point detection method 1000 described above, and further includes:
[0107] Step S2100, calculating the motion information of the target object based on the initial key points, the target key points and the shape of the reconstructed target point cloud;
[0108] Step S2200, controlling the robotic arm to operate the target object according to the motion information.
[0109] It should be understood that the steps shown in the target operation method 2000 based on key point detection are not exclusive, and other steps may be performed before, after, or between any of the steps shown. In addition, some of the steps shown may be performed simultaneously or in different steps. Figure 3 The following is performed in the order shown. Figures 3 to 5 Further description.
[0110] Step S2100, calculating the motion information of the target object based on the initial key points, the target key points and the shape of the reconstructed target point cloud.
[0111] like Figure 3 As shown in the figure, a 3D space perception framework is constructed based on the initial key points, target key points and the reconstructed target point cloud shape described above. The framework includes the key feature points of the object in the initial state, the position information of the object in the target state, and the target point cloud shape reconstructed by the deep learning model, which together constitute a description of the object's motion state.
[0112] The initial key points and the target key points can be matched to form mutually related key point pairs, and the singular value decomposition algorithm (SVD) is used to calculate the rotation matrix and displacement vector between each pair of matched key points. In the scene of calculating the motion information of the object, the matrix composed of the displacement vectors of the matched key point pairs is used to obtain the overall change trend of the object in the three-dimensional space.
[0113] For example, the displacement vector between each pair of matching key points can be calculated and combined into a matrix. The matrix is then subjected to singular value decomposition to obtain a left singular vector matrix, which corresponds to the rotation matrix of the object, and the displacement vector can be calculated from the translation component of the matrix, thereby decomposing the complex motion of the object in three-dimensional space into two basic components, rotation and translation, and providing an effective mathematical representation for subsequent motion control.
[0114] In order to ensure the accuracy and stability of the calculation, the results of the singular value decomposition can be processed in some embodiments, for example, the rotation matrix can be represented by quaternions to avoid the singularity problem of the Euler angle; in other embodiments, the displacement vector can be optimized using the least squares method to reduce the impact of noise. In addition, in order to improve the calculation efficiency, the incremental singular value decomposition or fast singular value decomposition algorithm can also be used to process large-scale point cloud data.
[0115] The rotation and translation of the target object in three-dimensional space can be calculated by SVD. As an example, the target object includes a hinged object, which relies on an articulated joint to achieve different states, such as a refrigerator with a hinge, which can realize the opening and closing of its refrigerator door through the hinge. The motion information of the hinged object includes at least one of the rotation axis information, angle information and surface normal of the local plane area where the key point is located. For the hinged object, the rotation axis information and angle information directly determine the opening and closing state and operation direction of the hinged object; the surface normal of the local plane area where the key point is located includes a vector perpendicular to the surface of the object, which reflects the direction of the surface of the object and the information of the curvature of the surface of the object. The rotation axis information of the hinged object can describe the position, direction and stability of the rotation axis, while the angle information describes the angle change and motion range of the object in different states. The rotation axis information and angle information directly determine the opening and closing state and operation direction of the hinged object, and are the basis for the machine to understand the movement of the hinged object. By calculating the rotation axis information and angle information, the dynamic behavior of the hinged object in three-dimensional space can be obtained, providing accurate parameters for subsequent robotic arm control and the operation of the hinged object. The surface normal of the local plane area where the key point is located helps to accurately locate the position of the key point, understand the local structure of the object surface and determine the best contact point of the gripper. At the same time, the gripping force can be adjusted according to the characteristics of the object surface to prevent the object from being damaged.
[0116] Step S2200, controlling the robotic arm to operate the target object according to the motion information.
[0117] Based on the calculated motion information, the robot arm is precisely controlled to operate the target object, including the hinged object. For example, the configuration of each joint of the robot arm is calculated based on the desired end effector position and posture. For example, the axis information and angle information can be converted into specific joint angle instructions, and the robot arm performs the required rotation operation to achieve effective control of the hinged object, whether it is opened, closed or rotated to a specific angle, with high precision and efficiency.
[0118] Exemplarily, the surface normal information of the local plane area where the key point is located can also be converted into an execution instruction of the end effector of the robot arm. The surface normal of the local plane area where the key point is located includes a vector perpendicular to the surface of the object, reflecting the direction of the surface of the object and the curvature information of the surface of the object. Based on the instruction information formed by the surface normal of the local plane area where the key point of the object is located, the end of the robot arm (such as the gripper) can accurately locate the position of the key point, understand the local structure of the surface of the object, and determine the optimal contact point of the gripper, and can adjust the gripping force of the gripper according to the surface characteristics of the object to prevent the target object from being damaged.
[0119] Figure 4 It is a schematic diagram of a key point guided translation operation of a target object according to an exemplary embodiment of the present application. Figure 5Schematic diagram of a key point guided rotation operation of a target object according to an exemplary embodiment of the present application. Since the key points detected in the exemplary embodiment of the present application have spatiotemporal consistency between the initial state and the target state of the object, the rotation operation or translation (i.e., linear movement) operation of the target object can be guided by the key points.
[0120] refer to Figure 5 As shown, the robot arm can be controlled to operate the target object, and the corresponding key points are used as path points of the trajectory of the end of the robot arm. When the rotation angle of the target object (such as a hinged object) is greater than 15°, a key point is interpolated as a path point every 15° starting from the initial angle according to the calculated axis information, and the line connecting the path points is the estimate of the end of the robot arm.
[0121] It should be noted that in order to ensure the safety and robustness of the operation, it is also necessary to consider the physical properties of the object, such as weight, shape and friction coefficient, as well as the constraints of the operating environment, such as the presence of obstacles and space limitations. This application combines sensor technology and real-time feedback control strategies to enable the robotic arm to adapt to changing operating conditions and achieve intelligent, flexible and safe control of the target object.
[0122] On the other hand, the present application provides a robot based on key point detection. Figure 6 is a block diagram of a robot based on key point detection according to an exemplary embodiment of the present application. Figure 6 As shown, the robot 900 includes a robotic arm 100 , a vision module 200 , a feature extraction module 300 , a key point detection module 400 , a feature reconstruction module 500 , and a control unit 600 .
[0123] In some embodiments, the vision module 200 is used to obtain the initial point cloud and the target point cloud of the scene. The vision module includes any one of a structured light camera, an RGB-D camera, a laser scanning device, a stereo camera, a ToF camera, and a binocular camera, or a combination of multiple thereof. The vision module 200 can be installed on the robot arm 100, or can be installed at other locations outside the robot arm 100 body.
[0124] Exemplarily, the initial point cloud includes the scene point cloud before operating the target object, and the target point cloud includes the scene point cloud after operating the target object. As an example, the target object includes at least one of a hinged object, a non-rigid object, and a dynamically changing object. A hinged object may refer to an item that uses a hinge mechanism to achieve opening and closing, rotation, translation, or folding functions. A hinge is a mechanical connection component that allows two objects to rotate relative to each other or move linearly around an axis; a non-rigid object includes an object that is easily deformed when subjected to an external force, and the deformation is reversible; a dynamically changing object refers to an object in which the same object forms different states at different times.
[0125] In some embodiments, the feature extraction module 300 is used to extract features from the initial point cloud and the target point cloud, and perform voxel grid projection on the extracted features to obtain initial voxel grid features and target voxel grid features.
[0126] In some embodiments, the key point detection module 400 is used to obtain the initial key points and the target key points. As an example, the key point detection module 400 includes a three-dimensional representation architecture, and the three-dimensional representation architecture is, for example, a three-dimensional composite architecture formed by the fusion of a PointNet network, a cross attention mechanism, and a 3D-UNet network. The initial point cloud and the target point cloud are respectively subjected to feature extraction by their own independent PointNet network encoders, and each point cloud is converted into, for example, a 256-dimensional feature vector. Based on the obtained point cloud features, the mutual influence between the two point clouds is calculated using the attention mechanism to generate an attention heat map, which not only strengthens the feature expression of the key points in the point cloud, but also further improves the accuracy of key point positioning by aggregating the geometric features of the two point clouds. Based on the attention heat map, key points in three-dimensional space can be identified, and the key points correspond to significant features in the scene, such as edges, corners, or texture changes. The existence of key points can realize the understanding and positioning of scene consistency when the viewing angle, lighting, or occlusion conditions change, and provide accurate geometric references for subsequent tasks such as three-dimensional reconstruction, object recognition, and robot operation.
[0127] The cross-attention mechanism is used in the 3D representation architecture to dynamically evaluate the importance of each point in the point cloud to the overall feature expression, and then assign a corresponding attention weight to each point. By calculating the similarity or correlation between points, the influence of each point in the scene is quantified. The attention-based weighted aggregation strategy enhances the feature expression of key points, suppresses irrelevant or redundant information, and achieves accuracy and efficiency in subsequent key point detection.
[0128] It should be noted that the three-dimensional representation architecture also integrates the 3D-UNet network, which is used to extract more detailed local information from voxel grid features, and perform feature fusion and upsampling to restore high-resolution details of key points. Through this integration, the architecture can not only process point cloud data, but also voxel grid data, realizing multi-level feature representation from points to voxels, further improving the accuracy and robustness of key point detection and positioning.
[0129] In some embodiments, the feature reconstruction module 500 is used to form a second voxel grid feature, and reconstruct the shape of the target point cloud based on the second voxel grid feature. Exemplarily, the feature reconstruction module 500 includes a clearing and transporting module 510, which is used to clear the features corresponding to the initial key points and the target key points in the initial voxel grid feature to form the first voxel grid feature; the clearing and transporting module 510 also transports the features corresponding to the target key points in the target voxel grid feature to the first voxel grid feature to form the second voxel grid feature.
[0130] The clearing and transporting module 510 is used to clear the features corresponding to the key points from the initial voxel grid features, and is also used to transport the key point features in the target voxel grid features to the updated voxel grid. The clearing and transporting module 510 locates the voxel positions corresponding to the initial key points and the target key points, and removes the feature representations of these voxels from the initial voxel grid features, which helps to reduce the repeated calculation of key point features, avoid information redundancy, and highlight the features of non-key areas. The first voxel grid feature only retains the information of the non-key point area; the second voxel grid feature contains both the detailed information of the non-key area and the precise features of the target key points, which can provide data support for subsequent tasks such as object recognition, scene understanding and three-dimensional reconstruction.
[0131] Exemplarily, the clearing and transporting module 510 may clear and transport features through a voxel grid processing algorithm, thereby ensuring the accuracy and robustness of feature clearing and transport. The voxel grid processing algorithm may include an efficient query and update mechanism, such as a hash table or a tree structure, which can achieve rapid access and modification of voxel grid features while maintaining the stability and consistency of the data structure.
[0132] Exemplarily, the voxel grid processing algorithm of the clearing and transporting module 510 includes a feature transporting algorithm, which satisfies:
[0133] φ + (o s )=(1-H(o s ))·(1-H(o t ))·φ(o s )+H(o t )·φ(o t ),
[0134] In the formula, φ + (o s ) is the second voxel grid feature, φ(o s ) is the initial voxel grid feature, φ(o t ) is the target voxel grid feature, H(o s ) is the heat map of the initial key point, H(ot ) is the heat map of the target key points.
[0135] In some embodiments, the feature reconstruction module 500 further includes an implicit neural decoder 520, which can map the feature vector of the query point to a corresponding occupancy probability to indicate whether the query point belongs to the surface of the target point cloud.
[0136] In some embodiments, the control unit 600 calculates the motion information of the target object based on the shape of the initial key points, the target key points, and the reconstructed target point cloud, and controls the robotic arm 100 to operate the target object according to the motion information to complete the opening or closing of the target object. The control unit 600 is connected to the robotic arm 100 in communication. The end of the robotic arm 100 may be connected to an actuator 110, and the actuator 110 is, for example, a gripper, and the robotic arm 100 may drive the movement of the gripper. As an example, the control unit 600 may also be connected to the gripper in communication, and control the movement of the robotic arm 100 and the opening and closing of the gripper fingers respectively by sending instructions.
[0137] For example, the configuration of each joint of the robot arm can be calculated according to the desired end effector position and posture. The control unit 600 converts the motion information of the target object into specific joint angle instructions, so that the robot arm 100 performs the required movement operation, such as approaching, opening, rotating or placing the object. At the same time, the control unit 600 can also adjust the opening and closing force of the gripper according to the size, shape and weight of the object to ensure that the object will not be damaged during the operation, and the object will not slip due to insufficient force.
[0138] In some embodiments, the robot 900 also includes a sensor assembly 700, which includes at least one of a force feedback sensor, an infrared sensor, and a distance sensor. The robot 900 can monitor the status of the robotic arm 100 and the gripper and changes in the operating environment in real time through the control unit 600 and the sensor assembly 700, such as sensing the presence of obstacles, changes in light or wind speed, etc.
[0139] Figure 7 It is a schematic diagram of a robot performing simulation operation based on key point detection according to an exemplary embodiment of the present application. Figure 7 The information of key point detection, feature extraction and shape reconstruction during the change of the hinge object from the initial state to the target state is shown. Figure 8This is a schematic diagram of a real machine operation of a robot based on key point detection according to an exemplary embodiment of the present application. By comparing the simulation results with the actual results, it can be seen that the key points detected by the robot 900 have temporal and spatial consistency at different time points / states (such as the initial state and the target state), which enables the robot 900 to track and understand dynamically changing objects. Due to the use of a self-supervised learning method, the robot 900 can be trained without manually annotated data, reducing the reliance on large-scale annotated data sets.
[0140] refer to Figure 8 , which shows an application example of robot 900 in a complex environment. In this example, robot 900 is deployed in a scene containing multiple dynamic hinged objects. Through real-time feedback of dynamic information, robot 900 can detect and track the key points of these hinged objects in different states in real time, and adjust its operation strategy according to the shape and dynamic changes of the objects. It should be noted that robot 900 based on key point detection can not only perform precise operations on a single object, but also coordinate between multiple objects. Figure 8 In the scenario shown, when the robot 900 detects that the hinged door is open, it will automatically adjust the position and posture of its end effector to facilitate further operations inside the cabinet or microwave. Similarly, when the door handle needs to be rotated, the robot 900 can also respond quickly to ensure stable contact between the end effector and the door handle. In addition, when lighting changes, occlusions, etc. occur, the robot 900 can still maintain stable detection of key points and adjust its operation strategy according to changes in the environment to ensure the completion of the task.
[0141] Figure 8 The real machine operation results demonstrate the generalization ability of Robot 900 on a variety of articulated objects by using the learned key points for closed-loop control. It also proves the effectiveness of this strategy in controlling the end effector of the robot arm in actual robot operation, which helps to achieve precise object manipulation.
[0142] This application provides a carrier suitable for 3D data, adopts 3D voxel representation, cross-attention mechanism, 3D feature transport and implicit reconstruction, etc., and applies the above multiple new expressions to 3D target objects, ensuring that the key points remain consistent in the time series and achieving the spatiotemporal consistency of the key points, which is beneficial to target recognition in dynamic scenes and can more accurately capture and track the characteristics of the target object that change over time.
[0143] It can be understood that the robot 900 has learning and adaptive capabilities, and can continuously optimize control strategies based on historical operating data and environmental changes to improve operating efficiency and accuracy.
[0144] Exemplarily, the robot 900 can be used in many scenarios such as medical care, home, retail, security, education, etc. For example, the robot 900 can be applied to cooking or automated food preparation processes in kitchen scenarios to achieve fast and accurate food placement and sorting.
[0145] The above description is only an implementation method of the present application and an explanation of the technical principles used. Those skilled in the art should understand that the scope of protection involved in the present application is not limited to the technical solution formed by the specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the technical concept. For example, the above features are replaced with the technical features with similar functions disclosed in this application (but not limited to) by each other to form a technical solution.
Claims
1. A key point detection method, characterized in that: The method comprises: Get the initial point cloud and target point cloud of the scene; Extracting features from the initial point cloud and the target point cloud respectively, and performing voxel grid projection to obtain initial voxel grid features and target voxel grid features; Constructing a three-dimensional representation architecture, and obtaining initial key points and target key points based on the initial point cloud and the target point cloud respectively; clearing features corresponding to the initial key point and the target key point in the initial voxel grid feature to form a first voxel grid feature, and transferring features corresponding to the target key point in the target voxel grid feature to the first voxel grid feature to form a second voxel grid feature; and The shape of the target point cloud is reconstructed based on the second voxel grid features.
2. The key point detection method according to claim 1, characterized in that: The method for reconstructing the target point cloud shape includes: Obtain query points in three-dimensional space through sampling strategies and characterize them into feature vectors of query points; Inputting the initial point cloud features and the feature vector of the query point into a deep neural network to output a volume field; and A surface of the target point cloud is extracted to obtain a geometric representation of the target point cloud.
3. The key point detection method according to claim 1, characterized in that: The method for performing the voxel grid projection comprises: dividing the three-dimensional space into a voxel grid including a plurality of voxels; Mapping each point in the initial point cloud and the target point cloud to the corresponding voxel; aggregating features of all points mapped to the voxel to generate a feature representation of the voxel; and The aggregated feature vectors are projected to corresponding positions of the voxel grid.
4. The key point detection method according to claim 1, characterized in that: The method of obtaining the initial key point and the target key point includes: Inputting the initial point cloud and the target point cloud into the three-dimensional representation framework; Using a cross attention mechanism to calculate the attention weight of each point in the initial point cloud and the target point cloud; Performing weighted aggregation on the features of each point in the initial point cloud and the target point cloud according to the attention weight to obtain aggregated features; identifying potential key points in the initial point cloud and the target point cloud using the aggregated features; and Matching of the potential key points is performed between the initial point cloud and the target point cloud to obtain the initial key points and the target key points.
5. The key point detection method according to claim 2, characterized in that: The deep neural network includes an implicit neural decoder, and the implicit neural decoder maps the feature vector of the query point to the distance of the query point from the target surface to indicate whether the query point belongs to the surface of the target point cloud.
6. The key point detection method according to claim 1, characterized in that: The three-dimensional representation architecture is composed of a fusion of a point feature network, a cross-attention mechanism and a three-dimensional U-shaped network.
7. The key point detection method according to claim 1, characterized in that: In forming the second voxel grid feature, the algorithm based on which the transfer is performed includes: φ + (o s )=(1-H(o s ))·(1-H(o t ))·φ(o s )+H(o t )·φ(o t ), Among them, φ + (o s ) is the second voxel grid feature, φ(o s ) is the initial voxel grid feature, φ(o t ) is the target voxel grid feature, H(o s ) is the thermal map of the initial key point, H(o t ) is the heat map of the target key points.
8. The key point detection method according to claim 7, characterized in that: The method of forming the heat map of the initial key point and the heat map of the target key point includes: Generate a corresponding distribution map according to the acquired initial key points and the target key points; Learning the variance of the Gaussian distribution of each key point in the initial key point heat map through a neural network to adaptively adjust the Gaussian distribution of each key point in the distribution map; and The distribution map is converted into a Gaussian distribution heat map with each key point as the center and fixed variance.
9. The key point detection method according to claim 5, characterized in that: The implicit neural decoder predicts the volume field in such a way that: where Ω is the implicit neural decoder, q e is the encoded feature vector of the query point, is a local feature obtained from the volume of the initial point cloud feature by trilinear interpolation, o s is the initial point cloud, Prob(q|o t ) is the query point q in the target point cloud o t The surface distance of the object, → means if If Prob(q|o t ) also holds true.
10. The key point detection method according to claim 4, characterized in that: The features of each point in the initial point cloud and the target point cloud are weightedly aggregated according to the attention weight to satisfy: f s =[f s ,z s ], f’ t =[f t ,z t ], Among them, z s is the point feature f of the initial point cloud s The attention weight vector, z t is the point feature f of the target point cloud t The attention weight vector, Yes t The transpose of Yes s The transpose of f' s and f' t are the feature vectors after the fusion of the initial point cloud and the target point cloud, softmax is the activation function, and d is the point feature.
11. The key point detection method according to claim 1, characterized in that: The initial point cloud includes a scene point cloud before operating the hinge object, and the target point cloud includes a scene point cloud after operating the hinge object.
12. A target operation method based on key point detection, characterized in that: include: The method according to any one of claims 1 to 11; Calculating motion information of a target object based on the initial key points, the target key points and the reconstructed shape of the target point cloud; as well as The robot arm is controlled to operate the target object according to the motion information.
13. The target operation method according to claim 12, characterized in that: Calculating the motion information of the target object includes calculating the homogeneous transformation matrix between each pair of matching key points using a singular value decomposition algorithm. Each pair of matched key points is formed by matching the initial key point with the target-shaped key point.
14. The target operation method according to claim 12, characterized in that: The target object includes a hinge object, and the motion information includes at least one of rotation axis information and angle information of the hinge object and a surface normal of a local plane area where the key point is located.
15. A robot based on key point detection, characterized in that: include: Robotic arm; The visual module is used to obtain the initial point cloud and target point cloud of the scene; A feature extraction module, used to extract features from the initial point cloud and the target point cloud, and perform voxel grid projection to obtain initial voxel grid features and target voxel grid features; Key point detection module, used to obtain initial key points and target key points; A feature reconstruction module, used for forming a second voxel grid feature and reconstructing the shape of the target point cloud based on the second voxel grid feature; as well as A control unit calculates motion information of the target object based on the initial key points, the target key points and the reconstructed shape of the target point cloud, and controls the robotic arm to operate the target object according to the motion information.
16. The robot according to claim 15, characterized in that: The feature reconstruction module includes a clearing and transferring module, and the clearing and transferring module is used to clear the features corresponding to the initial key points and the target key points in the initial voxel grid feature to form a first voxel grid feature, and transfer the features corresponding to the target key points in the target voxel grid feature to the first voxel grid feature to form the second voxel grid feature.
17. The robot according to claim 16, characterized in that: The removal and transport module includes a feature transport algorithm, and the feature transport algorithm satisfies: φ + (o s )=(1-H(o s ))·(1-H(o t ))·φ(o s )+H(o t )·φ(o t ), Among them, φ + (o s ) is the second voxel grid feature, φ(o s ) is the initial voxel grid feature, φ(o t ) is the target voxel grid feature, H(o s ) is the thermal map of the initial key point, H(o t ) is the heat map of the target key points.
Citation Information
Cited By
Robot-based hinged object control method, system and equipment and medium
CN120095827A
