An actionable component-aware approach to discover common interaction properties from large models

Abstract

The present invention relates to the technical field of computer vision, and proposes an operable component perception method for discovering common interactive attributes from a large model, comprising the following steps: obtaining an image-text pairing dataset corresponding to the operable components of objects in a real scene, and encoding to obtain visual-level embedding and text embedding corresponding to the image-text pairing dataset; maximizing the mutual information between the visual-level embedding and the text embedding by contrastive learning training, and obtaining trained visual-level embedding and text embedding; distilling image features corresponding to the trained visual-level embedding to a 3D space; performing instance segmentation on the operable components of objects in the real scene based on the distillation result and the trained text embedding, and obtaining a segmentation mask of the component; obtaining a predicted pose of the component by using a posture prediction network based on the segmentation mask of the component; and using the segmentation mask of the component and the predicted pose of the component to realize the interaction between embodied intelligence and objects in the real scene.

Description

Technical Field

[0001] The present invention relates to the technical field of computer vision, and more particularly, to an operable component perception method for discovering common interactive properties from a large model. Background Art

[0002] Identification of operable parts for different kinds of objects is crucial for manipulating objects, especially for the interaction of embodied intelligence with scenes in real-world scenarios. Current research usually formulates this problem as a robust 3D segmentation problem. In this paradigm, different kinds of objects are treated as different domains, and models are trained to learn domain-invariant representations of parts through adversarial training methods. However, these learned representations often face challenges in dealing with significant visual variations between the same part categories across different objects. In addition, they often fail to capture the subtle visual distinctions that distinguish one part category from another. Another feasible solution is to directly apply existing open-world segmentation methods to segment operable parts, which usually detect all visible object parts in an uncontrolled environment. However, operable part segmentation focuses on specifically identifying object parts that can be interacted with, such as handles, buttons, or switches, while existing methods focus on identifying the functional aspects of objects that support interaction. Therefore, existing methods have a weak perception of the operable parts of objects. Summary of the invention

[0003] In order to overcome the defect of the prior art in that the operable parts of an object are weakly perceived, the present invention provides an operable parts perception method that can accurately perceive the operable parts of an object and discover common interactive properties from a large model.

[0004] In order to solve the above technical problems, the technical solution of the present invention is as follows:

[0005] Using large language models and diffusion models, we obtain image-text pairing datasets corresponding to the operable parts of objects in real scenes.

[0006] Use the image encoder and text encoder to encode the image-text pairing dataset to obtain the visual level embedding and text embedding corresponding to the image-text pairing dataset;

[0007] Using contrastive learning to train the visual level embedding and the text embedding, and when the mutual information between the visual level embedding and the text embedding is maximized, stopping the training to obtain a trained visual level embedding and a trained text embedding;

[0008] Based on knowledge distillation, the image features corresponding to the trained visual level embedding are distilled into 3D space;

[0009] Based on the image features distilled to 3D space and the trained text embedding, the operable parts of objects in real scenes are segmented to obtain the segmentation masks of the parts;

[0010] Based on the segmentation mask of the component, the predicted pose of the component is predicted using the pose prediction network;

[0011] The segmentation masks of the parts and the predicted poses of the parts are used to realize the interaction between embodied intelligence and objects in the real scene.

[0012] The present invention also proposes a system for perceiving operable components with common interactive properties from a large model, which is used to implement the above-mentioned method for perceiving operable components with common interactive properties from a large model. The system includes:

[0013] A data set acquisition module, used to acquire an image-text pairing data set corresponding to operable parts of objects in real scenes by using a large language model and a diffusion model;

[0014] A dataset encoding module is used to encode the image-text pairing dataset using an image encoder and a text encoder to obtain visual level embedding and text embedding corresponding to the image-text pairing dataset;

[0015] A mutual information maximization module, used for training the visual level embedding and the text embedding by contrastive learning, and stopping the training when the mutual information between the visual level embedding and the text embedding is maximized, thereby obtaining the trained visual level embedding and the trained text embedding;

[0016] The knowledge distillation module is used to distill the image features corresponding to the trained visual level embedding into the 3D space based on knowledge distillation;

[0017] The instance segmentation module is used to perform instance segmentation on the operable parts of objects in real scenes based on the image features distilled to 3D space and the trained text embedding to obtain the segmentation mask of the parts;

[0018] A pose prediction module is used to predict the predicted pose of a component based on the segmentation mask of the component using a pose prediction network;

[0019] The interaction module uses the segmentation mask and predicted pose of the components to realize the interaction between embodied intelligence and objects in the real scene.

[0020] The present invention also proposes a computer device, including a memory and a processor, wherein the memory stores computer-readable instructions, wherein when the computer-readable instructions are executed by the processor, the processor executes the steps of the operational component perception method for discovering common interactive properties from a large model proposed by the present invention.

[0021] Compared with the prior art, the technical solution of the present invention has the following beneficial effects:

[0022] The present invention is based on the visual-level embeddings and text embeddings corresponding to the image-text pairing dataset corresponding to the operable parts of objects in real scenes, and uses contrastive learning for training to maximize the mutual information between the visual-level embeddings and the text embeddings, thereby ensuring the consistency and generalizability of the alignment between the 2D visual information and the text description of the operable parts. Based on knowledge distillation, the image features corresponding to the trained visual-level embeddings are distilled to the 3D space, and the common functional knowledge originally obtained at the 2D level is distilled into their 3D point feature equivalents, which enriches the three-dimensional point features and enhances their effective generalization ability. Finally, the image features and text embeddings distilled to the 3D space are used to obtain segmentation masks and predicted poses, thereby achieving accurate perception of the operable parts of the object. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 This is a flow chart of the method for discovering operable component perception of common interactive attributes from a large model proposed in Example 1;

[0024] Figure 2 A schematic diagram of a process for discovering common interactive attributes of operable components from a large model as proposed in Example 1;

[0025] Figure 3 A schematic diagram showing the simulation environment proposed in Example 1;

[0026] Figure 4 A second schematic diagram of the framework of the operable component perception method proposed in Example 1;

[0027] Figure 5 This is a schematic diagram of comparative experimental results proposed in Example 2;

[0028] Figure 6 This is a schematic diagram of the visualization result of the instance segmentation proposed in Example 2;

[0029] Figure 7 This is the overall framework diagram of the operational component perception system proposed in Example 3 for discovering common interactive attributes from a large model. DETAILED DESCRIPTION

[0030] The drawings are for illustrative purposes only and should not be construed as limiting the present patent;

[0031] In order to better illustrate the present embodiment, some parts in the drawings may be omitted, enlarged or reduced, and do not represent the size of the actual product;

[0032] It is understandable to those skilled in the art that some well-known structures and their descriptions may be omitted in the drawings.

[0033] The technical solution of the present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0034] Example 1

[0035] This embodiment proposes a method for discovering operable component perception of common interactive attributes from a large model. Figure 1 A schematic diagram of the process of discovering operational component perception of common interactive attributes from a large model proposed in this embodiment.

[0036] The method for discovering operable component perception of common interactive attributes from a large model proposed in this embodiment includes the following steps:

[0037] S1: Using large language models and diffusion models, we obtain image-text pairing datasets corresponding to the operable parts of objects in real scenes.

[0038] S2: Encode the image-text pairing dataset using the image encoder and the text encoder to obtain the visual level embedding and text embedding corresponding to the image-text pairing dataset;

[0039] S3: using contrastive learning to train the visual level embedding and the text embedding, and when the mutual information between the visual level embedding and the text embedding is maximized, stopping the training to obtain the trained visual level embedding and the trained text embedding;

[0040] S4: Based on knowledge distillation, the image features corresponding to the trained visual level embedding are distilled into 3D space;

[0041] S5: Based on the image features distilled to 3D space and the trained text embedding, the operable parts of objects in real scenes are instance segmented to obtain the segmentation masks of the parts;

[0042] S6: Based on the segmentation mask of the component, the predicted pose of the component is predicted using the pose prediction network;

[0043] S7: The segmentation mask of the parts and the predicted pose of the parts are used to realize the interaction between embodied intelligence and objects in the real scene.

[0044] In the specific implementation process, the visual-level embeddings and text embeddings corresponding to the image-text pairing dataset corresponding to the operable parts of objects in real scenes are trained using contrastive learning to maximize the mutual information between the visual-level embeddings and the text embeddings, ensuring the consistency and generalizability of the 2D visual information and text descriptions of the operable parts. Based on knowledge distillation, the image features corresponding to the trained visual-level embeddings are distilled to the 3D space, and the common functional knowledge initially obtained at the 2D level is distilled into their 3D point feature equivalents, enriching the three-dimensional point features and enhancing their effective generalization ability. Finally, the image features and text embeddings distilled to the 3D space are used to obtain the segmentation mask and predicted pose, thereby achieving accurate perception of the operable parts of the object.

[0045] In an optional embodiment, based on an object in a real scene, when obtaining an image-text pairing dataset corresponding to an operable component of an object in a real scene using a large language model and a diffusion model, obtaining a pth group of image-text pairing data in the image-text pairing dataset includes:

[0046] Collect all the operable component categories of objects in real scenes, and select the pth operable component category l from them p , using l p Generate query sentence T for large language model p ;

[0047] The query statement T p Input a large language model, and the large language model output contains l p Object labels for all objects in

[0048] Among them, T p =Please tell me some categories of objects with{part name},{part name}=l p , Indicates that it contains l p The object label of the ith object, Indicates that it contains l p The ^C p The object label of an object, also means that it contains l p The objects have a total of ^C p indivual;

[0049] Utilize l p and l p Corresponding object label Image generation sentence T for the generative diffusion model p , the image generates the sentence T pInput diffusion model, diffusion model output contains l p The RGB images corresponding to all objects I p ;

[0050] in, 1≤i≤^C p , Indicates that it contains l p The RGB image of the i-th object, Indicates that it contains l p The ^C p RGB image of an object;

[0051] Utilize I p and T p Generate the pth group of image-text pairing data {I p ; T p};

[0052] The expression of the image-text pairing dataset is:

[0053] A={{I1;T1},...,{I p ; T p}...,{I M ; T M}}, where A represents the image-text pairing dataset, M represents the total number of image-text pairing data in the image-text pairing dataset A, and I M Indicates the Mth operable component category l M The RGB images corresponding to all objects, T M Indicates that M The associated image generates a sentence.

[0054] In this optional embodiment, image-text pairing datasets are collected from two large models, a large language model (LLM) and a diffusion model, to query common affordance. The large language model associates each component category with a potential open-world object label, and the diffusion model generates object-component images with common functions. Common affordance refers to shared interaction functions in the same component category across different objects, which can be revealed by analyzing example objects containing these components. These examples usually provide clear indications of how objects and their components interact with each other. For example, doors and microwave ovens usually have "handles" that can open the doors. Doors and microwave ovens are objects, i.e., objects described in this optional embodiment, and "handles" that can open the doors are components, i.e., operable components described in this optional embodiment. Motivated by this observation, we explored broader and more adaptive common functional information present in objects. This exploration aims to grasp the functional aspects of objects.

[0055] In an optional embodiment, the steps of encoding the image-text pairing dataset using an image encoder and a text encoder to obtain visual level embedding and text embedding corresponding to the image-text pairing dataset include:

[0056] The RGB image in the image-text pairing dataset is input into the image encoder, and the image encoder outputs the visual level embedding; the image encoder includes an MLP layer and a pooling layer; the MLP layer is used as a domain adapter φ DA The pre-trained weights are adjusted to the perception of the operational components corresponding to the RGB image. The pooling layer integrates fine-grained image features using the output data of the MLP layer to form a visual level embedding V, where v i Represents the feature dimension data of the image features corresponding to the i-th RGB image, R d represents the d-dimensional real number space;

[0057] The image feature F 2D The image-generated sentence associated with the corresponding image is input into the text encoder, and the text encoder outputs the text embedding t i represents the text encoding corresponding to the sentence generated by the i-th image, R e Represents the real number space of dimension e.

[0058] In this alternative embodiment, we perform common visual queries and encode common visual information through contrastive learning training on image-text pairing datasets; use one MLP layer as the domain adapter φ DA(Domain Adaptor) adjusts the pre-trained weights to the perception of interactive components and uses a pooling layer to integrate fine-grained image features F 2D ∈R H ×W×d , and then the feature will become a visual level embedding in knowledge distillation At the same time, the textual hints associated with the image are encoded by the text encoder, and finally the text embedding is obtained

[0059] As an example, the image encoder E 2D The pre-trained ViT-L encoder (Vision Transformer) of LSeg (Language-driven semantic segmentation); the text encoder (text encoder) E text Pre-trained text converter for CLIP (Contrastive Language-Image Pre-training);

[0060] As an example, text embedding In the equation, e=512, e and The d in are not equal, and a linear layer is needed to map d to e.

[0061] In an optional embodiment, the step of training the visual level embedding and the text embedding using contrastive learning comprises:

[0062] Design a loss function L to maximize the mutual information between visual level embedding and text embedding 2D-text , the loss function L 2D-text The expression is:

[0063]

[0064]

[0065]

[0066] In the formula, v k represents the kth element of the visual level embedding, t k represents the kth element of text embedding, <·,·> represents the similarity between two vectors, τ represents the learnable temperature coefficient, Q k Represents the sampling dictionary of negative samples provided by positive samples, where v k The corresponding image and t kThe samples with common components in the corresponding texts are considered positive samples, and the rest are considered negative samples;

[0067] Based on the loss function L 2D-text , use contrastive learning to train the visual level embedding and text embedding until the loss function L 2D-text When the number of iterations is minimized or reaches a threshold, the training is stopped to obtain the trained visual level embedding and the trained text embedding. When the training is stopped, the mutual information between the visual level embedding and the text embedding is maximized, and the expression of the trained visual level embedding is: v′ k Indicates v when training is stopped k ; The expression of the trained text embedding is t i ′ represents the time when training is stopped i .

[0068] In this optional embodiment, we perform common visibility learning to provide fine-grained visual semantic alignment knowledge for component segmentation across different open-world objects, and maximize the protection of the mutual information of image-text pairs by minimizing the cross-modal contrastive learning loss (InfoNCE Loss). The contrastive loss is calculated as:

[0069]

[0070]

[0071] The common cross-modal visibility learning loss function is a weighted average of the two, which is finally calculated as follows:

[0072]

[0073] As an example, γ is set to 0.5; for Q k We adopt a sampling strategy: as long as there are common parts in the image and text, we consider them as positive samples, and the rest as negative samples. The common visibility learning process gives the two-dimensional image feature F 2D Three significant advantages: 1) Due to the encoder E 2D 1) The features are constructed from a pre-trained open-set segmentation network, so that they are fine-grained and exhibit universal pixel-wise semantic alignment; 2) Thanks to the auxiliary image-text pairs, these features have strong generalization ability in identifying functional interactions between parts and objects; 3) They are well adapted to the field of actionable part perception, where visual perception is dominated by object-centric images in the frontal view.

[0074] In an optional embodiment, based on knowledge distillation, the step of distilling the image features corresponding to the trained visual level embedding into the 3D space includes:

[0075] Based on the objects in the real scene, we obtain the point cloud voxels corresponding to the image features corresponding to the trained visual level embedding, and convert the point cloud voxels into voxel grids. We use 3DSparse Unet to extract the 3D features F in the voxel grids. 3D , where F 3D ∈R N×z , R N×z Represents a two-dimensional tensor, where N represents the number of voxels and z represents F 3D 3D feature dimension;

[0076] Using the coordinate mapping formula, the 3D feature F 3D The 3D coordinate x is mapped to the 2D coordinate x′, and the 3D feature F is obtained. 3D The corresponding 2D feature F 2D′ ; The expression of the coordinate mapping formula is: Where h(·) represents the mapping function that maps x to x′. Represents the augmentation matrix M used to prevent overfitting aug The inverse matrix of

[0077] Get the image features corresponding to the trained visual level embedding and use the semantic adapter φ SA Map the image features to a dimension with a dimension of d to obtain the mapped image features in, R H×W×d Represents a three-dimensional tensor, H represents The height of the corresponding RGB image, W represents The width of the corresponding RGB image;

[0078] Based on knowledge distillation, mapped image features and 2D feature F 2D′ Design the distillation loss function L 2D-3D , iteratively solve the distillation loss function L 2D-3D , until the distillation loss function L 2D-3D When the minimum value is reached or the number of iterations reaches the preset number, the iteration is stopped. When the iteration is stopped, the mapped image features are completed. Operations distilled to 3D space;

[0079] The distillation loss function L 2D-3D The expression is:

[0080]

[0081] Where L(·,·) represents the L2 loss function, represents the 2D feature at the height coordinate i and the width coordinate j, Represents the mapped image features at the height coordinate i and the width coordinate j.

[0082] In this alternative embodiment, we perform common visibility distillation. The goal of our perception network is to predict the 3D segmentation result of the operable part of the object to be manipulated. To achieve this, highly generalizable knowledge about common visibility is extracted from 2D pixel features to 3D voxel features. For a 3D coordinate x∈R 3 , we use h(·) as the mapping function to map x to 2D coordinates; during the training process, the 3D coordinates first need to be multiplied by an augmentation matrix M aug To prevent overfitting; the enhancement matrix is ​​randomly generated, which will cause an offset from the 2D coordinates. Therefore, we need to form an inverse matrix to maintain the alignment of 2D and 3D coordinates, that is We use a semantic adapter φ SA To map the 2D features to the appropriate dimension, denoted as Finally, the common visibility information is distilled from 2D to 3D via a distillation loss.

[0083] In an optional embodiment, based on the image features distilled to the 3D space and the trained text embedding, the operable parts of the objects in the real scene are instance segmented, and the step of obtaining the segmentation mask of the parts includes:

[0084] The image features The result of distillation to 3D space is recorded as Point-by-point calculation The cosine similarity between the feature corresponding to each point of and the trained text embedding will be The text encoding in the trained text embedding with the largest cosine similarity between the features corresponding to each point is regarded as the initial instance segmentation prediction result of the operable part of the object in the real scene corresponding to the point;

[0085] The initial instance segmentation prediction result C p Optimize to obtain an optimized instance segmentation prediction result, wherein the optimization steps include:

[0086] Based on the initial instance segmentation prediction result C p Design instance segmentation loss function L 3D-text ; Iteratively solve the loss function L 3D-text , until the loss function L 3D-textWhen the minimum value is reached or the number of iterations reaches the preset number, the iteration is stopped to obtain the optimized instance segmentation prediction result. The optimized instance segmentation prediction result is C when the iteration is stopped. p ; Group, score and calculate non-maximum suppression on the optimized instance segmentation prediction results in turn to obtain the segmentation mask of the component corresponding to the optimized instance segmentation prediction result;

[0087] The loss function L 3D-text The expression is:

[0088]

[0089] Where, L bce represents the binary cross entropy loss, Y 3D ∈R N is the true label of the operable part of the 3D point of the object in the real scene, Indicates that Each point corresponds to the feature C p Convert to text form.

[0090] In this optional embodiment, we perform interactive component perception and enrich the semantic information of the operable components into 3D voxel features through common visual distillation, making them identifiable in the visual-semantic cross-modal feature space; therefore, our model can predict the 3D part segmentation result C by calculating the cosine similarity between the 3D point-by-point features and the text embeddings of all part labels. p ; Given a 3D point cloud P∈N×3 of an object observation, where N represents the number of points, the intelligent agent first segments the operable parts based on the current observation of the object, and then estimates the pose of the parts for the subsequent object operation. In real-world scenarios, the operable parts on different objects usually have similar geometric meanings and functions. The operable part perception network is trained on visible object classes (such as the "handle" of a "door") and hopes to effectively generalize to invisible objects (such as the "handle" of a "teapot").

[0091] In an optional embodiment, the step of predicting the predicted pose of the component using a posture prediction network based on the segmentation mask of the component includes:

[0092] For each part segmentation mask, from the 3D feature F 3D The mask features corresponding to the segmentation mask are queried in , and the component coordinate space coordinates of the points corresponding to the mask features are normalized point by point using the posture prediction network, and the component coordinate space coordinates are subjected to a 7-dimensional rigid transformation to obtain the initial predicted component pose p npcs ;

[0093] The predicted pose p of the initial component npcsOptimize to obtain the predicted pose of the component, where the optimization steps include:

[0094] Based on the initial predicted pose p of the component npcs Design normalized loss function L NPCS ; Iteratively solve the loss function L NPCS , until the loss function L NPCS When the minimum value is reached or the number of iterations reaches the preset number, the iteration is stopped to obtain the predicted pose of the component. The predicted pose of the component is p when the iteration is stopped. npcs ;

[0095] The loss function L NPCS The expression is:

[0096] L NPCS =L1(p npcs , g npcs )

[0097] In the formula, L1 represents the L1 loss function, g npcs Ground data representing the true pose of the part.

[0098] In this optional embodiment, we perform component pose estimation, and for each predicted component segmentation mask M i , we start from the feature F 3D Query its mask features in Then, the part-oriented pose prediction network (Part Pose Estimation Network, PE) is used to regress the point-by-point normalized part coordinate space (Normalized Part Coordinate Space, NPCS) coordinates; the NPC loss L NPCS The design is similar to GAPartNet; during reasoning, we estimate the 7-dimensional rigid transformation and obtain the pose of the predicted part; in terms of object manipulation, the robot arm obtains the segmentation mask and pose of the target part, executes the corresponding control program, and completes operations such as crawling, pushing / pulling, and rotating.

[0099] In an optional embodiment, before using the segmentation mask of the component and the predicted pose of the component to realize the interaction between the embodied intelligence and the objects in the real scene, a loss function L is designed to optimize the segmentation mask of the component and the predicted pose of the component. total , iteratively solve the loss function L total , until the loss function L total When the minimum value is reached or the number of iterations reaches a preset number, the iteration is stopped, and the optimization results of the segmentation mask of the component and the predicted position and posture of the component are obtained, and the optimization results are used to realize the interaction between the embodied intelligence and the object components of the real scene;

[0100] The loss function L total The expression is:

[0101] L total =L aux +L seg +L NPCS

[0102]

[0103] Where, L match represents the Hungarian matching cost, y i Ground data representing real instance segmentation results of non-operational parts of objects in real scenes, Instance segmentation prediction results representing inoperable parts;

[0104] L aux =λ1L 2D-text +λ2L 2D-3D +λ3L 3D-text

[0105] In the formula, λ1 represents L 2D-text The weight of L 2D-3D The weight of L 3D-text The weight of .

[0106] In this alternative embodiment, our approach can be exploited to query, learn, and distill common information, developing more robust and unique representations for actionable component awareness, Figure 2 This is a first schematic diagram of the framework of the operable component perception method proposed in this embodiment. Figure 2We demonstrate that DisCo can use large language models to generate textual prompts for objects with parts that have common general human-computer interaction functions, generate corresponding 2D images through a large pre-trained diffusion model, and add 2D data as auxiliary data to the model's training framework to achieve the purpose of predicting the operable parts of open-domain objects. Finally, we input the prediction results into the simulation environment for simulation operations. Among them, we query image-text pair datasets from large language models and diffusion models as auxiliary data. This dataset provides comprehensive insights into the visual aspects and interactive functions of object parts, and shows the common visibility of different objects in the open world. Through common visibility learning, consistency is achieved between the 2D visual information and textual descriptions of objects and operable parts. The 3D point features are enriched by common visual learning, enhancing their ability to generalize effectively; subsequently, these common visual information are learned to encode into fine-grained image features by ensuring consistency between the embedding of the generated image and its corresponding object part text (e.g., "door with handle"); this process achieves consistent and generalizable alignment between the 2D visual information and text descriptions of objects and operable parts; then, the acquired common functional knowledge initially learned at the 2D pixel level is distilled into their 3D point feature equivalents; this enriches the 3D point features and enhances their ability to generalize effectively; finally, these fine 3D features are used to predict part segmentation and estimate pose, thereby facilitating robot target manipulation;

[0107] Figure 3 A schematic diagram showing a simulation environment proposed in this embodiment; Figure 3 The image shows the push, pull and rotation operations performed by the robot hand after the segmentation mask and estimated pose are input into the simulation environment. The first column successfully rotates the knob of the toaster to the correct position, the second column pulls open the drawer of the cabinet, and the third column opens the door in an arc.

[0108] The main challenge of actionable part perception lies in its ability to generalize to open-world objects in real-world environments; current methods train perception models on a limited set of source objects, making them susceptible to dataset bias and limiting their generalization capabilities; to overcome this limitation, we propose to leverage the scalability advantages of 2D image perception to enhance 3D actionable object part perception. This enhancement involves querying, learning, and distilling extensive public knowledge from large-scale pre-trained models. Figure 4 This is a second schematic diagram of the framework of the operable component perception method proposed in this embodiment. Figure 4 We present our approach to improve the performance of text-to-image diffusion models in Figure 4In the paper, the Common Affordance Querying module initially collects visual and textual insights about object-part interactions by leveraging a large language model (LLM) and a diffusion model (DM); then, the Common Affordance Learning and Distillation module integrates the queried knowledge into the perception model training to enhance the part perception ability, where for each actionable part i, the output of the perception network includes a segmentation mask M i , a class label l i ∈{1, ..., 9}, and a 7-dimensional rigid body transformation; we use the pre-trained ViT-L model of Lseg as our image encoder E 2D , CLIP’s pre-trained text transformer is used as our text encoder E text , and use Sparse UNet as our 3D backbone network E 3D , extract 3D features from point clouds. To maintain 2D-3D consistency, we use domain adapters and semantic adapters to project 2D and 3D features to the same dimension. Then we use 2D-3D alignment and 3D-text alignment to learn generalization capabilities from the pre-trained 2D model and obtain semantic labels for each voxel; to learn common features from the generated images, we use contrastive loss to align 2D features and text features; next, we use the Grouping and Scoring module to obtain instance segmentation predictions; then we estimate point-by-point poses through the pose estimation network, and operate on parts through the predicted poses and segmentation results.

[0109] Example 2

[0110] This embodiment applies the operability component perception method for discovering common interactive attributes from a large model proposed in Embodiment 1, and proposes the following performance comparison example:

[0111] This embodiment uses the GAPart dataset for performance comparison. The GAPart dataset is divided according to the degrees of freedom of the components. Each component is classified only based on its function and degrees of freedom, regardless of the body to which it is attached. The GAPart dataset divides the operable components into 9 categories based on their appearance and degrees of freedom.

[0112] Experimental results on the GAPart dataset show that compared with other baseline models (PointGroup, SoftGroup, AutoGAPart and GAPartNet), the model based on our proposed method (ours) can better learn the same parts of different objects and has better generalization results.

[0113] Specifically, Figure 5 Schematic diagram of comparative experimental results proposed in this embodiment; Figure 5 As shown, our model (ours) improves by 1.8% on the in-domain test set and 4.8% on the out-of-domain test set.

[0114] Figure 6 This is a schematic diagram of the visualization result of the instance segmentation proposed in this embodiment, as shown in Figure 6 As shown, our model can well predict semantic segmentation masks, instance segmentation masks and prediction boxes of object parts; Figure 6 The first two rows are samples from the out-of-domain test set in PartNet-Mobility, and the last two rows are samples from AKB-48.

[0115] Example 3

[0116] This embodiment proposes an operable component perception system for discovering common interactive attributes from a large model, which is used to implement the operable component perception method for discovering common interactive attributes from a large model proposed in Example 1.

[0117] Figure 7 The overall framework diagram of the operational component perception system proposed in this embodiment for discovering common interactive attributes from a large model.

[0118] The operational component perception system for discovering common interaction properties from a large model includes:

[0119] A data set acquisition module, used to acquire an image-text pairing data set corresponding to operable parts of objects in real scenes by using a large language model and a diffusion model;

[0120] A dataset encoding module is used to encode the image-text pairing dataset using an image encoder and a text encoder to obtain visual level embedding and text embedding corresponding to the image-text pairing dataset;

[0121] A mutual information maximization module, used for training the visual level embedding and the text embedding by contrastive learning, and stopping the training when the mutual information between the visual level embedding and the text embedding is maximized, thereby obtaining the trained visual level embedding and the trained text embedding;

[0122] The knowledge distillation module is used to distill the image features corresponding to the trained visual level embedding into the 3D space based on knowledge distillation;

[0123] The instance segmentation module is used to perform instance segmentation on the operable parts of objects in real scenes based on the image features distilled to 3D space and the trained text embedding to obtain the segmentation mask of the parts;

[0124] A pose prediction module is used to predict the predicted pose of a component based on the segmentation mask of the component using a pose prediction network;

[0125] The interaction module uses the segmentation mask and predicted pose of the components to realize the interaction between embodied intelligence and objects in the real scene.

[0126] This embodiment proposes a computer device, including a memory and a processor, wherein the memory stores computer-readable instructions, wherein when the computer-readable instructions are executed by the processor, the processor executes the steps of the method for perceiving operable components for discovering common interactive attributes from a large model proposed in Embodiment 1.

[0127] It can be understood that the system and computer device of this embodiment are applied to the method of the above-mentioned embodiment 1, and the options in the above-mentioned embodiment 1 are also applicable to this embodiment, so they will not be described repeatedly here.

[0128] The same or similar reference numerals correspond to the same or similar components;

[0129] The terms used in the drawings to describe positional relationships are only used for illustrative purposes and should not be construed as limiting this patent;

[0130] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the embodiments of the present invention. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the embodiments here. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the claims of the present invention.

Claims

1. A method for discovering operable component perception of common interactive attributes from a large model, characterized in that: The following steps are involved: Using large language models and diffusion models, we obtain image-text pairing datasets corresponding to the operable parts of objects in real scenes. Use the image encoder and text encoder to encode the image-text pairing dataset to obtain the visual level embedding and text embedding corresponding to the image-text pairing dataset; Using contrastive learning to train the visual level embedding and the text embedding, and when the mutual information between the visual level embedding and the text embedding is maximized, stopping the training to obtain a trained visual level embedding and a trained text embedding; Based on knowledge distillation, the image features corresponding to the trained visual level embedding are distilled into 3D space; Based on the image features distilled to 3D space and the trained text embedding, the operable parts of objects in real scenes are segmented to obtain the segmentation masks of the parts; Based on the segmentation mask of the component, the predicted pose of the component is predicted using the pose prediction network; The segmentation mask of the parts and the predicted pose of the parts are used to realize the interaction between embodied intelligence and objects in the real scene; The step of training the visual level embedding and text embedding using contrastive learning includes: Design a loss function L to maximize the mutual information between visual level embedding and text embedding 2D-text , the loss function L 2D-text The expression is: In the formula, v k represents the kth element of the visual level embedding, t k represents the kth element of text embedding, <·,·> represents the similarity between two vectors, τ represents the learnable temperature coefficient, Q k Represents the sampling dictionary of negative samples provided by positive samples, where v k The corresponding image and t k The samples with common components in the corresponding texts are considered positive samples, and the rest are considered negative samples; Based on the loss function L 2D-text , use contrastive learning to train the visual level embedding and text embedding until the loss function L 2D-text When the number of iterations is minimized or reaches a threshold, the training is stopped to obtain the trained visual level embedding and the trained text embedding. When the training is stopped, the mutual information between the visual level embedding and the text embedding is maximized, and the expression of the trained visual level embedding is: v ′ k Indicates v when training is stopped k ; The expression of the trained text embedding is t i ′ Indicates when to stop training i ; Based on the segmentation mask of the component, the steps of using the posture prediction network to predict the predicted pose of the component include: For each part segmentation mask, from the 3D feature F 3D The mask features corresponding to the segmentation mask are queried in , and the component coordinate space coordinates of the points corresponding to the mask features are normalized point by point using the posture prediction network, and the component coordinate space coordinates are subjected to a 7-dimensional rigid transformation to obtain the initial predicted component pose p npcs ; The predicted pose p of the initial component npcs Optimize to obtain the predicted pose of the component, where the optimization steps include: Based on the initial predicted pose p of the component npcs Design normalized loss function L NPCS ; Iteratively solve the loss function L NPCS , until the loss function L NPCS When the minimum value is reached or the number of iterations reaches the preset number, the iteration is stopped to obtain the predicted pose of the component. The predicted pose of the component is p when the iteration is stopped. npcs ; The loss function L NPCS The expression is: L NPCS =L1(p npcs ,g npcs ) In the formula, L1 represents the L1 loss function, g npcs Ground data representing the true pose of the part.

2. The method for discovering operable components with common interactive attributes from a large model according to claim 1, characterized in that: Based on the objects in the real scene, when the image-text pairing data set corresponding to the operable parts of the objects in the real scene is obtained by using the large language model and the diffusion model, the pth group of image-text pairing data in the image-text pairing data set is obtained, including: Collect all the operable component categories of objects in real scenes, and select the pth operable component category l from them p , using l p Generate query sentence T for large language model p ; The query statement T p Input a large language model, and the large language model output contains l p Object labels for all objects in Among them, T p =Please tell me some categories of objects with{part name},{partname}=l p , Indicates that it contains l p The object label of the ith object, Indicates that it contains l p The ^C p The object label of an object, also means that it contains l p The objects have a total of ^C p indivual; Utilize l p and l p Corresponding object label Image generation sentence T for the generative diffusion model p , the image generates the sentence T p Input diffusion model, diffusion model output contains l p The RGB images corresponding to all objects I p ; in, T p =a{object name}with a{part name}, Indicates that it contains l p The RGB image of the i-th object, Indicates that it contains l p The ^C p RGB image of an object; Utilize I p and T p Generate the pth group of image-text pairing data {I p ; T p }; The expression of the image-text pairing dataset is: A = {{I1; T1}, ..., {I p ; T p }...,{I M ; T M }}, where A represents the image-text pairing dataset, M represents the total number of image-text pairing data in the image-text pairing dataset A, and I M Indicates the Mth operable component category l M The RGB images corresponding to all objects, T M Indicates that M The associated image generates a sentence.

3. The method for discovering operable component perception of common interactive attributes from a large model according to claim 2, characterized in that: The steps of encoding the image-text pairing dataset using the image encoder and the text encoder to obtain the visual level embedding and text embedding corresponding to the image-text pairing dataset include: The RGB image in the image-text pairing dataset is input into the image encoder, and the image encoder outputs the visual level embedding; the image encoder includes an MLP layer and a pooling layer; the MLP layer is used as a domain adapter φ DA The pre-trained weights are adjusted to the perception of the operational components corresponding to the RGB image. The pooling layer integrates fine-grained image features using the output data of the MLP layer to form a visual level embedding V, where v i Represents the feature dimension data of the image features corresponding to the i-th RGB image, R d represents the d-dimensional real number space; The image feature F 2D The corresponding image-associated image generation sentence is input into the text encoder, and the text encoder outputs the text embedding t i represents the text encoding corresponding to the sentence generated by the i-th image, R e Represents the real number space of dimension e.

4. The method for discovering operable components with common interactive attributes from a large model according to any one of claims 1 to 3, characterized in that: Based on knowledge distillation, the steps of distilling the image features corresponding to the trained visual level embedding into the 3D space include: Based on the objects in the real scene, we obtain the point cloud voxels corresponding to the image features corresponding to the trained visual level embedding, and convert the point cloud voxels into voxel grids. We use 3DSparse Unet to extract the 3D features F in the voxel grids. 3D , where F 3D ∈R N×z , R N×z Represents a two-dimensional tensor, where N represents the number of voxels and z represents F 3D 3D feature dimension; Using the coordinate mapping formula, the 3D feature F 3D The 3D coordinates are mapped to 2D coordinates to obtain the 3D feature F 3D The corresponding 2D feature F 2D′ ; The expression of the coordinate mapping formula is: Where h(·) represents mapping x to x ′ The mapping function of Represents the augmentation matrix M used to prevent overfitting aug The inverse matrix of Get the image features corresponding to the trained visual level embedding and use the semantic adapter φ SA Map the image features to a dimension with a dimension of d to obtain the mapped image features in, R H×W×d Represents a three-dimensional tensor, H represents The height of the corresponding RGB image, W represents The width of the corresponding RGB image; Based on knowledge distillation, mapped image features and 2D feature F 2D′ Design the distillation loss function L 2D-3D , iteratively solve the distillation loss function L 2D-3D , until the distillation loss function L 2D-3D When the minimum value is reached or the number of iterations reaches the preset number, the iteration is stopped. When the iteration is stopped, the mapped image features are completed. Operations distilled to 3D space; The distillation loss function L 2D-3D The expression is: Where L(·,·) represents the L2 loss function, represents the 2D feature at the height coordinate i and the width coordinate j, Represents the mapped image features at the height coordinate i and the width coordinate j.

5. The method for discovering operable component perception of common interactive attributes from a large model according to claim 4, characterized in that: Based on the image features distilled to 3D space and the trained text embedding, the operable parts of objects in real scenes are segmented, and the steps of obtaining the segmentation mask of the parts include: The image features The result of distillation to 3D space is recorded as Point-by-point calculation The cosine similarity between the feature corresponding to each point of and the trained text embedding will be The text encoding in the trained text embedding with the largest cosine similarity between the features corresponding to each point is regarded as the initial instance segmentation prediction result of the operable part of the object in the real scene corresponding to the point; The initial instance segmentation prediction result C p Optimize to obtain an optimized instance segmentation prediction result, wherein the optimization steps include: Based on the initial instance segmentation prediction result C p Design instance segmentation loss function L 3D-text ; Iteratively solve the loss function L 3D-text , until the loss function L 3D-text When the minimum value is reached or the number of iterations reaches the preset number, the iteration is stopped to obtain the optimized instance segmentation prediction result. The optimized instance segmentation prediction result is C when the iteration is stopped. p ; Group, score and calculate non-maximum suppression on the optimized instance segmentation prediction results in turn to obtain the segmentation mask of the component corresponding to the optimized instance segmentation prediction result; The loss function L 3D-text The expression is: Where, L bce represents the binary cross entropy loss, Y 3D ∈R N is the true label of the operable part of the 3D point of the object in the real scene, Indicates that Each point corresponds to the feature C p Convert to text form.

6. The method for discovering operable component perception of common interactive attributes from a large model according to claim 5, characterized in that: Before using the segmentation mask and the predicted pose of the component to realize the interaction between embodied intelligence and objects in the real scene, a loss function L is designed to optimize the segmentation mask and the predicted pose of the component. total , iteratively solve the loss function L total , until the loss function L total When the minimum value is reached or the number of iterations reaches a preset number, the iteration is stopped, and the optimization results of the segmentation mask of the component and the predicted position and posture of the component are obtained, and the optimization results are used to realize the interaction between the embodied intelligence and the object components of the real scene; The loss function L total The expression is: L total =L aux +L seg +L NPCS Where, L match represents the Hungarian matching cost, y i Ground data representing real instance segmentation results of non-operational parts of objects in real scenes, Instance segmentation prediction results representing inoperable parts; L aux =λ1L 2D-text +λ2L 2D-3D +λ3L 3D-text In the formula, λ1 represents L 2D-text The weight of L 2D-3D The weight of L 3D-text The weight of .

7. A system for perceiving operable components for discovering common interactive properties from a large model, used to implement the method for perceiving operable components for discovering common interactive properties from a large model as claimed in any one of claims 1 to 6, characterized in that: include: A data set acquisition module, used to acquire an image-text pairing data set corresponding to operable parts of objects in real scenes by using a large language model and a diffusion model; A dataset encoding module is used to encode the image-text pairing dataset using an image encoder and a text encoder to obtain visual level embedding and text embedding corresponding to the image-text pairing dataset; A mutual information maximization module, used for training the visual level embedding and the text embedding by contrastive learning, and stopping the training when the mutual information between the visual level embedding and the text embedding is maximized, thereby obtaining the trained visual level embedding and the trained text embedding; The knowledge distillation module is used to distill the image features corresponding to the trained visual level embedding into the 3D space based on knowledge distillation; The instance segmentation module is used to perform instance segmentation on the operable parts of objects in real scenes based on the image features distilled to 3D space and the trained text embedding to obtain the segmentation mask of the parts; A pose prediction module is used to predict the predicted pose of a component based on the segmentation mask of the component using a pose prediction network; The interaction module uses the segmentation mask and predicted pose of the components to realize the interaction between embodied intelligence and objects in the real scene.

8. A computer device comprising a memory and a processor, wherein the memory stores computer-readable instructions, characterized in that: When the computer-readable instructions are executed by the processor, the processor performs the steps of the method for discovering operable component perception of common interactive properties from a large model as described in any one of claims 1 to 6.

Images