Three-dimensional affordability positioning method and device based on multi-graph invariant features and medium

Abstract

The invention relates to a three-dimensional affordability positioning method and device based on multi-image invariant features, and a medium. The method comprises the following steps: obtaining a plurality of person-object reference images; a multi-image guide invariant feature perception three-dimensional availability basic positioning model is utilized, an availability area of a three-dimensional object is positioned by identifying a common interaction mode in a plurality of person-object reference images, and the model comprises an invariant availability knowledge extraction module and an availability dictionary adaptive fusion module. The invariant accessibility knowledge extraction module extracts aligned accessibility knowledge from a plurality of reference images step by step by utilizing an iterative updating strategy and integrates the accessibility knowledge into an accessibility dictionary, and the accessibility dictionary self-adaptive fusion module fuses the extracted invariant accessibility knowledge with point clouds to obtain a point cloud fusion result; and obtaining a comprehensive point cloud feature representation considering all the affordability candidate items, and outputting an affordability positioning prediction result. Compared with the prior art, more accurate and robust three-dimensional object availability positioning is realized.

Description

Technical Field

[0001] The present invention relates to a three-dimensional object availability basic positioning technology, and in particular to a three-dimensional availability positioning method, device and medium based on multi-image invariant features. Background Art

[0002] 3D Affordance Grounding aims to predict functional regions on 3D objects and lays the foundation for a wide range of applications in robotics.

[0003] Currently, methods for predicting 3D object affordance can be divided into two categories. One category involves the use of reinforcement learning, which requires the agent to interact with objects repeatedly in a simulated environment. Such methods are usually time-consuming and their generalization performance is usually limited by the limited number of simulated scenarios. Another category of methods uses annotated 3D object affordance data to try to learn the mapping between objects and affordances in an end-to-end manner. This approach is more direct and has become increasingly popular in recent years. Specifically, 3D AffordanceNet first constructed a benchmark for basic positioning of 3D point cloud affordance. Subsequently, in order to adapt to the diversity of object affordance and incorporate references to human interactions, many methods have been proposed, including IAGNet and LASO, which either combine a single image or combine text descriptions with point clouds for 3D affordance prediction.

[0004] Despite the progress made in previous studies, existing techniques still have a significant limitation, which is the failure to fully utilize the strong correlations between objects in the same affordance category and the implicit invariant affordance knowledge. Figure 1 As shown in Figure 2, the affordance of an object is often determined by multiple reference images of people and objects. These real-world images vary significantly in appearance but represent the same object and affordance category. Therefore, they share invariant affordance knowledge and have strong internal relationships, providing important insights into the affordance of objects in different contexts. Similarly, predicting the affordance of objects requires invariant knowledge extracted from a large number of human interaction images. However, previous methods have failed to adequately address these limitations, often lacking visual information or focusing solely on mapping 3D regions to a single human interaction image, resulting in poor generalization and performance. Summary of the invention

[0005] The purpose of the present invention is to provide a three-dimensional availability positioning method, device and medium based on multi-image invariant features, which gradually extracts availability knowledge from multiple human-object reference images through a multi-image guided invariant feature-aware three-dimensional availability basic positioning (MIFAG) model, and effectively combines this invariant knowledge with point cloud representation to achieve accurate availability prediction.

[0006] The purpose of the present invention can be achieved by the following technical solutions:

[0007] A three-dimensional affordance localization method based on multi-image invariant features comprises the following steps:

[0008] Acquire multiple person-object reference images;

[0009] A three-dimensional affordance basic positioning model guided by multiple images and perceived by invariant features is used to locate the affordance area of ​​a three-dimensional object by identifying common interaction patterns in multiple human-object reference images. The three-dimensional affordance basic positioning model guided by multiple images and perceived by invariant features includes an invariant affordance knowledge extraction module and an affordance dictionary adaptive fusion module. The invariant affordance knowledge extraction module uses an iterative update strategy to gradually extract aligned affordance knowledge from multiple human-object reference images and integrate it into the affordance dictionary. The affordance dictionary adaptive fusion module fuses the extracted invariant affordance knowledge with the point cloud, and uses the point cloud to query each candidate in the affordance dictionary to obtain a comprehensive point cloud feature representation that takes into account all affordance candidates, and outputs the availability positioning prediction result of the three-dimensional object.

[0010] The invariant availability knowledge extraction module aligns multiple images based on their common availability categories and gradually extracts invariant availability knowledge through a multi-layer network and a dual-branch structure, wherein the dual-branch structure consists of an image feature branch and an availability knowledge branch.

[0011] The affordance knowledge branch performs the following steps:

[0012] Randomly initialize a series of learnable affordance query tokens Q to represent the invariant affordance knowledge learned from the reference images;

[0013] The availability query token is iteratively updated through a multi-layer network using the first multi-head criss-cross attention, with reference image features as keys and values. The formula is as follows:

[0014]

[0015] in, and denote the availability query and reference image features in the (l-1)th layer, MCA denotes the multi-head cross attention mechanism, and W q1 , W k1 , W v1 is the weight matrix of the first multi-head criss-cross attention mechanism, M represents the number of availability queries, C represents the hidden layer dimension of the first multi-head criss-cross attention, D represents the feature dimension, H represents the image height, and W represents the image width;

[0016] In order to exploit the inherent consistency among multiple affordance queries, all affordance queries of the current layer are aligned and aggregated based on the MLP layer to obtain the aggregated affordance query of the lth layer. It is expressed as:

[0017]

[0018] Where n is the number of reference images;

[0019] The aggregated availability query Feedback to the image feature branch through the second multi-head cross attention.

[0020] The image feature branch iteratively updates the reference image features through the multi-head self-attention layer and the second multi-head cross attention, which is expressed as the following formula:

[0021]

[0022]

[0023] in, MSA stands for multi-head self-attention, W stands for the weight matrix of the self-attention mechanism, W q2 , W k2 , W v2 is the weight matrix of the second multi-head cross attention mechanism.

[0024] The availability knowledge branch works in collaboration with the image feature branch, and the learnable availability query tags iteratively interact with the image features to add useful availability information to the image feature branch. Meanwhile, in the availability knowledge branch, under the guidance of the image feature branch, the invariant availability knowledge is gradually aggregated, and the output of the last layer of the availability knowledge branch forms an invariant availability dictionary, which encapsulates the availability knowledge in the reference image.

[0025] The affordance dictionary adaptive fusion module performs the following steps:

[0026] Using adaptive invariant perceptual query dictionary cross attention mechanism to fuse object point cloud features P in and invariant affordance knowledge, outputting fusion features The fused features contain all weighted invariant queries, N represents the number of points in the point cloud, and d represents the hidden layer dimension of the adaptive invariant perceptual query dictionary cross attention;

[0027] For the fusion feature P q The irrelevant affordance query tokens are discarded through the self-weighted attention layer to obtain the final adaptive fusion feature P based on the affordance dictionary. mix :

[0028]

[0029] Among them, SWA stands for self-weighted attention;

[0030] The adaptive fusion feature is combined with the original point cloud feature P in Combined, we get the prediction feature P out ;

[0031] The predicted feature P out Passed to decoder f d To obtain the three-dimensional availability positioning prediction result P pred :

[0032] P pred =f d (P out ),

[0033] in,

[0034] The adaptive invariant perceptual query dictionary cross attention mechanism is specifically:

[0035] Using point cloud feature P in Generate query Q for the third multi-head cross attention P , get the key from the invariant affordance knowledge dictionary Sum The details are as follows:

[0036] Q P =P in W q3 ,

[0037] K ID =[Q1, Q2, ..., Q n ]W k3 ,

[0038] V ID =[Q1, Q2, ..., Q n ]W v3 ,

[0039] in, Q i represents the availability query corresponding to the i-th reference image encapsulated in the invariant affordance dictionary; W q3 , W k3 , W v3 is the weight matrix of the third multi-head cross attention mechanism;

[0040] Apply the cross attention mechanism to calculate the dictionary attention matrix A:

[0041] A=SoftMax(Sim cos (QP , K ID )),

[0042] Among them, Sim cos represents the cosine similarity function, SoftMax represents the normalized exponential function, and the dictionary attention matrix represents the similarity between the point cloud and the invariant query in each image;

[0043] Output fusion feature P based on dictionary attention matrix A q :

[0044] P q =IQDCA(Q P , K ID , V ID )=A·V ID

[0045] Among them, IQDCA stands for Adaptive Invariant-aware Query Dictionary Cross-Attention Mechanism.

[0046] The total loss of the multi-image guided invariant feature-aware 3D affordance based localization model is expressed as:

[0047]

[0048] Among them, λ1, λ2 and λ3 are hyperparameters used to balance the total loss; Represents the actual value y of the availability category and the predicted value of the availability category The cross entropy loss between , where the affordance category prediction value It is calculated based on the image features output by the last layer of the invariant affordance knowledge extraction module; represents the image cosine similarity loss, which is calculated based on the image features extracted by the image feature branch of the invariant affordance knowledge extraction module; Combining the focal loss and dice loss, the 3D affordance localization prediction result P based on the model output pred and the three-dimensional affordance positioning truth value P label Calculated,point-by-point heatmap for supervision on point cloud.

[0049] A three-dimensional availability positioning device based on multi-image invariant features includes a memory, a processor, and a program stored in the memory. When the processor executes the program, the method described above is implemented.

[0050] A storage medium stores a program, which implements the method described above when executed.

[0051] Compared with the prior art, the present invention has the following beneficial effects:

[0052] (1) This invention makes full use of the strong connections between multiple reference images that were ignored by previous technologies, allowing the model to deeply explore the affordance information contained in multiple reference images, achieving more accurate and robust affordance positioning of three-dimensional objects. Moreover, the advancement of this technology can further promote the development of fields and applications such as robotic grasping and embodied intelligence.

[0053] (2) The present invention introduces a new MIFAG framework for three-dimensional object availability-based positioning, which can extract invariant availability knowledge from multiple reference images.

[0054] (3) We propose an invariant affordance knowledge extraction module that progressively extracts invariant affordance knowledge from images while minimizing the interference caused by appearance differences. We then propose an affordance dictionary adaptive fusion module that uses this knowledge to obtain comprehensive point cloud features that take into account all affordance candidates.

[0055] (4) This paper constructs a Multi-Image and Point Affordance (MIPA) benchmark, which can promote the study of affordance understanding in diverse visual data. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 It is a schematic diagram of invariant affordance knowledge;

[0057] Figure 2 It is a schematic diagram of the MIFAG model structure of the present invention;

[0058] Figure 3 The figure is a comparison chart of qualitative experimental results between the method of the present invention and the prior art in an embodiment. DETAILED DESCRIPTION

[0059] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process, but the protection scope of the present invention is not limited to the following embodiments.

[0060] In order to solve the problem that the geometric structure of 3D objects in the prior art is not always consistent with the objects in human-object interaction images, resulting in poor generalization ability, the present invention proposes a method for learning generalizable invariant affordance knowledge from multiple human-object interaction images of the same affordance category. Specifically, the present invention introduces a multi-image guided invariant feature-aware 3D affordance basic localization (MIFAG) model, which locates the affordance area of ​​3D objects by identifying common interaction patterns in multiple human-object interaction images. First, the invariant affordance knowledge extraction module (IAM) uses an iterative update strategy to gradually extract aligned affordance knowledge from multiple images and integrate it into the affordance dictionary. Then, the affordance dictionary adaptive fusion module (ADM) learns comprehensive point cloud representations that take into account all affordance candidate regions in multiple images. In addition, this embodiment also constructs a multi-image and point affordance (MIPA) benchmark test to prove that the method of the present invention is superior to the existing state-of-the-art methods in various experimental comparisons.

[0061] Specifically, this embodiment provides a three-dimensional availability positioning method based on multi-image invariant features, comprising the following steps:

[0062] S1, acquiring multiple person-object reference images;

[0063] S2, using the Multi-Image Guided Invariant Feature-Aware 3D Affordance Grounded Localization (MIFAG) model, locates the affordance region of a 3D object by identifying common interaction patterns in multiple human-object reference images.

[0064] like Figure 2 As shown in the figure, the multi-image guided invariant feature-aware 3D affordance-based localization (MIFAG) model includes an invariant affordance knowledge extraction module (IAM) and an affordance dictionary adaptive fusion module (ADM). The invariant affordance knowledge extraction module uses an iterative update strategy to gradually extract aligned affordance knowledge from multiple human-object reference images through a multi-layer network. Its dual-branch structure can minimize the interference caused by image appearance differences. The output of the last layer of IAM forms an invariant affordance knowledge dictionary that encapsulates all invariant affordance knowledge in these images. The affordance dictionary adaptive fusion module applies dictionary-based cross attention and self-weighted attention to fuse the extracted invariant affordance knowledge with the point cloud, and uses the point cloud to query each candidate in the affordance dictionary to obtain a comprehensive point cloud feature representation that takes into account all affordance candidates, and outputs the affordance localization prediction result of the 3D object.

[0065] (a) Invariant Affordance Knowledge Extraction Module (IAM)

[0066] Although the appearance of objects in the reference images varies significantly, they all share a common affordance category and provide important insights into how objects are used. To this end, the present invention constructs an invariant affordance knowledge extraction module, aligns multiple images based on their common affordance categories, and gradually extracts invariant affordance knowledge through a multi-layer network and a dual-branch structure.

[0067] like Figure 2 As shown in (a), the dual-branch structure consists of an image feature branch and an affordance knowledge branch.

[0068] The affordance knowledge branch performs the following steps:

[0069] S201, randomly initialize a series of learnable affordance query tags Q to represent the invariant affordance knowledge learned from the reference image.

[0070] S202, the availability query tag is iteratively updated through the multi-layer network using the first multi-head cross attention, with reference to the image features as the key and value, and the formula is as follows:

[0071]

[0072] in, and denote the availability query and reference image features in the (l-1)th layer, MCA denotes the multi-head cross attention mechanism, and W q1 , W k1 , W v1 is the weight matrix of the first multi-head criss-cross attention mechanism, M represents the number of availability queries, C represents the hidden layer dimension of the first multi-head criss-cross attention, D represents the feature dimension, H represents the image height, and W represents the image width.

[0073] S203, in order to utilize the inherent consistency among multiple availability queries, the present invention aligns and aggregates all the availability queries of the current layer based on the MLP layer to obtain the aggregated availability query of the lth layer It is expressed as:

[0074]

[0075] Where n is the number of reference images.

[0076] S204, query the aggregated availability Feedback to the image feature branch through the second multi-head cross attention.

[0077] In order to continuously extract consistent image features, the image feature branch iteratively updates the reference image features through the multi-head self-attention layer and the second multi-head cross attention, which is expressed as the following formula:

[0078]

[0079]

[0080] in, MSA stands for multi-head self-attention, W stands for the weight matrix of the self-attention mechanism, W q2 , W k2 , W v2 is the weight matrix of the second multi-head cross attention mechanism. At each iteration layer, in order to ensure that all images share the same affordance category, this embodiment calculates all image features The cosine similarity loss between them is used as part of the model loss.

[0081] In IAM, the affordance knowledge branch works with the image feature branch. The learnable affordance query tags interact iteratively with the image features to add useful affordance information to the image feature branch. At the same time, in the affordance knowledge branch, under the guidance of the image feature branch, the invariant affordance knowledge is gradually aggregated. This dual-branch structure minimizes the interference of image feature changes on affordance knowledge. The output of the last layer of the affordance knowledge branch forms an invariant affordance dictionary, which encapsulates the affordance knowledge in the reference image.

[0082] (b) Affordance Dictionary Adaptive Fusion Module (ADM)

[0083] In order to utilize the invariant affordance knowledge dictionary obtained in the IAM, this embodiment designs an ADM module, such as Figure 2 (b) shows how to adaptively integrate affordance knowledge into the point cloud representation.

[0084] Unlike existing multi-head self-attention mechanisms, which generate query, key and value tokens from the input features themselves, our method uses the obtained invariant affordance knowledge as an additional dictionary to seamlessly guide affordance learning on point clouds during the training phase. To this end, we propose an adaptive invariant-aware query dictionary cross-attention (IQDCA) mechanism.

[0085] First, use the point cloud feature P in Generate query Q for the third multi-head cross attention P , get the key from the invariant affordance knowledge dictionary Sum The details are as follows:

[0086] Q P =P in W q3 ,

[0087] KID =[Q1, Q2, ..., Q n ]W k3 ,

[0088] V ID =[Q1, Q2, ..., Q n ]W v3 ,

[0089] in, W i represents the availability query corresponding to the i-th reference image encapsulated in the invariant affordance dictionary; W q3 , W k3 , W v3 is the weight matrix of the third multi-head cross attention mechanism.

[0090] Next, the cross-attention mechanism is applied to calculate the dictionary attention matrix A:

[0091] A=SoftMax(Sim cos (Q P , K ID )),

[0092] Among them, Sim cos represents the cosine similarity function, SoftMax represents the normalized exponential function, and the dictionary attention matrix Representing the similarity between the point cloud and the invariant query in each image, this matrix serves as a guide to adaptively optimize the knowledge in the dictionary.

[0093] Output fusion feature P based on dictionary attention matrix A q :

[0094] P q =IQDCA(Q P , K ID , V ID )=A·V ID

[0095] Among them, IQDCA represents the adaptive invariant perceptual query dictionary cross attention mechanism, and the output of IQDCA is It contains all weighted invariant queries, N represents the number of points in the point cloud, and d represents the hidden layer dimension of the adaptive invariant-aware query dictionary cross-attention.

[0096] Then, the fusion feature P q The irrelevant affordance query tokens are discarded through the self-weighted attention layer to obtain the final adaptive fusion feature P based on the affordance dictionary. mix :

[0097]

[0098] Among them, SWA stands for self-weighted attention.

[0099] The adaptive fusion feature is combined with the original point cloud feature P in Combined, we get the prediction feature P out .

[0100] The predicted feature P out Passed to decoder f d To obtain the three-dimensional availability positioning prediction result P pred :

[0101] P pred =f d (P out ),

[0102] in,

[0103] In addition, the outputs F1, F2, ..., F of the last layer of IAM are n Aggregate to calculate the affordance category prediction score The loss function used to build the model.

[0104] In this embodiment, the total loss of the MIFAG model is expressed as:

[0105]

[0106] Among them, λ1, λ2 and λ3 are hyperparameters used to balance the total loss; Represents the actual value y of the availability category and the predicted value of the availability category The cross entropy loss between , where the affordance category prediction value It is calculated based on the image features output by the last layer of the invariant affordance knowledge extraction module; represents the image cosine similarity loss, which is calculated based on the image features extracted by the image feature branch of the invariant affordance knowledge extraction module; Combining the focal loss and dice loss, the 3D affordance localization prediction result P based on the model output pred and the three-dimensional affordance positioning truth value P label Calculated,point-by-point heatmap for supervision on point cloud.

[0107] Referring to the person-object interaction images, although they often differ significantly in appearance, they all reveal the same affordance category and therefore have strong internal relationships and intrinsic consistency. In order to utilize the invariant affordance knowledge between multiple images, the present invention constructs a multi-image and point affordance (MIPA) baseline, which is the first method to combine point clouds with multiple reference images. This embodiment verifies the feasibility and effectiveness of the proposed method through actual experiments on the point affordance (MIPA) baseline proposed in the present invention. In the experimental verification stage, this embodiment selected other image-point cloud cross-modal fusion methods for quantitative and qualitative experiments. In the quantitative experiment part, the experimental results are shown in Table 1. It can be found that the method proposed in the present invention has achieved the best in all evaluation indicators, and the results achieved are better than other related technical methods currently available; in the qualitative experiment, the experimental results are as follows Figure 3 As shown, it can be found that the framework proposed in the present invention achieves better visualization effect than other methods.

[0108] Table 1 Quantitative comparison experimental results

[0109]

[0110] This embodiment also provides a three-dimensional availability positioning device based on multi-image invariant features, including a memory, a processor, and a program stored in the memory, and the processor implements the method described above when executing the program.

[0111] If the above functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., which can store program code.

[0112] The preferred specific embodiments of the present invention are described in detail above. It should be understood that a person skilled in the art can make many modifications and changes based on the concept of the present invention without creative work. Therefore, any technical solution that can be obtained by a person skilled in the art through logical analysis, reasoning, or limited experiments based on the concept of the present invention on the basis of the prior art should be within the scope of protection determined by the claims.

Claims

1. A three-dimensional affordance localization method based on multi-image invariant features, characterized in that: The following steps are involved: Acquire multiple person-object reference images; A three-dimensional affordance basic positioning model guided by multiple images and perceived by invariant features is used to locate the affordance area of ​​a three-dimensional object by identifying common interaction patterns in multiple human-object reference images. The three-dimensional affordance basic positioning model guided by multiple images and perceived by invariant features includes an invariant affordance knowledge extraction module and an affordance dictionary adaptive fusion module. The invariant affordance knowledge extraction module uses an iterative update strategy to gradually extract aligned affordance knowledge from multiple human-object reference images and integrate it into the affordance dictionary. The affordance dictionary adaptive fusion module fuses the extracted invariant affordance knowledge with the point cloud, and uses the point cloud to query each candidate in the affordance dictionary to obtain a comprehensive point cloud feature representation that takes into account all affordance candidates, and outputs the availability positioning prediction result of the three-dimensional object.

2. A three-dimensional availability positioning method based on multi-image invariant features according to claim 1, characterized in that: The invariant availability knowledge extraction module aligns multiple images based on their common availability categories and gradually extracts invariant availability knowledge through a multi-layer network and a dual-branch structure, wherein the dual-branch structure consists of an image feature branch and an availability knowledge branch.

3. A three-dimensional availability positioning method based on multi-image invariant features according to claim 2, characterized in that: The affordance knowledge branch performs the following steps: Randomly initialize a series of learnable affordance query tokens Q to represent the invariant affordance knowledge learned from the reference images; The availability query token is iteratively updated through a multi-layer network using the first multi-head criss-cross attention, with reference image features as keys and values. The formula is as follows: in, and denote the availability query and reference image features in the (l-1)th layer, MCA denotes the multi-head cross attention mechanism, and W q1 , W k1 , W v1 is the weight matrix of the first multi-head criss-cross attention mechanism, M represents the number of availability queries, C represents the hidden layer dimension of the first multi-head criss-cross attention, D represents the feature dimension, H represents the image height, and W represents the image width; In order to exploit the inherent consistency among multiple affordance queries, all affordance queries of the current layer are aligned and aggregated based on the MLP layer to obtain the aggregated affordance query of the lth layer. It is expressed as: Where n is the number of reference images; The aggregated availability query Feedback to the image feature branch through the second multi-head cross attention.

4. The three-dimensional availability positioning method based on multi-image invariant features according to claim 3, characterized in that: The image feature branch iteratively updates the reference image features through the multi-head self-attention layer and the second multi-head cross attention, which is expressed as the following formula: in, MSA stands for multi-head self-attention, W stands for the weight matrix of the self-attention mechanism, W q2 , W k2 , W v2 is the weight matrix of the second multi-head cross attention mechanism.

5. A three-dimensional availability positioning method based on multi-image invariant features according to claim 4, characterized in that: The availability knowledge branch works in collaboration with the image feature branch, and the learnable availability query tags iteratively interact with the image features to add useful availability information to the image feature branch. Meanwhile, in the availability knowledge branch, under the guidance of the image feature branch, the invariant availability knowledge is gradually aggregated, and the output of the last layer of the availability knowledge branch forms an invariant availability dictionary, which encapsulates the availability knowledge in the reference image.

6. The three-dimensional availability positioning method based on multi-image invariant features according to claim 1, characterized in that: The affordance dictionary adaptive fusion module performs the following steps: Using adaptive invariant perceptual query dictionary cross attention mechanism to fuse object point cloud features P in and invariant affordance knowledge, outputting fusion features The fused features contain all weighted invariant queries, N represents the number of points in the point cloud, and d represents the hidden layer dimension of the adaptive invariant perceptual query dictionary cross attention; For the fusion feature P q The irrelevant affordance query tokens are discarded through the self-weighted attention layer to obtain the final adaptive fusion feature P based on the affordance dictionary. mix : Among them, SWA stands for self-weighted attention; The adaptive fusion feature is combined with the original point cloud feature P in Combined, we get the prediction feature P out ; The predicted feature P out Passed to decoder f d To obtain the three-dimensional availability positioning prediction result P pred : P pred =f d (P out ), in, 7. The three-dimensional availability positioning method based on multi-image invariant features according to claim 6, characterized in that: The adaptive invariant perceptual query dictionary cross attention mechanism is specifically: Using point cloud feature P in Generate query Q for the third multi-head cross attention P , get the key from the invariant affordance knowledge dictionary Sum The details are as follows: Q P =P in W q3 , K ID =[Q1,Q2,…,Q n ]W k2 , V ID =[Q1,Q2,…,Q n ]W v3 , in, K ID , Q i represents the availability query corresponding to the i-th reference image encapsulated in the invariant affordance dictionary; W q3 , W k3 , W v3 is the weight matrix of the third multi-head cross attention mechanism; Apply the cross attention mechanism to calculate the dictionary attention matrix A: A=SoftMax(Sim cos (Q P ,K ID )), Among them, Sim cos represents the cosine similarity function, SoftMax represents the normalized exponential function, and the dictionary attention matrix represents the similarity between the point cloud and the invariant query in each image; Output fusion feature P based on dictionary attention matrix A q : P q =IQDCA(Q P ,K ID ,V ID )=A·V ID Among them, IQDCA stands for Adaptive Invariant-aware Query Dictionary Cross-Attention Mechanism.

8. The three-dimensional availability positioning method based on multi-image invariant features according to claim 1, characterized in that: The total loss of the multi-image guided invariant feature-aware 3D affordance based localization model is expressed as: Among them, λ1, λ2 and λ3 are hyperparameters used to balance the total loss; Represents the actual value y of the availability category and the predicted value of the availability category The cross entropy loss between , where the affordance category prediction value It is calculated based on the image features output by the last layer of the invariant affordance knowledge extraction module; represents the image cosine similarity loss, which is calculated based on the image features extracted by the image feature branch of the invariant affordance knowledge extraction module; Combining the focal loss and dice loss, the 3D affordance localization prediction result P based on the model output pred and the three-dimensional affordance positioning truth value P label Calculated,point-by-point heatmap for supervision on point cloud.

9. A three-dimensional affordance positioning device based on multi-image invariant features, comprising a memory, a processor, and a program stored in the memory, characterized in that: When the processor executes the program, the method according to any one of claims 1 to 8 is implemented.

10. A storage medium having a program stored thereon, characterized in that: When the program is executed, the method according to any one of claims 1 to 8 is implemented.

Images