An object point-level interaction area prediction calibration method based on human contact part relationship field

Abstract

The application discloses a kind of object point level interaction area prediction calibration methods based on human contact part relation field, it is related to computer vision and intelligent interaction field, including: obtaining the three-dimensional point set of target object, initial prediction value and scene information;Part token library is constructed and each token activation weight is generated;The correlation strength of object point and each part is calculated, and point-part relation field is formed;After the relation field is sparsely normalized, the contact perception calibration vector of each point is obtained by weighted aggregation;According to point feature, initial prediction value and calibration vector, generate enhancement, inhibition and residual quantity, calibrate initial prediction value;According to the prediction value after calibration, determine the final interaction area.The application can reduce the deviation and false detection of interaction area prediction by introducing the explicit relationship constraint between human contact part and object point, and can improve the prediction accuracy and interpretability.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and intelligent interaction technology, specifically to a method for predicting and calibrating object point-level interaction regions based on the relationship field of human body contact points. Background Technology

[0002] With the development of technologies such as robot manipulation, augmented reality, and 3D scene understanding, the categories of objects to be identified are constantly increasing, and the operable areas on object surfaces are becoming increasingly complex. When robots or augmented reality interaction devices need to perform specific operations on different objects, only by accurately determining which areas of the object's surface are more suitable for interaction can reasonable interaction decisions be made for different objects and different operation requirements. One of the key technologies for achieving these reasonable interaction decisions is to predict the interactive areas on the target object's surface. By acquiring the object's 3D structure, interaction semantics, and relevant environmental information, the interactivity of each area on the object's surface can be determined.

[0003] Existing methods for predicting object interaction regions (or 3D availability understanding) typically take 3D point clouds as input. They first sample the object surface into several discrete points, then use point cloud feature extraction networks (such as PointNet++, DGCNN, etc.) to obtain the geometric features and neighborhood information of each point, and finally use point-level prediction networks to output the predicted value of each point belonging to the interaction region. The final interaction region is obtained by threshold filtering or clustering.

[0004] In recent years, some methods have attempted to incorporate multimodal information. For example, EgoChoir uses first-person video and head movements to infer interaction regions, while InteractVLM uses a visual language model to achieve availability localization for language guidance. These methods expand the input modalities and application scenarios for interaction region prediction.

[0005] However, existing technologies still have the following shortcomings: First, there is a lack of explicit constraints on the human contact points. Most existing methods treat interaction region prediction as a binary classification problem, without establishing a correspondence between points on the object's surface and specific human contact points. When an object has regions with similar shapes but different interactive functions (such as the handle and the body of a cup), the model is prone to confusing the two, leading to prediction bias or false detections.

[0006] Second, the initial prediction results lack effective calibration. Existing methods directly use the network output as the final result. Once the initial prediction is wrong, subsequent post-processing such as thresholding and smoothing is insufficient to fundamentally correct it, because these processes do not incorporate any prior knowledge related to human interaction.

[0007] Third, the prediction results lack interpretability. Existing methods only output the location of the interaction area, without explaining which part of the human body these areas correspond to (palm, fingers, or wrist), thus limiting the rationality of the interaction decisions.

[0008] To address the aforementioned issues, this invention proposes a point-level interactive region prediction and calibration method based on the human body contact area relationship field. By establishing the correspondence between object points and human body contact areas, the initial prediction values ​​are calibrated. Summary of the Invention

[0009] (a) Technical problems to be solved In view of the above-mentioned shortcomings and deficiencies of the prior art, the present invention provides a method for predicting and calibrating object point-level interactive regions based on the human body contact part relationship field, so as to solve the problems mentioned in the background art.

[0010] (II) Technical Solution To achieve the above objectives, the present invention provides the following technical solution: a method for predicting and calibrating point-level interactive regions of objects based on the relationship field of human body contact points, comprising the following steps: Obtain the set of 3D points of the target object, the initial interaction prediction value of each point, and the current interaction scene information; Construct a token library for human body contact points, which includes K tokens for human body contact points, where K is an integer greater than 1, and each token for human body contact point corresponds to a human body contact point. Based on the current interaction scenario information, generate the contact activation weight corresponding to each human contact part token; For each object point and each human contact part token, the contact association strength is calculated based on the object point characteristics, local geometric information, human contact part token, and the activation weight of the contact part. The point-part level relationship field is formed by all contact association strengths. The point-to-part level relation field is subjected to sparsification and normalization to obtain the processed relation field; Based on the processed relation field, the human body contact point tokens are weighted and aggregated to obtain the contact perception calibration vector corresponding to each object point. Based on the object point features of each object point, the initial interaction prediction value, and the contact perception calibration vector, the enhancement amount, the suppression amount, and the residual calibration amount are generated respectively. The interaction prediction values ​​are calibrated based on the initial interaction prediction values ​​of each object point and the corresponding enhancement, suppression and residual calibration values ​​to obtain the calibrated interaction prediction values. The final object interaction area is determined based on the interaction prediction values ​​after calibration of each object point.

[0011] Furthermore, the initial interaction prediction value is the pre-normalized prediction score output by the point-level prediction model; after calibration, the calibrated interaction prediction score is mapped to the object point interaction probability in the range of 0 to 1 using the Sigmoid function.

[0012] Furthermore, the activation weights of the contact points are generated based on at least one of action semantics, human posture, object category, task description, or external semantic information. Specifically, the aforementioned information is feature-encoded to obtain scene features, which are then input into the activation weight generation module to output K initial weight values. These values ​​are then normalized using the Softmax function to obtain the activation weights corresponding to each human contact point token.

[0013] Furthermore, the calculation method for the contact association strength is as follows: the object point features of the i-th object point are fused with local geometric information to obtain a comprehensive feature; the comprehensive feature is concatenated with the j-th human contact part token and input into the relationship strength calculation module to obtain the initial relationship strength; then, the initial relationship strength is weighted using the activation weight corresponding to the j-th human contact part to obtain the contact association strength r. ij .

[0014] Furthermore, the relationship strength calculation module is implemented using multilayer perceptron, similarity calculation, gating network, or attention calculation methods.

[0015] Furthermore, the sparsification process is as follows: for each object point, retain the M human contact parts with the highest contact correlation strength, where M is a positive integer less than K, and set the relationship strength corresponding to the other human contact parts to zero; the normalization process is as follows: normalize the retained relationship strengths so that the contact correlation strengths corresponding to the same object point remain numerically stable.

[0016] Furthermore, the contact sensing calibration vector c i The calculation method is as follows: ; Among them, c i Let represent the contact sensing calibration vector corresponding to the i-th object point. b represents the strength of the relationship between the i-th object point and the j-th human body contact point after processing. j This represents the token representing the contact point of the j-th person.

[0017] Furthermore, the enhancement, suppression, and residual calibration values ​​are generated by the predictive calibration module; the predictive calibration module includes a calibration feature fusion module, a calibration value generation module, and a calibration fusion module; the calibration feature fusion module is used to fuse object point features, initial interaction prediction values, and contact sensing calibration vectors to obtain calibration fusion features; the calibration value generation module generates enhancement, suppression, and residual calibration values ​​based on the calibration fusion features; the calibration fusion module is used to perform calibration based on the initial interaction prediction values ​​and the enhancement, suppression, and residual calibration values.

[0018] Furthermore, the calibration method of the calibration fusion module is as follows: ; Among them, z i Let z' represent the initial interaction prediction score for the i-th object point. i e represents the calibrated interactive prediction score. i Indicates the enhancement amount, s i Indicates the amount of inhibition, Δ i λ represents the residual calibration value. e λ represents the enhancement coefficient. s This represents the inhibition coefficient.

[0019] Furthermore, based on the processed relation field, the dominant human contact point corresponding to each object point is determined, and the interpretation result of the human contact point corresponding to the object interaction area is output.

[0020] (III) Beneficial Effects This invention provides a method for predicting and calibrating point-level interaction regions of objects based on the relationship field of human body contact points, which has the following beneficial effects: 1. This invention constructs a three-way calibration mechanism of "enhancement-suppression-residual" by enhancing, suppressing and residual calibration, and systematically corrects the initial prediction value. This solves the problem of the lack of effective calibration and difficulty in correction of the initial prediction result in the prior art, and achieves the effect of improving the prediction accuracy of the interactive region. 2. This invention constructs a point-to-part level relationship field between object points and human contact parts, records the contact correlation strength between each object point and each contact part, and strengthens object points that conform to human contact relationships and suppresses false detection points that do not conform to human contact relationships based on this relationship field during the calibration process. This solves the problem that existing technologies are prone to confusion between regions with similar shapes but different functions due to the lack of explicit constraints on human contact parts, and achieves the effect of reducing prediction area offset and false detection. 3. Based on the processed point-to-part level relationship field, this invention determines the dominant human contact part corresponding to each object point and outputs the corresponding contact association strength. This solves the problem that existing technologies can only output the location of the interaction area but cannot provide explanation information about the contact part, thus enhancing the interpretability of the prediction results. Compared with existing technologies, this invention introduces explicit relationship constraints between human contact parts and object points to systematically calibrate the initial prediction values, which can reduce the offset and false detection of the interaction area prediction and improve the prediction accuracy and interpretability. Attached Figure Description

[0021] Figure 1 This is a flowchart of the prediction calibration method of the present invention; Figure 2 This is a flowchart illustrating the data generation process for point-level relational fields in this invention. Figure 3 This is a block diagram of the object point-level interactive prediction value calibration function of the present invention. Detailed Implementation

[0022] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0023] The present invention will now be described in detail through specific embodiments, as follows: This embodiment provides a method for predicting and calibrating point-level interactive regions of objects based on the relationship field of human contact points. For example... Figure 1 As shown, the method includes the following steps.

[0024] 1. Obtain the 3D point set of the target object, the initial interaction prediction value, and the current interaction scene information. The target object consists of several object points, denoted as N, and the i-th object point is denoted as p. i Each object point includes at least one of the following: 3D coordinates, normal vector, local shape features, and neighborhood relationship features.

[0025] The set of 3D points of the target object can be acquired by a depth camera, sampling the surface of an existing 3D model, or from a 3D object dataset. In this embodiment, the surface of the 3D model of the target object is sampled to obtain a set of 3D points containing a preset number of object points; if necessary, the number of object points can be adjusted by voxel downsampling, random sampling, or farthest point sampling. The 3D coordinates of each object point are obtained from the sampling results, and the normal vector and local geometric information are calculated based on the surface information of the 3D model or the neighboring points of the corresponding object point. The neighborhood relationship is obtained by constructing local neighborhood connectivity relationships.

[0026] The current interaction scene information includes at least one of action semantics, human posture, object category, task description, or external semantic information, used to generate activation weights corresponding to tokens for each human contact point. Specifically, action semantic information consists of text or category labels describing the current interaction action; human posture information is extracted from the interaction scene image using human posture estimation methods; object category information is obtained by classifying and recognizing the 3D point cloud or image of the target object; and task description information consists of user input or system-preset natural language commands or operation parameters.

[0027] The set of 3D points of the target object is input into an existing point-level prediction model to obtain the initial interaction prediction value for each object point. The existing point-level prediction model refers to a current object point-level interaction region prediction model, which includes an object point feature extraction module and a point-level prediction module. The object point feature extraction module encodes the 3D coordinates, normal vectors, local shape features, and neighborhood relationship features of each object point of the target object to obtain object point features; the point-level prediction module outputs the initial interaction prediction value corresponding to each object point based on the object point features. In this embodiment, the existing point-level prediction model includes an object point feature extraction module and a point-level prediction module. The object point feature extraction module uses a Dynamic Graph Convolutional Neural Network (DGCNN) to construct a local neighborhood graph for each object point and construct and aggregate edge features based on the feature relationships between the central object point and neighboring object points to obtain the object point features corresponding to each object point; the point-level prediction module outputs the initial interaction prediction score for each object point through a point-by-point prediction layer based on the object point features. In other embodiments, the object point feature extraction module can also be implemented using PointNet, PointNet++, or other networks capable of feature encoding of a 3D point set.

[0028] The initial interaction prediction value for the i-th object point is denoted as z. iThe initial interaction prediction value represents the initial probability that the i-th object point belongs to the interaction region. In this embodiment, the initial interaction prediction value is the unnormalized prediction score output by the existing point-level prediction model, i.e., logit, which represents the initial prediction score that the i-th object point belongs to the current interaction region. This invention calibrates the initial interaction prediction value within the logit value domain, and after calibration, uses the Sigmoid function to map the calibrated interaction prediction score to the object point interaction probability in the range of 0 to 1.

[0029] In this invention, the following data structure is created: Object point data area: Used to store the three-dimensional coordinates, point-level features, and local geometric information of the target object.

[0030] Initial prediction results area: Used to store the initial interactive prediction values ​​of each object point.

[0031] Human Contact Token Library: Used to store tokens corresponding to different human contact points.

[0032] Point-to-part level relationship field: used to record the contact correlation strength between each object point and each human body contact part.

[0033] Calibration Results Area: Used to store the calibrated object point-level interactive prediction values.

[0034] II. Constructing a token treasury for human contact points The human contact site token pool contains K human contact site tokens, where K is an integer greater than 1, and the j-th human contact site token is denoted as b. j Human body contact point tokens correspond to the parts of the human body that can come into contact with or be manipulated by objects.

[0035] Human body contact point tokens are feature vectors of a preset dimension. Each human body contact point token corresponds to a pre-defined human body contact point, representing the contact function and contact attributes of that human body part during the interaction process. The human body contact points are determined according to a preset human body part division method, and their specific number is denoted as K.

[0036] The human body part prototype is the initial feature representation corresponding to the corresponding human body part. It is used to characterize the contact function and contact attributes of the human body part and can be used for the initialization or updating of human body contact part tokens.

[0037] For different interacting users, human body contact point tokens can be initialized or updated based on the user's posture information, combined with action semantic information or task description information, thus reflecting the differences in human body parts among different users and in different interaction scenarios. Subsequently, the activation weight of each contact point represents the degree of participation of each human body contact point in the current interaction scenario.

[0038] III. Generating Activation Weights for Contact Areas Different human body parts participate in the interaction in different scenarios. To avoid all human body contact parts having the same impact on the prediction result, this invention generates activation weights for each human body contact part token based on the current interaction scenario. The activation weight of the j-th human body contact part token is denoted as a. j This is used to indicate the activation level of the token at the human body contact point in the current interaction scenario.

[0039] Activation weights are generated based on at least one of action semantics, human posture, object category, task description, or external semantic information. Specifically, firstly, at least one of the action semantics, human posture, object category, task description, or external semantic information in the current interaction scene is feature-encoded to obtain corresponding scene features; when multiple information is used, the scene features are concatenated or fused. Then, the fused scene features are input into the activation weight generation module, which outputs K initial weight values ​​corresponding one-to-one with K human contact part tokens. The K initial weight values ​​are then normalized using the Softmax function to obtain the activation weight corresponding to each human contact part token. The value of each activation weight ranges from 0 to 1, and their sum is 1; the larger the activation weight, the higher the relative participation of the corresponding human contact part in the current interaction scene. The activation weight generation module can be composed of one or more fully connected networks.

[0040] By setting activation weights, tokens representing human contact points that are relevant to the current interaction scenario are given higher weights in subsequent relation field calculations, thereby reducing the impact of tokens representing human contact points that are less relevant to the current interaction scenario on the calibration results.

[0041] IV. Constructing a point-to-part level relationship field between object points and human body contact points. For each object point p i token b for each part of the human body in contact j Calculate the contact correlation strength r between the two. ij And the point-location level relationship field R is formed by all contact correlation strengths: ; Where N represents the number of object points, K represents the number of tokens at the human contact points, and r is the element in the i-th row and j-th column of R. ij This represents the contact strength between the i-th object point and the j-th human body contact point.

[0042] The contact association strength is calculated based on object point features, local geometric information, human contact site tokens, and contact site activation weights. Specifically, firstly, the object point features and local geometric information of the i-th object point are fused to obtain the comprehensive features of the object point; then, the comprehensive features are concatenated with the j-th human contact site token and input into the relationship strength calculation module to obtain the initial relationship strength between the i-th object point and the j-th human contact site; finally, the initial relationship strength is weighted using the activation weights corresponding to the j-th human contact site to obtain the contact association strength r between the i-th object point and the j-th human contact site. ij The stronger the contact correlation, the higher the probability of contact between the corresponding object point and the contact point on the human body.

[0043] In this embodiment, the calculation process can be implemented using a multilayer perceptron, similarity calculation, gating network, or attention calculation. Specifically, a relationship strength calculation module can be set up, taking object point features, local geometric information, human contact part tokens, and corresponding contact part activation weights as input, and outputting the corresponding contact association strength according to a preset calculation method. When using a multilayer perceptron, the above input information is fused and input into several fully connected layers, outputting a scalar form of contact association strength. When using a similarity calculation method, object point features and human contact part tokens are mapped to a feature space of the same dimension, and the contact association strength is determined based on their similarity. When using a gating network or attention calculation method, the degree of association between object points and human contact parts is adjusted through gating coefficients or attention weights, respectively. All of the above methods can be used to implement the relationship strength calculation module.

[0044] The point-part level relationship field uses object points as one dimension and human body contact part tokens as another dimension to represent the strength of the relationship between each object point and each human body contact part.

[0045] V. Perform sparsification and normalization processing on the point-part level relationship field. To reduce the influence of irrelevant human contact areas on calibration results, this invention performs sparsification and normalization on the relation field R to obtain the processed relation field. .

[0046] The sparsity reduction process involves: for each object point, retaining only a few human contact points with high relationship strength to that object point. Here, "a few" refers to a pre-set number of relationships to retain, denoted as M, where M is a positive integer less than K. Specifically, for each object point, the contact relationship strength between it and all human contact points is arranged in descending order, retaining only the M human contact points with the highest relationship strength, and setting the relationship strength of the remaining human contact points to zero.

[0047] In this embodiment, the retention quantity M is set to 3, meaning that only the three human contact points with the highest relationship strength are retained for each object point. The specific value of M can be determined by comparing different candidate values ​​using validation data.

[0048] In another embodiment of the present invention, sparsity processing is implemented using a preset threshold method. The preset threshold is denoted as τ. Specifically, when the contact association strength is greater than or equal to the preset threshold τ, the corresponding relationship strength is retained; when the contact association strength is less than the preset threshold τ, the corresponding relationship strength is set to zero or reduced. The preset threshold τ can be determined based on the statistical distribution of contact association strength, the number of human contact points, and verification data. When there are no human contact points that meet the preset threshold condition, at least the human contact point with the highest contact association strength is retained.

[0049] In a preferred embodiment, sparsification is performed by sorting the relationship strengths, that is, for each object point, the M human contact parts with the highest contact correlation strength are retained, and the retained contact correlation strengths are normalized.

[0050] Normalization includes normalizing the retained relationship strengths to keep the numerical values ​​of the contact association strengths corresponding to the same object point stable.

[0051] After the above processing, each object point is mainly constrained by the human contact parts that are highly correlated with it.

[0052] VI. Generate a contact sensing calibration vector based on the processed relation field. For the i-th object point, based on the processed relation field We perform weighted aggregation of tokens at the human contact points to obtain the contact sensing calibration vector c corresponding to that point. i : ; Among them, c i Let represent the contact sensing calibration vector corresponding to the i-th object point. b represents the strength of the relationship between the i-th object point and the j-th human body contact point after processing. j This represents the token representing the contact point of the j-th person.

[0053] The contact sensing calibration vector is used to represent the association information between the i-th object point and each contact part of the human body. Each object point generates a contact sensing calibration vector according to its corresponding relation field weight.

[0054] VII. Generation enhancement amount, suppression amount, and residual calibration amount After obtaining the contact sensing calibration vector, this invention generates enhancement, suppression, and residual calibration values ​​based on the object point features, initial predicted values, and the contact sensing calibration vector. The enhancement value corresponding to the i-th object point is denoted as e. i The inhibition amount is denoted as s. i The residual calibration value is denoted as Δ. i .

[0055] Enhancement amount e i Used to improve the predicted value of object points that conform to human contact relationships; suppression amount s i Used to reduce predicted values ​​of object points that do not conform to human contact relationships; residual calibration amount Δ i Used to correct the initial prediction.

[0056] The aforementioned enhancement, suppression, and residual calibration values ​​are generated by the predictive calibration module. The predictive calibration module includes a calibration feature fusion module, a calibration value generation module, and a calibration fusion module.

[0057] Specifically, the object point features, initial interaction prediction score, and contact perception calibration vector of the i-th object point are first input into the calibration feature fusion module. Through splicing or feature fusion processing, the calibration fusion features corresponding to the object point are obtained.

[0058] Then, the calibration fusion features are input into the calibration quantity generation module, and the enhancement quantity e is obtained through the enhancement quantity generation branch, the suppression quantity generation branch, and the residual calibration quantity generation branch, respectively. i Inhibition amount s i and residual calibration amount Δ i .

[0059] Among them, the enhancement and suppression values ​​are non-negatively processed and used to improve the prediction scores of object points that conform to the human body contact point relationship and reduce the prediction scores of object points that do not conform to the human body contact point relationship, respectively; the residual calibration value can be positive or negative and is used to supplement and correct the initial interaction prediction score.

[0060] The calibration feature fusion module, calibration quantity generation module, and calibration fusion module in the predictive calibration module can be implemented using neural networks, gating structures, or multilayer perceptrons. Specifically, the object point features, initial predicted values, and contact-sensing calibration vectors are first concatenated to obtain the calibration fusion features. When implemented using a neural network, the calibration fusion features are input into the neural network. After shared feature extraction, the enhanced quantity, suppressed quantity, and residual calibration quantity are obtained through enhancement, suppression, and residual calibration quantity output branches, respectively. When implemented using a gating structure, corresponding gating coefficients are generated based on the calibration fusion features, and these coefficients are used to adjust the output amplitudes of the enhanced quantity, suppressed quantity, and residual calibration quantity. When implemented using a multilayer perceptron, the calibration fusion features are input into one or more fully connected layers, and the enhanced quantity, suppressed quantity, and residual calibration quantity are output through three output branches. The enhanced and suppressed quantities are nonnegated, and the residual calibration quantity is processed through linear mapping or bounded activation.

[0061] 8. Calibrate the initial interaction prediction values. The calibration fusion module receives the initial interaction prediction score z of the i-th object point. i , Enhancement amount e i Inhibition amount s i and residual calibration amount Δ i The initial interactive prediction score is calibrated according to a preset calibration relationship to obtain the calibrated interactive prediction score.

[0062] Based on the aforementioned enhancement, suppression, and residual calibration values, the initial predicted value z for the i-th object point is... i The calibration is performed to obtain the calibrated predicted value z'. i : ; ; Among them, z i Let z' represent the initial interaction prediction score for the i-th object point. i e represents the calibrated interactive prediction score. i Indicates the enhancement amount, s i Indicates the amount of inhibition, Δ i λ represents the residual calibration value. e λ represents the enhancement coefficient. s p' represents the inhibition coefficient. i This represents the calibrated probability that the i-th object point belongs to the interaction region. The Sigmoid function is used to map the calibrated interaction prediction score to the range of 0 to 1.

[0063] The calibration process involves first weighting the enhancement amount according to a preset enhancement coefficient, and then adding the weighted enhancement amount to the initial predicted value of the i-th object point; next, weighting the suppression amount according to a preset suppression coefficient, and then subtracting the weighted suppression amount from the initial predicted value; finally, the residual calibration amount is added to obtain the calibrated predicted value of the object point. The enhancement amount is used to improve the predicted value of object points that conform to the human body contact point relationship, the suppression amount is used to reduce the predicted value of object points that do not conform to the human body contact point relationship, and the residual calibration amount is used to supplement and correct the aforementioned calibration results.

[0064] This calibration method enhances the predicted values ​​of object points that conform to the relationship with human contact points, and suppresses the predicted values ​​of object points that do not conform to the relationship with human contact points, thereby completing the calibration of the initial prediction results.

[0065] 9. Output the calibrated object interaction area After calibrating all object points, the calibrated probabilities p' of each object point obtained in step eight are used. i The final object interaction area is obtained through preset thresholds, probability sorting, or connected component analysis.

[0066] When using a preset threshold method, object points whose probability reaches the preset threshold are identified as candidate interaction points. When using a probability ranking method, object points are sorted from high to low probability, and a preset number or proportion of object points are retained. When using a connected component analysis method, several connected components are formed based on the 3D coordinates and neighborhood relationships of the candidate interaction points. Isolated points or regions with few points are removed, and connected components with a large number of points or a high average predicted probability are identified as the final object interaction regions. The preset threshold, the number of points to retain, or the proportion of points to retain can be determined based on the verification data and the specific interaction task. Simultaneously, the processed relationship field... Determine the dominant human contact point for each object point. Specifically, for the i-th object point, the contact point with the strongest contact correlation is identified as the dominant human contact point for that object point: Therefore, this invention outputs the interaction area of ​​the target object's surface and the interpretation result of the human contact parts corresponding to this interaction area. The interpretation result of the human contact parts refers to determining the dominant human contact parts corresponding to each object point within the object's interaction area based on the processed point-part level relationship field, and outputting the corresponding human contact part identifiers and contact association strengths. Specifically, for each object point, the human contact part with the highest contact association strength is determined as the dominant human contact part corresponding to that object point. For an interaction area composed of multiple object points, the main associated human contact parts of the interaction area can be further determined based on the contact association strengths corresponding to each object point. The interpretation result is used to explain the correspondence between the predicted object interaction area and each human contact part.

[0067] 10. Model Training Methods During the training phase, the existing point-level prediction model is loaded with pre-trained model parameters as initialization parameters and jointly trained end-to-end with the human contact site token, activation weight generation module, relationship strength calculation module, and prediction calibration module. During training, the parameters of the existing point-level prediction model are not frozen but are updated together with the parameters of the newly added modules through backpropagation.

[0068] The training samples include at least the 3D point data of the target object, the corresponding object point-level interaction region labels, human contact labels, and current interaction scene information. Human contact labels indicate whether each vertex of the human mesh is in contact with the object. Based on a preset mapping relationship between human mesh vertices and human contact points, the human contact labels are aggregated at the part level to obtain the contact score or part participation label corresponding to each human contact point. The object point-level interaction region prediction loss is backpropagated through the calibration fusion module to the enhancement generation branch, suppression generation branch, and residual calibration generation branch to jointly update the model parameters of the three generation branches.

[0069] During training, an object-level interaction region prediction loss is constructed based on the difference between the calibrated object-level prediction results and the actual object-level interaction region labels. A human contact part prediction loss is constructed based on the difference between the predicted human contact part participation level and the part participation labels obtained by aggregating human contact labels.

[0070] Furthermore, a point-to-part level relational field constraint loss is constructed based on the contact distribution of human body contact points and the object point-level interaction region labels. Specifically, the contact scores of each human body contact point are normalized to obtain the target distribution of human body contact points. The supervision weight of each object point in the relational field constraint loss is determined based on the object point-level interaction region labels, and the point-to-part level relational distribution corresponding to each object point within the interaction region is constrained to be consistent with the target distribution of human body contact points.

[0071] The object point-level interaction region prediction loss, human body contact part prediction loss, and point-part level relation field constraint loss are weighted and combined to obtain the total training loss. The parameters of the existing point-level prediction model, human body contact part token, activation weight generation module, relation strength calculation module, and prediction calibration module are jointly updated through backpropagation.

[0072] In one specific implementation, the weight corresponding to the human body contact site prediction loss is 0.1, and the weight corresponding to the point-site level relation field constraint loss is 0.05.

[0073] In this embodiment, the number of tokens K for human contact points is 24, the feature dimension of object points is 768, the dimension of tokens for human contact points is 768, and the retention quantity M is 3. The number of target object points N is a preset number of points obtained after sampling the original object point cloud, and the specific value is determined according to the input scale of the point-level prediction model.

[0074] This embodiment uses the prediction of the cup's grasping interaction area as an example to provide a detailed description of the technical solution of the present invention.

[0075] Taking the prediction of the cup's grasping interaction area as an example, firstly, a 3D point set of the cup is obtained, and the object point features of each object point are extracted using an existing point-level prediction model to obtain the initial interaction prediction value of each object point. A token library for human contact parts is constructed based on the current grasping interaction scenario, and activation weights corresponding to each human contact part token are generated based on action semantics, human posture, object category, or task description information. In the grasping scenario, tokens corresponding to contact parts such as the hand and wrist have a high activation level. The contact association strength between each object point of the cup and each human contact part token is calculated to form a point-part-level relationship field. After sparsification and normalization of the relationship field, the human contact part tokens are weighted and aggregated to obtain the contact perception calibration vector corresponding to each object point. The object point features, initial interaction prediction values, and contact perception calibration vectors are input into the prediction calibration module. The calibration feature fusion module generates calibration fusion features, and the calibration quantity generation module generates enhancement, suppression, and residual calibration quantities, and calibrates the initial interaction prediction value. Specifically, the predicted values ​​for the cup handle area, which has a strong contact relationship with the hand or wrist, are enhanced, while the predicted values ​​for falsely detected areas, which have a weaker relationship with the current grasping action, are suppressed. After calibration of all object points, the final grasping interaction area of ​​the cup is determined based on the calibrated prediction results. Then, based on the processed point-to-part level relationship field, the dominant human contact part and contact association strength corresponding to the interaction area are output.

[0076] The model used in this invention achieves a mean inter-region overlap (aIOU) of up to 0.1846 and a predicted distribution similarity (SIM) of up to 0.4560. In comparison, the aIOU and SIM of the original EgoChoir model are 0.1494 and 0.4360, respectively. The model used in this invention represents a relative improvement of approximately 23.6% in aIOU and approximately 4.6% in SIM; its AUC is 0.7732, which is essentially equivalent to the original model's 0.7802.

[0077] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Any modifications, alterations, substitutions, and variations made by those skilled in the art to the above embodiments are within the scope of the present invention.

Claims

1. A method for predicting and calibrating point-level interactive regions of objects based on the relationship field of human body contact points, characterized in that, Includes the following steps: Obtain the set of 3D points of the target object, the initial interaction prediction value of each point, and the current interaction scene information; Construct a token library for human body contact points, which includes K tokens for human body contact points, where K is an integer greater than 1, and each token for human body contact point corresponds to a human body contact point. Based on the current interaction scenario information, generate the contact activation weight corresponding to each human contact part token; For each object point and each human contact part token, the contact association strength is calculated based on the object point characteristics, local geometric information, human contact part token, and the activation weight of the contact part. The point-part level relationship field is formed by all contact association strengths. The point-to-part level relation field is subjected to sparsification and normalization to obtain the processed relation field; Based on the processed relation field, the human body contact point tokens are weighted and aggregated to obtain the contact perception calibration vector corresponding to each object point. Based on the object point features of each object point, the initial interaction prediction value, and the contact perception calibration vector, the enhancement amount, the suppression amount, and the residual calibration amount are generated respectively. The interaction prediction values ​​are calibrated based on the initial interaction prediction values ​​of each object point and the corresponding enhancement, suppression and residual calibration values ​​to obtain the calibrated interaction prediction values. The final object interaction area is determined based on the interaction prediction values ​​after calibration of each object point.

2. The object point-level interaction region prediction and calibration method based on the human body contact part relationship field according to claim 1, characterized in that, The initial interactive prediction value is the pre-normalized prediction score output by the point-level prediction model. After calibration, the Sigmoid function is used to map the calibrated interaction prediction score to the interaction probability of object points in the range of 0 to 1.

3. The object point-level interaction region prediction and calibration method based on the human body contact part relationship field according to claim 1, characterized in that, The activation weights of the contact points are generated based on at least one of action semantics, human posture, object category, task description, or external semantic information. Specifically, the aforementioned information is feature-encoded to obtain scene features, which are then input into the activation weight generation module to output K initial weight values. These values ​​are then normalized using the Softmax function to obtain the activation weights corresponding to each human contact point token.

4. The object point-level interaction region prediction and calibration method based on the human body contact part relationship field according to claim 1, characterized in that, The contact association strength is calculated as follows: the object point features of the i-th object point are fused with local geometric information to obtain a comprehensive feature; the comprehensive feature is concatenated with the j-th human contact part token and input into the relationship strength calculation module to obtain the initial relationship strength; then, the initial relationship strength is weighted using the activation weight corresponding to the j-th human contact part to obtain the contact association strength r. ij .

5. The object point-level interaction region prediction and calibration method based on the human body contact part relationship field according to claim 4, characterized in that, The relationship strength calculation module is implemented using multilayer perceptron, similarity calculation, gating network, or attention calculation.

6. The object point-level interaction region prediction and calibration method based on the human body contact part relationship field according to claim 1, characterized in that, The sparsification process is as follows: for each object point, retain the M human contact parts with the highest contact association strength, where M is a positive integer less than K, and set the relationship strength corresponding to the other human contact parts to zero; the normalization process is as follows: normalize the retained relationship strengths so that the contact association strengths corresponding to the same object point remain numerically stable.

7. The object point-level interaction region prediction and calibration method based on the human body contact part relationship field according to claim 1, characterized in that, The contact sensing calibration vector c i The calculation method is as follows: ； Among them, c i Let represent the contact sensing calibration vector corresponding to the i-th object point. b represents the strength of the relationship between the i-th object point and the j-th human body contact point after processing. j This represents the token representing the contact point of the j-th person.

8. The object point-level interaction region prediction and calibration method based on the human body contact part relationship field according to claim 1, characterized in that, The enhancement, suppression, and residual calibration values ​​are generated by the predictive calibration module; the predictive calibration module includes a calibration feature fusion module, a calibration value generation module, and a calibration fusion module. The calibration feature fusion module is used to fuse object point features, initial interaction prediction values, and contact sensing calibration vectors to obtain calibration fusion features; the calibration quantity generation module generates enhancement, suppression, and residual calibration quantities based on the calibration fusion features; the calibration fusion module is used to perform calibration based on the initial interaction prediction values, enhancement, suppression, and residual calibration quantities.

9. The object point-level interaction region prediction and calibration method based on the human body contact part relationship field according to claim 8, characterized in that, The calibration method of the calibration fusion module is as follows: ； Among them, z i Let z' represent the initial interaction prediction score for the i-th object point. i e represents the calibrated interactive prediction score. i Indicates the enhancement amount, s i Indicates the amount of inhibition, Δ i λ represents the residual calibration value. e λ represents the enhancement coefficient. s This represents the inhibition coefficient.

10. The object point-level interaction region prediction and calibration method based on the human body contact part relationship field according to claim 1, characterized in that, Also includes: Based on the processed relation field, determine the dominant human contact point corresponding to each object point, and output the interpretation result of the human contact point corresponding to the object interaction area.

Images