Hand contact state detection method and system

Abstract

The invention discloses a hand contact state detection method and system, and relates to the technical field of computer vision and artificial intelligence. The method comprises the following steps: inputting a to-be-detected image, and extracting bounding boxes and initial features of hands, objects and people in the to-be-detected image through a backbone network and an auxiliary detector; analyzing a spatial and semantic relationship between a hand and a detected object through a hand-object relationship module, and outputting a confidence score related to hand-object interaction; modeling an interaction relationship between the hand and the detected person through a contrast space refining module, improving the interpersonal contact recognition capability in a complex scene by the contrast space refining module based on contrast learning and spatial relationship enhancement, and outputting a confidence score related to hand-person interaction; and predicting the contact state of each detected hand in combination with the confidence score related to the hand-object interaction and the confidence score related to the hand-human interaction. The method can be applied to human-computer interaction, video monitoring, social behavior analysis, virtual reality and the like.

Description

Technical Field

[0001] The present invention relates to the field of computer vision and artificial intelligence technology, and in particular to a hand contact state detection method and system. Background Art

[0002] Accurately detecting hand contact status is crucial for understanding human behavior, intentions, and environmental interactions. In intelligent surveillance, it can be used to identify abnormal contact or unsafe behavior; in human-computer interaction systems, it enables more natural and refined command understanding; in virtual and augmented reality, it helps create a more realistic interactive experience; and in social robotics and medical assistance, it is also crucial for understanding social cues and assisting specific groups of people. With the advancement of deep learning technology, automated, high-precision hand contact status detection has become a research hotspot.

[0003] Hand contact state detection (HCSD) aims to identify and classify physical interactions between hands and objects, hands and one's own body parts, or hands and other people's body parts. This technology is crucial for building advanced human-computer interaction systems, understanding human social dynamics, and analyzing human behavior in various real-world applications.

[0004] However, the task of detecting hand contact states faces many inherent challenges: first, the hand area occupies a relatively small proportion in the image and provides limited direct visual information; second, severe occlusion often occurs in complex interactive scenes, especially in crowded environments where hands, bodies, and interactive objects easily occlude each other; third, the visual differences between different contact states (such as contact with an object, no contact, self-contact, and contact with others) may be very subtle and difficult to distinguish.

[0005] Academia and industry have already conducted research and developed models to address these challenges. For example, early methods attempted to directly use detected hand features for classification, but due to limited visual information about the hand, these methods often performed poorly. Subsequent approaches, such as CENet, introduced auxiliary object detectors to jointly analyze the hand and objects in the scene. By incorporating object context, these methods improved the analysis of hand-object interactions to a certain extent. These methods have made some progress in modeling hand-object interactions.

[0006] Despite progress, existing methods still have significant shortcomings in handling hand-human interactions, particularly distinguishing between hand contact with the body itself (self-contact) and hand contact with others (other-contact). Many models fail to fully exploit key contextual cues between the hand and its own body, as well as the bodies of other people in the scene. This lack of dedicated hand-human interaction modeling leads to significant performance degradation in situations involving interpersonal contact (especially contact with others) and in crowded scenes with multiple people, and is prone to confusion between different contact states.

[0007] Therefore, how to model hand-human interaction in a precise and robust manner and, based on this, accurately identify various hand contact states, especially in challenging scenarios such as occlusion and crowding, is a key issue that needs to be urgently addressed in the current field of hand contact state detection. Summary of the Invention

[0008] To address these issues, the present invention proposes a hand contact state detection method and system for identifying the physical interaction between a human hand and an object or other individual. This system utilizes a deep learning network to meticulously model the contextual relationships between hands, objects, and people. This system is particularly well-suited for detecting and understanding hand contact states in complex scenarios, such as those with occlusion or crowded environments. It has applications in human-computer interaction, video surveillance, social behavior analysis, and virtual reality.

[0009] The technical solution adopted in the present invention is as follows: A method for detecting a hand contact state, comprising: Input the image to be tested, and use the backbone network and auxiliary detector to extract the bounding boxes and initial features of hands, objects, and people in the image to be tested; The hand-object relationship module analyzes the spatial and semantic relationship between the hand and the detected object and outputs a confidence score related to the hand-object interaction; Modeling the interaction between hands and detected people through a contrastive space refinement module, which improves human contact recognition in complex scenarios based on contrastive learning and spatial relationship enhancement, and outputs a confidence score related to the hand-human interaction; The confidence scores related to hand-object interaction and hand-human interaction are combined to predict the contact state of each detected hand.

[0010] Furthermore, the hand-object relationship module includes: a spatial weighting unit, configured to capture the spatial distribution of interactions within the hand-object joint region; The cross-region attention unit is configured to capture the pairwise affine relationship between the hand region and the object region and model the contact probability of the region correspondence.

[0011] Furthermore, in the spatial weighting unit, capturing the spatial distribution of interactions within the hand-object joint area includes: highlighting the area most relevant to the contact state prediction by generating a spatial attention map.

[0012] Furthermore, in the cross-region attention unit, by capturing the paired affine relationship between the hand region and the object region, it includes: calculating the spatial correspondence between the hand features and the hand-object joint region features.

[0013] Furthermore, the contrast space refinement module includes: a contrastive geometric embedding unit, configured to enhance the discriminability of hand representations through geometric anchoring and contrastive disentanglement; Relative spatial positioning units are configured to enhance modeling of interacting regions by explicitly encoding spatial dependencies.

[0014] Furthermore, in the contrastive geometric embedding unit, the discriminability of hand representation is enhanced through geometric anchoring and contrastive disentanglement, including: Geometric anchoring: The hand center coordinates are used as persistent homologous points, and the feature maps are sampled using these homologous points to capture global hand semantic information; Contrastive Disentanglement: The contrastive loss function is used to bring different views of the same hand closer in the embedding space, that is, to maximize the mutual information between the geometric anchor features and their enhanced views; and to move features of different hands away from each other, that is, different hand identities are effectively separated in the embedding space.

[0015] Furthermore, in the relative spatial positioning unit, modeling of the interaction region is enhanced by explicitly encoding spatial dependencies, including: Relative position enhancement: For each hand-person pair, a normalized relative coordinate map is generated using the center coordinates of the bounding box. Appearance compatibility is captured by element-wise multiplication between hand features and body features, and channel-wise summation is performed to produce an aggregated feature map. The relative coordinate map and the aggregated feature map are combined to generate spatial features. Fusion and interaction scoring: Based on the geometric context features from the contrastive geometric embedding unit, the spatial features obtained by relative position enhancement are fused to generate enhanced human features that incorporate hand information.

[0016] Furthermore, the relative space positioning unit also includes bidirectional enhancement: based on the enhanced human body features fused with hand information, a bidirectional enhancement method is used to enhance the hand features, and a confidence score related to the hand-human interaction is calculated through a multi-layer perceptron.

[0017] Furthermore, in the predicted contact state of each detected hand, the contact state includes a no-contact state, a self-contact state, a contact state with others, and a contact state with an object.

[0018] A hand contact state detection system, comprising: The backbone network and auxiliary detectors are configured to extract bounding boxes and initial features of hands, objects, and people in the image to be tested; A hand-object relationship module is configured to analyze the spatial and semantic relationships between the hand and the detected object and output a confidence score related to the hand-object interaction; a contrastive space refinement module configured to model the interaction relationship between the hand and the detected person, the contrastive space refinement module improving the interpersonal contact recognition capability in complex scenarios based on contrastive learning and spatial relationship enhancement, and outputting a confidence score related to the hand-person interaction; The hand contact state prediction module is configured to combine the confidence score related to the hand-object interaction and the confidence score related to the hand-human interaction to predict the contact state of each detected hand.

[0019] The beneficial effects of the present invention are: 1. Enhanced hand-human interaction modeling: The contrast space refinement module of the present invention can explicitly model the complex interactions between hands and people, effectively distinguishing between self-contact and contact with others, especially in crowded and occluded scenes. It is significantly better than the existing method that only relies on general object detection processes to process the human body.

[0020] 2. Improved Accuracy in Complex Scenarios: The present invention's hand-object relationship module effectively captures key spatial and contextual cues of hand-object interactions through spatial weighting and cross-regional attention mechanisms. The contrastive spatial refinement module enhances the discernibility of hand features and accurately infers hand ownership by contrasting geometric embeddings and relative spatial positioning. This combination enables the present invention to accurately detect hand contact even in complex environments with frequent occlusions and overlapping multiple people.

[0021] 3. Comprehensive interaction understanding: This paper systematically captures the contextual semantics and geometric properties of hand-object and hand-human interactions by decoupling their feature learning processes, enabling the model to perform accurate contact state judgment based on specialized path-driven reasoning, rather than relying solely on appearance information.

[0022] 4. Significant performance improvement: Experiments have shown that the present invention has achieved leading performance on public benchmark datasets (such as ContactHands and 100DOH), especially in the detection accuracy of the "other contact" state, which is significantly improved compared with the existing technology, proving the effectiveness of the present invention in solving the challenges of hand contact state detection.

[0023] In summary, the present invention uses a deep learning network to finely model the contextual relationship between hands, objects, and people. It is particularly suitable for detecting and understanding hand contact status in complex scenarios (such as occlusion and crowded environments), and can be applied to human-computer interaction, video surveillance, social behavior analysis, and virtual reality. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 This is a schematic diagram of a hand contact state detection method according to embodiment 3 of the present invention.

[0025] Figure 2This is a schematic diagram of the hand-object relationship module according to embodiment 3 of the present invention.

[0026] Figure 3 This is a schematic diagram of the contrast space refinement module of Example 3 of the present invention. DETAILED DESCRIPTION

[0027] In order to have a clearer understanding of the technical features, purposes and effects of the present invention, the specific embodiments of the present invention are now described. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention. That is, the embodiments described are only part of the embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without making creative work are within the scope of protection of the present invention.

[0028] Example 1 This embodiment provides a method for detecting a hand contact state, including: Input the image to be tested, and use the backbone network and auxiliary detector to extract the bounding boxes and initial features of hands, objects, and people in the image to be tested; The hand-object relationship module analyzes the spatial and semantic relationship between the hand and the detected object and outputs a confidence score related to the hand-object interaction; The contrastive space refinement module models the interaction relationship between the hand and the detected person. The contrastive space refinement module improves the ability to recognize human contact in complex scenarios based on contrastive learning and spatial relationship enhancement, and outputs a confidence score related to the hand-human interaction. The confidence scores related to hand-object interaction and hand-human interaction are combined to predict the contact state of each detected hand (e.g., no contact, self-contact, contact with others, and contact with objects).

[0029] It should be noted that existing technologies lack specialized and sophisticated mechanisms for modeling hand-human interactions (especially those between hands and other people's bodies), resulting in low accuracy in distinguishing between states such as "self-contact" and "contact with others." To address the shortcomings of existing methods in hand-human interaction analysis, particularly the difficulty in distinguishing between self-contact and other-contact, this embodiment proposes a new Hand-Object-Human Interaction Network (HOPNet). This network meticulously models contextual information between hands and objects, as well as between hands and bodies (one's own and others' bodies), thereby improving the accuracy of identifying interpersonal contact states in complex scenarios.

[0030] Furthermore, in complex visual scenarios such as crowding and occlusion, existing technologies struggle to accurately infer hand ownership, distinguish individuals, and effectively utilize incomplete visual information to determine contact status, resulting in a significant performance degradation. To improve the accuracy of interpersonal contact status recognition in crowded scenarios, particularly, and to address the issues of unclear hand features and unclear ownership due to occlusion and overlap, this embodiment designs a contrastive spatial refinement (CSR) module. This module learns hand-body interactions through contrastive geometric embedding and handles spatial ambiguity through relative spatial enhancement, significantly improving the accuracy and reliability of interpersonal contact status recognition in crowded scenarios.

[0031] Preferably, the hand-object relationship module of this embodiment includes: a spatial weighting unit, configured to capture the spatial distribution of interactions within the hand-object joint region; The cross-region attention unit is configured to capture the pairwise affine relationship between the hand region and the object region and model the contact probability of the region correspondence.

[0032] More preferably, in the spatial weighting unit, capturing the spatial distribution of interactions within the hand-object joint region includes: highlighting the region most relevant to the contact state prediction by generating a spatial attention map.

[0033] More preferably, in the cross-region attention unit, by capturing the pairwise affine relationship between the hand region and the object region, the spatial correspondence between the hand features and the hand-object joint region features is calculated.

[0034] Preferably, the contrast space refinement module of this embodiment includes: a contrastive geometric embedding unit, configured to enhance the discriminability of hand representations through geometric anchoring and contrastive disentanglement; Relative spatial positioning units are configured to enhance modeling of interacting regions by explicitly encoding spatial dependencies.

[0035] More preferably, in the contrastive geometric embedding unit, the discriminability of the hand representation is enhanced by geometric anchoring and contrastive disentanglement, including: Geometric anchoring: The hand center coordinates are used as persistent homologous points, and the feature maps are sampled using homologous points to capture global hand semantic information; Contrastive Disentanglement: The contrastive loss function is used to bring different views of the same hand closer in the embedding space, that is, to maximize the mutual information between the geometric anchor features and their enhanced views; and to move features of different hands away from each other, that is, different hand identities are effectively separated in the embedding space.

[0036] More preferably, in the relative spatial positioning unit, modeling of the interaction region is enhanced by explicitly encoding the spatial dependency, comprising: Relative position enhancement: For each hand-person pair, a normalized relative coordinate map is generated using the center coordinates of the bounding box. Appearance compatibility is captured by element-wise multiplication between hand features and body features, and channel-wise summation is performed to produce an aggregated feature map. The relative coordinate map and the aggregated feature map are combined to generate spatial features. Fusion and interaction scoring: Based on the geometric context features from the contrastive geometric embedding unit, the spatial features obtained by relative position enhancement are fused to generate enhanced human features that incorporate hand information; Bidirectional enhancement: Based on the enhanced human features of the fused hand information, a bidirectional enhancement method is used to enhance the hand features, and the confidence score related to the hand-human interaction is calculated through a multi-layer perceptron.

[0037] In summary, the hand contact state detection method of this embodiment has the following characteristics: 1. Dual-Path Interaction Modeling Mechanism: This embodiment proposes a dual-path network architecture (HOPNet), comprising a Hand-Object Relationship (HOR) module specifically for hand-object interaction and a Contrastive Space Refinement (CSR) module specifically for hand-human interaction. This decoupled modeling approach systematically captures the contextual semantics and geometric characteristics of different interaction types, representing a key innovation that distinguishes it from existing single-path approaches or methods that lack dedicated hand-human interaction modeling.

[0038] 2. Specific Techniques in the Hand-Object Relationship (HOR) Module: The HOR module employs a spatial weighting mechanism to highlight key interaction regions within the hand-object joint region and incorporates a cross-region attention mechanism to capture detailed dependencies between the hand and object regions. The combination of these two techniques provides effective features for accurately analyzing hand-object interactions.

[0039] 3. Specific technologies in the Contrastive Spatial Refinement (CSR) module: The CSR module introduces Contrastive Geometric Embedding (CGE) technology to enhance the discriminability of hand features under occlusion through geometric anchoring and contrastive learning. Furthermore, the module employs Relative Spatial Localization (RSL) technology to explicitly model hand-person proximity by generating relative coordinate maps and spatial attention masks to address hand attribution ambiguity in crowded scenes. The Bidirectional Enhancement (BE) process in the CSR module further refines the correlation between hand and person features.

[0040] 4. Robustness in complex scenarios (especially crowding and occlusion): This embodiment significantly improves the accuracy and robustness of hand contact state detection in complex scenarios such as crowding, multi-person overlap, and partial occlusion through the collaborative work of the HOR module and the CSR module. In particular, the ability to distinguish between "self-contact" and "other-contact" far exceeds existing technologies.

[0041] Example 2 This embodiment is based on embodiment 1: This embodiment provides a method for detecting hand contact states. Given an input image, the method first extracts bounding boxes and initial features of hands, objects, and people in the image through a backbone network (such as ResNet-101 and Feature Pyramid Network (FPN)) and auxiliary detectors. These features are then fed into two specially designed relational reasoning modules: the Hand-Object Relation (HOR) module and the Contrastive Spatial Refinement (CSR) module. The Hand-Object Relation module analyzes the spatial and semantic relationships between the hand and the detected object and outputs a confidence score related to the hand-object interaction. The Contrastive Spatial Refinement module focuses on modeling the interaction between the hand and detected people (including the hand itself and others). It improves interpersonal contact recognition in complex scenarios through contrastive learning and spatial relationship enhancement, and outputs a confidence score related to the hand-person interaction. Finally, the interaction scores from these two modules are summed to obtain a final score, which is used to predict the contact state of each detected hand (e.g., no contact, self-contact, contact with another person, or contact with an object).

[0042] Specifically, the hand-object relationship module, contrast space refinement module, and their corresponding hand contact state prediction and loss function of this embodiment are described as follows.

[0043] 1. Hand-Object Relationship (HOR) Module The HOR module is designed to capture the unique interaction patterns and spatial dependencies between the hand and the object to address the problem of the small hand region and lack of contextual information. It adopts a two-branch architecture: spatial weighting and cross-region attention.

[0044] (1) Spatial weighting The spatial weighted branch focuses on capturing the spatial distribution of interactions within the hand-object joint region. To highlight the areas most relevant to contact state prediction. Spatial Attention Map The calculation formula is: ,in Is a 1×1 convolutional layer used to generate An attention map. It is Joint features of hand-object pairs. Projected features pass Obtain, among which Hand-object interaction score Element-wise product of the attention map and the projected features Calculate the mean of :

[0045] Final spatially weighted score The scores of all detected objects are averaged and passed through a multi-layer perceptron (MLP):

[0046] (2) Cross-regional attention The cross-region attention branch aims to model region-specific contact probability by capturing pairwise affinities between the hand region and the object region. It computes the hand features and hand-object joint regional features The spatial correspondence between attention weights Use 1×1 convolutional layer and calculate:

[0047] Where × represents matrix multiplication, Indicates transpose. The function ensures that the attention weights are normalized. Enhanced feature representation By combining the attention-weighted joint features with the original hand features: ,in Represents batch normalization.

[0048] To perform contact prediction, three flattened feature representations are first generated: the original hand features , joint regional context features , and features after attention fusion .

[0049] Final contact score The concatenated features are processed by the classifier. These features are passed through the fully connected layers (FC) before concatenation: .

[0050] 2. Contrastive Space Refinement (CSR) Module The CSR module is designed to enhance spatial reasoning in hand-human interaction analysis, specifically to accurately distinguish between self-contact and other-contact in crowded and occluded scenes. This module integrates Contrastive Geometric Embedding (CGE) and Relative Spatial Localization (RSL), and employs a bidirectional enhancement (BE) process to comprehensively capture the interrelationship between the hand and the person.

[0051] (1) Comparison with geometric embedding Contrastive geometric embeddings enhance the discriminability of hand representations through geometric anchoring.

[0052] Geometric Anchoring: The hand center coordinates As persistent homology points, these points can maintain structural integrity under hand movement and deformation. Sampling is performed to capture global hand semantic information in a way that is robust to partial occlusion. The sampling process is defined as: ,in It is Anchoring feature of the hand.

[0053] Contrastive Disentanglement: To maximize the mutual information between the geometric anchor features and their enhanced views and ensure that different hand identities are effectively separated in the embedding space, a contrastive loss function is used.

[0054] in It is One-handed Normalized eigenvector, is the temperature coefficient. This loss function encourages different views of the same hand to be closer in the embedding space, and features of different hands to be farther away from each other.

[0055] CGE will characterize the human body (No. Individual characteristics) and hand geometric prior features Perform element-wise multiplication (⊙) and sum it up and then perform Normalization (Norm) realizes geometrically perceived channel interaction and obtains enhanced human context features

[0056]

[0057] (2) Relative spatial positioning Relative spatial positioning enhances modeling of interaction regions by explicitly encoding spatial dependencies.

[0058] Relative Position Enhancement (RPE): For each hand-person pair (No. Hand and Individual), use the bounding box and its center coordinates to generate a normalized relative coordinate map The computation of such relative coordinates takes into account the intrinsic spatial information of each person, generating a scale-invariant geometric descriptor that robustly captures fine-scale positional interactions.

[0059] Hand features and human characteristics The element-wise product between captures the appearance compatibility, followed by channel-wise summation to produce the aggregated feature map

[0060]

[0061] Spatial Attention Mask By putting the relative coordinates and aggregate feature maps After concatenation, pass through a standard convolutional layer (used to reduce channel dimensionality) and a Activation Function generate

[0062] Refined human features By using this spatial attention mask Applied to original human features , then for all The results of the two hands are summed and carried out Normalized to get

[0063] Fusion and interaction scoring: the final enhanced human features is achieved by fusing geometric context features from CGE and spatial features from RSL The specific operation is to and After concatenation, pass through a 1×1 convolutional layer Then after the ReLU activation function and the maximum pooling (MaxPool) operation

[0064] It serves as an enhanced human representation that incorporates hand information.

[0065] Bidirectional Enhancement (BE): After obtaining the human features that are integrated with the hand information (in is the number of people) Then, we use a two-way enhancement method to enhance the hand features in turn In this process, the hand center features in formulas (5)-(9) are Replace with , human body characteristics Replace with hand features After the CGE and RSL processing, the enhanced hand features are obtained. , this feature contains the human body context information.

[0066] Human context score Finally, the enhanced hand features Input a multi-layer perceptron (MLP) to calculate

[0067] 3. Hand contact state prediction Preferably, the final hand contact state prediction score , which can be obtained by combining the spatial attention score from the HOR module and cross-region interaction scores , and the human context score from the CSR module Adding them together we get: This total score It is then used to classify the hand contact status into one of several types: No-Contact, Self-Contact, Other-Person Contact, or Object-Contact.

[0068] 4. Loss function design The method in this embodiment uses a multi-task loss function for end-to-end training, including the standard Mask R-CNN loss and the hand contact state prediction loss:

[0069] in, is the classification loss, is the bounding box regression loss, is the mask segmentation loss, and all three come from the Mask R-CNN framework. is a loss term specifically designed to optimize the accuracy of hand contact state prediction. Considering that a hand may have multiple potential contact states at the same time (although usually marked as a single main state), Use the binary cross-entropy loss (BCE) of each contact state and combine it with the contrast regularization term (defined in the CGE module) ) to calculate:

[0070] in, The model is The predicted score of each contact state (after deal with), is the corresponding true label (0 or 1). It is a balancing hyperparameter, set to 1, which is used to weigh the BCE loss term and the contrast loss term.

[0071] Example 3 This embodiment is based on embodiment 1: like Figure 1 As shown, this embodiment provides a hand contact state detection method, including input image processing, hand-object relationship reasoning and contrast space refinement, which can be specifically implemented in the following manner.

[0072] 1. Input image processing This embodiment processes the input image in three main stages: feature extraction, relational reasoning, and contact prediction. First, a ResNet-101 backbone network is used in conjunction with a feature pyramid network (FPN) to extract multi-scale feature maps from the input image. Second, a region proposal network (RPN) generates hand region proposals, which are refined through RolAlign (Region of Interest Align, a feature aggregation method for object detection tasks) and bounding box regression to produce Dimensional hand features ,in and Represent the height and width of the feature respectively. At the same time, the auxiliary detector based on Mask R-CNN will identify the objects and people in the image and generate their corresponding features and ,in and Represents the number of objects and people respectively.

[0073] 2. Hand-Object Relationship Reasoning like Figure 2 As shown in Figure 2, the interaction pattern between the hand and the object is captured by the hand-object relationship module. In this module, the spatial weighted branch processes the hand-object joint features through the convolutional layer. To generate the attention map and projection features Then, the preliminary spatial interaction score vector is calculated by element-wise multiplication and mean pooling, and then the score of each hand-object pair is obtained through MLP. The cross-region attention branches process hand features separately. and hand-object joint features , calculate the attention weight after convolution projection And use this weight to fuse features to get enhanced features Finally, the original , and enhanced The flattened features are fed into the fully connected layer and concatenated to obtain the cross-region interaction score through MLP. .

[0074] 3. Contrast Space Refinement like Figure 3 As shown in Figure 2, the spatial reasoning ability in hand-human interaction analysis is enhanced through the contrastive spatial refinement module. This module contains two main sub-modules: contrastive geometric embedding (CGE) and relative spatial localization (RSL), and adopts a bidirectional enhancement strategy.

[0075] Preferably, the CGE submodule uses geometric anchoring (using the hand center point to sample the hand geometric features from the global features) ) and contrastive learning loss ( ) to enhance the distinction between different hand features. At the same time, it and hand geometry prior Perform element-wise multiplication and normalization to obtain geometrically aware human context features .

[0076] Preferably, the RSL submodule is implemented for each opponent-person through the relative position enhancement block (RPE). Generate a normalized relative coordinate plot Hand features and human characteristics Perform element-wise multiplication and aggregation to obtain feature maps . and After splicing, the spatial attention mask is generated by convolution and Sigmoid function , used to enhance human features .

[0077] Geometric features and spatial characteristics Enhanced human representation after fusion Preferably, the hand representation is similarly enhanced through a bidirectional enhancement mechanism. .at last, Get person-context score through MLP .

[0078] IV. Experimental Setup This example uses the ContactHands and 100DOH datasets. ContactHands is an image dataset specifically constructed for hand contact state detection. These images are detailed with hand instances and their corresponding contact states, providing a rich and diverse range of real-world scenarios for model training and evaluation. The 100DOH dataset focuses on capturing hand interactions from continuous video streams and includes images extracted from 100,000 video frames to ensure the independence of data distribution, thereby more comprehensively evaluating the model's generalization ability in dynamic scenarios.

[0079] This example uses the standard PASCAL VOC Average Precision (AP) as the main evaluation metric, requiring the bounding box IoU ≥ 0.5 and the contact state to be accurately matched.

[0080] All experiments were conducted on an NVIDIA GeForce RTX 3090 based on the PyTorch framework. Mask R-CNN was used as the backbone network and initialized with pre-trained weights from Detectron2. The input image resolution and data augmentation strategy followed the configuration of the baseline method CENet for fair comparison. The network was optimized using a stochastic gradient descent (SGD) optimizer with momentum set to 0.9 and weight decay set to 0.0001. Following the configuration of CENet, the batch size was set to 1. For the auxiliary detector, the confidence threshold for human detection was set to 0.5, and the detection of other object categories maintained the default settings. The initial learning rate was set to , and adopt a step-by-step attenuation strategy: for the ContactHands dataset, the learning rate is decayed by multiplying by 0.1 at the 20th and 30th epochs respectively; for the 100DOH dataset, based on experience, the learning rate is decayed by multiplying by 0.1 at the 10th and 20th epochs respectively.

[0081] 5. Experimental Verification To validate the technical effectiveness of our method, we conducted a comprehensive comparison with current mainstream methods such as CENet on two public benchmark datasets: ContactHands and 100DOH. The experimental results demonstrate that our method effectively addresses the recognition bottlenecks of existing technologies in complex interaction scenarios and significantly enhances the model's accuracy and adaptability to different scenarios.

[0082] 1. Improved the recognition accuracy of key interaction categories Existing methods for processing hand interactions lack specialized modeling of hand-to-human interactions, resulting in recognition bottlenecks when distinguishing similar states such as "self-contact" and "other-contact." Based on this, this paper proposes a dual-branch decoupling architecture. By introducing a contrastive space refinement module (CSR), it significantly enhances the model's ability to understand and discriminate interpersonal interactions. Its effectiveness has been verified on two major public datasets: On the ContactHands dataset, the method of the present invention improves the recognition accuracy (AP) of the key category "other contact" from about 40% of the existing technology to more than 44%, achieving a steady performance increase.

[0083] On the 100DOH dataset, which has even lower baseline performance, the present invention achieved a breakthrough effect, improving the recognition accuracy of this category by more than 17 percentage points, doubling the performance.

[0084] This series of results fully demonstrates that the present invention effectively solves the bottleneck of the existing technology and ultimately surpasses the existing methods in terms of comprehensive performance indicators (mAP).

[0085] 2. Enhanced adaptability in complex scenes such as crowding and occlusion In crowded scenes with large crowds, frequent body occlusion and overlap make hand attribution determination extremely difficult, a major technical challenge for existing methods. Our method, through a unique modular design, enables the model to reason based on geometric spatial relationships rather than relying solely on vague appearance features, demonstrating strong adaptability in complex environments.

[0086] Taking a highly crowded scene with more than 15 people as an example, the recognition accuracy of the method of the present invention in the key difficult category of "contact with others" is improved by nearly 10 percentage points compared with the existing technology; at the same time, the mAP in this complex scene can still reach more than 44%, demonstrating stable recognition capabilities and overall superiority in extreme environments.

[0087] Example 4 This embodiment is based on embodiment 1: This embodiment provides a hand contact state detection system, including: The backbone network and auxiliary detectors are configured to extract bounding boxes and initial features of hands, objects, and people in the image to be tested; A hand-object relationship module is configured to analyze the spatial and semantic relationships between the hand and the detected object and output a confidence score related to the hand-object interaction; A contrastive space refinement module is configured to model the interaction relationship between the hand and the detected person. The contrastive space refinement module improves the interpersonal contact recognition ability in complex scenes based on contrastive learning and spatial relationship enhancement, and outputs a confidence score related to the hand-person interaction; The hand contact state prediction module is configured to combine the confidence score related to the hand-object interaction and the confidence score related to the hand-human interaction to predict the contact state of each detected hand.

[0088] Example 5 This embodiment is based on embodiment 1: This embodiment provides a computer device including a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, it implements a hand contact state detection method of embodiment 1. The computer program may be in source code form, object code form, executable file, or some intermediate form.

[0089] Example 6 This embodiment is based on embodiment 1: This embodiment provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements a hand contact state detection method of embodiment 1. The computer program may be in source code form, object code form, executable file, or some intermediate form. The storage medium includes: any entity or device capable of carrying computer program code, recording medium, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal, and software distribution medium. It should be noted that the content contained in the storage medium may be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, the storage medium does not include electric carrier signal and telecommunication signal.

[0090] It should be noted that, for the sake of simplicity, the aforementioned method embodiments are described as a series of action combinations. However, those skilled in the art should be aware that this application is not limited by the order of the actions described, because according to this application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in this specification are all preferred embodiments, and the actions and modules involved are not necessarily required by this application.

Claims

1. A method for detecting hand contact status, characterized in that: include: Input the image to be tested, and use the backbone network and auxiliary detector to extract the bounding boxes and initial features of hands, objects, and people in the image to be tested; The hand-object relationship module analyzes the spatial and semantic relationship between the hand and the detected object and outputs a confidence score related to the hand-object interaction; Modeling the interaction between hands and detected people through a contrastive space refinement module, which improves human contact recognition in complex scenarios based on contrastive learning and spatial relationship enhancement, and outputs a confidence score related to the hand-human interaction; The confidence scores related to hand-object interaction and hand-human interaction are combined to predict the contact state of each detected hand.

2. The hand contact state detection method according to claim 1, characterized in that: The hand-object relationship module includes: a spatial weighting unit, configured to capture the spatial distribution of interactions within the hand-object joint region; The cross-region attention unit is configured to capture the pairwise affine relationship between the hand region and the object region and model the contact probability of the region correspondence.

3. The hand contact state detection method according to claim 2, characterized in that: In the spatial weighting unit, the spatial distribution of interactions within the hand-object joint region is captured, including: highlighting the region most relevant to the contact state prediction by generating a spatial attention map.

4. The hand contact state detection method according to claim 2, characterized in that: In the cross-region attention unit, by capturing the paired affine relationship between the hand region and the object region, the method includes calculating the spatial correspondence between the hand features and the hand-object joint region features.

5. The hand contact state detection method according to claim 1, characterized in that: The contrast space refinement module includes: a contrastive geometric embedding unit, configured to enhance the discriminability of hand representations through geometric anchoring and contrastive disentanglement; Relative spatial positioning units are configured to enhance modeling of interacting regions by explicitly encoding spatial dependencies.

6. The hand contact state detection method according to claim 5, characterized in that: In the contrastive geometric embedding unit, the discriminability of hand representation is enhanced through geometric anchoring and contrastive disentanglement, including: Geometric anchoring: The hand center coordinates are used as persistent homologous points, and the feature maps are sampled using these homologous points to capture global hand semantic information; Contrastive Disentanglement: The contrastive loss function is used to bring different views of the same hand closer in the embedding space, that is, to maximize the mutual information between the geometric anchor features and their enhanced views; and to move features of different hands away from each other, that is, different hand identities are effectively separated in the embedding space.

7. The hand contact state detection method according to claim 5, characterized in that: In the relative spatial positioning unit, modeling of the interaction region is enhanced by explicitly encoding spatial dependencies, including: Relative position enhancement: For each hand-person pair, a normalized relative coordinate map is generated using the center coordinates of the bounding box. Appearance compatibility is captured by element-wise multiplication between hand features and body features, and channel-wise summation is performed to produce an aggregated feature map. The relative coordinate map and the aggregated feature map are combined to generate spatial features. Fusion and interaction scoring: Based on the geometric context features from the contrastive geometric embedding unit, the spatial features obtained by relative position enhancement are fused to generate enhanced human features that incorporate hand information.

8. The hand contact state detection method according to claim 7, characterized in that: The relative space positioning unit also includes bidirectional enhancement: based on the enhanced human body features fused with hand information, a bidirectional enhancement method is used to enhance the hand features, and a confidence score related to the hand-human interaction is calculated through a multi-layer perceptron.

9. The hand contact state detection method according to claim 1, characterized in that: In the predicting of the contact state of each detected hand, the contact state includes a non-contact state, a self-contact state, a contact state with others, and a contact state with an object.

10. A hand contact state detection system, characterized in that: include: The backbone network and auxiliary detectors are configured to extract bounding boxes and initial features of hands, objects, and people in the image to be tested; A hand-object relationship module is configured to analyze the spatial and semantic relationships between the hand and the detected object and output a confidence score related to the hand-object interaction; a contrastive space refinement module configured to model the interaction relationship between the hand and the detected person, the contrastive space refinement module improving the interpersonal contact recognition capability in complex scenarios based on contrastive learning and spatial relationship enhancement, and outputting a confidence score related to the hand-person interaction; The hand contact state prediction module is configured to combine the confidence score related to the hand-object interaction and the confidence score related to the hand-human interaction to predict the contact state of each detected hand.

Images