A body parts association method and system based on bidirectional collaborative reasoning

Abstract

The present invention relates to the field of computer vision technology, and discloses a body part association method and system based on bidirectional collaborative reasoning, wherein the method uses a bidirectional collaborative method to infer the association relationship between the hand and the body. The present invention solves the problems of unidirectional uncertainty and spatial ambiguity existing in the prior art.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a body part association method and system based on bidirectional collaborative reasoning. Background Art

[0002] With the continuous development of human-computer interaction technology and computer vision technology, it has brought unprecedented innovations and opportunities to many industries, including but not limited to human posture recognition, behavior analysis, medical image processing, etc. The purpose of hand-body association is to match the detected hand with the corresponding body in the image, which plays an important role in human-centered visual tasks. For example, in intelligent monitoring systems, accurate identification and association of the position and movement of different parts of the human body can be used for anomaly detection and behavior analysis, improving the security and intelligence level of the system. Therefore, the development of an efficient and accurate body part association method is of great significance to promoting the development of human-computer interaction technology, and also provides strong support for technological progress and application expansion in related fields.

[0003] Hand-body association (HBA) focuses on finding the location of the corresponding person for each hand detected in an image. In fact, hand-body association has great potential impact in many applications such as video surveillance, social behavior analysis, and scene understanding.

[0004] Traditional methods for associating body parts usually use one-way reasoning techniques, such as machine learning and pattern recognition methods. These methods mainly analyze and associate based on the characteristics of specific parts. However, this method is often limited by the acquisition of local information and the limitations of one-way reasoning, and cannot fully explore the potential associations between different parts of the body. For example, in human posture recognition, one-way reasoning methods can often only associate with nearby body parts, and cannot effectively handle long-distance associations between different parts.

[0005] In recent years, people have further explored hand-body association from a new perspective of relational reasoning. First, a hand-body association network (HBANet) is proposed to match the corresponding body using the semantic and spatial position relationship of the detected hands. In addition, a large-scale challenging dataset called BodyHands is introduced as a new benchmark, which contains unconstrained images with significant overlap and occlusion between people.

[0006] Another BPJDet method extends more body part associations, such as hand-body associations, head-body associations, face-body associations, and integrates the center position offset set of body or part associations, where body part associations are embedded into a unified semantic and spatial representation.

[0007] Although these methods have improved the performance, there are still some problems with hand-body association: insufficient use of semantic information, one-way uncertainty and spatial ambiguity. Summary of the invention

[0008] In order to overcome the deficiencies of the prior art, the present invention provides a body part association method and system based on bidirectional collaborative reasoning, which solves the problems of unidirectional uncertainty and spatial ambiguity existing in the prior art.

[0009] The technical solution adopted by the present invention to solve the above problems is:

[0010] A body parts association method based on bidirectional collaborative reasoning uses a bidirectional collaborative approach to reason about the association relationship between hands and body.

[0011] As a preferred technical solution, the following steps are included:

[0012] A1, hand-body pair matching: predicting the initial association score based on the appearance features and spatial information of the detected hand and body;

[0013] A2, hand-body association reasoning: predict the predicted position offset of the body relative to the hand, and then match the predicted position offset with the calculated theoretical position offset of each body relative to the hand to obtain a hand-based association score;

[0014] A3, body-hand association reasoning: predict the predicted position offset of the hand relative to the body, and then match the predicted position offset with the calculated theoretical position offset of each hand relative to the body to obtain a body-based association score;

[0015] A4, summary: summarize the initial association score, hand-based association score, and body-based association score to obtain the final matching score;

[0016] The execution order of step A2 and step A3 is one of the following: executing step A2 first and then executing step A3, executing step A3 first and then executing step A3, or executing step A2 and step A3 simultaneously.

[0017] As a preferred technical solution, in step A1, the calculation formula of the association score of the kth hand-body pair in the initial association score S1 is:

[0018]

[0019] Where k is the number of the hand-body pair, represents the association score of the kth hand-body pair in the initial association score S1, represents the weight of the k-th hand-body pair, and the superscript T represents the transposition operation. represents the appearance features of the kth hand-body pair after hand-body pairing matching, represents the bias parameter of the k-th hand-body pair, m represents the number of hands, and n represents the number of bodies.

[0020] As a preferred technical solution, in step A2, the association score of the kth hand-body pair is The calculation formula is:

[0021]

[0022] in, represents the association score of the k-th hand-body pair, represents the predicted position offset of the i-th hand corresponding to the body, i represents the hand number, i=1,…,m, represents the theoretical position offset of the j-th body relative to the ith hand, j represents the body number, j=1,…,n, ||*|| represents the L1 paradigm, and σ1 represents the hyperparameter.

[0023] As a preferred technical solution, in step A2, hand h i With body b j Theoretical position offset between The calculation formula is:

[0024]

[0025] in, Indicates b j The horizontal coordinate of the center, Indicates h i The horizontal coordinate of the center, Indicates h i The width of Indicates b j The ordinate of the center, Indicates h i The ordinate of the center, Indicates h i high, Indicates b j The width of Indicates b j Height, h i represents the i-th hand.

[0026] As a preferred technical solution, in step A3, the k-th hand-body pair The score is calculated as follows:

[0027]

[0028] in, Indicates hand h i Relative to body b j The theoretical position offset, represents the predicted position offset of hand 1 predicted by the body-hand association, represents the predicted position offset of hand 2 predicted by the body-hand association, and σ2 represents a hyperparameter;

[0029] The expressions are:

[0030]

[0031] in, represents the horizontal coordinate of the center of hand 1 predicted by the body-hand association, The ordinate of the center of hand 1 representing the predicted body-hand association, represents the width of hand 1 predicted by the body-hand association, represents the height of hand 1 predicted by the body-hand association, represents the horizontal coordinate of the center of hand 2 predicted by the body-hand association, The ordinate of the center of hand 2 representing the body-hand association prediction, represents the width of hand 2 predicted by the body-hand association, represents the height of hand 2 predicted by the body-hand association.

[0032] As a preferred technical solution, in step A3, hand h i Relative to body b j Theoretical position offset The calculation formula is:

[0033]

[0034] in, Indicates hand h i Relative to body b j Theoretical position offset.

[0035] As a preferred technical solution, in step A4, the initial association score, the hand-based association score, and the body-based association score are aggregated through dot products.

[0036] A body part association system based on bidirectional collaborative reasoning, used to implement a body part association method based on bidirectional collaborative reasoning, includes a bidirectional association reasoning network, and the bidirectional association reasoning network includes the following modules:

[0037] Hand-body pair matching module: used to predict the initial association score based on the appearance features and spatial information of the detected hands and bodies;

[0038] Hand-body association reasoning module: used to predict the predicted position offset of the body relative to the hand, and then match the predicted position offset with the calculated theoretical position offset of each body relative to the hand to obtain a hand-based association score;

[0039] Body-hand association reasoning module: used to predict the predicted position offset of the hand relative to the body, and then match the predicted position offset with the calculated theoretical position offset of each hand relative to the body to obtain a body-based association score;

[0040] Summarization module: used to aggregate the initial association score, the hand-based association score, and the body-based association score to obtain the final matching score;

[0041] The hand-body pairing matching module, the hand-body association reasoning module, and the body-hand association reasoning module are respectively connected to the summary module.

[0042] As a preferred technical solution, the loss function of the bidirectional association reasoning network is:

[0043] L BICORE =L HPM +L HAR +L BAR (8)

[0044] Where, L BICORE represents the bidirectional collaborative reasoning association loss, L HPM represents the hand-body pair matching loss, L HAR represents the hand-body association reasoning loss, L BAR Represents body-hand association reasoning loss;

[0045] in,

[0046]

[0047]

[0048]

[0049] In the formula, represents the S1 theoretical association score of the kth opponent-body pair, t represents the dimension number, represents the t-th dimension value of the predicted body position offset of the ith hand obtained through hand-body association reasoning, The t-th dimension value representing the theoretical position offset of the body corresponding to the i-th hand, represents the smoothed L1 loss function, where L1 represents the mean absolute error. Indicates body b jThe t-th dimension value of the predicted position offset of hand 1, Indicates body b j The t-th dimension value corresponding to the theoretical position offset value of hand 1, Indicates body b j The tth dimension corresponding to the predicted position offset of hand 2, Indicates body b j The t-th dimension value corresponding to the theoretical position offset of hand 2.

[0050] Compared with the prior art, the present invention has the following beneficial effects:

[0051] (1) In order to model joint and spatial consistency, the present invention uses the appearance features and spatial information of the hand and body in HPM to predict the relevant hand-body pairs, which enhances the semantic information of the features and facilitates the subsequent association operation;

[0052] (2) Considering the potential complementarity between the hand and body of the same person, the two association reasoning modules HAR and BAR of the present invention collaborate to learn the semantic relationship between the detected hand and body in a bidirectional manner, which can significantly improve the association accuracy and alleviate spatial ambiguity and uncertainty;

[0053] (3) Based on the existing methods, the network of the present invention can overcome the lack of semantic information in unidirectional relational reasoning and the spatial ambiguity caused by huge changes or mutual occlusion. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 This is a diagram of the implementation architecture of a method for associating body parts based on bidirectional collaborative reasoning according to the present invention;

[0055] Figure 2 It is a schematic diagram of the internal structure of HPM;

[0056] Figure 3 Schematic diagram of the structure of HAR and BAR. DETAILED DESCRIPTION

[0057] The present invention will be further described in detail below in conjunction with embodiments and drawings, but the embodiments of the present invention are not limited thereto.

[0058] Example 1

[0059] like Figures 1 to 3 As shown, the present invention provides a body part association method based on bidirectional collaborative reasoning, which can be applied to the fields of human-computer interaction, video behavior analysis, group activity recognition, etc., and is used to identify and associate the correlation between different parts of the human body, thereby supporting the realization of various application scenarios.

[0060] Terminology Note:

[0061] Hand-body prediction offset: refers to the predicted position offset of the body obtained from the hand. Therefore, it represents the predicted position offset of the body relative to the hand, which is the predicted position offset of the hand-body association reasoning module.

[0062] Similarly, the body-hand prediction offset refers to the predicted position offset of the hand obtained from the body, which indicates the predicted position offset of the hand relative to the body and is the predicted position offset of the body-hand association reasoning module.

[0063] Hand-body theoretical offset: The theoretical position offset of each body relative to the hand is calculated by the formula;

[0064] Body-hand theoretical offset: The theoretical position offset of each hand relative to the body is calculated by the formula.

[0065] The specific scheme of the present invention is as follows:

[0066] 1: In order to solve the problem of insufficient semantic information caused by one-way reasoning, we propose to combine appearance features and spatial information to enhance the semantic expression of hand-body pairs.

[0067] To handle different overlaps and occlusions between multiple people, previous methods rely on one-way information transfer, such as inferring the body from the detected hand or inferring the hand from the body. However, the detected hand regions are too similar and small to provide enough semantic information to directly locate the corresponding body, which inevitably leads to mismatches between hand-body pairs.

[0068] The hands and body of the same person often have locally consistent joint patterns and specific position distributions. Modeling the hand-body relationship from both appearance features and spatial position information can enhance semantic expression.

[0069] 2: In view of the uncertainty and spatial ambiguity of one-way reasoning, a two-way collaborative approach is used to infer the relationship between the detected hand and body to alleviate the uncertainty and spatial ambiguity.

[0070] In complex multi-person scenes, people overlap with each other, causing different people's hands to be confused or blocked. Moreover, a hand may be in a similar position in multiple body bounding boxes. Relying only on a one-way predicted position offset will inevitably lead to greater uncertainty and fail to match the corresponding body.

[0071] Considering the potential complementarity between the hand and body of the same person, two relational reasoning modules are established, namely hand-body association reasoning and body-hand association reasoning. Compared with existing methods, the proposed method can alleviate the uncertainty and spatial ambiguity caused by the huge deformation of a person in the one-way reasoning method.

[0072] 1: Overview:

[0073] Given an image, all bounding boxes of hands and body are obtained through object detectors such as Mask RCNN. This method mainly consists of three modules: hand-body pair matching (HPM), hand-body association reasoning (HAR), and body-hand association reasoning (BAR).

[0074] The whole framework consists of the following aspects: First, the hand-body pair matching module (HPM) is used to predict the initial association score based on the appearance features and spatial information of the detected hands and bodies; second, in order to find the corresponding body from each detected hand, the hand-body association reasoning (HAR) module is introduced to predict the hand-body predicted offset of the body relative to the hand in a regression manner, and this offset is matched with the detected hand-body theoretical offset to obtain the hand-based association score. Conversely, starting from each body, the body-hand association reasoning (BAR) module is proposed to predict the body-hand predicted offset for all visible hands, and then the body-based association score is obtained by maximizing (the more matching, the higher the score) this predicted offset and the calculated body-hand theoretical offset. Finally, the scores of the three modules are aggregated into the Hungarian algorithm distribution to obtain the final matching score.

[0075] 2: Specific plan:

[0076] 2.1 Hand-body Pair Matching (HPM):

[0077] Intuitively, there are latent joint patterns and locally restricted spatial relationships between the hands and body of the same person. Most hands are distributed in the upper half of the corresponding body, and the distribution of the center points of all hand bounding boxes relative to the center points of the body bounding box is also consistent. Therefore, it is feasible to directly consider appearance features and spatial information to obtain the initial association score S1.

[0078] Specifically, the RGB image I is input into the target detector Mask RCNN to detect the bounding box m and n represent the number of hands and body in image I, represents 4-dimensional data, so there are m*n hand-body pairs in total. Assume h i and b j The RoI-Aligned feature is expressed as and Appearance characteristics of all hand-body pairs The appearance features of the k-th hand-body pair can be obtained by concatenation. Can be defined as

[0079] In addition, the local spatial relationship between hand-body pairs is constrained by the relative positions of their center points and bounding boxes. Given the normalized center coordinates of the bounding box of the k-th hand-body pair, denoted as and It can be obtained from the four-dimensional vector composed of the upper left and lower right vertices. Spatial features of hand-body pairs Extracted by concatenating all geometric attributes, i.e. They represent the x-coordinate distance between the center points of the hand-body pair, the y-coordinate distance between the center points, the Euclidean coordinate distance, the ratio of the inverse trigonometric function, and the area ratio of the overlapping area between the two to the hand bounding box.

[0080] Comprehensive appearance features of hand-body pair matching module and spatial information To comprehensively consider the confidence score of the opponent-body pair, we can get S1.

[0081] Specifically, in order to integrate appearance features and spatial information, After using two convolutional layers and three fully connected layers (details as Figure 2 The output is After the fully connected layer, we get the same dimension The features after combining the two are It can be defined as For hand-body pair matching, Sent to the fully connected layer to obtain refined features The calculation formula of the association score of the k-th hand-body pair in the initial association score S1 is:

[0082]

[0083] and represents the weight and bias parameters of the k-th hand-body pair, which can be learned using binary cross entropy loss.

[0084] 2.2. Bidirectional Correlation Reasoning (BICORE):

[0085] The potential complementarity between hands and their corresponding bodies has not been explored in detail in previous methods. Since a person's hand is part of their body, the hand box should be contained in the body box of the same person. Compared with the hand, the human body region is larger and contains richer semantic information. In addition, the position of the hand can be approximated between multiple body bounding boxes. Associative reasoning from body to hand can suppress the uncertainty caused by matching based on hand offsets. The basic idea is to introduce a bidirectional associative reasoning strategy to enhance semantic information while reducing spatial uncertainty and ambiguity. Therefore, this method designs two core modules, namely hand-body associative reasoning (HAR) and body-hand associative reasoning (BAR).

[0086] 2.2.1 Hand-body association reasoning (HAR):

[0087] Given the detected bounding boxes H and B, the goal of HAR is to extract i The corresponding body in B is found from the angle of . Inspired by the position density module in HBANet, in the hand-body association inference module of this method, the position offset of each detected hand corresponding to the body is used to infer the hand-body association. Specifically, according to the bounding box regression formula in Faster RCNN, the detected hand h i With body b j Position offset between You can get:

[0088]

[0089] in and h i and b j The horizontal and vertical coordinates of the center. and Respectively represent h i and b j The width and height of the .

[0090] Use b i Relative to the detected hand h i The encoded bounding box of the corresponding body is then used to detect the hand features via hand-based regression and its bounding box position to get the predicted position offset of the body relative to the hand L HAR Loss optimizes the learnable parameters by minimizing Smooth-L1. Therefore, the associated score You can match the encoded body offset by Offset from position Get the k-th hand-body association score By hand iand for each detected body b j The calculation is defined as follows:

[0091]

[0092] In the formula, ||*|| represents the L1 norm. σ1 is a hyperparameter, which is set to 5 according to experience. It means hand h i The predicted position offset of the body relative to the hand predicted by the HAR module, It is the theoretical position offset of the body relative to the hand calculated by the regression formula. Here, it is used to judge the similarity between the two, which can be understood as is the predicted value, It is a theoretical value.

[0093] 2.2.2 Body-hand association reasoning (BAR):

[0094] As mentioned before, previous methods all proceed in a one-way reasoning manner, where they reason about associations from each detected hand to match all bodies or from each detected body to match all hands. In reality, people often overlap with each other, and hands of different people are mixed or occluded in real scenes. The one-way association strategy has two disadvantages in complex multi-person scenes:

[0095] i) Due to the scale variation and mutual occlusion between the hand and the body, the detected hand region is often too small or occluded to provide enough semantic information to find the correct body;

[0096] ii) A hand may be in similar positions in multiple body bounding boxes, and relying solely on the predicted position offsets of the hand relative to all predicted bodies will inevitably lead to larger uncertainty and fail to match the corresponding bodies.

[0097] Therefore, compared with HAR, BAR focuses on identifying the corresponding hand in H from the perspective of each detected body. Similarly, all detected bodies B and hands H can be paired with each other. i Relative to body b j The theoretical body-hand offset for:

[0098]

[0099] Unlike HAR, the association reasoning from body to hand is a one-to-many matching problem, because a person has two hands. The predicted offsets of the two hands can be defined as and For each detected body b jSpecifically, an 8-dimensional offset is generated for each detected body through training to represent the left and right hands respectively, and then the body-hand predicted offset is matched with the actual detected body-hand theoretical offset to calculate the error value, thereby calculating the association score S3.

[0100] Specifically, physical characteristics and its bounding box position are sent to two convolutional layers and two fully connected layers for body-based regression training (details are as follows Figure 3 These two prediction offsets and It can be expressed as follows:

[0101]

[0102] In formula (5), represents the horizontal coordinate of the center of hand 1 predicted by the body-hand association, The ordinate of the center of hand 1 representing the predicted body-hand association, represents the width of hand 1 predicted by the body-hand association, represents the height of hand 1 predicted by the body-hand association, represents the horizontal coordinate of the center of hand 2 predicted by the body-hand association, The ordinate of the center of hand 2 representing the body-hand association prediction, represents the width of hand 2 predicted by the body-hand association, represents the height of hand 2 predicted by the body-hand association.

[0103] However, in real multi-person occlusion scenarios, some bodies may be associated with a hand. This method adopts an intuitive way to deal with this problem: by copying the offset of the hand, it is forced to set two hands with the same offset. The body-based regression parameters can be obtained by minimizing the Smooth-L1 loss (L BAR ).

[0104] After obtaining the encoding offsets of the two hands, a greedy search strategy is used to find the best matching pair of the detected body and hands. and the theoretical position offset of the hand relative to the body Maximize the matching to get the body-hand association score The kth hand-body pair The score is calculated by each detected hand h i and body b j The calculation results are:

[0105]

[0106] σ2 is a hyperparameter and is empirically set to 2.

[0107] 2.3 Hand-body pair matching:

[0108] The basic idea of ​​our method is to aggregate different association scores from the three branches by using dot products. The final association score S of the k-th hand-body pair is k The calculation formula is:

[0109]

[0110] Finally, a two-part matching strategy based on the Hungarian algorithm is introduced to assign the detected hands and bodies, where S k Used as a hand i and body b j The weight between .

[0111] The model obtains a final score of S, which is an (m,n) score matrix, where m represents the number of hands and n represents the number of bodies. The Hungarian algorithm (the algorithm for finding the maximum matching) uses the score matrix S to assign which hand corresponds to which body.

[0112] 2.4Loss function design:

[0113] The bidirectional association inference network proposed in this method optimizes L Total =L CLS +L REG +L MASK +L BICORE , trained using the MaskRCNN framework. The SGD optimizer is used to learn the network parameters. The batch size is set to 1 and the base learning rate is 0.0001. Similar to HARNet, L CLS , L REG and L MASK denote the classification, bounding box regression, and segmentation mask losses from Mask RCNN, respectively. L BICORE represents the bidirectional collaborative reasoning association loss, which is defined as:

[0114] L BICORE =L HPM +L HAR +L BAR (8)

[0115] L HPM is the loss of the hand-body pair matching module. HAR and L BAR They represent hand-body association reasoning loss and body-hand association reasoning loss respectively.

[0116] L HPM To minimize the correlation score Its Ground truth Perform hand-body pair matching. Ideally, if the k-th hand-body pair belongs to the same person, is equal to 1, otherwise it is equal to 0. Therefore, the binary cross entropy loss is used to define L HPM ,as follows:

[0117]

[0118]

[0119] L HAR and L BAR The loss is used to train the HAR and BAR modules for body offset and hand offset During the training phase, the hand i The corresponding body and hands Relative to body b j is known. Ground truth position offset and It can be obtained according to the border regression formula of Faster RCNN. HAR and L BAR The loss between the position offset of the encoded offset and the actual true value is defined by minimizing Smooth-L1, that is:

[0120]

[0121]

[0122] Finally, in L Total These modules are trained collaboratively in an end-to-end manner under the constraints of

[0123] In the formula, represents the S1 theoretical association score of the k-th opponent-body pair (if the k-th opponent-body pair belongs to the same person, it is 1, otherwise it is 0), t represents the dimension number (i.e., the index of the four-dimensional offset), t∈{x,y,w,h}, represents the t-th dimension value of the predicted body position offset of the ith hand obtained through hand-body association reasoning, The t-th dimension value representing the theoretical position offset of the body corresponding to the i-th hand, represents the smoothed L1 loss function, where L1 represents the mean absolute error. Indicates body b j The t-th dimension value of the predicted position offset of hand 1, Indicates body bj The t-th dimension value corresponding to the theoretical position offset value of hand 1, Indicates body b j The tth dimension corresponding to the predicted position offset of hand 2, Indicates body b j The t-th dimension value corresponding to the theoretical position offset of hand 2.

[0124] The present invention has the following characteristics:

[0125] 1. The invention uses appearance features and spatial features to jointly realize hand-body reasoning technology:

[0126] Traditional methods for associating body parts often rely on only a single feature information, such as appearance features or spatial features, which limits the accuracy and reliability of the association method. This method can more comprehensively capture the association between the hand and the body by combining appearance features and spatial features, thereby improving the accuracy of the association. The combined application of this feature enables this method to maintain a high recognition accuracy in complex scenes and changing postures;

[0127] 2. The present invention uses a two-way reasoning technique on the hand-body association, that is, a combination of reasoning the body from the hand and reasoning the hand from the body:

[0128] Traditional association methods usually use one-way reasoning, that is, only reasoning from the hand to the body or from the body to the hand. However, this one-way reasoning method easily ignores the interaction and feedback between the hand and the body. This method proposes a two-way reasoning mechanism, which considers both the situation of reasoning from the hand to the body and the situation of reasoning from the body to the hand, and achieves a more accurate and comprehensive association between the hand and the body through two-way collaborative reasoning. This two-way reasoning method can make full use of the information between the hand and the body and improve the accuracy and stability of the association.

[0129] 3. Application prospects of this method in multiple fields:

[0130] Whether in human-computer interaction, computer vision or medical image analysis, body part association technology has a wide range of application needs. The method proposed in this paper is not only applicable to these fields, but also can further expand its application scope by optimizing algorithms and models to meet the needs of more practical scenarios.

[0131] Example 2

[0132] like Figures 1 to 3 As shown, as a further optimization of Example 1, based on Example 1, this embodiment also includes the following technical features:

[0133] 1: Overall architecture implementation:

[0134] Figure 1 The overall framework of the body part association method for bidirectional collaborative reasoning is presented. The specific process is to input an image, and after preprocessing, such as image normalization and padding, it is sent to the object detector (e.g., Mask RCNN) to generate bounding boxes for all hands and bodies. Box samples are used to extract their appearance features. Then, the hand-body pairs are sent to three modules respectively to obtain their association scores based on different clues. The hand-body paired matching (HPM) module generates a confidence score S1 by capturing appearance and spatial features; the hand-body association reasoning (HAR) and body-hand association reasoning (BAR) obtain their respective association scores S2 and S3 through regression-based offset matching. Finally, S1, S2, and S3 are multiplied by the dot product to obtain the final matching score, and then the Hungarian matching algorithm is used to obtain the hand-body association result.

[0135] 1.1: Hand-body pairing matching implementation (HPM):

[0136] The detailed configuration of the HPM module is as follows Figure 2 As shown. The appearance feature branch starts with two convolutional layers. The input is a 512-dimensional concatenation of body and hand features, and the input structure is (512, 7, 7). After a 3*3 convolution and a 1*1 convolution, the output structure is (256, 7, 7), which is then expanded to obtain a 12544-dimensional vector, which is sent to three fully connected layers FC to obtain appearance features. The spatial branch is used to model the mutual geometric constraints between hand-body pairs. Next, a full connectivity layer upgrades the five types of geometric information to three-dimensional spatial features. The fused features are obtained by connecting them. For the specific meaning, please refer to the technical solution. Finally, the two vectors are concatenated to obtain a 128-dimensional vector, and two fully connected layers are used to directly predict the confidence score of the hand-body pair.

[0137] 1.2: Bidirectional Reasoning Implementation (BICORE):

[0138] The detailed design of HAR and BAR is as follows Figure 3 shown.

[0139] Given the detected bounding boxes H and B and their RoI-Aligned features, the goal of HAR is to find the corresponding body in B from the perspective of each detected hand. After concatenating the visual features and bounding box coordinates of each hand, hand-based regression is performed by using two fully connected layers to obtain the body offset, that is, a 4-dimensional offset is generated for each detected hand, indicating the body offset corresponding to the hand. Compared with the hand, the human body region contains richer semantic information.

[0140] In contrast, BAR focuses on identifying the corresponding hand in H from the perspective of each detected body. In the BAR module, body features and bounding box positions are fed into two convolutional layers and two fully connected layers to perform body-based regression. Since body features contain richer global information, two additional convolutional layers are used here to capture potential high-level semantic information. Similarly, for each detected body, an 8-dimensional vector is generated after the body-hand reasoning module, which represents the offset of the left and right hands corresponding to the body.

[0141] 2: Experimental Setup

[0142] 2.1 Dataset selection:

[0143] Bodyhands is a large dataset containing unconstrained images with annotated positions and correspondences of hands and bodies, specifically designed for evaluating the association between hands and bodies.

[0144] COCO-WholeBody is an extension of the COCO dataset with full body annotations.

[0145] The dataset contains 133 keypoints and 4 types of bounding boxes (person box, face box, left box, and right box) for each person in the image.

[0146] 2.2: Evaluation Metrics:

[0147] The standard VOC Average Precision (AP) metric is used to evaluate the hand detection performance.

[0148] Defined by HBANet, two core metrics are considered to measure the performance of hand-body association, namely conditional accuracy (Condition Accuracy) for body association and joint accuracy (Joint AP) for hand detection and body association.

[0149] 2.3: Implementation details:

[0150] All experiments are conducted on NVIDIA GeForce GTX 4090 based on PyTorch framework. For fair comparison, HBANet is used as our baseline, while BICORE uses the same configuration in terms of input image size and data augmentation. The SGD optimizer is used to learn the network parameters with momentum = 0.9 and the batch size is set to 1. The initial learning rate for both datasets is set to 0.0001 and gradually decayed by a factor of 10 in the 20th and 25th epochs of the Bodyhands dataset and by a factor of 10 in the 10th and 20th epochs of the COCO WholeBody dataset. For hand and body detection, the confidence threshold of the prediction is set to 0.05 and the IoU threshold of NMS is set to 0.3.

[0151] As described above, the present invention can be preferably implemented.

[0152] All features disclosed in all embodiments in this specification, or steps in all methods or processes implicitly disclosed, except for mutually exclusive features and / or steps, can be combined and / or expanded or replaced in any manner.

[0153] The above description is only a preferred embodiment of the present invention and does not limit the present invention in any form. According to the technical essence of the present invention, within the spirit and principles of the present invention, any simple modification, equivalent replacement and improvement made to the above embodiment still falls within the protection scope of the technical solution of the present invention.

Claims

1. A body part association method based on bidirectional collaborative reasoning, characterized in that: Use a two-way collaborative approach to reason about the relationship between the hand and the body; The following steps are involved: A1, hand-body pair matching: predicting the initial association score based on the appearance features and spatial information of the detected hand and body; A2, hand-body association reasoning: predict the predicted position offset of the body relative to the hand, and then match the predicted position offset with the calculated theoretical position offset of each body relative to the hand to obtain a hand-based association score; A3, body-hand association reasoning: predict the predicted position offset of the hand relative to the body, and then match the predicted position offset with the calculated theoretical position offset of each hand relative to the body to obtain a body-based association score; A4, summary: summarize the initial association score, hand-based association score, and body-based association score to obtain the final matching score; The execution order of step A2 and step A3 is one of the following: executing step A2 first and then executing step A3, executing step A3 first and then executing step A3, or executing step A2 and step A3 simultaneously; In step A1, the calculation formula of the association score of the kth hand-body pair in the initial association score S1 is: Where k is the number of the hand-body pair, represents the association score of the kth hand-body pair in the initial association score S1, represents the weight of the k-th hand-body pair, and the superscript T represents the transposition operation. represents the appearance features of the kth hand-body pair after hand-body pairing matching, represents the bias parameter of the kth hand-body pair, m represents the number of hands, n represents the number of bodies; In step A2, the association score of the k-th hand-body pair is The calculation formula is: in, represents the association score of the k-th hand-body pair, represents the predicted position offset of the i-th hand corresponding to the body, i represents the hand number, i=1,…,m, represents the theoretical position offset of the jth body relative to the ith hand, j represents the body number, j=1,…,n, ||*|| represents the L1 paradigm, and σ1 represents the hyperparameter; In step A2, hand h i With body b j Theoretical position offset between The calculation formula is: in, Indicates b j The horizontal coordinate of the center, Indicates h i The horizontal coordinate of the center, Indicates h i The width of Indicates b j The ordinate of the center, Indicates h i The ordinate of the center, Indicates h i high, Indicates b j The width of Indicates b j Height, h i represents the i-th hand; In step A3, the k-th hand-body pair The score is calculated as follows: in, Indicates hand h i Relative to body b j The theoretical position offset, represents the predicted position offset of hand 1 predicted by the body-hand association, represents the predicted position offset of hand 2 predicted by the body-hand association, and σ2 represents a hyperparameter; The expressions are: in, represents the horizontal coordinate of the center of hand 1 predicted by the body-hand association, The ordinate of the center of hand 1 representing the predicted body-hand association, represents the width of hand 1 predicted by the body-hand association, represents the height of hand 1 predicted by the body-hand association, represents the horizontal coordinate of the center of hand 2 predicted by the body-hand association, The ordinate of the center of hand 2 representing the body-hand association prediction, represents the width of hand 2 predicted by the body-hand association, represents the height of hand 2 predicted by the body-hand association; In step A3, hand h i Relative to body b j Theoretical position offset The calculation formula is: in, Indicates hand h i Relative to body b j Theoretical position offset.

2. A method for associating body parts based on bidirectional collaborative reasoning according to claim 1, characterized in that: In step A4, the initial association score, the hand-based association score, and the body-based association score are aggregated through dot products.

3. A body part association system based on bidirectional collaborative reasoning, characterized in that: A method for associating body parts based on bidirectional collaborative reasoning according to claim 1 or 2 is used to implement the method, comprising a bidirectional association reasoning network, wherein the bidirectional association reasoning network comprises the following modules: Hand-body pair matching module: used to predict the initial association score based on the appearance features and spatial information of the detected hands and bodies; Hand-body association reasoning module: used to predict the predicted position offset of the body relative to the hand, and then match the predicted position offset with the calculated theoretical position offset of each body relative to the hand to obtain a hand-based association score; Body-hand association reasoning module: used to predict the predicted position offset of the hand relative to the body, and then match the predicted position offset with the calculated theoretical position offset of each hand relative to the body to obtain a body-based association score; Summarization module: used to aggregate the initial association score, the hand-based association score, and the body-based association score to obtain the final matching score; The hand-body pairing matching module, the hand-body association reasoning module, and the body-hand association reasoning module are respectively connected to the summary module.

4. The body part association system based on bidirectional collaborative reasoning according to claim 3 is characterized in that: The loss function of the bidirectional association inference network is: L BICORE =L HPM +L HAR +L BAR (8) Where, L BICORE represents the bidirectional collaborative reasoning association loss, L HPM represents the hand-body pair matching loss, L HAR represents the hand-body association reasoning loss, L BAR Represents body-hand association reasoning loss; in, In the formula, represents the S1 theoretical association score of the kth opponent-body pair, t represents the dimension number, represents the t-th dimension value of the predicted body position offset of the ith hand obtained through hand-body association reasoning, The t-th dimension value representing the theoretical position offset of the body corresponding to the i-th hand, represents the smoothed L1 loss function, where L1 represents the mean absolute error. Indicates body b j The t-th dimension value of the predicted position offset of hand 1, Indicates body b j The t-th dimension value corresponding to the theoretical position offset value of hand 1, Indicates body b j The tth dimension corresponding to the predicted position offset of hand 2, Indicates body b j The t-th dimension value corresponding to the theoretical position offset of hand 2.

Images