Human body and human hand association method, device, equipment and storage medium

Abstract

An embodiment of the present application provides a method, apparatus, device and storage medium for associating a human body with a human hand, wherein the method comprises: acquiring an image to be detected whose content includes a human body and a human hand; determining a human body detection frame of the human body and a hand detection frame of the human hand in the image to be detected; determining an association probability between the human body detection frame and the hand detection frame; determining an external frame of the human body detection frame and the hand detection frame; and determining a degree of association between the human body and the human hand in the external frame based on key points of the hand in the external frame and the association probability.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to Singapore patent application No. 10202106806S filed with the Intellectual Property Office of Singapore on June 22, 2021, the contents of which are incorporated herein by reference in their entirety. Technical Field

[0003] The embodiments of the present application relate to the field of image processing technology, and relate to but are not limited to a method, device, equipment and storage medium for associating a human body with a human hand. Background Art

[0004] In complex scenes with many people, there are usually people blocking each other or arms intertwining. In related technologies, sensors are usually used to determine the ownership of a hand based on the key points of the human body. This cannot guarantee the accuracy of the judgment and has a high judgment cost. Summary of the invention

[0005] The embodiment of the present application provides a technical solution for the association between the human body and the human hand.

[0006] The technical solution of the embodiment of the present application is implemented as follows:

[0007] The present application provides a method for associating a human body with a human hand, the method comprising:

[0008] Acquire an image to be detected including a human body and a human hand;

[0009] In the image to be detected, determining a human body detection frame of the human body and a hand detection frame of the human hand;

[0010] Determining an association probability between the human body detection frame and the hand detection frame;

[0011] Determine the circumscribed frames of the human body detection frame and the human hand detection frame;

[0012] Based on the key points of the human hand in the circumscribed frame and the association probability, the association degree between the human body and the human hand in the circumscribed frame is determined. In this way, the association degree between the human body and the human hand in the circumscribed frame is determined by the association probability between the human body detection frame and the human hand detection frame, and the key points of the human hand in the circumscribed frame of the human body detection frame and the human hand detection frame, so that the association accuracy between the human body detection frame and the human hand detection frame in the circumscribed frame can be improved.

[0013] In some embodiments, the determining the association probability between the human body detection frame and the hand detection frame includes: determining the human body features of the human body detection frame and the hand features of the hand detection frame; and determining the association probability between the human body detection frame and the hand detection frame based on the human body features and the hand features. In this way, by determining the association probability between the human body detection frame and the hand detection frame through the human body features and the hand features, the accuracy of determining the association between the human body detection frame and the hand detection frame can be improved.

[0014] In some embodiments, the determining of the bounding boxes of the human body detection box and the human hand detection box includes: determining the human body detection box and the human hand detection box whose association probability is greater than a preset probability threshold as a target detection box pair; and determining the bounding boxes of the human body detection box and the human hand detection box in the target detection box pair. In this way, by determining the bounding boxes of the human body detection box and the human hand detection box through the association probability, the bounding boxes of the human body detection box and the human hand detection box with a high degree of association can be efficiently and accurately given.

[0015] In some embodiments, the determining of the bounding box of the human body detection frame and the hand detection frame in the target detection frame pair includes: determining a candidate frame that surrounds the target detection frame pair; and adjusting the coverage area of ​​the candidate frame based on the relative position relationship between the human body detection frame and the hand detection frame in the target detection frame pair to obtain the bounding box. In this way, by determining the bounding box of the human body detection frame and the hand detection frame through the candidate frame that can surround the human body detection frame and the hand detection frame, a more accurate bounding box can be quickly and efficiently provided, and at the same time, based on the more accurate key points of the hand in the bounding box, the accuracy of the association between the human body detection frame and the hand detection frame that form the bounding box can be improved.

[0016] In some embodiments, the determining the degree of association between the human body and the human hand in the external frame based on the human hand key points in the external frame and the association probability includes: determining at least two target human hand key points in the external frame, the at least two target human hand key points include key points of two different human hands belonging to the same person; based on the at least two target human hand key points, determining the degree of association between the human body detection frame and the human hand detection frame forming the external frame. In this way, by comprehensively detecting the human hand key points in the external frame, the association accuracy between the human body detection frame and the human hand detection frame forming the external frame can be further optimized.

[0017] In some embodiments, the determination of the correlation between the human body detection frame and the hand detection frame forming the circumscribed frame based on the at least two target hand key points includes: determining the confidence that each target hand key point is a true value point; in the case where any of the confidences is greater than a preset threshold, determining the distance from the target hand key point with a confidence greater than the preset threshold to the hand detection frame forming the circumscribed frame; based on the distance, determining the target correlation probability between the human body detection frame and the hand detection frame forming the circumscribed frame; based on the target correlation probability, determining the correlation between the human body detection frame and the hand detection frame forming the circumscribed frame. In this way, in the case where the confidence that the target hand key point is a true value point is greater than the preset threshold, the distance from the target hand key point to the hand detection frame forming the circumscribed frame can further optimize the correlation accuracy between the human body detection frame and the hand detection frame forming the circumscribed frame based on more accurate parameters.

[0018] In some embodiments, the determining the target association probability between the human body detection frame and the hand detection frame forming the circumscribed frame based on the distance includes: when the distance is less than or equal to a preset distance, determining the association probability between the human body detection frame and the hand detection frame forming the circumscribed frame to obtain the in-frame association probability; and determining the in-frame association probability as the target association probability. In this way, by judging the numerical value of the distance, the target association probability between the human body detection frame and the hand detection frame can be determined quickly and conveniently, and unnecessary calculations can be reduced.

[0019] In some embodiments, the determining the target association probability between the human body detection frame and the hand detection frame forming the circumscribed frame based on the distance includes: when the distance is greater than the preset distance, reducing the association probability between the human body detection frame and the hand detection frame forming the circumscribed frame to obtain a corrected association probability; and determining the corrected association probability as the target association probability. In this way, by judging the numerical value of the distance, the relevant parameters can be quickly and conveniently adjusted to determine the target association probability between the human body detection frame and the hand detection frame, while reducing unnecessary calculations.

[0020] In some embodiments, when the distance is greater than the preset distance, reducing the association probability between the human body detection frame and the hand detection frame forming the external frame to obtain the corrected association probability includes: when the distance is greater than the preset distance, obtaining a preset adjustment parameter for correcting the association probability; using the preset adjustment parameter to reduce the association probability between the human body detection frame and the hand detection frame forming the external frame to obtain the corrected association probability. In this way, by reducing the relevant association probability through the preset adjustment parameter to obtain the corrected association probability, the corrected association probability can be determined quickly and conveniently, thereby improving the accuracy of determining the association between the human body detection frame and the hand detection frame in the external frame.

[0021] In some embodiments, the determination of the degree of association between the human body detection frame and the hand detection frame forming the circumscribed frame based on the at least two target hand key points includes: when there is no target hand key point with a confidence level greater than a preset threshold among the at least two target hand key points, reducing the probability of association between the human body detection frame and the hand detection frame forming the circumscribed frame to obtain a corrected probability of association; and determining the corrected probability of association as the target probability of association. In this way, when there is no target hand key point with a confidence level greater than a preset threshold among the target hand key points, reducing the probability of association between the human body detection frame and the hand detection frame forming the circumscribed frame can further optimize the accuracy of association between the human body detection frame and the hand detection frame forming the circumscribed frame.

[0022] In some embodiments, the target human hand key points include wrist key points, which can provide more accurate detection parameters and thus improve detection accuracy.

[0023] In some embodiments, when multiple human body detection frames and / or multiple hand detection frames are determined from the image to be detected, the method also includes: combining each of the detected human body detection frames with each of the detected hand detection frames to obtain multiple detection frame pairs; determining the circumscribed frames of the human body detection frames and the hand detection frames, includes: determining the circumscribed frames of the human body detection frames and the hand detection frames in each of the detection frame pairs; determining the correlation between the human body and the hand in the circumscribed frame based on the hand key points in the circumscribed frame and the association probability, includes: determining the correlation between the human body and the hand in each circumscribed frame to obtain a target correlation set; in the target correlation set, determining a candidate correlation whose correlation is greater than a preset correlation threshold; based on the target circumscribed frame corresponding to the candidate correlation, determining a hand that matches the human body in the target circumscribed frame. In this way, based on the target correlation corresponding to the bounding box formed by each human body and each hand, candidate correlations that meet certain conditions are screened, and then the matching hand for each human body is determined, that is, a more accurate matching relationship between the human body and the hand can be determined based on the optimized correlation.

[0024] The present application provides a device for associating a human body with a human hand, the device comprising:

[0025] The first acquisition module is used to acquire the image to be detected, the image content of which includes the human body and the human hand:

[0026] A first determination module is used to determine a human body detection frame of the human body and a hand detection frame of the human hand in the image to be detected;

[0027] A second determination module is used to determine the association probability between the human body detection frame and the hand detection frame;

[0028] A third determination module is used to determine the circumscribed frames of the human body detection frame and the human hand detection frame;

[0029] The fourth determination module is used to determine the degree of association between the human body and the human hand in the circumscribed frame based on the key points of the human hand in the circumscribed frame and the association probability.

[0030] In some embodiments, the second determination module includes: a feature determination submodule, used to determine the human features of the human detection frame and the hand features of the hand detection frame; and an association probability determination submodule, used to determine the association probability between the human detection frame and the hand detection frame based on the human features and the hand features.

[0031] In some embodiments, the third determination module includes: a detection frame determination submodule, used to determine that the human body detection frame and the hand detection frame whose association probability is greater than a preset probability threshold are a target detection frame pair; and an external frame determination submodule, used to determine the external frame of the human body detection frame and the hand detection frame in the target detection frame pair.

[0032] In some embodiments, the external frame determination submodule includes: a candidate frame determination subunit, used to determine a candidate frame surrounding the target detection frame pair; an external frame determination subunit, based on the relative position relationship between the human body detection frame and the hand detection frame within the target detection frame pair, adjusts the coverage area of ​​the candidate frame to obtain the external frame.

[0033] In some embodiments, the fourth determination module includes: a hand key point determination submodule, used to determine at least two target hand key points within the external frame, and the at least two target hand key points include key points of two different hands belonging to the same person; a correlation determination submodule, used to determine the correlation between the human body detection frame and the hand detection frame forming the external frame based on the at least two target hand key points.

[0034] In some embodiments, the association determination submodule includes: a confidence determination subunit, used to determine the confidence that each target hand key point is a true value point; a distance determination subunit, used to determine the distance from the target hand key point with a confidence greater than the preset threshold to the hand detection frame forming the external frame when any of the confidence is greater than the preset threshold; an association probability determination subunit, used to determine the target association probability between the human body detection frame and the hand detection frame forming the external frame based on the distance; and a correlation determination subunit, used to determine the correlation between the human body detection frame and the hand detection frame forming the external frame based on the target association probability.

[0035] In some embodiments, the association probability determination subunit is also used to determine the association probability between the human body detection frame and the hand detection frame forming the external frame when the distance is less than or equal to the preset distance, to obtain the in-frame association probability; and determine the in-frame association probability as the target association probability.

[0036] In some embodiments, the association probability determination subunit is also used to reduce the association probability between the human body detection frame and the hand detection frame forming the external frame when the distance is greater than the preset distance, so as to obtain a corrected association probability; and determine the corrected association probability as the target association probability.

[0037] In some embodiments, the association probability determination subunit is also used to obtain a preset adjustment parameter for correcting the association probability when the distance is greater than the preset distance; using the preset adjustment parameter, the association probability between the human body detection frame and the hand detection frame forming the external frame is reduced to obtain the corrected association probability.

[0038] In some embodiments, the association degree determination submodule is also used to reduce the association probability between the human body detection frame and the hand detection frame forming the external frame when there is no target hand key point with a confidence level greater than a preset threshold among the at least two target hand key points, so as to obtain a corrected association probability; and determine the corrected association probability as the target association probability.

[0039] In some embodiments, the target hand key points include wrist key points.

[0040] In some embodiments, when multiple human body detection frames and / or multiple hand detection frames are determined from the image to be detected, the detection frame determination submodule is further used to combine each detected human body detection frame with each detected hand detection frame to obtain multiple detection frame pairs; the external frame determination submodule is further used to determine the external frame of the human body detection frame and the hand detection frame in each detection frame pair; the fourth determination module is further used to determine the correlation between the human body and the hand in each external frame to obtain a target correlation set; in the target correlation set, determine the candidate correlation whose correlation is greater than a preset correlation threshold; based on the target external frame corresponding to the candidate correlation, determine the hand that matches the human body in the target external frame.

[0041] An embodiment of the present application provides a computer device, wherein the computer includes a memory and a processor, wherein the memory stores computer executable instructions, and when the processor runs the computer executable instructions on the memory, the above-mentioned method of associating a human body with a human hand can be implemented.

[0042] Correspondingly, an embodiment of the present application provides a computer storage medium, on which computer executable instructions are stored. After the computer executable instructions are executed, the above-mentioned method of associating the human body and the human hand can be implemented.

[0043] The embodiments of the present application provide a method, apparatus, device and storage medium for associating a human body with a human hand. First, a human body detection frame and a hand detection frame are determined from an image to be detected whose content includes a human body and a human hand. Second, the association probability between the human body detection frame and the hand detection frame is determined, as well as an external frame of the human body detection frame and the hand detection frame is determined. Finally, based on the key points of the hand in the external frame and the association probability, the degree of association between the human body and the hand in the external frame is determined. In this way, the degree of association between the human body and the person in the external frame is determined through the association probability between the human body detection frame and the hand detection frame, as well as the key points of the hand in the external frame of the human body detection frame and the hand detection frame, so that the association accuracy between the human body detection frame and the hand detection frame in the external frame can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative work, among which:

[0045] Figure 1 A schematic diagram of a flow chart of a method for associating a human body with a human hand provided in an embodiment of the present application;

[0046] Figure 2 A schematic flow chart of another method for associating a human body with a human hand provided in an embodiment of the present application;

[0047] Figure 3A A schematic flow chart of another method for associating a human body with a human hand provided in an embodiment of the present application;

[0048] Figure 3B A schematic flow chart of another method for associating a human body with a human hand provided in an embodiment of the present application;

[0049] Figure 4 A schematic diagram of an application scenario of the method for associating a human body and a human hand provided in an embodiment of the present application;

[0050] Figure 5 A schematic diagram of the structure of a device for associating a human body and a human hand provided in an embodiment of the present application;

[0051] Figure 6 A schematic diagram of the composition structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0052] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the specific technical solution of the invention will be further described in detail below in conjunction with the drawings in the embodiments of the present application. The following embodiments are used to illustrate the embodiments of the present application, but are not used to limit the scope of the embodiments of the present application.

[0053] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0054] In the following description, the terms "first\second\third" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described here can be implemented in an order other than that illustrated or described here.

[0055] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art of the technical field of the present application. The terms used herein are only for the purpose of describing the embodiments of the present application and are not intended to limit the embodiments of the present application.

[0056] Before further describing the embodiments of the present application in detail, the nouns and terms involved in the embodiments of the present application are explained. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations.

[0057] 1) Computer Vision (CV) is a science that studies how to make machines "see". It refers to using cameras and computers to replace human eyes to identify, track and measure targets, and further perform image processing.

[0058] 2) Deep Learning (DL) is a new research direction in the field of Machine Learning (ML). It is introduced into Machine Learning to make it closer to its original goal: Artificial Intelligence (AI). DL is the inherent rules and representation levels of learning sample data. The information obtained in the learning process is of great help in the interpretation of data such as text, images and sounds. Its ultimate goal is to enable machines to have analytical learning capabilities like humans and to recognize data such as text, images and sounds.

[0059] 3) Human Keypoints Detection (HKD), also known as human posture estimation, is a relatively basic task in CV and a prerequisite for human motion recognition, behavior analysis, human-computer interaction, etc.; generally, human keypoints detection can be subdivided into single / multi-person keypoints detection and two-dimensional / three-dimensional keypoints detection. At the same time, some algorithms will track key points after completing keypoint detection, which is also called human posture tracking.

[0060] 4) Association algorithm is an important type of algorithm in data mining; among them, association reflects the knowledge of dependence or connection between an event and other events.

[0061] 5) Confidence In statistics, the confidence interval of a probability sample is an interval estimate of a population parameter of the sample; the confidence interval shows the extent to which the true value of the parameter has a certain probability of falling around the measurement result; the confidence interval gives the range of credibility of the measured value of the measured parameter, that is, the "certain probability" required earlier; this probability is called the confidence level.

[0062] The following describes an exemplary application of the device for associating a human body with a human hand provided in an embodiment of the present application. The device provided in an embodiment of the present application can be implemented as various types of user terminals such as a laptop computer with an image acquisition function, a tablet computer, a desktop computer, a camera, a mobile device (e.g., a personal digital assistant, a dedicated messaging device, a portable gaming device), and can also be implemented as a server. The following describes an exemplary application when the device is implemented as a terminal or a server.

[0063] The method can be applied to a computer device, and the functions implemented by the method can be implemented by calling a program code through a processor in the computer device. Of course, the program code can be stored in a computer storage medium. It can be seen that the computer device at least includes a processor and a storage medium.

[0064] The present application embodiment provides a method for associating a human body with a human hand, such as Figure 1 As shown, combined with Figure 1 The steps shown are explained:

[0065] Step S101, obtaining an image to be detected, the image content of which includes a human body and a human hand.

[0066] In some embodiments, the image to be detected can be obtained by an associated device of the human body and the human hand through an internal image acquisition module, or it can be an image to be detected sent by a device or equipment that can interact with information; accordingly, the image to be detected can be a color image or a grayscale image, and the human body and the human hand can be located in the foreground area, middle ground area and background area of ​​the image to be detected. In the embodiments of the present application, no limitation is imposed on the areas where the human body and the human hand are located in the image to be detected.

[0067] In some embodiments, the human body in the image to be detected refers to a human body that can present all body information in the image to be detected, or a human body that presents partial body information; when the human body presents all body information in the image to be detected, such as: head, two arms, a torso and two legs, etc.; it can also present partial body information, such as: head, one arm and two legs, etc.; the human hand in the image to be detected can refer to the left hand information, right hand information, left and right hand information, etc. presented in the image to be detected.

[0068] In some embodiments, the posture information presented by the human body in the image to be detected may be: standing, walking, sitting, etc.; the posture information presented by the human hand in the image to be detected may be: open or closed, or partially closed, etc.

[0069] In some embodiments, the picture content of the image to be detected includes but is not limited to the human body and human hands; the picture content of the image to be detected can be the human body and human hands in any scene, such as: classroom, park, office or playground, etc.; at the same time, the picture content of the image to be detected includes the number of human bodies and the number of human hands, which can be one, two or more; the relative position relationship between the human bodies, between the human hands, and between the human body and the human hands presented in the picture content of the image to be detected can be: left and right, front and back, up and down, etc.

[0070] Step S102: determining a human body detection frame and a hand detection frame of a human hand in the image to be detected.

[0071] In some embodiments, the human body detection frame is a detection frame that can include the body part of the human body in the image to be detected. Exemplarily, assuming that all the body parts of the human body can be presented in the image to be detected, the human body detection frame is the smallest rectangular frame that can surround the head, two arms, a torso and two legs of the human body; at the same time, the human body detection frame can surround the key points of the human body, such as: head, neck, shoulders, elbows, hands, hips, knees and feet. Exemplarily, assuming that part of the human body can be presented in the image to be detected, the human body detection frame is the smallest rectangular frame that can surround the head, one arm, part of the torso, etc. of the human body.

[0072] In some embodiments, the hand detection frame of a human hand may be a portion of the human hand that can be included in the image to be detected. Exemplarily, assuming that the entire hand can be presented in the image to be detected, the hand detection frame is a minimum rectangular frame that can enclose the fingertips of the human hand, the joints of each phalange, etc. Assuming that partial information of the human hand can be presented in the image to be detected, the hand detection frame is a minimum rectangular frame that can enclose part of the fingertips of the human hand and part of the joints of the phalanges, etc.

[0073] In some embodiments, the positional relationship between the human body detection frame and the human hand detection frame can be partially overlapping or adjacent, or the hand detection frame can be located inside the human body detection frame, etc.; at the same time, the size of the human body detection frame is determined based on the size of the human body presented in the image to be detected; accordingly, the size of the hand detection frame is determined based on the size of the human hand presented in the image to be detected.

[0074] In some embodiments, assuming that the image to be detected includes multiple human bodies and multiple human hands, each human body corresponds to a human body detection frame, and the human body detection frames corresponding to different human bodies may overlap, partially overlap, and be adjacent to each other. Each human hand corresponds to a hand detection frame, and the hand detection frames corresponding to different human hands may overlap, partially overlap, and be adjacent to each other; at the same time, the hand detection frames can be divided into two categories, namely: left hand detection frame and right hand detection frame.

[0075] In some possible implementations, the human body and the person in the image to be detected are respectively detected by a trained detection model to determine the corresponding human body detection frame and human hand detection frame.

[0076] In some embodiments, the human body detection frame and the hand detection frame can be generated based on the target detection algorithm. Exemplarily, in the embodiments of the present application, a target detection algorithm such as Fast Region Convolutional Neural Networks (Fast R-CNN), Faster Region Convolutional Neural Networks (Faster R-CNN) or Region-based Fully Convolutional Network (R-FCN) can be used to detect the human body and the hand in the image to be detected, and obtain the human body detection frame and the hand detection frame of the human hand; wherein, the target detection is performed using Faster R-CNN, which can be achieved through the following four basic steps, such as: candidate region generation, feature extraction, classification and position refinement; in this way, a more accurate human body detection frame and hand detection frame can be quickly and conveniently given; at the same time, the amount of calculation can be reduced due to the use of the target detection algorithm.

[0077] Step S103, determining the association probability between the human body detection frame and the hand detection frame.

[0078] In some embodiments, the association probability may refer to the degree of association between the human body detection frame and the hand detection frame or the possibility of association between the two. Here, "association" means that there is an ownership relationship between the entity objects represented by the two, such as there is an ownership relationship between the hand and the human body. Among them, the degree of association or the possibility of association can be determined based on the color features, texture features, shape features, and spatial relationship features between the human body detection frame and the hand detection frame. Exemplarily, in an embodiment of the present application, if the association probability between the human body detection frame and the hand detection frame is high, it means that the degree of association of the relevant feature information between the human body detection frame and the hand detection frame is high; similarly, if the association probability between the human body detection frame and the hand detection frame is low, it means that the degree of association of the relevant feature information between the human body detection frame and the hand detection frame is low.

[0079] In some embodiments, the association probability between the human body detection frame and the hand detection frame can be determined by a relevant association algorithm. Accordingly, the association probability can be expressed by a decimal, a fraction, or a percentage, and the value of the association probability can be 90%, 0.75, or the like.

[0080] In a possible implementation, determining the association probability between the human body detection frame and the hand detection frame by acquiring the human body features of the human body detection frame and the hand features of the hand detection frame can be achieved through the following process:

[0081] The first step is to determine the human features of the human detection frame and the hand features of the hand detection frame.

[0082] In some embodiments, human body features may refer to the color features, texture features, shape features, and spatial relationship features of a human body in a human body detection frame. For example, it may refer to the posture information, relative position information, and color information of a human body in a human body detection frame; human hand features may refer to the color features, texture features, shape features, and spatial relationship features of a human hand in a human hand detection frame. For example, it may refer to the posture information, relative position information, and color information of a human hand in a human hand detection frame.

[0083] In some embodiments, the human features of the human body detection frame and the human hand features of the human hand detection frame can be extracted and determined by the feature extraction model. In this way, more accurate human features in the human body detection frame and human hand features in the human hand detection frame can be given, and then the accuracy of determining the association between the human body detection frame and the human hand detection frame can be improved based on the human body features and human hand features with higher accuracy.

[0084] The second step is to determine the association probability between the human body detection frame and the hand detection frame based on the human body features and hand features.

[0085] In some embodiments, the human body features and the human hand features are associated through an association algorithm to determine the association probability between the human body detection frame and the human hand detection frame. Exemplarily, the human body features and the human hand features may be matched with feature information to determine the association probability between the human body detection frame and the human hand detection frame. In this way, by matching information between the human body features and the human hand features to determine the association probability between the human body detection frame and the human hand detection frame, the accuracy of determining the association between the human body detection frame and the human hand detection frame can be improved.

[0086] In some possible embodiments, a feature extraction model is used to perform feature extraction on the human body detection frame and the hand detection frame respectively to obtain corresponding human body features and hand features; in this way, the human body features and hand features can be extracted quickly and accurately, and then the association probability between the human body detection frame and the hand detection frame is determined based on the extracted human body features and hand features; further improving the accuracy of determining the degree of association between the human body detection frame and the hand detection frame.

[0087] Step S104, determining the circumscribed frames of the human body detection frame and the hand detection frame.

[0088] In some embodiments, the external frame may refer to the smallest rectangular frame that can enclose the human body detection frame and the hand detection frame, and may also refer to the smallest rectangular frame that can enclose the human body detection frame and the hand detection frame, with the probability of association between the human body detection frame and the hand detection frame being greater than a preset probability threshold; exemplarily, the external frame may be determined based on the human body detection frame, the hand detection frame, and the relative position relationship between the human body detection frame and the hand detection frame; wherein, the information within the external frame includes but is not limited to: feature information of the human body detection frame, feature information of the hand detection frame; and at the same time, the hand detection frame within the external frame may be at any position within the external frame.

[0089] In some embodiments, the human hand detection frame includes a right hand detection frame and a left hand detection frame. The human body detection frame can form two external frames with the right hand detection frame and the left hand detection frame respectively; wherein, there may be position overlap or complete overlap between the two external frames.

[0090] In some embodiments, when there are multiple human body detection frames and multiple hand detection frames, each human body detection frame and each hand detection frame form an external frame. The size of the external frame is determined based on the size of the human body detection frame and the size of the hand detection frame; and the position of the external frame in the image to be detected is determined based on the human body detection frame and the hand detection.

[0091] Step S105 , determining the correlation between the human body and the human hand in the circumscribed frame based on the key points of the human hand in the circumscribed frame and the correlation probability.

[0092] In some embodiments, the key points of the human hand in the external frame may refer to the center information of the human hand palm or the key points of the human hand wrist in the external frame. Exemplarily, it is assumed that the key points of the human hand refer to the key points of the human hand wrist, that is, the information of the human hand wrist node; wherein the key points of the human hand wrist may include: the confidence that the key points of the human hand wrist are true value points, the distance from the key points of the human hand wrist to the human hand detection frame forming the external frame; for example, including: the confidence that the key points of the left wrist are true value points, the confidence that the key points of the right wrist are true value points, the distance from the key points of the right wrist to the human hand detection frame forming the external frame, the distance from the key points of the left wrist to the human hand detection frame forming the external frame, etc. The human hand wrist refers to the part of the human body connecting the palm and the forearm.

[0093] In some embodiments, the degree of association may refer to the degree of association between a human body, i.e., a human body detection frame, and a human hand, i.e., a human hand detection frame, within an external frame; wherein the degree of association may be represented by a numerical value, such as 0 to 100. Exemplarily, it is assumed that there is no association between the human body detection frame and the human hand detection frame, i.e., the numerical value corresponding to the degree of association is 0; it is assumed that there is a high association between the human body detection frame and the human hand detection frame, i.e., the numerical value corresponding to the degree of association may be represented by 80 to 100.

[0094] In some embodiments, the correlation between the human body and the human hand in the external frame is determined by judging the confidence that the human hand key point in the external frame is a true value point and the distance between the human hand key point and the human hand detection frame forming the external frame. Exemplarily, the correlation between the human body and the human hand in the external frame can also be expressed as match and mismatch, or high correlation and general correlation.

[0095] The method for associating a human body and a human hand provided in an embodiment of the present application first determines a human body detection frame and a hand detection frame of a human body from an image to be detected whose screen content includes a human body and a human hand; secondly, determines the association probability between the human body detection frame and the hand detection frame, and determines an external frame of the human body detection frame and the hand detection frame; finally, based on the key points of the hand in the external frame and the association probability, determines the degree of association between the human body and the hand in the external frame; in this way, by determining the degree of association between the human body and the person in the external frame through the association probability between the human body detection frame and the hand detection frame, and the key points of the hand in the external frame of the human body detection frame and the hand detection frame, the association accuracy of the human body detection frame and the hand detection frame in the external frame can be improved.

[0096] In a possible implementation, a target detection frame pair consisting of a human detection frame and a hand detection frame whose association probability is greater than a preset probability threshold is determined, and at the same time, a candidate frame that can surround the target detection frame pair is determined as the circumscribed frame of the human detection frame and the hand detection frame, that is, the above step S104, which can be implemented by the following process, such as Figure 2As shown, Figure 2 A flowchart of another method for associating a human body with a human hand provided in an embodiment of the present application, combined with Figure 1 and Figure 2 The steps shown are explained below:

[0097] Step S141, determining that a human body detection frame and a hand detection frame whose association probability is greater than a preset probability threshold are a target detection frame pair.

[0098] In some embodiments, the preset probability threshold is determined based on actual applications, or based on the attribute information of the image to be detected, or based on the human features of the human detection frame and the hand features of the hand detection frame; wherein the preset probability threshold may be 90%, 85%, and 88%, etc., and accordingly, the preset probability threshold may be expressed as a fraction, a decimal, and a percentage. wherein each target detection frame pair is composed of a human detection frame and a hand detection frame, and the association probability between the human detection frame and the hand detection frame is greater than the preset probability threshold.

[0099] Step S142, determining the circumscribed frames of the human body detection frame and the hand detection frame in the target detection frame pair.

[0100] In some embodiments, the bounding box may refer to a minimum rectangular box that can completely surround the human body detection box and the hand detection box in the target detection box pair. When the preset probability threshold is 90%, the probability of association between the human body detection box and the hand detection box in the bounding box is greater than 90%.

[0101] In some embodiments, a target detection frame pair formed by a human body detection frame and a hand detection frame is obtained through association probability screening, and then the external frames of the human body detection frame and the hand detection frame in the target detection frame pair are obtained; in this way, the external frames of the human body detection frame and the hand detection frame with a high degree of correlation can be given efficiently and accurately.

[0102] In some possible implementations, determining the bounding box of the human body detection box and the hand detection box in the target detection box pair, that is, the above step S142, can be implemented by the following process. Figure 1 and Figure 2 The steps shown are explained below:

[0103] The first step is to determine the candidate boxes that enclose the target detection box pair.

[0104] In some embodiments, when the association probability is greater than a preset probability threshold, the human body detection frame and the hand detection frame are combined based on the relative position relationship between the human body detection frame and the hand detection frame to obtain a candidate frame that can surround the target detection frame pair, that is, the human body detection frame and the hand detection frame; wherein the size and shape of the candidate frame are not limited in any way in the embodiments of the present application.

[0105] In some embodiments, when multiple human body detection frames and multiple hand detection frames are determined, and the associated probability is greater than a preset probability threshold, the first human body detection frame and the first right hand detection frame are combined to obtain a first target detection frame pair, and a first candidate frame that can surround the first target detection frame pair is determined. In some possible implementations, the first human body detection frame and the second right hand detection frame are combined to obtain a second target detection frame pair, and a second candidate frame that can surround the second target detection frame pair is determined; the second human body detection frame and the third right hand detection frame are combined to obtain a third target detection frame pair, and a third candidate frame that can surround the third target detection frame pair is determined, etc.; the second human body detection frame and the second left hand detection frame are combined to obtain a fourth target detection frame pair, and a fourth candidate frame that can surround the fourth target detection frame is determined, etc.

[0106] In the second step, based on the relative position relationship between the human body detection frame and the hand detection frame within the target detection frame, the coverage area of ​​the candidate frame is adjusted to obtain the external frame.

[0107] In some embodiments, based on the relative positional relationship between the human body detection frame and the hand detection frame within the target detection frame pair, the coverage area of ​​the candidate frame is adjusted to obtain an external frame; wherein the external frame is the smallest rectangular frame that can surround the human body detection frame and the hand detection frame. Exemplarily, adjusting the coverage area of ​​the candidate frame includes but is not limited to: reducing the coverage area of ​​the candidate frame according to the shape of the candidate frame or reducing it according to the human body information inside the candidate frame. In this way, a more accurate external frame can be given quickly and efficiently, so as to improve the accuracy of the association between the human body detection frame and the hand detection frame that form the external frame based on more accurate hand key points within the external frame.

[0108] Based on the above description, it can be known that the association probability between the human body detection frame and the hand detection frame in the external frame is greater than the preset probability threshold; therefore, there is a certain correlation between the human body and the hand in the external frame, that is, there will be no external frame whose value corresponding to the degree of correlation between the human body detection frame and the hand detection frame is 0.

[0109] In some possible implementations, by determining at least two target hand key points and the association probability within the bounding box, the association degree between the human body detection box and the hand detection box forming the bounding box is determined, that is, the above step S105 can be implemented by the following process, and then combined with Figure 1 and Figure 2 The steps shown are explained below:

[0110] Step S151, determining at least two target hand key points within the bounding box.

[0111] The at least two target hand key points include key points of two different hands belonging to the same person.

[0112] In some embodiments, at least two target hand key points in the external frame may refer to key points of two different hands belonging to the same person in the external frame; wherein the key points of two different hands of the same person may be located at any position in the external frame. Thus, by providing at least two target hand key points in the external frame, the error that may occur when only detecting the key points of one person to determine the correlation degree can be reduced.

[0113] In some embodiments, at least two target human hand key points within the circumscribed frame may be determined using a human key point detection model.

[0114] In a possible implementation, the target hand key points include wrist key points.

[0115] In some embodiments, when the target hand key point includes a wrist key point, the at least two target hand key points in the circumscribed frame may be wrist key points of two different hands in the circumscribed frame that belong to the same person. In the following embodiments of the present application, wrist key points are used to refer to target hand key points.

[0116] In some embodiments, the key points of the human hand are determined as the key points of the wrist, so that more accurate detection parameters can be given, thereby improving the detection accuracy.

[0117] Step S152, based on at least two target hand key points, determining the correlation between the human body detection frame and the hand detection frame forming the circumscribed frame.

[0118] In some embodiments, based on the key points of different wrists, such as the key points of the right wrist and the key points of the left wrist, the correlation between the human body detection frame and the human hand detection frame bracket forming the external frame is jointly determined. In this way, under the premise of not determining whether the human hand detection frame forming the external frame is the left hand detection frame or the right hand detection frame, the key points of the left wrist and the key points of the right wrist are detected at the same time, which can reduce the error that may occur when only detecting the key points of one wrist to determine the relevant correlation degree, that is, the correlation accuracy between the human body detection frame and the human hand detection frame forming the external frame can be further optimized by comprehensively detecting the key points of the wrists in the external frame.

[0119] In some embodiments, determining the correlation between the human body detection frame and the hand detection frame forming the circumscribed frame through at least two target hand key points, i.e., key points of different wrists, i.e., step S152, can be implemented by the following steps: Figure 3A and 3B As shown, Figure 3A and 3B A flowchart of another method for associating a human body with a human hand provided in an embodiment of the present application, combined with Figures 1 to 3B The steps shown are explained below:

[0120] Step S201, determining the confidence level of each target hand key point as a true value point.

[0121] In some embodiments, the confidence level of each wrist key point as a true value point is determined; wherein the true value point may refer to information that each wrist key point in the external frame is a true value, i.e., a real value. Exemplarily, it may refer to that under certain conditions, the true value point is an actual value objectively existing in the external frame after measurement; wherein the confidence level of each wrist key point as a true value point may be obtained from the image to be detected through an image feature advance model or a target detection algorithm.

[0122] In some embodiments, the confidence level of each wrist key point being a true point may refer to the confidence level of each wrist key point being a true point. For example, the confidence interval of the right wrist key point being a true point at the confidence level of 0.95 is (0.5, 0.6), and then there is a 95% probability that the right wrist key point is a true point falling between 50% and 60%.

[0123] In some embodiments, the confidence that each target hand key point in the circumscribed frame is a true value point may refer to the confidence that the left wrist key point is a true value point and the confidence that the left wrist key point is a true value point.

[0124] In a possible implementation, the target association probability between the human body detection frame and the hand detection frame forming the outer frame is determined by judging the confidence that the target hand key point is a true value point. It can be divided into two cases:

[0125] In the case where any confidence level is greater than a preset threshold, the following steps S202, S203 and S206 are performed, and combined Figure 1 and Figure 2 ,as well as Figure 3A The steps shown are described below.

[0126] In the case that there is no confidence greater than the preset threshold, the following steps S204 to S206 are performed, combined with Figure 1 and Figure 2 ,as well as Figure 3B The steps shown are described below.

[0127] Step S202, when any confidence level is greater than a preset threshold, determine the distance from the target hand key point with a confidence level greater than the preset threshold to the hand detection frame forming the circumscribed frame.

[0128] In some embodiments, the expression form of the preset threshold and the expression form of the confidence have a mapping relationship. The preset threshold can be set according to the requirements of the clarity and / or recognizability of the wrist key point. For example, the higher the requirements for the clarity and recognizability of the wrist key point, the larger the preset threshold is set. Exemplarily, the preset threshold can be set to 0.85. When the confidence is greater than the preset threshold, the distance from the wrist key point with a confidence greater than the preset threshold to the hand detection frame forming the external frame is determined; wherein, the distance can be the distance from each wrist key point to the center point of the hand detection frame forming the external frame, the distance from each wrist key point to any bone node of the hand in the hand detection frame forming the external frame, or the distance from each wrist key point to the center point of the hand in the hand detection frame forming the external frame; it can also be the distance from each wrist key point to the boundary area of ​​the hand detection frame forming the external frame.

[0129] In a possible implementation, the first distance and the second distance from different wrist key points to the hand detection frame forming the circumscribed frame may be determined, and then the correlation between the human body detection frame and the hand detection frame forming the circumscribed frame may be determined based on the first distance and / or the second distance. That is, in the above step S202, when the different wrist key points include: a first key point of a first wrist and a second key point of a second wrist, and the distances include: the first distance and the second distance, the following process may be used to implement the process:

[0130] In the first step, when the confidence level corresponding to the first key point is greater than a preset threshold, a first distance from the first key point to a hand detection frame forming an outer frame is determined.

[0131] In some embodiments, when the confidence level corresponding to the first key point of the first wrist is greater than a preset threshold, a first distance from the first key point to the human hand detection frame forming the external frame is determined. The first wrist may refer to any one of the left and right wrists of the human body in the external frame. If the first wrist is the left wrist, the second wrist corresponds to the right wrist; if the first wrist is the right wrist, the second wrist corresponds to the left wrist.

[0132] In some embodiments, assuming that the first wrist refers to the right wrist, when the confidence corresponding to the first key point of the right wrist is greater than a preset threshold, the first distance can be determined as: the distance from the first key point of the right wrist to the boundary area of ​​the human hand detection frame forming the external frame, or the distance from the first key point of the right wrist to the center point of the human hand detection frame forming the external frame, or the distance from the first key point of the right wrist to any bone node of the human hand forming the human hand detection frame forming the external frame, or the distance from the first key point of the right wrist to the boundary area of ​​the human hand detection frame forming the external frame, etc.

[0133] In the second step, when the confidence level corresponding to the second key point is greater than a preset threshold, a second distance from the second key point to the hand detection frame forming the circumscribed frame is determined.

[0134] In some embodiments, when the confidence level corresponding to the second key point of the second wrist is greater than a preset threshold, the second distance from the second key point to the hand detection frame forming the external frame is determined. Exemplarily, assuming that the second wrist is the left wrist, the second distance may be: the distance from the second key point of the left wrist to the boundary area of ​​the hand detection frame forming the external frame, or the distance from the second key point of the left wrist to the center point of the hand detection frame forming the external frame, or the distance from the second key point of the left wrist to any bone node of the hand forming the hand detection frame forming the external frame, or the distance from the second key point of the left wrist to the boundary area of ​​the hand detection frame forming the external frame, etc. The first distance and the second distance may be the same or different.

[0135] In some embodiments, by respectively determining the confidences and related distances corresponding to the first key point of the first wrist and the second key point of the second wrist, the accuracy of determining the target association probability based on the distance can be further improved under the premise of providing more accurate distance reference parameters.

[0136] Step S203 : determining the target association probability between the human body detection frame and the hand detection frame forming the circumscribed frame based on the distance.

[0137] In some embodiments, based on the distance from each wrist key point to the hand detection frame forming the circumscribed frame, the target association probability between the human body detection frame and the hand detection frame forming the circumscribed frame is determined. The target association probability may be the same as or different from the association probability between the human body detection frame and the hand detection frame forming the circumscribed frame.

[0138] In some embodiments, the association probability between the human body detection frame and the hand detection frame forming the circumscribed frame is adjusted based on the distance; the association probability between the human body detection frame and the hand detection frame forming the circumscribed frame may be increased, decreased or maintained, thereby obtaining the target association probability.

[0139] In a possible implementation, by judging the distance, determining the target association probability between the human body detection frame and the hand detection frame of the circumscribed frame, that is, the above step S203, can be implemented by dividing into the following two cases based on the distance:

[0140] In the first case, the following steps S231 and S232 can be used to implement the above:

[0141] Step S231, when the distance is less than or equal to the preset distance, determining the association probability between the human body detection frame and the hand detection frame forming the circumscribed frame, and obtaining the in-frame association probability.

[0142] In some embodiments, by judging the numerical value of the distance, when the distance is less than or equal to the preset distance, the corresponding in-frame association probability is obtained by calculating the association probability between the human body detection frame and the hand detection frame forming the external frame. Exemplarily, the in-frame association probability may be the original association probability between the human body detection frame and the hand detection frame forming the external frame, and the in-frame association probability may also be obtained by adaptively increasing the original association probability between the human body detection frame and the hand detection frame forming the external frame.

[0143] Step S232, determining the association probability within the frame as the target association probability.

[0144] In some embodiments, the in-frame association probability is determined as the target association probability, and the in-frame association probability can be directly determined as the target association probability. Exemplarily, it is assumed that the in-frame association probability is the original association probability of the human body detection frame and the hand detection frame that form the circumscribed frame, that is, the in-frame association probability is greater than the preset probability threshold; it can also be an adaptive adjustment of the in-frame association probability, such as: increasing the in-frame association probability, and determining the increased in-frame association probability as the target association probability. In this way, by judging the numerical value of the distance, the target association probability between the human body detection frame and the hand detection frame can be quickly and conveniently determined, and unnecessary calculations can be reduced.

[0145] The second situation can be achieved by following steps S233 and S234:

[0146] Step S233 , when the distance is greater than a preset distance, reducing the association probability between the human body detection frame and the hand detection frame forming the circumscribed frame to obtain a corrected association probability.

[0147] In some embodiments, by judging the value of the distance, when the distance is greater than a preset distance, the association probability between the human body detection frame and the human hand detection frame forming the circumscribed frame is reduced. Exemplarily, when the distance is greater than a preset distance, the association probability between the human body detection frame and the human hand detection frame forming the circumscribed frame is reduced to obtain a corrected association probability; wherein, the original association probability between the human body detection frame and the human hand detection frame forming the circumscribed frame may be reduced, that is, the corrected association probability may be the same as, less than, or greater than the preset association probability.

[0148] In some embodiments, the parameters on which the association probability between the human body detection frame and the hand detection frame forming the circumscribed frame is corrected may be fixed or dynamically changed based on the absolute difference between the distance and the preset distance.

[0149] In a possible implementation, the association probability between the human body detection frame and the hand detection frame forming the circumscribed frame may be corrected by preset adjustment parameters, that is, the above step S233 may be implemented by the following steps:

[0150] The first step is to obtain a preset adjustment parameter for correcting the association probability when the distance is greater than a preset distance.

[0151] In some embodiments, when the distance is greater than a preset distance, a preset adjustment parameter for correcting the association probability is obtained. The preset adjustment parameter may be a constant, and its corresponding presentation form may be a decimal, a fraction, or a percentage. The presentation form of the numerical value corresponding to the preset adjustment parameter corresponds to the numerical presentation form corresponding to the association probability; the preset adjustment parameter may be 1, 0.4, or 5%, etc.

[0152] In some embodiments, the preset adjustment parameters for correcting the association probability can be jointly determined based on the distance from the left hand wrist key point and / or the right hand wrist key point to the hand detection frame forming the external frame; can also be determined based on the attribute information of the external frame; can also be determined based on the size parameters of the hand detection frame forming the external frame; can also be jointly determined based on the size of the hand inside the hand detection frame forming the external frame, and the size of the human body inside the human body detection frame forming the external frame, etc.

[0153] In the second step, the preset adjustment parameters are used to reduce the association probability between the human body detection frame and the hand detection frame that form the external frame to obtain the corrected association probability.

[0154] In some embodiments, a preset adjustment parameter is used to reduce the association probability between the human body detection frame and the hand detection frame forming the circumscribed frame. Exemplarily, the association probability between the human body detection frame and the hand detection frame forming the circumscribed frame can be subtracted from the preset adjustment parameter; wherein the preset adjustment parameter can be any positive number; or the association probability between the human body detection frame and the hand detection frame forming the circumscribed frame can be subtracted from the preset adjustment parameter to obtain a corrected association probability. wherein the preset adjustment parameter can be any positive integer.

[0155] The corrected association probability is lower than the original association probability between the human body detection frame and the hand detection frame that form the external frame. The corrected association probability may be greater than the preset association probability, or may be less than or equal to the preset association probability. In this way, by reducing the relevant association probability through the preset adjustment parameters, the corrected association probability is obtained, which can achieve fast and convenient determination of the corrected association probability, thereby improving the accuracy of determining the association between the human body detection frame and the hand detection frame in the external frame.

[0156] Step S234, determining the corrected association probability as the target association probability.

[0157] In some embodiments, the corrected association probability may be directly determined as the target association probability. In this way, by judging the numerical value of the distance, the relevant parameters can be quickly and conveniently adjusted to determine the target association probability between the human body detection frame and the hand detection frame, while reducing unnecessary calculations.

[0158] Step S204, when there is no target hand key point with a confidence level greater than a preset threshold among at least two target hand key points, reducing the association probability between the human body detection frame and the hand detection frame forming the circumscribed frame to obtain a corrected association probability.

[0159] In some embodiments, when there are no key points with a confidence level greater than a preset threshold among the right hand wrist key points and the left hand wrist key points, the association probability between the human body detection frame and the hand detection frame forming the external frame is reduced or decreased to obtain a corrected association probability.

[0160] Step S205: determining the corrected association probability as the target association probability.

[0161] In some embodiments, when it is determined that there are no target hand key points with a confidence level greater than a preset threshold among the target hand key points, reducing the association probability between the human body detection frame and the hand detection frame forming the external frame can further optimize the association accuracy between the human body detection frame and the hand detection frame forming the external frame.

[0162] Step S206, determining the degree of association between the human body detection frame and the hand detection frame forming the circumscribed frame based on the target association probability.

[0163] In some embodiments, based on the target association probability, that is, based on the uncorrected in-frame association probability or the corrected association probability, the association degree between the human body detection frame and the hand detection frame forming the external frame is determined; for example, the target association probability is determined to be 80%, that is, the corresponding association degree between the human body detection frame and the hand detection frame forming the external frame is relatively strong; the target association probability is determined to be 60%, that is, the corresponding association degree between the human body detection frame and the hand detection frame forming the external frame is relatively weak; the target association probability is determined to be 90%, that is, the corresponding association degree between the human body detection frame and the hand detection frame forming the external frame is very strong. In this way, when the confidence that the key point of the wrist is a true value point is greater than a preset threshold, the association probability between the human body detection frame and the hand detection frame in the external frame is adaptively adjusted through the distance from the key point of the wrist to the hand detection frame forming the external frame, or when the confidence is less than or equal to the preset threshold, the association probability between the human body detection frame and the hand detection frame forming the external frame is reduced, which can further optimize the association accuracy between the human body detection frame and the hand detection frame forming the external frame.

[0164] In a possible implementation, when multiple human body detection frames and / or multiple hand detection frames are determined from the image to be detected, the above method for associating human body and human hand can also determine the matching relationship between human body and human hand in the relevant external frame through the following process:

[0165] In the first step, each detected human body detection frame is combined with each detected hand detection frame to obtain multiple detection frame pairs.

[0166] In some embodiments, it is assumed that two human body detection frames and three hand detection frames are determined from the image to be detected, and based on each human body detection frame and each hand detection frame detected, six detection frame pairs are obtained, such as: human body detection frame 1-hand detection frame 1, human body detection frame 1-hand detection frame 2, human body detection frame 1-hand detection frame 3, human body detection frame 2-hand detection frame 1, human body detection frame 2-hand detection frame 2, human body detection frame 2-hand detection frame 3.

[0167] The second step is to determine the human body detection frame in each detection frame pair and the circumscribed frame of each hand detection frame.

[0168] In some embodiments, the second step corresponds to step S104 provided in the above method for associating a human body with a human hand, namely, determining the circumscribed frames of the human body detection frame and the human hand detection frame.

[0169] In some embodiments, the circumscribed frame of the human body detection frame and each human hand detection frame in each detection frame pair is determined; wherein each circumscribed frame can be obtained by combining the human body detection frame of each human body and the human hand detection frame of each human hand. Exemplarily, the human body detection frame of each human body and the human hand detection frame of each human hand can be determined first; secondly, the association probability of each human body detection frame and each human hand detection frame is calculated, and when the association probability is greater than a preset probability threshold, the human body detection frame and the human hand detection frame that meet certain conditions are combined to form a circumscribed frame.

[0170] In some embodiments, the specific size and shape of the circumscribed frame of each human body detection frame and each human hand detection frame can be jointly determined based on the human body detection frame of each human body and the human hand detection frame of each human hand.

[0171] In some embodiments, each human body may correspond to multiple external frames. For example, each human body detection frame forms two corresponding external frames with at least two hand detection frames, such as a right hand detection frame and a left hand detection frame; at the same time, each hand detection frame may correspond to one external frame, or two or more external frames.

[0172] After the first and second steps are completed, step S105 provided by the above method for associating a human body with a human hand may be executed, that is, the following steps:

[0173] The third step is to determine the correlation between the human body and the hand in each external frame and obtain the target correlation set.

[0174] In some embodiments, the degree of association between the human body and the human hand in each external frame is calculated and determined, wherein the target association degree can be determined based on different human hand key points in each external frame. It can also be determined based on different wrist key points in each external frame. Exemplarily, it can be determined based on the confidence that the key point of each wrist in each external frame is a true value point, and the distance from the key point of each wrist to the human hand detection frame forming the external frame. The details of the relevant implementation scheme can be referred to the description of determining the target association probability of the external frame in the above embodiment, which will not be repeated here.

[0175] The fourth step is to determine, in the target relevance set, candidate relevances whose relevances are greater than a preset relevance threshold.

[0176] In some embodiments, in the target association set, the associations are screened by a preset association threshold. The associations in the target association set can be screened based on the preset association threshold to determine the corresponding candidate associations; or the associations in the target association set can be screened by parameters lower than the preset association threshold to determine the corresponding candidate associations.

[0177] The fifth step is to determine the human hand that matches the human body in the target bounding box based on the target bounding box corresponding to the candidate association degree.

[0178] In some embodiments, based on the target external bounding box corresponding to the candidate correlation degree; wherein the target external bounding box may include each human body and two corresponding hand detection boxes, such as: two external bounding boxes formed by a left hand detection box and a right hand detection box; or each human body and multiple corresponding hand detection boxes, such as: multiple external bounding boxes formed by multiple left hand detection boxes and / or multiple right hand detection boxes. In this way, based on the target correlation degree corresponding to the external bounding box formed by each human body and each hand, the candidate correlation degree that meets certain conditions is screened, and then the matching human hand of each human body is determined, that is, it is possible to determine a more accurate matching relationship between human bodies and human hands based on the optimized correlation degree.

[0179] In some embodiments, there are three target external frames of the first human body A, which are the A1 external frame formed by the first human body A and the first hand, and the corresponding candidate association degrees are respectively: 90%; the A2 external frame formed by the first human body A and the second hand, and the corresponding candidate association degrees are respectively: 99%; the A3 external frame formed by the first human body and the third hand, and the corresponding candidate association degrees are respectively 70%; there are four target external frames of the second human body B, which are the B1 external frame formed by the second human body B and the first hand, and the corresponding candidate association degrees are respectively: 60%; the B2 external frame formed by the second human body B and the second hand, and the corresponding candidate association degrees are respectively: 70%; the B3 external frame formed by the second human body B and the fourth hand, and the corresponding candidate association degrees are respectively 99%; the B4 external frame formed by the first human body B and the fifth hand, and the corresponding candidate association degrees are respectively 95%; the embodiment of the present application is based on the greedy algorithm or the maximum weight matching (Kuhn and Munkres, KM) algorithm etc. calculate the above parameters to determine the hands that match each human body. For example, the calculation result may be: the hand that matches the first human body A is the second hand, and the hands that match the second human body B are the fourth hand and the fifth hand.

[0180] The above-mentioned method of associating the human body and the human hand is described below in conjunction with a specific embodiment. However, it is worth noting that this specific embodiment is only for better illustrating the embodiment of the present application and does not constitute an improper limitation on the embodiment of the present application.

[0181] In complex scenes with a large number of people, such as in game venues, there is usually a phenomenon of occlusion between people or arms interlacing with each other. At the same time, due to the large number of people in the field of view, the corresponding arm crossing situation is often more complicated; the related technology uses the detection association algorithm of the human body and the human hand to predict the association probability of each pair of human bodies and human hands, or uses the human body key points to detect the hand key points of each human body, and combines other sensors to determine the ownership of the human hand; it is difficult to solve the situation of reaching out and crossing arms by using the human body and the human hand detection association algorithm; among them, the human body key point detection model corresponding to the human body and the human hand detection association algorithm is very dependent on the accuracy of the human body detection frame. If the human body detection frame is inaccurate and the hand is not included, the human body key point detection model will output an incorrect result. Moreover, using the human body key points to detect the hand key points of each human body, it is impossible to obtain the position and size of the entire human hand. At the same time, it is necessary to combine other sensors to determine the ownership of the human hand, which cannot guarantee the accuracy of the judgment and the judgment cost is high.

[0182] Based on this, the embodiment of the present application provides a method for associating a human body with a human hand. In a gaming venue, the method for associating a human body with a human hand provided in the embodiment of the present application can be used to provide a more accurate matching relationship between a human body and a human hand; Figure 4 , which is a schematic diagram of an application scenario of the method for associating a human body and a human hand provided in an embodiment of the present application; Figure 4 The method of associating each human body and each human hand in the image can be achieved by the following steps:

[0183] The first step is to use the detection model to predict Figure 4 All human detection frames and hand detection frames in Figure 4 As shown in , multiple human body detection frames with facial information are given, and the human body detection frame is body, the human hand detection frame is hand, the human face detection frame is face, and the playing card detection frame is chip. For example, 401, 402, 403, and 404 are human bodies in the image to be detected, and 405, 406, and 407 are human hands.

[0184] In the second step, the association probability between the human body detection frame of each human body and the hand detection frame of each hand determined in the first step is calculated.

[0185] The third step is to determine the external frames of the human body detection frame and the hand detection frame whose association probability is greater than the preset probability threshold.

[0186] In the fourth step, the obtained external frame is input into the human body key point model for detection to obtain information of at least two target hand key points in each external frame, such as: the confidence that each target hand key point is a true value point, and the distance from the target hand key point whose confidence is greater than the preset threshold to the hand detection frame forming the external frame.

[0187] The fifth step is to take an external frame as an example for explanation. The wrist key points inside the external frame, where the wrist key points can be the left wrist key points and the right wrist key points belonging to the same person, respectively. If the confidence of the right wrist key points and the left wrist key points as true value points is greater than the preset threshold, x and y are used to represent the right wrist as the true value point and the left wrist as the true value point, respectively. When the confidence of any corresponding x and y is greater than a preset threshold thres_kp, the distance from x and y to the hand detection frame forming the external frame is determined; when the distance is less than or equal to the preset distance, no correction is made to the association probability between the human body detection frame and the hand detection frame forming the external frame, and the uncorrected association probability is obtained. When the distance is greater than the preset distance, the association probability between the human body detection frame and the hand detection frame forming the external frame is reduced to obtain a reduced association probability; at the same time, when the confidence of any corresponding x and y is not greater than a preset threshold thres_kp, the association probability between the human body detection frame and the hand detection frame forming the external frame is reduced to obtain a reduced association probability. At the same time, as Figure 4 As shown, the confidence parameters corresponding to each detection box are given, such as: face 1.00, hand1.00, and hand 0.97.

[0188] In the sixth step, the human body detection frame and the hand detection frame in the external frame corresponding to the reduced association probability and the uncorrected association probability are matched by a greedy algorithm or a KM algorithm to obtain the hand corresponding to each human body.

[0189] In this way, in the game venue, the correlation between the human body and the human being forming the external frame is determined through the key points of the human hand in the external frame and the correlation probability between the human body detection frame and the human hand detection frame, which can greatly improve the correlation accuracy between the human body and the human hand.

[0190] In a specific example, the image to be detected is an image collected in a game place, wherein the human body is a player's human body and the hand is a player's hand. Through the above-mentioned first to sixth steps, first, the association probability between the human body detection frame of each player's human body and the hand detection frame of each player's hand is determined. Secondly, when the association probability is greater than the preset probability threshold, the human body detection frame of the player's human body and the hand detection frame of the player's hand are combined to generate an external frame. Secondly, by comparing the confidence of the hand key points in the external frame, such as the wrist key points as the true value points with the preset threshold, when the confidence is greater than the preset threshold, the distance from the key points of each wrist to the hand detection frame forming the external frame is determined; and based on the distance, the association probability between the human body detection frame and the hand detection frame in the external frame is adaptively adjusted, or when the confidence is less than or equal to the preset threshold, the association probability between the human body detection frame and the hand detection frame forming the external frame is reduced. Finally, the player's hand corresponding to each player's human body is determined based on the association probability, that is, the pairing of the player's human body and the player's hand is achieved. In this way, the hands and bodies of players in complex gaming venues can be matched, and the process of players issuing game coins or making compensations during the game can be monitored more effectively.

[0191] In an embodiment of the present application, the degree of association between the human body and the hand in the external frame is determined by the key points of the human hand in the external frame and the association probability between the human body detection frame and the hand detection frame that form the external frame. In this way, the facilities for obtaining the specific position information of the human hand can be simplified; at the same time, the association accuracy between the human body detection frame and the hand detection frame in the external frame can be improved.

[0192] Based on the above embodiments, the present application provides a device for associating a human body with a human hand. Figure 5 A schematic diagram of the structure of a human body and human hand association device provided in an embodiment of the present application, such as Figure 5 As shown, the human body and human hand association device 500 includes:

[0193] A first acquisition module 510 is used to acquire an image to be detected whose contents include a human body and a human hand;

[0194] A first determination module 520 is used to determine a human body detection frame of the human body and a hand detection frame of the human hand in the image to be detected;

[0195] A second determination module 530 is used to determine the association probability between the human body detection frame and the hand detection frame;

[0196] A third determination module 540 is used to determine the circumscribed frames of the human body detection frame and the human hand detection frame;

[0197] The fourth determination module 550 is used to determine the degree of association between the human body and the human hand in the circumscribed frame based on the key points of the human hand in the circumscribed frame and the association probability.

[0198] In some embodiments, the second determination module 530 includes:

[0199] A feature determination submodule 531 is used to determine the human features of the human body detection frame and the hand features of the hand detection frame;

[0200] The association probability determination submodule 532 is used to determine the association probability between the human body detection frame and the human hand detection frame based on the human body features and the human hand features.

[0201] In some embodiments, the third determination module 540 includes:

[0202] A detection frame determination submodule 541 is used to determine that the human body detection frame and the hand detection frame whose association probability is greater than a preset probability threshold are a target detection frame pair;

[0203] The circumscribed frame determination submodule 542 is used to determine the circumscribed frames of the human body detection frame and the hand detection frame in the target detection frame pair.

[0204] In some embodiments, the external frame determination submodule 542 includes:

[0205] A candidate box determination subunit 5421 is used to determine a candidate box surrounding the target detection box pair;

[0206] The circumscribed frame determination subunit 5422 is used to adjust the coverage area of ​​the candidate frame based on the relative position relationship between the human body detection frame and the hand detection frame within the target detection frame pair to obtain the circumscribed frame.

[0207] In some embodiments, the fourth determination module 550 includes:

[0208] A hand key point determination submodule 551 is used to determine at least two target hand key points within the circumscribed frame, wherein the at least two target hand key points include key points of two different hands belonging to the same person;

[0209] The correlation determination submodule 552 is used to determine the correlation between the human body detection frame and the hand detection frame forming the circumscribed frame based on the at least two target hand key points.

[0210] In some embodiments, the association degree determination submodule 552 includes:

[0211] A confidence determination subunit 5521 is used to determine the confidence that each target hand key point is a true value point;

[0212] The distance determination subunit 5522 is used to determine the distance from the target hand key point with the confidence greater than the preset threshold to the hand detection frame forming the circumscribed frame when any of the confidences is greater than the preset threshold;

[0213] An association probability determination subunit 5523, configured to determine, based on the distance, a target association probability between a human body detection frame and a hand detection frame forming the circumscribed frame;

[0214] The association degree determination subunit 5524 is used to determine the association degree between the human body detection frame and the hand detection frame forming the circumscribed frame based on the target association probability.

[0215] In some embodiments, the association probability determination subunit 5523 is also used to determine the association probability between the human body detection frame and the hand detection frame forming the external frame when the distance is less than or equal to the preset distance, and obtain the in-frame association probability; and determine the in-frame association probability as the target association probability.

[0216] In some embodiments, the association probability determination subunit 5523 is also used to reduce the association probability between the human body detection frame and the hand detection frame forming the external frame when the distance is greater than the preset distance, so as to obtain a corrected association probability; and determine the corrected association probability as the target association probability.

[0217] In some embodiments, the association probability determination subunit 5523 is also used to obtain a preset adjustment parameter for correcting the association probability when the distance is greater than the preset distance; using the preset adjustment parameter, the association probability between the human body detection frame and the hand detection frame forming the external frame is reduced to obtain the corrected association probability.

[0218] In some embodiments, the association probability determination subunit 5523 is also used to reduce the association probability between the human body detection frame and the hand detection frame forming the external frame when there is no target hand key point with a confidence level greater than a preset threshold among the at least two target hand key points, so as to obtain a corrected association probability; and determine the corrected association probability as the target association probability.

[0219] In some embodiments, the target hand key points include wrist key points.

[0220] In some embodiments, when a plurality of human body detection frames and / or a plurality of human hand detection frames are determined from the image to be detected,

[0221] The detection frame determination submodule 541 is further used to combine each of the detected human body detection frames with each of the detected human hand detection frames to obtain a plurality of detection frame pairs;

[0222] The circumscribed frame determining submodule 542 is further used to determine the circumscribed frames of the human body detection frame and the human hand detection frame in each of the detection frame pairs;

[0223] The fourth determination module 550 is further used to determine the correlation between the human body and the human hand in each of the external bounding boxes to obtain a target correlation set; in the target correlation set, determine a candidate correlation whose correlation is greater than a preset correlation threshold; based on the target external bounding box corresponding to the candidate correlation, determine a human hand that matches the human body in the target external bounding box.

[0224] It should be noted here that the description of the above device embodiment is similar to the description of the above method embodiment and has similar beneficial effects as the method embodiment. For technical details not disclosed in the device embodiment of the present application, please refer to the description of the method embodiment of the present application for understanding.

[0225] It should be noted that in the embodiments of the present application, if the above-mentioned method of associating the human body with the human hand is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application can be essentially or partly reflected in the form of a software product that contributes to the relevant technology. The computer software product is stored in a storage medium, including several instructions to enable an electronic device (which can be a smart phone with a camera, a tablet computer, etc.) to execute all or part of the methods described in each embodiment of the embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a U disk, a mobile hard disk, a read-only memory (ROM), a magnetic disk or an optical disk. In this way, the embodiments of the present application are not limited to any specific combination of hardware and software.

[0226] Based on the same technical concept, an embodiment of the present application provides a computer device for implementing the method for associating a human body and a human hand recorded in the above method embodiment. Figure 6 A schematic diagram of the structure of a computer device provided in an embodiment of the present application is shown in FIG. Figure 6As shown, the computer device 600 includes: a processor 601, at least one communication bus, a communication interface 602, at least one external communication interface and a memory 603. The communication interface 602 is configured to realize the connection and communication between these components. The communication interface 602 may include a display screen, and the external communication interface may include a standard wired interface and a wireless interface. The processor 601 is configured to execute the human body and human hand association program in the memory to implement the human body and human hand association method provided in the above embodiment.

[0227] Correspondingly, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the method for associating a human body with a human hand described in any of the above embodiments is implemented.

[0228] Correspondingly, in an embodiment of the present application, a chip is also provided, which includes a programmable logic circuit and / or program instructions. When the chip is running, it is used to implement the method of associating the human body and the human hand as described in any of the above embodiments.

[0229] Correspondingly, in an embodiment of the present application, a computer program product is also provided. When the computer program product is executed by a processor of an electronic device, it is used to implement the method of associating the human body and the human hand described in any of the above embodiments.

[0230] The description of the above embodiments of the device for associating the human body with the human hand, the computer device, and the storage medium is similar to the description of the above method embodiment, and has similar technical descriptions and beneficial effects as the corresponding method embodiment. Due to space limitations, the above method embodiment can be referred to, so it will not be repeated here. For technical details not disclosed in the embodiments of the device for associating the human body with the human hand, the computer device, and the storage medium in the embodiments of the present application, please refer to the description of the method embodiment of the present application for understanding.

[0231] It should be understood that the "one embodiment" or "one embodiment" mentioned throughout the specification means that the specific features, structures or characteristics related to the embodiment are included in at least one embodiment of the embodiment of the present application. Therefore, "in one embodiment" or "in one embodiment" appearing throughout the specification may not necessarily refer to the same embodiment. In addition, these specific features, structures or characteristics can be combined in one or more embodiments in any suitable manner. It should be understood that in various embodiments of the embodiment of the present application, the size of the sequence number of the above-mentioned processes does not mean the order of execution, and the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present application. The above-mentioned sequence number of the embodiment of the present application is only for description and does not represent the advantages and disadvantages of the embodiment. It should be noted that in this article, the term "include", "include" or any other variant thereof is intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements that are not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of more restrictions, the elements defined by the sentence "include one..." do not exclude the existence of other identical elements in the process, method, article or device including the element.

[0232] In several embodiments provided in the embodiments of the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored, or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0233] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units; they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0234] In addition, all functional units in the embodiments of the present application can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the above integrated unit can be implemented in the form of hardware or in the form of hardware plus software functional units. It can be understood by ordinary technicians in this field that all or part of the steps of implementing the above method embodiments can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above method embodiments; and the aforementioned storage medium includes: mobile storage devices, read-only memory (ROM), disks or optical disks, etc. Various media that can store program codes.

[0235] Alternatively, if the above-mentioned integrated unit of the embodiment of the present application is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the methods described in each embodiment of the embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROMs, magnetic disks, or optical disks. The above is only a specific implementation method of the embodiment of the present application, but the protection scope of the embodiment of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or replacements within the technical scope disclosed in the embodiment of the present application, which should be covered within the protection scope of the embodiment of the present application. Therefore, the protection scope of the embodiment of the present application should be based on the protection scope of the claims.

Claims

1. A method for associating a human body with a human hand, the method comprising: Acquire an image to be detected including a human body and a human hand; In the image to be detected, determining a human body detection frame of the human body and a hand detection frame of the human hand; Determining an association probability between the human body detection frame and the hand detection frame; Determine the circumscribed frames of the human body detection frame and the human hand detection frame; The circumscribed frame is a minimum rectangular frame surrounding the human body detection frame and the hand detection frame; Determining the degree of association between the human body and the human hand in the circumscribed frame based on the key points of the human hand in the circumscribed frame and the association probability; Determining the circumscribed frames of the human body detection frame and the human hand detection frame; Determining the degree of association between the human body and the human hand in the circumscribed frame based on the key points of the human hand in the circumscribed frame and the association probability includes: Determine a bounding box surrounding the human body detection box and the hand detection box based on the association probability; Determine the confidence level that the key points of the human hand are the true value points; When the confidence level is greater than a preset threshold, determining a distance from the key point of the human hand to a human hand detection frame forming the circumscribed frame; Based on the distance, the association probability is adjusted to determine a target association probability between a human body detection frame and a hand detection frame of the circumscribed frame; Based on the target association probability, a degree of association between a human body detection frame and a hand detection frame forming the circumscribed frame is determined.

2. The method according to claim 1, wherein: The determining the association probability between the human body detection frame and the hand detection frame includes: Determine the human body features of the human body detection frame and the hand features of the hand detection frame; Based on the human body features and the human hand features, an association probability between the human body detection frame and the human hand detection frame is determined.

3. The method according to claim 1 or 2, wherein: The determining of a bounding box surrounding the human body detection box and the hand detection box based on the association probability includes: Determine a human body detection frame and a hand detection frame whose association probability is greater than a preset probability threshold as a target detection frame pair; Determine the circumscribed frames of the human body detection frame and the hand detection frame in the target detection frame pair.

4. The method according to claim 3, wherein: The determining of the circumscribed frames of the human body detection frame and the human hand detection frame in the target detection frame pair includes: Determine a candidate frame surrounding the target detection frame pair; Based on the relative position relationship between the human body detection frame and the hand detection frame within the target detection frame pair, the coverage area of ​​the candidate frame is adjusted to obtain the circumscribed frame.

5. The method according to claim 1, wherein: The confidence level of determining that the key point of the human hand is a true value point includes: Determine at least two target human hand key points within the circumscribed frame, wherein the at least two target human hand key points include key points of two different human hands belonging to the same person; Determine the confidence level that each target hand keypoint is a true value point.

6. The method according to claim 1, wherein: The step of adjusting the association probability based on the distance to determine the target association probability between the human body detection frame and the hand detection frame of the circumscribed frame includes: When the distance is less than or equal to the preset distance, determining an association probability between a human body detection frame and a hand detection frame forming the circumscribed frame, and obtaining an in-frame association probability; The in-frame association probability is determined as the target association probability.

7. The method according to claim 6, wherein: The step of adjusting the association probability based on the distance to determine the target association probability between the human body detection frame and the hand detection frame of the circumscribed frame further includes: When the distance is greater than the preset distance, reducing the association probability between the human body detection frame and the hand detection frame forming the circumscribed frame to obtain a corrected association probability; The corrected association probability is determined as the target association probability.

8. The method according to claim 7, wherein: When the distance is greater than the preset distance, reducing the association probability between the human body detection frame and the hand detection frame forming the circumscribed frame to obtain a corrected association probability, includes: When the distance is greater than the preset distance, obtaining a preset adjustment parameter for correcting the association probability; The preset adjustment parameters are used to reduce the association probability between the human body detection frame and the hand detection frame forming the circumscribed frame, thereby obtaining the corrected association probability.

9. The method according to claim 5, further comprising: When there is no target hand key point whose confidence level is greater than a preset threshold value among the at least two target hand key points, reducing the association probability between the human body detection frame and the hand detection frame forming the circumscribed frame to obtain a corrected association probability; The corrected association probability is determined as the target association probability.

10. The method according to claim 5, wherein: The target hand key points include wrist key points.

11. The method according to any one of claims 1, 2, 4 to 10, wherein: In the case where a plurality of human body detection frames and / or a plurality of human hand detection frames are determined from the image to be detected, the method further includes: Combining each of the detected human body detection frames with each of the detected human hand detection frames to obtain a plurality of detection frame pairs; The determining the circumscribed frames of the human body detection frame and the human hand detection frame comprises: determining the circumscribed frames of the human body detection frame and the human hand detection frame in each of the detection frame pairs; The determining the degree of association between the human body and the human hand in the circumscribed frame based on the key points of the human hand in the circumscribed frame and the association probability includes: Determine the correlation between the human body and the human hand in each of the external frames to obtain a target correlation set; In the target relevance set, determining a candidate relevance whose relevance is greater than a preset relevance threshold; Based on the target bounding box corresponding to the candidate association degree, a human hand matching the human body in the target bounding box is determined.

12. A device for associating a human body with a human hand, the device comprising: The first acquisition module is used to acquire the image to be detected, the image content of which includes the human body and the human hand: A first determination module is used to determine a human body detection frame of the human body and a hand detection frame of the human hand in the image to be detected; A second determination module is used to determine the association probability between the human body detection frame and the hand detection frame; A third determination module is used to determine the circumscribed frames of the human body detection frame and the human hand detection frame; The circumscribed frame is a minimum rectangular frame surrounding the human body detection frame and the hand detection frame; A fourth determination module, configured to determine the degree of association between the human body and the human hand in the circumscribed frame based on the key points of the human hand in the circumscribed frame and the association probability; The second determination module is further used to determine a bounding box surrounding the human body detection box and the human hand detection box based on the association probability; The fourth determination module is further used to determine the confidence that the human hand key point is a true value point; when the confidence is greater than a preset threshold, determine the distance from the human hand key point to the human hand detection frame forming the circumscribed frame; based on the distance, adjust the association probability to determine the target association probability between the human body detection frame and the human hand detection frame of the circumscribed frame; Based on the target association probability, a degree of association between a human body detection frame and a hand detection frame forming the circumscribed frame is determined.

13. The device according to claim 12, wherein: The second determining module includes: A feature determination submodule, used to determine the human features of the human body detection frame and the hand features of the hand detection frame; The association probability determination submodule is used to determine the association probability between the human body detection frame and the human hand detection frame based on the human body features and the human hand features.

14. The device according to claim 12 or 13, wherein: The third determining module comprises: A detection frame determination submodule, used to determine that the human body detection frame and the hand detection frame whose association probability is greater than a preset probability threshold are a target detection frame pair; The circumscribed frame determination submodule is used to determine the circumscribed frames of the human body detection frame and the hand detection frame in the target detection frame pair.

15. The device according to claim 14, wherein: The external frame determination submodule includes: A candidate box determination subunit, used to determine a candidate box surrounding the target detection box pair; The circumscribed frame determining subunit adjusts the coverage area of ​​the candidate frame based on the relative position relationship between the human body detection frame and the hand detection frame within the target detection frame pair to obtain the circumscribed frame.

16. The device according to claim 12, wherein: The fourth determining module includes: A hand key point determination submodule, used to determine at least two target hand key points within the external frame, wherein the at least two target hand key points include key points of two different hands belonging to the same person; The confidence determination subunit is used to determine the confidence that each target hand key point is a true value point.

17. A computer device, comprising a memory and a processor, wherein the memory stores computer executable instructions, and when the processor runs the computer executable instructions on the memory, it can implement the method for associating a human body and a human hand as described in any one of claims 1 to 11.

18. A computer storage medium having computer executable instructions stored thereon, which, after being executed, can implement the method for associating a human body with a human hand as described in any one of claims 1 to 11.

Images