Gesture posture determination method and device, equipment, medium and product

Abstract

The invention provides a gesture posture determination method and device, equipment, a medium and a product. According to one specific embodiment, the method comprises the steps that first position information of a first joint point and second position information of a second joint point of a hand in a target image are obtained, the first joint point is a joint point between a middle phalanx and a proximal phalanx, and the second joint point is a joint point between a distal phalanx and the middle phalanx; constructing a first position matrix corresponding to the first joint point and a second position matrix corresponding to the second joint point based on the first position information and the second position information; calculating a cosine similarity value between the first position matrix and the second position matrix; and determining a gesture posture corresponding to the target image based on the cosine similarity value. According to the scheme, the matrix vector of the fingertip (the second joint point) is calculated by taking the finger joint point (the first joint point) as the reference coordinate point, and the final gesture posture is matched through the cosine similarity of the two joint points, so that the accuracy of the recognition result of the finger posture is improved.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to methods, devices, equipment, media and products for determining hand gestures. Background Art

[0002] Hand gesture recognition refers to accurately identifying the position of the key points of a hand gesture from an image. Hand gesture recognition has been widely used and developed in human-computer interaction fields such as games, movies, virtual reality (VR), augmented reality (AR), and mixed reality (MR).

[0003] Most of the existing hand gesture estimation methods currently use the recognition and calculation of images obtained by depth cameras or multi-cameras to obtain the 3D full joint point information of the hand in the image, and then calculate the hand gesture based on the 3D full joint point information. However, the vectors used in the above calculation process are all based on the hand anchor space, which will lead to excessive differences in the joint directions (fingertip joint vectors) of the fingertips during the gesture change process, and inaccurate similarity calculation results, which will lead to inaccurate determined hand gestures. Summary of the invention

[0004] The present disclosure provides a method, device, equipment, medium and product for determining a hand gesture posture, which can improve the recognition efficiency of hand gesture posture and the accuracy of the results.

[0005] According to a first aspect of the present disclosure, a method for determining a gesture posture is provided, comprising: obtaining first position information of a first joint point of a hand and second position information of a second joint point in a target image, wherein the first joint point is a joint point between a middle phalanx and a proximal phalanx, and the second joint point is a joint point between a distal phalanx and a middle phalanx; based on the first position information and the second position information, constructing a first position matrix corresponding to the first joint point and a second position matrix corresponding to the second joint point; calculating a cosine similarity value between the first position matrix and the second position matrix; and determining the gesture posture corresponding to the target image based on the cosine similarity value.

[0006] According to a second aspect of the present disclosure, a device for determining a gesture posture is provided, comprising an acquisition module, a construction module, a calculation module and a determination module, wherein: the acquisition module is configured to acquire first position information of a first joint point of a hand and second position information of a second joint point in a target image, wherein the first joint point is a joint point between a middle phalanx and a proximal phalanx, and the second joint point is a joint point between a distal phalanx and a middle phalanx; the construction module is configured to construct a first position matrix corresponding to the first joint point and a second position matrix corresponding to the second joint point based on the first position information and the second position information; the calculation module is configured to calculate a cosine similarity value between the first position matrix and the second position matrix; the determination module is configured to determine the gesture posture corresponding to the target image based on the cosine similarity value.

[0007] According to a third aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method provided by the first aspect.

[0008] According to a fourth aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, where the computer instructions are used to cause a computer to execute the method provided in the first aspect.

[0009] According to a fifth aspect of the present disclosure, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the computer program implements the method provided according to the first aspect.

[0010] According to the method for determining gesture posture provided by the present invention, the method changes the coordinate system with the wrist joint point as the origin to a coordinate system with the first joint point (finger joint point) as the reference coordinate point, that is, the matrix vector of the fingertip (second joint point) is calculated with the finger joint point as the reference coordinate point, and the cosine similarity between the two joint points is calculated. The final gesture posture is matched through the cosine similarity, thereby improving the accuracy of the finger posture recognition results.

[0011] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0013] Figure 1 There are three types of hand gesture posture diagrams;

[0014] Figure 2is an exemplary system architecture diagram to which the method for determining a gesture posture of the present disclosure can be applied;

[0015] Figure 3 is a schematic diagram of a first embodiment of a method for determining a gesture posture according to the present disclosure;

[0016] Figure 4a This is a diagram of the bone structure of the hand;

[0017] Figure 4b This is the hand skeleton tracking result image;

[0018] Figure 5 is a schematic diagram of a second embodiment of a method for determining a hand gesture posture according to the present disclosure;

[0019] Figure 6 is a schematic diagram of a third embodiment of a method for determining a gesture posture according to the present disclosure;

[0020] Figure 7 is a schematic diagram of an implementation scenario of the method for determining a hand gesture posture of the present disclosure;

[0021] Figure 8 is a schematic diagram of an embodiment of a device for determining a gesture posture according to the present disclosure;

[0022] Fig. 9 It is a block diagram of an electronic device used to implement the method for determining the gesture posture of the embodiment of the present disclosure. DETAILED DESCRIPTION

[0023] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0024] Gesture recognition has been widely used and developed in the fields of human-computer interaction such as games, movies, virtual reality, augmented reality, and mixed reality. For example, it is applied to the VisionOS system, which is an AR glasses system that uses the cosine similarity algorithm to calculate the similarity of gestures to determine matching gestures.

[0025] The cosine similarity algorithm is very effective in calculating gesture similarity, especially when dealing with complex gestures. Since each person's hand size and joint complexity are different, simple joint distance calculation may not be accurate enough, so cosine similarity is more applicable. The calculation of cosine similarity is based on the dot product of vectors. After normalization, it can better reflect the degree of "parallelism" between two vectors.

[0026] However, the VisionOS system calculates cosine similarity based on the hand anchor space matrix. This will cause the fingertip joint directions (fingertip joint vectors) to differ too much during the potential change process, and the similarity calculation results will be inaccurate, resulting in a mismatch between the recognized gesture posture and the desired gesture.

[0027] For reference Figure 1 , Figure 1 Three hand gestures are shown, which may be three adjacent frame images or any three frame images. Figure 1 From left to right, the fingers are fully extended (i.e. not bent), half bent, and fully bent. In each gesture diagram, the black dot is the origin of the coordinate system, the green and red arrows are in the direction of the x-axis and y-axis, and the blue arrow indicates the direction of the fingertip. Figure 1 In the figure, the first finger is not bent and has no similarity with the second and third fingers. However, the second finger is half bent and the third finger is fully bent. Both gestures belong to the state of finger bending, but the degree of bending is different, so they have certain similarities. But relative to the coordinate origin (the black dot in the figure), the fingertip vectors are almost opposite in direction, resulting in: their cosine similarity results are close to 0. It can be seen that when the cosine similarity of the fingertips is calculated with the wrist as the origin, the calculation result does not match the actual situation.

[0028] The present disclosure provides a method for determining a hand gesture posture, which changes the coordinate system with the wrist joint point as the origin to a coordinate system with the first joint point (finger joint point) as the reference coordinate point, that is, the matrix vector of the fingertip (second joint point) is calculated with the finger joint point as the reference coordinate point, and the cosine similarity between the two joint points is calculated. The final hand gesture posture is matched through the cosine similarity, thereby improving the accuracy of the finger posture recognition result.

[0029] It should be pointed out that in the technical solution of the present disclosure, the acquisition, storage and application of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0030] Figure 2 An exemplary system architecture 200 is shown to which an embodiment of a method or apparatus for determining a gesture posture of the present disclosure may be applied.

[0031] like Figure 2 As shown, system architecture 200 may include terminal device 201, network 202 and server 203. Network 202 is used to provide a communication link between terminal device 201 and server 203, and may include various connection types, such as wired communication link, wireless communication link or optical fiber cable.

[0032] The user can use the terminal device 201 to interact with the server 203 through the network 202 to receive or send information, etc. Various client applications can be installed on the terminal device 201, such as social applications, human-computer interaction applications, leisure or entertainment applications, etc.

[0033] The terminal device 201 can be hardware or software. When the terminal device 201 is hardware, it can be various electronic devices, including but not limited to independent electronic devices such as smart phones, tablet computers, laptop computers and desktop computers. When the terminal device 201 is software, it can be installed in the above electronic devices. It can be implemented as multiple software or software modules, or as a single software or software module. No specific limitation is made here.

[0034] The server 203 may be hardware or software. When the server 203 is hardware, it may be implemented as a distributed server cluster consisting of multiple servers, or it may be implemented as a single server. When the server 203 is software, it may be implemented as multiple software or software modules (for example, for providing distributed services), or it may be implemented as a single software or software module. No specific limitation is made here.

[0035] The method for determining the gesture posture provided in the embodiment of the present disclosure is generally executed by the server 203 , and accordingly, the device for determining the gesture posture is generally disposed in the server 203 .

[0036] It should be noted that Figure 2 The number of terminal devices 201, networks 202 and servers 203 in the embodiment is only for illustration. Any number of terminal devices 201, networks 202 and servers 203 may be provided according to implementation requirements.

[0037] Figure 3 A process 300 of an embodiment of a method for determining a gesture posture according to the present disclosure is shown, referring to Figure 3 As shown, the method for determining the gesture posture includes the following steps:

[0038] Step S301, obtaining first position information of a first joint point and second position information of a second joint point of a hand in a target image.

[0039] In the embodiment of the present disclosure, the execution subject of the method for determining the gesture posture, for example Figure 2The server 203 shown will obtain the first position information of the first joint point of the hand in the target image and the second position information of the second joint point, wherein the first joint point is the joint point between the middle phalanx and the proximal phalanx, and the second joint point is the joint point between the distal phalanx and the middle phalanx.

[0040] Here, the execution subject will first obtain a target image, which may be an RGB image, a grayscale image, a fisheye image, etc., which is not limited in the disclosed embodiment. The obtained target image is an image containing a hand, and the execution subject will first determine the image portion corresponding to the hand from the target image.

[0041] Since the hand has multiple joints, these joints are distributed on the palm and fingers. Figure 4a , Figure 4a A schematic diagram of the hand skeleton is shown, which includes the proximal phalanx, the middle phalanx and the distal phalanx for the index finger, the middle finger, the ring finger and the little finger. The first joint point in this embodiment is the joint point between the middle phalanx and the proximal phalanx, and the second joint point is the joint point between the distal phalanx and the middle phalanx, that is, the first joint point is the parent joint of the second joint point.

[0042] The execution subject will recognize the hand image to determine the skeletal joints of the hand, and establish a three-dimensional coordinate system with the wrist as the coordinate origin to generate the position coordinates of each joint point. The execution subject will obtain the position coordinates of the first joint point and the position coordinates of the second joint point.

[0043] It should be noted that the first joint point and the second joint point belong to the same finger, and position information of multiple pairs of the first joint point and the second joint point can be obtained here.

[0044] In some implementations of the embodiments of the present disclosure, the method for determining the above-mentioned gesture posture also includes: tracking the hand skeleton of the target image; and identifying various joints of the hand from the target image based on the result of the hand skeleton tracking.

[0045] In this implementation, after acquiring the target image, the execution subject will determine the hand part in the target image and use a skeleton tracking algorithm to perform skeleton tracking on it, thereby identifying multiple joints of the hand based on the skeleton tracking results of the hand.

[0046] For further reference Figure 4b , Figure 4bThe hand skeleton tracking result diagram is shown. The above execution subject will establish a three-dimensional coordinate system with the wrist joint point (0,25) as the coordinate origin, thereby generating the position coordinates of each joint point. The above execution subject will obtain the position coordinates of the first joint point (which can be 7,12,17,22) and the position coordinates of the second joint point (correspondingly, it can be 8,13,18,23).

[0047] Therefore, the various joints of the hand are identified through the hand skeleton tracking algorithm to generate the position information of each joint point. Identifying the various joint points of the hand is the basis for subsequent hand posture matching.

[0048] Step S302: constructing a first position matrix corresponding to the first joint point and a second position matrix corresponding to the second joint point based on the first position information and the second position information.

[0049] In the disclosed embodiment, the execution subject will construct a first position matrix corresponding to the first joint point and a second position matrix corresponding to the second joint point based on the first position information and the second position information. After identifying the various joints of the hand, a three-dimensional coordinate system will be established based on the wrist joint point. The first position information and the second position information are the position information of the first joint point and the second joint point in the three-dimensional coordinate system. The above-mentioned execution subject will convert the first position information and the second position information into matrices, namely the first position matrix and the second position matrix, respectively, to calculate the cosine similarity based on the first position matrix and the second position matrix.

[0050] In some implementations of the embodiments of the present disclosure, the first position information includes: the direction vector of the first joint point in the directions of each coordinate axis in the three-dimensional space; the second position information includes: the direction vector of the second joint point in the directions of each coordinate axis in the three-dimensional space.

[0051] In this implementation, the first position information includes direction vectors of the first joint point in the x-axis, y-axis and z-axis directions. It can be understood that the vector in the x-axis direction is the vector of the first joint point between the x-axis and the coordinate origin, the vector in the y-axis direction is the vector of the first joint point between the y-axis and the coordinate origin, and the vector in the z-axis direction is the vector of the first joint point between the z-axis and the coordinate origin.

[0052] The second position information includes direction vectors of the second joint point in the x-axis, y-axis and z-axis directions. It can be understood that the vector in the x-axis direction is the vector of the second joint point between the x-axis and the coordinate origin, the vector in the y-axis direction is the vector of the second joint point between the y-axis and the coordinate origin, and the vector in the z-axis direction is the vector of the second joint point between the z-axis and the coordinate origin.

[0053] Afterwards, each vector in the first position information is combined to obtain a first position matrix, and each vector in the second position information is combined to obtain a second position matrix.

[0054] Step S303: Calculate the cosine similarity value between the first position matrix and the second position matrix.

[0055] In the embodiment of the present disclosure, the execution entity calculates the cosine similarity between the first position matrix and the second position matrix. Cosine similarity uses the cosine value of the angle between two vectors in the vector space as a measure of the size of the difference between two individuals. The closer the cosine value is to 1, the closer the angle is to 0 degrees, that is, the more similar the two vectors are. Specifically, the above-mentioned execution entity performs a dot multiplication operation on the first position matrix and the second position matrix, thereby determining the cosine similarity between the first joint point and the second joint point based on the result of the dot multiplication operation. Matrix dot multiplication is an operation used to calculate the scalar product of vectors, which can be expressed as the multiplication and addition of the coordinates of two vectors.

[0056] Step S304: determining the hand gesture posture corresponding to the target image based on the cosine similarity value.

[0057] In the disclosed embodiment, the execution subject determines the gesture posture corresponding to the target image based on the calculated cosine similarity value. Here, the execution subject predefines multiple cosine value ranges and gesture postures corresponding to each cosine value range. Therefore, after calculating the cosine similarity between the first joint point and the second joint point, the execution subject matches the cosine value with the multiple cosine ranges defined by the cosine to determine the cosine range corresponding to the cosine value, which is the target cosine range, and then obtains the gesture posture corresponding to the target cosine range, and uses it as the gesture posture corresponding to the target image.

[0058] As an example, the predefined cosine range and the gesture posture corresponding to each cosine range are: cosine range -1 to -0.5 corresponds to gesture posture 1, cosine range -0.5 to 0 corresponds to gesture posture 2, cosine value 0 corresponds to gesture posture 3, cosine range 0-0.5 corresponds to gesture posture 4, and cosine range 0.5-1 corresponds to gesture posture 5. By calculating the cosine similarity between the first joint point and the second joint point as 0.6, it can be determined that 0.6 belongs to the range of 0.5 to 1, so the gesture posture corresponding to the cosine range 0.5 to 1 will be used as the gesture posture of the target image, that is, gesture posture 5 is the gesture posture corresponding to the target image.

[0059] It should be noted that when obtaining the first joint points and the second joint points of multiple fingers, the cosine similarity can be calculated based on the first joint points and the second joint points of each finger respectively, and then the cosine similarities of the multiple fingers are averaged, and the average is used as the cosine similarity value of the entire hand, and then the corresponding gesture posture is matched. When obtaining the first joint points and the second joint points of multiple fingers, the first position matrix and the second position matrix of the multiple fingers can also be merged to obtain the hand matrix of the entire hand, which contains the first position matrix and the second position matrix of each finger in the multiple fingers, and then the cosine similarity is calculated based on the hand matrix, and the gesture posture corresponding to the calculated cosine similarity is matched.

[0060] According to the method for determining the gesture posture provided by the embodiment of the present disclosure, the first position information of the first joint point of the hand and the second position information of the second joint point in the target image are first obtained, and then based on the first position information and the second position information, the first position matrix corresponding to the first joint point and the second position matrix corresponding to the second joint point are constructed, and then the cosine similarity value between the first position matrix and the second position matrix is ​​calculated, and finally the gesture posture corresponding to the target image is determined based on the cosine similarity value. This method uses the finger joint point (first joint point) as the reference coordinate point, that is, the matrix vector of the fingertip (second joint point) is calculated using the finger joint point as the reference coordinate point, and the cosine similarity between the two joint points is calculated, and the final gesture posture is matched through the cosine similarity, thereby improving the accuracy of the recognition result of the finger posture.

[0061] Figure 5 The process 500 of the second embodiment of the method for determining a gesture posture according to the present disclosure is shown, referring to Figure 5 As shown, the method for determining the gesture posture includes the following steps:

[0062] Step S501, obtaining first position information of a first joint point and second position information of a second joint point of a hand in a target image.

[0063] In the embodiment of the present disclosure, the execution subject of the method for determining the gesture posture, for example Figure 2 The server 203 shown will obtain the first position information of the first joint point of the hand in the target image and the second position information of the second joint point, wherein the first joint point is the joint point between the middle phalanx and the proximal phalanx, and the second joint point is the joint point between the distal phalanx and the middle phalanx.

[0064] Step S501 is substantially the same as step S301 in the aforementioned embodiment, and reference may be made to the relevant description of step S301, which will not be repeated here.

[0065] Step S502: merge the direction vectors of the first joint point in the directions of the respective coordinate axes in the three-dimensional space to obtain a first position matrix.

[0066] In the embodiment of the present disclosure, since the first position information includes direction vectors of the first joint point in the x-axis, y-axis and z-axis directions, the execution entity will merge the vectors in the first position information to obtain the first position matrix.

[0067] Step S503: merge the direction vectors of the second joint point in the directions of the respective coordinate axes in the three-dimensional space to obtain a second position matrix.

[0068] In the embodiment of the present disclosure, since the second position information includes the direction vectors of the second joint point in the x-axis, y-axis and z-axis directions, the execution entity will merge the vectors in the second position information to obtain the second position matrix.

[0069] The first position matrix is ​​obtained by merging the various direction vectors of the first joint point, and the second position matrix is ​​obtained by merging the various direction vectors of the second joint point, so that the cosine similarity can be subsequently calculated based on the position matrices of the two joint points, thereby improving the calculation efficiency of the cosine similarity and the accuracy of the calculation results.

[0070] Step S504: perform a dot product operation on the first position matrix and the second position matrix, and determine a cosine similarity value based on the dot product operation result.

[0071] In the embodiment of the present disclosure, the execution subject performs a dot product operation on the first position matrix and the second position matrix, and determines the cosine similarity value based on the dot product operation result. Cosine similarity uses the cosine value of the angle between two vectors in the vector space as a measure of the size of the difference between two individuals. The closer the cosine value is to 1, the closer the angle is to 0 degrees, that is, the more similar the two vectors are.

[0072] Specifically, the execution subject performs a dot multiplication operation on the first position matrix and the second position matrix, thereby determining the cosine similarity between the first joint point and the second joint point based on the dot multiplication operation result. Matrix dot multiplication is an operation used to calculate the scalar product of vectors, which can be expressed as the multiplication and addition of the coordinates of two vectors. Thus, the cosine similarity value between the first joint point and the second joint point is accurately calculated.

[0073] Step S505, determining the target finger to which the first joint point and the second joint point belong.

[0074] In the embodiment of the present disclosure, the execution subject will determine the finger to which the first joint point and the second joint point belong, that is, the target finger. Since the embodiment of the present disclosure determines the bending condition of the gesture, the reference fingers are the fingers other than the thumb, that is, the index finger, middle finger, ring finger and little finger. Each finger corresponds to a first joint point and a second joint point, and the above-mentioned execution subject can obtain the first joint points and second joint points of multiple fingers. Therefore, the above-mentioned execution subject will first determine the finger to which the first joint point and the second joint point belong, that is, determine whether the first joint point and the second joint point belong to the index finger, middle finger, ring finger or little finger, and use it as the target finger.

[0075] Step S506, determining the cosine value range corresponding to the cosine similarity value, determining the posture corresponding to the cosine value range from the posture set corresponding to the target finger, and obtaining the posture of the target finger.

[0076] In the disclosed embodiment, the execution entity predefines a corresponding gesture set for each finger, and each gesture in the gesture set still corresponds to a different cosine range, that is, for the same cosine range, different fingers correspond to different finger postures.

[0077] Therefore, the above-mentioned execution entity will obtain the posture set corresponding to the target finger, and after calculating the cosine similarity between the first joint point and the second joint point, determine the cosine range corresponding to the cosine similarity, and determine the finger posture corresponding to the cosine range from the posture set corresponding to the target finger, and use it as the posture of the target finger.

[0078] Step S507: determining the gesture posture corresponding to the target image based on the posture of the target finger.

[0079] In the disclosed embodiment, the execution subject determines the gesture posture corresponding to the target image based on the gesture of the target finger. When there is only one target finger, the finger gesture of the target finger is the gesture posture corresponding to the target image.

[0080] from Figure 5 It can be seen that Figure 3 Compared with the corresponding embodiments, the method for determining gesture posture in this embodiment first determines the fingers to which the first joint point and the second joint point belong, and then calculates the matrix vector of the fingertip (second joint point) with the finger joint point of the target finger as the reference coordinate point, and calculates the cosine similarity between the two joint points. The finger posture of the target finger is matched through the cosine similarity, thereby improving the accuracy of the gesture posture recognition results.

[0081] Figure 6 The process 600 of the third embodiment of the method for determining a gesture posture according to the present disclosure is shown, referring to Figure 6 As shown, the method for determining the gesture posture includes the following steps:

[0082] Step S601, obtaining first position information of a first joint point and second position information of a second joint point of a hand in a target image.

[0083] In the embodiment of the present disclosure, the execution subject of the method for determining the gesture posture, for example Figure 2 The server 203 shown will obtain the first position information of the first joint point of the hand in the target image and the second position information of the second joint point, wherein the first joint point is the joint point between the middle phalanx and the proximal phalanx, and the second joint point is the joint point between the distal phalanx and the middle phalanx.

[0084] Step S602: merge the direction vectors of the first joint point in the directions of the respective coordinate axes in the three-dimensional space to obtain a first position matrix.

[0085] In the embodiment of the present disclosure, the execution entity merges the direction vectors of the first joint point in the directions of each coordinate axis in the three-dimensional space to obtain a first position matrix.

[0086] Step S603: merge the direction vectors of the second joint point in the directions of the respective coordinate axes in the three-dimensional space to obtain a second position matrix.

[0087] In the embodiment of the present disclosure, the execution entity will merge the direction vectors of the second joint point in the directions of each coordinate axis in the three-dimensional space to obtain a second position matrix.

[0088] Step S604: perform a dot product operation on the first position matrix and the second position matrix, and determine a cosine similarity value based on the dot product operation result.

[0089] In the embodiment of the present disclosure, the execution entity performs a dot product operation on the first position matrix and the second position matrix, and determines the cosine similarity value based on the dot product operation result.

[0090] Step S605, determining the target finger to which the first joint point and the second joint point belong.

[0091] In the embodiment of the present disclosure, the execution entity determines the target finger to which the first joint point and the second joint point belong.

[0092] Step S606, determining the cosine value range corresponding to the cosine similarity value, determining the posture corresponding to the cosine value range from the posture set corresponding to the target finger, and obtaining the posture of the target finger.

[0093] In the disclosed embodiment, the execution entity determines the cosine value range corresponding to the cosine similarity value, determines the posture corresponding to the cosine value range from the posture set corresponding to the target finger, and obtains the posture of the target finger.

[0094] Steps S601-S606 are substantially consistent with steps S501-S506 in the aforementioned embodiment, and reference may be made to the relevant description of steps S501-S506, which will not be repeated here.

[0095] Step S607: combining the postures of multiple target fingers to obtain a combined posture.

[0096] In the embodiment of the present disclosure, when there are multiple target fingers, the execution entity first determines the finger posture of each target finger, and then combines the postures of the multiple target fingers to obtain a combined posture.

[0097] Step S608 , determining the hand gesture posture corresponding to the combined posture from the predefined posture set, and obtaining the hand gesture posture corresponding to the target image.

[0098] In the disclosed embodiment, the execution subject predefines a multi-finger gesture set, that is, defines a gesture posture after a combination of multiple fingers. Therefore, after obtaining the combined gesture, the execution subject determines the gesture posture corresponding to the combined gesture from the predefined gesture set and uses it as the gesture posture corresponding to the target image.

[0099] from Figure 6 It can be seen that Figure 5 Compared with the corresponding embodiments, the method for determining gesture posture in this embodiment, when the number of target fingers is multiple, the method first combines the postures of multiple target fingers, and then determines the gesture posture corresponding to the combined posture from a predefined posture set, and uses it as the gesture posture of the target image, thereby improving the recognition accuracy of gesture posture.

[0100] Figure 7 A schematic diagram of an implementation scenario of the method for determining a hand gesture of the present disclosure is shown. In this implementation scenario, the execution subject calculates the cosine similarity between the finger joints in the acquired image in real time. Figure 7 The three images in can be three adjacent frames or three discontinuous frames. The execution body will first calculate Figure 7 The cosine similarity between the finger joints in the first image is used to determine the gesture posture that matches the image. Then calculate Figure 7 The cosine similarity between the finger joints in the second image is used to determine the gesture posture that matches the image, and finally the cosine similarity between the finger joints in the third image is calculated to determine the gesture posture that matches the image. That is, the calculation process of the three images is the same, and the above execution body will calculate and determine the gesture posture corresponding to each frame of the image in real time.

[0101] Specifically, the above-mentioned execution entity will first determine the joints of the hand, and then use the joint point between the middle phalanx and the proximal phalanx (i.e. the first joint point, the black dot in the figure) as a reference point to determine the position information of the second joint point, which is the child node of the first joint point (i.e. the joint point between the distal phalanx and the middle phalanx).

[0102] Afterwards, the position vectors of the first joint point are combined to obtain the corresponding first matrix, and the position vectors of the second joint point are combined to obtain the corresponding second matrix, and the cosine similarity between the first matrix and the second matrix is ​​calculated, so as to determine the gesture posture corresponding to the image based on the cosine similarity. That is, the cosine similarity is matched with a plurality of pre-set ranges to determine the cosine range to which it belongs, and then the gesture posture corresponding to the pre-defined cosine range is determined as the gesture posture corresponding to the image.

[0103] Here, the range of cosine values ​​is between [-1,1]. The closer the value is to 1, the closer the directions of the two vectors are; the closer it is to -1, the more opposite the directions of the two vectors are; and the closer it is to 0, the more orthogonal the two vectors are.

[0104] As an implementation of the method for determining the gesture posture shown in the above figures, Figure 8 An embodiment of a device for determining a gesture posture according to the present disclosure is shown. The device for determining a gesture posture 800 is Figure 3 Corresponding to the method embodiment shown, the device can be applied to various electronic devices.

[0105] Reference Figure 8 As shown, the gesture determination device 800 provided by the embodiment of the present disclosure includes: an acquisition module 801, a construction module 802, a calculation module 803 and a determination module 804. The acquisition module 801 is configured to acquire the first position information of the first joint point of the hand and the second position information of the second joint point in the target image, wherein the first joint point is the joint point between the middle phalanx and the proximal phalanx, and the second joint point is the joint point between the distal phalanx and the middle phalanx; the construction module 802 is configured to construct a first position matrix corresponding to the first joint point and a second position matrix corresponding to the second joint point based on the first position information and the second position information; the calculation module 803 is configured to calculate the cosine similarity value between the first position matrix and the second position matrix; the determination module 804 is configured to determine the gesture corresponding to the target image based on the cosine similarity value.

[0106] In the hand gesture determination device 800 of the embodiment of the present disclosure, the specific processing of the acquisition module 801, the construction module 802, the calculation module 803 and the determination module 804 and the technical effects thereof can be respectively referred to in Figure 3The relevant descriptions of steps S301 to S304 in the corresponding embodiment are not repeated here.

[0107] In some implementations of the embodiments of the present disclosure, the device for determining the above-mentioned gesture posture also includes: a skeleton tracking module and a joint point determination module, wherein the skeleton tracking module is configured to perform hand skeleton tracking on the target image; the joint point determination module is configured to identify various joint points of the hand from the target image based on the results of hand skeleton tracking.

[0108] In some implementations of the embodiments of the present disclosure, the above-mentioned gesture posture determination device also includes: a target finger determination module, which is configured to determine the target finger to which the first joint point and the second joint point belong; and the determination module is further configured to: determine the cosine value range corresponding to the cosine similarity value; determine the posture corresponding to the cosine value range from the posture set corresponding to the target finger to obtain the posture of the target finger; determine the gesture posture corresponding to the target image based on the posture of the target finger

[0109] In the gesture determination device of the embodiment of the present disclosure, the specific processing of the determination module and the technical effects brought about by it can be referred to Figure 5 The relevant descriptions of steps S505-S507 in the corresponding embodiment are not repeated here.

[0110] In some implementations of the embodiments of the present disclosure, the number of target fingers is multiple, and the determination module is further configured to: combine the postures of multiple target fingers to obtain a combined posture; determine the gesture posture corresponding to the combined posture from a predefined posture set to obtain the gesture posture corresponding to the target image.

[0111] In the gesture determination device of the embodiment of the present disclosure, the specific processing of the determination module and the technical effects brought about by it can be referred to Figure 6 The relevant descriptions of steps S607-S608 in the corresponding embodiment are not repeated here.

[0112] In some implementations of the disclosed embodiments, the first position information includes: direction vectors of the first joint point in the directions of each coordinate axis in the three-dimensional space; the second position information includes: direction vectors of the second joint point in the directions of each coordinate axis in the three-dimensional space. And the construction module is further configured to: merge the direction vectors of the first joint point in the directions of each coordinate axis in the three-dimensional space to obtain a first position matrix; merge the direction vectors of the second joint point in the directions of each coordinate axis in the three-dimensional space to obtain a second position matrix.

[0113] In the gesture determination device of the embodiment of the present disclosure, the specific processing of the construction module and the technical effects brought about by it can be referred to Figure 5The relevant descriptions of steps S502-S503 in the corresponding embodiment are not repeated here.

[0114] In some implementations of the embodiments of the present disclosure, the calculation module is further configured to: perform a dot product operation on the first position matrix and the second position matrix; and determine a cosine similarity value based on the dot product operation result.

[0115] In the gesture determination device of the embodiment of the present disclosure, the specific processing of the calculation module and the technical effects brought about by it can be referred to Figure 5 The relevant description of step S504 in the corresponding embodiment will not be repeated here.

[0116] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0117] Fig. 9 A schematic block diagram of an example electronic device 900 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0118] like Fig. 9 As shown, the device 900 includes a computing unit 901, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 902 or a computer program loaded from a storage unit 908 into a random access memory (RAM) 903. In the RAM 903, various programs and data required for the operation of the device 900 can also be stored. The computing unit 901, the ROM 902, and the RAM 903 are connected to each other via a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0119] A number of components in the device 900 are connected to the I / O interface 905, including: an input unit 906, such as a keyboard, a mouse, etc.; an output unit 907, such as various types of displays, speakers, etc.; a storage unit 908, such as a disk, an optical disk, etc.; and a communication unit 909, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 909 allows the device 900 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0120] The computing unit 901 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 901 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 901 performs the various methods and processes described above, such as a method for determining a gesture posture. For example, in some embodiments, the method for determining a gesture posture may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 908. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 900 via the ROM 902 and / or the communication unit 909. When the computer program is loaded into the RAM 903 and executed by the computing unit 901, one or more steps of the method for determining a gesture posture described above may be performed. Alternatively, in other embodiments, the computing unit 901 may be configured to perform a method for determining a gesture posture in any other appropriate manner (e.g., by means of firmware).

[0121] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0122] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0123] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0124] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0125] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0126] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0127] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0128] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A method for determining a gesture posture, comprising: Acquire first position information of a first joint point and second position information of a second joint point of a hand in a target image, wherein the first joint point is a joint point between a middle phalanx and a proximal phalanx, and the second joint point is a joint point between a distal phalanx and the middle phalanx; Based on the first position information and the second position information, construct a first position matrix corresponding to the first joint point and a second position matrix corresponding to the second joint point; Calculating a cosine similarity value between the first position matrix and the second position matrix; The hand gesture posture corresponding to the target image is determined based on the cosine similarity value.

2. The method according to claim 1, further comprising: Performing hand skeleton tracking on the target image; Based on the result of the hand skeleton tracking, each joint point of the hand is identified from the target image.

3. The method according to claim 1, wherein: Also includes: Determine the target finger to which the first joint point and the second joint point belong; as well as The determining the hand gesture posture corresponding to the target image based on the cosine similarity value includes: Determine a cosine value range corresponding to the cosine similarity value; Determine the posture corresponding to the cosine value range from the posture set corresponding to the target finger to obtain the posture of the target finger; The gesture posture corresponding to the target image is determined based on the posture of the target finger.

4. The method according to claim 3, wherein: The number of the target fingers is multiple, and determining the gesture posture corresponding to the target image based on the posture of the target fingers includes: Combining the postures of multiple target fingers to obtain a combined posture; The hand gesture posture corresponding to the combined posture is determined from a predefined posture set to obtain the hand gesture posture corresponding to the target image.

5. The method according to claim 1, wherein: The first position information includes: the direction vector of the first joint point in the direction of each coordinate axis in the three-dimensional space; the second position information includes: the direction vector of the second joint point in the direction of each coordinate axis in the three-dimensional space; and The constructing a first position matrix corresponding to the first joint point and a second position matrix corresponding to the second joint point based on the first position information and the second position information includes: Merging the direction vectors of the first joint point in the directions of each coordinate axis in the three-dimensional space to obtain the first position matrix; The direction vectors of the second joint point in the directions of each coordinate axis in the three-dimensional space are combined to obtain the second position matrix.

6. The method according to claim 1, wherein: The calculating the cosine similarity value between the first position matrix and the second position matrix includes: Performing a dot multiplication operation on the first position matrix and the second position matrix; The cosine similarity value is determined based on the dot product operation result.

7. A device for determining a gesture posture, comprising: An acquisition module is configured to acquire first position information of a first joint point and second position information of a second joint point of a hand in a target image, wherein the first joint point is a joint point between a middle phalanx and a proximal phalanx, and the second joint point is a joint point between a distal phalanx and the middle phalanx; A construction module, configured to construct a first position matrix corresponding to the first joint point and a second position matrix corresponding to the second joint point based on the first position information and the second position information; a calculation module, configured to calculate a cosine similarity value between the first position matrix and the second position matrix; The determination module is configured to determine the gesture posture corresponding to the target image based on the cosine similarity value.

8. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 6.

9. A non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 6.

10. A computer program product, comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 6.

Images