Method, apparatus, device and computer-readable medium for generating gesture depth information

Abstract

This disclosure provides embodiments of a method, apparatus, device, and computer-readable medium for generating gesture depth information. One specific implementation of the method includes: controlling an associated monocular RGB camera to acquire a target gesture image; performing gesture keypoint detection on the target gesture image to obtain a set of keypoint coordinate information, wherein the keypoint coordinate information set corresponds to the gesture region included in the target gesture image; generating a set of bone length information based on the set of keypoint coordinate information; inputting the bone length information set into a pre-trained hand shape parameter generation model to obtain hand shape parameter information; and for each keypoint coordinate information set in the set of keypoint coordinate information, generating gesture depth information based on the keypoint coordinate information set, the hand shape parameter information, and the bone length information set. This implementation can reduce the power consumption of the camera device and simplify user operation.

Description

Technical Field

[0001] Embodiments of this disclosure relate to the field of computer technology, and more specifically to methods, apparatus, devices, and computer-readable media for generating gesture depth information. Background Technology

[0002] Gesture depth information, which characterizes the depth features of a gesture image, plays a crucial role in solving technical problems such as gesture recognition and tracking. Currently, the common method for generating gesture depth information is to acquire gesture depth images using a depth camera or multi-view camera.

[0003] However, the inventors discovered that when generating gesture depth information using the above method, the following technical problems often exist: when capturing gesture depth images through a depth camera or multi-camera, the camera consumes a lot of power, which leads to the need for frequent charging of the camera device and makes the user operation more cumbersome.

[0004] The information disclosed in this background section is only intended to enhance the understanding of the background of the inventive concept, and therefore may contain information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0005] The summary portion of this disclosure is intended to provide a brief overview of the concepts, which will be described in detail in the detailed description portion. This summary portion is not intended to identify key or essential features of the claimed technical solutions, nor is it intended to limit the scope of the claimed technical solutions.

[0006] Some embodiments of this disclosure provide methods, apparatus, electronic devices, and computer-readable media for generating gesture depth information to address one or more of the technical problems mentioned in the background section above.

[0007] In a first aspect, some embodiments of this disclosure provide a method for generating gesture depth information based on a monocular RGB camera. The method includes: controlling an associated monocular RGB camera to acquire a target gesture image; performing gesture keypoint detection on the target gesture image to obtain a set of keypoint coordinate information, wherein the keypoint coordinate information set includes keypoint coordinate information groups that correspond to the gesture regions included in the target gesture image; generating a set of bone length information based on the keypoint coordinate information set; inputting the bone length information set into a pre-trained hand shape parameter generation model to obtain hand shape parameter information; and generating gesture depth information for each keypoint coordinate information set in the keypoint coordinate information set, based on the keypoint coordinate information set, the hand shape parameter information, and the bone length information set.

[0008] Optionally, the above-mentioned gesture key point detection of the target gesture image to obtain a set of key point coordinate information includes: inputting the target gesture image into a key point coordinate information generation model to obtain a set of key point coordinate information.

[0009] Optionally, for each keypoint coordinate information group in the aforementioned keypoint coordinate information set, generating gesture depth information based on the keypoint coordinate information group, the hand shape parameter information, and the bone length information group includes: for each keypoint coordinate information group in the aforementioned keypoint coordinate information set, performing the following steps: inputting the hand shape parameter information and preset initial posture parameter information into a preset parameterized model to obtain a standard keypoint coordinate information set; generating wrist point depth information based on the bone length information group and the standard keypoint coordinate information set; generating posture parameter information based on the keypoint coordinate information group and the standard keypoint coordinate information set; inputting the hand shape parameter information and the posture parameter information into the preset parameterized model to obtain hand full degrees of freedom information; and generating gesture depth information based on the wrist point depth information and the hand full degrees of freedom information.

[0010] Optionally, the key point coordinate information in each key point coordinate information group included in the above-mentioned key point coordinate information set includes a key point coordinate sequence; and the above-mentioned generation of posture parameter information based on the above-mentioned key point coordinate information group and the above-mentioned standard key point coordinate information set includes: for each key point coordinate information in the above-mentioned key point coordinate information group, selecting key point coordinates that meet preset position conditions from the key point coordinate sequence included in the above-mentioned key point coordinate information as palm key point coordinates, thereby obtaining a palm key point coordinate set; for each palm key point coordinate in the above-mentioned palm key point coordinate set, selecting the corresponding palm key point from the above-mentioned standard key point coordinate information set. The standard keypoint coordinates of the coordinate system are used as the standard keypoint coordinates of the hand, resulting in a set of standard keypoint coordinates for the hand. Based on this set of hand keypoint coordinates and the standard keypoint coordinates, hand rotation information is generated. For each keypoint coordinate included in the keypoint coordinate information set, the following steps are performed: the standard keypoint coordinates corresponding to the keypoint coordinates in the set of standard keypoint coordinates are determined as the target standard keypoint coordinates; keypoint rotation information is generated based on the keypoint coordinates and the target standard keypoint coordinates; and the hand rotation information and the generated keypoint rotation information are determined as attitude parameter information.

[0011] Optionally, the above method further includes: inputting the generated gesture depth information into a pre-trained gesture recognition model to obtain gesture types; determining the operation information corresponding to the gesture types as target operation information; and executing the application operation corresponding to the target operation information.

[0012] Optionally, the aforementioned keypoint coordinate information generation model is trained through the following steps: obtaining a sample set, wherein the samples in the sample set include sample gesture images and a set of sample keypoint coordinate information corresponding to the sample gesture images; performing the following training steps based on the sample set: inputting the sample gesture images of at least one sample in the sample set into an initial neural network to obtain a set of keypoint coordinate information corresponding to each of the at least one sample; comparing the set of keypoint coordinate information corresponding to each of the at least one sample with the corresponding set of sample keypoint coordinate information; determining whether the initial neural network has reached a preset optimization objective based on the comparison result; and, in response to determining that the initial neural network has reached the optimization objective, using the initial neural network as the trained keypoint coordinate information generation model.

[0013] Optionally, the above training steps further include: in response to determining that the initial neural network has not achieved the above optimization objective, adjusting the parameters of the initial neural network, forming a sample set using unused samples, using the adjusted initial neural network as the initial neural network, and performing the above training steps again.

[0014] Secondly, some embodiments of this disclosure provide a gesture depth information generation device based on a monocular RGB camera. The device includes: a control unit configured to control an associated monocular RGB camera to acquire a target gesture image; a detection unit configured to perform gesture keypoint detection on the target gesture image to obtain a set of keypoint coordinate information, wherein the keypoint coordinate information set includes keypoint coordinate information groups that correspond to the gesture regions included in the target gesture image; a first generation unit configured to generate a set of bone length information based on the set of keypoint coordinate information; an input unit configured to input the set of bone length information into a pre-trained hand shape parameter generation model to obtain hand shape parameter information; and a second generation unit configured to generate gesture depth information for each keypoint coordinate information set in the set of keypoint coordinate information, based on the keypoint coordinate information set, the hand shape parameter information, and the bone length information set.

[0015] Optionally, the detection unit can be further configured to input the target gesture image into the key point coordinate information generation model to obtain a set of key point coordinate information.

[0016] Optionally, the second generation unit can be further configured to perform the following steps for each keypoint coordinate information group in the aforementioned keypoint coordinate information set: inputting the aforementioned hand shape parameter information and preset initial posture parameter information into a preset parameterized model to obtain a standard keypoint coordinate information set; generating wrist point depth information based on the aforementioned bone length information group and the aforementioned standard keypoint coordinate information set; generating posture parameter information based on the aforementioned keypoint coordinate information group and the aforementioned standard keypoint coordinate information set; inputting the aforementioned hand shape parameter information and the aforementioned posture parameter information into the aforementioned preset parameterized model to obtain hand full degree of freedom information; and generating gesture depth information based on the aforementioned wrist point depth information and the aforementioned hand full degree of freedom information.

[0017] Optionally, the key point coordinate information in each key point coordinate information group included in the above-mentioned key point coordinate information set includes a key point coordinate sequence; the second generation unit can be further configured to, for each key point coordinate information in the above-mentioned key point coordinate information group, select key point coordinates that meet preset position conditions from the key point coordinate sequence included in the above-mentioned key point coordinate information as palm key point coordinates, thereby obtaining a palm key point coordinate set; for each palm key point coordinate in the above-mentioned palm key point coordinate set, select standard key point coordinates corresponding to the above-mentioned palm key point coordinates from the above-mentioned standard key point coordinate information set as the hand key point coordinates. The standard keypoint coordinates of the palm are obtained to form a set of standard keypoint coordinates for the palm. Based on the set of standard keypoint coordinates, palm rotation information is generated. For each keypoint coordinate included in the set of keypoint coordinate information, the following steps are performed: the standard keypoint coordinates corresponding to the keypoint coordinates in the set of standard keypoint coordinate information are determined as target standard keypoint coordinates; keypoint rotation information is generated based on the keypoint coordinates and the target standard keypoint coordinates; and the palm rotation information and the generated keypoint rotation information are determined as attitude parameter information.

[0018] Optionally, the gesture depth information generation device 500 based on a monocular RGB camera may further include: a gesture depth information input unit, a determination unit, and an execution unit. The gesture depth information input unit is configured to input the generated gesture depth information into a pre-trained gesture recognition model to obtain gesture types; the determination unit is configured to determine the operation information corresponding to the gesture types as target operation information; and the execution unit is configured to execute the application operation corresponding to the target operation information.

[0019] Optionally, the aforementioned keypoint coordinate information generation model is trained through the following steps: obtaining a sample set, wherein the samples in the sample set include sample gesture images and a set of sample keypoint coordinate information corresponding to the sample gesture images; performing the following training steps based on the sample set: inputting the sample gesture images of at least one sample in the sample set into an initial neural network to obtain a set of keypoint coordinate information corresponding to each of the at least one sample; comparing the set of keypoint coordinate information corresponding to each of the at least one sample with the corresponding set of sample keypoint coordinate information; determining whether the initial neural network has reached a preset optimization objective based on the comparison result; and, in response to determining that the initial neural network has reached the optimization objective, using the initial neural network as the trained keypoint coordinate information generation model.

[0020] Optionally, the above training steps further include: in response to determining that the initial neural network has not achieved the above optimization objective, adjusting the parameters of the initial neural network, forming a sample set using unused samples, using the adjusted initial neural network as the initial neural network, and performing the above training steps again.

[0021] Thirdly, some embodiments of this disclosure provide an electronic device, including: one or more processors; a storage device storing one or more programs thereon; a monocular RGB camera; wherein when the one or more programs are executed by the one or more processors, the one or more processors control the monocular RGB camera to implement the method described in any implementation of the first aspect.

[0022] Fourthly, some embodiments of this disclosure provide a computer-readable medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the method described in any of the implementations of the first aspect above.

[0023] The above-described embodiments of this disclosure have the following beneficial effects: The gesture depth information generation method based on a monocular RGB camera according to some embodiments of this disclosure can reduce the power consumption of the camera device and simplify user operation. Specifically, the reason why camera devices need frequent charging and user operation is relatively cumbersome is that when acquiring gesture depth images through a depth camera or multi-camera, the camera power consumption is high, resulting in frequent charging of the camera device and cumbersome user operation. Based on this, the gesture depth information generation method based on a monocular RGB camera according to some embodiments of this disclosure first controls the associated monocular RGB camera to acquire a target gesture image. This results in a gesture image acquired by a monocular RGB camera with lower power consumption and higher applicability, thereby reducing the power consumption of the camera device during image acquisition and thus reducing the number of times the user needs to charge the camera device. Secondly, gesture keypoint detection is performed on the target gesture image to obtain a set of keypoint coordinate information. The keypoint coordinate information set includes keypoint coordinate information groups that correspond to the gesture regions included in the target gesture image. Therefore, the coordinates of each joint point corresponding to each gesture area in the relative coordinate system with the wrist point as the origin can be obtained, which can be used to predict the 3D coordinates of each joint point in the camera coordinate system. Then, based on the above keypoint coordinate information set, a bone length information set is generated. This allows us to obtain the bone lengths between each hand joint point in the relative coordinate system with the wrist point as the origin, which can be used to predict hand shape parameters. Furthermore, based on the principle that bone length is constant in the camera coordinate system, the bone lengths between each hand joint point in the camera coordinate system are obtained, which can then be used to predict the 3D coordinates of each joint point in the camera coordinate system. Next, the above bone length information set is input into a pre-trained hand shape parameter generation model to obtain hand shape parameter information. This yields hand shape parameter information representing the hand shape, which can be used to predict the 3D coordinates of each joint point in the camera coordinate system. Finally, for each keypoint coordinate information set in the above keypoint coordinate information set, gesture depth information is generated based on the above keypoint coordinate information set, the above hand shape parameter information, and the above bone length information set. Therefore, the three-dimensional coordinates of each joint of the gesture in the camera coordinate system can be obtained from the gesture image captured by a monocular RGB camera. This eliminates the need to use a depth camera or multiple cameras to capture depth images to generate gesture depth information. Furthermore, because the gesture depth information is generated based on the gesture image captured by a monocular RGB camera, the need for a depth camera or multiple cameras is eliminated. This reduces the number of times the user needs to charge the camera device, thereby reducing power consumption and simplifying user operation. Attached Figure Description

[0024] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and elements are not necessarily drawn to scale.

[0025] Figure 1 This is a schematic diagram of an application scenario of the gesture depth information generation method based on a monocular RGB camera, which is one of the embodiments of this disclosure;

[0026] Figure 2 This is a flowchart of some embodiments of the gesture depth information generation method based on a monocular RGB camera according to the present disclosure;

[0027] Figure 3 This is a flowchart of some further embodiments of the gesture depth information generation method based on a monocular RGB camera according to the present disclosure;

[0028] Figure 4 This is a flowchart of some other embodiments of the gesture depth information generation method based on a monocular RGB camera according to the present disclosure;

[0029] Figure 5 This is a schematic diagram of the structure of some embodiments of the gesture depth information generation device based on a monocular RGB camera according to the present disclosure;

[0030] Figure 6 This is a schematic diagram of the structure of an electronic device suitable for implementing some embodiments of the present disclosure. Detailed Implementation

[0031] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0032] It should also be noted that, for ease of description, only the parts relevant to the invention are shown in the accompanying drawings. Unless otherwise specified, the embodiments and features described in this disclosure can be combined with each other.

[0033] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are used only to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependencies.

[0034] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0035] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0036] This disclosure will now be described in detail with reference to the accompanying drawings and embodiments.

[0037] Figure 1 This is a schematic diagram of an application scenario of the gesture depth information generation method based on a monocular RGB camera, which is one of the embodiments of this disclosure.

[0038] exist Figure 1 In the application scenario, firstly, the computing device 101 can control the associated monocular RGB camera 102 to acquire a target gesture image 103. Secondly, the computing device 101 can perform gesture keypoint detection on the target gesture image 103 to obtain a keypoint coordinate information set 104. The keypoint coordinate information set 104 corresponds to the gesture region included in the target gesture image 103. Then, the computing device 101 can generate a bone length information set 105 based on the keypoint coordinate information set 104. Afterwards, the computing device 101 can input the bone length information set 105 into a pre-trained hand shape parameter generation model 106 to obtain hand shape parameter information 107. Finally, for each keypoint coordinate information set in the keypoint coordinate information set 104, the computing device 101 can generate gesture depth information 108 based on the keypoint coordinate information set 104, the hand shape parameter information 107, and the bone length information set 105.

[0039] It should be noted that the aforementioned computing device 101 can be either hardware or software. When the computing device is hardware, it can be implemented as a distributed cluster consisting of multiple servers or terminal devices, or as a single server or a single terminal device. When the computing device is software, it can be installed within the hardware devices listed above. It can be implemented as, for example, multiple software programs or software modules used to provide distributed services, or as a single software program or software module. No specific limitations are made here.

[0040] It should be understood that Figure 1 The number of computing devices shown is merely illustrative. Any number of computing devices can be used depending on implementation needs.

[0041] Continue to refer to Figure 2The diagram illustrates a flow 200 of some embodiments of a gesture depth information generation method based on a monocular RGB camera according to the present disclosure. This gesture depth information generation method based on a monocular RGB camera includes the following steps:

[0042] Step 201: Control the associated monocular RGB camera to acquire target gesture images.

[0043] In some embodiments, the execution entity of the gesture depth information generation method based on a monocular RGB camera (e.g., Figure 1 The computing device 101 shown can control an associated monocular RGB camera to capture a target gesture image. The executing entity can also be a head-mounted display device, in which case the associated monocular RGB camera can be mounted on the head-mounted display device. The head-mounted display device can be AR glasses or VR glasses. Alternatively, the executing entity can be a mobile phone, in which case the associated monocular RGB camera can be the camera on the mobile phone. The target gesture image can be an image to be detected that includes the user's gesture. The target gesture image can include at least one gesture area. The gesture area included in the at least one gesture area can be an area displaying a hand.

[0044] Step 202: Perform gesture key point detection on the target gesture image to obtain a set of key point coordinate information.

[0045] In some embodiments, the execution entity can perform gesture keypoint detection on the target gesture image to obtain a set of keypoint coordinate information. The set of keypoint coordinate information corresponds to the gesture region included in the target gesture image. The correspondence between the set of keypoint coordinate information and the gesture region included in the target gesture image can be one-to-one. The set of keypoint coordinate information can represent the coordinates of each keypoint of a hand. The keypoints can be joints of the hand. Each keypoint corresponds to one keypoint coordinate. The number of keypoints in a hand can be a preset number. For example, the number of keypoints in a hand can be 21. These 21 keypoints can include the four joints of each finger and the joints of the wrist. The keypoint coordinates can be the coordinates of the keypoints in the wrist coordinate system. The keypoint coordinates can be two-dimensional coordinates or three-dimensional coordinates. The wrist coordinate system described above can be a three-dimensional coordinate system with the wrist joint as the origin, a plane parallel to the hand as the xy-plane, and a direction perpendicular to the hand as the z-axis. Alternatively, the wrist coordinate system can be a two-dimensional coordinate system with the wrist joint as the origin and a plane parallel to the hand as the xy-plane.

[0046] The aforementioned keypoint coordinate information set may include first keypoint coordinate information and second keypoint coordinate information. The first keypoint coordinate information may represent the coordinates of a wrist joint. The first keypoint coordinate information may include, but is not limited to, at least one of the following: a wrist point identifier and first keypoint coordinates. The wrist point identifier may uniquely identify the wrist point. The wrist point may be a wrist joint. For example, the wrist point identifier may be "left wrist point" or "left 0". The first keypoint coordinates may be the keypoint coordinates corresponding to the wrist point. The second keypoint coordinate information may represent the coordinates of each keypoint on each finger of a hand. The second keypoint coordinate information may include a second keypoint coordinate set and a second keypoint identifier set. The second keypoint coordinates in the second keypoint coordinate set correspond to the second keypoint identifiers included in the second keypoint identifier set. The second keypoint identifier may uniquely identify the second keypoint. The second keypoint may be a joint on a finger. The second keypoint identifier may be a combination of hand type, finger name, and position information. The combination method may be character concatenation. The aforementioned location information can characterize the position of the second keypoint on the finger. For example, the aforementioned location information can be a number. The aforementioned hand type can be, but is not limited to, one of the following: left hand, right hand. The aforementioned finger name can be the name of the finger. For example, the aforementioned finger name can be the index finger. The aforementioned number can be a numerical number for each second keypoint on each finger, ordered sequentially from the palm to the fingertip. For example, the aforementioned second keypoint identifier can be "right index finger 01". In practice, firstly, the aforementioned executing entity can send the aforementioned target gesture image to the server for gesture keypoint detection to generate a set of keypoint coordinate information. Then, it receives the set of keypoint coordinate information corresponding to the aforementioned target gesture image sent by the server.

[0047] In some optional implementations of certain embodiments, the executing entity can input the target gesture image into a keypoint coordinate information generation model to obtain a set of keypoint coordinate information. The keypoint coordinate information generation model can characterize the correspondence between the gesture image and the set of keypoint coordinate information. For example, the keypoint coordinate information generation model can be a classification model that takes a gesture image as input and a set of keypoint coordinate information as output. The keypoint coordinate information generation model can sequentially compare the target gesture image with each preset gesture image in a preset gesture image set. Each preset gesture image in the preset gesture image set corresponds to a preset set of keypoint coordinate information. The preset gesture images in the preset gesture image set can be pre-defined gesture images. The preset keypoint coordinate information sets in the preset keypoint coordinate information set can be pre-defined keypoint coordinate information sets. If the similarity between any preset gesture image in the preset gesture image set and the target gesture image is less than a pre-defined threshold, then the preset keypoint coordinate information set corresponding to any preset gesture image can be determined as the keypoint coordinate information set corresponding to the target gesture image.

[0048] For example, the keypoint coordinate information generation model described above can be a neural network that takes a gesture image as input and outputs a set of keypoint coordinate information. This neural network can be, but is not limited to, one of the following: a convolutional neural network or a recurrent neural network.

[0049] Therefore, the execution subject can perform gesture key point detection on the target gesture image and directly obtain the set of key point coordinate information.

[0050] Optionally, the above-mentioned key point coordinate information generation model can be trained through the following steps:

[0051] The first step is to obtain a sample set. This sample set may include sample gesture images and a set of keypoint coordinate information corresponding to those images.

[0052] The second step involves performing the following training steps based on the aforementioned sample set: First, the gesture images of at least one sample in the sample set are input into an initial neural network to obtain a set of keypoint coordinate information for each sample in the at least one sample. This initial neural network can be any type of neural network capable of generating keypoint coordinate information sets from gesture images; for example, it can be, but is not limited to, a convolutional neural network or a recurrent neural network. Second, the set of keypoint coordinate information for each sample in the at least one sample is compared with the set of keypoint coordinate information for the corresponding sample. Then, based on the comparison result, it is determined whether the initial neural network has achieved a preset optimization objective. For example, when the difference between the set of keypoint coordinate information for a sample and the set of keypoint coordinate information for the corresponding sample is less than a preset difference threshold, the set of keypoint coordinate information is considered accurate. In this case, the optimization objective can refer to the accuracy of the keypoint coordinate information set generated by the initial neural network being greater than a preset accuracy threshold. Finally, in response to determining that the initial neural network has achieved the optimization objective, the initial neural network is used as the trained keypoint coordinate information generation model.

[0053] Optionally, the above training steps further include: in response to determining that the initial neural network has not achieved the above optimization objective, adjusting the parameters of the initial neural network, and using unused samples to form a sample set, using the adjusted initial neural network as the initial neural network, and performing the above training steps again. As an example, the back propagation algorithm and gradient descent method (e.g., mini-batch gradient descent algorithm) can be used to adjust the network parameters of the initial neural network.

[0054] It should be noted that the implementation method of performing gesture key point detection on the server side to generate key point coordinate information set can be the same as the above technical solution, so it will not be described again.

[0055] Step 203: Generate a bone length information set based on the key point coordinate information set.

[0056] In some embodiments, the execution entity can generate a bone length information set based on the aforementioned keypoint coordinate information set. The bone length information set includes bone length information that can characterize the bone length between two adjacent keypoints of each finger and the length between the wrist point and the joint point of each finger on the palm. In practice, firstly, the execution entity can determine any keypoint coordinate information set from the aforementioned keypoint coordinate information set as the target keypoint coordinate information set. Then, for the second target keypoint coordinate information in the target keypoint coordinate information set, the following steps are performed:

[0057] The first step is to identify two second target key point identifiers in the second target key point identifier set that satisfy the preset key point identifier conditions as the target key point identifier group, thus obtaining the target key point identifier group set. The preset key point identifier conditions can be that the two second target key point identifiers in the second target key point identifier set are two second target key point identifiers that represent two corresponding second target key points that are geographically adjacent.

[0058] The second step is to determine the coordinates of two target key points corresponding to the target key point identification group in the second target key point coordinate set as key point coordinate pairs for each target key point identification group included in the above target key point identification group set.

[0059] The third step involves generating a finger bone length value for each determined keypoint coordinate pair, based on the coordinates of the two keypoints included in the pair. This finger bone length value can be the bone length between two adjacent keypoints of the same finger. In practice, the executing entity can determine the distance between the two keypoint coordinates in the keypoint pair as the bone length value using the Euclidean distance method.

[0060] The fourth step is to combine the generated bone length values ​​to obtain bone length information.

[0061] Step 204: Input the bone length information group into the pre-trained hand shape parameter generation model to obtain hand shape parameter information.

[0062] In some embodiments, the execution entity can input the bone length information set into a pre-trained hand shape parameter generation model to obtain hand shape parameter information. The hand shape parameter generation model can represent the correspondence between the bone length information set and the hand shape parameter information. The hand shape parameter information can represent the shape of the hand. The hand shape parameter information can be 10-dimensional shape parameters. The shape parameters can be parameters representing the shape of the hand. For example, the hand shape parameter information can represent the length, thickness, etc. of the hand. For example, the hand shape parameter generation model can be a neural network that takes the bone length information set as input and outputs the hand shape parameter information. The neural network can be, but is not limited to, one of the following: a convolutional neural network or a recurrent neural network.

[0063] As an example, the above hand shape parameter generation model was trained through the following model training steps:

[0064] The first step is to obtain a skeleton sample set. This set includes skeleton length information groups and corresponding hand shape parameter information.

[0065] The second step involves performing the following training steps based on the aforementioned bone sample set: First, the sample bone length information group of at least one bone sample in the aforementioned bone sample set is input into an initial neural network to obtain the hand shape parameter information corresponding to each bone sample in the aforementioned at least one bone sample. The initial neural network can be any neural network capable of obtaining hand shape parameter information from the bone length information group; for example, the initial neural network can be, but is not limited to, one of the following: a convolutional neural network or a recurrent neural network. Second, the hand shape parameter information corresponding to each bone sample in the aforementioned at least one bone sample is compared with the corresponding sample hand shape parameter information. Then, based on the comparison result, it is determined whether the initial neural network has reached a preset optimization objective. As an example, when the difference between the hand shape parameter information corresponding to a bone sample and the corresponding sample hand shape parameter information is less than a preset difference threshold, the hand shape parameter information is considered accurate. In this case, the aforementioned optimization objective can refer to the accuracy rate of the hand shape parameter information generated by the aforementioned initial neural network being greater than a preset accuracy threshold. Finally, in response to determining that the aforementioned initial neural network has reached the aforementioned optimization objective, the aforementioned initial neural network is used as the trained hand shape parameter generation model.

[0066] Optionally, the above training steps further include: in response to determining that the initial neural network has not achieved the above optimization objective, adjusting the parameters of the initial neural network, and using unused bone samples to form a bone sample set, using the adjusted initial neural network as the initial neural network, and performing the above training steps again. As an example, the back propagation algorithm and gradient descent method (e.g., mini-batch gradient descent algorithm) can be used to adjust the network parameters of the initial neural network.

[0067] For example, the aforementioned hand shape parameter generation model can be a classification model that outputs a set of bone length information and a set of hand shape parameter information. Specifically, this model compares the set of bone length information sequentially with each preset bone length information set included in a preset set of bone length information. The preset bone length information sets included in the preset set of bone length information can be pre-defined bone length information sets. Each preset bone length information set in the preset set of bone length information corresponds to preset hand shape parameter information. The preset hand shape parameter information can be pre-defined hand shape parameter information. If the difference between any bone length information set in the preset set of bone length information and the given bone length information set is less than a pre-defined threshold, then the preset hand shape parameter information corresponding to that bone length information set can be determined as the hand shape parameter information corresponding to the target gesture image.

[0068] Step 205: For each keypoint coordinate information group in the keypoint coordinate information group set, generate gesture depth information based on the keypoint coordinate information group, hand shape parameter information, and bone length information group.

[0069] In some embodiments, the execution entity can generate gesture depth information for each keypoint coordinate information group in the keypoint coordinate information set, based on the keypoint coordinate information group, the hand shape parameter information, and the bone length information group. The gesture depth information can characterize the positions of each hand keypoint and each hand mesh point in the camera coordinate system. The gesture depth information can include a hand keypoint coordinate set and a hand mesh point coordinate set. The hand keypoint coordinate set can include 21 hand keypoint coordinates. The hand keypoint coordinates can be the three-dimensional coordinates of the keypoints in the camera coordinate system. The hand mesh point coordinate set can include the three-dimensional coordinates of the hand mesh points in the camera coordinate system. The mesh points can be the vertices required to construct the hand mesh. In practice, the execution entity can first perform the following steps for each keypoint coordinate information group in the keypoint coordinate information set:

[0070] The first step involves inputting the aforementioned hand shape parameters and preset initial posture parameters into the keypoint classification model to obtain a standard keypoint coordinate information set and a standard grid point coordinate information set. The initial posture parameters represent the hand's posture when extended. The standard keypoint coordinate information set represents the position of keypoints in the wrist coordinate system when the hand is extended. This set may include standard keypoint coordinates and keypoint identifiers. The standard keypoint coordinates are the coordinates of keypoints in the wrist coordinate system when the hand is extended. The standard grid point coordinate information set represents the positions of the 778 grid points needed to construct the hand model in the wrist coordinate system when the hand is extended. This set may include standard grid point coordinates and grid point identifiers. The standard grid point coordinates are the coordinates of grid points in the wrist coordinate system when the hand is extended. The grid point identifiers uniquely identify each grid point; for example, they can be numbers. The aforementioned keypoint classification model can be a classification model that takes hand shape parameters and posture parameters as input, and outputs a keypoint coordinate information set and a grid point coordinate information set. This keypoint classification model sequentially compares the hand shape parameters with each preset hand shape parameter in a preset hand shape parameter information set. Then, it sequentially compares the initial posture parameters with each preset posture parameter in a preset posture parameter information set. The preset posture parameters in the preset posture parameter information set can be pre-defined posture parameters. The preset hand shape parameters in the preset hand shape parameter information set can be pre-defined. The preset hand shape parameters in the preset hand shape parameter information set correspond to the preset posture parameters in the preset posture parameter information set. The correspondence between the preset hand shape parameters in the preset hand shape parameter information set and the preset posture parameters in the preset posture parameter information set can be one-to-one. Each corresponding set of preset hand shape parameters and preset posture parameters corresponds to a preset keypoint coordinate information set and a preset grid point coordinate information set. The preset key point coordinate information set mentioned above includes preset key point coordinate information that can be pre-defined key point coordinate information. The preset grid point coordinate information set mentioned above includes preset grid point coordinate information that can be pre-defined grid point coordinate information.If the difference between any preset hand shape parameter information in the aforementioned preset hand shape parameter information set and any preset posture parameter information in the aforementioned preset posture parameter information set and the hand shape parameter information and posture parameter information is less than a preset threshold, then the preset key point coordinate information set and preset grid point coordinate information set corresponding to any preset hand shape parameter information and the aforementioned preset posture parameter information can be determined as the standard key point coordinate information set and standard grid point coordinate information set corresponding to the hand shape parameter information and posture parameter information.

[0071] The second step involves inputting the aforementioned keypoint coordinate information set and the aforementioned standard keypoint coordinate information set into the posture information classification model to obtain keypoint posture information. This keypoint posture information can characterize the rotation angles of each keypoint and the hand. The keypoint posture information can include the hand rotation angle and a set of keypoint rotation angles. The hand rotation angle can be the rotation angle of the hand itself. The keypoint rotation angle set can include the rotation angles of keypoints. The aforementioned input posture information classification model can be a classification model that takes the keypoint coordinate information set and the standard keypoint coordinate information set as input and outputs the keypoint posture information. This input posture information classification model can sequentially compare the keypoint coordinate information set and the standard keypoint coordinate information set with multiple sets of keypoint coordinate information and standard keypoint coordinate information sets in a preset posture information relationship table. This preset posture information relationship table can be created based on the analysis of a large number of keypoint coordinate information sets and standard keypoint coordinate information sets. Each set of keypoint coordinate information sets and standard keypoint coordinate information sets corresponds to preset keypoint posture information. The aforementioned preset keypoint attitude information can be pre-defined keypoint attitude information. If the difference between any set of keypoint coordinate information and the standard keypoint coordinate information set in the aforementioned preset attitude information relationship table and the keypoint coordinate information set is less than a pre-defined threshold, then the preset keypoint attitude information corresponding to any set of keypoint coordinate information and the standard keypoint coordinate information set can be determined as the keypoint attitude information corresponding to that keypoint coordinate information set and the standard keypoint coordinate information set.

[0072] The third step is to input the above-mentioned hand shape parameter information and the above-mentioned key point posture information into the above-mentioned key point classification model to obtain the target key point coordinate information set and the target grid point coordinate information set.

[0073] Fourth, based on the preset wrist point depth value, perform coordinate transformation on the coordinates of each target key point included in the target key point coordinate information set and the coordinates of each target grid point included in the aforementioned target grid point coordinate information set to obtain the target hand grid point coordinate set and the target hand key point coordinate set. The target hand grid point coordinates included in the aforementioned target hand grid point coordinate set can be the three-dimensional coordinates of the grid points in the camera coordinate system. The target hand key point coordinates included in the aforementioned target hand key point coordinate set can be the three-dimensional coordinates of the key points in the camera coordinate system. The aforementioned wrist point depth value can be the z-axis coordinate value of the wrist point in the camera coordinate system.

[0074] The fifth step is to combine the above-mentioned target hand grid point coordinate set and target hand key point coordinate set to obtain gesture depth information. The combination method can be character concatenation.

[0075] The above-described embodiments of this disclosure have the following beneficial effects: the gesture depth information generation method based on a monocular RGB camera according to some embodiments of this disclosure can simplify user operation. Specifically, the reason why camera devices need to be charged frequently and user operation is relatively cumbersome is that when capturing gesture depth images through a depth camera or multi-camera, the camera power consumption is relatively high, resulting in the need for frequent charging of the camera device and cumbersome user operation. Based on this, the gesture depth information generation method based on a monocular RGB camera according to some embodiments of this disclosure first controls the associated monocular RGB camera to capture the target gesture image. Thus, gesture images captured by a monocular RGB camera with low power consumption and high applicability can be obtained, thereby reducing the power consumption of the camera device during image acquisition and reducing the number of times the user needs to charge the camera device. Secondly, gesture keypoint detection is performed on the target gesture image to obtain a set of keypoint coordinate information. The keypoint coordinate information set includes a set of keypoint coordinate information that corresponds to the gesture region included in the target gesture image. Thus, the coordinates of each joint point corresponding to each gesture region in the relative coordinate system with the wrist point as the origin can be obtained, which can be used to predict the three-dimensional coordinates of each joint point in the camera coordinate system. Then, based on the aforementioned keypoint coordinate information set, a bone length information set is generated. This allows us to obtain the bone lengths between each hand joint in a relative coordinate system with the wrist point as the origin, which can be used to predict hand shape parameters. Furthermore, based on the principle that bone length is constant in the camera coordinate system, the bone lengths between each hand joint in the camera coordinate system are obtained, which can then be used to predict the 3D coordinates of each joint in the camera coordinate system. Next, the aforementioned bone length information set is input into a pre-trained hand shape parameter generation model to obtain hand shape parameter information. This yields hand shape parameter information characterizing the hand shape, which can then be used to predict the 3D coordinates of each joint in the camera coordinate system. Finally, for each keypoint coordinate information set in the aforementioned keypoint coordinate information set, gesture depth information is generated based on the aforementioned keypoint coordinate information set, the aforementioned hand shape parameter information, and the aforementioned bone length information set. This allows us to obtain the 3D coordinates of each joint in the gesture image captured by a monocular RGB camera in the camera coordinate system, thus eliminating the need to use a depth camera or multi-camera to capture depth images for generating gesture depth information. Because the gesture depth information is obtained by processing the gesture image captured by a monocular RGB camera through a series of algorithms, there is no need to use a depth camera or multi-camera to capture depth images. This reduces the number of times users need to charge the camera device, thus simplifying user operation.

[0076] Further reference Figure 3The document illustrates a flowchart 300 of another embodiment of a gesture depth information generation method based on a monocular RGB camera. The flowchart 300 of this gesture depth information generation method based on a monocular RGB camera includes the following steps:

[0077] Step 301: Control the associated monocular RGB camera to acquire target gesture images.

[0078] Step 302: Perform gesture key point detection on the target gesture image to obtain a set of key point coordinate information.

[0079] Step 303: Generate a bone length information set based on the above key point coordinate information set.

[0080] Step 304: Input the bone length information group into the pre-trained hand shape parameter generation model to obtain hand shape parameter information.

[0081] Step 305: For each key point coordinate information group in the above key point coordinate information group set, generate gesture depth information based on the above key point coordinate information group, the above hand shape parameter information and the above bone length information group.

[0082] In some embodiments, the specific implementation of steps 301-305 and the resulting technical effects can be found in [reference needed]. Figure 2 Steps 201-205 in the corresponding embodiments will not be repeated here.

[0083] Step 306: Input the generated depth information of each gesture into the pre-trained gesture recognition model to obtain the gesture type.

[0084] In some embodiments, the aforementioned execution entity (e.g. Figure 1The computing device 101 shown can input the generated gesture depth information into a pre-trained gesture recognition model to obtain gesture types. The gesture type can represent the operation performed by the gesture. The gesture type can be, but is not limited to, one of the following: a click gesture, a swipe gesture. The gesture recognition model can represent the correspondence between gesture depth information and gesture types. For example, the gesture recognition model can be a classification model that takes gesture depth information as input and gesture type as output. The gesture recognition model can sequentially compare the gesture depth information with each preset gesture depth information in a preset gesture depth information set. The preset gesture depth information in the preset gesture depth information set can be pre-defined gesture depth information. The preset gesture type can be a pre-defined gesture type. Each preset gesture depth information in the preset gesture depth information set corresponds to a preset gesture type. If the difference between any preset gesture depth information in the preset gesture depth information set and the given gesture depth information is less than a pre-defined threshold, then the preset gesture type corresponding to any preset gesture depth information can be determined as the gesture type corresponding to that gesture depth information.

[0085] For example, the gesture recognition model described above can be a neural network that takes gesture depth information as input and gesture type as output. This neural network can be, but is not limited to, one of the following: a convolutional neural network or a recurrent neural network.

[0086] Optionally, the above gesture recognition model is trained through the following gesture recognition steps:

[0087] The first step is to obtain a gesture sample set. This set includes gesture depth information and the gesture type corresponding to the gesture image.

[0088] The second step involves performing the following training steps based on the aforementioned gesture sample set: First, the gesture depth information of at least one gesture sample in the aforementioned gesture sample set is input into an initial neural network to obtain the gesture type corresponding to each gesture sample in the aforementioned at least one gesture sample. The initial neural network can be any neural network capable of obtaining the gesture type based on the gesture depth information; for example, the initial neural network can be, but is not limited to, one of the following: a convolutional neural network or a recurrent neural network. Second, the gesture type corresponding to each gesture sample in the aforementioned at least one gesture sample is compared with the corresponding sample gesture type. Then, based on the comparison result, it is determined whether the initial neural network has reached a preset optimization objective. As an example, when the difference between the gesture type corresponding to a sample and the corresponding sample gesture type is less than a preset difference threshold, the gesture type is considered accurate. In this case, the aforementioned optimization objective can refer to the accuracy of the gesture type generated by the aforementioned initial neural network being greater than a preset accuracy threshold. Finally, in response to determining that the aforementioned initial neural network has reached the aforementioned optimization objective, the aforementioned initial neural network is used as the trained gesture recognition model.

[0089] Optionally, the above training steps further include: in response to determining that the initial neural network has not achieved the above optimization objective, adjusting the parameters of the initial neural network, and using unused gesture samples to form a gesture sample set, using the adjusted initial neural network as the initial neural network, and performing the gesture recognition steps again. As an example, the back propagation algorithm and gradient descent method (e.g., mini-batch gradient descent algorithm) can be used to adjust the network parameters of the initial neural network.

[0090] Step 307: Determine the operation information corresponding to the gesture type as the target operation information.

[0091] In some embodiments, the executing entity may determine the operation information corresponding to the gesture type as the target operation information. The operation information may be information about an operation performed by the user. For example, the target operation information may be a "click operation".

[0092] Step 308: Execute the application operation corresponding to the target operation information.

[0093] In some embodiments, the execution entity may perform the application operation corresponding to the target operation information. The application operation may be an operation applied to an application. The application operation may include, but is not limited to, one of the following: clicking or swiping.

[0094] from Figure 3 It can be seen from this that, with Figure 2 Compared to the description of some corresponding embodiments, Figure 3 The flow 300 of the gesture depth information generation method based on a monocular RGB camera in some corresponding embodiments extends the steps of how to apply the gesture depth information. Therefore, the schemes described in these embodiments can determine the user's operation on the application through the generated gesture depth information, thereby enabling human-computer interaction.

[0095] Further reference Figure 4 The document illustrates a flowchart 400 of another embodiment of a gesture depth information generation method based on a monocular RGB camera. The flowchart 400 of this gesture depth information generation method based on a monocular RGB camera includes the following steps:

[0096] Step 401: Control the associated monocular RGB camera to acquire target gesture images.

[0097] Step 402: Perform gesture key point detection on the target gesture image to obtain a set of key point coordinate information.

[0098] Step 403: Generate a bone length information set based on the above key point coordinate information set.

[0099] Step 404: Input the bone length information group into the pre-trained hand shape parameter generation model to obtain hand shape parameter information.

[0100] In some embodiments, the specific implementation of steps 401-404 and the resulting technical effects can be found in [reference needed]. Figure 2 Steps 201-204 in the corresponding embodiments will not be repeated here.

[0101] Step 405: For each keypoint coordinate information group in the keypoint coordinate information group set, perform the following steps:

[0102] Step 4051: Input the hand shape parameter information and the preset initial posture parameter information into the preset parametric model to obtain the standard key point coordinate information set.

[0103] In some embodiments, the aforementioned execution entity (e.g. Figure 1 The computing device 101 shown can input the aforementioned hand shape parameter information and preset initial posture parameter information into a preset parametric model to obtain a standard key point coordinate information set. The aforementioned initial posture parameter information can characterize the posture when the hand is extended. The aforementioned preset parametric model can be a pre-defined parametric model. The aforementioned preset parametric model can be a Mano model.

[0104] Step 4052: Generate wrist point depth information based on the bone length information set and the standard key point coordinate information set.

[0105] In some embodiments, the execution entity can generate wrist point depth information based on the bone length information set and the standard keypoint coordinate information set. The wrist point depth information can represent the three-dimensional coordinates of the wrist point in the camera coordinate system. In practice, firstly, the execution entity can select a second target keypoint identifier that meets preset finger conditions from the second target keypoint identifier set as the middle finger root keypoint identifier. The preset finger conditions can represent the second target keypoint identifier as the joint point of the middle finger at the palm. Secondly, using the Euclidean distance method, the distance between the coordinates of the second target keypoint corresponding to the middle finger root keypoint identifier and the coordinates of the first target keypoint included in the first target keypoint coordinate information is determined as the bone length distance. Then, camera intrinsic parameters, the pixel coordinates of the middle finger root keypoint, and the pixel coordinates of the wrist point are obtained. The pixel coordinates of the middle finger root keypoint can be the coordinates of the joint point of the middle finger at the palm in the camera coordinate system. The pixel coordinates of the wrist point can be the three-dimensional coordinates of the wrist point in the camera coordinate system. Then, by solving the following equation using the least squares method, the z-coordinate of the wrist point in the camera coordinate system is obtained as the wrist point depth information:

[0106]

[0107] Among them, l ref This can represent the bone length distance mentioned above. K can represent the camera intrinsic parameters mentioned above. r This can represent the z-coordinate of the wrist point in the camera coordinate system. r This can represent the x-coordinate value in the pixel coordinates of the key point at the base of the middle finger mentioned above. r This can represent the y-coordinate value of the key pixel coordinates at the base of the middle finger. r can represent the joint point of the middle finger on the palm. w can represent the wrist point. u w This can represent the x-coordinate value in the above wrist point pixel coordinates. w This can represent the y-coordinate value in the aforementioned wrist point pixel coordinates. d w It can represent the z-coordinate value of the second target key point coordinates corresponding to the key point of the middle finger root above, which is to be solved.

[0108] Step 4053: Generate attitude parameter information based on the key point coordinate information group and the standard key point coordinate information set.

[0109] In some embodiments, the executing entity can generate posture parameter information based on the keypoint coordinate information set and the standard keypoint coordinate information set. The posture parameter information can characterize the hand's posture. In practice, the executing entity can input the keypoint coordinate information set and the standard keypoint coordinate information set into the posture information classification model to obtain the posture parameter information.

[0110] In some optional implementations of certain embodiments, the keypoint coordinate information in each keypoint coordinate information group included in the aforementioned keypoint coordinate information set may include a keypoint coordinate sequence. Each keypoint coordinate information in the aforementioned keypoint coordinate information group can represent the coordinates of each keypoint on each finger, starting from the wrist point and ending at the fingertip. The aforementioned keypoint coordinate sequence can be a sequence in which the coordinates of each keypoint on each finger are ordered sequentially from the wrist point to the fingertip. The aforementioned execution entity can generate posture parameter information based on the aforementioned keypoint coordinate information group and the aforementioned standard keypoint coordinate information set through the following steps:

[0111] The first step is to select keypoint coordinates that meet preset position conditions from the keypoint coordinate sequence included in the above keypoint coordinate information group as the hand keypoint coordinates, thereby obtaining the hand keypoint coordinate set. The preset position condition can be that the keypoint coordinates are positioned second in the corresponding keypoint coordinate sequence.

[0112] The second step involves selecting, for each hand keypoint coordinate in the aforementioned hand keypoint coordinate set, the corresponding standard keypoint coordinates from the aforementioned standard keypoint coordinate information set as the hand standard keypoint coordinates, thus obtaining the hand standard keypoint coordinate set. The standard keypoint coordinates corresponding to the aforementioned hand keypoint coordinates can be the same as the keypoint identifiers of the aforementioned hand keypoint coordinates.

[0113] The third step is to generate hand rotation information based on the aforementioned set of key hand coordinates and the aforementioned set of standard key hand coordinates. This hand rotation information represents the rotation angle of the hand. In practice, the executing entity can first use the least squares method to solve for the aforementioned set of key hand coordinates and the aforementioned set of standard key hand coordinates to obtain the hand rotation angle as the hand rotation information.

[0114] Fourth, for each keypoint coordinate included in the keypoint coordinate information group above, perform the following sub-steps:

[0115] The first sub-step involves determining the standard keypoint coordinates corresponding to the aforementioned keypoint coordinates from the set of standard keypoint coordinate information as the target standard keypoint coordinates. Specifically, the standard keypoint coordinates corresponding to the aforementioned keypoint coordinates can be the standard keypoint coordinates corresponding to the same keypoint as the aforementioned keypoint coordinates.

[0116] The second sub-step involves generating keypoint rotation information based on the aforementioned keypoint coordinates and the target standard keypoint coordinates. This keypoint rotation information represents the rotation angle of the keypoint. In practice, the executing entity can use the cosine formula for the difference between two angles to calculate the cosine of the keypoint coordinates and the target standard keypoint coordinates as the cosine of the rotation angle. Then, the rotation angle corresponding to the cosine of the rotation angle is calculated using inverse trigonometric functions as the keypoint rotation information.

[0117] The fifth step involves determining the hand rotation information and the generated keypoint rotation information as attitude parameter information. In practice, the executing entity can combine the hand rotation information and the generated keypoint rotation information to obtain the attitude parameter information.

[0118] Therefore, attitude parameter information can be obtained by pre-setting a parameterized model, thereby reducing the computational complexity and shortening the computation time.

[0119] Step 4054: Input the hand shape parameter information and posture parameter information into the preset parametric model to obtain the hand's full degree of freedom information.

[0120] In some embodiments, the execution entity can input the hand shape parameter information and the posture parameter information into the preset parameterized model to obtain the hand's full degrees of freedom information. This hand full degrees of freedom information can characterize the three-dimensional coordinates of each grid point of the hand in the wrist coordinate system and the three-dimensional coordinates of each keypoint in the wrist coordinate system. The hand full degrees of freedom information can include a set of grid point coordinates and a set of keypoint coordinates. The set of grid point coordinates can be the three-dimensional coordinates of 778 grid points of the hand in the wrist coordinate system. The set of keypoint coordinates can be the three-dimensional coordinates of 21 keypoints of the hand in the wrist coordinate system.

[0121] Step 4055: Generate gesture depth information based on wrist point depth information and hand full degrees of freedom information.

[0122] In some embodiments, the execution entity can generate gesture depth information based on the wrist point depth information and the hand's full degrees of freedom information. In practice, the execution entity can perform coordinate transformation on the grid point coordinate set and keypoint coordinate set included in the hand's full degrees of freedom information based on the wrist point depth information to obtain a hand depth grid point coordinate set and a hand depth keypoint coordinate set. The hand depth grid point coordinate set can be the three-dimensional coordinates of the hand's grid points in the camera coordinate system. The hand depth keypoint coordinate set can be the three-dimensional coordinates of the keypoints in the camera coordinate system.

[0123] from Figure 4 It can be seen from this that, with Figure 2 Compared to the description of some corresponding embodiments, Figure 4 The flowchart 400 of the gesture depth information generation method based on a monocular RGB camera in some corresponding embodiments illustrates the steps of generating gesture depth information based on key point coordinate information, hand shape parameter information, and bone length information. Thus, the schemes described in these embodiments can obtain posture parameter information and hand full degree of freedom information through a preset parameterized model, and obtain wrist point depth information based on the principle that bone length is constant in the camera coordinate system, thereby obtaining the gesture depth information of the target gesture image captured by the monocular RGB camera. Furthermore, gesture depth information can be generated through a simple calculation scheme, shortening the computation time and improving the efficiency of gesture depth information generation.

[0124] Further reference Figure 5 As an implementation of the methods shown in the above figures, this disclosure provides some embodiments for generating gesture depth information based on a monocular RGB camera. These device embodiments are similar to... Figure 2 Corresponding to the method embodiments shown, the device can be specifically applied to various electronic devices.

[0125] like Figure 5 As shown, a gesture depth information generation device 500 based on a monocular RGB camera in some embodiments includes: a control unit 501, a detection unit 502, a first generation unit 503, an input unit 504, and a second generation unit 505. The control unit 501 is configured to control an associated monocular RGB camera to acquire a target gesture image; the detection unit 502 is configured to perform gesture keypoint detection on the target gesture image to obtain a set of keypoint coordinate information, wherein the keypoint coordinate information set corresponds to the gesture region included in the target gesture image; the first generation unit 503 is configured to generate a set of bone length information based on the set of keypoint coordinate information; the input unit 504 is configured to input the set of bone length information into a pre-trained hand shape parameter generation model to obtain hand shape parameter information; and the second generation unit 505 is configured to generate gesture depth information for each keypoint coordinate information set in the set of keypoint coordinate information, based on the keypoint coordinate information set, the hand shape parameter information, and the bone length information set.

[0126] Optionally, the detection unit 502 can be further configured to input the target gesture image into the key point coordinate information generation model to obtain a set of key point coordinate information.

[0127] Optionally, the second generation unit 505 can be further configured to perform the following steps for each keypoint coordinate information group in the aforementioned keypoint coordinate information set: inputting the aforementioned hand shape parameter information and preset initial posture parameter information into a preset parameterized model to obtain a standard keypoint coordinate information set; generating wrist point depth information based on the aforementioned bone length information group and the aforementioned standard keypoint coordinate information set; generating posture parameter information based on the aforementioned keypoint coordinate information group and the aforementioned standard keypoint coordinate information set; inputting the aforementioned hand shape parameter information and the aforementioned posture parameter information into the aforementioned preset parameterized model to obtain hand full degree of freedom information; and generating gesture depth information based on the aforementioned wrist point depth information and the aforementioned hand full degree of freedom information.

[0128] Optionally, the key point coordinate information in each key point coordinate information group included in the above-mentioned key point coordinate information set includes a key point coordinate sequence; the second generation unit 505 can be further configured to, for each key point coordinate information in the above-mentioned key point coordinate information group, select key point coordinates that meet preset position conditions from the key point coordinate sequence included in the above-mentioned key point coordinate information as palm key point coordinates, thereby obtaining a palm key point coordinate set; for each palm key point coordinate in the above-mentioned palm key point coordinate set, select standard key point coordinates corresponding to the above-mentioned palm key point coordinates from the above-mentioned standard key point coordinate information set as... The standard keypoint coordinates of the palm are obtained to form a set of standard keypoint coordinates for the palm. Based on the set of standard keypoint coordinates, palm rotation information is generated. For each keypoint coordinate included in the keypoint coordinate information set, the following steps are performed: the standard keypoint coordinates corresponding to the keypoint coordinates in the set of standard keypoint coordinates are determined as target standard keypoint coordinates; keypoint rotation information is generated based on the keypoint coordinates and the target standard keypoint coordinates; and the palm rotation information and the generated keypoint rotation information are determined as posture parameter information.

[0129] Optionally, the gesture depth information generation device 500 based on a monocular RGB camera may further include: a gesture depth information input unit, a determination unit, and an execution unit (not shown in the figure). The gesture depth information input unit is configured to input the generated gesture depth information into a pre-trained gesture recognition model to obtain the gesture type; the determination unit is configured to determine the operation information corresponding to the gesture type as the target operation information; and the execution unit is configured to execute the application operation corresponding to the target operation information.

[0130] Optionally, the aforementioned keypoint coordinate information generation model is trained through the following steps: obtaining a sample set, wherein the samples in the sample set include sample gesture images and a set of sample keypoint coordinate information corresponding to the sample gesture images; performing the following training steps based on the sample set: inputting the sample gesture images of at least one sample in the sample set into an initial neural network to obtain a set of keypoint coordinate information corresponding to each of the at least one sample; comparing the set of keypoint coordinate information corresponding to each of the at least one sample with the corresponding set of sample keypoint coordinate information; determining whether the initial neural network has reached a preset optimization objective based on the comparison result; and, in response to determining that the initial neural network has reached the optimization objective, using the initial neural network as the trained keypoint coordinate information generation model.

[0131] Optionally, the above training steps further include: in response to determining that the initial neural network has not achieved the above optimization objective, adjusting the parameters of the initial neural network, forming a sample set using unused samples, using the adjusted initial neural network as the initial neural network, and performing the above training steps again.

[0132] It is understandable that the units and references described in the gesture depth information generation device 500 based on a monocular RGB camera are similar. Figure 2 The steps in the described method correspond accordingly. Therefore, the operations, features, and beneficial effects described above for the method also apply to the device 500 and the units contained therein, and will not be repeated here.

[0133] The following is for reference. Figure 6 It illustrates an electronic device 600 suitable for implementing some embodiments of the present disclosure (e.g., Figure 1 The diagram shows the structure of a computing device. In some embodiments of this disclosure, the electronic device may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 6 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments of this disclosure.

[0134] like Figure 6As shown, electronic device 600 may include a processing device (e.g., a central processing unit, a graphics processor, etc.) 601, which can perform various appropriate actions and processes according to a program stored in read-only memory (ROM) 602 or a program loaded from storage device 608 into random access memory (RAM) 603. RAM 603 also stores various programs and data required for the operation of electronic device 600. Processing device 601, ROM 602, and RAM 603 are interconnected via bus 604. Input / output (I / O) interface 605 is also connected to bus 604.

[0135] Typically, the following devices can be connected to I / O interface 605: input devices 606 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, monocular RGB cameras, etc.; output devices 607 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 608 including, for example, magnetic tapes, hard disks, etc.; and communication devices 609. Communication device 609 allows electronic device 600 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 6 An electronic device 600 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively. Figure 6 Each box shown can represent a device or multiple devices as needed.

[0136] In particular, according to some embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, some embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 609, or installed from a storage device 608, or installed from a ROM 602. When the computer program is executed by the processing device 601, it performs the functions defined above in the methods of some embodiments of this disclosure.

[0137] It should be noted that, in some embodiments of this disclosure, the computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium may be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In some embodiments of this disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In some embodiments of this disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0138] In some implementations, clients and servers can communicate using any currently known or future-developed network protocol such as HTTP (Hypertext Transfer Protocol) and can interconnect with digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), the Internet (e.g., the Internet of Things), and end-to-end networks (e.g., ad hoc end-to-end networks), as well as any currently known or future-developed networks.

[0139] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device. The aforementioned computer-readable medium carries one or more programs that, when executed by the electronic device, cause the electronic device to: control an associated monocular RGB camera to acquire a target gesture image; perform gesture keypoint detection on the target gesture image to obtain a set of keypoint coordinate information, wherein the keypoint coordinate information set includes keypoint coordinate information groups corresponding to the gesture regions included in the target gesture image; generate a set of bone length information based on the set of keypoint coordinate information; input the set of bone length information into a pre-trained hand shape parameter generation model to obtain hand shape parameter information; and for each keypoint coordinate information group in the set of keypoint coordinate information, generate gesture depth information based on the keypoint coordinate information group, the hand shape parameter information, and the bone length information group.

[0140] Computer program code for performing operations of some embodiments of this disclosure can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0141] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0142] The units described in some embodiments of this disclosure can be implemented in software or hardware. The described units can also be housed in a processor; for example, a processor may be described as including a control unit, a detection unit, a first generation unit, an input unit, and a second generation unit. The names of these units do not necessarily limit the specific unit; for example, a control unit may also be described as "a unit that controls an associated monocular RGB camera to acquire target gesture images."

[0143] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), System-on-Chip (SoCs), Complex Programmable Logic Devices (CPLDs), and so on.

[0144] The above description is merely a selection of preferred embodiments of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in the embodiments of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described inventive concept. For example, technical solutions formed by substituting the above-described features with (but not limited to) technical features with similar functions disclosed in the embodiments of this disclosure.

Claims

1. A method for generating gesture depth information based on a monocular RGB camera, comprising: Control the associated monocular RGB camera to capture target gesture images; Gesture key point detection is performed on the target gesture image to obtain a set of key point coordinate information, wherein the key point coordinate information set includes key point coordinate information groups that correspond to the gesture regions included in the target gesture image; Generate a bone length information set based on the set of key point coordinate information; The bone length information group is input into a pre-trained hand shape parameter generation model to obtain hand shape parameter information; For each keypoint coordinate information group in the keypoint coordinate information set, gesture depth information is generated based on the keypoint coordinate information group, the hand shape parameter information, and the bone length information group, including: For each key point coordinate information group in the key point coordinate information group set, perform the following steps: The hand shape parameter information and the preset initial posture parameter information are input into the preset parameterized model to obtain a standard key point coordinate information set. The hand shape parameter information represents the shape of the hand. Based on the bone length information set and the standard key point coordinate information set, wrist point depth information is generated; Based on the key point coordinate information group and the standard key point coordinate information set, posture parameter information is generated, and the posture parameter information represents the posture of the hand. The hand shape parameter information and the posture parameter information are input into the preset parameterized model to obtain the hand full degree of freedom information, which includes a grid point coordinate set and a key point coordinate set; Based on the wrist point depth information and the hand's full degrees of freedom information, gesture depth information is generated.

2. The method according to claim 1, wherein, The step of detecting key points in the target gesture image to obtain a set of key point coordinate information includes: The target gesture image is input into the key point coordinate information generation model to obtain a set of key point coordinate information.

3. The method according to claim 1, wherein, The key point coordinate information set includes a key point coordinate sequence in each key point coordinate information group; as well as The step of generating attitude parameter information based on the key point coordinate information group and the standard key point coordinate information set includes: For each key point coordinate information in the key point coordinate information group, select key point coordinates that meet the preset position conditions from the key point coordinate sequence included in the key point coordinate information as palm key point coordinates to obtain palm key point coordinate set. For each palm key point coordinate in the palm key point coordinate set, the standard key point coordinate corresponding to the palm key point coordinate is selected from the standard key point coordinate information set as the palm standard key point coordinate, thus obtaining the palm standard key point coordinate set. Based on the set of key hand points and the set of standard key hand points, generate hand rotation information; For each key point coordinate included in the key point coordinate information group, perform the following steps: The standard key point coordinates corresponding to the key point coordinates in the standard key point coordinate information set are determined as the target standard key point coordinates; Based on the coordinates of the key points and the coordinates of the target standard key points, key point rotation information is generated; The hand rotation information and the generated rotation information of each key point are determined as posture parameter information.

4. The method according to claim 1, wherein, The method further includes: The generated depth information of each gesture is input into a pre-trained gesture recognition model to obtain the gesture type; The operation information corresponding to the gesture type is determined as the target operation information; Execute the application operation corresponding to the target operation information.

5. The method according to claim 2, wherein, The key point coordinate information generation model is trained through the following steps: Obtain a sample set, wherein the samples in the sample set include sample gesture images and a set of sample key point coordinate information corresponding to the sample gesture images; The following training steps are performed based on the aforementioned sample set: The hand gesture image of at least one sample in the sample set is input into the initial neural network to obtain a set of key point coordinate information corresponding to each sample in the at least one sample. Compare the set of key point coordinate information corresponding to each sample in the at least one sample with the set of key point coordinate information of the corresponding sample. Based on the comparison results, determine whether the initial neural network has achieved the preset optimization objective; In response to determining that the initial neural network has reached the optimization objective, the initial neural network is used as a model to generate key point coordinate information after training.

6. The method according to claim 5, wherein, The training steps also include: In response to determining that the initial neural network has not reached the optimization objective, the parameters of the initial neural network are adjusted, and a sample set is formed using unused samples. The adjusted initial neural network is then used as the initial neural network, and the training steps are performed again.

7. A gesture depth information generation device based on a monocular RGB camera, comprising: The control unit is configured to control the associated monocular RGB camera to capture images of the target's gestures; The detection unit is configured to perform gesture key point detection on the target gesture image to obtain a set of key point coordinate information, wherein the key point coordinate information set includes key point coordinate information groups that correspond to the gesture regions included in the target gesture image. The first generation unit is configured to generate a bone length information set based on the set of key point coordinate information. The input unit is configured to input the bone length information group into a pre-trained hand shape parameter generation model to obtain hand shape parameter information; The second generation unit is configured to generate gesture depth information for each keypoint coordinate information group in the keypoint coordinate information set, based on the keypoint coordinate information group, the hand shape parameter information, and the bone length information group, including: For each key point coordinate information group in the key point coordinate information group set, perform the following steps: The hand shape parameter information and the preset initial posture parameter information are input into the preset parameterized model to obtain a standard key point coordinate information set. The hand shape parameter information represents the shape of the hand. Based on the bone length information set and the standard key point coordinate information set, wrist point depth information is generated; Based on the key point coordinate information group and the standard key point coordinate information set, posture parameter information is generated, and the posture parameter information represents the posture of the hand. The hand shape parameter information and the posture parameter information are input into the preset parameterized model to obtain the hand full degree of freedom information, which includes a grid point coordinate set and a key point coordinate set; Based on the wrist point depth information and the hand's full degrees of freedom information, gesture depth information is generated.

8. An electronic device, comprising: One or more processors; Storage device, on which one or more programs are stored, A single RGB camera, When the one or more programs are executed by the one or more processors, the one or more processors control the monocular RGB camera to implement the method as described in any one of claims 1-6.

9. A computer-readable medium having a computer program stored thereon, wherein, When the computer program is executed by a processor, it implements the method as described in any one of claims 1-6.

Images