Palm depth detection method, device, computer equipment and storage medium

Abstract

The embodiment of the present invention provides a palm depth detection method, device, computer equipment and storage medium. The method includes: collecting image data, the image data contains palm data, converting the palm data into a first palm model, initializing a second palm model used to describe the palm, adjusting the second palm model to match the first palm model, and if the adjustment is completed, detecting key points on the second palm model as the depth information of the palm. This embodiment proposes a framework for detecting palm depth. The first palm model can better reflect the posture of the user's real palm, and the second palm model is constructed independently of the image data and can be in accordance with the shape of a human palm, so that the second palm model is as close to the posture of the user's real palm as possible while conforming to the shape of the user's real palm. At this time, the depth information read from the second palm model has a higher accuracy.

Description

Technical Field

[0001] Embodiments of the present invention relate to the technical field of computer vision, and in particular to a palm depth detection method, apparatus, computer equipment, and storage medium. Background Art

[0002] In scenarios such as industrial inspection, autonomous driving, medical imaging, and facial beauty, image data is usually collected from the user's palm and the depth information of the palm is restored from the image data.

[0003] At present, the depth information of the palm is mainly obtained by collecting image data with the palm as samples, manually marking the depth of the palm, and then training the neural network. On the one hand, the number of training samples is limited, and the angle, light, etc. when collecting image data of the palm are different, resulting in distortion of the depth information detected by the neural network, which does not conform to the normal shape of the human palm, for example, the fingers are too long or too short, the finger gaps are too large or too small, etc. On the other hand, the structure of the neural network is single, resulting in a single detection capability, and the neural network needs to be retrained for each update, and the operation and maintenance cost is high. Summary of the invention

[0004] The embodiments of the present invention provide a palm depth detection method, device, computer equipment and storage medium to solve the problems of distortion of palm depth information, single detection capability and high operation and maintenance cost when using a neural network to detect the palm depth.

[0005] In a first aspect, an embodiment of the present invention provides a palm depth detection method, comprising:

[0006] Collecting image data, wherein the image data includes palm data;

[0007] Converting the palm data into a first palm model;

[0008] Initialize a second palm model for describing the palm;

[0009] adjusting the second palm model to match the first palm model;

[0010] If the adjustment is completed, key points on the second palm model are detected as depth information of the palm.

[0011] In a second aspect, an embodiment of the present invention further provides a palm depth detection device, comprising:

[0012] An image data acquisition module, used for acquiring image data, wherein the image data includes palm data;

[0013] A first palm model conversion module, used for converting the palm data into a first palm model;

[0014] A second palm model initialization module, used for initializing a second palm model for describing a palm;

[0015] A palm model adjustment module, used for adjusting the second palm model to match the first palm model;

[0016] The depth information reading module is used to detect key points on the second palm model as the depth information of the palm if the adjustment is completed.

[0017] In a third aspect, an embodiment of the present invention further provides a computer device, the computer device comprising:

[0018] one or more processors;

[0019] a memory for storing one or more programs,

[0020] When the one or more programs are executed by the one or more processors, the one or more processors implement the palm depth detection method as described in the first aspect.

[0021] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the method for detecting the depth of a palm as described in the first aspect is implemented.

[0022] In this embodiment, image data is collected, and the image data contains palm data. The palm data is converted into a first palm model, a second palm model for describing the palm is initialized, and the second palm model is adjusted to match the first palm model. If the adjustment is completed, the key points on the second palm model are detected as the depth information of the palm. This embodiment proposes a framework for detecting the depth of the palm, and the palm data can be observed from the image data to be converted into the first palm model. The first palm model can better reflect the posture of the user's real palm, and the second palm model is constructed independently of the image data and can be in accordance with the shape of the human palm. The second palm model is adjusted with the posture of the first palm model as a reference, so that the second palm model is as close to the posture of the user's real palm as possible while conforming to the shape of the user's real palm. At this time, the depth information read from the second palm model has a high accuracy. The framework does not rely on a specific neural network, can adapt to business scenarios, and can arbitrarily match neural networks to convert palm data into the first palm model. The detection capability is diversified, the existing neural network is reused, and the operation and maintenance cost is low. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 A flowchart of a palm depth detection method provided in Embodiment 1 of the present invention;

[0024] Figure 2A A schematic diagram of a palm skeleton provided in Embodiment 1 of the present invention;

[0025] Figure 2B An example diagram of a joint point provided in the first embodiment of the present invention;

[0026] Figure 3 is a flow chart of a palm depth detection method provided by Embodiment 2 of the present invention;

[0027] Figure 4 is an example diagram of a vector projection provided by the second embodiment of the present invention;

[0028] Figure 5 is an example graph of gradient descent provided by the second embodiment of the present invention;

[0029] Figure 6 A schematic diagram of the structure of a palm depth detection device provided in Embodiment 3 of the present invention;

[0030] Figure 7 A schematic diagram of the structure of a computer device provided in Embodiment 4 of the present invention. DETAILED DESCRIPTION

[0031] The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the present invention, rather than to limit the present invention. It should also be noted that, for ease of description, only parts related to the present invention, rather than all structures, are shown in the accompanying drawings.

[0032] Embodiment 1

[0033] Figure 1 This is a flowchart of a palm depth detection method provided in Embodiment 1 of the present invention. This embodiment can be applied to the case where a second palm model conforming to the normal shape of a human palm is adjusted based on a first palm model observed in image data, so as to read depth information from the second palm model. The method can be executed by a palm depth detection device, which can be implemented by software and / or hardware and can be configured in a computer device, such as a server, a workstation, a personal computer, a mobile terminal (such as a mobile phone, a tablet, etc.), a smart wearable device (such as a smart watch, smart glasses, etc.), etc., and specifically includes the following steps:

[0034] Step 101: Collect image data.

[0035] If this embodiment is applied to a computer device such as a mobile terminal or a smart wearable device as a client, applications that support image processing can be installed in the computer device, such as a live broadcast application, an image editing application, a camera application, an instant messaging tool, a gallery application, and the like.

[0036] Applications such as image editing applications, instant messaging tools, and gallery applications may have a UI (User Interface) that may provide imported controls. Users may operate the imported controls through touch operations or mouse operations to select locally stored video data or image data (represented by thumbnails or paths), or select network stored video data or image data (represented by URLs (Uniform Resource Locators)).

[0037] Applications such as live broadcast applications, image editing applications, camera applications, and instant messaging tools may provide controls for taking photos and recording videos in their UIs. Users may operate the controls by touch or mouse operations to notify the application to call the camera to collect video data or image data.

[0038] If this embodiment is applied to a computer device such as a server or a workstation as a server, it can receive video data or image data sent by a device as a client.

[0039] The video data contains multiple frames of image data, and the computer device can extract the image data from the video data.

[0040] Step 102: Convert the palm data into a first palm model.

[0041] In this embodiment, the image data contains palm data, namely, pixels representing the palm of a human body. Based on the palm data observable in the image data, a model describing the palm data is created, which is recorded as the handpose palm model of the first hand mediapipe.

[0042] In one embodiment of the present invention, step 102 may include the following steps:

[0043] Step 1021: Load the gesture recognition model.

[0044] In this embodiment, the gesture recognition model (structure) and its parameters may be pre-stored. When the palm data is modeled, the gesture recognition model and its parameters may be loaded into the memory to run the gesture recognition mode.

[0045] Among them, the gesture recognition model is usually a neural network, such as CNN (Convolutional Neural Networks), DNN (Deep Neural Networks), RNN (Recurrent Neural Network), etc., which can be used to identify the joint points representing the palm (represented by numbers, coordinate points, etc.) to obtain the palm gesture.

[0046] Furthermore, the gesture recognition model may be a gesture recognition model trained by a third party, such as Handpose of MediaPipe, etc., or a gesture recognition model trained by a third party may be used as a pre-trained model (Pre_trainedmodel), and the gesture recognition model may be fine-tuned or continuously learned, or the structure of the gesture recognition model may be customized and the gesture recognition model may be trained, etc. This embodiment does not limit this.

[0047] Different gesture recognition models have different structures, training samples, and training methods, which makes the detection capabilities (such as accuracy, detection time, etc.) of gesture recognition models in different environments (such as light intensity, background complexity, etc.) different. In this implementation, a suitable gesture recognition model can be selected according to the needs of business scenarios (such as live broadcast, remote action mapping, etc.).

[0048] Step 1022: input the image data into a gesture recognition model to identify coordinate points representing joint points in the palm data.

[0049] The current image data is input into the gesture recognition model, and the gesture recognition model processes the image data according to its own logic, places the palm data in a coordinate system such as the world coordinate system, and identifies the coordinate points representing the joint points in the palm data.

[0050] Since different gesture recognition models may output different formats, after the gesture recognition model outputs the coordinate points representing the joint points, the coordinate points representing the joint points may be mapped to a unified format.

[0051] For example, Figure 2A As shown, generally, a human palm includes a thumb 210 , an index finger 220 , a middle finger 230 , a third finger (ring finger) 240 , and a little finger 250 .

[0052] The palm bones at the base of the palm include the apical bone 201, which includes the pisiform bone, triquetrum bone, lunate bone, scaphoid bone, hamate bone, capitate bone, trapezium bone, trapezium bone, etc. The palm bones in the index finger 220, middle finger 230, ring finger 240, and little finger 250 include the metacarpal bone 202, proximal phalanx 203, middle phalanx 204, and distal phalanx 205. The thumb 210 includes the metacarpal bone 202, proximal phalanx 203, and distal phalanx 205.

[0053] There is a root joint (joint point) between the metacarpal bone 201 and the metacarpal bone 202, a middle interphalangeal joint (joint point) between the metacarpal bone 202 and the proximal phalanx 203, a proximal interphalangeal joint (joint point) between the proximal phalanx 203 and the middle phalanx 204, a distal interphalangeal joint (joint point) between the middle phalanx 204 and the distal phalanx 205, and an interphalangeal joint (joint point) between the proximal phalanx 203 and the distal phalanx 205.

[0054] In addition, to facilitate modeling of palm data, the root joints between the metacarpal bone 201 and the metacarpal bone 202 in the thumb 210, index finger 220, middle finger 230, ring finger 240, and little finger 250 can be merged into one root joint, and fingertip joints (joint points) are set at the end of the distal phalanx 205 of the thumb 210, index finger 220, middle finger 230, ring finger 240, and little finger 250, respectively, and a new joint point is added for the thumb 210 near the trapezium, which is recorded as the root mid-joint.

[0055] Therefore, if Figure 2B As shown, the coordinate points of the palm data detection representing the joint points include:

[0056] Thumb: coordinate point 260 representing the root joint, coordinate point 261 representing the root-middle interphalangeal joint, coordinate point 262 representing the middle interphalangeal joint, coordinate point 263 representing the interphalangeal joint, coordinate point 264 representing the fingertip joint;

[0057] Index finger: coordinate point 260 representing the root joint, coordinate point 265 representing the middle interphalangeal joint, coordinate point 266 representing the proximal interphalangeal joint, coordinate point 267 representing the distal interphalangeal joint, coordinate point 268 representing the fingertip joint;

[0058] Middle finger: coordinate point 260 representing the root joint, coordinate point 269 representing the middle interphalangeal joint, coordinate point 270 representing the proximal interphalangeal joint, coordinate point 271 representing the distal interphalangeal joint, coordinate point 272 representing the fingertip joint;

[0059] Ring finger: coordinate point 260 representing the root joint, coordinate point 273 representing the middle interphalangeal joint, coordinate point 274 representing the proximal interphalangeal joint, coordinate point 275 representing the distal interphalangeal joint, coordinate point 276 representing the fingertip joint;

[0060] Little finger: coordinate point 260 indicating the root joint, coordinate point 277 indicating the middle interphalangeal joint, coordinate point 278 indicating the proximal interphalangeal joint, coordinate point 279 indicating the distal interphalangeal joint, and coordinate point 280 indicating the fingertip joint.

[0061] Step 1023: Generate a first vector in the first palm model based on adjacent coordinate points.

[0062] In a specific implementation, the coordinate points on each finger can be sorted from the palm to the fingertip. For two adjacent coordinate points, the data point sorted first can be used as the starting point to point to the data point sorted later to obtain a first vector, so that the first vector represents the palm bones between the joint points corresponding to the adjacent coordinate points. These first vectors are combined into a specific data structure to represent the first palm model.

[0063] For example, for the palm data shown in FIG2 , a first palm model with the following tree structure may be constructed:

[0064]

[0065] Furthermore, the settings for each finger are as follows:

[0066] thumb:

[0067] index finger:

[0068] Middle finger:

[0069] Ring finger:

[0070] Little finger:

[0071] Thumb index finger Middle finger Ring finger little finger Then the first palm model can be expressed as

[0072] In some cases, considering that the thumb is relatively flexible, the index finger, middle finger, ring finger, and little finger can roughly reflect the posture of the palm. The thumb can be omitted and the index finger, middle finger, ring finger, and little finger can be used to model the palm data. In this case, the first palm model It can be expressed as

[0073] Step 103: Initialize a second palm model for describing the palm.

[0074] In this embodiment, a 3D engine can be started, for example, the Unity engine, the Unreal engine, the Cocos2dx engine, a self-developed engine, etc., and the 3D engine can render a virtual palm that conforms to the shape of a human palm (such as the length ratio between each finger, the range of bones, the range of finger gaps, the degree of bending of the joints, etc.), and the virtual palm can be described by a second palm model.

[0075] Furthermore, the form of describing the first palm model is consistent with the form of describing the second palm model. For example, if the first palm model is represented as Then the second palm model can be expressed as

[0076] The gesture data describing the palm is transmitted to the 3D engine and mapped to a virtual palm, so that the virtual palm appears as the gesture data.

[0077] Among them, the posture data can be a default value, or it can be set according to ecological parameters (such as gender, age, height, etc.) input by the current user and related to the palm shape, so that the virtual palm is closer to the user's real palm shape, etc. This embodiment does not limit this.

[0078] Taking the Unity engine as an example, the Unity engine receives data packets (i.e., posture data) through the network, unpacks the data and converts the data into Unity's quaternion. If the quaternion is not in the world coordinate system, the quaternion can be transformed into Unity's world coordinate system to facilitate subsequent processing.

[0079] In the Unity engine, the virtual palm can be generated by means of skinned mesh animation. The so-called skinned mesh animation refers to the dynamic calculation of the vertices of the skinned mesh through vertex blending under the control of bones, and the movement of the bones is relative to their parent bones and driven by animation keyframe data. A skinned mesh animation usually includes bone hierarchy data, mesh data, mesh skin data (skin info) and bone animation (keyframe) data.

[0080] In the Unity engine, for the posture data passed in by the specified virtual palm, the BindPose algorithm is used to render the virtual palm. The BindPose algorithm is as follows:

[0081] OneBoneBindPose=bone.worldToLocalMatrix*transform.localToWorldMatrix

[0082] The pose is the world-to-local coordinate matrix of the bone, worldToLocalMatrix, multiplied by the local-to-world matrix of the Mesh, localToWorldMatrix.

[0083] The process of calculating the changes in the vertices of a Mesh affected by the bone is usually called skinning. The general process is:

[0084] Transform vertices from mesh space to bone space:

[0085] v bone =v mesh *bindpose

[0086] Transform the vertices in the bone space from the bone space to the mesh space through the current bone's transformation matrix:

[0087] v out =v bone *boneToMeshMatrix

[0088] A vertex may be affected by multiple bones, so the final result is the transformation of the affected bones, multiplied by the weights in boneWeights and added together to get the result:

[0089] v out0 =boneToMeshMatrix0*bindpose0*v mesh

[0090] v out1 =boneToMeshMatrix1*bindpose1*v mesh

[0091] v out2 =boneToMeshMatrix2*bindpose2*v mesh

[0092] v out3 =boneToMeshMatrix3*bindpose3*v mesh

[0093] v out =v out0 *weight0+v out1 *weight1+vout2 *weight2+v out3 *weight3

[0094] Step 104: adjust the second palm model to match the first palm model.

[0095] The first palm model is observed from the image data. Although there is more or less distortion, it can still better reflect the posture of the user's real palm. The second palm model conforms to the shape of the human palm, but the posture during initialization is a default posture, which is different from the posture of the user's real palm. Therefore, in this embodiment, the posture of the first palm model can be used as a reference to adjust the second palm model so that the second palm model matches the first palm model, that is, the posture expressed by the second palm model is similar to the posture expressed by the first palm model.

[0096] Here, the so-called similarity may mean that the difference between the second palm model and the first palm model is within a reasonable error range.

[0097] In one embodiment of the present invention, the first palm model has multiple first vectors representing palm bones, and the second palm model has multiple second vectors representing palm bones. For the index finger, middle finger, ring finger, and little finger, the palm bones include metacarpal bones, proximal phalanges, middle phalanges, and distal phalanges in sequence from the base of the palm to the fingertips.

[0098] In a specific implementation, for each finger, the second vector representing the metacarpal bone can be rotated according to a pattern (such as a rotation matrix) until the second vector representing the metacarpal bone matches the first vector representing the metacarpal bone, so that the metacarpal bone in the second finger model matches the metacarpal bone in the first finger.

[0099] If the rotation of the second vector representing the metacarpal bone is completed, then for each finger, the second vector representing the proximal phalanx can be rotated according to a pattern (such as a rotation matrix) until the second vector representing the proximal phalanx matches the first vector representing the proximal phalanx, so that the proximal phalanx in the second finger model matches the proximal phalanx in the first finger.

[0100] If the rotation of the second vector representing the proximal phalanx is completed, the second vector representing the middle phalanx can be rotated according to a pattern (such as a rotation matrix) for each finger until the second vector representing the middle phalanx matches the first vector representing the middle phalanx, so that the middle phalanx in the second finger model matches the middle phalanx in the first finger.

[0101] If the rotation of the second vector representing the middle phalanx is completed, then for each finger, the second vector representing the distal phalanx can be rotated according to a pattern (such as a rotation matrix) until the second vector representing the distal phalanx matches the first vector representing the distal phalanx, so that the distal phalanx in the second finger model matches the distal phalanx in the first finger.

[0102] If the rotation of the second vector representing the distal phalanx is completed, it can be considered that the posture of the second palm model is adjusted to match the posture of the first palm model.

[0103] The so-called matching of the second vector with the first vector may refer to that the difference between the second vector and the first vector is within a reasonable error range.

[0104] In this embodiment, the joints in the second palm model are rotated in sequence according to the order of bones in the human palm, so that the posture of the second palm model is adjusted to match the posture of the first palm model while ensuring that the second palm model conforms to the shape of the human palm.

[0105] Step 105: If the adjustment is completed, key points on the second palm model are detected as depth information of the palm.

[0106] By adjusting the second palm model, the second palm model can be made to conform to the shape of the user's real palm and be as close to the posture of the user's real palm as possible. At this time, the key points in the adjusted second palm model can be output as the depth information of the palm.

[0107] In this embodiment, image data is collected, and the image data contains palm data. The palm data is converted into a first palm model, a second palm model for describing the palm is initialized, and the second palm model is adjusted to match the first palm model. If the adjustment is completed, the key points on the second palm model are detected as the depth information of the palm. This embodiment proposes a framework for detecting the depth of the palm, and the palm data can be observed from the image data to be converted into the first palm model. The first palm model can better reflect the posture of the user's real palm, and the second palm model is constructed independently of the image data and can be in accordance with the shape of the human palm. The second palm model is adjusted with the posture of the first palm model as a reference, so that the second palm model is as close to the posture of the user's real palm as possible while conforming to the shape of the user's real palm. At this time, the depth information read from the second palm model has a high accuracy. The framework does not rely on a specific neural network, can adapt to business scenarios, and can arbitrarily match neural networks to convert palm data into the first palm model. The detection capability is diversified, the existing neural network is reused, and the operation and maintenance cost is low.

[0108] Embodiment 2

[0109] Figure 3 This is a flow chart of a palm depth detection method provided in Embodiment 2 of the present invention. This embodiment is based on the above embodiment and further refines the operation of adjusting the second palm model. The method specifically includes the following steps:

[0110] Step 301: Collect image data.

[0111] The image data includes palm data.

[0112] Step 302: Convert the palm data into a first palm model.

[0113] Step 303: Initialize a second palm model for describing the palm.

[0114] Step 304: Calculate the rotation mode for the first vector and the second vector representing the same palm bone.

[0115] In this embodiment, the first palm model has a plurality of first vectors representing palm bones. The second palm model has a plurality of second vectors representing palm bones These first vectors can be Second vector For each finger, it is divided into the first vector representing the same palm bone With the second vector Denoted as vector pair.

[0116] For each vector pair, the first vector The posture and the second vector The posture is consistent, so that the first vector With the second vector Matching, this process can be regarded as the following optimization problem:

[0117]

[0118] Second vector After rotating according to the rotation mode R (such as the third-order rotation matrix), it is rotated with the second vector Minimize the difference between them.

[0119] In one embodiment of the present invention, step 304 includes the following steps:

[0120] Step 3041: Project the first vector onto a preset plane to obtain a first projection vector.

[0121] In the first vector set In the example, given the first vector d=(d x ,d y ,d z), it can be projected to any plane in the world coordinate system. At this time, the first projection vector forms a first angle with the specified axis on the plane.

[0122] For example, Figure 4 As shown, the first vector d=(d x ,d y ,d z ) is projected onto the preset XY plane to obtain the first projection vector d xy =(d x ,d y ,0), where the first projection vector d xy Forming a first angle α with the X axis on the XY plane d , at this time, X=(1,0,0), then:

[0123]

[0124] Of course, the first vector may also be projected onto the YZ plane or the XZ plane to obtain a first projection vector, and the first projection vector forms a first angle with the Y axis or the Z axis, which is not limited in this embodiment.

[0125] Step 3042: Project the second vector rotated by the target angle around the axis onto the plane to obtain a second projection vector.

[0126] In the second vector set In the example, given the second vector m=(m x ,m y ,m z ), it can be projected to any plane in the world coordinate system. At this time, the second projection vector forms a second angle with the axis on the plane.

[0127] Furthermore, the plane projected by the first vector is the same as the plane projected by the second vector, and the axis forming the first angle is the same as the axis forming the second angle.

[0128] For example, Figure 4 As shown, the second vector m=(m x ,m y ,m z ) is projected onto the XY plane to obtain the second projection vector m xy =(m x ,m y ,0), where the second projection vector m xy Forming a second angle α with the X axis on the XY plane m .

[0129] Of course, the second vector may also be projected onto the YZ plane or the XZ plane to obtain a second projection vector, and the second projection vector forms a second angle with the Y axis or the Z axis, which is not limited in this embodiment.

[0130] In this embodiment, the goal of adjusting the second vector is to rotate the second vector m around a specified axis (such as the X axis) by a target angle θ so that α m =α d .

[0131] The target angle may initially be a random value, a default value, etc., which is not limited in this embodiment.

[0132] Rotates the second vector around the specified axis by the target angle. At this time, the rotation mode changes.

[0133] For example, Figure 4 As shown, the rotation mode R is a third-order rotation matrix, which rotates the second vector around the X-axis by the target angle θ. The rotation matrix R can be obtained by Rodriguez's formula:

[0134]

[0135] Therefore, the second vector m is Figure 4 After the dotted line in is rotated, the second vector m R for:

[0136]

[0137] Then, the second vector m after rotation is R The second projection vector m projected on the XY plane Rxy for:

[0138]

[0139] Step 3043: Set the objective function to be the difference between the second angle and the first angle.

[0140] In this embodiment, an objective function is set, and the objective function uses the target angle θ as a parameter, denoted as f(θ), to measure the difference between the second angle and the first angle.

[0141] Exemplarily, the objective function may be set to the square of the angle difference, where the angle difference is the difference between the second angle and the first angle.

[0142] Then the objective function f(θ) is expressed as:

[0143] f(θ)=(α m -α d ) 2

[0144] Step 3044, taking minimizing the difference as the optimization goal, solving the target angle.

[0145] When the objective function f(θ) is minimum, the first vector and the second vector can be matched. At this time, the value of the target angle θ is the optimal rotation amount θ best :

[0146]

[0147] In a specific implementation, the objective function can be optimized by using optimization methods such as gradient descent method, stochastic gradient descent method, Newton's method, etc. to solve the optimal value of the target angle.

[0148] Taking the gradient descent method as an example, the objective function can be used to derive the target angle to obtain the gradient value.

[0149] Multiply the preset step size by the gradient value to obtain the update amount, and subtract the update amount from the target angle to obtain the new target angle.

[0150] Determine whether the target angle meets a preset iteration condition, for example, the number of iterations is greater than or equal to a preset first threshold, and the difference between the second angle and the first angle is less than or equal to a preset second threshold.

[0151] If yes, output the target angle.

[0152] If not, return to using the objective function to derive the target angle and obtain the gradient.

[0153] Then, in this example, the iterative process of the target angle can be expressed as:

[0154]

[0155] Among them, lr is the step size.

[0156] like Figure 5 As shown, the points are the values ​​of the target angle θ in each iteration, and the lines are the minimum value area of ​​the objective function f(θ). In this example, when the target angle θ is approximately 15°, the objective function f(θ) is minimum, and the rotation is basically correct.

[0157] Step 305: Rotate the second vector according to the rotation mode to make the second vector match the first vector.

[0158] For each vector pair, if the optimal rotation mode is solved, the second vector can be rotated according to the rotation mode, so that the first vector matches the second vector.

[0159] Furthermore, the rotation mode may be to rotate a target angle around a specified axis, which may be represented by a third-order rotation matrix. In this case, the second vector may be multiplied by the rotation matrix to obtain the rotated second vector.

[0160] Step 306: If the rotation is completed, detect key points on the second palm model as depth information of the palm.

[0161] If the rotation of each second vector is completed, the key points on the second palm model can be read and recorded as the depth information of the palm.

[0162] It should be noted that, for the sake of simplicity, the method embodiments are described as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present invention are not limited by the order of the actions described, because according to the embodiments of the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present invention.

[0163] Embodiment 3

[0164] Figure 6 This is a structural block diagram of a palm depth detection device provided in Embodiment 3 of the present invention, which may specifically include the following modules:

[0165] An image data acquisition module 601 is used to acquire image data, wherein the image data includes palm data;

[0166] A first palm model conversion module 602, configured to convert the palm data into a first palm model;

[0167] A second palm model initialization module 603, used to initialize a second palm model used to describe a palm;

[0168] A palm model adjustment module 604, configured to adjust the second palm model to match the first palm model;

[0169] The depth information reading module 605 is used to detect key points on the second palm model as the depth information of the palm if the adjustment is completed.

[0170] In one embodiment of the present invention, the first palm model conversion module 602 includes:

[0171] A gesture recognition model loading module, used to load a gesture recognition model, wherein the gesture recognition model is used to recognize joint points representing the palm;

[0172] A gesture recognition model calling module, used for inputting the image data into the gesture recognition model and identifying coordinate points representing joint points in the palm data;

[0173] A vector generation module is used to generate a first vector in a first palm model based on the adjacent coordinate points, wherein the first vector represents the palm bones between the joint points corresponding to the adjacent coordinate points.

[0174] In one embodiment of the present invention, the first palm model has a plurality of first vectors representing palm bones, the second palm model has a plurality of second vectors representing palm bones, and the palm bones sequentially include metacarpal bones, proximal phalanges, middle phalanges, and distal phalanges;

[0175] The palm model adjustment module 604 includes:

[0176] A metacarpal bone rotation module, configured to rotate the second vector representing the metacarpal bone until the second vector representing the metacarpal bone matches the first vector representing the metacarpal bone;

[0177] A proximal phalanx rotation module, configured to rotate the second vector representing the proximal phalanx if the rotation of the second vector representing the metacarpal bone is completed, until the second vector representing the proximal phalanx matches the first vector representing the proximal phalanx;

[0178] A middle phalanx rotation module, configured to rotate the second vector representing the middle phalanx if the rotation of the second vector representing the proximal phalanx is completed, until the second vector representing the middle phalanx matches the first vector representing the middle phalanx;

[0179] The distal phalanx rotation module is used to rotate the second vector representing the distal phalanx if the rotation of the second vector representing the middle phalanx is completed, until the second vector representing the distal phalanx matches the first vector representing the distal phalanx.

[0180] In another embodiment of the present invention, the first palm model has a plurality of first vectors representing palm bones, and the second palm model has a plurality of second vectors representing palm bones;

[0181] The palm model adjustment module 604 includes:

[0182] A rotation mode calculation module, used for calculating a rotation mode for the first vector and the second vector representing the same palm bone;

[0183] A vector rotation module is used to rotate the second vector according to the rotation mode so that the second vector matches the first vector.

[0184] In one embodiment of the present invention, the rotation mode calculation module includes:

[0185] A first vector projection module, used for projecting the first vector onto a preset plane to obtain a first projection vector, wherein the first projection vector forms a first angle with a specified axis on the plane;

[0186] A second vector projection module, used for projecting the second vector rotated around the axis by a target angle onto the plane to obtain a second projection vector, wherein the second projection vector forms a second angle with the axis on the plane;

[0187] An objective function setting module, used to set the objective function as the difference between the second angle and the first angle;

[0188] The target angle solving module is used to solve the target angle with minimizing the difference as the optimization goal.

[0189] In an example of an embodiment of the present invention, the first vector projection module is further used for:

[0190] Projecting the first vector onto a preset XY plane to obtain a first projection vector, wherein the first projection vector forms a first angle with an X-axis on the XY plane;

[0191] The second vector projection module is also used for:

[0192] The second vector is projected onto the XY plane to obtain a second projection vector, wherein the second projection vector forms a second angle with the X-axis on the XY plane.

[0193] In an example of an embodiment of the present invention, the objective function setting module is further used to:

[0194] The objective function is set to the square of the angle difference, where the angle difference is the difference between the second angle and the first angle.

[0195] In an example of an embodiment of the present invention, the target angle solving module is further used for:

[0196] Using the objective function to derive the target angle, to obtain a gradient value;

[0197] Multiplying the preset step length by the gradient value to obtain an update amount;

[0198] Subtract the update amount from the target angle to obtain a new target angle;

[0199] Determining whether the target angle satisfies a preset iteration condition;

[0200] If yes, output the target angle;

[0201] If not, return to execute the process of using the objective function to derive the target angle to obtain the gradient.

[0202] The palm depth detection device provided in the embodiment of the present invention can execute the palm depth detection method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0203] Embodiment 4

[0204] Figure 7 A schematic diagram of the structure of a computer device provided in Embodiment 4 of the present invention. Figure 7 A block diagram of an exemplary computer device 12 suitable for use in implementing embodiments of the present invention is shown. Figure 7 The computer device 12 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0205] like Figure 7 As shown, the computer device 12 is in the form of a general-purpose computing device. The components of the computer device 12 may include, but are not limited to: one or more processors or processing units 16, a system memory 28, and a bus 18 that connects various system components (including the system memory 28 and the processing unit 16).

[0206] Bus 18 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processor or a local bus using any of a variety of bus architectures. By way of example, these architectures include, but are not limited to, an Industry Standard Architecture (ISA) bus, a Micro Channel Architecture (MAC) bus, an Enhanced ISA bus, a Video Electronics Standards Association (VESA) local bus, and a Peripheral Component Interconnect (PCI) bus.

[0207] The computer device 12 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by the computer device 12, including volatile and non-volatile media, removable and non-removable media.

[0208] The system memory 28 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 30 and / or cache memory 32. The computer device 12 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, the storage system 34 may be used to read and write non-removable, non-volatile magnetic media ( Figure 7 not shown, usually called a "hard drive"). Although Figure 7 Not shown in the figure, a disk drive for reading and writing to a removable non-volatile disk (e.g., a "floppy disk"), and an optical disk drive for reading and writing to a removable non-volatile optical disk (e.g., a CD-ROM, a DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to the bus 18 via one or more data medium interfaces. The memory 28 may include at least one program product having a set (e.g., at least one) of program modules that are configured to perform the functions of the various embodiments of the present invention.

[0209] A program / utility 40 having a set (at least one) of program modules 42 may be stored, for example, in the memory 28, such program modules 42 including, but not limited to, an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment. The program modules 42 generally perform the functions and / or methods of the embodiments described herein.

[0210] The computer device 12 may also communicate with one or more external devices 14 (e.g., keyboards, pointing devices, displays 24, etc.), may communicate with one or more devices that enable a user to interact with the computer device 12, and / or may communicate with any device that enables the computer device 12 to communicate with one or more other computing devices (e.g., network cards, modems, etc.). Such communication may be performed via an input / output (I / O) interface 22. Furthermore, the computer device 12 may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via a network adapter 20. As shown, the network adapter 20 communicates with other modules of the computer device 12 via a bus 18. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the computer device 12, including, but not limited to, microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0211] The processing unit 16 executes various functional applications and data processing by running the programs stored in the system memory 28, such as implementing the palm depth detection method provided in the embodiment of the present invention.

[0212] Embodiment 5

[0213] Embodiment 5 of the present invention further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the various processes of the above-mentioned palm depth detection method are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be described here.

[0214] Among them, computer-readable storage media may include, for example, but are not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or devices, or any combination of the above. More specific examples of computer-readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this document, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, device or device.

[0215] Note that the above are only preferred embodiments of the present invention and the technical principles used. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and that various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in more detail through the above embodiments, the present invention is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the present invention, and the scope of the present invention is determined by the scope of the appended claims.

Claims

1. A palm depth detection method, characterized in that: include: Collecting image data, wherein the image data includes palm data; Converting the palm data into a first palm model; Initialize a second palm model for describing the palm; adjusting the second palm model to match the first palm model; If the adjustment is completed, detecting key points on the second palm model as depth information of the palm; The first palm model has a plurality of first vectors representing palm bones, and the second palm model has a plurality of second vectors representing palm bones; wherein the second vectors match the first vectors, and the difference between the second vectors and the first vectors is within a reasonable error range; The converting the palm data into a first palm model comprises: Loading a gesture recognition model, wherein the gesture recognition model is used to identify joint points representing a palm; Inputting the image data into the gesture recognition model, identifying coordinate points representing joint points in the palm data; A first vector in a first palm model is generated based on the adjacent coordinate points, wherein the first vector represents the palm bones between the joint points corresponding to the adjacent coordinate points.

2. The method according to claim 1, characterized in that The palm bones include metacarpal bones, proximal phalanges, middle phalanges, and distal phalanges in sequence; The adjusting the second palm model to match the first palm model comprises: Rotating the second vector representing the metacarpal bone until the second vector representing the metacarpal bone matches the first vector representing the metacarpal bone; If the rotation of the second vector representing the metacarpal bone is completed, the second vector representing the proximal phalanx is rotated until the second vector representing the proximal phalanx matches the first vector representing the proximal phalanx; If the rotation of the second vector representing the proximal phalanx is completed, the second vector representing the middle phalanx is rotated until the second vector representing the middle phalanx matches the first vector representing the middle phalanx; If the rotation of the second vector representing the middle phalanx is completed, the second vector representing the distal phalanx is rotated until the second vector representing the distal phalanx matches the first vector representing the distal phalanx.

3. The method according to any one of claims 1 to 2, characterized in that: The first palm model has a plurality of first vectors representing palm bones, and the second palm model has a plurality of second vectors representing palm bones; The adjusting the second palm model to match the first palm model comprises: Calculate a rotation mode for the first vector and the second vector representing the same palm bone; The second vector is rotated according to the rotation mode so that the second vector matches the first vector.

4. The method according to claim 3, characterized in that The step of calculating the rotation mode for the first vector and the second vector representing the same palm bone comprises: Projecting the first vector onto a preset plane to obtain a first projection vector, wherein the first projection vector forms a first angle with a specified axis on the plane; Projecting the second vector rotated by a target angle around the axis onto the plane to obtain a second projection vector, wherein the second projection vector forms a second angle with the axis on the plane; Setting the objective function to be the difference between the second angle and the first angle; The target angle is solved by minimizing the difference as the optimization goal.

5. The method according to claim 4, characterized in that The projecting the first vector onto a preset plane to obtain a first projection vector includes: Projecting the first vector onto a preset XY plane to obtain a first projection vector, wherein the first projection vector forms a first angle with an X-axis on the XY plane; The step of projecting the second rotated vector onto the plane to obtain a second projection vector includes: The second vector is projected onto the XY plane to obtain a second projection vector, wherein the second projection vector forms a second angle with the X-axis on the XY plane.

6. The method according to claim 4, characterized in that The setting objective function is the difference between the second angle and the first angle, comprising: The objective function is set to the square of the angle difference, where the angle difference is the difference between the second angle and the first angle.

7. The method according to claim 4, characterized in that The step of solving the target angle with minimizing the difference as the optimization goal includes: Using the objective function to derive the target angle, to obtain a gradient value; Multiplying the preset step length by the gradient value to obtain an update amount; Subtract the update amount from the target angle to obtain a new target angle; Determining whether the target angle satisfies a preset iteration condition; If yes, output the target angle; If not, return to execute the process of using the objective function to derive the target angle to obtain the gradient.

8. A palm depth detection device, characterized in that: include: An image data acquisition module, used for acquiring image data, wherein the image data includes palm data; A first palm model conversion module, used for converting the palm data into a first palm model; A second palm model initialization module, used for initializing a second palm model for describing a palm; A palm model adjustment module, used for adjusting the second palm model to match the first palm model; A depth information reading module, configured to detect key points on the second palm model as depth information of the palm if the adjustment is completed; The first palm model has a plurality of first vectors representing palm bones, and the second palm model has a plurality of second vectors representing palm bones; wherein the second vectors match the first vectors, and the difference between the second vectors and the first vectors is within a reasonable error range; The first palm model conversion module includes: A gesture recognition model loading module, used to load a gesture recognition model, wherein the gesture recognition model is used to recognize joint points representing the palm; A gesture recognition model calling module, used for inputting the image data into the gesture recognition model and identifying coordinate points representing joint points in the palm data; A vector generation module is used to generate a first vector in a first palm model based on the adjacent coordinate points, wherein the first vector represents the palm bones between the joint points corresponding to the adjacent coordinate points.

9. A computer device, characterized in that: The computer device comprises: one or more processors; a memory for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the palm depth detection method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the palm depth detection method according to any one of claims 1 to 7 is implemented.

Images