Hand key point detection method and device, computer device and storage medium

Abstract

This application discloses a method, apparatus, computer device, and storage medium for detecting hand key points, belonging to the field of computer technology. The method includes: acquiring feature information from a hand image; performing feature transformation on the feature information to obtain pose parameters and shape parameters corresponding to the hand; constructing a three-dimensional hand model based on the pose and shape parameters; determining the three-dimensional hand key points contained in the three-dimensional hand model; and projecting the three-dimensional hand key points onto the hand image to obtain two-dimensional hand key points in the hand image. The method provided in this application constructs a three-dimensional hand model contained in the hand image to constrain the positions of the three-dimensional hand key points in the three-dimensional hand model, thereby ensuring the accuracy of the three-dimensional hand key points. Then, using the three-dimensional hand key points as constraints, the corresponding two-dimensional hand key points are projected onto the hand image, thereby improving the accuracy of the two-dimensional hand key points.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a method, apparatus, computer device, and storage medium for detecting key points of a hand. Background Technology

[0002] With the development of computer technology, image processing technology is being applied more and more widely. In various scenarios, such as sign language translation or motion generation, multiple key points in a hand image are identified to determine the hand posture represented by these key points, thereby enabling the understanding of the hand image.

[0003] In related technologies, keypoint detection is performed directly on a hand image to obtain two-dimensional hand keypoints. However, directly performing keypoint detection on the hand image results in poor accuracy of the detected two-dimensional hand keypoints. Summary of the Invention

[0004] This application provides a method, apparatus, computer device, and storage medium for detecting key points of a hand, which can improve the accuracy of two-dimensional hand key points. The technical solution is as follows:

[0005] On the one hand, a method for detecting key points of a hand is provided, the method comprising:

[0006] Obtain feature information from a hand image, wherein the hand image includes a hand;

[0007] The feature information is transformed to obtain the posture parameters and shape parameters corresponding to the hand.

[0008] Based on the posture parameters and the shape parameters, a three-dimensional hand model is constructed, and the three-dimensional hand key points contained in the three-dimensional hand model are determined.

[0009] The three-dimensional hand key points are projected onto the hand image to obtain the two-dimensional hand key points in the hand image.

[0010] In one possible implementation, the step of calling the keypoint detection model to perform keypoint detection on the sample hand image, and obtaining the sample two-dimensional hand keypoints and the confidence scores corresponding to the sample two-dimensional hand keypoints, includes:

[0011] The key point detection model is invoked to perform key point detection on the sample hand image, resulting in multiple heatmaps corresponding to the sample hand image. Each heatmap includes multiple location points and a confidence score for each location point. The confidence score is used to indicate the probability that the location point in the sample hand image is a key point of the sample two-dimensional hand.

[0012] The location point corresponding to the maximum confidence level in each heatmap is determined as the location point corresponding to the sample hand image, and is respectively identified as the sample two-dimensional hand key point.

[0013] In another possible implementation, the viewpoint parameters include translation parameters, rotation parameters, and scale parameters. The translation parameters are used to indicate the position of the 3D hand model relative to the origin of the camera coordinate system. The rotation parameters are used to indicate the rotation angle of the 3D hand model relative to the coordinate axes of the camera coordinate system. The scale parameters are used to indicate the scaling scale of the 3D hand model.

[0014] The step of transforming the 3D hand key points to the camera coordinate system based on the viewpoint parameters to obtain the reference 3D hand key points includes:

[0015] Determine the first product of the scale parameter and the coordinates of the three-dimensional hand key points;

[0016] Determine the first product and the second product of the rotation parameter;

[0017] The coordinates of the reference three-dimensional hand key points are determined by summing the second product corresponding to the three-dimensional hand key points with the translation parameter.

[0018] In another possible implementation, the viewpoint parameters include translation parameters, rotation parameters, and scale parameters. The translation parameters indicate the position of the 3D hand model relative to the origin of the camera coordinate system. The rotation parameters indicate the rotation angle of the 3D hand model relative to the coordinate axes of the camera coordinate system. The scale parameters indicate the scaling scale of the 3D hand model. The viewpoint parameter extraction layer includes a translation parameter extraction layer, a rotation parameter extraction layer, and a scale parameter extraction layer.

[0019] The step of calling the viewpoint parameter extraction layer in the parameter extraction model to perform feature transformation on the reference feature vector to obtain the viewpoint parameters includes:

[0020] The translation parameter extraction layer, the rotation parameter extraction layer, and the scale parameter extraction layer are invoked to perform feature transformation on the reference feature vector to obtain the translation parameter, the rotation parameter, and the scale parameter.

[0021] In another possible implementation, after constructing a three-dimensional hand model based on the pose parameters and the shape parameters, and determining the three-dimensional hand key points included in the three-dimensional hand model, the method further includes:

[0022] The three-dimensional hand model is transformed into the camera coordinate system to obtain the target three-dimensional hand model in the camera coordinate system.

[0023] On the other hand, a hand key point detection device is provided, the device comprising:

[0024] The first acquisition module is used to acquire feature information of a hand image, wherein the hand image includes a hand;

[0025] The feature transformation module is used to perform feature transformation on the feature information to obtain the posture parameters and shape parameters corresponding to the hand.

[0026] The determination module is used to construct a three-dimensional hand model of the hand based on the posture parameters and the shape parameters, and to determine the three-dimensional hand key points contained in the three-dimensional hand model;

[0027] The projection module is used to project the three-dimensional hand key points into the hand image to obtain the two-dimensional hand key points in the hand image.

[0028] In one possible implementation, the determining module includes:

[0029] The first determining unit is used to call the hand construction model, process the posture parameters and the shape parameters to obtain the three-dimensional hand model, and determine the three-dimensional hand key points contained in the three-dimensional hand model.

[0030] In another possible implementation, the three-dimensional hand model is composed of multiple vertices connected together; the first determining unit is used to determine the vertex number of the multiple vertices; and the vertex corresponding to the key point number among the determined vertex numbers is determined as the key point of the three-dimensional hand.

[0031] In another possible implementation, the device further includes:

[0032] The second acquisition module is used to acquire a sample hand image, and the sample pose parameters and sample shape parameters corresponding to the sample hand contained in the sample hand image;

[0033] The detection module is also used to call the key point detection model to perform key point detection on the sample hand image and obtain the sample two-dimensional hand key points in the sample hand image;

[0034] The determining module is further configured to call the hand construction model, process the sample pose parameters and the sample shape parameters to obtain a predicted three-dimensional hand model, and determine the predicted three-dimensional hand key points contained in the predicted three-dimensional hand model.

[0035] The projection module is also used to project the predicted three-dimensional hand key points into the sample hand image to obtain the predicted two-dimensional hand key points in the sample hand image.

[0036] The training module is used to train the hand construction model based on the sample two-dimensional hand key points and the predicted two-dimensional hand key points.

[0037] In another possible implementation, the training module includes:

[0038] The second determining unit is used to determine the loss value of the hand construction model based on the sample two-dimensional hand key points and the predicted two-dimensional hand key points;

[0039] A training unit is used to train the hand construction model based on the loss value.

[0040] In another possible implementation, the detection module includes:

[0041] The detection unit is used to call the key point detection model to perform key point detection on the sample hand image, and obtain the sample two-dimensional hand key points and the confidence level corresponding to the sample two-dimensional hand key points;

[0042] The second determining unit is used to perform weighted processing on the distance between the sample two-dimensional hand key points and the predicted two-dimensional hand key points according to the confidence level, so as to obtain the loss value.

[0043] In another possible implementation, the sample two-dimensional hand key points include multiple points, and the predicted two-dimensional hand key points include multiple points;

[0044] The second determining unit is used to determine the distance between each sample two-dimensional hand key point and the corresponding predicted two-dimensional hand key point; determine the product of each distance and the confidence level corresponding to the sample two-dimensional hand key point; and determine the sum of the determined multiple products as the loss value.

[0045] In another possible implementation, the detection unit is used to invoke the keypoint detection model to perform keypoint detection on the sample hand image, obtaining multiple heatmaps corresponding to the sample hand image. Each heatmap includes multiple location points and a confidence level corresponding to each location point. The confidence level is used to represent the probability that the location point corresponding to the location point in the sample hand image is a sample two-dimensional hand keypoint. The location point corresponding to the maximum confidence level in each heatmap is determined as the location point corresponding to the sample hand image, respectively, as the sample two-dimensional hand keypoint.

[0046] In another possible implementation, the projection module includes:

[0047] The coordinate transformation unit is used to transform the three-dimensional hand key points into the camera coordinate system to obtain reference three-dimensional hand key points;

[0048] The projection unit is used to project the reference three-dimensional hand key points into the hand image according to the shooting parameters corresponding to the hand image, so as to obtain the two-dimensional hand key points. The shooting parameters are the shooting parameters used when shooting the hand image.

[0049] In another possible implementation, the coordinate transformation unit is used to obtain the viewpoint parameters corresponding to the hand image, the viewpoint parameters being used to indicate the display position of the hand in the camera coordinate system; based on the viewpoint parameters, the three-dimensional hand key points are transformed to the camera coordinate system to obtain the reference three-dimensional hand key points.

[0050] In another possible implementation, the viewpoint parameters include translation parameters, rotation parameters, and scale parameters. The translation parameters are used to indicate the position of the 3D hand model relative to the origin of the camera coordinate system. The rotation parameters are used to indicate the rotation angle of the 3D hand model relative to the coordinate axes of the camera coordinate system. The scale parameters are used to indicate the scaling scale of the 3D hand model.

[0051] The coordinate transformation unit is used to determine the first product of the scale parameter and the coordinates of the three-dimensional hand key point; determine the second product of the first product and the rotation parameter; and sum the second product corresponding to the three-dimensional hand key point with the translation parameter to determine the coordinates of the reference three-dimensional hand key point.

[0052] In another possible implementation, the coordinate transformation unit is used to call the dimension transformation layer in the parameter extraction model to perform dimension transformation on the feature information to obtain a reference feature vector; and to call the viewpoint parameter extraction layer in the parameter extraction model to perform feature transformation on the reference feature vector to obtain the viewpoint parameters.

[0053] In another possible implementation, the viewpoint parameters include translation parameters, rotation parameters, and scale parameters. The translation parameters indicate the position of the 3D hand model relative to the origin of the camera coordinate system. The rotation parameters indicate the rotation angle of the 3D hand model relative to the coordinate axes of the camera coordinate system. The scale parameters indicate the scaling scale of the 3D hand model. The viewpoint parameter extraction layer includes a translation parameter extraction layer, a rotation parameter extraction layer, and a scale parameter extraction layer.

[0054] The coordinate transformation unit is used to call the translation parameter extraction layer, the rotation parameter extraction layer and the scale parameter extraction layer to perform feature transformation on the reference feature vector respectively to obtain the translation parameter, the rotation parameter and the scale parameter.

[0055] In another possible implementation, the feature transformation module includes:

[0056] The dimension transformation unit is used to call the dimension transformation layer in the parameter extraction model to perform dimension transformation on the feature information and obtain a reference feature vector.

[0057] The feature transformation unit is used to call the pose parameter extraction layer and the shape parameter extraction layer in the parameter extraction model to perform feature transformation on the reference feature vector to obtain the pose parameters and the shape parameters.

[0058] In another possible implementation, the device further includes:

[0059] The coordinate transformation module transforms the three-dimensional hand model into the camera coordinate system to obtain the target three-dimensional hand model in the camera coordinate system.

[0060] In another possible implementation, the device further includes:

[0061] The sign language recognition module is used to perform sign language recognition on the hand image based on the two-dimensional hand key points to obtain the text information corresponding to the hand image. The text information is used to represent the sign language meaning represented by the hand in the hand image.

[0062] On the other hand, a computer device is provided, the computer device including a processor and a memory, the memory storing at least one computer program, the at least one computer program being loaded and executed by the processor to perform the operations performed in the hand key point detection method as described above.

[0063] On the other hand, a computer-readable storage medium is provided that stores at least one computer program, which is loaded and executed by a processor to perform the operations performed in the hand keypoint detection method as described above.

[0064] In another aspect, a computer program product or computer program is provided, the computer program product or computer program including computer program code stored in a computer-readable storage medium. A processor of a computer device reads the computer program code from the computer-readable storage medium, and the processor executes the computer program code, causing the computer device to perform the operations performed in the hand keypoint detection method described above.

[0065] The beneficial effects of the technical solutions provided in this application include at least the following:

[0066] The methods, apparatus, computer devices, and storage media provided in this application incorporate relative positional constraints between various points contained in the hand in the three-dimensional hand model. Therefore, by constructing a three-dimensional hand model containing the hand in the hand image, the positions of the three-dimensional hand key points in the three-dimensional hand model are constrained, thereby ensuring the accuracy of the three-dimensional hand key points. Then, the three-dimensional hand key points are used as constraints to project the corresponding two-dimensional hand key points in the hand image, thereby improving the accuracy of the two-dimensional hand key points. Attached Figure Description

[0067] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0068] Figure 1 This is a schematic diagram of the structure of an implementation environment provided in an embodiment of this application;

[0069] Figure 2 This is a flowchart of a hand key point detection method provided in an embodiment of this application;

[0070] Figure 3 This is a flowchart of a hand key point detection method provided in an embodiment of this application;

[0071] Figure 4 This is a schematic diagram of key points of the hand provided in an embodiment of this application;

[0072] Figure 5 This is a flowchart of a hand construction model training method provided in an embodiment of this application;

[0073] Figure 6 This is a flowchart of a hand construction model training method provided in an embodiment of this application;

[0074] Figure 7 This is a schematic diagram of the structure of a hand key point detection device provided in an embodiment of this application;

[0075] Figure 8 This is a schematic diagram of the structure of a hand key point detection device provided in an embodiment of this application;

[0076] Figure 9 This is a schematic diagram of the structure of a terminal provided in an embodiment of this application;

[0077] Figure 10 This is a schematic diagram of the structure of a server provided in an embodiment of this application. Detailed Implementation

[0078] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the implementation methods of this application will be further described in detail below with reference to the accompanying drawings.

[0079] The terms “first,” “second,” “third,” “fourth,” “fifth,” “sixth,” etc., used in this application may be used to describe various concepts, but unless otherwise stated, these concepts are not limited by these terms. These terms are used only to distinguish one concept from another. For example, without departing from the scope of this application, a first product can be referred to as a second product, and similarly, a second product can be referred to as a first product.

[0080] As used in this application, the terms "at least one," "multiple," "each," and "any" mean that at least one includes one, two, or more; multiple includes two or more; each refers to each of the corresponding multiple; and any means refers to any one of the multiple. For example, multiple two-dimensional hand keypoints include three two-dimensional hand keypoints, where each refers to each of the three two-dimensional hand keypoints, and any means refers to any one of the three two-dimensional hand keypoints, which could be the first two-dimensional hand keypoint, or the second two-dimensional hand keypoint, or the third two-dimensional hand keypoint.

[0081] Artificial intelligence (AI) is the theory, methods, technology, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to possess the functions of perception, reasoning, and decision-making.

[0082] Artificial intelligence (AI) is a comprehensive discipline encompassing a wide range of fields, including both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies primarily include computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0083] Machine Learning (ML) is a multidisciplinary field involving probability theory, statistics, approximation theory, convex analysis, and algorithm complexity theory. It specifically studies how computers can simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to endow computers with intelligence; its applications span all areas of artificial intelligence. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and instruction-based learning.

[0084] The solution provided in this application embodiment is based on artificial intelligence machine learning technology, which can train a hand construction model and a parameter extraction model, and use the trained parameter extraction model and hand construction model to obtain two-dimensional hand key points in a hand image.

[0085] The hand key point detection method provided in this application embodiment can be used in computer devices. Optionally, the computer device can be a terminal or a server. Optionally, the server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. Optionally, the terminal can be a smartphone, tablet computer, laptop computer, desktop computer, smart speaker, smartwatch, etc., but is not limited to these.

[0086] Figure 1 This is a schematic diagram of an implementation environment provided in an embodiment of this application. See also... Figure 1 The implementation environment includes a terminal 101 and a server 102, which are connected via a wireless or wired network.

[0087] A target application, provided by server 102, is installed on terminal 101. Terminal 101 can perform functions such as data transmission and message interaction through this target application. Optionally, the target application is a target application within the operating system of terminal 101, or a target application provided by a third party. For example, the target application is an image processing application with image processing capabilities. This image processing application may also have other functions, such as shopping, navigation, and gaming. Optionally, server 102 is a backend server for the target application or a cloud server providing cloud computing and cloud storage services.

[0088] Terminal 101 is used to acquire a hand image and send the hand image to server 102 through a target application. Server 102 is used to receive the hand image, process the hand image, and acquire two-dimensional hand key points in the hand image.

[0089] Figure 2 This is a flowchart of a method for obtaining key hand points according to an embodiment of this application, which is applied in a computer device, such as... Figure 2 As shown, the method includes:

[0090] 201. Computer devices acquire feature information from hand images.

[0091] The hand image contains the hand itself, and this feature information describes the features contained in the hand image. By obtaining the feature information of the hand image, the pose and shape parameters of the hand contained in the hand image can be subsequently obtained.

[0092] 202. The computer device performs feature transformation on the feature information to obtain the posture parameters and shape parameters corresponding to the hand.

[0093] The pose parameter indicates the posture of the hand in the hand image. Different poses of the hand in the hand image result in different pose parameters. The shape parameter indicates the shape of the hand in the hand image. Different people may have different hand shapes and sizes, so different people may have different shape parameters.

[0094] 203. The computer equipment constructs a three-dimensional hand model based on the posture parameters and shape parameters, and determines the key points of the three-dimensional hand included in the three-dimensional hand model.

[0095] By using the pose and shape parameters corresponding to the hand, a 3D hand model is constructed to match the hand in the hand image, ensuring the accuracy of the 3D hand model. Then, the key points of the 3D hand contained in the 3D hand model are determined, thus ensuring the accuracy of determining the key points of the 3D hand from the 3D hand model.

[0096] 204. The computer equipment projects the three-dimensional key points of the hand onto the hand image to obtain the two-dimensional key points of the hand in the hand image.

[0097] By projecting 3D hand key points onto a hand image, the corresponding positions of the 3D hand key points in the hand image are determined, and the determined positions are used as 2D hand key points in the hand image.

[0098] The method provided in this application incorporates relative positional constraints between various points contained in the hand in the three-dimensional hand model. Therefore, by constructing a three-dimensional hand model containing the hand in the hand image, the positions of the three-dimensional hand key points in the three-dimensional hand model are constrained, thereby ensuring the accuracy of the three-dimensional hand key points. Then, the three-dimensional hand key points are used as constraints to project the corresponding two-dimensional hand key points in the hand image, thereby improving the accuracy of the two-dimensional hand key points.

[0099] Figure 3 This is a flowchart of a method for obtaining key hand points according to an embodiment of this application, which is applied in a computer device, such as... Figure 3 As shown, the method includes:

[0100] 301. Computer equipment acquires feature information from hand images.

[0101] The hand image includes the hand itself, and optionally, it is an RGB (Red, Green, Blue) image. The feature information describes the characteristics of the hand image; optionally, this feature information is represented by a feature vector. By obtaining the feature information of the hand image, the pose and shape parameters of the hand contained within the image can be subsequently obtained.

[0102] In one possible implementation, step 301 includes: acquiring a hand image, extracting features from the hand image, and obtaining feature information of the hand image.

[0103] After acquiring the hand image, feature extraction is used to obtain the feature information of the hand image to ensure the accuracy of the obtained feature information.

[0104] Alternatively, the process of acquiring a hand image includes the following three methods:

[0105] In the first method, the computer device is equipped with a camera: the hand is photographed using the camera to obtain an image of the hand.

[0106] The second method involves a computer receiving an image of a hand from another device. This other device can be any device.

[0107] The third method is to acquire the target video, perform frame extraction on the target video to obtain multiple video frames, perform hand detection on the multiple video frames, and obtain hand images that include the hand in the multiple video frames.

[0108] The target video can be any video, such as a sign language video, which includes hands. After extracting multiple video frames from the video, a hand detection method is used to filter out hand images containing hands from the multiple video frames.

[0109] Optionally, the process of obtaining feature information includes: calling a feature extraction model to encode the hand image and obtain the feature information of the hand image. The feature extraction model is used to extract the feature information of the hand image; optionally, the feature extraction model is EfficientNet-b0 (a multi-dimensional hybrid model). Using a feature extraction model to obtain the feature information of the hand image ensures the accuracy of the obtained feature information.

[0110] In one possible implementation, step 301 includes: the computer device receiving feature information of a hand image sent by another device.

[0111] 302. The computer device calls the dimension transformation layer in the parameter extraction model to perform dimension transformation on the feature information and obtain the reference feature vector.

[0112] In this embodiment of the application, the parameter extraction model is a network model used to extract parameters. The parameter extraction model includes a dimension transformation layer, a pose parameter extraction layer, and a shape parameter extraction layer. During the parameter extraction process, the output of the dimension transformation extraction layer is used as the input of the pose parameter extraction layer and the shape parameter extraction layer, respectively. The pose parameter extraction layer and the shape parameter extraction layer output the pose parameter and the shape parameter, respectively.

[0113] The dimension transformation layer is used to transform the dimensions of the feature information. Optionally, the dimension transformation layer is a fully connected layer. The dimension of the reference feature vector is different from the dimension of the feature information. For example, the feature information is a high-dimensional feature vector of 7×7×1536, while the reference feature vector is a feature vector of 512 dimensions. By transforming the feature information, the features containing the hand in the hand image are highlighted to ensure the accuracy of the parameters obtained subsequently.

[0114] In one possible implementation, the dimension transformation layer includes a pooling layer, a first dimension transformation layer, and a second dimension transformation layer. Then, step 302 includes: the computer device calls the pooling layer to perform pooling processing on the feature information to obtain a first reference feature vector corresponding to the feature information; calls the first dimension transformation layer to perform dimension transformation on the first reference feature vector to obtain a second reference feature vector; and calls the second dimension transformation layer to perform dimension transformation on the second reference feature vector to obtain a third reference feature vector.

[0115] The first, second, and third reference feature vectors all have different dimensions. For example, if the feature information is a high-dimensional feature vector of 7×7×1536, when the pooling layer processes this feature information, the largest value of 7×7 in the high-dimensional feature vector is selected, resulting in a first reference feature vector with a dimension of 1536. Inputting the first reference feature vector into the first dimension transformation layer yields a second reference feature vector with a dimension of 1024, and inputting the second reference feature vector into the second dimension transformation layer yields a third reference feature vector with a dimension of 512.

[0116] Optionally, during the process of obtaining the third reference feature vector at the dimensionality transformation layer, when outputting the reference feature vector at each dimensionality transformation layer, data standardization and normalization are performed on the reference feature vector to ensure the accuracy of the output reference feature vector. Specifically, when performing data standardization on the reference feature vector, BatchNormalization (a standardization function) or other functions can be used; when performing normalization, ReLU (Rectified Linear Unit, an activation function) or other functions can be used.

[0117] 303. The computer device calls the attitude parameter extraction layer and shape parameter extraction layer in the parameter extraction model, and performs feature transformation on the reference feature vector to obtain the attitude parameters and shape parameters.

[0118] The pose parameter indicates the hand's posture in the hand image; different poses result in different pose parameters. The shape parameter indicates the hand's shape in the hand image; different people have different hand shapes and sizes, thus different people may have different shape parameters. A pose parameter extraction layer is used to obtain the pose parameters, and a shape parameter extraction layer is used to obtain the shape parameters. Optionally, both the pose parameter extraction layer and the shape parameter extraction layer are fully connected layers.

[0119] After the reference feature vector output by the dimension transformation layer, the reference feature vector is input to the pose parameter extraction layer and the shape parameter extraction layer, respectively. Then, feature transformations are performed on the reference feature vector by the pose parameter extraction layer and the shape parameter extraction layer to obtain the pose parameters and shape parameters. The pose parameters and shape parameters are obtained through the pose parameter extraction layer and the shape parameter extraction layer to ensure the accuracy of the obtained pose parameters and shape parameters. In one possible implementation, the pose parameter extraction layer includes a third dimension transformation layer and a fourth dimension transformation layer. In this case, step 303 includes: calling the third dimension transformation layer to perform a dimension transformation on the reference feature vector to obtain a first feature vector, and calling the fourth dimension transformation layer to perform a dimension transformation on the first feature vector to obtain the pose parameters.

[0120] Optionally, both the third and fourth dimension transformation layers are fully connected layers. Through these layers, the reference feature vector undergoes multiple dimension transformations to obtain the pose parameters. For example, if the reference feature vector has a dimension of 512, the first feature vector has a dimension of 128, and the pose parameters have a dimension of 30, then the third and fourth dimension transformation layers transform the dimension of the reference feature vector from 512 to 30. The resulting feature vector after dimension transformation is the pose parameter.

[0121] In one possible implementation, the shape parameter extraction layer includes a fifth dimension transformation layer and a sixth dimension transformation layer. Then, step 303 includes: calling the fifth dimension transformation layer to perform dimension transformation on the reference feature vector to obtain a second feature vector, and calling the sixth dimension transformation layer to perform dimension transformation on the second feature vector to obtain shape parameters.

[0122] Optionally, both the fifth and sixth dimension transformation layers are fully connected layers. The fifth and sixth dimension transformation layers perform multiple dimensional transformations on the reference feature vector to obtain the shape parameter. For example, if the reference feature vector has a dimension of 512, the third feature vector has a dimension of 128, and the shape parameter has a dimension of 10, then the fifth and sixth dimension transformation layers transform the dimension of the reference feature vector from 512 to 10, and the transformed feature vector is the shape parameter.

[0123] It should be noted that in this embodiment, the posture parameters and shape parameters are obtained by calling the parameter extraction model. In another embodiment, steps 302-303 do not need to be executed, and other methods can be used to perform feature transformation on the feature information to obtain the posture parameters and shape parameters corresponding to the hand.

[0124] 304. The computer device calls the hand model, processes the posture and shape parameters to obtain a three-dimensional hand model, and determines the three-dimensional hand key points contained in the three-dimensional hand model.

[0125] The hand model is used to construct a 3D model of a human hand. Optionally, this hand model is a MANO (a type of hand model) or other network model. The 3D hand keypoints are arbitrary keypoints; for example, 3D hand keypoints are the positions of the hand's joints within the 3D hand model. These 3D hand keypoints also represent the 3D hand pose, indicating the pose of the 3D hand model. Optionally, there may be multiple 3D hand keypoints, such as... Figure 4 As shown, there are 21 key points in the three-dimensional hand, and all 21 key points are key points in the hand.

[0126] By using the pose and shape parameters corresponding to the hand, a 3D hand model is constructed to match the hand in the hand image, ensuring the accuracy of the 3D hand model. Then, the key points of the 3D hand contained in the 3D hand model are determined, thus ensuring the accuracy of determining the key points of the 3D hand from the 3D hand model.

[0127] In one possible implementation, the 3D hand model is composed of multiple vertices connected together. The process of determining the key points of the 3D hand includes: determining the vertex numbers of the multiple vertices, and determining the vertex corresponding to the key point number among the determined vertex numbers as the 3D hand key points.

[0128] The key point number is the number corresponding to the key point of the 3D hand. This key point number is any vertex number among multiple vertex numbers. For example, the vertex with vertex number 5 is a key point of the 3D hand.

[0129] In this embodiment, the three-dimensional hand model is composed of multiple connected vertices; that is, the three-dimensional hand model is a mesh model composed of multiple connected vertices. For example, the three-dimensional hand model includes 778 vertices, and multiple connected vertices form 1538 planes, which together constitute the three-dimensional hand model. In this three-dimensional hand model, each vertex corresponds to a vertex number, and for different three-dimensional hand models, the vertex corresponding to the same vertex number has the same position in the three-dimensional hand model. For example, for the vertex with vertex number 1, in the first three-dimensional hand model, vertex 1 indicates the fingertip position of the thumb, and in the second three-dimensional hand model, vertex 1 also indicates the fingertip position of the thumb.

[0130] Furthermore, among the multiple vertices contained in the 3D hand model, the key point numbers of the 3D hand key points are also fixed. Therefore, after determining the vertex numbers corresponding to multiple vertices, the vertex corresponding to the key point number among the multiple vertex numbers is determined as the 3D hand key point.

[0131] In one possible implementation, there are multiple 3D hand keypoints. When determining the 3D hand keypoints, the vertices corresponding to multiple keypoint indices from the multiple vertex indices are identified as 3D hand keypoints, thus obtaining multiple 3D hand keypoints.

[0132] It should be noted that in this embodiment of the application, a three-dimensional hand model is constructed and the key points of the three-dimensional hand are determined by calling the hand construction model. In another embodiment, step 304 is not required, and other methods can be used to construct a three-dimensional hand model based on posture parameters and shape parameters, and determine the key points of the three-dimensional hand contained in the three-dimensional hand model.

[0133] 305. The computer device calls the viewpoint parameter extraction layer in the parameter extraction model to perform feature transformation on the reference feature vector to obtain the viewpoint parameters.

[0134] In this embodiment, the parameter extraction model further includes a viewpoint parameter extraction layer. This layer extracts viewpoint parameters corresponding to the hand image. These viewpoint parameters indicate the display position of the hand in the camera coordinate system, i.e., the position of the hand in the camera coordinate system when the camera captures the hand image. The camera coordinate system is the coordinate system in which the camera is located when the hand image is captured. For example, the camera coordinate system can be obtained by taking the position of the camera when the hand image is captured as the origin, using the main line-of-sight direction of the camera as any coordinate axis, and using any two mutually perpendicular directions perpendicular to the main line-of-sight direction as the other two coordinate axes.

[0135] After the dimension transformation layer in the above parameter extraction model outputs the reference feature vector, the reference feature vector is input into the viewpoint parameter extraction layer. Then, the viewpoint parameter extraction layer performs feature transformation on the reference feature vector to obtain the viewpoint parameter.

[0136] In one possible implementation, the viewpoint parameter extraction layer includes a translation parameter extraction layer, a rotation parameter extraction layer, and a scale parameter extraction layer. Step 305 includes: calling the translation parameter extraction layer, the rotation parameter extraction layer, and the scale parameter extraction layer to perform feature transformation on the reference feature vector respectively to obtain the translation parameters, rotation parameters, and scale parameters.

[0137] The viewpoint parameters include translation, rotation, and scale parameters. Translation parameters indicate the position of the 3D hand model relative to the origin of the camera coordinate system; rotation parameters indicate the rotation angle of the 3D hand model relative to the coordinate axes of the camera coordinate system; and scale parameters indicate the scaling of the 3D hand model. Optionally, translation, rotation, and scale parameters are all represented by vectors: translation parameters are 3D feature vectors, rotation parameters are 3D feature vectors, and scale parameters are 1D feature vectors. A translation parameter extraction layer is used to obtain translation parameters, a rotation parameter extraction layer is used to obtain rotation parameters, and a scale parameter extraction layer is used to obtain scale parameters.

[0138] In the process of extracting parameters in this parameter extraction model, the reference feature vector output by the dimension transformation layer is input to the translation parameter extraction layer, the rotation parameter extraction layer and the scale parameter extraction layer respectively. Then, the translation parameter extraction layer, the rotation parameter extraction layer and the scale parameter extraction layer perform feature transformation on the reference feature vector respectively, and output the translation parameter, the rotation parameter and the scale parameter.

[0139] Optionally, the translation parameter extraction layer includes a seventh-dimensional transformation layer, an eighth-dimensional transformation layer, and a ninth-dimensional transformation layer. The process of obtaining the translation parameters includes: calling the seventh-dimensional transformation layer to perform dimensional transformation on the reference feature vector to obtain the fourth feature vector; calling the eighth-dimensional transformation layer to perform dimensional transformation on the fourth feature vector to obtain the fifth feature vector; and calling the ninth-dimensional transformation layer to perform dimensional transformation on the fifth feature vector to obtain the translation parameters.

[0140] For example, the seventh, eighth, and ninth dimension transformation layers are all fully connected layers. Through these layers, the reference feature vector undergoes multiple dimensional transformations to obtain the translation parameter. For instance, if the reference feature vector has a dimension of 512, the fourth feature vector has a dimension of 128, the fifth feature vector has a dimension of 32, and the translation parameter has a dimension of 3, then these layers transform the reference feature vector from 512 to 3. The resulting feature vector after the dimensional transformation is the translation parameter.

[0141] Optionally, both the rotation parameter extraction layer and the scale parameter extraction layer include three-dimensional transformation layers.

[0142] For example, when obtaining rotation parameters, the rotation parameter extraction layer includes three dimension transformation layers that perform multiple dimension transformations on the reference feature vector to obtain the rotation parameters. For instance, if the dimension of the reference feature vector is 512, the three dimension transformation layers will yield a 128-dimensional feature vector, a 32-dimensional feature vector, and a 3-dimensional feature vector, respectively. That is, the dimension of the reference feature vector is transformed from 512 to 3, and the feature vector after dimension transformation is the rotation parameter.

[0143] For example, when obtaining the scale parameter, the scale parameter extraction layer includes three dimension transformation layers that perform multiple dimension transformations on the reference feature vector to obtain the scale parameter. For instance, if the dimension of the reference feature vector is 512, the three dimension transformation layers will produce a 128-dimensional feature vector, a 32-dimensional feature vector, and a 1-dimensional feature vector, respectively. In other words, the dimension of the reference feature vector is transformed from 512 to 1, and the feature vector after dimension transformation is the scale parameter.

[0144] The process of obtaining rotation parameters through the three-dimensional transformation layers included in the rotation parameter extraction layer, and the process of obtaining scale parameters through the three-dimensional transformation layers included in the scale parameter extraction layer, are the same as the process of obtaining translation parameters through the seventh, eighth, and ninth dimension transformation layers, as described above, and will not be repeated here.

[0145] It should be noted that in steps 302, 303 and 305 above, the parameter extraction model called by the computer device is either the model configured by the computer device itself or the model configured by other devices.

[0146] In one possible implementation, the parameter extraction model is configured in the computer device. That is, when the computer device executes steps 302, 303, and 305 above, it calls its own configured parameter extraction model to process the feature information to obtain the pose parameters, shape parameters, and viewpoint parameters.

[0147] In another possible implementation, the computer device is any device node in the blockchain system, and other device nodes in the blockchain system are configured with parameter extraction models. Then, steps 302, 303, and 305 above include: the computer device sending feature information to the other device node; the other device node receiving the feature information sent by the computer device; calling the dimension transformation layer in the parameter extraction model configured by the other device node to perform dimension transformation on the feature information to obtain a reference feature vector; calling the posture parameter extraction layer and shape parameter extraction layer in the parameter extraction model to perform feature transformation on the reference feature vector respectively to obtain posture parameters and shape parameters; calling the viewpoint parameter extraction layer in the parameter extraction model to perform feature transformation on the reference feature vector to obtain viewpoint parameters; and sending the posture parameters, shape parameters, and viewpoint parameters to the computer device; the computer device receiving the posture parameters, shape parameters, and viewpoint parameters sent by the other device node.

[0148] The blockchain system comprises multiple device nodes that communicate peer-to-peer via a network. The computer device is any one of these nodes and can perform peer-to-peer transmission with other device nodes configured with parameter extraction models. Through this peer-to-peer transmission, feature information is sent to these other device nodes for processing, and the other nodes return the processing results to the computer device. This allows the computer device to access the parameter extraction models of other node devices.

[0149] It should be noted that the present application embodiment obtains the viewpoint parameters through a parameter extraction model, while in another embodiment, it is not necessary to perform the above steps 302 and 305, and other methods can be used to obtain the viewpoint parameters corresponding to the hand image.

[0150] It should be noted that in the above embodiment, after obtaining the reference feature vector, the posture parameters and shape parameters are obtained first, and the hand construction model is called to process the posture parameters and shape parameters to obtain a three-dimensional hand model and determine the three-dimensional hand key points contained in the three-dimensional hand model. Then, the parameter extraction model is called to obtain the viewpoint parameters for description. In another embodiment, after step 302, step 305 is executed first, followed by steps 303 and 304, or step 303 is executed first, followed by steps 305 and 304.

[0151] 306. The computer equipment transforms the 3D hand key points into the camera coordinate system based on the viewpoint parameters to obtain the reference 3D hand key points.

[0152] Here, the camera coordinate system refers to the coordinate system in which the camera is positioned when the hand image is captured. Since the hand image is captured by the camera in this camera coordinate system, the viewpoint parameter is used to indicate the display position of the hand contained in the hand image in the camera coordinate system. Based on this viewpoint parameter, the 3D hand keypoints are transformed into the camera coordinate system, and the resulting reference 3D hand keypoints are the positions of the 3D hand keypoints in the hand in the camera coordinate system when the hand image is captured. Both the reference 3D hand keypoints and the hand image are represented in the camera coordinate system so that the reference 3D hand keypoints can be projected onto the hand image later.

[0153] In one possible implementation, the viewpoint parameters include translation parameters, rotation parameters, and scale parameters. In this case, step 306 includes: determining the first product of the scale parameters and the coordinates of the three-dimensional hand key points; determining the second product of the first product and the rotation parameters; summing the second product corresponding to the three-dimensional hand key points with the translation parameters; and determining the coordinates of the reference three-dimensional hand key points.

[0154] Optionally, if there are multiple 3D hand key points, then each 3D hand key point is transformed into the camera coordinate system in the manner described above to obtain the coordinates of multiple reference 3D hand key points.

[0155] Optionally, the translation parameters, rotation parameters, scale parameters, coordinates of the 3D hand keypoints, and coordinates of the reference 3D hand keypoints satisfy the following relationship:

[0156] J = sJ0R + T

[0157] Where J represents the coordinates of the reference 3D hand keypoint, s represents the scale parameter, J0 represents the coordinates of the 3D hand keypoint, R represents the rotation parameter, and T represents the translation parameter.

[0158] It should be noted that in this embodiment, the three-dimensional hand key points are transformed into the camera coordinate system based on the viewpoint parameters. However, in another embodiment, steps 305-306 are not required, and other methods can be used to transform the three-dimensional hand key points into the camera coordinate system to obtain reference three-dimensional hand key points.

[0159] Furthermore, the above only describes the transformation of the obtained 3D hand key points to the camera coordinate system to obtain reference 3D hand key points. In another embodiment, after obtaining the 3D hand model, the 3D hand model is transformed to the camera coordinate system to obtain a reference 3D hand model in the camera coordinate system. The process of transforming the 3D hand model to the camera coordinate system is the same as the process of transforming the 3D hand key points to the camera coordinate system described above, and will not be repeated here.

[0160] 307. The computer equipment projects reference three-dimensional hand key points onto the hand image based on the shooting parameters corresponding to the hand image, thereby obtaining two-dimensional hand key points in the hand image.

[0161] The shooting parameters are those used when capturing the hand image. These two-dimensional hand keypoints represent the two-dimensional hand pose, indicating the hand's posture within the image. By projecting reference three-dimensional hand keypoints onto the hand image, the corresponding positions of these keypoints are determined, thus obtaining the two-dimensional hand keypoints within the image. The accuracy of the obtained two-dimensional hand keypoints is ensured by projecting the reference three-dimensional hand keypoints onto the hand image using the shooting parameters corresponding to the hand image.

[0162] In one possible implementation, the shooting parameters include a principal point and focal length. The principal point represents the intersection of the camera's main line of sight and the hand image when the hand image is captured. This principal point can be any point within the hand image, such as the midpoint or any other arbitrary point. The focal length indicates the distance between the camera and the hand image when the hand image is captured.

[0163] Optionally, step 307 includes: determining the planar image corresponding to the hand image in the camera coordinate system based on the principal point and focal length included in the shooting parameters; projecting the reference three-dimensional hand key point into the planar image to obtain the two-dimensional hand key point in the planar image; and determining the position point corresponding to the two-dimensional hand key point in the planar image as the two-dimensional hand key point in the hand image.

[0164] In this design, the planar image has the same dimensions as the hand image. For example, when determining the planar image corresponding to the hand image in the camera coordinate system, a planar image with the same dimensions as the hand image is constructed. This planar image is perpendicular to the camera's main line of sight, and the principal point corresponding to the planar image is perpendicular to the main line of sight, thus obtaining the planar image. By projecting the reference 3D hand keypoints onto the hand image in the camera coordinate system, the accuracy of the obtained 2D hand keypoints is ensured.

[0165] In one possible implementation, after step 307, the method further includes: performing sign language recognition on the hand image based on two-dimensional hand key points to obtain text information corresponding to the hand image.

[0166] The text information is used to represent the sign language meaning of the hands in the hand image. By using two-dimensional hand key points, text information representing the sign language meaning of the hands in the hand image is obtained, thus realizing the understanding of the hand movements contained in the hand image.

[0167] It should be noted that in this embodiment, the three-dimensional hand key points are first transformed into the camera coordinate system, and then the reference three-dimensional hand key points in the camera coordinate system are projected onto the hand image to obtain the two-dimensional hand key points in the hand image. In another embodiment, steps 306-307 do not need to be performed, and other methods can be used to project the three-dimensional hand key points onto the hand image to obtain the two-dimensional hand key points in the hand image.

[0168] The method provided in this application incorporates relative positional constraints between various points contained in the hand in the three-dimensional hand model. Therefore, by constructing a three-dimensional hand model containing the hand in the hand image, the positions of the three-dimensional hand key points in the three-dimensional hand model are constrained, thereby ensuring the accuracy of the three-dimensional hand key points. Then, the three-dimensional hand key points are used as constraints to project the corresponding two-dimensional hand key points in the hand image, thereby improving the accuracy of the two-dimensional hand key points.

[0169] Taking sign language recognition as an example, a sign language recognition process is provided, which includes:

[0170] 1. The terminal has a sign language recognition application installed. The terminal acquires sign language action videos and, based on the sign language recognition application, sends the sign language action videos to the server that provides sign language recognition services.

[0171] 2. The server performs frame extraction on the sign language video to obtain hand images, and then extracts features from the hand images to obtain the feature information of the hand images.

[0172] 3. The server calls the dimension transformation layer in the parameter extraction model to perform dimension transformation on the feature information to obtain the reference feature vector. It then calls the pose parameter extraction layer and shape parameter extraction layer in the parameter extraction model to perform feature transformation on the reference feature vector to obtain the pose parameters and shape parameters. Finally, it calls the hand construction model to process the pose parameters and shape parameters to obtain the 3D hand model and determine the 3D hand key points contained in the 3D hand model.

[0173] 4. The server calls the viewpoint parameter extraction layer in the parameter extraction model, performs feature transformation on the reference feature vector to obtain viewpoint parameters, and transforms the 3D hand key points into the camera coordinate system according to the viewpoint parameters to obtain reference 3D hand key points. According to the shooting parameters corresponding to the hand image, the reference 3D hand key points are projected onto the hand image to obtain the 2D hand key points in the hand image.

[0174] 5. The server performs sign language recognition on the hand image based on the two-dimensional hand key points, obtains the corresponding text information, and sends the text information to the terminal. The text information represents the sign language meaning represented by the hand in the hand image.

[0175] 6. The terminal receives the text information sent by the server and displays the text information for the user to view, thereby realizing the understanding of sign language actions in the sign language video.

[0176] Taking sign language generation as an example, a process for generating sign language based on virtual objects is provided, which includes:

[0177] 1. The computer device acquires video of sign language gestures, performs frame extraction on the video to obtain multiple hand images, and extracts features from each hand image to obtain feature information for each hand image.

[0178] 2. For each hand image, the computer device calls the dimension transformation layer in the parameter extraction model to perform dimension transformation on the feature information to obtain a reference feature vector. Then, it calls the pose parameter extraction layer and shape parameter extraction layer in the parameter extraction model to perform feature transformation on the reference feature vector to obtain pose parameters and shape parameters. Finally, it calls the hand construction model to process the pose parameters and shape parameters to obtain a three-dimensional hand model and determine the three-dimensional hand key points contained in the three-dimensional hand model.

[0179] 3. The computer device calls the viewpoint parameter extraction layer in the parameter extraction model, performs feature transformation on the reference feature vector to obtain the viewpoint parameters, and transforms the 3D hand key points into the camera coordinate system based on the viewpoint parameters to obtain the reference 3D hand key points.

[0180] 4. The computer device transfers the reference 3D hand key points to the hand of the 3D virtual object based on the reference 3D hand key points corresponding to each hand image, so that the pose of the hand of the 3D virtual object is the same as the pose of the hand contained in the corresponding hand image.

[0181] 5. The computer equipment renders the hand of a three-dimensional virtual object to obtain a motion video representing the hand based on the three-dimensional virtual object.

[0182] exist Figure 3 Based on the illustrated embodiment, the hand construction model needs to be trained before calling the hand construction model. The training process is detailed in the following embodiment. Figure 5 This is a flowchart of a hand model training method provided in an embodiment of this application, applied in a computer device, such as... Figure 5 As shown, the method includes:

[0183] 501. The computer device acquires a sample hand image, and the sample pose parameters and sample shape parameters corresponding to the sample hand contained in the sample hand image.

[0184] The sample hand image is an image containing the hand. The process of obtaining the sample pose parameters and sample shape parameters corresponding to the sample image is the same as the process of obtaining the pose parameters and shape parameters corresponding to the hand in the above embodiment, and will not be repeated here.

[0185] 502. The computer device calls the key point detection model to perform key point detection on the sample hand image, and obtains the sample two-dimensional hand key points in the sample hand image.

[0186] The keypoint detection model is used to detect two-dimensional hand keypoints in hand images. Optionally, this keypoint detection model is the OpenPose (human pose recognition) model.

[0187] In one possible implementation, step 502 includes: calling a keypoint detection model to perform keypoint detection on the sample hand image, and obtaining the sample two-dimensional hand keypoints and the confidence scores corresponding to the sample two-dimensional hand keypoints.

[0188] The confidence level is used to represent the probability that the detected two-dimensional hand keypoints in the sample are true two-dimensional hand keypoints.

[0189] Optionally, if there are multiple key points of the sample two-dimensional hand, the process of obtaining multiple key points of the sample two-dimensional hand includes: calling the key point detection model to perform key point detection on the sample hand image, obtaining multiple heatmaps corresponding to the sample hand image, and determining the position point corresponding to the maximum confidence in each heatmap as the position point corresponding to the sample hand image.

[0190] Each heatmap includes multiple location points and a confidence level for each location point. The confidence level is used to indicate the probability that the location point in the sample hand image is a key point of the sample two-dimensional hand.

[0191] In the embodiments of this application, the hand corresponds to multiple different types of hand key points. Different types of hand key points are used to indicate different hand positions. For example, one type of hand key point indicates the fingertip position of the thumb, another type of hand key point indicates the fingertip position of the index finger, and another type of hand key point indicates the position where the middle finger connects to the palm. Thus, the sample hand image includes multiple sample two-dimensional hand key points belonging to different types, and the multiple heat maps obtained correspond to different types.

[0192] For any given heatmap, the multiple location points contained in the heatmap, corresponding to the location points in the sample hand image, may be sample 2D hand keypoints belonging to the type corresponding to that heatmap. The confidence level of each location point represents the probability that the location point in the sample hand image belongs to the sample 2D hand keypoint of the type corresponding to that heatmap. Based on the multiple location points included in the heatmap and the confidence level corresponding to each location point, the location point with the highest confidence level in the heatmap, corresponding to the location point in the sample hand image, is determined to be a sample 2D hand keypoint belonging to the type corresponding to that heatmap. Following the above method, sample 2D hand keypoints belonging to the type corresponding to each heatmap are determined, thereby obtaining multiple sample 2D hand keypoints. Furthermore, the highest confidence level in the heatmap is determined as the confidence level corresponding to the sample 2D hand keypoint of that heatmap.

[0193] The heatmap is the same size as the sample hand image, and each point in the heatmap corresponds to a point in the sample hand image. Optionally, different colors are used in the heatmap to represent the confidence levels corresponding to multiple points. For example, in any heatmap, multiple points may be represented by different colors; for instance, the confidence level of a point represented in red may be higher than that of a point represented in green. By using the colors corresponding to multiple points in the heatmap, the point corresponding to the highest confidence level can be determined.

[0194] After obtaining multiple heatmaps, the key points of the sample hand in the sample hand image are determined based on the location of the highest confidence in each heatmap. This ensures that each key point of the sample hand has the highest confidence, thereby guaranteeing the accuracy of the key points of the sample hand.

[0195] Furthermore, this keypoint detection model includes multiple keypoint detection sub-models. Each keypoint detection sub-model is used to obtain a heatmap, that is, each keypoint detection sub-model is used to detect the location points corresponding to a certain type of two-dimensional hand keypoints in the sample hand image and the confidence score of each location point. The process of obtaining multiple heatmaps includes: calling multiple keypoint detection sub-models to perform keypoint detection on the sample hand image respectively, and obtaining multiple heatmaps corresponding to the sample hand image.

[0196] 503. The computer device calls the hand construction model, processes the sample posture parameters and sample shape parameters to obtain the predicted three-dimensional hand model, and determines the predicted three-dimensional hand key points contained in the predicted three-dimensional hand model.

[0197] This step is the same as step 304 above, and will not be repeated here.

[0198] 504. The computer equipment projects the predicted three-dimensional hand key points onto the sample hand image to obtain the predicted two-dimensional hand key points in the sample hand image.

[0199] This step is similar to the process of converting the 3D hand key points to the camera coordinate system in the above embodiment, and will not be repeated here.

[0200] 505. The computer equipment determines the loss value of the hand construction model based on the sample two-dimensional hand key points and the predicted two-dimensional hand key points.

[0201] The loss value of the hand construction model is determined based on the difference between the sample 2D hand key points and the predicted 2D hand key points, so that the hand construction model can be trained based on the loss value.

[0202] In one possible implementation, after obtaining the sample two-dimensional hand key points and their corresponding confidence levels, step 505 includes: weighting the distance between the sample two-dimensional hand key points and the predicted two-dimensional hand key points based on the confidence levels corresponding to the sample two-dimensional hand key points to obtain a loss value.

[0203] Optionally, the sample two-dimensional hand keypoints include multiple points, and the predicted two-dimensional hand keypoints also include multiple points; then the process of determining the loss value includes: determining the distance between each sample two-dimensional hand keypoint and the corresponding predicted two-dimensional hand keypoint, determining the product of each distance and the confidence level corresponding to the corresponding sample two-dimensional hand keypoint, and summing the determined products to determine the loss value.

[0204] Among them, multiple sample two-dimensional hand key points correspond one-to-one with multiple predicted two-dimensional hand key points.

[0205] Optionally, multiple sample 2D hand keypoints, multiple predicted 2D hand keypoints, and loss values ​​satisfy the following relationship:

[0206]

[0207] Where E represents the loss value, K represents the total number of multiple two-dimensional hand keypoints, i represents the index of the multiple two-dimensional hand keypoints, and con i This represents the confidence level corresponding to the two-dimensional hand keypoints of the i-th sample. Let represent the coordinates of the two-dimensional hand key points of the i-th sample. This represents the coordinates of the i-th predicted 2D hand keypoint.

[0208] 506. The computer equipment trains the hand model based on the loss value.

[0209] After obtaining the loss value, the hand model is trained based on the loss value to improve the accuracy of the hand model.

[0210] It should be noted that in this embodiment, the loss value is first determined based on the sample two-dimensional hand key points and the predicted two-dimensional hand key points, and then the hand construction model is trained based on the determined loss value. In another embodiment, steps 505-506 are not required, and other methods can be used to train the hand construction model based on the sample two-dimensional hand key points and the predicted two-dimensional hand key points.

[0211] The method provided in this application adopts an unsupervised approach when training the hand construction model. It is based on a key point detection model to train the hand construction model, so as to ensure the accuracy of the hand construction model. Furthermore, during the training process, there is no need for manual annotation of training samples, which reduces the cost of manual annotation and thus saves labor costs.

[0212] Figure 6 This is a flowchart of a hand model training method provided in an embodiment of this application, such as... Figure 6 As shown, the process includes:

[0213] 601. Computer equipment acquires sample hand images.

[0214] 602. The computer device calls the key point detection model to perform key point detection on the sample hand image, and obtains the sample two-dimensional hand key points in the sample hand image.

[0215] 603. The computer device calls the feature extraction model to extract features from the sample hand image, obtains the feature information of the sample hand image, calls the dimension transformation layer in the parameter extraction model to perform dimension transformation on the feature information, obtains the reference feature vector, calls the pose parameter extraction layer and shape parameter extraction layer in the parameter extraction model to perform feature transformation on the reference feature vector respectively, obtains the sample pose parameters and sample shape parameters, calls the viewpoint parameter extraction layer in the parameter extraction model to perform feature transformation on the reference feature vector, obtains the sample viewpoint parameters.

[0216] 604. The computer device calls the hand construction model, processes the sample pose parameters and sample shape parameters to obtain the predicted three-dimensional hand model, and determines the predicted three-dimensional hand key points contained in the predicted three-dimensional hand model. The predicted three-dimensional hand key points are projected onto the sample hand image to obtain the predicted two-dimensional hand key points in the sample hand image.

[0217] 605. The computer equipment determines the loss value of the hand construction model based on the sample two-dimensional hand key points and the predicted two-dimensional hand key points, and trains the hand construction model based on the loss value.

[0218] In addition, based on the hand construction model training method provided in the embodiments of this application, the hand construction model can be trained on any dataset, such as FreiHAND (a dataset).

[0219] The keypoint detection model provided in this application is a two-dimensional image detection model. The 2D MPJPE (Mean Per Joint Position Error) of this keypoint detection model on this dataset is 9.84 pixels. In the hand reconstruction model training method provided in this application embodiment, the sample two-dimensional hand keypoints output by this keypoint detection model are used as training supervision to train the hand reconstruction model. Then, based on the hand keypoint detection method provided in this application embodiment, the trained hand reconstruction model is used to obtain two-dimensional hand keypoints in the hand image. The obtained two-dimensional hand keypoints have a 2D MPJPE of 9.55 pixels on this dataset, indicating that the hand keypoint detection method provided in this application has higher accuracy.

[0220] Furthermore, on this dataset, the 3D MPJPE (Mean Per Joint Position Error) of the trained hand reconstruction model is 1.22 mm. In related technologies, the 3D MPJPE for obtaining 3D hand keypoints is 3.50 mm and 1.33 mm. In the hand reconstruction model training method provided in this application embodiment, only the sample 2D hand keypoints output by the keypoint detection model are used as supervision to train the hand reconstruction model, eliminating the need for labeled sample hand images. This achieves an unsupervised training method, and compared with related technologies, the hand keypoint detection method provided in this application embodiment has higher accuracy.

[0221] Figure 7 This is a schematic diagram of the structure of a hand key point detection device provided in an embodiment of this application, as shown below. Figure 7 As shown, the device includes:

[0222] The first acquisition module 701 is used to acquire feature information of a hand image, wherein the hand image contains a hand;

[0223] The feature transformation module 702 is used to perform feature transformation on the feature information to obtain the posture parameters and shape parameters corresponding to the hand.

[0224] The determination module 703 is used to construct a three-dimensional hand model based on posture parameters and shape parameters, and to determine the three-dimensional hand key points contained in the three-dimensional hand model;

[0225] The projection module 704 is used to project three-dimensional hand key points onto the hand image to obtain two-dimensional hand key points in the hand image.

[0226] In one possible implementation, such as Figure 8 As shown, module 703 includes:

[0227] The first determining unit 7031 is used to call the hand construction model, process the posture parameters and shape parameters to obtain the three-dimensional hand model, and determine the three-dimensional hand key points contained in the three-dimensional hand model.

[0228] In another possible implementation, the three-dimensional hand model is composed of multiple vertices connected together; the first determining unit 7031 is used to determine the vertex number of the multiple vertices; the vertex corresponding to the key point number among the determined vertex numbers is determined as the three-dimensional hand key point.

[0229] In another possible implementation, such as Figure 8 As shown, the device also includes:

[0230] The second acquisition module 705 is also used to acquire a sample hand image, and the sample pose parameters and sample shape parameters corresponding to the sample hand contained in the sample hand image;

[0231] The detection module 706 is also used to call the key point detection model to perform key point detection on the sample hand image and obtain the sample two-dimensional hand key points in the sample hand image;

[0232] The determination module 703 is also used to call the hand construction model, process the sample pose parameters and sample shape parameters to obtain the predicted three-dimensional hand model, and determine the predicted three-dimensional hand key points contained in the predicted three-dimensional hand model.

[0233] The projection module 704 is also used to project the predicted three-dimensional hand key points onto the sample hand image to obtain the predicted two-dimensional hand key points in the sample hand image.

[0234] Training module 707 is used to train a hand construction model based on sample two-dimensional hand key points and predicted two-dimensional hand key points.

[0235] In another possible implementation, such as Figure 8 As shown, training module 707 includes:

[0236] The second determining unit 7071 is used to determine the loss value of the hand construction model based on the sample two-dimensional hand key points and the predicted two-dimensional hand key points.

[0237] Training unit 7072 is used to train the hand model based on the loss value.

[0238] In another possible implementation, such as Figure 8 As shown, the detection module 706 includes:

[0239] The detection unit 7061 is used to call the key point detection model to perform key point detection on the sample hand image, and obtain the sample two-dimensional hand key points and the confidence level corresponding to the sample two-dimensional hand key points;

[0240] The second determining unit 7071 is used to perform weighted processing on the distance between the sample two-dimensional hand key points and the predicted two-dimensional hand key points according to the confidence level, so as to obtain the loss value.

[0241] In another possible implementation, the sample two-dimensional hand keypoints include multiple points, and the predicted two-dimensional hand keypoints also include multiple points.

[0242] The second determining unit 7071 is used to determine the distance between each sample two-dimensional hand keypoint and the corresponding predicted two-dimensional hand keypoint; determine the product of each distance and the confidence level corresponding to the sample two-dimensional hand keypoint; and determine the sum of the determined multiple products as the loss value.

[0243] In another possible implementation, the detection unit 7061 is used to call the key point detection model to perform key point detection on the sample hand image, and obtain multiple heatmaps corresponding to the sample hand image. Each heatmap includes multiple location points and the confidence level corresponding to each location point. The confidence level is used to represent the probability that the location point corresponding to the location point in the sample hand image is a key point of the sample two-dimensional hand. The location point corresponding to the maximum confidence level in each heatmap is determined as the key point of the sample two-dimensional hand in the sample hand image.

[0244] In another possible implementation, such as Figure 8 As shown, the projection module 704 includes:

[0245] The coordinate transformation unit 7041 is used to transform the 3D hand key points to the camera coordinate system to obtain the reference 3D hand key points;

[0246] The projection unit 7042 is used to project reference three-dimensional hand key points onto the hand image according to the shooting parameters corresponding to the hand image to obtain two-dimensional hand key points. The shooting parameters are the shooting parameters used when shooting the hand image.

[0247] In another possible implementation, coordinate transformation unit 7041 is used to obtain the viewpoint parameters corresponding to the hand image. The viewpoint parameters are used to indicate the display position of the hand in the camera coordinate system. Based on the viewpoint parameters, the three-dimensional hand key points are transformed into the camera coordinate system to obtain reference three-dimensional hand key points.

[0248] In another possible implementation, the viewpoint parameters include translation parameters, rotation parameters, and scale parameters. The translation parameters are used to indicate the position of the 3D hand model relative to the origin of the camera coordinate system, the rotation parameters are used to indicate the rotation angle of the 3D hand model relative to the coordinate axes of the camera coordinate system, and the scale parameters are used to indicate the scaling scale of the 3D hand model.

[0249] The coordinate transformation unit 7041 is used to determine the first product of the scale parameter and the coordinates of the three-dimensional hand key points; determine the second product of the first product and the rotation parameter; and sum the second product corresponding to the three-dimensional hand key points with the translation parameter to determine the coordinates of the reference three-dimensional hand key points.

[0250] In another possible implementation, the coordinate transformation unit 7041 is used to call the dimension transformation layer in the parameter extraction model to perform dimension transformation on the feature information to obtain the reference feature vector; and to call the view parameter extraction layer in the parameter extraction model to perform feature transformation on the reference feature vector to obtain the view parameters.

[0251] In another possible implementation, the viewpoint parameters include translation parameters, rotation parameters, and scale parameters. The translation parameters are used to indicate the position of the 3D hand model relative to the origin of the camera coordinate system, the rotation parameters are used to indicate the rotation angle of the 3D hand model relative to the coordinate axes of the camera coordinate system, and the scale parameters are used to indicate the scaling scale of the 3D hand model. The viewpoint parameter extraction layer includes a translation parameter extraction layer, a rotation parameter extraction layer, and a scale parameter extraction layer.

[0252] The coordinate transformation unit 7041 is used to call the translation parameter extraction layer, rotation parameter extraction layer and scale parameter extraction layer to perform feature transformation on the reference feature vector to obtain translation parameters, rotation parameters and scale parameters.

[0253] In another possible implementation, such as Figure 8 As shown, the feature transformation module 702 includes:

[0254] The dimension transformation unit 7021 is used to call the dimension transformation layer in the parameter extraction model to perform dimension transformation on the feature information and obtain the reference feature vector.

[0255] The feature transformation unit 7022 is used to call the attitude parameter extraction layer and shape parameter extraction layer in the parameter extraction model to perform feature transformation on the reference feature vector to obtain the attitude parameters and shape parameters.

[0256] In another possible implementation, such as Figure 8 As shown, the device also includes:

[0257] The coordinate transformation module 708 transforms the 3D hand model into the camera coordinate system, resulting in the target 3D hand model in the camera coordinate system.

[0258] In another possible implementation, such as Figure 8 As shown, the device also includes:

[0259] The sign language recognition module 709 is used to perform sign language recognition on a hand image based on two-dimensional hand key points to obtain text information corresponding to the hand image. The text information is used to represent the sign language meaning represented by the hand in the hand image.

[0260] It should be noted that the hand key point detection device provided in the above embodiments is only an example of the division of the above functional modules. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the computer device can be divided into different functional modules to complete all or part of the functions described above. In addition, the hand key point detection device and the hand key point detection method embodiments provided in the above embodiments belong to the same concept, and the specific implementation process can be found in the method embodiments, which will not be repeated here.

[0261] This application also provides a computer device, which includes a processor and a memory. The memory stores at least one computer program, which is loaded and executed by the processor to perform the operations performed in the hand key point detection method of the above embodiments.

[0262] Optionally, the computer device is provided as a terminal. Figure 9 A structural block diagram of a terminal 900 provided in an exemplary embodiment of this application is shown. The terminal 900 may be a portable mobile terminal, such as a smartphone, tablet computer, MP3 player (Moving Picture Experts Group Audio Layer III), MP4 player (Moving Picture Experts Group Audio Layer IV), laptop computer, or desktop computer. The terminal 900 may also be referred to as a user device, portable terminal, laptop terminal, desktop terminal, or other names.

[0263] The terminal 900 includes a processor 901 and a memory 902.

[0264] Processor 901 may include one or more processing cores, such as a quad-core processor, an octa-core processor, etc. Processor 901 may be implemented using at least one hardware form selected from DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). Processor 901 may also include a main processor and a coprocessor. The main processor, also known as a CPU (Central Processing Unit), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, processor 901 may integrate a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content required to be displayed on the screen. In some embodiments, processor 901 may also include an AI (Artificial Intelligence) processor, which is used to handle computational operations related to machine learning.

[0265] The memory 902 may include one or more computer-readable storage media, which may be non-transitory. The memory 902 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In some embodiments, the non-transitory computer-readable storage media in the memory 902 are used to store at least one computer program, which is executed by the processor 901 to implement the hand key point detection method provided in the method embodiments of this application.

[0266] In some embodiments, the terminal 900 may also optionally include a peripheral device interface 903 and at least one peripheral device. The processor 901, memory 902, and peripheral device interface 903 can be connected via a bus or signal line. Each peripheral device can be connected to the peripheral device interface 903 via a bus, signal line, or circuit board. Specifically, the peripheral device includes at least one of the following: a radio frequency circuit 904, a display screen 905, a camera assembly 906, an audio circuit 907, a positioning assembly 908, and a power supply 909.

[0267] Peripheral device interface 903 can be used to connect at least one I / O (Input / Output) related peripheral device to processor 901 and memory 902. In some embodiments, processor 901, memory 902 and peripheral device interface 903 are integrated on the same chip or circuit board; in some other embodiments, any one or two of processor 901, memory 902 and peripheral device interface 903 can be implemented on separate chips or circuit boards, which is not limited in this embodiment.

[0268] The radio frequency (RF) circuit 904 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The RF circuit 904 communicates with communication networks and other communication devices via electromagnetic signals. The RF circuit 904 converts electrical signals into electromagnetic signals for transmission, or converts received electromagnetic signals back into electrical signals. Optionally, the RF circuit 904 includes: an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a user identity module card, etc. The RF circuit 904 can communicate with other terminals through at least one wireless communication protocol. This wireless communication protocol includes, but is not limited to: the World Wide Web, metropolitan area networks, intranets, various generations of mobile communication networks (2G, 3G, 4G, and 5G), wireless local area networks, and / or WiFi (Wireless Fidelity) networks. In some embodiments, the RF circuit 904 may also include circuitry related to NFC (Near Field Communication), which is not limited in this application.

[0269] Display screen 905 is used to display a UI (User Interface). This UI may include graphics, text, icons, videos, and any combination thereof. When display screen 905 is a touch display screen, it also has the ability to collect touch signals on or above its surface. These touch signals can be input as control signals to processor 901 for processing. In this case, display screen 905 can also be used to provide virtual buttons and / or a virtual keyboard, also known as soft buttons and / or a soft keyboard. In some embodiments, there may be one display screen 905, disposed on the front panel of terminal 900; in other embodiments, there may be at least two display screens 905, disposed on different surfaces of terminal 900 or in a folded design; in other embodiments, display screen 905 may be a flexible display screen, disposed on a curved or folded surface of terminal 900. Furthermore, display screen 905 may be configured as a non-rectangular irregular shape, i.e., a non-rectangular screen. Display screen 905 may be made of materials such as LCD (Liquid Crystal Display) or OLED (Organic Light-Emitting Diode).

[0270] The camera assembly 906 is used to acquire images or videos. Optionally, the camera assembly 906 includes a front-facing camera and a rear-facing camera. The front-facing camera is disposed on the front panel of the terminal, and the rear-facing camera is disposed on the back of the terminal. In some embodiments, there are at least two rear-facing cameras, which are any one of a main camera, a depth-sensing camera, a wide-angle camera, and a telephoto camera, to achieve background blurring by fusion of the main camera and the depth-sensing camera, panoramic shooting by fusion of the main camera and the wide-angle camera, VR (Virtual Reality) shooting, or other fusion shooting functions. In some embodiments, the camera assembly 906 may also include a flash. The flash may be a single-color temperature flash or a dual-color temperature flash. A dual-color temperature flash refers to a combination of a warm light flash and a cool light flash, which can be used for light compensation at different color temperatures.

[0271] The audio circuit 907 may include a microphone and a speaker. The microphone is used to collect sound waves from the user and the environment, converting them into electrical signals that are input to the processor 901 for processing, or to the radio frequency circuit 904 for voice communication. For stereo sound acquisition or noise reduction purposes, multiple microphones may be used, each positioned at a different location on the terminal 900. The microphone may also be an array microphone or an omnidirectional microphone. The speaker is used to convert electrical signals from the processor 901 or the radio frequency circuit 904 into sound waves. The speaker may be a conventional diaphragm speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, it can convert electrical signals not only into audible sound waves but also into inaudible sound waves for purposes such as distance measurement. In some embodiments, the audio circuit 907 may also include a headphone jack.

[0272] The positioning component 908 is used to determine the current geographic location of the terminal 900 in order to enable navigation or LBS (Location Based Service). The positioning component 908 can be a positioning component based on the US GPS (Global Positioning System), China's BeiDou system, or Russia's Galileo system.

[0273] Power supply 909 is used to supply power to the various components in terminal 900. Power supply 909 can be AC ​​power, DC power, a disposable battery, or a rechargeable battery. When power supply 909 includes a rechargeable battery, the rechargeable battery can be a wired rechargeable battery or a wireless rechargeable battery. A wired rechargeable battery is a battery that is charged via a wired line, while a wireless rechargeable battery is a battery that is charged via a wireless coil. The rechargeable battery can also be used to support fast charging technology.

[0274] In some embodiments, the terminal 900 further includes one or more sensors 910. The one or more sensors 910 include, but are not limited to: an accelerometer 911, a gyroscope 912, a pressure sensor 913, a fingerprint sensor 914, an optical sensor 915, and a proximity sensor 916.

[0275] Accelerometer 911 can detect the magnitude of acceleration along the three coordinate axes of a coordinate system established by terminal 900. For example, accelerometer 911 can be used to detect the components of gravitational acceleration along the three coordinate axes. Processor 901 can control display screen 905 to display the user interface in either a landscape or portrait view based on the gravitational acceleration signal acquired by accelerometer 911. Accelerometer 911 can also be used for games or for acquiring user motion data.

[0276] The gyroscope sensor 912 can detect the orientation and rotation angle of the terminal 900. The gyroscope sensor 912, in conjunction with the accelerometer sensor 911, can collect the user's 3D movements on the terminal 900. Based on the data collected by the gyroscope sensor 912, the processor 901 can perform the following functions: motion sensing (e.g., changing the UI based on the user's tilt), image stabilization during shooting, game control, and inertial navigation.

[0277] The pressure sensor 913 can be disposed on the side bezel of the terminal 900 and / or the lower layer of the display screen 905. When the pressure sensor 913 is disposed on the side bezel of the terminal 900, it can detect the user's grip signal on the terminal 900, and the processor 901 can perform left / right hand recognition or quick operation based on the grip signal collected by the pressure sensor 913. When the pressure sensor 913 is disposed on the lower layer of the display screen 905, the processor 901 can control the operable controls on the UI interface based on the user's pressure operation on the display screen 905. The operable controls include at least one of button controls, scroll bar controls, icon controls, and menu controls.

[0278] The fingerprint sensor 914 is used to collect the user's fingerprint. The processor 901 identifies the user's identity based on the fingerprint collected by the fingerprint sensor 914, or vice versa. When the user's identity is identified as trusted, the processor 901 authorizes the user to perform relevant sensitive operations, including unlocking the screen, viewing encrypted information, downloading software, making payments, and changing settings. The fingerprint sensor 914 can be located on the front, back, or side of the terminal 900. When the terminal 900 has physical buttons or a manufacturer's logo, the fingerprint sensor 914 can be integrated with the physical buttons or manufacturer's logo.

[0279] An optical sensor 915 is used to collect ambient light intensity. In one embodiment, the processor 901 can control the display brightness of the display screen 905 based on the ambient light intensity collected by the optical sensor 915. Specifically, when the ambient light intensity is high, the display brightness of the display screen 905 is increased; when the ambient light intensity is low, the display brightness of the display screen 905 is decreased. In another embodiment, the processor 901 can also dynamically adjust the shooting parameters of the camera assembly 906 based on the ambient light intensity collected by the optical sensor 915.

[0280] The proximity sensor 916, also known as a distance sensor, is installed on the front panel of the terminal 900. The proximity sensor 916 is used to detect the distance between the user and the front of the terminal 900. In one embodiment, when the proximity sensor 916 detects that the distance between the user and the front of the terminal 900 is gradually decreasing, the processor 901 controls the display screen 905 to switch from a screen-on state to a screen-off state; when the proximity sensor 916 detects that the distance between the user and the front of the terminal 900 is gradually increasing, the processor 901 controls the display screen 905 to switch from a screen-off state to a screen-on state.

[0281] Those skilled in the art will understand that Figure 9 The structure shown does not constitute a limitation on terminal 900, and may include more or fewer components than shown, or combine certain components, or use different component arrangements.

[0282] Optionally, the computer equipment is provided as a server. Figure 10 This is a schematic diagram of a server structure provided in an embodiment of this application. The server 1000 can vary significantly due to differences in configuration or performance. It may include one or more Central Processing Units (CPUs) 1001 and one or more memories 1002. The memories 1002 store at least one computer program, which is loaded and executed by the processor 1001 to implement the methods provided in the above-described method embodiments. Of course, the server may also have wired or wireless network interfaces, a keyboard, and input / output interfaces for input and output. The server may also include other components for implementing device functions, which will not be elaborated upon here.

[0283] This application also provides a computer-readable storage medium storing at least one computer program, which is loaded and executed by a processor to perform the operations performed in the hand key point detection method of the above embodiments.

[0284] This application also provides a computer program product or computer program, which includes computer program code stored in a computer-readable storage medium. A processor of a computer device reads the computer program code from the computer-readable storage medium and executes the computer program code, causing the computer device to perform the operations performed in the hand keypoint detection method of the above embodiments.

[0285] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0286] The above description is only an optional embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present application should be included within the protection scope of the present application.

Claims

1. A hand keypoint detection method, characterized in that, The method comprises: obtaining feature information of a hand image, the hand image containing a hand; performing feature transformation on the feature information to obtain pose parameters and shape parameters corresponding to the hand; calling a hand construction model to process the pose parameters and the shape parameters to obtain a three-dimensional hand model of the hand and determine three-dimensional hand key points contained in the three-dimensional hand model; projecting the three-dimensional hand key points in the hand image to obtain two-dimensional hand key points in the hand image; The method further comprises: obtaining a sample hand image and sample pose parameters and sample shape parameters corresponding to a sample hand contained in the sample hand image; calling a key point detection model to perform key point detection on the sample hand image to obtain sample two-dimensional hand key points in the sample hand image and confidence degrees corresponding to the sample two-dimensional hand key points; calling the hand construction model to process the sample pose parameters and the sample shape parameters to obtain a predicted three-dimensional hand model and determine predicted three-dimensional hand key points contained in the predicted three-dimensional hand model; projecting the predicted three-dimensional hand key points in the sample hand image to obtain predicted two-dimensional hand key points in the sample hand image; and training the hand construction model according to the sample two-dimensional hand key points and the predicted two-dimensional hand key points. The training of the hand construction model according to the sample two-dimensional hand key points and the predicted two-dimensional hand key points comprises: performing weighted processing on distances between the sample two-dimensional hand key points and the predicted two-dimensional hand key points according to the confidence degrees to obtain a loss value of the hand construction model; and training the hand construction model according to the loss value.

2. The method of claim 1, wherein, The three-dimensional hand model is formed by a plurality of vertex connections; and the determination of the three-dimensional hand key points contained in the three-dimensional hand model comprises: determining vertex serial numbers of the plurality of vertices; determining, as the three-dimensional hand key points, vertices corresponding to key point serial numbers in the determined plurality of vertex serial numbers.

3. The method of claim 1, wherein, The sample two-dimensional hand key points comprise a plurality of sample two-dimensional hand key points, and the predicted two-dimensional hand key points comprise a plurality of predicted two-dimensional hand key points. The weighted processing of the distances between the sample two-dimensional hand key points and the predicted two-dimensional hand key points according to the confidence degrees to obtain the loss value comprises: determining distances between each sample two-dimensional hand key point and a corresponding predicted two-dimensional hand key point; determining products of each distance and a confidence degree corresponding to the corresponding sample two-dimensional hand key point; determining a sum of the determined plurality of products as the loss value.

4. The method of claim 1, wherein, The projection of the three-dimensional hand key points in the hand image to obtain the two-dimensional hand key points in the hand image comprises: converting the three-dimensional hand key points to a camera coordinate system to obtain reference three-dimensional hand key points; projecting the reference three-dimensional hand key points in the hand image according to shooting parameters corresponding to the hand image to obtain the two-dimensional hand key points, the shooting parameters being shooting parameters used when the hand image is shot.

5. The method of claim 4, wherein, The converting the three-dimensional hand key points into a camera coordinate system to obtain reference three-dimensional hand key points comprises: Obtaining a view angle parameter corresponding to the hand image, the view angle parameter being used to indicate a display position of the hand in a camera coordinate system; According to the view angle parameter, the three-dimensional hand key points are converted into the camera coordinate system to obtain the reference three-dimensional hand key points.

6. The method of claim 5, wherein, The obtaining of the view angle parameter corresponding to the hand image comprises: Calling a dimension transformation layer in a parameter extraction model to perform dimension transformation on the feature information to obtain a reference feature vector; Calling a view angle parameter extraction layer in the parameter extraction model to perform feature transformation on the reference feature vector to obtain the view angle parameter.

7. The method of claim 1, wherein, The feature transformation on the feature information to obtain the gesture parameter and the shape parameter corresponding to the hand comprises: Calling a dimension transformation layer in a parameter extraction model to perform dimension transformation on the feature information to obtain a reference feature vector; Calling a gesture parameter extraction layer and a shape parameter extraction layer in the parameter extraction model to perform feature transformation on the reference feature vector to obtain the gesture parameter and the shape parameter.

8. The method according to any one of claims 1 to 7, characterized in that, After the projecting the three-dimensional hand key points in the hand image to obtain two-dimensional hand key points in the hand image, the method further comprises: According to the two-dimensional hand key points, performing sign language recognition on the hand image to obtain text information corresponding to the hand image, the text information being used to represent a sign language meaning represented by the hand in the hand image.

9. A hand keypoint detection apparatus, comprising: The device comprises: A first obtaining module is configured to obtain feature information of a hand image, the hand image containing a hand; A feature transformation module is configured to perform feature transformation on the feature information to obtain a gesture parameter and a shape parameter corresponding to the hand; A determining module is configured to construct a three-dimensional hand model of the hand according to the gesture parameter and the shape parameter, and determine three-dimensional hand key points contained in the three-dimensional hand model; A projecting module is configured to project the three-dimensional hand key points in the hand image to obtain two-dimensional hand key points in the hand image; The determining module comprises a first determining unit configured to call a hand construction model to process the gesture parameter and the shape parameter to obtain the three-dimensional hand model and determine the three-dimensional hand key points contained in the three-dimensional hand model; The device further comprises: A second obtaining module is configured to obtain a sample hand image and sample gesture parameters and sample shape parameters corresponding to a sample hand contained in the sample hand image; A detecting module is further configured to call a key point detection model to perform key point detection on the sample hand image to obtain sample two-dimensional hand key points in the sample hand image; The determining module is further configured to call the hand construction model to process the sample gesture parameters and the sample shape parameters to obtain a predicted three-dimensional hand model and determine predicted three-dimensional hand key points contained in the predicted three-dimensional hand model; The projection module is further configured to project the predicted three-dimensional hand key points in the sample hand image to obtain predicted two-dimensional hand key points in the sample hand image. The training module is configured to train the hand construction model according to the sample two-dimensional hand key points and the predicted two-dimensional hand key points. The training module comprises a second determination unit configured to determine a loss value of the hand construction model according to the sample two-dimensional hand key points and the predicted two-dimensional hand key points; and a training unit configured to train the hand construction model according to the loss value. The detection module is configured to call the key point detection model to perform key point detection on the sample hand image to obtain the sample two-dimensional hand key points and confidence degrees corresponding to the sample two-dimensional hand key points. The second determination unit is configured to perform weighted processing on distances between the sample two-dimensional hand key points and the predicted two-dimensional hand key points according to the confidence degrees to obtain the loss value.

10. A computer device, comprising: The computer device comprises a processor and a memory, and the memory stores at least one computer program, which is loaded and executed by the processor to implement operations performed in the hand key point detection method according to any one of claims 1 to 9.

11. A computer readable storage medium, characterized in that, The computer readable storage medium stores at least one computer program, which is loaded and executed by the processor to implement operations performed in the hand key point detection method according to any one of claims 1 to 9.

Images