Hand key point detection method and mobile terminal

Abstract

The present application provides a hand key point detection method and mobile terminal for improving the accuracy of detected hand key points. The method comprises: performing hand detection on any image in a video captured by a mobile terminal using a pre-trained hand detection model to obtain a hand region image; inputting the hand region image into the pre-trained hand key point detection model to perform hand key point detection to obtain hand key points; wherein the hand key point detection model is trained in the following manner: training a hand heat map detection model based on a first training sample to obtain target model parameters of the hand heat map detection model; setting model parameters of the hand key point detection model using the target model parameters to obtain a set hand key point detection model; and training the set hand key point detection model using a second training sample to obtain a trained hand key point detection model.

Description

Technical Field

[0001] The present application relates to the field of virtual reality technology, and in particular to a hand key point detection method and a mobile terminal. Background Art

[0002] With the rapid development of the metaverse, demand for AR (Augmented Reality) and VR (Virtual Reality) is increasing. Currently, AR / VR technology is primarily present in people's lives through smartphones or AR / VR devices. However, AR / VR primarily relies on visual perception, making its technology highly dependent on data and algorithms. With the rapid development of visual algorithms, many traditional vision algorithms are gradually being replaced by deep learning, such as those for object detection, object segmentation, and face recognition. Therefore, the development of deep learning-related algorithms for mobile terminals has become an inevitable demand.

[0003] This is to further enhance the application of mobile terminals in scenarios such as 3D (Three Dimensions) ranging, environmental recognition, and gesture recognition. In the field of AR glasses, due to the structure of AR glasses, operating AR glasses requires manually pressing relevant buttons to realize corresponding functions. There are also methods to replace interface operations by using their posture. Obviously, this interaction method is not very natural and the experience is average. To solve the interaction problem, gesture recognition is added to realize natural gesture interaction. Therefore, the interaction between gestures and mobile terminals is indispensable. Therefore, it is necessary to detect the key points of the hand.

[0004] In the prior art, the method for detecting hand key points in mobile terminals uses a deep learning palm detector (target detection algorithm) to detect the palm and the corresponding hand key points. A tracking algorithm is then applied to track the palm. If tracking fails, palm detection is performed again. However, due to the diversity of hand postures, hand area tracking errors or loss are easily caused. Furthermore, the tracking process struggles to capture the absolute area of ​​the hand, resulting in low-quality input for the extraction of hand key points and distorted key point detection. Consequently, the accuracy of hand key point detection is low. Summary of the Invention

[0005] The present application provides a hand key point detection method and a mobile terminal, which are used to automatically detect hand key points in a mobile terminal and improve the accuracy of the detected hand key points.

[0006] In a first aspect, an embodiment of the present application provides a hand key point detection method, which is applied to a mobile terminal. The method includes:

[0007] For any frame of image in the video acquired by the mobile terminal, perform hand detection on the image using a pre-trained hand detection model to obtain a hand area image;

[0008] Inputting the hand area image into a pre-trained hand key point detection model to perform hand key point detection to obtain hand key points;

[0009] The hand key point detection model is trained in the following manner:

[0010] Acquire a first training sample, wherein the first training sample includes a plurality of hand images and hand heat maps corresponding to the plurality of hand images respectively;

[0011] Training a hand heat map detection model based on the first training sample to obtain target model parameters of the hand heat map detection model; wherein the hand heat map detection model and the hand key point detection model have different structures in the last layer, and other model structures except the structure of the last layer are the same;

[0012] Setting each model parameter of the hand key point detection model using the target model parameters to obtain a set hand key point detection model;

[0013] Acquire a second training sample, wherein the second training sample includes a plurality of hand images and hand key points corresponding to the hand images respectively;

[0014] The set hand key point detection model is trained using the second training sample to obtain a trained hand key point detection model.

[0015] A second aspect of the present application provides a mobile terminal, including a processor and a memory, wherein the processor and the memory are connected via a bus;

[0016] The memory stores a computer program, and the processor is configured to perform the following operations based on the computer program:

[0017] For any frame of image in the video acquired by the mobile terminal, perform hand detection on the image using a pre-trained hand detection model to obtain a hand area image;

[0018] Inputting the hand area image into a pre-trained hand key point detection model to perform hand key point detection to obtain hand key points;

[0019] The hand key point detection model is trained in the following manner:

[0020] Acquire a first training sample, wherein the first training sample includes a plurality of hand images and hand heat maps corresponding to the plurality of hand images respectively;

[0021] Training a hand heat map detection model based on the first training sample to obtain target model parameters of the hand heat map detection model; wherein the hand heat map detection model and the hand key point detection model have different structures in the last layer, and other model structures except the structure of the last layer are the same;

[0022] Setting each model parameter of the hand key point detection model using the target model parameters to obtain a set hand key point detection model;

[0023] Acquire a second training sample, wherein the second training sample includes a plurality of hand images and hand key points corresponding to the hand images respectively;

[0024] The set hand key point detection model is trained using the second training sample to obtain a trained hand key point detection model.

[0025] According to a third aspect of an embodiment of the present invention, a computer storage medium is provided, wherein the computer storage medium stores a computer program, and the computer program is used to execute the method according to the first aspect.

[0026] In the above-mentioned embodiment of the present application, hand detection is performed on each frame of the video acquired in the mobile terminal, and then hand key point detection is performed on the detected image to obtain the hand key points. Therefore, in this embodiment, hand detection and hand key point detection can be performed on each frame of the image, thereby achieving accurate tracking of the hand and improving the accuracy of the detected hand key points. In addition, in this embodiment, the hand heat map detection model is first trained based on the first training sample to obtain the target model parameters, and then the model parameters of the hand key point detection model are set based on the target model parameters before the hand key point detection model is trained. Since the hand heat map obtained by the heat map detection model can more accurately determine the position of the hand key points, the target model parameters are determined by the hand map detection model to set the model parameters of the hand key point detection model, so that the recognition accuracy of the hand key point detection model is more accurate, further improving the recognition accuracy of the hand key point detection model. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0028] Figure 1 One of the flowcharts of the hand key point detection method provided in an embodiment of the present application is exemplarily shown;

[0029] Figure 2 The following is a schematic diagram of a hand area image provided by an embodiment of the present application;

[0030] Figure 3 The following is an exemplary diagram of the structure of a hand detection model provided in an embodiment of the present application;

[0031] Figure 4 The following is a schematic diagram illustrating a process of hand detection according to an embodiment of the present application;

[0032] Figure 5 The following is a schematic diagram showing the process of feature extraction performed by the first feature extraction layer provided in an embodiment of the present application;

[0033] Figure 6 The following is an exemplary diagram of the key points of the hand provided in an embodiment of the present application;

[0034] Figure 7 The following is a schematic diagram illustrating the structure of a hand key point detection model provided in an embodiment of the present application;

[0035] Figure 8 The following is a schematic diagram illustrating a process for determining key points of a hand according to an embodiment of the present application;

[0036] Figure 9 The following is a schematic diagram illustrating a process of extracting features from a hand region feature map provided in an embodiment of the present application;

[0037] Figure 10 The following is a schematic diagram illustrating a specific process of training a hand key point detection model according to an embodiment of the present application;

[0038] Figure 11 The following is an exemplary diagram of the structure of a hand heat map detection model provided in an embodiment of the present application;

[0039] Figure 12 The second flowchart of the hand key point detection method provided in the embodiment of the present application is exemplarily shown;

[0040] Figure 13The following is a schematic diagram of a hand key point detection device according to an embodiment of the present application;

[0041] Figure 14 The hardware structure diagram of the mobile terminal provided in the embodiment of the present application is exemplified. DETAILED DESCRIPTION

[0042] In order to make the purpose, implementation mode and advantages of the present application clearer, the exemplary implementation mode of the present application will be clearly and completely described below in conjunction with the drawings in the exemplary embodiments of the present application. Obviously, the described exemplary embodiments are only part of the embodiments of the present application, not all of the embodiments.

[0043] All other embodiments derived by persons of ordinary skill in the art based on the exemplary embodiments described herein without inventive effort are within the scope of protection of the claims appended hereto. Furthermore, although the disclosure herein is presented based on one or more exemplary embodiments, it should be understood that each aspect of the disclosure may constitute a complete embodiment on its own.

[0044] It should be noted that the brief descriptions of terms in this application are only for the purpose of facilitating the understanding of the embodiments described below, and are not intended to limit the embodiments of this application. Unless otherwise specified, these terms should be understood according to their ordinary and usual meanings.

[0045] The terms "first," "second," and the like in the specification and claims of this application and in the accompanying drawings are used to distinguish similar objects and are not necessarily used to describe a particular order or sequential sequence. It should be understood that the terms used in this manner are interchangeable where appropriate, so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. In addition, the terms "including" and "having," and any variations thereof, are intended to cover, but not exclude, inclusion. For example, a product or device comprising a list of components is not necessarily limited to those components explicitly listed, but may include other components not explicitly listed or inherent to the product or device.

[0046] The term "module" as used in this application refers to any known or later developed hardware, software, firmware, artificial intelligence, fuzzy logic or combination of hardware and / or software code that is capable of performing the functions associated with that element.

[0047] The following is an overview of the concepts of the embodiments of the present application.

[0048] Currently, hand keypoint detection in mobile terminals relies on a deep learning palm detector to detect the palm and its corresponding keypoints. A tracking algorithm is then applied to track the palm, and palm detection is performed again if tracking fails. However, due to the diversity of hand postures, hand tracking errors or loss are common. Furthermore, the tracking process struggles to accurately capture the hand's absolute area, resulting in low-quality input for hand keypoint extraction and distorted keypoint detection. Consequently, the accuracy of hand keypoint detection is low.

[0049] Based on the problem of low hand key point detection accuracy in the prior art, an embodiment of the present application provides a hand key point detection method. This method performs hand detection on each frame of a video captured by a mobile terminal, then performs hand key point detection on the detected image to obtain hand key points. Thus, this embodiment can perform hand detection and hand key point detection on each frame, achieving accurate hand tracking and improving the accuracy of detected hand key points. Furthermore, in this embodiment, a hand heat map detection model is first trained based on a first training sample to obtain target model parameters. The model parameters of the hand key point detection model are then set based on the target model parameters before the hand key point detection model is trained again. Because the hand heat map obtained by the heat map detection model can more accurately determine the location of hand key points, the target model parameters are determined by the hand map detection model and used to set the model parameters of the hand key point detection model. This increases the recognition accuracy of the hand key point detection model and further improves the recognition accuracy of the hand key point detection model. The embodiments of the present application are described in detail below with reference to the accompanying drawings.

[0050] like Figure 1 FIG. 1 is a flow chart of a hand key point detection method, which may include the following steps:

[0051] Step 101: For any frame of image in a video acquired by a mobile terminal, perform hand detection on the image using a pre-trained hand detection model to obtain a hand region image;

[0052] like Figure 2 The figure below shows a schematic diagram of a hand region image. As can be seen from the figure, the hand detection model will frame the hand region in the image. Then, the hand key point detection algorithm is used to detect the hand key points in the hand region.

[0053] in, Figure 3 This is a structural diagram of the hand detection model. Figure 3It can be seen that the hand detection model includes a SliceM layer, a first feature extraction layer, a second feature extraction layer, a third feature extraction layer, a target detection detect layer, a depth-separable convolution layer, an upsampling layer, and multiple connection concat layers.

[0054] Next, the hand region image obtained in step 101 is described in detail. Figure 4 FIG. 1 is a flow chart showing how to use a pre-trained hand detection model to perform hand detection on an image to obtain a hand region image, which may include the following steps:

[0055] Step 401: using the SliceM layer to divide the image into a specified number of target images, and performing downsampling and feature extraction on each target image to obtain a specified number of feature maps, and then splicing the specified number of feature maps to obtain a first feature map;

[0056] Step 402: Continuously perform multiple feature extractions on the first feature map using a first feature extraction layer to obtain a second feature map, a third feature map, and a fourth feature map, wherein the third feature map is obtained based on the second feature map, and the fourth feature map is obtained based on the third feature map;

[0057] from Figure 3 It can be seen that the first feature extraction layer includes four first intermediate feature extraction layers, two second intermediate feature extraction layers, an upsampling layer and a third intermediate feature extraction layer, wherein the first intermediate feature extraction layer includes a convolution conv layer and a feature extraction CSP layer, the second intermediate feature extraction layer includes a depth-separable convolution layer, and the third intermediate feature layer includes a conv layer and a feature extraction SPP layer;

[0058] In one embodiment, Figure 5 FIG. 1 is a flow chart showing how to continuously perform multiple feature extractions on the first feature map using the first feature extraction layer, which may include the following steps:

[0059] Step 501: sequentially using three consecutive first intermediate feature extraction layers to perform feature extraction three times on the first feature map to obtain the second feature map and the third feature map, wherein the second feature map is obtained by the second first intermediate feature extraction layer, and the third feature map is obtained by the third first intermediate feature extraction layer;

[0060] In one embodiment, for any first intermediate feature extraction layer, the conv layer is first used to extract features from the image, and then the CSP layer is used to extract features again.

[0061] Step 502: Using the second intermediate feature extraction layer to perform feature extraction on the third feature map to obtain a first sub-feature map;

[0062] In one embodiment, step 502 may be implemented as follows: performing feature extraction on the third feature map using a depthwise separable convolutional layer to obtain the first sub-feature map.

[0063] Step 503: performing feature extraction on the first sub-feature map through the third intermediate feature extraction layer to obtain a second sub-feature map;

[0064] In one embodiment, step 503 may be implemented as follows: first performing feature extraction on the first sub-feature map using a conv layer, and then performing feature extraction again using an SPP layer to obtain the second sub-feature map.

[0065] Step 504: performing feature extraction on the second sub-feature map using the first intermediate feature extraction layer to obtain a third sub-feature map;

[0066] Step 505: After performing feature extraction on the third sub-feature map using the second intermediate feature extraction layer, upsampling the image using the upsampling layer to obtain the fourth feature map.

[0067] In one embodiment, performing feature extraction on the third sub-feature map using the second intermediate feature extraction layer in step 505 may be implemented as: performing feature extraction on the third sub-feature map using a depthwise separable convolutional layer on the second intermediate feature extraction layer.

[0068] Step 403: concatenate the third feature map and the fourth feature map through a concat layer to obtain a fifth feature map, perform feature extraction on the fifth feature map through the depthwise separable convolution layer, and then perform image upsampling through an upsampling layer to obtain a sixth feature map;

[0069] Step 404: Connecting the sixth feature map and the second feature map through a concat layer to obtain a seventh feature map, and performing multiple feature extractions on the seventh feature map using the second feature extraction layer to obtain a first intermediate feature map and a second intermediate feature map, wherein the second intermediate feature map is obtained based on the first intermediate feature map;

[0070] wherein the second feature extraction layer includes a CSP layer and a depthwise separable convolutional layer, and the third feature extraction layer includes a CSP layer;

[0071] In one embodiment, step 404 uses the second feature extraction layer to perform multiple feature extractions on the seventh feature map to obtain a first intermediate feature map and a second intermediate feature map, which can be implemented as follows: using the CSP layer to extract features on the seventh feature map to obtain the first intermediate feature map, and performing feature extraction on the first intermediate feature map again through the depthwise separable convolutional layer to obtain the second intermediate feature map.

[0072] Step 405: Using a concat layer, the second intermediate feature map and the sixth feature map are connected to obtain an eighth feature map, and the second feature extraction layer is used to perform multiple feature extractions on the eighth feature map to obtain a third intermediate feature map and a fourth intermediate feature map, wherein the fourth intermediate feature map is obtained based on the third intermediate feature map;

[0073] In one embodiment, in step 405, performing multiple feature extractions on the eighth feature map using the second feature extraction layer to obtain a third intermediate feature map and a fourth intermediate feature map can be implemented as follows: performing feature extraction on the eighth feature map using the CSP layer to obtain the third intermediate feature map, and performing feature extraction on the third intermediate feature map again through the depthwise separable convolutional layer to obtain the fourth intermediate feature map.

[0074] Step 406: using a concat layer to connect the fourth intermediate feature map and the fourth feature map to obtain a fifth intermediate feature map, and using the third feature extraction layer to extract features from the fifth intermediate feature map to obtain a target feature map;

[0075] In one embodiment, in step 406, performing feature extraction on the fifth intermediate feature map using the third feature extraction layer to obtain the target feature map can be implemented as follows: performing feature extraction on the fifth intermediate feature map using the CSP layer to obtain the target feature map.

[0076] Step 407: Perform hand detection on the target feature map, the first intermediate feature map, and the second intermediate feature map through the detect layer to obtain the hand area image.

[0077] Among them, the detect layer in this embodiment is the detect layer in YOLOv5, which is a structure in the prior art and will not be described in detail in this embodiment.

[0078] Step 102: Inputting the hand region image into a pre-trained hand key point detection model to perform hand key point detection to obtain hand key points;

[0079] Among them, the number of hand key points in this embodiment is 21, such as Figure 6The following table shows the hand key points identified in this embodiment. However, this embodiment does not limit the number of hand key points; the number can be set based on actual conditions. Furthermore, because the hand detection model and hand key point detection model in this embodiment are lightweight, the difficulty of deployment on mobile terminal platforms is reduced.

[0080] like Figure 7 As shown in the figure, it is a structural diagram of the hand key point detection model. Figure 7 As can be seen in the figure, the hand key point detection model includes SliceM layer, the first feature extraction layer, the fourth feature extraction layer, the regression layer, the depth-wise separable convolution layer, the upsampling layer and the concat layer;

[0081] Next, the method for determining the key points of the hand in step 102 is described in detail. Figure 8 The figure shows a flow chart of determining the key points of the hand, which includes the following steps:

[0082] Step 801: using the SliceM layer to divide the hand region image into a specified number of target hand region images, downsampling and feature extraction are performed on each target hand region image to obtain a specified number of feature maps, and then the specified number of feature maps are spliced ​​to obtain a hand region feature map;

[0083] Step 802: Continuously perform multiple feature extractions on the hand region feature map using the first feature extraction layer to obtain a first hand region feature map, a second hand region feature map, and a third hand region feature map, wherein the second hand region feature map is obtained based on the first hand region feature map, and the third hand region feature map is obtained based on the second hand region feature map.

[0084] The first feature extraction layer includes four first intermediate feature extraction layers, two second intermediate feature extraction layers, an upsampling layer and a third intermediate feature extraction layer, wherein the first intermediate feature extraction layer includes a convolutional layer and a feature extraction CSP layer, the second intermediate feature extraction layer includes a depth-separable convolutional layer, and the third intermediate feature layer includes a conv layer and a feature extraction SPP layer;

[0085] In one embodiment, Figure 9 FIG. 8 is a flow chart showing how to continuously perform multiple feature extractions on the hand region feature map using the first feature extraction layer in step 802, which may include the following steps:

[0086] Step 901: performing feature extraction three times on the first feature map using three consecutive first intermediate feature extraction layers to obtain the first hand region feature map and the second hand region feature map, wherein the first hand region feature map is obtained by the second first intermediate feature extraction layer, and the second hand region feature map is obtained by the third first intermediate feature extraction layer;

[0087] Step 902: performing feature extraction on the second hand region feature map using the second intermediate feature extraction layer to obtain a first sub-hand region feature map;

[0088] In one embodiment, step 902 may be implemented as follows: performing feature extraction on the second hand region feature map using a depthwise separable convolutional layer to obtain the first sub-hand region feature map.

[0089] Step 903: extracting features from the first sub-hand region feature map through the third intermediate feature extraction layer to obtain a second sub-hand region feature map;

[0090] In one embodiment, step 903 may be implemented as follows: firstly performing feature extraction on the first sub-hand region feature map using a conv layer, and then performing feature extraction again using the SPP layer to obtain the second sub-hand region feature map.

[0091] Step 904: performing feature extraction on the second sub-hand region feature map using the first intermediate feature extraction layer to obtain a third sub-hand region feature map;

[0092] In one embodiment, step 904 may be implemented as follows: performing feature extraction on the second sub-hand region feature map using a conv layer, and then performing feature extraction again using a CSP layer to obtain the third sub-hand region feature map.

[0093] Step 905: After performing feature extraction on the third sub-hand region feature map using the second intermediate feature extraction layer, the upsampling layer is used to upsample the image to obtain the third hand region feature map.

[0094] In one embodiment, after performing feature extraction on the third sub-hand region feature map using the second intermediate feature extraction layer in step 905, performing image upsampling using the upsampling layer can be implemented as follows: performing feature extraction on the third sub-hand region feature map using a depthwise separable convolutional layer.

[0095] Step 803: concatenating the second hand region feature map and the third hand region feature map through the concat layer to obtain a fourth hand region feature map, performing feature extraction on the fourth hand region feature map through the depthwise separable convolution layer, and then performing image upsampling on the fourth hand region feature map after feature extraction through the upsampling layer to obtain a fifth hand region feature map;

[0096] Step 804: using the concat layer to connect the fifth hand region feature map and the first hand region feature map to obtain a sixth hand region feature map;

[0097] Step 805: performing feature extraction on the sixth hand region feature map using the fourth feature extraction layer to obtain a target hand region feature map;

[0098] Wherein, the fourth feature extraction layer includes a feature extraction CSP layer and two CPM layers;

[0099] In one embodiment, step 805 can be implemented as follows: using the CSP layer to perform feature extraction on the sixth hand area feature map to obtain a fourth sub-hand area feature map; using the CPM layer to perform feature extraction on the fourth sub-hand area feature map to obtain a fifth sub-hand area feature map, and using the CPM layer to perform feature extraction on the fifth sub-hand area feature map to obtain the target hand area feature map.

[0100] Among them, such as Figure 9 As shown in the figure, it is a schematic diagram of the structure of the CPM layer. Figure 9 As can be seen from the figure, the CPM layer includes two convolutional layers and three depth-separable convolutional layers;

[0101] In one embodiment, using the CPM layer to perform feature extraction on the fourth sub-hand area feature map to obtain the fifth sub-hand area feature map can be specifically implemented as follows: using a convolutional layer to perform feature extraction on the fourth sub-hand area feature map to obtain a first intermediate hand area feature map; and using three depthwise separable convolutional layers to perform feature extraction on the fourth sub-hand area feature map three times in a row to obtain a second intermediate hand area feature map; fusing the first intermediate hand area feature map and the second intermediate hand area feature map to obtain a third intermediate hand area feature map; and using a convolutional layer to perform feature extraction on the third intermediate hand area feature map to obtain the fifth sub-hand area feature map.

[0102] It should be noted that the step of using the CPM layer to extract features from the fifth sub-hand area feature map to obtain the target hand area feature map is the same as the step of obtaining the fifth sub-hand area feature map in the above embodiment, and will not be repeated here in this embodiment.

[0103] Step 806: Use the regression layer to perform regression processing on the target hand area feature map to obtain the hand key points.

[0104] The regression layer in this embodiment uses a linear fully connected layer, but the specific regression layer can be set according to actual conditions, and this embodiment does not limit the specific structure of the regression layer. In addition, the number of hand key points in this embodiment is 21, but the number of hand key points can be set according to actual conditions, and this embodiment does not limit the number of hand key points.

[0105] The following is a detailed introduction to the training method of the hand key point detection model. Figure 10 The figure shows a specific process diagram for training a hand key point detection model, which may include the following steps:

[0106] Step 1001: Acquire a first training sample, wherein the first training sample includes a plurality of hand images and hand heat maps corresponding to the plurality of hand images respectively;

[0107] Step 1002: Training a hand heat map detection model based on the first training sample to obtain target parameters of the hand heat map detection model; wherein the hand heat map detection model and the hand key point detection model have different structures in the last layer, and other model structures except the structure of the last layer are the same;

[0108] The target model parameters are parameters of other model structures except the last layer structure in the hand heat map detection model.

[0109] like Figure 11 As shown in the figure, it is a structural diagram of the hand heat map detection model, which includes a SliceM layer, a first feature extraction layer, a fourth feature extraction layer, a convolution conv layer, a depth-separable convolution layer, an upsampling layer and a concat layer. As can be seen from the figure, the last layer of the hand heat map detection model is a conv layer. However, the last layer of the hand key point detection model is a regression layer. Therefore, in this embodiment, the hand heat map detection model is first trained based on the first training sample to obtain the target model parameters, and then the model parameters of the hand key point detection model are set before the hand key point detection model is trained. In this way, the recognition accuracy of the hand key point detection model is further improved.

[0110] Step 1003: setting each model parameter of the hand key point detection model using the target model parameters to obtain a set hand key point detection model;

[0111] In one embodiment, step 1003 may be implemented as follows: using the target model parameters to set parameters of other model structures except the last layer model structure in the hand key point detection model. Specifically, the parameters of the other model structures are set to be the same as the target model parameters.

[0112] Step 1004: Acquire a second training sample, wherein the second training sample includes a plurality of hand images and hand key points corresponding to the hand images;

[0113] Step 1005: Use the second training sample to train the set hand key point detection model to obtain a trained hand key point detection model.

[0114] In one embodiment, step 1005 may be implemented as follows: using the second training sample to train the last layer of the set hand key point detection model to obtain a trained hand key point detection model.

[0115] It should be noted that the hand key point detection method in this embodiment is performed in a mobile terminal, so it needs to be deployed in the mobile terminal. Specifically, in this embodiment, the hand key point detection method in this embodiment is deployed on a mobile terminal platform such as Android using an NCNN or TNN framework. However, this embodiment does not limit the deployment method, and the specific deployment method can be set according to actual circumstances.

[0116] In order to further connect the technical solutions in this application, Figure 12 A detailed description may include the following steps:

[0117] Step 1201: For any frame of image in the video acquired by the mobile terminal, the SliceM layer in the pre-trained hand detection model is used to divide the image into a specified number of target images, and each target image is downsampled and feature extracted. After obtaining a specified number of feature maps, the specified number of feature maps are spliced ​​to obtain a first feature map;

[0118] Step 1202: Continuously perform multiple feature extractions on the first feature map using a first feature extraction layer to obtain a second feature map, a third feature map, and a fourth feature map, wherein the third feature map is obtained based on the second feature map, and the fourth feature map is obtained based on the third feature map;

[0119] Step 1203: concatenate the third feature map and the fourth feature map through a concat layer to obtain a fifth feature map, perform feature extraction on the fifth feature map through the depthwise separable convolution layer, and then perform image upsampling through an upsampling layer to obtain a sixth feature map;

[0120] Step 1204: concatenate the sixth feature map and the second feature map through a concat layer to obtain a seventh feature map, and perform multiple feature extractions on the seventh feature map using the second feature extraction layer to obtain a first intermediate feature map and a second intermediate feature map, wherein the second intermediate feature map is obtained based on the first intermediate feature map;

[0121] Step 1205: Using a concat layer, concatenate the second intermediate feature map and the sixth feature map to obtain an eighth feature map, and use the second feature extraction layer to perform multiple feature extractions on the eighth feature map to obtain a third intermediate feature map and a fourth intermediate feature map, wherein the fourth intermediate feature map is obtained based on the third intermediate feature map;

[0122] Step 1206: using a concat layer to connect the fourth intermediate feature map and the fourth feature map to obtain a fifth intermediate feature map, and using the third feature extraction layer to extract features from the fifth intermediate feature map to obtain a target feature map;

[0123] Step 1207: Perform hand detection on the target feature map, the first intermediate feature map, and the second intermediate feature map through a detect layer to obtain the hand region image;

[0124] Step 1208: Using the SliceM layer in the pre-trained hand key point detection model, the hand region image is divided into a specified number of target hand region images, and each target hand region image is downsampled and feature extracted. After obtaining a specified number of feature maps, the specified number of feature maps are spliced ​​together to obtain a hand region feature map.

[0125] Step 1209: Continuously perform multiple feature extractions on the hand region feature map using the first feature extraction layer to obtain a first hand region feature map, a second hand region feature map, and a third hand region feature map, wherein the second hand region feature map is obtained based on the first hand region feature map, and the third hand region feature map is obtained based on the second hand region feature map.

[0126] Step 1210: Concatenating the second hand region feature map and the third hand region feature map using the concat layer to obtain a fourth hand region feature map, performing feature extraction on the fourth hand region feature map using the depthwise separable convolution layer, and then upsampling the fourth hand region feature map after feature extraction using the upsampling layer to obtain a fifth hand region feature map;

[0127] Step 1211: using the concat layer to connect the fifth hand region feature map and the first hand region feature map to obtain a sixth hand region feature map;

[0128] Step 1212: performing feature extraction on the sixth hand region feature map using the fourth feature extraction layer to obtain a target hand region feature map;

[0129] Step 1213: Use the regression layer to perform regression processing on the target hand area feature map to obtain the hand key points.

[0130] Based on the same inventive concept, the hand key point detection method disclosed above can also be implemented by a hand key point detection device. The effect of the hand key point detection device is similar to that of the aforementioned method, and will not be repeated here.

[0131] Figure 13 Schematic diagram of the structure of a hand key point detection device according to an embodiment of the present disclosure.

[0132] like Figure 13 As shown, the hand key point detection device 1300 of the present disclosure may include a hand detection module 1310 , a hand key point detection module 1320 and a hand key point detection model training module 1330 .

[0133] The hand detection module 1310 is configured to perform hand detection on any frame of a video acquired by the mobile terminal using a pre-trained hand detection model to obtain a hand region image.

[0134] A hand key point detection module 1320 is configured to input the hand region image into a pre-trained hand key point detection model to perform hand key point detection and obtain hand key points;

[0135] The hand key point detection model training module 1330 is used to train the hand key point detection model in the following manner:

[0136] Acquire a first training sample, wherein the first training sample includes a plurality of hand images and hand heat maps corresponding to the plurality of hand images respectively;

[0137] Training a hand heat map detection model based on the first training sample to obtain target model parameters of the hand heat map detection model; wherein the hand heat map detection model and the hand key point detection model have different structures in the last layer, and other model structures except the structure of the last layer are the same;

[0138] Setting each model parameter of the hand key point detection model using the target model parameters to obtain a set hand key point detection model;

[0139] Acquire a second training sample, wherein the second training sample includes a plurality of hand images and hand key points corresponding to the hand images respectively;

[0140] The set hand key point detection model is trained using the second training sample to obtain a trained hand key point detection model.

[0141] In one embodiment, the hand detection model includes a SliceM layer, a first feature extraction layer, a second feature extraction layer, a third feature extraction layer, a target detection layer, a depth-separable convolution layer, an upsampling layer, and a plurality of concat layers;

[0142] The hand detection module 1310 is specifically configured to:

[0143] Using the SliceM layer to divide the image into a specified number of target images, downsampling and feature extraction are performed on each target image to obtain a specified number of feature maps, and then the specified number of feature maps are spliced ​​to obtain a first feature map;

[0144] Continuously performing multiple feature extractions on the first feature map using a first feature extraction layer to obtain a second feature map, a third feature map, and a fourth feature map, wherein the third feature map is obtained based on the second feature map, and the fourth feature map is obtained based on the third feature map;

[0145] The third feature map and the fourth feature map are connected by a concat layer to obtain a fifth feature map, and the fifth feature map is extracted by the depthwise separable convolution layer, and then the image is upsampled by an upsampling layer to obtain a sixth feature map;

[0146] Connecting the sixth feature map and the second feature map through a concat layer to obtain a seventh feature map, and performing multiple feature extractions on the seventh feature map using the second feature extraction layer to obtain a first intermediate feature map and a second intermediate feature map, wherein the second intermediate feature map is obtained based on the first intermediate feature map;

[0147] Using a concat layer to connect the second intermediate feature map and the sixth feature map to obtain an eighth feature map, and using the second feature extraction layer to perform multiple feature extractions on the eighth feature map to obtain a third intermediate feature map and a fourth intermediate feature map, wherein the fourth intermediate feature map is obtained based on the third intermediate feature map;

[0148] Using a concat layer to connect the fourth intermediate feature map and the fourth feature map to obtain a fifth intermediate feature map, and using the third feature extraction layer to extract features from the fifth intermediate feature map to obtain a target feature map;

[0149] Hand detection is performed on the target feature map, the first intermediate feature map, and the second intermediate feature map through a detect layer to obtain the hand area image.

[0150] In one embodiment, the first feature extraction layer includes four first intermediate feature extraction layers, two second intermediate feature extraction layers, an upsampling layer, and a third intermediate feature extraction layer, wherein the first intermediate feature extraction layer includes a convolutional layer and a feature extraction CSP layer, the second intermediate feature extraction layer includes a depthwise separable convolutional layer, and the third intermediate feature layer includes a conv layer and a feature extraction SPP layer;

[0151] The hand detection module 1310 performs the feature extraction on the first feature map using the first feature extraction layer for multiple times to obtain a second feature map, a third feature map, and a fourth feature map, specifically for:

[0152] sequentially performing feature extraction three times on the first feature map using three consecutive first intermediate feature extraction layers to obtain the second feature map and the third feature map, wherein the second feature map is obtained by the second first intermediate feature extraction layer, and the third feature map is obtained by the third first intermediate feature extraction layer;

[0153] Performing feature extraction on the third feature map using the second intermediate feature extraction layer to obtain a first sub-feature map;

[0154] Performing feature extraction on the first sub-feature map through the third intermediate feature extraction layer to obtain a second sub-feature map;

[0155] performing feature extraction on the second sub-feature map using the first intermediate feature extraction layer to obtain a third sub-feature map;

[0156] After using the second intermediate feature extraction layer to extract features from the third sub-feature map, an upsampling layer is used to upsample the image to obtain the fourth feature map.

[0157] In one embodiment, the second feature extraction layer includes a CSP layer and a depthwise separable convolutional layer, and the third feature extraction layer includes a CSP layer;

[0158] The hand detection module 1310 performs the multiple feature extractions on the seventh feature map using the second feature extraction layer to obtain a first intermediate feature map and a second intermediate feature map, specifically for:

[0159] Performing feature extraction on the seventh feature map using the CSP layer to obtain the first intermediate feature map, and performing feature extraction on the first intermediate feature map again using the depthwise separable convolutional layer to obtain the second intermediate feature map;

[0160] The hand detection module 1310 performs the multiple feature extractions on the eighth feature map using the second feature extraction layer to obtain a third intermediate feature map and a fourth intermediate feature map, specifically for:

[0161] Performing feature extraction on the eighth feature map using the CSP layer to obtain the third intermediate feature map, and performing feature extraction again on the third intermediate feature map using the depthwise separable convolutional layer to obtain the fourth intermediate feature map;

[0162] The hand detection module 1310 performs feature extraction on the fifth intermediate feature map using the third feature extraction layer to obtain a target feature map, specifically for:

[0163] The CSP layer is used to perform feature extraction on the fifth intermediate feature map to obtain the target feature map.

[0164] In one embodiment, the hand key point detection model includes a SliceM layer, a first feature extraction layer, a fourth feature extraction layer, a regression layer, a depth-separable convolution layer, an upsampling layer, and a concat layer;

[0165] The hand key point detection module 1320 is specifically configured to:

[0166] Using the SliceM layer, the hand region image is divided into a specified number of target hand region images, and downsampling and feature extraction are performed on each target hand region image to obtain a specified number of feature maps, and then the specified number of feature maps are spliced ​​to obtain a hand region feature map;

[0167] Continuously performing multiple feature extractions on the hand region feature map using the first feature extraction layer to obtain a first hand region feature map, a second hand region feature map, and a third hand region feature map, wherein the second hand region feature map is obtained based on the first hand region feature map, and the third hand region feature map is obtained based on the second hand region feature map;

[0168] concatenating the second hand region feature map and the third hand region feature map through the concat layer to obtain a fourth hand region feature map, performing feature extraction on the fourth hand region feature map using the depthwise separable convolution layer, and then performing image upsampling on the fourth hand region feature map after feature extraction using the upsampling layer to obtain a fifth hand region feature map;

[0169] Using the concat layer to connect the fifth hand region feature map and the first hand region feature map to obtain a sixth hand region feature map;

[0170] performing feature extraction on the sixth hand region feature map using the fourth feature extraction layer to obtain a target hand region feature map;

[0171] The regression layer is used to perform regression processing on the target hand area feature map to obtain the hand key points.

[0172] In one embodiment, the first feature extraction layer includes four first intermediate feature extraction layers, two second intermediate feature extraction layers, an upsampling layer, and a third intermediate feature extraction layer, wherein the first intermediate feature extraction layer includes a convolutional layer and a feature extraction CSP layer, the second intermediate feature extraction layer includes a depthwise separable convolutional layer, and the third intermediate feature layer includes a conv layer and a feature extraction SPP layer;

[0173] The hand key point detection module 1320 performs the feature extraction on the hand region feature map using the first feature extraction layer for multiple times to obtain a first hand region feature map, a second hand region feature map, and a third hand region feature map, specifically for:

[0174] Sequentially performing feature extraction three times on the first feature map using three consecutive first intermediate feature extraction layers to obtain a first hand region feature map and a second hand region feature map, wherein the first hand region feature map is obtained by the second first intermediate feature extraction layer, and the second hand region feature map is obtained by the third first intermediate feature extraction layer;

[0175] performing feature extraction on the second hand region feature map using the second intermediate feature extraction layer to obtain a first sub-hand region feature map;

[0176] performing feature extraction on the first sub-hand region feature map through the third intermediate feature extraction layer to obtain a second sub-hand region feature map;

[0177] performing feature extraction on the second sub-hand region feature map using the first intermediate feature extraction layer to obtain a third sub-hand region feature map;

[0178] After using the second intermediate feature extraction layer to extract features from the third sub-hand region feature map, the upsampling layer is used to upsample the image to obtain the third hand region feature map.

[0179] In one embodiment, the fourth feature extraction layer includes a feature extraction CSP layer and two CPM layers;

[0180] The hand key point detection module 1320 performs feature extraction on the sixth hand region feature map using the fourth feature extraction layer to obtain a target hand region feature map, specifically for:

[0181] performing feature extraction on the sixth hand region feature map using the CSP layer to obtain a fourth sub-hand region feature map;

[0182] The fourth sub-hand region feature map is subjected to feature extraction using the CPM layer to obtain a fifth sub-hand region feature map, and the fifth sub-hand region feature map is subjected to feature extraction using the CPM layer to obtain the target hand region feature map.

[0183] In one embodiment, the CPM layer includes two convolutional layers and three depth-wise separable convolutional layers;

[0184] The hand key point detection module 1320 performs feature extraction on the fourth sub-hand region feature map using the CPM layer to obtain a fifth sub-hand region feature map, specifically for:

[0185] Performing feature extraction on the fourth sub-hand region feature map using a convolutional layer to obtain a first intermediate hand region feature map; and

[0186] performing feature extraction three times continuously on the fourth sub-hand region feature map using three depthwise separable convolutional layers to obtain a second intermediate hand region feature map;

[0187] fusing the first intermediate hand region feature map and the second intermediate hand region feature map to obtain a third intermediate hand region feature map;

[0188] A convolutional layer is used to perform feature extraction on the third intermediate hand region feature map to obtain the fifth sub-hand region feature map.

[0189] After introducing a hand key point detection method and apparatus according to an exemplary embodiment of the present invention, a mobile terminal according to another exemplary embodiment of the present invention is introduced next.

[0190] Those skilled in the art will appreciate that various aspects of the present invention may be implemented as systems, methods, or program products. Therefore, various aspects of the present invention may be implemented in the following forms: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software implementations, which may be collectively referred to herein as "circuits," "modules," or "systems."

[0191] In some possible implementations, the mobile terminal according to the present invention may include at least one processor and at least one computer storage medium. The computer storage medium stores program code, and when the program code is executed by the processor, the processor executes the steps of the hand key point detection method according to various exemplary embodiments of the present invention described above. For example, the processor may execute the following steps: Figure 1 Steps 101-102 shown in FIG.

[0192] Refer to the following Figure 14 The mobile terminal 1400 according to this embodiment of the present invention will be described. Figure 14 The mobile terminal 1400 shown is merely an example and should not limit the functionality and scope of use of the embodiments of the present invention.

[0193] like Figure 14 As shown, mobile terminal 1400 is a general-purpose mobile terminal. Components of mobile terminal 1400 may include, but are not limited to, at least one processor 1401, at least one computer storage medium 1402, and a bus 1403 connecting various system components (including computer storage medium 1402 and processor 1401).

[0194] Bus 1403 represents one or more of several types of bus structures, including a computer storage media bus or computer storage media controller, a peripheral bus, a processor, or a local bus using any of a variety of bus architectures.

[0195] Computer storage media 1402 may include readable media in the form of volatile computer storage media, such as random access computer storage media (RAM) 1421 and / or cache storage media 1422 , and may further include read-only computer storage media (ROM) 1423 .

[0196] The computer storage medium 1402 may also include a program / utility 1425 having a set (at least one) of program modules 1424, such program modules 1424 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0197] Mobile terminal 1400 can also communicate with one or more external devices 1404 (e.g., keyboards, pointing devices, etc.), one or more devices that enable a user to interact with mobile terminal 1400, and / or any device that enables mobile terminal 1400 to communicate with one or more other mobile terminals (e.g., routers, modems, etc.). Such communication can occur via input / output (I / O) interface 1405. Furthermore, mobile terminal 1400 can communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks such as the Internet) via network adapter 1406. As shown, network adapter 1406 communicates with other modules within mobile terminal 1400 via bus 1403. It should be understood that, although not shown, other hardware and / or software modules can be used in conjunction with mobile terminal 1400, including but not limited to microcode, device drivers, redundant processors, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0198] In some possible implementations, various aspects of a hand key point detection method provided by the present invention may also be implemented in the form of a program product, which includes program code. When the program product is run on a computer device, the program code is used to enable the computer device to execute the steps of the hand key point detection method according to various exemplary embodiments of the present invention described above in this specification.

[0199] Obviously, those skilled in the art may make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if such changes and modifications fall within the scope of the claims and their equivalents, the present invention is intended to include such changes and modifications.

Claims

1. A hand key point detection method, characterized in that: Applied in a mobile terminal, the method includes: For any frame of image in the video acquired by the mobile terminal, perform hand detection on the image using a pre-trained hand detection model to obtain a hand area image; Inputting the hand area image into a pre-trained hand key point detection model to perform hand key point detection to obtain hand key points; The hand key point detection model is trained in the following manner: Acquire a first training sample, wherein the first training sample includes a plurality of hand images and hand heat maps corresponding to the plurality of hand images respectively; Training a hand heat map detection model based on the first training sample to obtain target model parameters of the hand heat map detection model; wherein the hand heat map detection model and the hand key point detection model have different structures in the last layer, and other model structures except the structure of the last layer are the same; Setting each model parameter of the hand key point detection model using the target model parameters to obtain a set hand key point detection model; Acquire a second training sample, wherein the second training sample includes a plurality of hand images and hand key points corresponding to the hand images respectively; The set hand key point detection model is trained using the second training sample to obtain a trained hand key point detection model.

2. The method according to claim 1, characterized in that The hand detection model includes a SliceM layer, a first feature extraction layer, a second feature extraction layer, a third feature extraction layer, a target detection layer, a depth-separable convolution layer, an upsampling layer, and multiple connection concat layers; The method of performing hand detection on the image using a pre-trained hand detection model to obtain a hand region image includes: Using the SliceM layer to divide the image into a specified number of target images, downsampling and feature extraction are performed on each target image to obtain a specified number of feature maps, and then the specified number of feature maps are spliced ​​to obtain a first feature map; Continuously performing multiple feature extractions on the first feature map using a first feature extraction layer to obtain a second feature map, a third feature map, and a fourth feature map, wherein the third feature map is obtained based on the second feature map, and the fourth feature map is obtained based on the third feature map; The third feature map and the fourth feature map are connected by a concat layer to obtain a fifth feature map, and the fifth feature map is extracted by the depthwise separable convolution layer, and then the image is upsampled by an upsampling layer to obtain a sixth feature map; Connecting the sixth feature map and the second feature map through a concat layer to obtain a seventh feature map, and performing multiple feature extractions on the seventh feature map using the second feature extraction layer to obtain a first intermediate feature map and a second intermediate feature map, wherein the second intermediate feature map is obtained based on the first intermediate feature map; Using a concat layer to connect the second intermediate feature map and the sixth feature map to obtain an eighth feature map, and using the second feature extraction layer to perform multiple feature extractions on the eighth feature map to obtain a third intermediate feature map and a fourth intermediate feature map, wherein the fourth intermediate feature map is obtained based on the third intermediate feature map; Using a concat layer to connect the fourth intermediate feature map and the fourth feature map to obtain a fifth intermediate feature map, and using the third feature extraction layer to extract features from the fifth intermediate feature map to obtain a target feature map; Hand detection is performed on the target feature map, the first intermediate feature map, and the second intermediate feature map through a detect layer to obtain the hand area image.

3. The method according to claim 2, characterized in that The first feature extraction layer includes four first intermediate feature extraction layers, two second intermediate feature extraction layers, an upsampling layer and a third intermediate feature extraction layer, wherein the first intermediate feature extraction layer includes a convolution conv layer and a feature extraction CSP layer, the second intermediate feature extraction layer includes a depth-separable convolution layer, and the third intermediate feature extraction layer includes a conv layer and a feature extraction SPP layer; The method of continuously performing multiple feature extractions on the first feature map using the first feature extraction layer to obtain a second feature map, a third feature map, and a fourth feature map includes: sequentially performing feature extraction three times on the first feature map using three consecutive first intermediate feature extraction layers to obtain the second feature map and the third feature map, wherein the second feature map is obtained by the second first intermediate feature extraction layer, and the third feature map is obtained by the third first intermediate feature extraction layer; Performing feature extraction on the third feature map using the second intermediate feature extraction layer to obtain a first sub-feature map; Performing feature extraction on the first sub-feature map through the third intermediate feature extraction layer to obtain a second sub-feature map; performing feature extraction on the second sub-feature map using the first intermediate feature extraction layer to obtain a third sub-feature map; After using the second intermediate feature extraction layer to extract features from the third sub-feature map, an upsampling layer is used to upsample the image to obtain the fourth feature map.

4. The method according to claim 2, characterized in that The second feature extraction layer includes a CSP layer and a depthwise separable convolutional layer, and the third feature extraction layer includes a CSP layer; The step of performing multiple feature extractions on the seventh feature map using the second feature extraction layer to obtain a first intermediate feature map and a second intermediate feature map includes: Performing feature extraction on the seventh feature map using a CSP layer to obtain a first intermediate feature map, and performing feature extraction again on the first intermediate feature map using the depthwise separable convolutional layer to obtain a second intermediate feature map; The step of performing multiple feature extractions on the eighth feature map using the second feature extraction layer to obtain a third intermediate feature map and a fourth intermediate feature map includes: Performing feature extraction on the eighth feature map using the CSP layer to obtain the third intermediate feature map, and performing feature extraction again on the third intermediate feature map using the depthwise separable convolutional layer to obtain the fourth intermediate feature map; The step of extracting features from the fifth intermediate feature map using the third feature extraction layer to obtain a target feature map includes: The CSP layer is used to perform feature extraction on the fifth intermediate feature map to obtain the target feature map.

5. The method according to claim 1, wherein The hand key point detection model includes a SliceM layer, a first feature extraction layer, a fourth feature extraction layer, a regression layer, a depth-separable convolution layer, an upsampling layer, and a concat layer; Inputting the hand region image into a pre-trained hand key point detection model to perform hand key point detection to obtain hand key points includes: Using the SliceM layer, the hand region image is divided into a specified number of target hand region images, and downsampling and feature extraction are performed on each target hand region image to obtain a specified number of feature maps, and then the specified number of feature maps are spliced ​​to obtain a hand region feature map; Continuously performing multiple feature extractions on the hand region feature map using the first feature extraction layer to obtain a first hand region feature map, a second hand region feature map, and a third hand region feature map, wherein the second hand region feature map is obtained based on the first hand region feature map, and the third hand region feature map is obtained based on the second hand region feature map; concatenating the second hand region feature map and the third hand region feature map through the concat layer to obtain a fourth hand region feature map, performing feature extraction on the fourth hand region feature map using the depthwise separable convolution layer, and then performing image upsampling on the fourth hand region feature map after feature extraction using the upsampling layer to obtain a fifth hand region feature map; Using the concat layer to connect the fifth hand region feature map and the first hand region feature map to obtain a sixth hand region feature map; performing feature extraction on the sixth hand region feature map using the fourth feature extraction layer to obtain a target hand region feature map; The regression layer is used to perform regression processing on the target hand area feature map to obtain the hand key points.

6. The method according to claim 5, characterized in that The first feature extraction layer includes four first intermediate feature extraction layers, two second intermediate feature extraction layers, an upsampling layer and a third intermediate feature extraction layer, wherein the first intermediate feature extraction layer includes a convolution conv layer and a feature extraction CSP layer, the second intermediate feature extraction layer includes a depth-separable convolution layer, and the third intermediate feature extraction layer includes a conv layer and a feature extraction SPP layer; The step of continuously performing multiple feature extractions on the hand region feature map using the first feature extraction layer to obtain a first hand region feature map, a second hand region feature map, and a third hand region feature map includes: sequentially performing feature extraction three times on the hand region feature map using three consecutive first intermediate feature extraction layers to obtain the first hand region feature map and the second hand region feature map, wherein the first hand region feature map is obtained by the second first intermediate feature extraction layer, and the second hand region feature map is obtained by the third first intermediate feature extraction layer; performing feature extraction on the second hand region feature map using the second intermediate feature extraction layer to obtain a first sub-hand region feature map; performing feature extraction on the first sub-hand region feature map through the third intermediate feature extraction layer to obtain a second sub-hand region feature map; performing feature extraction on the second sub-hand region feature map using the first intermediate feature extraction layer to obtain a third sub-hand region feature map; After using the second intermediate feature extraction layer to extract features from the third sub-hand region feature map, the upsampling layer is used to upsample the image to obtain the third hand region feature map.

7. The method according to claim 5, characterized in that The fourth feature extraction layer includes a feature extraction CSP layer and two CPM layers; The step of extracting features from the sixth hand region feature map using the fourth feature extraction layer to obtain a target hand region feature map includes: Performing feature extraction on the sixth hand region feature map using the CSP layer to obtain a fourth sub-hand region feature map; The fourth sub-hand region feature map is subjected to feature extraction using the CPM layer to obtain a fifth sub-hand region feature map, and the fifth sub-hand region feature map is subjected to feature extraction using the CPM layer to obtain the target hand region feature map.

8. The method according to claim 7, characterized in that The CPM layer includes two convolutional layers and three depth-separable convolutional layers; The step of extracting features from the fourth sub-hand region feature map using the CPM layer to obtain a fifth sub-hand region feature map includes: Performing feature extraction on the fourth sub-hand region feature map using a convolutional layer to obtain a first intermediate hand region feature map; and performing feature extraction three times continuously on the fourth sub-hand region feature map using three depthwise separable convolutional layers to obtain a second intermediate hand region feature map; fusing the first intermediate hand region feature map and the second intermediate hand region feature map to obtain a third intermediate hand region feature map; A convolutional layer is used to perform feature extraction on the third intermediate hand region feature map to obtain the fifth sub-hand region feature map.

9. A mobile terminal, characterized in that: comprising a processor and a memory, wherein the processor and the memory are connected via a bus; The memory stores a computer program, and the processor is configured to perform the following operations based on the computer program: For any frame of image in the video acquired by the mobile terminal, perform hand detection on the image using a pre-trained hand detection model to obtain a hand area image; Inputting the hand area image into a pre-trained hand key point detection model to perform hand key point detection to obtain hand key points; The hand key point detection model is trained in the following manner: Acquire a first training sample, wherein the first training sample includes a plurality of hand images and hand heat maps corresponding to the plurality of hand images respectively; Training a hand heat map detection model based on the first training sample to obtain target model parameters of the hand heat map detection model; wherein the hand heat map detection model and the hand key point detection model have different structures in the last layer, and other model structures except the structure of the last layer are the same; Setting each model parameter of the hand key point detection model using the target model parameters to obtain a set hand key point detection model; Acquire a second training sample, wherein the second training sample includes a plurality of hand images and hand key points corresponding to the hand images respectively; The set hand key point detection model is trained using the second training sample to obtain a trained hand key point detection model.

10. The mobile terminal according to claim 9, wherein: The hand detection model includes a SliceM layer, a first feature extraction layer, a second feature extraction layer, a third feature extraction layer, a target detection layer, a depth-separable convolution layer, an upsampling layer, and multiple connection concat layers; The processor performs hand detection on the image using the pre-trained hand detection model to obtain a hand area image, and is specifically configured to: Using the SliceM layer to divide the image into a specified number of target images, downsampling and feature extraction are performed on each target image to obtain a specified number of feature maps, and then the specified number of feature maps are spliced ​​to obtain a first feature map; Continuously performing multiple feature extractions on the first feature map using a first feature extraction layer to obtain a second feature map, a third feature map, and a fourth feature map, wherein the third feature map is obtained based on the second feature map, and the fourth feature map is obtained based on the third feature map; The third feature map and the fourth feature map are connected by a concat layer to obtain a fifth feature map, and the fifth feature map is extracted by the depthwise separable convolution layer, and then the image is upsampled by an upsampling layer to obtain a sixth feature map; Connecting the sixth feature map and the second feature map through a concat layer to obtain a seventh feature map, and performing multiple feature extractions on the seventh feature map using the second feature extraction layer to obtain a first intermediate feature map and a second intermediate feature map, wherein the second intermediate feature map is obtained based on the first intermediate feature map; Using a concat layer to connect the second intermediate feature map and the sixth feature map to obtain an eighth feature map, and using the second feature extraction layer to perform multiple feature extractions on the eighth feature map to obtain a third intermediate feature map and a fourth intermediate feature map, wherein the fourth intermediate feature map is obtained based on the third intermediate feature map; Using a concat layer to connect the fourth intermediate feature map and the fourth feature map to obtain a fifth intermediate feature map, and using the third feature extraction layer to extract features from the fifth intermediate feature map to obtain a target feature map; Hand detection is performed on the target feature map, the first intermediate feature map, and the second intermediate feature map through a detect layer to obtain the hand area image.

Images