Key point detection method, device, electronic device and computer readable storage medium

By extracting hand image features and generating spatial heat maps, attention maps and depth maps, combined with transposed convolution module, the accuracy of key point position detection in hand images is solved, and the accuracy of gesture recognition and model Generalization ability.

CN113298040BActive Publication Date: 2024-11-22TSINGHUA UNIVERSITY +1
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202110687209.X
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-06-21
Publication Date
2024-11-22
Estimated Expiration
2041-06-21

AI Technical Summary

Technical Problem

The prior art is difficult to accurately detect key points positions in hand images, especially in the case of multiple viewing angles and different hand shapes, resulting in a decrease in the accuracy of gesture recognition.

Method used

By extracting the image features of the hand image, a spatial heat map, a spatial attention map and a depth map are obtained, and these image features are processed using the transposed convolution module to determine the three-dimensional position information of the key points in the hand image. This method includes training a key point detection model, using pseudo-labels and visual consistency optimization model, and improving the generalization ability of the model in different data domains.

Benefits of technology

It realizes accurate detection of key points in hand images, improves the accuracy of gesture recognition and generalization ability of model, and is suitable for situations with multiple perspectives and different hand types.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN113298040B_ABST
    Figure CN113298040B_ABST
Patent Text Reader

Abstract

The present disclosure relates to a key point detection method, device, electronic device and computer-readable storage medium, and belongs to the field of deep learning technology. After extracting the image features of the hand image, the present disclosure obtains the spatial heat map, spatial attention map and depth map corresponding to the image features respectively, thereby determining the three-dimensional position information of multiple key points in the hand image based on the spatial heat map, spatial attention map and depth map, and thus determining the position of each key point in the hand image.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure relates to the field of deep learning technology, and in particular to a key point detection method, device, electronic device and computer-readable storage medium. Background Art

[0002] Key point detection is a method for determining key points in specific positions or areas of the human body, and is the basis for human body analysis and editing tasks. For example, in the process of controlling smart devices through gestures, the hand image is collected, and the positions of the key points of the hand are determined through hand key point detection to identify the gestures in the hand image, so as to control the smart device according to the identified gestures. Therefore, a key point detection method is urgently needed to determine the positions of the key points in the hand image. Summary of the invention

[0003] The present disclosure provides a key point detection method, device, electronic device and computer-readable storage medium to determine the three-dimensional position information of each key point in a hand image, thereby determining the position of each key point in the hand image. The technical solution of the present disclosure is as follows:

[0004] According to a first aspect of an embodiment of the present disclosure, a key point detection method is provided, the method comprising:

[0005] Extracting image features of the hand image;

[0006] Acquire multiple spatial heat maps, multiple spatial attention maps, and multiple depth maps corresponding to the image feature, wherein one of the spatial heat maps corresponds to one key point, the spatial heat map is used to represent the probability distribution of a pixel point in the hand image being the corresponding key point, the spatial attention map is used to represent the probability distribution of adjacent key points appearing in a corresponding area of ​​the hand image, and the depth map is used to represent an estimated value of a relative depth difference between adjacent key points when the adjacent key points appear at any position of the hand image;

[0007] Based on the multiple spatial heat maps, the multiple spatial attention maps, and the multiple depth maps, three-dimensional position information of multiple key points in the hand image is determined.

[0008] The solution provided by the embodiment of the present disclosure obtains the spatial heat map, spatial attention map and depth map corresponding to the image features after extracting the image features of the hand image, so as to determine the three-dimensional position information of multiple key points in the hand image based on the spatial heat map, spatial attention map and depth map, thereby determining the position of each key point in the hand image.

[0009] In some embodiments, obtaining multiple spatial heat maps, multiple spatial attention maps, and multiple depth maps corresponding to the image feature includes:

[0010] The image features are respectively input into a first transposed convolution module, a second transposed convolution module and a third transposed convolution module of a key point detection model, and the multiple spatial heat maps, the multiple spatial attention maps and the multiple depth maps are respectively output through the first transposed convolution module, the second transposed convolution module and the third transposed convolution module.

[0011] By processing the image features through the first transposed convolution module, the second transposed convolution module and the third transposed convolution module of the key point detection model respectively, multiple spatial heat maps, multiple spatial attention maps and multiple depth maps can be obtained.

[0012] In some embodiments, the first transposed convolution module, the second transposed convolution module and the third transposed convolution module have the same structure, and the first transposed convolution module, the second transposed convolution module and the third transposed convolution module each include at least one transposed convolution layer.

[0013] By setting at least one transposed convolution layer in the first transposed convolution module, the second transposed convolution module and the third transposed convolution module, the image features can be processed by the at least one transposed convolution layer. In addition, by setting the first transposed convolution module, the second transposed convolution module and the third transposed convolution module to the same structure, the image scales of the spatial heat map output by the first transposed convolution module, the spatial heat map output by the second device convolution module and the depth map output by the third transposed convolution module are the same, so that the spatial heat map, the spatial attention map and the depth map can be processed subsequently.

[0014] In some embodiments, the three-dimensional position information includes two-dimensional position information and depth information;

[0015] The method of determining three-dimensional position information of a plurality of key points in the hand image based on the plurality of spatial heat maps, the plurality of spatial attention maps, and the plurality of depth maps includes:

[0016] Determine the target position in each of the spatial heat maps, and use the two-dimensional position information of the target positions of the multiple spatial heat maps as the two-dimensional position information of the multiple key points, respectively. The target position is the position with the highest probability that the corresponding pixel point is the key point;

[0017] The multiple spatial attention maps are multiplied with the corresponding depth maps respectively, and the multiplication results are summed to obtain the depth information of the multiple key points.

[0018] The target position with the highest probability that the corresponding pixel point in each spatial heat map is the key point is most likely the location of the key point. Therefore, by determining the two-dimensional position information of the target position in each spatial heat map, the two-dimensional position information of each key point can be obtained. By multiplying the spatial attention map and the depth map correspondingly and then summing them, the relative depth information of each key point relative to its adjacent key points can be obtained as the depth information of each key point, thereby realizing the determination of the three-dimensional position information of each key point.

[0019] In some embodiments, the key point detection method is implemented by a key point detection model, and the method further includes:

[0020] Acquire a first sample image set and a second sample image set, wherein the first sample image set includes a plurality of first sample images and corresponding first sample labels, and the second sample image set includes a plurality of second sample images;

[0021] Perform training based on the first sample image set to obtain an intermediate key point detection model;

[0022] Processing the second sample image set by the intermediate key point detection model to obtain pseudo labels of the plurality of second sample images;

[0023] Based on the second sample image set and the pseudo labels of the plurality of second sample images, the intermediate key point detection model is trained to obtain the key point detection model.

[0024] By obtaining a first sample image set labeled with a first sample label, preliminary training is first performed on the first sample image set to obtain an intermediate key point detection model, and then the second sample image set without a label is processed by the intermediate key point detection model to obtain pseudo labels of multiple second sample images, so that the intermediate key point detection model can be further trained based on the second sample image set and the pseudo labels of the second sample images, thereby obtaining a key point detection model with higher accuracy.

[0025] In some embodiments, the second sample image set includes second sample images corresponding to a plurality of hand shapes, and the second sample image corresponding to each hand shape corresponds to a different viewing angle;

[0026] The method of training the intermediate key point detection model based on the second sample image set and the pseudo labels of the plurality of second sample images to obtain the key point detection model comprises:

[0027] For a target hand shape among the multiple hand shapes, acquiring a first target image from a second sample image corresponding to the target hand shape;

[0028] Performing image conversion on the first target image to obtain a third sample image;

[0029] Determining a predicted label of the third sample image by using the intermediate key point detection model;

[0030] Based on the pseudo label of the second target image and the predicted label, determining the pseudo label of the third sample image, the second target image being any second sample image other than the first target image among the second sample images corresponding to the target hand shape;

[0031] Based on the third sample image and the pseudo label of the third sample image, the intermediate key point detection model is trained to obtain the key point detection model.

[0032] The first target image is obtained from the second sample images corresponding to each hand shape in the second sample images of the multiple hand shapes included in the second sample image set, and then the third sample image obtained by image transformation of the first target image is processed through the intermediate key point detection model to obtain the predicted label of the third sample image, so as to determine the pseudo-label of the third sample image according to the pseudo-label of any second sample image other than the first target image in the second sample images corresponding to each hand shape, so as to continue to train the key point detection model based on the third sample image and the pseudo-label of the third sample image to further improve the accuracy of the key point detection model.

[0033] In some embodiments, determining the pseudo label of the third sample image based on the pseudo label of the second target image and the predicted label includes:

[0034] Determine the visual consistency between any second target image and the third sample image, where the visual consistency is used to represent the similarity between the two images after rotation;

[0035] If the visual consistency between any second target image and the third sample image meets the target condition, the pseudo label of any second target image is determined as the pseudo label of the third sample image.

[0036] By determining the visual consistency of any second target image and the third sample image, and then when the visual consistency meets the target condition, the label of the third sample image is updated by the predicted label determined by the model to the pseudo label of any second target image, thereby improving the accuracy of the label of the third sample image and thus improving the model training effect.

[0037] In some embodiments, determining the visual consistency between any second target image and the third sample image includes:

[0038] Taking the third sample image as a reference, rotating any second target image;

[0039] The similarity between the rotated second target image and the third sample image is determined as the visual consistency between the second target image and the third sample image.

[0040] The second target image is rotated by taking the third sample image as a reference, so as to determine the similarity between the second target image and the third sample image, thereby updating the label of the third sample image.

[0041] According to a second aspect of an embodiment of the present disclosure, a key point detection device is provided, the device comprising:

[0042] An extraction unit, configured to extract image features of a hand image;

[0043] an acquisition unit, configured to execute acquisition of a plurality of spatial heat maps, a plurality of spatial attention maps, and a plurality of depth maps corresponding to the image feature, wherein one of the spatial heat maps corresponds to a key point, the spatial heat map is used to represent the probability distribution of a pixel point in the hand image being a corresponding key point, the spatial attention map is used to represent the probability distribution of adjacent key points appearing in a corresponding area of ​​the hand image, and the depth map is used to represent an estimated value of a relative depth difference between adjacent key points when the adjacent key points appear at any position of the hand image;

[0044] A determination unit is configured to determine three-dimensional position information of multiple key points in the hand image based on the multiple spatial heat maps, the multiple spatial attention maps and the multiple depth maps.

[0045] In some embodiments, the acquisition unit is configured to execute a first transposed convolution module, a second transposed convolution module, and a third transposed convolution module that respectively input the image features into a key point detection model, and output the multiple spatial heat maps, the multiple spatial attention maps, and the multiple depth maps through the first transposed convolution module, the second transposed convolution module, and the third transposed convolution module, respectively.

[0046] In some embodiments, the first transposed convolution module, the second transposed convolution module and the third transposed convolution module have the same structure, and the first transposed convolution module, the second transposed convolution module and the third transposed convolution module each include at least one transposed convolution layer.

[0047] In some embodiments, the three-dimensional position information includes two-dimensional position information and depth information;

[0048] The determination unit is configured to determine the target position in each of the spatial heat maps, and use the two-dimensional position information of the target positions of the multiple spatial heat maps as the two-dimensional position information of the multiple key points, where the target position is the position with the highest probability that the corresponding pixel point is the key point; multiply the multiple spatial attention maps with the corresponding depth maps, sum the results of the multiplication, and obtain the depth information of the multiple key points.

[0049] In some embodiments, the key point detection method is implemented by a key point detection model;

[0050] The acquisition unit is further configured to acquire a first sample image set and a second sample image set, wherein the first sample image set includes a plurality of first sample images and corresponding first sample labels, and the second sample image set includes a plurality of second sample images;

[0051] The device also includes:

[0052] A training unit is configured to perform training based on the first sample image set to obtain an intermediate key point detection model;

[0053] A processing unit is configured to process the second sample image set by using the intermediate key point detection model to obtain pseudo labels of the plurality of second sample images;

[0054] The training unit is further configured to execute training on the intermediate key point detection model based on the second sample image set and the pseudo labels of the plurality of second sample images to obtain the key point detection model.

[0055] In some embodiments, the second sample image set includes second sample images corresponding to a plurality of hand shapes, and the second sample image corresponding to each hand shape corresponds to a different viewing angle;

[0056] The training unit includes an acquisition subunit, a processing subunit, a determination subunit and a training subunit;

[0057] The acquisition subunit is configured to acquire a first target image from a second sample image corresponding to a target hand shape among the multiple hand shapes;

[0058] The processing subunit is configured to perform image conversion on the first target image to obtain a third sample image;

[0059] The determination subunit is configured to determine a predicted label of the third sample image through the intermediate key point detection model;

[0060] The determination subunit is further configured to determine the pseudo label of the third sample image based on the pseudo label of the second target image and the predicted label, wherein the second target image is any second sample image other than the first target image in the second sample images corresponding to the target hand shape;

[0061] The training subunit is configured to perform training on the intermediate key point detection model based on the third sample image and the pseudo label of the third sample image to obtain the key point detection model.

[0062] In some embodiments, the determination subunit is configured to determine the visual consistency between any second target image and the third sample image, where the visual consistency is used to represent the similarity between the two images after rotation;

[0063] The determination subunit is further configured to determine the pseudo label of any second target image as the pseudo label of the third sample image if the visual consistency between the any second target image and the third sample image meets the target condition.

[0064] In some embodiments, the determination subunit is configured to rotate the any second target image with the third sample image as a reference; and determine the similarity between the any second target image and the third sample image after rotation as the visual consistency between the any second target image and the third sample image.

[0065] According to a third aspect of an embodiment of the present disclosure, there is provided an electronic device, including:

[0066] processor;

[0067] a memory for storing instructions executable by the processor;

[0068] The processor is configured to execute the instruction to implement the above-mentioned key point detection method.

[0069] According to a fourth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided. When instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform the above-mentioned key point detection method.

[0070] According to a fifth aspect of an embodiment of the present disclosure, a computer program product is provided, including a computer program, which implements the above-mentioned key point detection method when executed by a processor of an electronic device.

[0071] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0072] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.

[0073] Figure 1 It is a schematic diagram of an implementation environment of a key point detection method according to an exemplary embodiment.

[0074] Figure 2 The figure is a flow chart of a key point detection method according to an exemplary embodiment.

[0075] Figure 3 The figure is a flow chart of a key point detection method according to an exemplary embodiment.

[0076] Figure 4 is a schematic diagram showing the distribution of key points of a human hand according to an exemplary embodiment.

[0077] Figure 5 It is a schematic diagram of the processing process of a key point detection model according to an exemplary embodiment.

[0078] Figure 6 is a schematic diagram of a spatial attention map according to an exemplary embodiment.

[0079] Figure 7 The figure is a flowchart of a method for training a key point detection model according to an exemplary embodiment.

[0080] Figure 8 It is a schematic diagram of a training process of a key point detection model according to an exemplary embodiment.

[0081] Fig. 9 is a block diagram of a key point detection device according to an exemplary embodiment.

[0082] Fig.10 It is a block diagram of an electronic device according to an exemplary embodiment.

[0083] Fig.11 It is a block diagram of an electronic device according to an exemplary embodiment. DETAILED DESCRIPTION

[0084] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0085] Below, the technical terms and nouns involved in this disclosure are explained:

[0086] Source domain: The data domain with labeled data, which is composed of data and the labeled data corresponding to the data. The data distribution of the source domain data is different from the data distribution of the data used in the model testing process.

[0087] Target domain: a data domain without labeled data or containing partial labeled data. The target domain data consists of the data used in the model testing process.

[0088] For example, source domain data often consists of virtual data and labels automatically generated by electronic devices based on the virtual data, while target domain data consists of data in real scenarios.

[0089] In addition, the data involved in the present disclosure are data authorized by the user or fully authorized by all parties, and the images involved in the present disclosure are also images fully authorized by the user or all parties.

[0090] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings.

[0091] Figure 1 is a schematic diagram of an implementation environment of a key point detection method according to an exemplary embodiment, see Figure 1 , the implementation environment includes: a terminal 101 and a server 102.

[0092] Terminal 101 is at least one of a smart phone, a tablet computer, an e-book reader, a smart watch, a desktop computer, a laptop, an MP3 (Moving Picture Experts Group Audio Layer III, Moving Picture Experts Group Audio Layer 3) player, an MP4 (Moving Picture Experts Group Audio Layer IV, Moving Picture Experts Group Audio Layer 4) player, a virtual reality terminal, an augmented reality terminal, a wireless terminal, and a laptop computer. Terminal 101 generally refers to one of a plurality of terminals, and the embodiments of the present disclosure only take terminal 101 as an example. Those skilled in the art will appreciate that the number of the above-mentioned terminals may be more or less. For example, the above-mentioned terminals may be only a few, or the above-mentioned terminals may be dozens or hundreds, or a greater number. The embodiments of the present disclosure do not limit the number and device types of terminals 101.

[0093] The terminal 101 has a built-in or external camera component, which is not limited in the embodiments of the present disclosure. The terminal 101 obtains the hand image through the camera component, and then processes the obtained hand image to obtain the three-dimensional position information of each key point in the hand image, realizes the recognition of the positions of multiple key points in the hand image, and then realizes the recognition of gestures, so as to perform corresponding operations according to the recognized gestures. Alternatively, the terminal 101 is connected to the server 102 through wired or wireless communication, and then after obtaining the hand image through the camera component, the obtained hand image is sent to the server 102, and the server 102 processes the hand image and receives the gesture recognition result returned by the server 102, and then performs corresponding operations according to the received gesture recognition result.

[0094] Server 102 is an independent physical server, or server 102 is a server cluster or distributed file system composed of multiple physical servers, or server 102 is a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. Server 102 and terminal 101 are connected via wired or wireless communication, which is not limited in the embodiments of the present disclosure. Optionally, the number of the above servers is more or less, which is not limited in the embodiments of the present disclosure. Of course, in more possible implementations, server 102 also includes other functional servers to provide more comprehensive and diversified services.

[0095] Server 102 receives the hand image sent by terminal 101, and then processes the received hand image to obtain three-dimensional position information of each key point in the hand image, thereby recognizing the positions of multiple key points in the hand image, and then recognizing the gesture, and sending the gesture recognition result to terminal 101, so that terminal 101 performs corresponding operations based on the received gesture recognition result.

[0096] It should be noted that in Figure 1 In the implementation environment shown, the terminal 101 and the server 102 can both process the hand image to obtain the three-dimensional position information of each key point in the hand image, and the terminal 101 and the server 102 are both electronic devices. Therefore, in the subsequent description of the technical solution provided by the embodiment of the present disclosure, the execution subject is taken as an electronic device as an example for explanation. In some embodiments, the electronic device is provided as a terminal, and in other embodiments, the electronic device is provided as a server. The embodiment of the present disclosure does not limit this.

[0097] Figure 2 is a flow chart of a key point detection method according to an exemplary embodiment. Figure 2 As shown, the key point detection method is executed by an electronic device and includes the following steps.

[0098] In step S201, the electronic device extracts image features of a hand image.

[0099] In step S202, the electronic device obtains multiple spatial heat maps, multiple spatial attention maps, and multiple depth maps corresponding to the image feature, where one of the spatial heat maps corresponds to a key point, the spatial heat map is used to represent the probability distribution of pixels in the hand image being corresponding key points, the spatial attention map is used to represent the probability distribution of adjacent key points appearing in corresponding areas of the hand image, and the depth map is used to represent an estimated value of the relative depth difference between adjacent key points when the adjacent key points appear at any position in the hand image.

[0100] In step S203, the electronic device determines three-dimensional position information of multiple key points in the hand image based on the multiple spatial heat maps, the multiple spatial attention maps, and the multiple depth maps.

[0101] The solution provided by the embodiment of the present disclosure obtains the spatial heat map, spatial attention map and depth map corresponding to the image features after extracting the image features of the hand image, so as to determine the three-dimensional position information of multiple key points in the hand image based on the spatial heat map, spatial attention map and depth map, thereby determining the position of each key point in the hand image.

[0102] In some embodiments, obtaining multiple spatial heat maps, multiple spatial attention maps, and multiple depth maps corresponding to the image feature includes:

[0103] The image features are respectively input into a first transposed convolution module, a second transposed convolution module and a third transposed convolution module of a key point detection model, and the multiple spatial heat maps, the multiple spatial attention maps and the multiple depth maps are respectively output through the first transposed convolution module, the second transposed convolution module and the third transposed convolution module.

[0104] In some embodiments, the first transposed convolution module, the second transposed convolution module and the third transposed convolution module have the same structure, and the first transposed convolution module, the second transposed convolution module and the third transposed convolution module each include at least one transposed convolution layer.

[0105] In some embodiments, the three-dimensional position information includes two-dimensional position information and depth information;

[0106] The method of determining three-dimensional position information of a plurality of key points in the hand image based on the plurality of spatial heat maps, the plurality of spatial attention maps, and the plurality of depth maps includes:

[0107] Determine the target position in each of the spatial heat maps, and use the two-dimensional position information of the target positions of the multiple spatial heat maps as the two-dimensional position information of the multiple key points, respectively. The target position is the position with the highest probability that the corresponding pixel point is the key point;

[0108] The multiple spatial attention maps are multiplied with the corresponding depth maps respectively, and the multiplication results are summed to obtain the depth information of the multiple key points.

[0109] In some embodiments, the key point detection method is implemented by a key point detection model, and the method further includes:

[0110] Acquire a first sample image set and a second sample image set, wherein the first sample image set includes a plurality of first sample images and corresponding first sample labels, and the second sample image set includes a plurality of second sample images;

[0111] Perform training based on the first sample image set to obtain an intermediate key point detection model;

[0112] Processing the second sample image set by the intermediate key point detection model to obtain pseudo labels of the plurality of second sample images;

[0113] Based on the second sample image set and the pseudo labels of the plurality of second sample images, the intermediate key point detection model is trained to obtain the key point detection model.

[0114] In some embodiments, the second sample image set includes second sample images corresponding to a plurality of hand shapes, and the second sample image corresponding to each hand shape corresponds to a different viewing angle;

[0115] The method of training the intermediate key point detection model based on the second sample image set and the pseudo labels of the plurality of second sample images to obtain the key point detection model comprises:

[0116] For a target hand shape among the multiple hand shapes, acquiring a first target image from a second sample image corresponding to the target hand shape;

[0117] Performing image conversion on the first target image to obtain a third sample image;

[0118] Determining a predicted label of the third sample image by using the intermediate key point detection model;

[0119] Based on the pseudo label of the second target image and the predicted label, determining the pseudo label of the third sample image, the second target image being any second sample image other than the first target image among the second sample images corresponding to the target hand shape;

[0120] Based on the third sample image and the pseudo label of the third sample image, the intermediate key point detection model is trained to obtain the key point detection model.

[0121] In some embodiments, determining the pseudo label of the third sample image based on the pseudo label of the second target image and the predicted label includes:

[0122] Determine the visual consistency between any second target image and the third sample image, where the visual consistency is used to represent the similarity between the two images after rotation;

[0123] If the visual consistency between any second target image and the third sample image meets the target condition, the pseudo label of any second target image is determined as the pseudo label of the third sample image.

[0124] In some embodiments, determining the visual consistency between any second target image and the third sample image includes:

[0125] Taking the third sample image as a reference, rotating any second target image;

[0126] The similarity between the rotated second target image and the third sample image is determined as the visual consistency between the second target image and the third sample image.

[0127] Above Figure 2The process shown is only a basic process of the present disclosure. The following is a further description of the solution provided by the present disclosure based on a specific implementation method. Figure 3 , Figure 3 is a flow chart of a key point detection method according to an exemplary embodiment. Figure 3 As shown, the key point detection method is executed by an electronic device and includes the following steps.

[0128] In step S301, the electronic device extracts image features of a hand image.

[0129] In some embodiments, step S301 and the following steps S302 to S304 are all implemented through a key point detection model, which includes a feature extractor, a first transposed convolution module, a second transposed convolution module, and a third transposed convolution module. Among them, the first transposed convolution module, the second transposed convolution module, and the third transposed convolution module are parallel structures, that is, the output of the feature extractor will be input in parallel to the first transposed convolution module, the second transposed convolution module, and the third transposed convolution module, and then processed in parallel by the first transposed convolution module, the second transposed convolution module, and the third transposed convolution module to obtain different output results. In other embodiments, the key point detection model has other structures, which are not limited by the embodiments of the present disclosure.

[0130] It should be noted that the key point detection model is used to detect the position of each key point in the hand image, and in order to facilitate the representation of the position of each key point, the key point detection model outputs the three-dimensional position information of each key point, so that the electronic device can determine the position of each key point based on the three-dimensional position information output by the key point detection model. The three-dimensional position information of each key point consists of two-dimensional position information and depth information.

[0131] Among them, the human hand includes 21 key points. When the electronic device detects the position of each key point in the hand image through the key point detection model, it is to determine the three-dimensional position information of these 21 key points. Figure 4 , Figure 4 is a schematic diagram showing the distribution of key points of a human hand according to an exemplary embodiment. Figure 4 In the human hand shown, the positions of 21 key points numbered 0 to 20 are marked.

[0132] The following is an example in which the key point detection model includes a feature extractor, a first transposed convolution module, a second transposed convolution module and a third transposed convolution module for explanation.

[0133] In some embodiments, the electronic device inputs the hand image into a feature extractor of a key point detection model, and outputs image features of the hand image through the feature extractor.

[0134] Wherein, the feature extractor is a residual network (Residual Network, ResNet), for example, the feature extractor is a residual network (ResNet101) with 101 layers, then the feature extractor includes 100 convolution layers, 1 average pooling layer and 1 fully connected layer, then in some embodiments, the electronic device inputs the hand image into the feature extractor of the key point detection model, and convolution processing is performed on the hand image through the 100 convolution layers of the feature extractor to obtain the convolution feature of the hand image, and then the convolution feature is processed through the average pooling layer to achieve dimensionality reduction of the convolution feature, and the convolution feature after dimensionality reduction is obtained, and finally the convolution feature after dimensionality reduction is processed through the fully connected layer to obtain the image feature. Wherein, the image feature is a two-dimensional feature.

[0135] For example, see Figure 5 , Figure 5 It is a schematic diagram of the processing process of the key point detection model according to an exemplary embodiment. By inputting the hand image into the feature extractor, the image feature can be obtained.

[0136] It should be noted that the above is only an exemplary method of obtaining image features through a feature extractor. In other embodiments, other types of networks are used as feature extractors to process the hand image to obtain the image features of the hand image. For example, residual networks with different depths are used as feature extractors (such as ResNet50), or high-resolution networks (HRNet) are used as feature extractors. The embodiments of the present disclosure are not limited to this.

[0137] In step S302, the electronic device obtains multiple spatial heat maps, multiple spatial attention maps, and multiple depth maps corresponding to the image feature, where one of the spatial heat maps corresponds to a key point, the spatial heat map is used to represent the probability distribution of pixels in the hand image being corresponding key points, the spatial attention map is used to represent the probability distribution of adjacent key points appearing in corresponding areas of the hand image, and the depth map is used to represent an estimated value of the relative depth difference between adjacent key points when the adjacent key points appear at any position in the hand image.

[0138] In some embodiments, the electronic device inputs the image features into the first transposed convolution module, the second transposed convolution module and the third transposed convolution module of the key point detection model respectively, and processes the image features in parallel through the first transposed convolution module, the second transposed convolution module and the third transposed convolution module, and outputs the multiple spatial heat maps, the multiple spatial attention maps and the multiple depth maps respectively.

[0139] Among them, the first transposed convolution module, the second transposed convolution module and the third transposed convolution module have the same structure, and the first transposed convolution module, the second transposed convolution module and the third transposed convolution module each include at least one transposed convolution layer, that is, the first transposed convolution module, the second transposed convolution module and the third transposed convolution module each include the same number of device convolution layers.

[0140] By processing the image features respectively through the first transposed convolution module, the second transposed convolution module and the third transposed convolution module of the key point detection model, multiple spatial heat maps, multiple spatial attention maps and multiple depth maps can be obtained. By setting at least one transposed convolution layer in the first transposed convolution module, the second transposed convolution module and the third transposed convolution module, the image features can be processed through the at least one transposed convolution layer. In addition, the first transposed convolution module, the second transposed convolution module and the third transposed convolution module are set to the same structure, so that the spatial heat map output by the first transposed convolution module, the spatial heat map output by the second device convolution module and the depth map output by the third transposed convolution module have the same image scale, so that the spatial heat map, the spatial attention map and the depth map can be processed later.

[0141] It should be noted that although the structures of the first transposed convolution module, the second transposed convolution module and the third transposed convolution module are the same, the parameters adopted by the first transposed convolution module, the second transposed convolution module and the third transposed convolution module are different, so that the images output by the first transposed convolution module, the second transposed convolution module and the third transposed convolution module are different.

[0142] Taking the processing process of the first transposed convolution module as an example, in some embodiments, the first transposed convolution module includes three transposed convolution layers, and the electronic device inputs the image feature into the first transposed convolution layer of the first transposed convolution module, and performs transposed convolution processing on the image feature through the first transposed convolution layer to achieve upsampling of the image feature to obtain the first transposed convolution feature, and then inputs the first transposed convolution feature into the second transposed convolution layer of the first transposed convolution module, and processes the first transposed convolution feature through the second transposed convolution layer to achieve upsampling of the first transposed convolution feature to obtain the second transposed convolution feature, and then inputs the second transposed convolution feature into the third transposed convolution layer, and processes the second transposed convolution feature through the third transposed convolution layer to achieve upsampling of the second transposed convolution feature to obtain the spatial heat map.

[0143] It should be noted that the above process is only described by taking the processing process of the first transposed convolution module including three transposed convolution layers as an example. In other embodiments, the number of transposed convolution layers included in the first transposed convolution module is more or less. When the number of transposed convolution layers included in the first transposed convolution module is other values, the specific processing process is the same as the above process and will not be repeated here. In addition, the processing process of the second transposed convolution module and the third transposed convolution module is also the same as the above process and will not be repeated here.

[0144] Since one spatial heat map corresponds to one key point, and the spatial attention map and depth map are used to represent the content related to the adjacent key points, when the hand includes 21 key points, the first transposed convolution module outputs 21 spatial heat maps, the second transposed convolution module outputs 20 spatial attention maps, and the third transposed convolution module outputs 20 depth maps. The spatial heat map, spatial attention map, and depth map are all two-dimensional images.

[0145] Among them, the spatial heat map corresponding to each key point includes a Gaussian heat map of the area corresponding to each key point. The Gaussian heat map is used to represent the Gaussian heat distribution of the area where the corresponding key point is located, and the normalized result of the heat distribution of the area can represent the probability of each pixel point in the area being a key point, so that the spatial heat map can represent the probability distribution of the pixel points in the hand image being key points, so as to determine the two-dimensional position information of the key points according to the spatial heat map.

[0146] Each pair of adjacent key points corresponds to a spatial attention map, which is also a heat map. Each spatial attention map includes an elliptical Gaussian heat map, which is used to represent the Gaussian heat distribution of adjacent key points in this elliptical area. The normalized result of the heat distribution of the elliptical area can represent the probability of adjacent key points appearing in the elliptical area. Figure 6 , Figure 6 is a schematic diagram of a spatial attention map according to an exemplary embodiment. Figure 6 In the two spatial attention maps shown, the area marked by the ellipse is the area where adjacent key points may appear in the hand image.

[0147] Each pair of adjacent key points corresponds to a depth map, and the estimated value of the relative depth difference between any two positions in the depth map can be determined, so that no matter where the adjacent key points appear in the hand image, the estimated value of the relative depth difference between the two positions can be obtained to determine the relative depth difference of the adjacent key points.

[0148] Among them, adjacent key points are two key points in the hand image that are adjacent to each other. Figure 4 As an example, for the distribution of key points of human hands shown in Figure 4 Among the 21 key points marked in the human hand, the key points labeled 0 and 1, 5, 9, 13, and 17 are all adjacent key points, the key points labeled 1 and 2 are adjacent key points, and so on. In the hand image including 21 key points, these 21 key points can form 20 pairs of adjacent key points, and each pair of adjacent key points can correspond to a spatial attention map and a depth map.

[0149] In step S303, the electronic device determines the target position in each of the spatial heat maps, and uses the two-dimensional position information of the target positions of the multiple spatial heat maps as the two-dimensional position information of the multiple key points, respectively. The target position is the position where the probability that the corresponding pixel point is the key point is the highest.

[0150] It should be noted that since the spatial heat map is a two-dimensional image and the spatial heat map can represent the probability distribution of the pixel points in the hand image corresponding to the key points, the two-dimensional position information of the key points can be determined only by the spatial heat map.

[0151] In some embodiments, for any one of the multiple spatial heat maps, the electronic device normalizes the any one of the spatial heat maps to obtain the probability value corresponding to each position in the spatial heat map, and then determines the position with the highest probability that the corresponding pixel point in the spatial heat map is a key point as the target position, thereby obtaining the two-dimensional position information of the target position as the two-dimensional position information of the key point corresponding to the any one of the spatial heat maps.

[0152] The two-dimensional position information is the horizontal position information and the vertical position information, that is, the horizontal and vertical coordinates.

[0153] It should be noted that the above process is only explained by taking the process of processing a spatial heat map to obtain the two-dimensional position information of a key point as an example. The process of determining the two-dimensional position information of other key points is the same as the above process and will not be repeated here.

[0154] In step S304, the electronic device multiplies the multiple spatial attention maps with the corresponding depth maps respectively, sums the multiplication results, and obtains the depth information of the multiple key points.

[0155] It should be noted that, since the spatial attention map can represent the probability distribution of adjacent key points appearing in the corresponding area of ​​the hand image, the target position with the highest probability of the corresponding pixel point being the key point in each spatial heat map is most likely the position where the key point is located. Therefore, by determining the two-dimensional position information of the target position in each spatial heat map, the two-dimensional position information of each key point can be obtained, and the depth map can represent the estimated value of the relative depth difference between any two positions in the hand image. Then, by multiplying the spatial attention map and the depth map and then summing them, the relative depth difference between the two positions with the highest probability of the corresponding pixel point being the key point in the corresponding area can be obtained, which can be used as the relative depth difference of adjacent key points, thereby obtaining the relative depth information of each key point relative to its adjacent key points as the depth information of each key point, and then realizing the determination of the three-dimensional position information of each key point.

[0156] In some embodiments, for any one of the multiple spatial attention maps, the electronic device multiplies the feature values ​​of each pixel point in the corresponding depth map by the any one of the spatial attention maps, and then sums the multiplication results to obtain the relative depth difference of the adjacent key points corresponding to the any one of the spatial attention maps, so as to take one of the adjacent key points as a reference point to obtain the relative depth information of the other key point, thereby obtaining the depth information of two of the adjacent key points.

[0157] Among them, the feature values ​​of each pixel in any spatial attention map and the corresponding depth map are multiplied correspondingly, that is, the dot product of any spatial attention map and the corresponding depth map is performed. Figure 5 ,The electronic device can obtain the depth information by performing dot product of the spatial attention map and the depth map, and then,spatial summation.

[0158] It should be noted that the above process is only explained by taking the process of processing a spatial heat map to obtain the two-dimensional position information of a key point as an example. The process of determining the two-dimensional position information of other key points is the same as the above process and will not be repeated here.

[0159] Since when the depth information of two adjacent key points is determined based on the relative depth difference of the adjacent key points, one key point is used as a reference point (for example, the depth information of the key point used as the reference point is 0 by default), so as to determine the relative depth information of another key point relative to the reference point, and the reference points used by different adjacent key points may be different. Therefore, in some embodiments, after determining the depth information of each pair of adjacent key points, any one key point is selected from the 21 key points as a reference point, and the relative depth information of the remaining 20 key points relative to the reference point is determined, so as to obtain the depth information of the 21 key points, so that the determined depth information is relative to the same reference point, which is convenient for subsequent processing and reduces the processing pressure of subsequent electronic devices.

[0160] The solution provided by the embodiment of the present disclosure, after extracting the image features of the hand image, respectively obtains the spatial heat map, spatial attention map and depth map corresponding to the image features, thereby determining the two-dimensional position information and depth information of multiple key points in the hand image based on the spatial heat map, spatial attention map and depth map, and the three-dimensional position information of multiple key points in the hand image can be obtained, so that the position of each key point in the hand image can be determined. In addition, the solution provided by the embodiment of the present disclosure, by introducing the spatial attention map in the process of determining the depth information of the key points, makes it possible for the second transposed convolution module for outputting the spatial attention map to only complete the task of determining the probability distribution, that is, the task of distinguishing the key points, and the third transposed convolution module for outputting the depth map to only complete the task of estimating the relative depth difference, that is, the task of depth prediction, thereby realizing the decoupling of the depth prediction task and the key point distinction task, reducing the task difficulty of each module, thereby reducing the nonlinearity of the key point detection model, improving the generalization ability of the key point detection model between different data domains, and facilitating model migration.

[0161] Above Figure 3The process shown is only a process of processing the hand image through the key point detection model to obtain the three-dimensional position information of multiple key points. In some embodiments, the key point detection model is obtained by pre-training. The training process of the key point detection model is shown in Figure 7 , Figure 7 is a flowchart of a method for training a key point detection model according to an exemplary embodiment. Figure 7 As shown, the training method of the key point detection model is executed by an electronic device and includes the following steps.

[0162] In step S701, the electronic device obtains a first sample image set and a second sample image set, wherein the first sample image set includes multiple first sample images and corresponding first sample labels, and the second sample image set includes second sample images corresponding to multiple hand shapes, and the second sample image corresponding to each hand shape corresponds to a different viewing angle.

[0163] Among them, the first sample image set is an image set of the source domain, each first sample image in the first sample image set is source domain data, and each first sample image is a hand image, each first sample image has been marked with a first sample label, and the first sample label is automatically generated by an electronic device, and the first sample label is used to indicate the three-dimensional position information of each key point in the first sample image. For example, the first sample image set is a rendered hand dataset (RenderedHand Dataset, RHD), or the first sample image set is a FreiHand dataset. Optionally, the first sample image set uses other types of hand image sets, such as a monocular total capture (Monocular Total Capture, MTC) dataset, a stereo hand pose tracking benchmark (Stereo Hand Pose Tracking Benchmark, STB) dataset, etc., and the embodiments of the present disclosure are not limited to this.

[0164] The second sample image set is an image set of the target domain, and each second sample image in the second sample image set is not target domain data, and each second sample image is a hand image, and each second sample image is not annotated with a label indicating the three-dimensional position information of the key point. The second sample image set is a three-dimensional hand (Hand3D, H3D) data set. Optionally, the second sample image set adopts other types of hand image sets, such as MTC data sets, STB data sets, etc., and the embodiments of the present disclosure do not limit this. Regardless of the type of data set adopted by the second sample image set, it is necessary to ensure that the second sample image set includes second sample images corresponding to multiple hand shapes. For example, if the H3D data set is used as the second sample image set, the second sample image set includes second sample images corresponding to multiple hand shapes, and there are 15 second sample images corresponding to the same hand shape, that is, each hand shape corresponds to 15 second sample images of different perspectives, while in other data sets, each hand shape corresponds to more or less image perspectives, and the embodiments of the present disclosure do not limit this.

[0165] In step S702, the electronic device performs training based on the first sample image set to obtain an intermediate key point detection model.

[0166] In some embodiments, the electronic device trains the initial key point detection model based on the first sample image to obtain the intermediate key point detection model. The training process of the intermediate key point detection model includes: in the first iteration process, the electronic device inputs the first first sample image into the initial key point detection model, processes the first first sample image through the initial key point detection model, obtains the three-dimensional position information of each key point in the first first sample image, and then adjusts the model parameters in the initial key point detection model based on the first sample label corresponding to the first first sample image and the three-dimensional position information of each key point in the first first sample image obtained by the model processing, and uses the adjusted model parameters as the model parameters for the next iteration, and continues to train the model until the training conditions are met, and the model that meets the training conditions is used as the intermediate key point detection model. The training condition is that the accuracy of the three-dimensional position information output by the model meets the iteration cutoff condition, or the loss function value of the model meets the iteration cutoff condition, or the number of iterations reaches a preset number. The embodiment of the present disclosure does not limit which specific condition is used.

[0167] Among them, the model parameters in the initial key point detection model are adjusted, including adjusting the parameters of each transposed convolution module. When adjusting the parameters of different transposed convolution modules, the same loss function is used, or different loss functions are used, which is not limited in the embodiments of the present disclosure. The loss function corresponding to the first transposed convolution module is recorded as the heat map loss, the loss function corresponding to the second transposed convolution module is recorded as the attention map loss, and the loss function corresponding to the third transposed convolution module is recorded as the depth loss. In some embodiments, the heat map loss, the attention map loss and the depth loss all use the mean squared error (MSE) loss function, or the heat map loss, the attention map loss and the depth loss can all use other types of loss functions, which is not limited in the embodiments of the present disclosure.

[0168] Optionally, after obtaining the intermediate key point detection model through training with the first sample image set, the electronic device continues to train the intermediate key point detection model using a self-supervised learning method to obtain a key point detection model with better accuracy.

[0169] In some embodiments, the electronic device uses the intermediate key point detection model as a teacher model, and then migrates the parameters of the intermediate key point detection model to a model with the same structure as the intermediate key point detection model, thereby obtaining a new intermediate key point detection model as a student model, so as to train the intermediate key point detection model through the student model and the teacher model using a self-supervised learning method.

[0170] In step S703, the electronic device processes the second sample image set by using the intermediate key point detection model to obtain pseudo labels for the plurality of second sample images.

[0171] In some embodiments, after the electronic device obtains an intermediate key point detection model through training of the first sample image set, it processes multiple second sample images in the second sample image set through the intermediate key point detection model to obtain three-dimensional position information of multiple key points in each second sample image as pseudo labels for the multiple second sample images.

[0172] If the training process of the key point detection model adopts a training method based on self-supervised learning, the electronic device uses the intermediate key point detection model as a student model to process the second sample image to obtain pseudo labels of multiple second sample images.

[0173] In step S704, for a target hand shape among the multiple hand shapes, the electronic device obtains a first target image from the second sample images corresponding to the target hand shape.

[0174] In some embodiments, the second sample image set is still taken as an H3D data set as an example for explanation. For the multiple second sample images included in the second sample image set, the multiple second sample images correspond to multiple hand shapes, and each hand shape corresponds to 15 second sample images. The electronic device randomly extracts a second sample image from the second sample images corresponding to each hand shape as the first target image.

[0175] It should be noted that since the second sample image set includes multiple hand shapes corresponding to the second sample images, and when obtaining the first target image, a first target image is obtained from the second sample image corresponding to each hand shape, the number of first target images obtained is multiple.

[0176] In step S705, the electronic device performs image conversion on the first target image to obtain a third sample image.

[0177] In some embodiments, the electronic device performs data augmentation processing on the first target image to achieve image transformation of the first target image, thereby obtaining a third sample image.

[0178] Among them, the data augmentation processing includes horizontal flipping processing, vertical flipping processing, rotation processing, scaling processing, cropping processing, translation processing, adding noise, etc. The embodiment of the present disclosure does not limit which specific operation or operations are used to perform image transformation on the first target image.

[0179] It should be noted that since there are multiple first target images, when performing image transformation on the first target images, image transformation is performed on each first target image to obtain corresponding third sample images respectively, so that the number of third sample images is also multiple.

[0180] In step S706, the electronic device determines a predicted label of the third sample image through the intermediate key point detection model.

[0181] In some embodiments, after acquiring the third sample image, the electronic device inputs the third sample image into an intermediate key point detection model, processes the third sample image through the intermediate key point detection model, and obtains the three-dimensional position information of each key point in the third sample image as a predicted label for the third sample image.

[0182] If the training process of the key point detection model adopts a training method based on self-supervised learning, the electronic device uses the intermediate key point detection model as a student model to process the third sample image to obtain predicted labels of multiple third sample images.

[0183] In step S707, the electronic device determines the pseudo label of the third sample image based on the pseudo label of the second target image and the predicted label, where the second target image is any second sample image other than the first target image in the second sample images corresponding to the target hand shape.

[0184] In some embodiments, the electronic device determines the visual consistency between any second target image and the third sample image, and the visual consistency is used to represent the similarity between the two images after rotation; if the visual consistency between any second target image and the third sample image meets the target condition, the pseudo label of any second target image is determined as the pseudo label of the third sample image.

[0185] By determining the visual consistency of any second target image and the third sample image, and then when the visual consistency meets the target condition, the label of the third sample image is updated by the predicted label determined by the model to the pseudo label of any second target image, thereby improving the accuracy of the label of the third sample image and thus improving the model training effect.

[0186] Still taking the second sample image set as the H3D data set as an example, if the first target image is denoted as x_0, the second target images corresponding to the first target image are denoted as x_1, x_2, x_3, ..., x_14 respectively, and the third sample image corresponding to the first target image is denoted as A(x), then when the electronic device determines the visual consistency between any second target image and the third sample image, it determines an image with the highest visual consistency with the hand shapes included in the other images from the multiple images A(x), x_1, x_2, x_3, ..., x_14, and then uses the pseudo label of the determined image as the pseudo label of the third sample image.

[0187] The visual consistency between the two images is the similarity between the skeletons of the hands in the two images after rotating the other image at a certain angle with one of the two images as a reference. The similarity is determined based on the distance between the skeletons of the hands in the two images.

[0188] In some embodiments, when determining the visual consistency between any second target image and the third sample image, the electronic device rotates the any second target image with reference to the third sample image; and determines the similarity between the rotated any second target image and the third sample image as the visual consistency between the any second target image and the third sample image.

[0189] The second target image is rotated by taking the third sample image as a reference, so as to determine the similarity between the second target image and the third sample image, thereby updating the label of the third sample image.

[0190] The similarity between the rotated second target image and the third sample image is determined, that is, the distance between the skeleton in the rotated second sample image and the skeleton in the third sample image is determined, so as to obtain the visual consistency between the second target image and the third sample image. In some embodiments, the following formula (1) is used to determine the visual consistency between image X and image Y:

[0191]

[0192] Among them, r(X,Y) represents the visual consistency between image X and image Y, R represents the rotation matrix, and F represents the squared distance between image X and image Y.

[0193] For humans, visual consistency actually refers to the consistency of the relative positions of key points in different skeletons, such as the same angle or length ratio of two fingers, rather than the consistency of the absolute positions of key points in different skeletons. Therefore, when determining the visual consistency between two images, relative coordinates are used to represent the positions of each key point in the image, and when determining the relative coordinates of any key point, the key point adjacent to the key point can be used as a reference point to determine the relative coordinates of the key point. Moreover, experiments have shown that this method of using the key point adjacent to the key point as a reference point to determine the relative coordinates of the key point determines the visual consistency with higher accuracy, thereby making the pseudo-label of the third sample image more accurate.

[0194] Using relative coordinates to determine visual consistency can make the model output consistent results to humans, and it can also evenly distribute consistency constraints to different key points, making the determined visual consistency more accurate.

[0195] It should be noted that, when determining the pseudo-label of each third sample image, the pseudo-label corresponding to the image with the highest visual consistency with other images is used as the pseudo-label of the third sample image. Therefore, as the training progresses, r(X,Y) is monotonically decreasing, so that the accuracy of the pseudo-label of the third sample image is increasingly higher, thereby improving the model training effect. In addition, since the pseudo-label of the third sample image is determined based on the pseudo-label of the second target image, and the second target image has not undergone image transformation, and the second target image is an image in a real scene, the pseudo-label of the third sample image determined can represent a more realistic hand structure, thereby avoiding the problem of model collapse during model training.

[0196] By performing image transformation on the first target image and then performing model training based on the third sample image obtained after the image transformation, the generalization ability of the model can be improved. Without performing image transformation on the second target image, the accuracy of the determined pseudo-label can be improved, thereby improving the training effect of the model.

[0197] In step S708, the electronic device trains the intermediate key point detection model based on the third sample image and the pseudo label of the third sample image to obtain the key point detection model.

[0198] In some embodiments, the training process of the key point detection model includes: in the first iteration process, the first third sample image of the electronic device is input into the intermediate key point detection model, the first third sample image is processed by the intermediate key point detection model to obtain the three-dimensional position information of each key point in the first third sample image, and then based on the pseudo label of the first third sample image and the three-dimensional position information of each key point in the first third sample image obtained by model processing, the model parameters in the intermediate key point detection model are adjusted, and the adjusted model parameters are used as the model parameters in the next iteration, and the model training is continued until the training conditions are met, and the model that meets the training conditions is used as the key point detection model. Among them, the training conditions are the same as the training conditions in the above step S702, and will not be repeated here.

[0199] If the training process of the key point detection model adopts a training method based on self-supervised learning, when the electronic device trains the model based on the third sample image and the pseudo-label of the third sample image, the trained model is an intermediate key point detection model serving as a student model, thereby obtaining a key point detection model, and then migrating the parameters of the key point detection model to the intermediate key point detection model serving as a teacher model to obtain the key point detection model used in actual use.

[0200] In the above steps S704 to S708, the first target image is obtained from the second sample image corresponding to each hand shape in the second sample images of the multiple hand shapes included in the second sample image set, and then the third sample image obtained by image transformation of the first target image is processed through the intermediate key point detection model to obtain the predicted label of the third sample image, so as to determine the pseudo-label of the third sample image according to the pseudo-label of any second sample image other than the first target image in the second sample image corresponding to each hand shape, so as to continue to train the key point detection model based on the third sample image and the pseudo-label of the third sample image to further improve the accuracy of the key point detection model.

[0201] The process shown in the above steps S701 to S708 can be found in Figure 8 , Figure 8 is a schematic diagram of a training process of a key point detection model according to an exemplary embodiment. Figure 8 As shown, the electronic device uses the intermediate key point detection model obtained by training the first sample image set as the teacher model f, and then processes each second sample image x_i in the second sample image set through the teacher model to obtain a predicted label f(x_i) corresponding to each second sample image as a pseudo label of each second sample image, and processes the third sample image A(x) obtained by image transformation through the intermediate key point detection model as the student model g to obtain a predicted label g(A(x)) corresponding to the third sample image, and then determines the pseudo label of the third sample image based on g(A(x)) and f(x_i), thereby obtaining the second sample image, the third sample image and the corresponding pseudo label h(x) for model training, wherein the self-supervised loss L is used during model training. MSE As the loss function, the specific training process refers to the above steps S701 to S708, which will not be repeated here.

[0202] The applicant used the existing key point detection model and the key point detection model provided by the embodiment of the present disclosure to conduct experiments based on source domain data and target domain data. The experimental results are shown in Table 1 below:

[0203] Table 1

[0204]

[0205] It can be determined from the above Table 1 that, through the key point detection model provided by the embodiment of the present disclosure, for the key point detection model trained by the ordinary training method, the depth error on the target domain is reduced by 6%, and for the key point detection model trained by the self-supervised learning method, the depth error is further reduced by 16.7%. It can be seen that the solution provided by the embodiment of the present disclosure can effectively improve the accuracy of the key point detection model.

[0206] It should be noted that in the above experimental process, Figure 4 The key points 9 marked in the human hand are used as reference points to determine the depth error.

[0207] The solution provided by the embodiment of the present disclosure obtains a first sample image set labeled with a first sample label, the first sample image set is a source domain data set, and preliminary training is first performed through the first sample image set to obtain an intermediate key point detection model, that is, the source domain data set is used to train the intermediate key point detection model, and then the second sample image set without labeled labels is processed through the intermediate key point detection model to obtain pseudo labels of multiple second sample images, and the second sample image set without labeled labels is also a target domain data set. Based on the second sample image set and the pseudo labels of the second sample images, the intermediate key point detection model is further trained, that is, the model training is continued based on the target domain data set to further improve the accuracy of the model, thereby obtaining a key point detection model with higher accuracy. In addition, a self-supervision mechanism can be introduced in the training process to make full use of the information of multiple views in the data set, thereby improving the accuracy of the model in the target domain.

[0208] Fig. 9 is a block diagram of a key point detection device according to an exemplary embodiment. Fig. 9 , the device comprises:

[0209] An extraction unit 901 is configured to extract image features of a hand image;

[0210] An acquisition unit 902 is configured to execute acquisition of a plurality of spatial heat maps, a plurality of spatial attention maps, and a plurality of depth maps corresponding to the image feature, wherein one of the spatial heat maps corresponds to one key point, the spatial heat map is used to represent the probability distribution of a pixel point in the hand image being a corresponding key point, the spatial attention map is used to represent the probability distribution of adjacent key points appearing in a corresponding area of ​​the hand image, and the depth map is used to represent an estimated value of a relative depth difference between adjacent key points when the adjacent key points appear at any position of the hand image;

[0211] The determination unit 903 is configured to determine the three-dimensional position information of multiple key points in the hand image based on the multiple spatial heat maps, the multiple spatial attention maps and the multiple depth maps.

[0212] The device provided by the embodiment of the present disclosure obtains the spatial heat map, spatial attention map and depth map corresponding to the image features after extracting the image features of the hand image, so as to determine the three-dimensional position information of multiple key points in the hand image based on the spatial heat map, spatial attention map and depth map, thereby determining the position of each key point in the hand image.

[0213] In some embodiments, the acquisition unit 902 is configured to execute a first transposed convolution module, a second transposed convolution module, and a third transposed convolution module that respectively input the image features into a key point detection model, and output the multiple spatial heat maps, the multiple spatial attention maps, and the multiple depth maps through the first transposed convolution module, the second transposed convolution module, and the third transposed convolution module, respectively.

[0214] In some embodiments, the first transposed convolution module, the second transposed convolution module and the third transposed convolution module have the same structure, and the first transposed convolution module, the second transposed convolution module and the third transposed convolution module each include at least one transposed convolution layer.

[0215] In some embodiments, the three-dimensional position information includes two-dimensional position information and depth information;

[0216] The determination unit 903 is configured to determine the target position in each of the spatial heat maps, and use the two-dimensional position information of the target positions of the multiple spatial heat maps as the two-dimensional position information of the multiple key points, where the target position is the position with the highest probability that the corresponding pixel point is the key point; multiply the multiple spatial attention maps with the corresponding depth maps, sum the multiplication results, and obtain the depth information of the multiple key points.

[0217] In some embodiments, the key point detection method is implemented by a key point detection model;

[0218] The acquisition unit 902 is further configured to acquire a first sample image set and a second sample image set, wherein the first sample image set includes a plurality of first sample images and corresponding first sample labels, and the second sample image set includes a plurality of second sample images;

[0219] The device also includes:

[0220] A training unit is configured to perform training based on the first sample image set to obtain an intermediate key point detection model;

[0221] A processing unit is configured to process the second sample image set by using the intermediate key point detection model to obtain pseudo labels of the plurality of second sample images;

[0222] The training unit is further configured to execute training on the intermediate key point detection model based on the second sample image set and the pseudo labels of the plurality of second sample images to obtain the key point detection model.

[0223] In some embodiments, the second sample image set includes second sample images corresponding to a plurality of hand shapes, and the second sample image corresponding to each hand shape corresponds to a different viewing angle;

[0224] The training unit includes an acquisition subunit, a processing subunit, a determination subunit and a training subunit;

[0225] The acquisition subunit is configured to acquire a first target image from a second sample image corresponding to a target hand shape among the multiple hand shapes;

[0226] The processing subunit is configured to perform image conversion on the first target image to obtain a third sample image;

[0227] The determination subunit is configured to determine a predicted label of the third sample image through the intermediate key point detection model;

[0228] The determination subunit is further configured to determine the pseudo label of the third sample image based on the pseudo label of the second target image and the predicted label, wherein the second target image is any second sample image other than the first target image in the second sample images corresponding to the target hand shape;

[0229] The training subunit is configured to perform training on the intermediate key point detection model based on the third sample image and the pseudo label of the third sample image to obtain the key point detection model.

[0230] In some embodiments, the determination subunit is configured to determine the visual consistency between any second target image and the third sample image, where the visual consistency is used to represent the similarity between the two images after rotation;

[0231] The determination subunit is further configured to determine the pseudo label of any second target image as the pseudo label of the third sample image if the visual consistency between the any second target image and the third sample image meets the target condition.

[0232] In some embodiments, the determination subunit is configured to rotate the any second target image with the third sample image as a reference; and determine the similarity between the any second target image and the third sample image after rotation as the visual consistency between the any second target image and the third sample image.

[0233] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0234] In an exemplary embodiment, an electronic device is further provided, and optionally, the electronic device is provided as a terminal, or the electronic device is provided as a server, which is not limited in the embodiment of the present disclosure. The structures of the electronic device provided as a terminal and a server are introduced below.

[0235] Fig.10 1 is a block diagram of an electronic device according to an exemplary embodiment. The electronic device is provided as a terminal, denoted as terminal 1000, and the terminal 1000 may be: a smart phone, a tablet computer, an MP3 player (Moving Picture Experts Group Audio Layer III, Moving Picture Experts Group Audio Layer 3), an MP4 (Moving Picture Experts Group Audio Layer IV, Moving Picture Experts Group Audio Layer 4) player, a laptop computer or a desktop computer. The terminal 1000 may also be called a user device, a portable terminal, a laptop terminal, a desktop terminal, or other names.

[0236] Typically, the terminal 1000 includes: one or more processors 1001 and one or more memories 1002 .

[0237] The processor 1001 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 1001 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). The processor 1001 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 1001 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 1001 may also include an AI (Artificial Intelligence) processor, which is used to process computing operations related to machine learning.

[0238] The memory 1002 may include one or more computer-readable storage media, which may be non-transitory. The memory 1002 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 1002 is used to store at least one program code, which is used to be executed by the processor 1001 to implement the key point detection method provided in the method embodiment of the present disclosure.

[0239] In some embodiments, the terminal 1000 may further optionally include: a peripheral device interface 1003 and at least one peripheral device. The processor 1001, the memory 1002 and the peripheral device interface 1003 may be connected via a bus or a signal line. Each peripheral device may be connected to the peripheral device interface 1003 via a bus, a signal line or a circuit board. Specifically, the peripheral device includes: at least one of a radio frequency circuit 1004, a display screen 1005, a camera 1006, an audio circuit 1007, a positioning component 1008 and a power supply 1009.

[0240] The peripheral device interface 1003 may be used to connect at least one peripheral device related to I / O (Input / Output) to the processor 1001 and the memory 1002. In some embodiments, the processor 1001, the memory 1002, and the peripheral device interface 1003 are integrated on the same chip or circuit board; in some other embodiments, any one or two of the processor 1001, the memory 1002, and the peripheral device interface 1003 may be implemented on a separate chip or circuit board, which is not limited in this embodiment.

[0241] The radio frequency circuit 1004 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The radio frequency circuit 1004 communicates with the communication network and other communication devices through electromagnetic signals. The radio frequency circuit 1004 converts electrical signals into electromagnetic signals for transmission, or converts received electromagnetic signals into electrical signals. In some embodiments, the radio frequency circuit 1004 includes: an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a user identity module card, and the like. The radio frequency circuit 1004 can communicate with other terminals through at least one wireless communication protocol. The wireless communication protocol includes, but is not limited to: a metropolitan area network, various generations of mobile communication networks (2G, 3G, 4G and 5G), a wireless local area network and / or a WiFi (Wireless Fidelity) network. In some embodiments, the radio frequency circuit 1004 may also include circuits related to NFC (Near Field Communication), which is not limited in the present disclosure.

[0242] The display screen 1005 is used to display a UI (User Interface). The UI may include graphics, text, icons, videos, and any combination thereof. When the display screen 1005 is a touch display screen, the display screen 1005 also has the ability to collect touch signals on the surface or above the surface of the display screen 1005. The touch signal can be input to the processor 1001 as a control signal for processing. At this time, the display screen 1005 can also be used to provide virtual buttons and / or virtual keyboards, also known as soft buttons and / or soft keyboards. In some embodiments, the display screen 1005 can be one, and the front panel of the terminal 1000 is set; in other embodiments, the display screen 1005 can be at least two, which are respectively set on different surfaces of the terminal 1000 or are folded; in some other embodiments, the display screen 1005 can be a flexible display screen, which is set on the curved surface or folded surface of the terminal 1000. Even, the display screen 1005 can also be set to a non-rectangular irregular shape, that is, a special-shaped screen. The display screen 1005 can be made of materials such as LCD (Liquid Crystal Display) and OLED (Organic Light-Emitting Diode).

[0243] The camera assembly 1006 is used to capture images or videos. In some embodiments, the camera assembly 1006 includes a front camera and a rear camera. Typically, the front camera is disposed on the front panel of the terminal, and the rear camera is disposed on the back of the terminal. In some embodiments, there are at least two rear cameras, which are any one of a main camera, a depth of field camera, a wide-angle camera, and a telephoto camera, so as to realize the fusion of the main camera and the depth of field camera to realize the background blur function, the fusion of the main camera and the wide-angle camera to realize the panoramic shooting and VR (Virtual Reality) shooting function or other fusion shooting functions. In some embodiments, the camera assembly 1006 may also include a flash. The flash may be a monochrome temperature flash or a dual-color temperature flash. A dual-color temperature flash refers to a combination of a warm light flash and a cold light flash, which can be used for light compensation at different color temperatures.

[0244] The audio circuit 1007 may include a microphone and a speaker. The microphone is used to collect sound waves from the user and the environment, and convert the sound waves into electrical signals and input them into the processor 1001 for processing, or input them into the radio frequency circuit 1004 to achieve voice communication. For the purpose of stereo acquisition or noise reduction, there may be multiple microphones, which are respectively arranged at different parts of the terminal 1000. The microphone may also be an array microphone or an omnidirectional acquisition microphone. The speaker is used to convert the electrical signal from the processor 1001 or the radio frequency circuit 1004 into sound waves. The speaker may be a traditional film speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, it can not only convert the electrical signal into sound waves audible to humans, but also convert the electrical signal into sound waves inaudible to humans for purposes such as ranging. In some embodiments, the audio circuit 1007 may also include a headphone jack.

[0245] The positioning component 1008 is used to locate the current geographical location of the terminal 1000 to implement navigation or LBS (Location Based Service). The positioning component 1008 can be a positioning component based on the US GPS (Global Positioning System), China's Beidou system, Russia's Grenas system or the European Union's Galileo system.

[0246] The power supply 1009 is used to power various components in the terminal 1000. The power supply 1009 can be an alternating current, a direct current, a disposable battery, or a rechargeable battery. When the power supply 1009 includes a rechargeable battery, the rechargeable battery can support wired charging or wireless charging. The rechargeable battery can also be used to support fast charging technology.

[0247] In some embodiments, the terminal 1000 further includes one or more sensors 1010 , including but not limited to: an acceleration sensor 1011 , a gyroscope sensor 1012 , a pressure sensor 1013 , a fingerprint sensor 1014 , an optical sensor 1015 , and a proximity sensor 1016 .

[0248] The acceleration sensor 1011 can detect the magnitude of acceleration on the three coordinate axes of the coordinate system established by the terminal 1000. For example, the acceleration sensor 1011 can be used to detect the components of gravity acceleration on the three coordinate axes. The processor 1001 can control the display screen 1005 to display the user interface in a horizontal view or a vertical view according to the gravity acceleration signal collected by the acceleration sensor 1011. The acceleration sensor 1011 can also be used to collect motion data of games or users.

[0249] The gyro sensor 1012 can detect the body direction and rotation angle of the terminal 1000, and the gyro sensor 1012 can cooperate with the acceleration sensor 1011 to collect the user's 3D actions on the terminal 1000. The processor 1001 can implement the following functions based on the data collected by the gyro sensor 1012: motion sensing (such as changing the UI according to the user's tilt operation), image stabilization during shooting, game control, and inertial navigation.

[0250] The pressure sensor 1013 can be set on the side frame of the terminal 1000 and / or the lower layer of the display screen 1005. When the pressure sensor 1013 is set on the side frame of the terminal 1000, it can detect the user's holding signal of the terminal 1000, and the processor 1001 performs left and right hand recognition or shortcut operation according to the holding signal collected by the pressure sensor 1013. When the pressure sensor 1013 is set on the lower layer of the display screen 1005, the processor 1001 controls the operability controls on the UI interface according to the user's pressure operation on the display screen 1005. The operability controls include at least one of a button control, a scroll bar control, an icon control, and a menu control.

[0251] The fingerprint sensor 1014 is used to collect the user's fingerprint, and the processor 1001 identifies the user's identity based on the fingerprint collected by the fingerprint sensor 1014, or the fingerprint sensor 1014 identifies the user's identity based on the collected fingerprint. When the user's identity is identified as a trusted identity, the processor 1001 authorizes the user to perform relevant sensitive operations, which include unlocking the screen, viewing encrypted information, downloading software, paying, and changing settings. The fingerprint sensor 1014 can be set on the front, back, or side of the terminal 1000. When a physical button or a manufacturer logo is set on the terminal 1000, the fingerprint sensor 1014 can be integrated with the physical button or the manufacturer logo.

[0252] The optical sensor 1015 is used to collect the ambient light intensity. In one embodiment, the processor 1001 can control the display brightness of the display screen 1005 according to the ambient light intensity collected by the optical sensor 1015. Specifically, when the ambient light intensity is high, the display brightness of the display screen 1005 is increased; when the ambient light intensity is low, the display brightness of the display screen 1005 is reduced. In another embodiment, the processor 1001 can also dynamically adjust the shooting parameters of the camera assembly 1006 according to the ambient light intensity collected by the optical sensor 1015.

[0253] The proximity sensor 1016, also called a distance sensor, is usually arranged on the front panel of the terminal 1000. The proximity sensor 1016 is used to collect the distance between the user and the front of the terminal 1000. In one embodiment, when the proximity sensor 1016 detects that the distance between the user and the front of the terminal 1000 is gradually decreasing, the processor 1001 controls the display screen 1005 to switch from the screen-on state to the screen-off state; when the proximity sensor 1016 detects that the distance between the user and the front of the terminal 1000 is gradually increasing, the processor 1001 controls the display screen 1005 to switch from the screen-off state to the screen-on state.

[0254] Those skilled in the art will understand that Fig.10 The structure shown in the figure does not constitute a limitation on the terminal 1000, and the terminal 1000 may include more or less components than those shown in the figure, or combine some components, or adopt a different component arrangement.

[0255] Fig.11 It is a block diagram of an electronic device according to an exemplary embodiment, the electronic device is provided as a server, denoted as server 1100, the server 1100 may have relatively large differences due to different configurations or performances, and may include one or more processors (Central Processing Units, CPU) 1101 and one or more memories 1102, wherein the one or more memories 1102 store at least one program code, and the at least one program code is loaded and executed by the one or more processors 1101 to implement the key point detection method provided by the above-mentioned various method embodiments. Of course, the server 1100 may also have components such as a wired or wireless network interface, a keyboard, and an input and output interface for input and output, and the server 1100 may also include other components for implementing device functions, which will not be repeated here.

[0256] In an exemplary embodiment, a computer-readable storage medium including instructions is also provided, such as a memory 1002 including instructions or a memory 1102 including instructions, and the above instructions can be executed by the processor 1001 of the terminal 1000 to complete the above key point detection method, or the above instructions can be executed by the processor 1101 of the server 1100 to complete the above key point detection method. Optionally, the computer-readable storage medium can be a read-only memory (ROM), a random access memory (RAM), a compact disc read-only memory (CD-ROM), a magnetic tape, a floppy disk, an optical data storage device, etc.

[0257] In an exemplary embodiment, a computer program product is also provided, including a computer program, which can be executed by the processor 1001 of the terminal 1000 to complete the method steps of the key point detection method provided in the above embodiment; or, the computer program can be executed by the processor 1101 of the server 1100 to complete the method steps of the key point detection method provided in the above embodiment.

[0258] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0259] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A key point detection method, characterized in that: The method comprises: Extracting image features of the hand image; Acquire multiple spatial heat maps, multiple spatial attention maps, and multiple depth maps corresponding to the image features, wherein one of the spatial heat maps corresponds to one key point, the spatial heat map is used to represent the probability distribution of a pixel point in the hand image being a corresponding key point, the spatial attention map is used to represent the probability distribution of adjacent key points appearing in a corresponding area of ​​the hand image, and the depth map is used to represent an estimated value of a relative depth difference between adjacent key points when the adjacent key points appear at any position of the hand image; Determine a target position in each of the spatial heat maps, and use the two-dimensional position information of the target positions of the multiple spatial heat maps as the two-dimensional position information of multiple key points in the hand image, wherein the target position is the position where the probability that the corresponding pixel point is the key point is the highest; The multiple spatial attention maps are multiplied with the corresponding depth maps respectively, and the multiplication results are summed to obtain the depth information of multiple key points in the hand image.

2. The key point detection method according to claim 1, characterized in that: The obtaining of multiple spatial heat maps, multiple spatial attention maps, and multiple depth maps corresponding to the image features includes: The image features are respectively input into a first transposed convolution module, a second transposed convolution module and a third transposed convolution module of a key point detection model, and the multiple spatial heat maps, the multiple spatial attention maps and the multiple depth maps are respectively output through the first transposed convolution module, the second transposed convolution module and the third transposed convolution module.

3. The key point detection method according to claim 2, characterized in that: The first transposed convolution module, the second transposed convolution module and the third transposed convolution module have the same structure, and the first transposed convolution module, the second transposed convolution module and the third transposed convolution module each include at least one transposed convolution layer.

4. The key point detection method according to claim 1, characterized in that: The key point detection method is implemented by a key point detection model, and the method further includes: Acquire a first sample image set and a second sample image set, wherein the first sample image set includes a plurality of first sample images and corresponding first sample labels, and the second sample image set includes a plurality of second sample images; Performing training based on the first sample image set to obtain an intermediate key point detection model; Processing the second sample image set by the intermediate key point detection model to obtain pseudo labels of the plurality of second sample images; The intermediate key point detection model is trained based on the second sample image set and the pseudo labels of the plurality of second sample images to obtain the key point detection model.

5. The key point detection method according to claim 4, characterized in that: The second sample image set includes second sample images corresponding to a plurality of hand shapes, and the second sample image corresponding to each hand shape corresponds to a different viewing angle; The step of training the intermediate key point detection model based on the second sample image set and the pseudo labels of the plurality of second sample images to obtain the key point detection model comprises: For a target hand shape among the multiple hand shapes, acquiring a first target image from a second sample image corresponding to the target hand shape; Performing image conversion on the first target image to obtain a third sample image; Determining a predicted label of the third sample image by using the intermediate key point detection model; Determine the pseudo label of the third sample image based on the pseudo label of the second target image and the predicted label, wherein the second target image is any second sample image other than the first target image among the second sample images corresponding to the target hand shape; The intermediate key point detection model is trained based on the third sample image and the pseudo label of the third sample image to obtain the key point detection model.

6. The key point detection method according to claim 5, characterized in that: The step of determining the pseudo label of the third sample image based on the pseudo label of the second target image and the predicted label includes: Determine visual consistency between any second target image and a third sample image, wherein the visual consistency is used to represent the similarity between the two images after rotation; If the visual consistency between any second target image and the third sample image meets the target condition, the pseudo label of any second target image is determined as the pseudo label of the third sample image.

7. The key point detection method according to claim 6, characterized in that: The determining of the visual consistency between any second target image and the third sample image comprises: Taking the third sample image as a reference, rotating any second target image; The similarity between the rotated second target image and the third sample image is determined as the visual consistency between the second target image and the third sample image.

8. A key point detection device, characterized in that: The device comprises: An extraction unit, configured to extract image features of a hand image; an acquisition unit, configured to execute acquisition of a plurality of spatial heat maps, a plurality of spatial attention maps, and a plurality of depth maps corresponding to the image features, wherein one of the spatial heat maps corresponds to one key point, the spatial heat map is used to represent the probability distribution of a pixel point in the hand image being a corresponding key point, the spatial attention map is used to represent the probability distribution of adjacent key points appearing in a corresponding area of ​​the hand image, and the depth map is used to represent an estimated value of a relative depth difference between adjacent key points when the adjacent key points appear at any position of the hand image; A determination unit is configured to determine the target position in each of the spatial heat maps, and use the two-dimensional position information of the target positions of the multiple spatial heat maps as the two-dimensional position information of multiple key points in the hand image, wherein the target position is the position where the probability of the corresponding pixel point being the key point is the highest; and multiply the multiple spatial attention maps with the corresponding depth maps, and sum the multiplied results to obtain the depth information of the multiple key points in the hand image.

9. The key point detection device according to claim 8, characterized in that: The acquisition unit is configured to execute a first transposed convolution module, a second transposed convolution module, and a third transposed convolution module that respectively input the image features into a key point detection model, and output the multiple spatial heat maps, the multiple spatial attention maps, and the multiple depth maps through the first transposed convolution module, the second transposed convolution module, and the third transposed convolution module.

10. The key point detection device according to claim 9, characterized in that: The first transposed convolution module, the second transposed convolution module and the third transposed convolution module have the same structure, and the first transposed convolution module, the second transposed convolution module and the third transposed convolution module each include at least one transposed convolution layer.

11. The key point detection device according to claim 8, characterized in that: The key point detection method is implemented by a key point detection model; The acquisition unit is further configured to acquire a first sample image set and a second sample image set, wherein the first sample image set includes a plurality of first sample images and corresponding first sample labels, and the second sample image set includes a plurality of second sample images; The device also includes: A training unit, configured to perform training based on the first sample image set to obtain an intermediate key point detection model; a processing unit configured to process the second sample image set by using the intermediate key point detection model to obtain pseudo labels of the plurality of second sample images; The training unit is further configured to perform training on the intermediate key point detection model based on the second sample image set and the pseudo labels of the plurality of second sample images to obtain the key point detection model.

12. The key point detection device according to claim 11, characterized in that: The second sample image set includes second sample images corresponding to a plurality of hand shapes, and the second sample image corresponding to each hand shape corresponds to a different viewing angle; The training unit includes an acquisition subunit, a processing subunit, a determination subunit and a training subunit; The acquisition subunit is configured to acquire, for a target hand shape among the multiple hand shapes, a first target image from a second sample image corresponding to the target hand shape; The processing subunit is configured to perform image conversion on the first target image to obtain a third sample image; The determination subunit is configured to determine a predicted label of the third sample image through the intermediate key point detection model; The determination subunit is further configured to determine the pseudo label of the third sample image based on the pseudo label of the second target image and the predicted label, wherein the second target image is any second sample image other than the first target image in the second sample images corresponding to the target hand shape; The training subunit is configured to perform training on the intermediate key point detection model based on the third sample image and the pseudo label of the third sample image to obtain the key point detection model.

13. The key point detection device according to claim 12, characterized in that: The determination subunit is configured to determine the visual consistency between any second target image and the third sample image, wherein the visual consistency is used to represent the similarity between the two images after rotation; The determining subunit is further configured to determine the pseudo label of any second target image as the pseudo label of the third sample image if the visual consistency between any second target image and the third sample image meets the target condition.

14. The key point detection device according to claim 13, characterized in that: The determining subunit is configured to rotate any second target image with reference to the third sample image; The similarity between the rotated second target image and the third sample image is determined as the visual consistency between the second target image and the third sample image.

15. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the key point detection method as described in any one of claims 1 to 7.

16. A computer-readable storage medium, characterized in that: When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform the key point detection method as claimed in any one of claims 1 to 7.

17. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor of an electronic device, the key point detection method according to any one of claims 1 to 7 is implemented.