3D gesture detection method and electronic equipment

Abstract

The invention relates to the technical field of gesture interaction, and provides a 3D gesture detection method and electronic equipment, which are used for improving the accuracy and robustness of 3D gesture detection. In model training, a synthetic gesture sample subset is added to supplement complex hand gestures under the condition of lack of self-shielding of a real gesture sample subset, and a key point field adaptation technology is designed to calculate a domain difference loss value of a real gesture sample and the synthetic gesture sample. Therefore, the problem of domain difference between a real gesture sample and a synthetic gesture sample is relieved, and generalization of a model to a complex hand posture with serious self-shielding is improved through the synthetic gesture sample; meanwhile, a background sample subset without a hand is introduced, a negative sample regularization constraint item is constructed according to the background sample subset, and a negative sample distribution loss value and a positive and negative sample distribution difference loss value are calculated, so that the distribution difference between the background information and the hand key point is enlarged, the interference of the background information on the hand key point is reduced, and the user experience is improved. And the 3D gesture detection accuracy is improved.

Description

Technical Field

[0001] The present application relates to the field of gesture interaction technology and provides a 3D gesture detection method and an electronic device. Background Art

[0002] With the continuous attention and investment in the concept of the metaverse at home and abroad, the development of the AR / VR industry has gradually become more popular. At present, improving user experience is one of the key purposes of AR / VR devices. 3D gesture interaction technology is a technology that does not rely on external devices to achieve human-computer interaction. It is more user-friendly. Therefore, 3D gesture interaction has become a research hotspot for AR / VR devices.

[0003] During the interaction process, the stability of 3D gestures is the main factor affecting the gesture interaction experience of AR / VR devices. During the interaction process, some key points of the hand may be invisible from a certain perspective due to the occlusion of the hand itself, which will seriously affect the recognition of 3D gestures. Therefore, solving the complex hand self-occlusion problem is a difficult problem that needs to be overcome in 3D gesture interaction technology.

[0004] At present, related technologies usually use multi-camera devices to collect gesture images and manually annotate to generate 3D gesture datasets. However, for complex gestures with severe self-occlusion, annotation is difficult and the output ratio of effective data is not high. Therefore, when performing gesture detection based on the 3D gesture dataset generated in this way, the accuracy of gesture detection will be affected, resulting in erroneous interactions or interaction failures, which will reduce the user experience. Summary of the invention

[0005] The embodiments of the present application provide a 3D gesture detection method and an electronic device, which are used to improve the accuracy of 3D gesture detection.

[0006] On the one hand, an embodiment of the present application provides a 3D gesture detection method, including:

[0007] Acquire a 3D gesture sample set; the 3D gesture sample set includes a background sample subset, a real gesture sample subset and a synthetic gesture sample subset, the real gesture samples and the synthetic gesture samples are used as positive samples, the background samples are used as negative samples, the label of each negative sample obeys a discrete uniform distribution, and the label of each positive sample obeys a Gaussian normal distribution;

[0008] According to the 3D gesture sample set, the 3D gesture detection model to be trained is iterated for at least one time to obtain a target gesture detection model, wherein each iteration process performs the following operations:

[0009] Performing feature extraction on multiple samples in the 3D gesture sample set to obtain a heat map set of corresponding samples;

[0010] Determine, according to respective heat maps of the real gesture sample and the synthetic gesture sample in the plurality of samples, a domain difference loss value between the real gesture sample and the synthetic gesture sample;

[0011] According to the heat maps of the positive samples and the negative samples in the multiple samples and the labels of the corresponding samples, respectively determine the positive sample distribution loss value, the negative sample distribution loss value, and the positive and negative sample distribution difference loss value;

[0012] A target loss value for 3D gesture detection is determined according to the domain difference loss value, the positive sample distribution loss value, the negative sample distribution loss value, and the positive and negative sample distribution difference loss value, and model parameters are adjusted according to the target loss value.

[0013] On the other hand, an embodiment of the present application provides an electronic device, including a processor, a memory, and a communication interface, wherein the communication interface, the memory, and the processor are connected via a bus;

[0014] The communication interface is used to communicate with the terminal device and the server;

[0015] The memory stores a computer program, and the processor performs the following operations according to the computer program:

[0016] Acquire a 3D gesture sample set; the 3D gesture sample set includes a background sample subset, a real gesture sample subset and a synthetic gesture sample subset, the real gesture samples and the synthetic gesture samples are used as positive samples, the background samples are used as negative samples, the label of each negative sample obeys a discrete uniform distribution, and the label of each positive sample obeys a Gaussian normal distribution;

[0017] According to the 3D gesture sample set, the 3D gesture detection model to be trained is iterated for at least one time to obtain a target gesture detection model, wherein each iteration process performs the following operations:

[0018] Performing feature extraction on multiple samples in the 3D gesture sample set to obtain a heat map set of corresponding samples;

[0019] Determine, according to respective heat maps of the real gesture sample and the synthetic gesture sample in the plurality of samples, a domain difference loss value between the real gesture sample and the synthetic gesture sample;

[0020] According to the heat maps of the positive samples and the negative samples in the multiple samples and the labels of the corresponding samples, respectively determine the positive sample distribution loss value, the negative sample distribution loss value, and the positive and negative sample distribution difference loss value;

[0021] A target loss value for 3D gesture detection is determined according to the domain difference loss value, the positive sample distribution loss value, the negative sample distribution loss value, and the positive and negative sample distribution difference loss value, and model parameters are adjusted according to the target loss value.

[0022] Optionally, the processor determines the domain difference loss value between the real gesture sample and the synthetic gesture sample according to the respective heat map sets of the real gesture sample and the synthetic gesture sample in the multiple samples, and the specific operation is:

[0023] For each positive sample in each real gesture sample and each synthetic gesture sample, project each heat map in the heat map set of the positive sample onto the U axis and V axis of the image coordinate system respectively to obtain a U coordinate classification vector and a V coordinate classification vector;

[0024] Determine the inter-class similarity and intra-class similarity of each positive sample according to the U coordinate classification vector and the V coordinate classification vector corresponding to each positive sample;

[0025] A domain difference loss value between the real gesture sample and the synthetic gesture sample is determined according to the intra-class similarity and the inter-class similarity.

[0026] Optionally, the intra-class similarity includes U-coordinate intra-class similarity and V-coordinate intra-class similarity, and the inter-class similarity includes U-coordinate inter-class similarity and V-coordinate inter-class similarity;

[0027] The processor determines the inter-class similarity and intra-class similarity of each positive sample according to the U coordinate classification vector and the V coordinate classification vector corresponding to each positive sample, and the specific operation is:

[0028] Calculating a first U coordinate intra-class similarity according to the U coordinate classification vectors corresponding to the respective real gesture samples, and calculating a first V coordinate intra-class similarity according to the V coordinate classification vectors corresponding to the respective real gesture samples;

[0029] Calculating a second U-coordinate intra-class similarity according to the U-coordinate classification vector of each of the synthesized gesture samples, and calculating a second V-coordinate intra-class similarity according to the V-coordinate classification vector of each of the synthesized gesture samples;

[0030] The U coordinate inter-class similarity is calculated based on the U coordinate classification vectors of the real gesture samples and the synthetic gesture samples, and the V coordinate intra-class similarity is calculated based on the V coordinate classification vectors of the real gesture samples and the synthetic gesture samples.

[0031] Optionally, the intra-class similarity includes U coordinate intra-class similarity and V coordinate intra-class similarity, the inter-class similarity includes U coordinate inter-class similarity and V coordinate inter-class similarity, and the calculation formula of the domain difference loss value is:

[0032]

[0033] Among them, Dist DA (·) represents the key point domain difference function between the real gesture sample and the synthetic gesture sample, N s Represents the number of real gesture samples, N T represents the number of synthetic gesture samples, x s represents the i-th true gesture sample, x t represents the jth synthetic gesture sample, represents the similarity within the U coordinate class, represents the similarity within the V coordinate class, represents the U coordinate inter-class similarity between the real gesture sample and the synthetic gesture sample, Represents the inter-class similarity of the V coordinates between the real gesture sample and the synthetic gesture sample.

[0034] Optionally, the processor determines the positive sample distribution loss value, the negative sample distribution loss value, and the positive and negative sample distribution difference loss value respectively according to the heat map sets of the positive samples and the negative samples in the multiple samples and the labels of the corresponding samples, and the specific operation is:

[0035] For each sample among the multiple samples, subtract the element value in each heat map of the sample from the element value at the corresponding position in the label of the sample to obtain the element difference at the corresponding position;

[0036] Determining the negative sample distribution loss value according to the element difference corresponding to each heat map of the negative samples in the multiple samples;

[0037] Determining the positive sample distribution loss value according to the element difference values ​​corresponding to each heat map of the positive sample in the multiple samples;

[0038] The KL divergence is adopted to determine the positive and negative sample distribution difference loss value according to the element value of each heat map of the multiple samples and the element value of the corresponding position in the label of the corresponding sample.

[0039] Optionally, the formula for the negative sample distribution loss value is expressed as:

[0040]

[0041] Wherein, x represents the element difference, and smooth1(·) represents a smoothing function.

[0042] Optionally, the formula for the positive and negative sample distribution difference loss value is expressed as:

[0043]

[0044] Among them, H s+t is the heat map corresponding to each key point contained in the positive sample in the multiple samples, H r Represents the heat map corresponding to each key point contained in the negative samples in multiple samples, h p (x) represents an element value in the heat map, h q (x) represents the element value at the corresponding position in the label, and KL(·) represents the KL divergence function.

[0045] Optionally, after obtaining the target gesture detection model, the processor further executes:

[0046] Acquire a gesture image during the gesture interaction process, and input the gesture image into the target gesture detection model;

[0047] Extract key points in the gesture image according to the target gesture detection model to obtain a 3D gesture;

[0048] According to the 3D gesture, the display of the virtual image is controlled.

[0049] On the other hand, an embodiment of the present application provides a head-mounted display, in which a target gesture detection model is deployed, the head-mounted display includes a processor, a memory, a display and a camera, and the camera, the display, the memory and the processor are connected via a bus;

[0050] The camera is used to obtain gesture images during gesture interaction;

[0051] The memory stores a computer program, and the processor performs the following operations according to the computer program:

[0052] Inputting the gesture image acquired by the camera into the target gesture detection model;

[0053] Extract key points in the gesture image according to the target gesture detection model to obtain a 3D gesture;

[0054] According to the 3D gesture, a virtual image displayed on the display is controlled.

[0055] On the other hand, an embodiment of the present application provides a computer-readable storage medium, which stores computer-executable instructions, and the computer-executable instructions are used to enable a computer device to execute the steps of a 3D gesture detection method provided in an embodiment of the present application.

[0056] A 3D gesture detection method and electronic device provided in an embodiment of the present application involve a target gesture detection model for gesture detection. The 3D gesture sample set used by the model, in addition to positive samples of hands, also includes negative samples containing only background but not hands (i.e., background samples). According to different ways of generating positive samples, the 3D gesture sample set is divided into a real gesture sample subset and a synthetic gesture sample subset. Synthetic gesture samples are used to enrich complex gestures that are difficult to collect in actual scenes, and background samples are used to alleviate the interference of the background in the environment on gesture detection, thereby improving the accuracy and robustness of complex hand gesture recognition. Different from conventional sample labels, the label of each negative sample in the 3D gesture sample set obeys a discrete uniform distribution, and the label of each positive sample obeys a Gaussian normal distribution, thereby well describing the distribution characteristics of the key points of the hand, which helps to improve the accuracy of gesture detection. On the other hand, during the model training process, the domain difference between real gesture samples and synthetic gesture samples is taken into account. Through domain adaptation technology, the domain difference loss value between the two is calculated according to the heat maps of real gesture samples and the heat maps of synthetic gesture samples, so as to achieve the generalization problem of complex hand postures with severe self-occlusion through synthetic gesture samples; at the same time, the regularization constraint term is constructed by introducing negative samples, and the positive sample distribution loss value, negative sample distribution loss value and positive and negative sample distribution difference loss value are calculated according to the heat maps of positive and negative samples and the labels of the corresponding samples, so that the model can fully learn the differences between the background and the key points of the hand, thereby improving the robustness and stability of the model.

[0057] Other features and advantages of the present application will be described in the following description, and partly become apparent from the description, or understood by practicing the present application. The purpose and other advantages of the present application can be realized and obtained by the structures specifically pointed out in the written description, claims, and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0059] Figure 1 Provide a schematic diagram of hand key point detection jitter for an embodiment of the present application;

[0060] Figure 2 The overall architecture diagram of the training target gesture detection model provided in the embodiment of the present application;

[0061] Figure 3 A flow chart of a method for training a target gesture detection model provided in an embodiment of the present application;

[0062] Figure 4 A schematic diagram of a 3D gesture sample set provided in an embodiment of the present application;

[0063] Figure 5A A heat map schematic diagram corresponding to a positive sample provided in an embodiment of the present application;

[0064] Figure 5B A heat map schematic diagram corresponding to a negative sample provided in an embodiment of the present application;

[0065] Figure 6 A flow chart of a method for calculating domain difference loss value based on key point domain adaptation technology provided in an embodiment of the present application;

[0066] Figure 7 A schematic diagram of the heat map projection results provided in the embodiment of the present application;

[0067] Fig. 8A A schematic diagram of an autocorrelation matrix provided in an embodiment of the present application;

[0068] Figure 8B Another schematic diagram of an autocorrelation matrix provided in an embodiment of the present application;

[0069] Fig. 9 A flow chart of a method for constructing a negative sample regularization constraint item for model training provided in an embodiment of the present application;

[0070] Fig.10 A 3D gesture effect diagram detected by the target gesture detection model provided by an embodiment of the present application;

[0071] Fig.11 A structural diagram of an electronic device provided in an embodiment of the present application;

[0072] Fig.12 A structural diagram of a head-mounted display provided in an embodiment of the present application. DETAILED DESCRIPTION

[0073] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the technical solution of the present application, rather than all of the embodiments. Based on the embodiments recorded in the application documents, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the technical solution of the present application.

[0074] As one of the core technologies of 3D gesture interaction technology, hand key point detection is the key to the interactive experience. In particular, hand key point detection has poor generalization in complex hand gestures with severe self-occlusion, resulting in abnormal jitter of hand key points, such as Figure 1 As shown, it will affect the success rate of interaction and the actual experience effect.

[0075] In order to improve the generalization of hand key point detection in complex gesture data with severe self-occlusion, the most direct method is to add relevant complex gesture data when training the gesture detection model. However, if multiple camera devices are used to collect complex gesture data, there will be a problem of difficulty in labeling, and the gesture accuracy is difficult to guarantee.

[0076] In view of this, the embodiment of the present application provides a 3D gesture detection method from the two aspects of domain adaptation and negative sample regularization to solve the jitter problem of complex hand gestures in severe self-occlusion and improve the accuracy and stability of gesture detection. The method enriches the complex gestures that are difficult to collect in actual scenes by introducing synthetic gesture samples, and adopts domain adaptation technology to reduce the distribution distance between real gesture samples and synthetic gesture samples, thereby achieving the generalization problem of complex hand gestures with severe self-occlusion by synthetic gesture samples. At the same time, by introducing negative samples that only contain background but not hands, the interference of background information on hand key points is alleviated, so that the model can fully learn the difference between background and hand key points, improve the robustness and stability of the algorithm, and use negative samples as regularization constraints to effectively amplify the distribution difference between background and hand key points, further improving the accuracy and robustness of gesture detection.

[0077] See also Figure 2, which is the overall architecture diagram of the training target gesture detection model provided in the embodiment of the present application, including five parts: preparing a training sample set, building a gesture detection model to be trained, key point feature extraction and model training. Among them, the sample set preparation part is used to generate a 3D gesture sample set for model training. The 3D gesture sample set includes a subset of real gesture samples collected conventionally and a subset of synthetic gesture samples of complex hand postures under severe self-occlusion. Both real gesture samples and synthetic gesture samples contain hand key points and can be used as positive samples for model training. At the same time, the 3D gesture sample set also includes a subset of background samples that do not contain hands, and the background samples do not contain hand key points and can be used as negative samples for model training. The network used in the part of building the gesture detection model to be trained only requires the feature output head to be able to output a heat map, and there is no restrictive requirement for the network structure. For example, Resnet, Hourglass and HRNet with simple network structures can be used as the backbone network to meet the needs of real-time interaction. The feature extraction part is used to extract features from the 3D gesture sample set and obtain the heat map set of each sample, where the number of heat maps corresponding to each sample can be set according to the number of key points of the hand. The model training part uses key point field adaptation technology and negative sample regularization constraint technology. According to the heat map set of each sample and the label of the corresponding sample, the target loss value is calculated, and the model parameters are adjusted according to the target loss value until the model converges, and the converged model is output as the target gesture detection model.

[0078] The training process of the above target gesture detection model is as follows Figure 3 The above mainly includes the following steps:

[0079] S301: Acquire a 3D gesture sample set.

[0080] In one example, a conventional camera is used to collect hand gestures to obtain a subset of real gesture samples, such as Figure 4 Considering that there may be complex hand gestures with self-occlusion during the interaction process, it is difficult to collect and annotate them through multi-camera equipment. Therefore, the embodiment of the present application dynamically synthesizes some high-precision complex hand gesture data based on the 3D hand model to obtain a synthetic gesture sample subset to supplement the complex hand gestures missing in the real gesture sample subset, such as Figure 4 As shown in (b) in . Since the real gesture samples in the real gesture sample subset and the synthetic gesture samples in the synthetic gesture sample subset both contain hand key points, they can be used as positive samples for model training.

[0081] For complex hand gestures with severe self-occlusion, there will be blurred edges under monochrome camera imaging, that is, it is impossible to identify whether the hand is included, which will affect the stability of hand key point detection. In addition, if only the subset of real gesture samples and synthetic gesture samples containing hand key points are used, when an image without hand key points is input during the interaction process, the model will still mistakenly identify the hand key points, which will also affect the accuracy and robustness of hand key point detection.

[0082] In order to alleviate the interference of background information on hand key points, in one example, a background sample subset that does not contain hands is generated, such as Figure 4 As shown in (c) in the figure, since the background sample does not contain hand key points, it can be used as a negative sample for model training.

[0083] Different from the labeling method of setting the positive sample label to 1 and the negative sample label to 0 in the conventional classification model and detection model, the labeling method of the sample labels in the embodiment of the present application fully considers the heat map characteristics of the hand key points extracted by the features, and sets the label of each positive sample to obey the Gaussian normal distribution, and the label of each negative sample to obey the discrete uniform distribution. This is because there are no hand key points in the negative samples. In theory, there will be no points with a relatively concentrated distribution in the heat map of the negative samples. Therefore, the labels of the negative samples are set to obey the discrete uniform distribution to indicate that the element values ​​at all positions in the heat map are equal and very low, where the element value is the confidence that the point at the position predicted by the model is a hand key point. There are hand key points in the positive samples, and there will be points with a relatively concentrated distribution in the heat map of the positive samples. Therefore, the labels of the positive samples are set to obey the Gaussian normal distribution to indicate that the element values ​​at different positions in the heat map are unequal.

[0084] In an embodiment of the present application, the labels of the samples may be represented by a matrix, and the size of the matrix is ​​the same as the size of the heat map.

[0085] Taking negative samples as an example, the discrete uniform distribution formula obeyed by the negative sample labels is expressed as:

[0086]

[0087] Assuming that the size of the heat map is k*k (k is an integer greater than 1), the matrix of the negative sample label is represented as:

[0088]

[0089] After obtaining the real gesture sample subset, the synthetic gesture sample subset and the background sample subset, these subsets are combined in a certain ratio to obtain a 3D gesture sample set for model training.

[0090] Optionally, the ratio of the real gesture samples, the synthetic gesture samples and the background samples in the 3D gesture sample set may be 1:1:1. In an example, the number of each sample may be adjusted according to actual conditions.

[0091] It should be noted that the adjustment of the ratio of real gesture samples, synthetic gesture samples and background samples at the same level has little impact on the model training results and can generally be ignored.

[0092] S302: Perform at least one iterative training on the 3D gesture detection model to be trained according to the 3D gesture sample set to obtain a target gesture detection model.

[0093] When building a 3D gesture detection model to be trained, some model parameters are usually set, such as the number of iterations, learning rate, batch data size, optimizer used and other hyperparameters.

[0094] For example, set the data size of each training batch to 64 and the initial value of the learning rate to 3e -4 , the number of iterations is 100, and the optimizer used is the Adma optimizer.

[0095] According to the acquired 3D gesture sample set, the 3D gesture detection model to be trained is iteratively trained for at least one time, and the model that reaches the iteration number or convergence (that is, the target loss value remains unchanged) is used as the final target gesture detection model.

[0096] Each training process includes the following steps:

[0097] S3021: Perform feature extraction on multiple samples in the 3D gesture sample set to obtain a heat map set of corresponding samples.

[0098] In an embodiment of the present application, the number of heat maps in the heat map set corresponding to each sample is the same and equal to the number of hand key points. For example, when the model can extract 21 hand key points, 21 heat maps will be extracted for each sample to obtain a heat map set.

[0099] like Figure 5A As shown in , it is a schematic diagram of a heat map corresponding to a positive sample. The points in the heat map are distributed relatively concentratedly, and the positions of these concentrated points are usually the positions of the key points of the hand. Figure 5B As shown, it is a schematic diagram of a heat map corresponding to a negative sample. The distribution of points in the heat map is relatively scattered, indicating that it does not contain hand key points.

[0100] S3022: Determine a domain difference loss value between the real gesture sample and the synthetic gesture sample according to the respective heat map sets of the real gesture sample and the synthetic gesture sample in the multiple samples.

[0101] The synthetic gesture sample subset generated based on the 3D hand model has the problem of hand self-occlusion in the synthetic gesture samples, which will cause the occlusion of the hand key points, and there is a domain gap (DomainGap) between the hand key points in the real gesture samples. In order to solve the domain gap problem of key points in the real gesture samples and the synthetic gesture samples, the embodiment of the present application designs a key point domain adaptation technology, which reduces the distribution distance of key points in the real gesture samples and the synthetic gesture samples by calculating the domain gap loss value between the real gesture samples and the synthetic gesture samples.

[0102] The domain difference loss value calculation process based on key point domain adaptation technology is as follows Figure 6 As shown, it mainly includes the following steps:

[0103] S3022_1: For each positive sample in each real gesture sample and each synthetic gesture sample, project each heat map in the heat map set of the positive sample onto the U axis and V axis of the image coordinate system respectively to obtain a U coordinate classification vector and a V coordinate classification vector.

[0104] The key point domain adaptation technology abstracts the heat map information regression task into a U and V coordinate classification task. In the specific implementation, for each positive sample in each real gesture sample and each synthetic gesture sample, take a heat map in the heat map set of the positive sample as an example, assuming that the size of the heat map is k*k, project the heat map onto the U axis and V axis of the image coordinate system respectively, and obtain a 1*k-dimensional U coordinate classification vector and a 1*k-dimensional V coordinate classification vector. Among them, the position of the maximum value in the classification vector is used to represent the position of the key point of the hand in the current heat map.

[0105] For example, Figure 7 The figure shows a schematic diagram of the heat map projection result, in which the maximum value 0.8 in the U coordinate classification vector is located at the third position in the 1*k dimensional vector, indicating that the U coordinate of the hand key point on the heat map is at the third position of 0 to k-1 pixels, and the maximum value 0.8 in the V coordinate classification vector is located at the fifth position in the 1*k dimensional vector, indicating that the V coordinate of the hand key point on the heat map is at the fifth position of 0 to k-1 pixels.

[0106] S3022_2: Determine the inter-class similarity and intra-class similarity of each positive sample according to the U-coordinate classification vector and the V-coordinate classification vector corresponding to each sample.

[0107] Among them, the intra-class similarity includes the U-coordinate intra-class similarity and the V-coordinate intra-class similarity, and the inter-class similarity includes the U-coordinate inter-class similarity and the V-coordinate inter-class similarity.

[0108] In one example, the process of determining the intra-class similarity includes: calculating the U coordinate intra-class similarity of the real gesture samples according to the U coordinate classification vector corresponding to each real gesture sample, which is recorded as the first U coordinate intra-class similarity, and calculating the V coordinate intra-class similarity of the real gesture samples according to the V coordinate classification vector corresponding to each real gesture sample, which is recorded as the first V coordinate intra-class similarity. Similarly, calculating the U coordinate intra-class similarity of the synthetic gesture samples according to the U coordinate classification vector corresponding to each synthetic gesture sample, which is recorded as the second U coordinate intra-class similarity, and calculating the V coordinate intra-class similarity of the synthetic gesture samples according to the V coordinate classification vector corresponding to each synthetic gesture sample, which is recorded as the second V coordinate intra-class similarity.

[0109] Taking the real gesture sample as an example, assuming there are a real gesture samples, the formula of the U coordinate classification vector of these a real gesture samples is expressed as Among them, Z U ∈R a*k ,against And for The formula for the V coordinate classification vector of this a real gesture sample is expressed as Among them, Z V ∈R a*k ,against And for At this time, the similarity within the U coordinate class The formula is: V coordinate similarity within the class The formula is: in, Represents the U coordinate classification vector of two different real gesture samples and Represents the V coordinate classification vector of two different real gesture samples and

[0110] Similarly, the U-coordinate intra-class similarity and V-coordinate intra-class similarity of the synthetic gesture samples can be solved using the same method.

[0111] In one example, the process of determining the inter-class similarity includes: calculating the U coordinate inter-class similarity between the real gesture sample and the synthetic gesture sample based on the U coordinate classification vector of each real gesture sample and each synthetic gesture sample, and calculating the V coordinate intra-class similarity between the real gesture sample and the synthetic gesture sample based on the V coordinate classification vector of each real gesture sample and each synthetic gesture sample.

[0112] For example, assuming there are a real gesture samples and a synthetic gesture samples, the formula for the U coordinate classification vector of each sample is expressed as Among them, ZU ∈R a*k ,against And for The formula for the V coordinate classification vector of each sample is expressed as Among them, Z V ∈R a*k ,against And for At this time, the similarity between U coordinate classes The formula is: V coordinate similarity between classes The formula is: in, Represents the U coordinate classification vector of a real gesture sample and the U coordinate classification vector of a synthetic gesture sample Represents the V coordinate classification vector of a real gesture sample and the V coordinate classification vector of a synthetic gesture sample

[0113] In the embodiment of the present application, the U coordinate classification vector and the V coordinate classification vector of the real gesture sample and the synthetic gesture sample, the intra-class similarity and the intra-class similarity in the classification task can be represented by an autocorrelation matrix.

[0114] For example, Fig. 8A and Figure 8B They are schematic diagrams of the autocorrelation matrix. When the intra-class similarity between the real gesture samples and the synthetic gesture samples is higher than the inter-class similarity between the real gesture samples and the synthetic gesture samples, the values ​​in the autocorrelation matrix are mainly concentrated in the diagonal position. When the intra-class similarity between the real gesture samples and the synthetic gesture samples is lower than the inter-class similarity between the real gesture samples and the synthetic gesture samples, the distribution of values ​​in the autocorrelation matrix is ​​irregular.

[0115] S3022_3: Determine the domain difference loss value between the real gesture sample and the synthetic gesture sample based on the intra-class similarity and the inter-class similarity.

[0116] In one example, a key point domain difference function between the real gesture sample and the synthetic gesture sample is constructed based on the intra-class similarity between the real gesture sample and the synthetic gesture sample, and the inter-class similarity between the real gesture sample and the synthetic gesture sample. in, represents the similarity within the U coordinate class, represents the similarity within the V coordinate class, represents the inter-class similarity of the U coordinates between the real gesture samples and the synthetic gesture samples, Represents the inter-class similarity of the V coordinates between the real gesture samples and the synthetic gesture samples.

[0117] Furthermore, based on the key point domain difference function, the domain difference loss function loss between the real gesture sample and the synthetic gesture sample is obtained. DA , the formula is as follows:

[0118]

[0119] Among them, N s Represents the number of real gesture samples, N T represents the number of synthetic gesture samples, x s represents the i-th true gesture sample, x t Represents the jth synthetic gesture sample. By using Formula 2, the domain difference loss value between the real gesture sample and the synthetic gesture sample can be calculated.

[0120] In an embodiment of the present application, a synthetic gesture sample subset containing complex hand gestures generated based on a 3D gesture model can supplement the complex hand gestures that are missing in the real gesture sample subset, so that the generalization of the trained model for complex hand gestures with severe hand self-occlusion is enhanced; at the same time, key point domain adaptation technology is adopted, and adversarial training is performed by calculating the domain difference loss value between the real gesture samples and the synthetic gesture samples, so that the model converges quickly on the real gesture samples and the synthetic gesture samples, thereby alleviating the domain difference problem between the real gesture samples and the synthetic gesture samples, effectively learning the complex hand gesture features in the synthetic gesture samples, and improving the accuracy and robustness of 3D gesture detection.

[0121] S3023: According to the respective heat maps of the positive samples and the negative samples in the multiple samples and the labels of the corresponding samples, respectively determine the positive sample distribution loss value, the negative sample distribution loss value, and the positive and negative sample distribution difference loss value.

[0122] In order to alleviate the interference of background information on the key points of the hand, a subset of background samples without hands is included in the 3D gesture sample set. Therefore, the negative sample regularization constraint for model training can be constructed based on the background samples. By calculating the positive sample distribution loss value, the negative sample distribution loss value, and the positive and negative sample distribution difference loss value, the model can fully learn the difference between the background information and the key points of the hand, and further improve the accuracy and robustness of 3D gesture detection.

[0123] See also Fig. 9 , which is the process of constructing negative sample regularization constraints for model training based on background samples, mainly includes the following steps:

[0124] S3023_1: For each sample among the multiple samples, subtract the element value in each heat map of the sample from the element value at the corresponding position in the label of the sample to obtain the element difference at the corresponding position.

[0125] Taking a negative sample as an example, it can be seen from the above embodiment that the label of the negative sample can be represented by a matrix as follows: The heat map of the negative sample can also be represented as a k*k matrix. Therefore, the element difference can be obtained by subtracting the element values ​​at the corresponding positions of the heat map matrix of the negative sample predicted by the model from the label matrix of the negative sample.

[0126] Similarly, the label of the positive sample can also be represented by a k*k matrix. Therefore, the same method can be used to obtain the element difference between the heat map matrix of the positive sample and the label matrix.

[0127] S3023_2: Determine the negative sample distribution loss value according to the element difference corresponding to each heat map of the negative sample in the multiple samples.

[0128] During the model training process, supervised learning can be performed on the heat map of the negative samples predicted by the model according to the labels of the negative samples, so that the model can learn the distribution characteristics of the heat map of the negative samples. Specifically, the negative sample distribution loss value can be determined according to the element difference corresponding to each heat map of the negative sample.

[0129] In one example, a smoothing technique can be used to calculate the negative sample distribution loss value loss hp , the formula is as follows:

[0130]

[0131] Where x represents the element difference between the heat map matrix and the label matrix at the corresponding position, and smooth1(·) represents the smoothing function.

[0132] S3023_3: Determine the positive sample distribution loss value according to the element difference corresponding to each heat map in the positive sample in the multiple samples.

[0133] During the model training process, the heat map of the positive sample predicted by the model can be supervised based on the label of the positive sample, so that the model can learn the distribution characteristics of the heat map of the positive sample. Specifically, the positive sample distribution loss value can be determined based on the element difference corresponding to each heat map of the positive sample.

[0134] Optionally, the calculation method of the positive sample distribution loss value may be the same as or different from the calculation method of the negative sample distribution loss value. For example, a Gaussian function may be used to calculate the positive sample distribution loss value.

[0135] The embodiment of the present application uses negative samples as constraints for model training, so that the model can fully learn the distribution differences of heat maps of positive samples and negative samples, thereby reducing the interference of background information on key points of the hand.

[0136] S3023_4: Use KL divergence to determine the positive and negative sample distribution difference loss value based on the element values ​​in each heat map of multiple samples and the element values ​​at the corresponding positions in the labels of the corresponding samples.

[0137] On the basis of supervised learning of the model based on the labels of negative samples, in order to better distinguish the distribution difference of the heat maps of positive samples and negative samples, the regularization method can be used to maximize the distribution difference of positive samples and the distribution difference of negative samples.

[0138] In one example, KL divergence can be used to measure the difference between the heat map distribution of positive samples and the heat map distribution of negative samples. The regularization process can be achieved by constructing a loss function, which is expressed as follows:

[0139]

[0140] Among them, H s+t is the heat map of positive samples (including real gesture samples and synthetic gesture samples) in multiple samples, H r Represents the heat map of negative samples in multiple samples, h p (x) represents an element value in the heat map, h q (x) represents the element value of the corresponding position in the label of the corresponding sample, and KL(·) represents the KL divergence function. Since we need to maximize the distribution difference between positive and negative samples, we need to set the distribution difference term h p (x) performs the inversion operation to achieve the distribution difference KL(H s+t , H r ) During the training process, as loss reg The process of decreasing and increasing.

[0141] The embodiment of the present application constructs a regularization constraint term through negative samples, which effectively amplifies the distribution difference between background information and hand key point features, thereby improving the robustness of the heat map distribution of hand key points. In addition, through regularization processing, the distribution difference between background information and hand key points is maximized, which helps to improve the stability of the overall algorithm. In practical applications, by combining a nonlinear optimization module, the false detection results of the front end can be filtered out to improve the accuracy of 3D gesture detection.

[0142] S3024: Determine a target loss value for 3D gesture detection based on the domain difference loss value, the positive sample distribution loss value, the negative sample distribution loss value, and the positive and negative sample distribution difference loss value, and adjust the model parameters based on the target loss value.

[0143] In one example, the target loss value is calculated as follows:

[0144] loss total =loss DA +losshp +loss hm +loss reg Formula 5

[0145] Among them, loss DA Indicates the domain difference loss value, loss hm Represents the positive sample distribution loss value, loss hp Represents the negative sample distribution loss value, loss reg Indicates the difference loss value of positive and negative sample distribution, loss total Represents the target loss value.

[0146] Different from the problem of improving the accuracy and robustness of 3D gesture detection by amplifying real gesture samples in the related art, the target gesture detection model provided in the embodiment of the present application increases the subset of synthetic gesture samples and adopts the key point domain adaptation technology to reduce the domain difference problem between real gesture samples and synthetic gesture samples, so that the model can effectively learn the complex hand posture features in the synthetic gesture samples, and improve the generalization of complex hand postures with severe self-occlusion through synthetic gesture samples; at the same time, by increasing the subset of background samples without hands and adopting the negative sample regularization technology, the difference between background information and hand key point features is increased, so that the robustness of the heat map output by the model to the complex hand postures with severe self-occlusion is enhanced, thereby enhancing the generalization of the model for complex hand postures with severe self-occlusion at a lower cost and shortening the training time of the model. The model can be used to assist in filtering 3D gestures that are misdetected by the front end to solve the problem of screen jitter caused by complex hand postures with severe self-occlusion. In addition, the target gesture detection model provided by the implementation of the present application is a universal optimization method for improving the stability of complex hand posture detection, which is not affected by the specific model structure design. It only requires the feature output head of the model to output the heat map, and it is more applicable.

[0147] The target gesture detection model provided by the implementation of this application can be deployed in a head-mounted display (such as AR glasses, VR glasses, etc.) to achieve 3D gesture interaction.

[0148] In specific implementation, the head-mounted display obtains gesture images in the gesture interaction process in real time through the camera thereon, and inputs the gesture images into the target gesture detection model. According to the target gesture detection model, the key points in the gesture images are extracted to obtain the 3D gestures, and the display of the virtual screen is controlled according to the 3D gestures.

[0149] like Fig.10 As shown, this is a 3D gesture effect diagram detected by the target gesture detection model provided in the embodiment of the present application. For hand grasping actions with occlusion, the hand key points can be accurately detected, thereby improving the accuracy of gesture interaction.

[0150] Based on the same technical concept, an embodiment of the present application provides an electronic device, which can be a terminal device, such as a laptop computer, a desktop computer, a head-mounted display, etc., or a server, such as a server cluster, a cloud server, etc. The electronic device can implement the steps of the above-mentioned 3D gesture detection method and can achieve the same technical effect.

[0151] See also Fig.11 , the electronic device includes a processor 1101, a memory 1102 and a communication interface 1103, and the communication interface 1103, the memory 1102 and the processor 1101 are connected via a bus 1104;

[0152] The memory 1102 may be a volatile memory, such as a random-access memory (RAM); the memory 1102 may also be a non-volatile memory, such as a read-only memory, a flash memory, a hard disk drive (HDD) or a solid-state drive (SSD); or the memory 1102 may be any other medium that can be used to carry or store a desired computer program in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory 1102 may be a combination of the above memories;

[0153] The processor 1101 may include one or more central processing units (CPU) or a digital processing unit, etc.;

[0154] The communication interface 1103 is used to communicate with other terminal devices and other servers;

[0155] The memory 1102 stores a computer program, and the processor 1101 performs the following operations according to the computer program:

[0156] Acquire a 3D gesture sample set; the 3D gesture sample set includes a background sample subset, a real gesture sample subset and a synthetic gesture sample subset, the real gesture samples and the synthetic gesture samples are used as positive samples, the background samples are used as negative samples, the label of each negative sample obeys a discrete uniform distribution, and the label of each positive sample obeys a Gaussian normal distribution;

[0157] According to the 3D gesture sample set, the 3D gesture detection model to be trained is iterated for at least one time to obtain a target gesture detection model, wherein each iteration process performs the following operations:

[0158] Performing feature extraction on multiple samples in the 3D gesture sample set to obtain a heat map set of corresponding samples;

[0159] Determine, according to respective heat maps of the real gesture sample and the synthetic gesture sample in the plurality of samples, a domain difference loss value between the real gesture sample and the synthetic gesture sample;

[0160] According to the heat maps of the positive samples and the negative samples in the multiple samples and the labels of the corresponding samples, respectively determine the positive sample distribution loss value, the negative sample distribution loss value, and the positive and negative sample distribution difference loss value;

[0161] A target loss value for 3D gesture detection is determined according to the domain difference loss value, the positive sample distribution loss value, the negative sample distribution loss value, and the positive and negative sample distribution difference loss value, and model parameters are adjusted according to the target loss value.

[0162] Optionally, the processor 1101 determines the domain difference loss value between the real gesture sample and the synthetic gesture sample according to the respective heat map sets of the real gesture sample and the synthetic gesture sample in the multiple samples, and the specific operation is:

[0163] For each positive sample in each real gesture sample and each synthetic gesture sample, project each heat map in the heat map set of the positive sample onto the U axis and V axis of the image coordinate system respectively to obtain a U coordinate classification vector and a V coordinate classification vector;

[0164] Determine the inter-class similarity and intra-class similarity of each positive sample according to the U coordinate classification vector and the V coordinate classification vector corresponding to each positive sample;

[0165] A domain difference loss value between the real gesture sample and the synthetic gesture sample is determined according to the intra-class similarity and the inter-class similarity.

[0166] Optionally, the intra-class similarity includes U-coordinate intra-class similarity and V-coordinate intra-class similarity, and the inter-class similarity includes U-coordinate inter-class similarity and V-coordinate inter-class similarity;

[0167] The processor 1101 determines the inter-class similarity and intra-class similarity of each positive sample according to the U coordinate classification vector and the V coordinate classification vector corresponding to each positive sample. The specific operation is:

[0168] Calculating a first U coordinate intra-class similarity according to the U coordinate classification vectors corresponding to the respective real gesture samples, and calculating a first V coordinate intra-class similarity according to the V coordinate classification vectors corresponding to the respective real gesture samples;

[0169] Calculating a second U-coordinate intra-class similarity according to the U-coordinate classification vector of each of the synthesized gesture samples, and calculating a second V-coordinate intra-class similarity according to the V-coordinate classification vector of each of the synthesized gesture samples;

[0170] The U coordinate inter-class similarity is calculated based on the U coordinate classification vectors of the real gesture samples and the synthetic gesture samples, and the V coordinate intra-class similarity is calculated based on the V coordinate classification vectors of the real gesture samples and the synthetic gesture samples.

[0171] Optionally, the intra-class similarity includes U coordinate intra-class similarity and V coordinate intra-class similarity, the inter-class similarity includes U coordinate inter-class similarity and V coordinate inter-class similarity, and the calculation formula of the domain difference loss value is:

[0172]

[0173] Among them, Dist DA (·) represents the key point domain difference function between the real gesture sample and the synthetic gesture sample, N s Represents the number of real gesture samples, N T represents the number of synthetic gesture samples, x s represents the i-th true gesture sample, x t represents the jth synthetic gesture sample, represents the similarity within the U coordinate class, represents the similarity within the V coordinate class, represents the U coordinate inter-class similarity between the real gesture sample and the synthetic gesture sample, Represents the inter-class similarity of the V coordinates between the real gesture sample and the synthetic gesture sample.

[0174] Optionally, the processor 1101 determines the positive sample distribution loss value, the negative sample distribution loss value, and the positive and negative sample distribution difference loss value respectively according to the heat map sets of the positive samples and the negative samples in the multiple samples and the labels of the corresponding samples, and the specific operation is:

[0175] For each sample among the multiple samples, subtract the element value in each heat map of the sample from the element value at the corresponding position in the label of the sample to obtain the element difference at the corresponding position;

[0176] Determining the negative sample distribution loss value according to the element difference corresponding to each heat map of the negative samples in the multiple samples;

[0177] Determining the positive sample distribution loss value according to the element difference values ​​corresponding to each heat map of the positive sample in the multiple samples;

[0178] The KL divergence is adopted to determine the positive and negative sample distribution difference loss value according to the element value of each heat map of the multiple samples and the element value of the corresponding position in the label of the corresponding sample.

[0179] Optionally, the formula for the negative sample distribution loss value is expressed as:

[0180]

[0181] Wherein, x represents the element difference, and smooth1(·) represents a smoothing function.

[0182] Optionally, the formula for the positive and negative sample distribution difference loss value is expressed as:

[0183]

[0184] Among them, H s+t is the heat map corresponding to each key point contained in the positive sample in the multiple samples, H r Represents the heat map corresponding to each key point contained in the negative samples in multiple samples, h p (x) represents an element value in the heat map, h q (x) represents the element value at the corresponding position in the label, and KL(·) represents the KL divergence function.

[0185] Optionally, after obtaining the target gesture detection model, the processor 1101 further executes:

[0186] Acquire a gesture image during the gesture interaction process, and input the gesture image into the target gesture detection model;

[0187] Extract key points in the gesture image according to the target gesture detection model to obtain a 3D gesture;

[0188] According to the 3D gesture, the display of the virtual image is controlled.

[0189] It should be noted that Fig.11 This is just an example, and provides the necessary hardware for an electronic device to execute the steps of a 3D gesture detection method provided in an embodiment of the present application. If not shown, the electronic device may also include conventional hardware such as a speaker, a pickup, a display, and a power supply.

[0190] Embodiments of the present application Fig.11The processor involved may be a central processing unit (CPU), a general-purpose processor, a graphics processing unit (GPU), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof.

[0191] like Fig.12 As shown, an example of an electronic device being a head-mounted display, the head-mounted display comprises a processor 1201, a memory 1202, a display 1203 and a camera 1204, the camera 1204, the display 1203, the memory 1202 and the processor 1201 are connected via a bus 1205, and a target gesture detection model is deployed in the head-mounted display;

[0192] The camera 1204 is used to obtain gesture images during the gesture interaction process;

[0193] The memory 1202 stores a computer program, and the processor 1201 performs the following operations according to the computer program:

[0194] Inputting the gesture image acquired by the camera 1204 into the target gesture detection model;

[0195] According to the target gesture detection model, the key points in the gesture image are extracted to obtain the 3D gesture;

[0196] According to the 3D gesture, the virtual image displayed on the display 1203 is controlled.

[0197] An embodiment of the present application further provides a computer-readable storage medium for storing some instructions, which, when executed, can complete a method for generating a 3D gesture data set in the aforementioned embodiment.

[0198] An embodiment of the present application further provides a computer program product for storing a computer program, wherein the computer program is used to execute a method for generating a 3D gesture data set in the aforementioned embodiment.

[0199] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0200] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0201] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0202] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0203] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A 3D gesture detection method, It is characterized in that include: Get a 3D gesture sample set; The 3D gesture sample set includes a background sample subset, a real gesture sample subset and a synthetic gesture sample subset, the real gesture samples and the synthetic gesture samples are used as positive samples, the background samples are used as negative samples, the label of each negative sample obeys a discrete uniform distribution, and the label of each positive sample obeys a Gaussian normal distribution; According to the 3D gesture sample set, the 3D gesture detection model to be trained is iterated for at least one time to obtain a target gesture detection model, wherein each iteration process performs the following operations: Extracting features from multiple samples in the 3D gesture sample set to obtain a heat map set of corresponding samples; Determine, according to respective heat maps of the real gesture sample and the synthetic gesture sample in the plurality of samples, a domain difference loss value between the real gesture sample and the synthetic gesture sample; According to the heat maps of the positive samples and the negative samples in the multiple samples and the labels of the corresponding samples, respectively determine the positive sample distribution loss value, the negative sample distribution loss value, and the positive and negative sample distribution difference loss value; A target loss value for 3D gesture detection is determined according to the domain difference loss value, the positive sample distribution loss value, the negative sample distribution loss value, and the positive and negative sample distribution difference loss value, and model parameters are adjusted according to the target loss value.

2. The method according to claim 1, It is characterized in that The determining, according to the respective heat maps of the real gesture sample and the synthetic gesture sample in the multiple samples, the domain difference loss value between the real gesture sample and the synthetic gesture sample comprises: For each positive sample in each real gesture sample and each synthetic gesture sample, project each heat map in the heat map set of the positive sample onto the U axis and V axis of the image coordinate system respectively to obtain a U coordinate classification vector and a V coordinate classification vector; Determine the inter-class similarity and intra-class similarity of each positive sample according to the U coordinate classification vector and the V coordinate classification vector corresponding to each positive sample; A domain difference loss value between the real gesture sample and the synthetic gesture sample is determined according to the intra-class similarity and the inter-class similarity.

3. The method according to claim 2, It is characterized in that The intra-class similarity includes the U-coordinate intra-class similarity and the V-coordinate intra-class similarity, and the inter-class similarity includes the U-coordinate inter-class similarity and the V-coordinate inter-class similarity; Determining the inter-class similarity and intra-class similarity of each positive sample according to the U coordinate classification vector and the V coordinate classification vector corresponding to each positive sample includes: Calculating a first U coordinate intra-class similarity according to the U coordinate classification vectors corresponding to the respective real gesture samples, and calculating a first V coordinate intra-class similarity according to the V coordinate classification vectors corresponding to the respective real gesture samples; Calculating a second U-coordinate intra-class similarity according to the U-coordinate classification vector of each of the synthesized gesture samples, and calculating a second V-coordinate intra-class similarity according to the V-coordinate classification vector of each of the synthesized gesture samples; The U coordinate inter-class similarity is calculated based on the U coordinate classification vectors of the real gesture samples and the synthetic gesture samples, and the V coordinate intra-class similarity is calculated based on the V coordinate classification vectors of the real gesture samples and the synthetic gesture samples.

4. The method according to claim 2, It is characterized in that The intra-class similarity includes the U coordinate intra-class similarity and the V coordinate intra-class similarity, the inter-class similarity includes the U coordinate inter-class similarity and the V coordinate inter-class similarity, and the calculation formula of the domain difference loss value is: Among them, Dist DA (·) represents the key point domain difference function between the real gesture sample and the synthetic gesture sample, N s Represents the number of real gesture samples, N T represents the number of synthetic gesture samples, x s represents the i-th true gesture sample, x t represents the jth synthetic gesture sample, represents the similarity within the U coordinate class, represents the similarity within the V coordinate class, represents the U coordinate inter-class similarity between the real gesture sample and the synthetic gesture sample, Represents the inter-class similarity of the V coordinates between the real gesture sample and the synthetic gesture sample.

5. The method according to claim 1, It is characterized in that The method of determining the positive sample distribution loss value, the negative sample distribution loss value, and the positive and negative sample distribution difference loss value respectively according to the heat maps of the positive samples and the negative samples in the multiple samples and the labels of the corresponding samples includes: For each sample among the multiple samples, subtract the element value in each heat map of the sample from the element value at the corresponding position in the label of the sample to obtain the element difference at the corresponding position; Determining the negative sample distribution loss value according to the element difference corresponding to each heat map of the negative samples in the multiple samples; Determining the positive sample distribution loss value according to the element difference corresponding to each heat map of the positive sample in the multiple samples; The KL divergence is adopted to determine the positive and negative sample distribution difference loss value according to the element value of each heat map of the multiple samples and the element value of the corresponding position in the label of the corresponding sample.

6. The method according to claim 5, It is characterized in that The formula for the negative sample distribution loss value is expressed as: Among them, x represents the element difference, smooth 1 (·) represents a smooth function.

7. The method according to claim 5, It is characterized in that The formula for the positive and negative sample distribution difference loss value is expressed as: Among them, H s+t is the heat map corresponding to each key point contained in the positive sample in the multiple samples, H r Represents the heat map corresponding to each key point contained in the negative samples in multiple samples, h p (x) represents an element value in the heat map, h q (x) represents the element value at the corresponding position in the label, and KL(·) represents the KL divergence function.

8. The method according to any one of claims 1 to 7, It is characterized in that After obtaining the target gesture detection model, the method further includes: Acquire a gesture image during the gesture interaction process, and input the gesture image into the target gesture detection model; Extract key points in the gesture image according to the target gesture detection model to obtain a 3D gesture; According to the 3D gesture, the display of the virtual image is controlled.

9. An electronic device, It is characterized in that It includes a processor, a memory and a communication interface, wherein the communication interface, the memory and the processor are connected via a bus; The communication interface is used to communicate with the terminal device and the server; The memory stores a computer program, and the processor performs the following operations according to the computer program: Acquire a 3D gesture sample set; the 3D gesture sample set includes a background sample subset, a real gesture sample subset and a synthetic gesture sample subset, the real gesture samples and the synthetic gesture samples are used as positive samples, the background samples are used as negative samples, the label of each negative sample obeys a discrete uniform distribution, and the label of each positive sample obeys a Gaussian normal distribution; According to the 3D gesture sample set, the 3D gesture detection model to be trained is iterated for at least one time to obtain a target gesture detection model, wherein each iteration process performs the following operations: Extracting features from multiple samples in the 3D gesture sample set to obtain a heat map set of corresponding samples; Determine, according to respective heat maps of the real gesture sample and the synthetic gesture sample in the plurality of samples, a domain difference loss value between the real gesture sample and the synthetic gesture sample; According to the heat maps of the positive samples and the negative samples in the multiple samples and the labels of the corresponding samples, respectively determine the positive sample distribution loss value, the negative sample distribution loss value, and the positive and negative sample distribution difference loss value; A target loss value for 3D gesture detection is determined according to the domain difference loss value, the positive sample distribution loss value, the negative sample distribution loss value, and the positive and negative sample distribution difference loss value, and model parameters are adjusted according to the target loss value.

10. A head mounted display, It is characterized in that A target gesture detection model as described in any one of claims 1 to 7 is deployed in a head-mounted display, the head-mounted display comprises a processor, a memory, a display and a camera, and the camera, the display, the memory and the processor are connected via a bus; The camera is used to obtain gesture images during gesture interaction; The memory stores a computer program, and the processor performs the following operations according to the computer program: inputting the gesture image acquired by the camera into the target gesture detection model; extracting key points in the gesture image according to the target gesture detection model to obtain a 3D gesture; and controlling the virtual screen displayed on the display according to the 3D gesture.

Images