3D gesture detection method and electronic device

Abstract

This application relates to the field of gesture interaction technology, providing a 3D gesture detection method and electronic device to improve the accuracy and robustness of 3D gesture detection. During model training, a subset of synthetic gesture samples is added to supplement complex hand poses under self-occlusion conditions where the real gesture sample subset is missing. A keypoint domain adaptation technique is designed to calculate the domain difference loss value between real and synthetic gesture samples, thereby alleviating the domain difference problem between real and synthetic gesture samples and improving the model's generalization ability to complex hand poses with severe self-occlusion through synthetic gesture samples. Simultaneously, a background sample subset excluding hands is introduced, and a negative sample regularization constraint term is constructed based on the background sample subset. The negative sample distribution loss value and the positive-negative sample distribution difference loss value are calculated, thereby widening the distribution difference between background information and hand keypoints, reducing the interference of background information on hand keypoints, and improving the accuracy of 3D gesture detection.

Description

Technical Field

[0001] This application relates to the field of gesture interaction technology, and provides a 3D gesture detection method and electronic device. Background Technology

[0002] With continued attention and investment in the metaverse concept both domestically and internationally, the AR / VR industry is experiencing a surge in popularity. Currently, improving user experience is one of the key objectives of AR / VR devices. 3D gesture interaction technology is a human-computer interaction technology that does not rely on external devices, making it more user-friendly. Therefore, 3D gesture interaction has become a research hotspot for AR / VR devices.

[0003] During interaction, the stability of 3D gestures is the main factor affecting the gesture interaction experience of AR / VR devices. During the interaction, some key points of the hand may be invisible from a certain perspective due to the hand's own occlusion, which will seriously affect the recognition of 3D gestures. Therefore, solving the complex problem of hand self-occlusion is a difficult problem that needs to be overcome in 3D gesture interaction technology.

[0004] Currently, related technologies typically use multiple cameras to capture gesture images and manually annotate them to generate 3D gesture datasets. However, for complex gestures with severe self-occlusion, annotation is difficult, resulting in a low proportion of effective data. Consequently, when performing gesture detection on 3D gesture datasets generated in this way, the accuracy of gesture detection will be affected, leading to incorrect interactions or interaction failures, and degrading the user experience. Summary of the Invention

[0005] This application provides a 3D gesture detection method and electronic device to improve the accuracy of 3D gesture detection.

[0006] On one hand, embodiments of this application provide a 3D gesture detection method, including:

[0007] Obtain a 3D gesture sample set; the 3D gesture sample set includes a background sample subset, a real gesture sample subset, and a synthetic gesture sample subset. The real gesture samples and synthetic gesture samples are used as positive samples, and the background samples are used as negative samples. The label of each negative sample follows a discrete uniform distribution, and the label of each positive sample follows a Gaussian normal distribution.

[0008] Based on the 3D gesture sample set, the 3D gesture detection model to be trained is trained at least once to obtain the target gesture detection model, wherein the following operations are performed in each iteration:

[0009] Feature extraction is performed on multiple samples in the 3D gesture sample set to obtain the heat map set of the corresponding samples;

[0010] Based on the heatmap sets of the real gesture samples and the synthetic gesture samples in the plurality of samples, the domain difference loss value between the real gesture samples and the synthetic gesture samples is determined;

[0011] Based on the heatmap sets of positive and negative samples in the multiple samples and the corresponding sample labels, the distribution loss value of positive samples, the distribution loss value of negative samples, and the distribution difference loss value of positive and negative samples are determined respectively.

[0012] Based on the domain difference loss value, the positive sample distribution loss value, the negative sample distribution loss value, and the positive and negative sample distribution difference loss value, the target loss value for 3D gesture detection is determined, and the model parameters are adjusted according to the target loss value.

[0013] On the other hand, embodiments of this application provide an electronic device, including a processor, a memory, and a communication interface, wherein the communication interface, the memory, and the processor are connected via a bus;

[0014] The communication interface is used to communicate with terminal devices and servers;

[0015] The memory stores a computer program, and the processor performs the following operations according to the computer program:

[0016] Obtain a 3D gesture sample set; the 3D gesture sample set includes a background sample subset, a real gesture sample subset, and a synthetic gesture sample subset. The real gesture samples and synthetic gesture samples are used as positive samples, and the background samples are used as negative samples. The label of each negative sample follows a discrete uniform distribution, and the label of each positive sample follows a Gaussian normal distribution.

[0017] Based on the 3D gesture sample set, the 3D gesture detection model to be trained is trained at least once to obtain the target gesture detection model, wherein the following operations are performed in each iteration:

[0018] Feature extraction is performed on multiple samples in the 3D gesture sample set to obtain the heat map set of the corresponding samples;

[0019] Based on the heatmap sets of the real gesture samples and the synthetic gesture samples in the plurality of samples, the domain difference loss value between the real gesture samples and the synthetic gesture samples is determined;

[0020] Based on the heatmap sets of positive and negative samples in the multiple samples and the corresponding sample labels, the distribution loss value of positive samples, the distribution loss value of negative samples, and the distribution difference loss value of positive and negative samples are determined respectively.

[0021] Based on the domain difference loss value, the positive sample distribution loss value, the negative sample distribution loss value, and the positive and negative sample distribution difference loss value, the target loss value for 3D gesture detection is determined, and the model parameters are adjusted according to the target loss value.

[0022] Optionally, the processor determines the domain difference loss value between the real gesture samples and the synthetic gesture samples based on the respective heatmap sets of the real gesture samples and the synthetic gesture samples from the plurality of samples. Specifically, the operation is as follows:

[0023] For each positive sample in each real gesture sample and each synthetic gesture sample, each heatmap in the heatmap set of the positive sample is projected onto the U-axis and V-axis of the image coordinate system to obtain a U-coordinate classification vector and a V-coordinate classification vector.

[0024] Based on the U-coordinate classification vector and V-coordinate classification vector corresponding to each positive sample, the inter-class similarity and intra-class similarity of each positive sample are determined.

[0025] Based on the intra-class similarity and the inter-class similarity, the domain difference loss value between the real gesture sample and the synthetic gesture sample is determined.

[0026] Optionally, the intra-class similarity includes U-coordinate intra-class similarity and V-coordinate intra-class similarity, and the inter-class similarity includes U-coordinate inter-class similarity and V-coordinate inter-class similarity;

[0027] The processor determines the inter-class similarity and intra-class similarity of each positive sample based on the U-coordinate classification vector and V-coordinate classification vector corresponding to each positive sample. The specific operation is as follows:

[0028] Based on the U-coordinate classification vector corresponding to each of the real gesture samples, calculate the first U-coordinate intra-class similarity, and based on the V-coordinate classification vector of each of the real gesture samples, calculate the first V-coordinate intra-class similarity.

[0029] Based on the U-coordinate classification vector of each synthesized gesture sample, calculate the second U-coordinate intra-class similarity; and based on the V-coordinate classification vector of each synthesized gesture sample, calculate the second V-coordinate intra-class similarity.

[0030] Based on the U-coordinate classification vectors of each real gesture sample and each synthetic gesture sample, the inter-class similarity of the U-coordinate is calculated, and based on the V-coordinate classification vectors of each real gesture sample and each synthetic gesture sample, the intra-class similarity of the V-coordinate is calculated.

[0031] Optionally, the intra-class similarity includes intra-class similarity in U-coordinates and intra-class similarity in V-coordinates, and the inter-class similarity includes inter-class similarity in U-coordinates and inter-class similarity in V-coordinates. The formula for calculating the domain difference loss value is as follows:

[0032]

[0033] Among them, Dist DA (·) represents the keypoint domain difference function between the real gesture sample and the synthetic gesture sample. N s N represents the number of real gesture samples. T x represents the number of synthesized gesture samples. s Let x represent the i-th real gesture sample. t This represents the j-th synthesized gesture sample. Indicates the intra-class similarity of the U-coordinate. V-coordinate represents intra-class similarity. This represents the inter-class similarity of the U-coordinate between the real gesture sample and the synthetic gesture sample. The V-coordinate represents the inter-class similarity between the real gesture sample and the synthetic gesture sample.

[0034] Optionally, the processor determines the positive sample distribution loss value, the negative sample distribution loss value, and the positive-negative sample distribution difference loss value based on the heatmap sets of the positive and negative samples and the corresponding sample labels, respectively. The specific operation is as follows:

[0035] For each of the plurality of samples, the element value in each heatmap of the sample is subtracted from the element value at the corresponding position in the label of the sample to obtain the element difference at the corresponding position.

[0036] The distribution loss value of the negative sample is determined based on the element differences corresponding to each heatmap of the negative sample among the multiple samples.

[0037] The distribution loss value of the positive samples is determined based on the element differences corresponding to each heatmap of the positive samples among the multiple samples.

[0038] Using KL divergence, the distribution difference loss value of the positive and negative samples is determined based on the element values ​​of each heatmap of the multiple samples and the element values ​​of the corresponding positions in the labels of the corresponding samples.

[0039] Optionally, the formula for the negative sample distribution loss value is expressed as:

[0040]

[0041] Where x represents the element difference, and smooth1(·) represents the smoothing function.

[0042] Optionally, the formula for the loss value of the positive and negative sample distribution differences is expressed as follows:

[0043]

[0044] Among them, H s+t H is a heatmap corresponding to each key point contained in the positive samples among the multiple samples. r h represents a heatmap of keypoints contained in a negative sample across multiple samples. p (x) represents an element value in the heatmap, h q (x) represents the element value at the corresponding position in the label, and KL(·) represents the KL divergence function.

[0045] Optionally, after obtaining the target gesture detection model, the processor further performs:

[0046] Acquire gesture images during the gesture interaction process and input the gesture images into the target gesture detection model;

[0047] Based on the target gesture detection model, key points are extracted from the gesture image to obtain a 3D gesture.

[0048] The display of the virtual screen is controlled based on the 3D gestures.

[0049] On the other hand, embodiments of this application provide a head-mounted display in which a target gesture detection model is deployed. The head-mounted display includes a processor, a memory, a display, and a camera. The camera, the display, the memory, and the processor are connected via a bus.

[0050] The camera is used to acquire gesture images during the gesture interaction process;

[0051] The memory stores a computer program, and the processor performs the following operations according to the computer program:

[0052] The gesture image acquired by the camera is input into the target gesture detection model;

[0053] Based on the target gesture detection model, key points are extracted from the gesture image to obtain a 3D gesture.

[0054] The virtual images displayed on the monitor are controlled based on the 3D gestures.

[0055] On the other hand, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions for causing a computer device to perform the steps of a 3D gesture detection method provided in embodiments of this application.

[0056] The 3D gesture detection method and electronic device provided in this application include a target gesture detection model for gesture detection. The 3D gesture sample set used by this model includes positive samples of the hand and negative samples (i.e., background samples) containing only the background and not the hand. The positive samples are further divided into a subset of real gesture samples and a subset of synthetic gesture samples according to the different generation methods. The synthetic gesture samples enrich the complex gestures that are difficult to collect in real-world scenarios, and the background samples alleviate the interference of the background on gesture detection, thereby improving the accuracy and robustness of complex hand pose recognition. Unlike conventional sample labels, the labels of each negative sample in the 3D gesture sample set follow a discrete uniform distribution, and the labels of each positive sample follow a Gaussian normal distribution, which effectively describes the distribution characteristics of key hand points and helps to improve the accuracy of gesture detection. On the other hand, during model training, considering the domain difference between real and synthetic gesture samples, a domain adaptation technique is used to calculate the domain difference loss between the two based on the heatmaps of real and synthetic gesture samples. This enables the generalization problem of complex hand poses with severe self-occlusion to be solved by using synthetic gesture samples. At the same time, a regularization constraint term is constructed by introducing negative samples. Based on the heatmaps of positive and negative samples and the labels of the corresponding samples, the distribution loss of positive samples, the distribution loss of negative samples, and the distribution difference loss of positive and negative samples are calculated. This allows the model to fully learn the differences between the background and the key points of the hand, thereby improving the robustness and stability of the model.

[0057] Other features and advantages of this application will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the application. The objectives and other advantages of this application may be realized and obtained by means of the structures particularly pointed out in the written description, claims, and drawings. Attached Figure Description

[0058] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0059] Figure 1 This application provides a schematic diagram of hand key point tremor detection for embodiments of the present application;

[0060] Figure 2 This is an overall architecture diagram of the training target gesture detection model provided in the embodiments of this application;

[0061] Figure 3 A flowchart illustrating the training method for the target gesture detection model provided in this application embodiment;

[0062] Figure 4 This is a schematic diagram of a 3D gesture sample set provided in an embodiment of this application;

[0063] Figure 5A This application provides a schematic diagram of a heatmap corresponding to a positive sample in an embodiment of the present application.

[0064] Figure 5B This application provides a schematic diagram of a heatmap corresponding to a negative sample in an embodiment of the present application.

[0065] Figure 6 A flowchart illustrating the domain difference loss calculation method based on key point domain adaptation technology provided in this application embodiment;

[0066] Figure 7 This is a schematic diagram of the thermal projection results provided in an embodiment of this application;

[0067] Figure 8A A schematic diagram of an autocorrelation matrix provided for an embodiment of this application;

[0068] Figure 8B Another schematic diagram of an autocorrelation matrix provided for an embodiment of this application;

[0069] Figure 9 A flowchart illustrating the method for constructing negative sample regularization constraints for model training provided in this application embodiment;

[0070] Figure 10 This is a 3D gesture effect image detected using the target gesture detection model provided in the embodiments of this application;

[0071] Figure 11 A structural diagram of the electronic device provided in the embodiments of this application;

[0072] Figure 12 This is a structural diagram of a head-mounted display provided in an embodiment of this application. Detailed Implementation

[0073] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings of the embodiments of this application. Obviously, the described embodiments are only some embodiments of the technical solutions of this application, and not all embodiments. Based on the embodiments recorded in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the technical solutions of this application.

[0074] Hand keypoint detection, as one of the core technologies of 3D gesture interaction, is crucial to the interactive experience. However, in complex hand poses with severe self-occlusion, hand keypoint detection exhibits poor generalization, leading to abnormal hand keypoint jitter issues, such as… Figure 1 As shown, this will affect the success rate of the interaction and the actual user experience.

[0075] To improve the generalization of hand keypoint detection on complex gesture data with severe self-occlusion, the most direct method is to add relevant complex gesture data when training the gesture detection model. However, if multiple camera devices are used to collect complex gesture data, there will be a problem of high annotation difficulty, and the gesture accuracy will be difficult to guarantee.

[0076] In view of this, this application provides a 3D gesture detection method from two aspects: domain adaptation and negative sample regularization, to solve the jitter problem of complex hand poses under severe self-occlusion, and improve the accuracy and stability of gesture detection. This method introduces synthetic gesture samples to enrich the collection of complex gestures that are difficult to collect in real-world scenarios, and employs domain adaptation technology to reduce the distribution distance between real gesture samples and synthetic gesture samples, thereby achieving the generalization problem of solving complex hand poses with severe self-occlusion through synthetic gesture samples. Simultaneously, by introducing negative samples that only contain the background and not the hand, the interference of background information on hand keypoints is alleviated, allowing the model to fully learn the differences between the background and hand keypoints, improving the robustness and stability of the algorithm. Using negative samples as a regularization constraint effectively amplifies the distribution differences between the background and hand keypoints, further improving the accuracy and robustness of gesture detection.

[0077] See Figure 2This document presents the overall architecture of the gesture detection model provided in this embodiment, comprising five parts: preparing a training sample set, building the gesture detection model to be trained, key point feature extraction, and model training. The sample set preparation part generates a 3D gesture sample set for model training. This 3D gesture sample set includes a subset of conventionally collected real gesture samples and a subset of synthetic gesture samples representing complex hand poses under severe self-occlusion. Both real and synthetic gesture samples contain hand key points and can be used as positive samples for model training. Simultaneously, the 3D gesture sample set also includes a subset of background samples that do not contain hands. These background samples, lacking hand key points, can be used as negative samples for model training. The network used in building the gesture detection model to be trained only needs to output a heatmap; there are no restrictive requirements on the network structure. For example, simple network structures such as ResNet, Hourglass, and HRNet can be used as the backbone to meet the needs of real-time interaction. The feature extraction section extracts features from the 3D gesture sample set, obtaining a heatmap for each sample. The number of heatmaps for each sample can be set according to the number of hand keypoints. The model training section employs keypoint neighborhood adaptation and negative sample regularization constraints. Based on the heatmaps and labels of each sample, the target loss value is calculated, and the model parameters are adjusted according to the target loss value until the model converges. The converged model is then output as the target gesture detection model.

[0078] The training process of the above target gesture detection model is as follows: Figure 3 The process mainly includes the following steps:

[0079] S301: Obtain a 3D gesture sample set.

[0080] In one example, hand gestures are captured using a conventional camera to obtain a subset of real hand gesture samples, such as... Figure 4 As shown in (a) above. Considering the possibility of complex hand poses with self-occlusion during interaction, which are difficult to collect and annotate using multiple camera devices, this application embodiment dynamically synthesizes some high-precision complex hand pose data based on a 3D hand model to obtain a subset of synthesized gesture samples to supplement the complex hand poses missing in the subset of real gesture samples, such as... Figure 4 As shown in (b) in the figure. Since both the real gesture samples in the real gesture sample subset and the synthetic gesture samples in the synthetic gesture sample subset contain hand key points, they can be used as positive samples for model training.

[0081] For complex hand poses with severe self-occlusion, monochrome camera imaging can result in blurred edges, making it impossible to identify whether a hand is present, thus affecting the stability of hand keypoint detection. Furthermore, if only subsets of real and synthetic gesture samples containing hand keypoints are used, the model will still incorrectly identify hand keypoints when an image without hand keypoints is input during interaction, further impacting the accuracy and robustness of hand keypoint detection.

[0082] To mitigate the interference of background information on key hand features, in one example, a subset of background samples excluding the hand was generated, such as... Figure 4 As shown in (c), since the background samples do not contain hand keypoints, they can be used as negative samples for model training.

[0083] Unlike conventional classification and detection models that label positive samples with a value of 1 and negative samples with a value of 0, the labeling method in this embodiment fully considers the heatmap characteristics of the extracted hand key points. It sets the label for each positive sample to follow a Gaussian normal distribution, and the label for each negative sample to follow a discrete uniform distribution. This is because negative samples do not contain hand key points, and theoretically, their heatmaps would not show concentrated distributions of points. Therefore, setting the label for negative samples to follow a discrete uniform distribution indicates that the element values ​​at all locations in the heatmap are equal and very low, where the element value represents the confidence level of the model's prediction that the point at that location is a hand key point. Conversely, positive samples contain hand key points, and their heatmaps would show concentrated distributions of points. Therefore, setting the label for positive samples to follow a Gaussian normal distribution indicates that the element values ​​at different locations in the heatmap are unequal.

[0084] In the embodiments of this application, the labels of the samples can be represented by a matrix, the size of which is the same as the size of the heatmap.

[0085] Taking negative samples as an example, the formula for the discrete uniform distribution that the labels of negative samples follow is expressed as:

[0086]

[0087] Assuming the heatmap has a size of k*k (where k is an integer greater than 1), the matrix representation of the labels for negative samples is as follows:

[0088]

[0089] After obtaining subsets of real gesture samples, synthetic gesture samples, and background samples, these subsets are combined in a certain proportion to obtain a 3D gesture sample set for model training.

[0090] Optionally, the ratio of real gesture samples, synthetic gesture samples, and background samples in the 3D gesture sample set can be 1:1:1. In one example, the number of each type of sample can also be adjusted according to the actual situation.

[0091] It should be noted that adjusting the proportions of real gesture samples, synthetic gesture samples, and background samples at the same order of magnitude has little impact on the model training results and can generally be ignored.

[0092] S302: Based on the 3D gesture sample set, perform at least one iteration of training on the 3D gesture detection model to be trained to obtain the target gesture detection model.

[0093] When building a 3D gesture detection model to be trained, some model parameters are usually set, such as the number of iterations, learning rate, batch size, and the optimizer used.

[0094] For example, the data size for each training batch is set to 64, and the initial learning rate is set to 3e. -4 The number of iterations is 100, and the optimizer used is the Adma optimizer.

[0095] Based on the acquired 3D gesture sample set, the 3D gesture detection model to be trained is trained at least once. The model that reaches the required number of iterations or converges (i.e., the target loss value remains unchanged) is taken as the final target gesture detection model.

[0096] Each training session includes the following steps:

[0097] S3021: Extract features from multiple samples in the 3D gesture sample set to obtain the heatmap set of the corresponding samples.

[0098] In the embodiments of this application, the number of heatmaps in the heatmap set corresponding to each sample is the same and equal to the number of hand key points. For example, when the model can extract 21 hand key points, 21 heatmaps will be extracted for each sample, resulting in a heatmap set.

[0099] like Figure 5A The image shown is a heatmap diagram corresponding to a positive sample. The points in this heatmap are relatively concentrated, and the locations of these concentrated points correspond to the locations of key points on the hand. Figure 5B The image shown is a heatmap diagram corresponding to a negative sample. The points in the heatmap are relatively scattered, indicating that it does not contain key points of the hand.

[0100] S3022: Determine the domain difference loss value between real gesture samples and synthetic gesture samples based on the heatmap sets of real gesture samples and synthetic gesture samples from multiple samples.

[0101] A subset of synthetic gesture samples generated based on a 3D hand model suffers from hand self-occlusion, leading to occlusion of key points and a domain gap between the synthetic and real gesture samples. To address this domain gap issue, this application employs a key point domain adaptation technique. This technique calculates the domain gap loss between the real and synthetic gesture samples to reduce the distribution distance of key points in both samples.

[0102] The calculation process of the domain difference loss value based on key point domain adaptation technology is as follows: Figure 6 As shown, it mainly includes the following steps:

[0103] S3022_1: For each positive sample in each real gesture sample and each synthetic gesture sample, project each heatmap in the heatmap set of the positive sample onto the U-axis and V-axis of the image coordinate system to obtain a U-coordinate classification vector and a V-coordinate classification vector.

[0104] Keypoint domain adaptation technology abstracts the heatmap information regression task into a U- and V-coordinate classification task. In practice, for each positive sample among all real and synthetic gesture samples, taking a heatmap from the heatmap set of that positive sample as an example, assuming the heatmap size is k*k, the heatmap is projected onto the U-axis and V-axis of the image coordinate system, respectively, resulting in a 1*k-dimensional U-coordinate classification vector and a 1*k-dimensional V-coordinate classification vector. The position of the maximum value in the classification vector represents the location of the hand keypoint in the current heatmap.

[0105] For example, such as Figure 7 The diagram shows the result of the heatmap projection. The maximum value of 0.8 in the U-coordinate classification vector is located at the 3rd position in the 1*k-dimensional vector, indicating that the hand key point is located at the 3rd position of the U-coordinate of 0 to k-1 pixels in the heatmap. The maximum value of 0.8 in the V-coordinate classification vector is located at the 5th position in the 1*k-dimensional vector, indicating that the hand key point is located at the 5th position of the V-coordinate of 0 to k-1 pixels in the heatmap.

[0106] S3022_2: Determine the inter-class similarity and intra-class similarity of each positive sample based on the U-coordinate classification vector and V-coordinate classification vector corresponding to each sample.

[0107] Intra-class similarity includes intra-class similarity in U-coordinates and intra-class similarity in V-coordinates, while inter-class similarity includes inter-class similarity in U-coordinates and inter-class similarity in V-coordinates.

[0108] In one example, the process of determining intra-class similarity includes: calculating the intra-class similarity of the real gesture samples in U-coordinates based on the U-coordinate classification vectors corresponding to each real gesture sample, denoted as the first intra-class similarity in U-coordinates; and calculating the intra-class similarity of the real gesture samples in V-coordinates based on the V-coordinate classification vectors corresponding to each real gesture sample, denoted as the first intra-class similarity in V-coordinates. Similarly, calculating the intra-class similarity of the synthesized gesture samples in U-coordinates based on the U-coordinate classification vectors corresponding to each synthesized gesture sample, denoted as the second intra-class similarity in U-coordinates; and calculating the intra-class similarity of the synthesized gesture samples in V-coordinates based on the V-coordinate classification vectors corresponding to each synthesized gesture sample, denoted as the second intra-class similarity in V-coordinates.

[0109] Taking real gesture samples as an example, assuming there are *a* real gesture samples, the formula for the U-coordinate classification vector of these *a* real gesture samples is expressed as: Among them, Z U ∈R a*k ,against And targeting The formula for the V-coordinate classification vector of these *a* real gesture samples is expressed as follows: Among them, Z V ∈R a*k ,against And targeting At this point, the intra-class similarity of the U-coordinates The formula is expressed as: V-coordinate intra-class similarity The formula is expressed as: in, U-coordinate classification vectors representing two different real gesture samples and V-coordinate classification vectors representing two different real gesture samples and

[0110] Similarly, the intra-class similarity of the U-coordinate and intra-class similarity of the synthesized gesture samples can be solved using the same method.

[0111] In one example, the process of determining inter-class similarity includes: calculating the inter-class similarity of U coordinates between real gesture samples and synthetic gesture samples based on the U coordinate classification vectors of each real gesture sample and each synthetic gesture sample; and calculating the intra-class similarity of V coordinates between real gesture samples and synthetic gesture samples based on the V coordinate classification vectors of each real gesture sample and each synthetic gesture sample.

[0112] For example, assuming there are *a* real gesture samples and *a* synthetic gesture samples, the formula for the U-coordinate classification vector of each sample is expressed as: Among them, ZU ∈R a*k ,against And targeting The formula for the V-coordinate classification vector of each sample is expressed as follows: Among them, Z V ∈R a*k ,against And targeting At this point, the inter-class similarity of the U-coordinates The formula is expressed as: V-coordinate inter-class similarity The formula is expressed as: in, A U-coordinate classification vector representing a real gesture sample A U-coordinate classification vector of a synthetic gesture sample V-coordinate classification vector representing a real gesture sample A V-coordinate classification vector of a synthetic gesture sample

[0113] In the embodiments of this application, the U-coordinate classification vectors and V-coordinate classification vectors of real gesture samples and synthetic gesture samples, and their intra-class similarity and intra-class similarity in the classification task can be represented by an autocorrelation matrix.

[0114] For example, Figure 8A and Figure 8B The diagrams show the autocorrelation matrix. When the intra-class similarity between real and synthetic gesture samples is higher than the inter-class similarity between real and synthetic gesture samples, the values ​​in the autocorrelation matrix are mainly concentrated on the diagonal. However, when the intra-class similarity between real and synthetic gesture samples is lower than the inter-class similarity between real and synthetic gesture samples, the distribution of values ​​in the autocorrelation matrix is ​​irregular.

[0115] S3022_3: Determine the domain difference loss value between real gesture samples and synthetic gesture samples based on intra-class similarity and inter-class similarity.

[0116] In one example, a keypoint domain difference function is constructed between real and synthetic gesture samples based on the intra-class similarity between real and synthetic gesture samples, and the inter-class similarity between real and synthetic gesture samples. in, Indicates the intra-class similarity of the U-coordinate. V-coordinate represents intra-class similarity. The U-coordinate represents the inter-class similarity between real and synthetic gesture samples. The V-coordinate represents the inter-class similarity between real and synthetic gesture samples.

[0117] Furthermore, based on the keypoint domain difference function, the domain difference loss function between real gesture samples and synthetic gesture samples is obtained. DA The formula is expressed as follows:

[0118]

[0119] Where, N s N represents the number of real gesture samples. T x represents the number of synthesized gesture samples. s Let x represent the i-th real gesture sample. t Let j represent the j-th synthesized gesture sample. The domain difference loss value between the real gesture sample and the synthesized gesture sample can be calculated using Formula 2.

[0120] In the embodiments of this application, the subset of synthetic gesture samples containing complex hand poses generated based on the 3D gesture model can supplement the complex hand poses missing in the subset of real gesture samples, thereby enhancing the generalization ability of the trained model for complex hand poses with severe self-occlusion. At the same time, the key point domain adaptation technique is adopted to perform adversarial training by calculating the domain difference loss value between real gesture samples and synthetic gesture samples, so that the model converges quickly on real gesture samples and synthetic gesture samples, thereby alleviating the domain difference problem between real gesture samples and synthetic gesture samples, effectively learning the complex hand pose features in synthetic gesture samples, and improving the accuracy and robustness of 3D gesture detection.

[0121] S3023: Based on the heatmaps of positive and negative samples in multiple samples and the corresponding sample labels, determine the distribution loss value of positive samples, the distribution loss value of negative samples, and the distribution difference loss value of positive and negative samples respectively.

[0122] To mitigate the interference of background information on hand key points, a subset of background samples excluding hands is included in the 3D gesture sample set. Therefore, a negative sample regularization constraint term can be constructed based on the background samples for model training. By calculating the positive sample distribution loss, negative sample distribution loss, and positive-negative sample distribution difference loss, the model can fully learn the differences between background information and hand key points, further improving the accuracy and robustness of 3D gesture detection.

[0123] See Figure 9 The process of constructing negative sample regularization constraints for model training based on background samples mainly includes the following steps:

[0124] S3023_1: For each of the multiple samples, subtract the element value in each heatmap of the sample from the element value at the corresponding position in the label of the sample to obtain the element difference at the corresponding position.

[0125] Taking a negative sample as an example, as can be seen from the aforementioned embodiments, the label of the negative sample can be represented by a matrix as follows: The heatmap of the negative sample can also be represented as a k*k matrix. Therefore, the element difference can be obtained by subtracting the element values ​​at corresponding positions in the heatmap matrix of the negative sample predicted by the model from the element values ​​at corresponding positions in the label matrix of the negative sample.

[0126] Similarly, the labels of positive samples can also be represented by a k*k matrix. Therefore, the element differences between the heatmap matrix and the label matrix of positive samples can be obtained using the same method.

[0127] S3023_2: Determine the negative sample distribution loss value based on the element differences corresponding to each heatmap of the negative samples in multiple samples.

[0128] During model training, supervised learning can be performed on the heatmaps of negative samples predicted by the model based on the labels of the negative samples, enabling the model to learn the distribution characteristics of the heatmaps of negative samples. Specifically, the distribution loss value of negative samples can be determined based on the element differences corresponding to each heatmap of the negative samples.

[0129] In one example, a smoothing technique can be used to calculate the loss value of the negative sample distribution. hp The formula is expressed as follows:

[0130]

[0131] Where x represents the element difference between the heatmap matrix and the label matrix at the corresponding positions, and smooth1(·) represents the smoothing function.

[0132] S3023_3: Determine the positive sample distribution loss value based on the element differences corresponding to each heatmap in the positive samples among multiple samples.

[0133] During model training, supervised learning can be performed on the heatmaps of positive samples predicted by the model based on the labels of the positive samples, enabling the model to learn the distribution characteristics of the heatmaps of positive samples. Specifically, the distribution loss value of positive samples can be determined based on the element differences corresponding to each heatmap of the positive samples.

[0134] Optionally, the method for calculating the loss value of the positive sample distribution can be the same as or different from that for the loss value of the negative sample distribution. For example, a Gaussian function can also be used to calculate the loss value of the positive sample distribution.

[0135] This application embodiment uses negative samples as constraints for model training, enabling the model to fully learn the distribution differences of heatmaps of positive and negative samples, thereby reducing the interference of background information on key points of the hand.

[0136] S3023_4: Using KL divergence, the loss value of positive and negative sample distribution difference is determined based on the element values ​​in each heatmap of multiple samples and the element values ​​at the corresponding positions in the labels of the corresponding samples.

[0137] Based on supervised learning of the model using labels of negative samples, in order to better distinguish the distribution differences of heatmaps of positive and negative samples, regularization can be used to maximize the distribution differences of positive and negative samples.

[0138] In one example, KL divergence can be used to measure the difference between the heatmap distributions of positive and negative samples. Regularization can be achieved by constructing a loss function, expressed as:

[0139]

[0140] Among them, H s+t H is a heatmap of positive samples (including real gesture samples and synthetic gesture samples) from multiple samples. r h represents a heatmap of negative samples among multiple samples. p (x) represents an element value in the heatmap, h q (x) represents the element value at the corresponding position in the label of the corresponding sample, and KL(·) represents the KL divergence function. Since we need to maximize the difference in distribution between positive and negative samples, we need to consider the distribution difference term h. p (x) is negated to achieve the distribution difference KL(H) s+t H r During the training process, as the loss... reg The process of decreasing and increasing.

[0141] This application's embodiments construct regularization constraints using negative samples, effectively amplifying the distribution differences between background information and hand keypoint features, thereby improving the robustness of the heatmap distribution of hand keypoints. Furthermore, regularization maximizes the distribution differences between background information and hand keypoints, contributing to improved overall algorithm stability. In practical applications, combining this with a nonlinear optimization module can filter out false detections from the front end, improving the accuracy of 3D gesture detection.

[0142] S3024: Determine the target loss value for 3D gesture detection based on the domain difference loss value, positive sample distribution loss value, negative sample distribution loss value, and positive-negative sample distribution difference loss value, and adjust the model parameters according to the target loss value.

[0143] In one example, the formula for calculating the target loss value is as follows:

[0144] loss total =loss DA +losshp +loss hm +loss reg Formula 5

[0145] Where, loss DA This represents the domain difference loss value. hm This represents the loss value for a positive sample distribution. hp The loss value represents the negative sample distribution. reg The loss value represents the difference in distribution between positive and negative samples. total This represents the target loss value.

[0146] Unlike related technologies that improve the accuracy and robustness of 3D gesture detection by amplifying real gesture samples, the target gesture detection model provided in this application reduces the domain difference between real and synthetic gesture samples by adding a subset of synthetic gesture samples and employing keypoint domain adaptation technology. This allows the model to effectively learn complex hand posture features from synthetic gesture samples, thereby improving the generalization of complex hand postures with severe self-occlusion through synthetic gesture samples. Simultaneously, by adding a subset of background samples without hands and employing negative sample regularization technology, the difference between background information and hand keypoint features is increased, enhancing the robustness of the model's output heatmap to those containing hand keypoints. This improves the model's generalization to complex hand postures with severe self-occlusion at a lower cost and shortens the model's training time. This model can be used to assist in filtering falsely detected 3D gestures at the front end, solving the image jitter problem caused by complex hand postures with severe self-occlusion. Furthermore, the target gesture detection model provided in this application is a general optimization method for improving the stability of complex hand posture detection. It is not affected by specific model structure design; it only requires that the model's feature output head can output heatmaps, making it more applicable.

[0147] The target gesture detection model provided in this application can be deployed in head-mounted displays (such as AR glasses, VR glasses, etc.) to achieve 3D gesture interaction.

[0148] In practice, the head-mounted display uses its camera to acquire gesture images in real time during the gesture interaction process, and inputs the gesture images into the target gesture detection model. Based on the target gesture detection model, key points in the gesture images are extracted to obtain 3D gestures, and the display of virtual images is controlled based on the 3D gestures.

[0149] like Figure 10 As shown, this is a 3D gesture effect diagram detected by the target gesture detection model provided in the embodiment of this application. For hand grasping actions with occlusion, it can accurately detect key points of the hand, thereby improving the accuracy of gesture interaction.

[0150] Based on the same technical concept, this application provides an electronic device, which can be a terminal device, such as a laptop, desktop computer, head-mounted display, etc., or a server, such as a server cluster, cloud server, etc. The electronic device can implement the steps of the above-mentioned 3D gesture detection method and achieve the same technical effect.

[0151] See Figure 11 The electronic device includes a processor 1101, a memory 1102 and a communication interface 1103, which are connected via a bus 1104.

[0152] Memory 1102 may be volatile memory, such as random-access memory (RAM); memory 1102 may also be non-volatile memory, such as read-only memory, flash memory, hard disk drive (HDD), or solid-state drive (SSD); or memory 1102 may be any other medium capable of carrying or storing a desired computer program having the form of instructions or data structures and accessible by a computer, but is not limited thereto. Memory 1102 may be a combination of the above-described memories;

[0153] The processor 1101 may include one or more central processing units (CPUs) or digital processing units, etc.

[0154] Communication interface 1103 is used to communicate with other terminal devices and other servers;

[0155] The memory 1102 stores a computer program, and the processor 1101 performs the following operations according to the computer program:

[0156] Obtain a 3D gesture sample set; the 3D gesture sample set includes a background sample subset, a real gesture sample subset, and a synthetic gesture sample subset. The real gesture samples and synthetic gesture samples are used as positive samples, and the background samples are used as negative samples. The label of each negative sample follows a discrete uniform distribution, and the label of each positive sample follows a Gaussian normal distribution.

[0157] Based on the 3D gesture sample set, the 3D gesture detection model to be trained is trained at least once to obtain the target gesture detection model, wherein the following operations are performed in each iteration:

[0158] Feature extraction is performed on multiple samples in the 3D gesture sample set to obtain the heat map set of the corresponding samples;

[0159] Based on the heatmap sets of the real gesture samples and the synthetic gesture samples in the plurality of samples, the domain difference loss value between the real gesture samples and the synthetic gesture samples is determined;

[0160] Based on the heatmap sets of positive and negative samples in the multiple samples and the corresponding sample labels, the distribution loss value of positive samples, the distribution loss value of negative samples, and the distribution difference loss value of positive and negative samples are determined respectively.

[0161] Based on the domain difference loss value, the positive sample distribution loss value, the negative sample distribution loss value, and the positive and negative sample distribution difference loss value, the target loss value for 3D gesture detection is determined, and the model parameters are adjusted according to the target loss value.

[0162] Optionally, the processor 1101 determines the domain difference loss value between the real gesture sample and the synthetic gesture sample based on the respective heatmap sets of the real gesture sample and the synthetic gesture sample in the plurality of samples, specifically as follows:

[0163] For each positive sample in each real gesture sample and each synthetic gesture sample, each heatmap in the heatmap set of the positive sample is projected onto the U-axis and V-axis of the image coordinate system to obtain a U-coordinate classification vector and a V-coordinate classification vector.

[0164] Based on the U-coordinate classification vector and V-coordinate classification vector corresponding to each positive sample, the inter-class similarity and intra-class similarity of each positive sample are determined.

[0165] Based on the intra-class similarity and the inter-class similarity, the domain difference loss value between the real gesture sample and the synthetic gesture sample is determined.

[0166] Optionally, the intra-class similarity includes U-coordinate intra-class similarity and V-coordinate intra-class similarity, and the inter-class similarity includes U-coordinate inter-class similarity and V-coordinate inter-class similarity;

[0167] The processor 1101 determines the inter-class similarity and intra-class similarity of each positive sample based on the U-coordinate classification vector and V-coordinate classification vector corresponding to each positive sample. The specific operation is as follows:

[0168] Based on the U-coordinate classification vector corresponding to each of the real gesture samples, calculate the first U-coordinate intra-class similarity, and based on the V-coordinate classification vector of each of the real gesture samples, calculate the first V-coordinate intra-class similarity.

[0169] Based on the U-coordinate classification vector of each synthesized gesture sample, calculate the second U-coordinate intra-class similarity; and based on the V-coordinate classification vector of each synthesized gesture sample, calculate the second V-coordinate intra-class similarity.

[0170] Based on the U-coordinate classification vectors of each real gesture sample and each synthetic gesture sample, the inter-class similarity of the U-coordinate is calculated, and based on the V-coordinate classification vectors of each real gesture sample and each synthetic gesture sample, the intra-class similarity of the V-coordinate is calculated.

[0171] Optionally, the intra-class similarity includes intra-class similarity in U-coordinates and intra-class similarity in V-coordinates, and the inter-class similarity includes inter-class similarity in U-coordinates and inter-class similarity in V-coordinates. The formula for calculating the domain difference loss value is as follows:

[0172]

[0173] Among them, Dist DA (·) represents the keypoint domain difference function between the real gesture sample and the synthetic gesture sample. N s N represents the number of real gesture samples. T x represents the number of synthesized gesture samples. s Let x represent the i-th real gesture sample. t This represents the j-th synthesized gesture sample. Indicates the intra-class similarity of the U-coordinate. V-coordinate represents intra-class similarity. This represents the inter-class similarity of the U-coordinate between the real gesture sample and the synthetic gesture sample. The V-coordinate represents the inter-class similarity between the real gesture sample and the synthetic gesture sample.

[0174] Optionally, the processor 1101 determines the positive sample distribution loss value, the negative sample distribution loss value, and the positive-negative sample distribution difference loss value based on the heatmap sets of the positive and negative samples in the plurality of samples and the corresponding sample labels, respectively. Specifically, the operation is as follows:

[0175] For each of the plurality of samples, the element value in each heatmap of the sample is subtracted from the element value at the corresponding position in the label of the sample to obtain the element difference at the corresponding position.

[0176] The distribution loss value of the negative sample is determined based on the element differences corresponding to each heatmap of the negative sample among the multiple samples.

[0177] The distribution loss value of the positive samples is determined based on the element differences corresponding to each heatmap of the positive samples among the multiple samples.

[0178] Using KL divergence, the distribution difference loss value of the positive and negative samples is determined based on the element values ​​of each heatmap of the multiple samples and the element values ​​of the corresponding positions in the labels of the corresponding samples.

[0179] Optionally, the formula for the negative sample distribution loss value is expressed as:

[0180]

[0181] Where x represents the element difference, and smooth1(·) represents the smoothing function.

[0182] Optionally, the formula for the loss value of the positive and negative sample distribution differences is expressed as follows:

[0183]

[0184] Among them, H s+t H is a heatmap corresponding to each key point contained in the positive samples among the multiple samples. r h represents a heatmap of keypoints contained in a negative sample across multiple samples. p (x) represents an element value in the heatmap, h q (x) represents the element value at the corresponding position in the label, and KL(·) represents the KL divergence function.

[0185] Optionally, after obtaining the target gesture detection model, the processor 1101 further executes:

[0186] Acquire gesture images during the gesture interaction process and input the gesture images into the target gesture detection model;

[0187] Based on the target gesture detection model, key points are extracted from the gesture image to obtain a 3D gesture.

[0188] The display of the virtual screen is controlled based on the 3D gestures.

[0189] It should be noted that, Figure 11 This is merely an example illustrating the hardware necessary for an electronic device to perform the steps of a 3D gesture detection method provided in this application embodiment. Not shown, the electronic device may also include conventional hardware such as a speaker, microphone, display, and power supply.

[0190] Examples of this application Figure 11The processor involved can be a central processing unit (CPU), a general-purpose processor, a graphics processing unit (GPU), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof.

[0191] like Figure 12 As shown, an example of an electronic device is a head-mounted display. The head-mounted display includes a processor 1201, a memory 1202, a display 1203, and a camera 1204. The camera 1204, the display 1203, the memory 1202, and the processor 1201 are connected via a bus 1205. A target gesture detection model is deployed in the head-mounted display.

[0192] Camera 1204 is used to acquire gesture images during gesture interaction;

[0193] The memory 1202 stores a computer program, and the processor 1201 performs the following operations according to the computer program:

[0194] The gesture image acquired by camera 1204 is input into the target gesture detection model;

[0195] Based on the target gesture detection model, key points are extracted from the gesture image to obtain 3D gestures;

[0196] Control the virtual screen displayed on monitor 1203 based on 3D gestures.

[0197] This application also provides a computer-readable storage medium for storing instructions that, when executed, can complete a method for generating a 3D gesture dataset as described in the foregoing embodiments.

[0198] This application also provides a computer program product for storing a computer program for executing a method for generating a 3D gesture dataset as described in the foregoing embodiments.

[0199] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0200] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0201] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0202] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0203] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A 3D gesture detection method, characterized in that, include: Obtain a 3D gesture sample set; The 3D gesture sample set includes a background sample subset, a real gesture sample subset, and a synthetic gesture sample subset. Real gesture samples and synthetic gesture samples are used as positive samples, and background samples are used as negative samples. The label of each negative sample follows a discrete uniform distribution, and the label of each positive sample follows a Gaussian normal distribution. Based on the 3D gesture sample set, the 3D gesture detection model to be trained is trained at least once to obtain the target gesture detection model, wherein the following operations are performed in each iteration: Feature extraction is performed on multiple samples in the 3D gesture sample set to obtain the heat map set of the corresponding samples; For each positive sample in each real gesture sample and each synthetic gesture sample, each heatmap in the heatmap set of the positive sample is projected onto the U-axis and V-axis of the image coordinate system to obtain a U-coordinate classification vector and a V-coordinate classification vector. Based on the U-coordinate classification vector and V-coordinate classification vector corresponding to each positive sample, the inter-class similarity and intra-class similarity of each positive sample are determined. The intra-class similarity includes U-coordinate intra-class similarity and V-coordinate intra-class similarity, and the inter-class similarity includes U-coordinate inter-class similarity and V-coordinate inter-class similarity. The formula for calculating the domain difference loss value is as follows: in, This represents the keypoint domain difference function between the real gesture sample and the synthetic gesture sample. , This indicates the number of real gesture samples. This indicates the number of synthesized gesture samples. This represents the i-th real gesture sample. This represents the j-th synthesized gesture sample. Indicates the intra-class similarity of the U-coordinate. V-coordinate represents intra-class similarity. This represents the inter-class similarity of the U-coordinate between the real gesture sample and the synthetic gesture sample. This represents the inter-class similarity value of the V-coordinate between the real gesture sample and the synthetic gesture sample; Based on the heatmaps of the positive and negative samples from the multiple samples and their corresponding labels, the distribution loss value for positive samples, the distribution loss value for negative samples, and the distribution difference loss value for positive and negative samples are determined respectively. The distribution difference loss value for positive and negative samples is obtained by using KL divergence, based on the element values ​​of each heatmap of the multiple samples and the element values ​​at the corresponding positions in the labels of the corresponding samples. Based on the domain difference loss value, the positive sample distribution loss value, the negative sample distribution loss value, and the positive and negative sample distribution difference loss value, the target loss value for 3D gesture detection is determined, and the model parameters are adjusted according to the target loss value.

2. The method as described in claim 1, characterized in that, The intra-class similarity includes U-coordinate intra-class similarity and V-coordinate intra-class similarity, and the inter-class similarity includes U-coordinate inter-class similarity and V-coordinate inter-class similarity; The step of determining the inter-class similarity and intra-class similarity of each positive sample based on its corresponding U-coordinate classification vector and V-coordinate classification vector includes: Based on the U-coordinate classification vector corresponding to each of the real gesture samples, calculate the first U-coordinate intra-class similarity, and based on the V-coordinate classification vector of each of the real gesture samples, calculate the first V-coordinate intra-class similarity. Based on the U-coordinate classification vector of each synthesized gesture sample, calculate the second U-coordinate intra-class similarity; and based on the V-coordinate classification vector of each synthesized gesture sample, calculate the second V-coordinate intra-class similarity. Based on the U-coordinate classification vectors of each real gesture sample and each synthetic gesture sample, the inter-class similarity of the U-coordinate is calculated, and based on the V-coordinate classification vectors of each real gesture sample and each synthetic gesture sample, the intra-class similarity of the V-coordinate is calculated.

3. The method as described in claim 1, characterized in that, The step of determining the positive sample distribution loss value, the negative sample distribution loss value, and the positive-negative sample distribution difference loss value based on the heatmap sets of positive and negative samples and the corresponding sample labels in the plurality of samples includes: For each of the plurality of samples, the element value in each heatmap of the sample is subtracted from the element value at the corresponding position in the label of the sample to obtain the element difference at the corresponding position. The distribution loss value of the negative sample is determined based on the element differences corresponding to each heatmap of the negative sample among the multiple samples. The distribution loss value of the positive samples is determined based on the element differences corresponding to each heatmap of the positive samples among the multiple samples. Using KL divergence, the distribution difference loss value of the positive and negative samples is determined based on the element values ​​of each heatmap of the multiple samples and the element values ​​of the corresponding positions in the labels of the corresponding samples.

4. The method as described in claim 3, characterized in that, The formula for the negative sample distribution loss value is expressed as follows: in, This represents the difference between the elements. This represents a smoothing function.

5. The method as described in claim 3, characterized in that, The formula for the loss value of the difference between the positive and negative sample distributions is expressed as follows: in, This is a heatmap corresponding to each key point contained in the positive samples among the multiple samples. This represents a heatmap showing the key points corresponding to each point in a negative sample across multiple samples. This represents the value of an element in a heatmap. This represents the value of the element at the corresponding position within the tag. This represents the KL divergence function.

6. The method according to any one of claims 1-5, characterized in that, After obtaining the target gesture detection model, the method further includes: Acquire gesture images during the gesture interaction process and input the gesture images into the target gesture detection model; Based on the target gesture detection model, key points are extracted from the gesture image to obtain a 3D gesture. The display of the virtual screen is controlled based on the 3D gestures.

7. An electronic device, characterized in that, It includes a processor, a memory, and a communication interface, wherein the communication interface, the memory, and the processor are connected via a bus; The communication interface is used to communicate with terminal devices and servers; The memory stores a computer program, and the processor performs the following operations according to the computer program: Obtain a 3D gesture sample set; the 3D gesture sample set includes a background sample subset, a real gesture sample subset, and a synthetic gesture sample subset. The real gesture samples and synthetic gesture samples are used as positive samples, and the background samples are used as negative samples. The label of each negative sample follows a discrete uniform distribution, and the label of each positive sample follows a Gaussian normal distribution. Based on the 3D gesture sample set, the 3D gesture detection model to be trained is trained at least once to obtain the target gesture detection model, wherein the following operations are performed in each iteration: Feature extraction is performed on multiple samples in the 3D gesture sample set to obtain the heat map set of the corresponding samples; For each positive sample in each real gesture sample and each synthetic gesture sample, each heatmap in the heatmap set of the positive sample is projected onto the U-axis and V-axis of the image coordinate system to obtain a U-coordinate classification vector and a V-coordinate classification vector. Based on the U-coordinate classification vector and V-coordinate classification vector corresponding to each positive sample, the inter-class similarity and intra-class similarity of each positive sample are determined. The intra-class similarity includes U-coordinate intra-class similarity and V-coordinate intra-class similarity, and the inter-class similarity includes U-coordinate inter-class similarity and V-coordinate inter-class similarity. The formula for calculating the domain difference loss value is as follows: in, This represents the keypoint domain difference function between the real gesture sample and the synthetic gesture sample. , This indicates the number of real gesture samples. This indicates the number of synthesized gesture samples. This represents the i-th real gesture sample. This represents the j-th synthesized gesture sample. Indicates the intra-class similarity of the U-coordinate. V-coordinate represents intra-class similarity. This represents the inter-class similarity of the U-coordinate between the real gesture sample and the synthetic gesture sample. This represents the inter-class similarity value of the V-coordinate between the real gesture sample and the synthetic gesture sample; Based on the heatmaps of the positive and negative samples from the multiple samples and their corresponding labels, the distribution loss value for positive samples, the distribution loss value for negative samples, and the distribution difference loss value for positive and negative samples are determined respectively. The distribution difference loss value for positive and negative samples is obtained by using KL divergence, based on the element values ​​of each heatmap of the multiple samples and the element values ​​at the corresponding positions in the labels of the corresponding samples. Based on the domain difference loss value, the positive sample distribution loss value, the negative sample distribution loss value, and the positive and negative sample distribution difference loss value, the target loss value for 3D gesture detection is determined, and the model parameters are adjusted according to the target loss value.

8. A head-mounted display, characterized in that, The head-mounted display is equipped with a target gesture detection model as described in any one of claims 1-6, the head-mounted display including a processor, a memory, a display and a camera, the camera, the display, the memory and the processor being connected via a bus; The camera is used to acquire gesture images during the gesture interaction process; The memory stores a computer program, and the processor performs the following operations according to the computer program: The gesture image acquired by the camera is input into the target gesture detection model; Based on the target gesture detection model, key points are extracted from the gesture image to obtain a 3D gesture. The virtual images displayed on the monitor are controlled based on the 3D gestures.

Images