Digital twin WEB application air gesture interaction method and system

Abstract

The invention belongs to the technical field of human-computer interaction, and discloses a digital twin WEB application air gesture interaction method and system. Comprising the following steps: collecting a real-time video stream of a camera of user side equipment, and preprocessing the real-time video stream; x hand region frames are obtained; inputting the X hand region frames into a preset hand key point detection model based on a generative adversarial network to obtain hand key point coordinate data; acquiring a hand skeleton model based on the hand key point coordinate data; acquiring a joint angle and a hand motion track according to the hand skeleton model; mapping the hand skeleton model into a digital twin virtual scene, presetting n gesture instruction modes in the digital twin virtual scene, and matching the hand skeleton model, the joint angle and the hand motion track with the gesture instruction modes; personalized and extensible gesture interaction capability is provided, and man-machine interaction modes are enriched.

Description

Technical Field

[0001] The present invention relates to the field of human-computer interaction technology, and more specifically, to a digital twin WEB application air gesture interaction method and system. Background Art

[0002] The patent with application publication number CN112394811A discloses an air gesture interaction method and electronic device, which relate to the fields of terminal technology and AI technology, such as the field of AI image recognition, and can improve the user experience when interacting with electronic devices through the air and reduce the error rate of air gestures; including: the electronic device displays a first display interface of a first application; the electronic device detects through a camera that a user inputs a first air gesture; in response to the first air gesture, the electronic device moves an indication icon in the first display interface along the movement trajectory of the first air gesture in a preset direction; in response to the first air gesture, the electronic device displays a second display interface of the first application.

[0003] However, traditional human-computer interaction methods, such as keyboards, mice and touch screens, lack naturalness and convenience, and cannot achieve an interactive experience anytime and anywhere; users need to directly contact hardware devices, and the interaction method is single, which brings many inconveniences to users; existing gesture recognition technology has problems such as low accuracy and poor robustness, and it is difficult to adapt to complex application scenarios; the accuracy of hand key point detection and gesture classification in gesture recognition needs to be improved; and there are certain difficulties in accurately detecting the position of hand key points and distinguishing similar gestures, which affects the overall recognition effect; existing gesture recognition technology lacks temporal continuity analysis of dynamic changes of gestures, and cannot well characterize and capture the dynamic process of gestures, resulting in limited recognition capabilities; lack of personalization and scalability, single interaction method, and cannot meet the diverse needs of different users and application scenarios.

[0004] In view of this, the present invention proposes a digital twin WEB application air gesture interaction method and system to solve the above problems. Summary of the invention

[0005] In order to overcome the above-mentioned defects of the prior art and to achieve the above-mentioned purpose, the present invention provides the following technical solutions: A digital twin WEB application remote gesture interaction method, comprising: S1, collecting the real-time video stream of the camera of the user terminal device, and preprocessing the real-time video stream; obtaining X hand area frames;

[0006] S2, inputting X hand region frames into a preset hand key point detection model based on a generative adversarial network to obtain hand key point coordinate data;

[0007] S3, obtaining a hand skeleton model based on the hand key point coordinate data; obtaining joint angles and hand motion trajectory according to the hand skeleton model;

[0008] S4. Map the hand skeleton model to the digital twin virtual scene, preset n gesture command modes in the digital twin virtual scene, and match the hand skeleton model, joint angles, and hand motion trajectories with the gesture command modes.

[0009] Furthermore, the method of preprocessing the real-time video stream includes:

[0010] Extract image frames frame by frame from the real-time video stream; input the image frames into the pre-trained hand detection model to obtain the bounding box coordinates of the hand; crop the hand region box containing only the hand from the original image frame according to the bounding box coordinates; and use the ROI adjustment algorithm to adjust the size and position of the hand region box in real time;

[0011] The training method of the hand detection model includes:

[0012] Initially define the basic framework of the hand detection model, which includes the backbone network, feature pyramid network, detection head network and anchor box;

[0013] The ResNet-101 network is used as the backbone network to output feature maps. The ResNet-101 network consists of b stages, each of which consists of m residual blocks. A feature pyramid network is constructed based on the feature maps output by the backbone network.

[0014] Based on the feature pyramid network, an independent detection head sub-network is designed for each level of the pyramid feature map; the detection head includes 4 convolutional layers and 2 parallel fully connected layers; the 2 parallel fully connected layers include a classification fully connected layer and a regression fully connected layer;

[0015] On each level of the pyramid feature map, 9 predefined anchor boxes are set for each pixel position; the size of the anchor box is defined as {32 2 , 64 2 , 128 2 , 256 2 , 512 2} pixels; and the aspect ratio is {1:2, 1:1, 2:1};

[0016] The loss function of the hand detection model is defined as a classification loss function and a regression loss function.

[0017] Classification loss function FL(p c )=-(1-p c ) γ ·log(p c ), where γ is the regulatory factor;

[0018] Regression loss function

[0019] Collect k images of hands to form an image dataset; annotate the images in the image dataset and generate labels, where the labels are the bounding box coordinates of the hand in each image, i.e., the annotation box;

[0020] The image dataset is divided into U batches; the batches are input into the hand detection model, and the values ​​of the classification loss function and the regression loss function after each batch is input are counted, and the parameters in the hand detection model are continuously adjusted; the values ​​of the classification loss function and the regression loss function no longer decrease in Y consecutive batches, and the training is stopped; a trained hand detection model is obtained.

[0021] Furthermore, the formula of the output feature map is: y = F(x, W i )+x; where y is the feature map output by the residual block; x is the input image frame; F(x,W i ) is the residual function; W i is the learnable parameter of the i-th residual block;

[0022] F(x,W i )=ReLU(Conv(x,W i ));Wherein, Conv represents the convolution operation; ReLU is the rectified linear unit activation function;

[0023] The construction method of the feature pyramid network includes:

[0024] By constructing horizontally connected and vertically connected paths, a pyramid feature map with 7 levels is generated;

[0025] The formula for lateral connection is: P b =Conv(C b ,256)+Upsample(P b+1 ), where P b The b-th level pyramid feature map; C b The feature map output by the b-th stage of the backbone network; b is a positive integer greater than or equal to 1 and less than or equal to 7; Conv(C b ,256) means C b Perform a 1×1 convolution operation to change the number of channels to 256; Upsample (P b+1 ) represents the upsampling operation, P b+1 The size is upsampled to the same size as P b Same size; P b+1 is the b+1th level pyramid feature map;

[0026] The formula for vertical connection is: Among them, P6 is the sixth level pyramid feature map; P7 is the seventh level pyramid feature map; the feature map output by the fifth stage of the C5 backbone network; Maxpool (P6) means the maximum pooling operation on P6;

[0027] The classification fully connected layer uses a Softmax activation function to calculate the probability that each anchor box belongs to the hand or the background;

[0028] The probability is calculated as: Among them, W c T is the weight parameter W of the classification fully connected layer c The transpose of is the feature vector from the feature pyramid network; B c is the bias parameter of the classification fully connected layer; p c The probability of each predicted anchor box belonging to the hand or the background;

[0029] The regression fully connected layer has no activation function and is used to obtain the offset fu from the predicted anchor box to the annotation box.

[0030] Furthermore, the method of adjusting the size and position of the hand area frame in real time includes:

[0031] Using Kalman filtering or motion estimation algorithm, the position of the hand in the continuous image frames is tracked; the displacement and movement speed of the hand in the coordinate system of the image frame are obtained; the displacement includes vertical displacement and horizontal displacement; the movement speed includes horizontal movement speed and vertical movement speed;

[0032] Set the size of the initial hand region box to W0×H0. The size of the initial hand region box surrounds the detected hand bounding box and leaves a fixed margin.

[0033] Adjust the size of the hand area box according to the movement speed;

[0034] The formula for adjusting the size of the hand area box is: Among them, α is the adjustment coefficient; W t H is the width of the hand region frame that changes according to the time frame t; t is the height of the hand region box that changes according to the time frame t; v h is the horizontal movement speed of the hand in the coordinate system of the image frame; v z is the vertical movement speed of the hand in the coordinate system of the image frame;

[0035] Adjust the center coordinates of the hand area frame according to the displacement;

[0036] The formula for adjusting the center coordinates of the hand area box is: Among them, h t z is the horizontal coordinate of the center coordinate of the hand area frame according to the time frame t; t is the ordinate of the center coordinate of the hand region frame according to the change of time frame t; h0 is the abscissa of the hand region frame of the previous time frame; z0 is the ordinate of the hand region frame of the previous time frame; Δh is the horizontal displacement; Δz is the vertical displacement;

[0037] Set the maximum size threshold and the minimum size threshold of the hand area frame; if the size of the adjusted hand area frame exceeds the range of the maximum size threshold and the minimum size threshold, it will be cropped to the maximum size threshold or the minimum size threshold;

[0038] Use the adjusted hand region box size (W t ,H t ) and the center coordinates (h t ,z t ) as the initial hand region frame of the next image frame.

[0039] Furthermore, the preset method of the hand key point detection model includes:

[0040] The basic structure of the hand key point detection model is defined to include the generator network structure and the discriminator network structure; and adversarial training is performed on the generator network structure and the discriminator network structure;

[0041] The generator network structure adopts an encoder-decoder structure. The input is the image of the hand region frame, and the output is the generated hand key point heat map y_fake. The encoder consists of a convolutional layer and a downsampling layer to extract image features. The decoder consists of an upsampling layer and a convolutional layer to generate a hand key point heat map.

[0042] The goal of the generator network structure is to minimize the difference LC between the generated hand key point heat map y_fake and the real heat map y_real. The difference is calculated using the L1 or L2 loss at the pixel level.

[0043] The input of the discriminator network structure is the hand key point heat map y_fake or the real heat map y_real generated by the generator, and the output is the probability value of the hand key point heat map y_fake or the real heat map y_real being real or generated;

[0044] The discriminator network structure adopts a convolutional neural network structure to extract features and classify the input hand key point heat map y_fake or real heat map y_real;

[0045] The goal of the discriminator network structure is to maximize the probability that the real heat map is judged as real, and the probability that the generated hand key point heat map is judged as generated; that is, the binary loss function LD that defines the discriminator network structure;

[0046] in, is the negative logarithmic expectation of the probability that the real heat map y_real is correctly judged as real; D y_real Indicates the probability that the discriminator network structure believes that y_real is the real heat map;

[0047] is the negative logarithmic expectation of the probability that the generated hand key point heat map y_fake is correctly judged as generated; D y_fake Indicates the probability that the discriminator thinks y_fake is a real heat map;

[0048] Initialize the parameters of the generator network structure and the discriminator network structure; collect a training data set containing hand images and corresponding real heat maps; for the training data set, shuffle the training data set and divide it into R batches; sample by batch; for each batch, sample the real heat map from the batch; input the current generator network structure into the hand image to generate y_fake; input the real heat map and y_fake into the discriminator network structure; and calculate the binary loss function LD of the discriminator network structure;

[0049] Sample hand images from the batch; input the hand images into the generator network structure to generate y_fake; input y_fake into the current discriminator network structure; calculate the LC of the generator network structure; iteratively update the parameters of the generator network structure and the discriminator network structure and meet the training objectives; that is, complete the preset of the hand key point detection model.

[0050] Furthermore, the method of performing adversarial training includes:

[0051] The training goal is to find a Nash equilibrium point, and update the parameters of the generator network structure and the discriminator network structure; the Nash equilibrium point is to minimize the difference between the generated hand key point heat map y_fake and the real heat map y_real while maximizing LD; during training, the generator network structure and the discriminator network structure are alternately optimized to update their respective network parameters;

[0052] The method of acquiring the hand motion trajectory and the hand three-dimensional model data includes:

[0053] The three-dimensional coordinates of the key points of the hand are calculated by combining the coordinate data of the key points of the hand with the intrinsic parameters of the camera of the user end device and the known hand size constraints;

[0054] Based on the three-dimensional coordinates of the key points of the hand, a hand skeleton model is constructed; the hand skeleton model consists of J skeleton chain nodes, each skeleton chain node corresponds to a key point; the skeleton chain nodes are connected by virtual bones;

[0055] The method of obtaining the joint angle includes:

[0056] For each set of two adjacent bone chain nodes, obtain the two bone vectors they are connected to; use vector dot multiplication to calculate the angle between the two bone vectors, i.e. the joint angle; the joint angle describes the posture and movement of the hand;

[0057] The method of obtaining the hand motion trajectory includes:

[0058] For each key point in the hand skeleton model, a motion estimation algorithm is used based on continuous image frames to obtain a three-dimensional motion trajectory, namely, the hand motion trajectory.

[0059] Furthermore, the method of calculating the three-dimensional coordinates of the key points of the hand includes:

[0060] Construct an intrinsic parameter matrix K based on the intrinsic parameters of the camera of the user end device;

[0061] Among them: fh is the horizontal focal length of the camera of the user end device; fz is the vertical focal length of the camera of the user end device; ch is the horizontal coordinate of the principal point of the camera of the user end device; cz is the vertical coordinate of the principal point of the camera of the user end device;

[0062] The hand key point coordinate data is used to remove the scale factor using the parameter matrix K to obtain the normalized image plane coordinates;

[0063] The formula for removing the scale factor is: Among them, hk j zk is the horizontal coordinate of the jth hand key point image plane coordinate; j is the ordinate of the jth hand key point image plane coordinate; uk j vk is the horizontal coordinate of the jth hand key point in the hand key point heat map; j is the ordinate of the jth hand key point in the hand key point heat map;

[0064] Given the distance constraints of the hand key points in three-dimensional space, a nonlinear least squares optimization problem is constructed to calculate the vertical coordinates of each key point.

[0065] The formula for calculating the vertical coordinate of the ruler is: Among them, d jl is the known distance constraint between the jth and lth hand key points; Zh jZh is the vertical coordinate of the jth hand key point; l is the vertical coordinate of the ruler of the lth hand key point;

[0066] And project the image plane coordinates according to the vertical coordinates of the ruler to obtain the three-dimensional coordinates of the key points of the hand;

[0067] The projection formula is: Among them, (Hk j ,Zk j ,Ck j ) is the three-dimensional coordinate of the hand key point of the jth hand key point.

[0068] Furthermore, the preset methods of the n gesture command modes include:

[0069] Collect gesture data, which comes from real user gesture capture or is generated by motion capture equipment or computer animation modeling; gesture data includes information about the hand skeleton model, joint angles, and hand motion trajectory;

[0070] Manually annotate the collected gesture data, and identify the gesture instructions corresponding to each gesture data; the gesture data is a continuous time data sequence; separate the sequence composed of each gesture action from the gesture data; and align the separated sequences at the same time;

[0071] The sequence of each gesture action is clustered to obtain n categories; for each category, a gesture instruction pattern is defined including gesture instructions, hand skeleton model, joint angles and hand motion trajectory.

[0072] Furthermore, the method of matching the hand skeleton model, joint angles and hand motion trajectory with the gesture instruction pattern includes:

[0073] The hand skeleton model, hand motion trajectory and joint angle are converted into a multi-dimensional feature sequence as a sequence to be matched; each time step of the multi-dimensional feature sequence corresponds to a feature vector;

[0074] For each preset gesture command pattern, its corresponding hand skeleton model, hand motion trajectory and joint angle are also converted into a multi-dimensional feature sequence as a gesture template sequence;

[0075] By calculating the time warping distance between the multi-dimensional feature sequence and the gesture command pattern, the best matching gesture command pattern is found;

[0076] The time warp distance is calculated by:

[0077] Calculate the local distance matrix between the sequence to be matched and each gesture template sequence; the size of the local distance matrix is ​​M×N, where M is the length of the sequence to be matched and N is the length of the gesture template sequence; each element D(λ,β) in the local distance matrix represents the distance between the sequence to be matched at time step λ and the gesture template sequence at time step β;

[0078] Based on the local distance matrix, a cumulative distance matrix is ​​calculated. Each element D'(λ,β) in the cumulative distance matrix represents the minimum cumulative distance from time step 0 to the sequence to be matched at time step λ and the gesture template sequence at time step β.

[0079] D'(λ,β)=D(λ,β)+min(D'(λ-1,β),D'(λ,β-1),D'(λ-1,β-1));

[0080] Among them, D'(λ-1,β) represents the minimum cumulative distance from time step 0 to the sequence to be matched at time step λ-1 and the gesture template sequence at time step β; D'(λ,β-1) represents the minimum cumulative distance from time step 0 to the sequence to be matched at time step λ and the gesture template sequence at time step β-1; D'(λ-1,β-1) represents the minimum cumulative distance from time step 0 to the sequence to be matched at time step λ-1 and the gesture template sequence at time step β-1;

[0081] Starting from the lower right corner element of the cumulative distance matrix, backtrack along the direction of the minimum cumulative distance to find an optimal path from time step 0 to the lower right corner element;

[0082] The sum of the minimum cumulative distances of all elements on the optimal path is obtained, which is the dynamic time warping distance between the sequence to be matched and the gesture template sequence; the dynamic time warping distance between the sequence to be matched and all gesture template sequences is calculated; the gesture instruction pattern corresponding to the gesture template sequence with the smallest dynamic time warping distance is selected as the best matching gesture instruction pattern; the matching is completed.

[0083] A digital twin WEB application air gesture interaction system, used to implement a digital twin WEB application air gesture interaction method, comprising: a data acquisition and processing module, used to acquire a real-time video stream from a camera of a user terminal device, and pre-process the real-time video stream; obtain X hand area frames;

[0084] A key point detection module is used to input X hand region frames into a preset hand key point detection model based on a generative adversarial network to obtain hand key point coordinate data;

[0085] The motion detection module obtains the hand skeleton model based on the coordinate data of the key points of the hand; obtains the joint angles and hand motion trajectory according to the hand skeleton model;

[0086] The best matching module is used to map the hand skeleton model to the digital twin virtual scene, preset n gesture command modes in the digital twin virtual scene, and match the hand skeleton model, joint angles and hand motion trajectory with the gesture command mode; each module is connected by wired and / or wireless means.

[0087] The technical effects and advantages of the digital twin WEB application air gesture interaction method and system of the present invention are as follows:

[0088] The present invention realizes natural and convenient human-computer interaction; adopts generative adversarial network to realize high-precision hand key point detection, improves the accuracy and robustness of gesture recognition; can adapt to different gestures and complex application scenarios; by adaptively adjusting the hand area frame and combining the ROI adjustment algorithm, the accuracy of hand detection and key point positioning is improved; the recognition difficulty caused by hand movement is reduced; a gesture recognition method based on hand skeleton model and time warp distance is proposed, and the time continuity analysis of gesture is introduced; by performing nonlinear alignment on the time axis, the dynamic changes of gestures can be better characterized and the accuracy of gesture classification can be improved; multiple gesture command modes are preset and dynamic updates are supported; personalized and scalable gesture interaction capabilities are provided, enriching the human-computer interaction mode; gesture interaction is combined with digital twin WEB applications to expand the application scenarios of gesture interaction. BRIEF DESCRIPTION OF THE DRAWINGS

[0089] Figure 1 A schematic diagram of a digital twin WEB application air gesture interaction method of the present invention;

[0090] Figure 2 This is a schematic diagram of a digital twin WEB application air gesture interaction system of the present invention. DETAILED DESCRIPTION

[0091] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0092] Example 1

[0093] See also Figure 1 As shown, the digital twin WEB application air gesture interaction method described in this embodiment includes:

[0094] S1, collecting the real-time video stream of the camera of the user terminal device, and pre-processing the real-time video stream; obtaining X hand area frames;

[0095] S2, inputting X hand region frames into a preset hand key point detection model based on a generative adversarial network to obtain hand key point coordinate data;

[0096] S3, obtaining a hand skeleton model based on the hand key point coordinate data; obtaining joint angles and hand motion trajectory according to the hand skeleton model;

[0097] S4. Map the hand skeleton model to the digital twin virtual scene, preset n gesture command modes in the digital twin virtual scene, and match the hand skeleton model, joint angles, and hand motion trajectories with the gesture command modes.

[0098] Furthermore, the method of preprocessing the real-time video stream includes:

[0099] Extract image frames frame by frame from the real-time video stream; input the image frames into a pre-trained hand detection model to obtain the bounding box coordinates of the hand; and crop a hand region box containing only the hand from the original image frame according to the bounding box coordinates;

[0100] The ROI adjustment algorithm is used to adjust the size and position of the hand area box in real time; when the hand moves, the ROI adjustment algorithm can adaptively expand or shrink the hand area box to ensure that the hand is completely included.

[0101] It should be explained that the hand region frame is cropped from the original image frame and is essentially also an image.

[0102] The training method of the hand detection model includes:

[0103] Initially define the basic framework of the hand detection model, which includes the backbone network, feature pyramid network, detection head network and anchor box;

[0104] The ResNet-101 network is used as the backbone network to output feature maps; the ResNet-101 network consists of b stages, each of which consists of m residual blocks.

[0105] The formula of the output feature map is: y = F(x, W i )+x; where y is the feature map output by the residual block; x is the input image frame; F(x,W i ) is the residual function; W i is the learnable parameter of the i-th residual block; F(x,W i )=ReLU(Conv(x,W i)); where Conv represents the convolution operation; ReLU is the rectified linear unit activation function; specifically, the residual function adds the result of the convolution operation on x to x, and then passes it through ReLU;

[0106] Construct a feature pyramid network based on the feature map output by the backbone network;

[0107] The construction method of the feature pyramid network includes:

[0108] By constructing horizontal and vertical connection paths, feature maps of different scales are fused to generate a pyramid feature map with 7 levels;

[0109] The formula for lateral connection is: P b =Conv(C b ,256)+Upsample(P b+1 ), where P b The b-th level pyramid feature map; C b The feature map output by the b-th stage of the backbone network; b is a positive integer greater than or equal to 1 and less than or equal to 7; Conv(C b ,256) means C b Perform a 1×1 convolution operation to change the number of channels to 256; Upsample (P b+1 ) represents the upsampling operation, P b+1 The size is upsampled to the same size as P b Same size; P b+1 is the b+1th level pyramid feature map;

[0110] The formula for vertical connection is: Among them, P6 is the sixth-level pyramid feature map; P7 is the seventh-level pyramid feature map; the feature map output by the fifth stage of the C5 backbone network; Maxpool (P6) means the maximum pooling operation on P6; building vertical connections from P6 and P7 can enable the model to better handle small-sized targets, while larger targets can be processed in horizontal connections through lower-level features; it can more comprehensively cover targets of various scales and improve the performance of the detection model.

[0111] Based on the feature pyramid network, an independent detection head sub-network is designed for each level of the pyramid feature map; the detection head includes 4 convolutional layers and 2 parallel fully connected layers; the 2 parallel fully connected layers include a classification fully connected layer and a regression fully connected layer;

[0112] The classification fully connected layer uses the Softmax activation function to calculate the probability that each anchor box belongs to the hand or the background;

[0113] The probability is calculated as: Among them, W c T is the weight parameter W of the classification fully connected layer c The transpose of is the feature vector from the feature pyramid network; B c is the bias parameter of the classification fully connected layer; p c The probability of each predicted anchor box belonging to the hand or the background;

[0114] The regression fully connected layer has no activation function and is used to obtain the offset fu from the predicted anchor box to the annotation box.

[0115] On each level of the pyramid feature map, 9 predefined anchor boxes are set for each pixel position; the size of the anchor box is defined as {32 2 , 64 2 , 128 2 , 256 2 , 512 2} pixels; and the aspect ratio is {1:2, 1:1, 2:1};

[0116] The loss function of the hand detection model is defined as a classification loss function and a regression loss function.

[0117] Classification loss function FL(p c )=-(1-p c ) γ ·log(p c ); where γ is the adjustment factor; it is used to balance the weights of positive and negative samples; in most cases, negative samples (anchor boxes that do not contain the hand) are far more than positive samples (anchor boxes that contain the hand), directly calculating the classification loss may cause the model to be biased towards predicting negative samples, thereby affecting the accuracy of hand detection; by setting a larger adjustment factor value (usually between 0.5-0.9), the weight of positive samples can be increased, so that the model pays more attention to positive samples and improves the performance of hand detection.

[0118] Regression loss function

[0119] Collect k images of hands to form an image dataset, ensuring that k is large enough and the images of hands are diverse; annotate the images in the image dataset and generate labels, which are the bounding box coordinates of the hand in each image, i.e., the annotation box;

[0120] The image dataset is divided into U batches; the batches are input into the hand detection model, and the values ​​of the classification loss function and the regression loss function after each batch is input are counted, and the parameters in the hand detection model are continuously adjusted, such as using an optimizer, SGD or Adam, etc.; the values ​​of the classification loss function and the regression loss function no longer decrease in Y consecutive batches, and the training is stopped; and a trained hand detection model is obtained.

[0121] The method of adjusting the size and position of the hand area frame in real time includes:

[0122] The position of the hand in continuous image frames is tracked by using Kalman filtering or motion estimation algorithm; the displacement and movement speed of the hand in the coordinate system of the image frame are obtained; the displacement includes vertical displacement and horizontal displacement; the movement speed includes horizontal movement speed and vertical movement speed.

[0123] Set the size of the initial hand region box to W0×H0. The size of the initial hand region box surrounds the detected hand bounding box and leaves a fixed margin.

[0124] Adjust the size of the hand area box according to the movement speed;

[0125] The formula for adjusting the size of the hand area box is: Among them, α is the adjustment coefficient, which is used to control the change rate of the hand area frame size; W t H is the width of the hand region frame that changes according to the time frame t; t is the height of the hand region box that changes according to the time frame t; v h is the horizontal movement speed of the hand in the coordinate system of the image frame; v z is the vertical movement speed of the hand in the coordinate system of the image frame.

[0126] Adjust the center coordinates of the hand area frame according to the displacement;

[0127] The formula for adjusting the center coordinates of the hand area box is: Among them, h t z is the horizontal coordinate of the center coordinate of the hand area frame according to the time frame t; t is the ordinate of the center coordinate of the hand area frame according to the change of time frame t; h0 is the abscissa of the hand area frame of the previous time frame; z0 is the ordinate of the hand area frame of the previous time frame; Δh is the horizontal displacement; Δz is the vertical displacement.

[0128] Set the maximum size threshold and the minimum size threshold of the hand area frame; if the size of the adjusted hand area frame exceeds the range of the maximum size threshold and the minimum size threshold, it will be cropped to the maximum size threshold or the minimum size threshold.

[0129] It should be noted that the maximum size threshold and the minimum size threshold are set by collecting a large amount of hand image data and counting the size distribution of the hand area box; based on the statistical results, appropriate percentiles are selected as the maximum size threshold and the minimum size threshold, such as 95% and 5%.

[0130] Use the adjusted hand region box size (W t ,H t ) and the center coordinates (h t ,z t ) as the initial hand region frame of the next image frame; repeat the above process on the new image frame to achieve adaptive adjustment of the hand region frame.

[0131] The hand region box is scaled to a fixed resolution to meet the input requirements of the hand key point detection model, and reasonable scaling helps to retain sufficient detail information while reducing computational complexity.

[0132] Convert the scaled-resolution hand region bounding boxes to the input format required by the hand keypoint detection model; this conversion helps speed up the model inference process.

[0133] Furthermore, the preset method of the hand key point detection model includes:

[0134] The basic structure of the hand key point detection model is defined to include a generator network structure and a discriminator network structure; and adversarial training is performed on the generator network structure and the discriminator network structure.

[0135] The generator network structure adopts an encoder-decoder structure. The input is the image of the hand region frame, and the output is the generated hand key point heat map y_fake. The encoder consists of a convolutional layer and a downsampling layer to extract image features. The decoder consists of an upsampling layer and a convolutional layer to generate a hand key point heat map.

[0136] The goal of the generator network structure is to minimize the difference LC between the generated hand key point heat map y_fake and the real heat map y_real. The difference is calculated using L1 or L2 loss at the pixel level.

[0137] The input of the discriminator network structure is the hand key point heat map y_fake or the real heat map y_real generated by the generator, and the output is the probability value of the hand key point heat map y_fake or the real heat map y_real being real or generated;

[0138] The discriminator network structure adopts a convolutional neural network structure to extract features and classify the input hand key point heat map y_fake or real heat map y_real.

[0139] The goal of the discriminator network structure is to maximize the probability that the real heat map is judged as real, and the probability that the generated hand key point heat map is judged as generated; that is, the binary loss function LD that defines the discriminator network structure;

[0140] in, is the negative logarithmic expectation of the probability that the real heat map y_real is correctly judged as real; D y_real It represents the output of the discriminator network structure for the real heat map y_real, which is a value between 0 and 1, indicating the probability that the discriminator network structure believes that y_real is the real heat map;

[0141] is the negative logarithmic expectation of the probability that the generated hand key point heat map y_fake is correctly judged as generated; D y_fake It represents the output of the discriminator network structure for the generated hand key point heat map y_fake, which is a value between 0 and 1, indicating the probability that the discriminator believes that y_fake is a real heat map.

[0142] The method of performing adversarial training includes:

[0143] The training goal is to find a Nash equilibrium point, and update the parameters of the generator network structure and the discriminator network structure so that the heat map of the hand key points generated by the network structure cannot be distinguished from the discriminator network structure;

[0144] The Nash equilibrium point is to minimize the difference between the generated hand key point heat map y_fake and the real heat map y_real while maximizing LD; during training, the generator network structure and the discriminator network structure are alternately optimized to update their respective network parameters.

[0145] Initialize the parameters of the generator network structure and the discriminator network structure; usually random initialization is used;

[0146] Collect a training data set containing hand images and corresponding real heat maps; for the training data set, shuffle the training data set and divide it into R batches; sample by batch; for each batch, sample the real heat map from the batch; input the current generator network structure into the hand image to generate y_fake; input the real heat map and y_fake into the discriminator network structure; and calculate the binary loss function LD of the discriminator network structure;

[0147] Sample hand images from the batch; input the hand images into the generator network structure to generate y_fake; input y_fake into the current discriminator network structure; calculate the LC of the generator network structure; iteratively update the parameters of the generator network structure and the discriminator network structure and meet the training objectives; that is, complete the preset of the hand key point detection model.

[0148] The coordinates of all hand key points in the hand key point heat map generated by the final generator network structure are extracted to form the hand key point coordinate data.

[0149] During the training process, the generator network structure and the discriminator network structure compete with each other. The generator tries to generate more and more realistic hand key point heat maps to deceive the discriminator, while the discriminator strives to distinguish between real and generated heat maps. Through this adversarial training, a high-quality generator model can be obtained in the end for hand key point coordinate detection.

[0150] Furthermore, the method of acquiring the hand motion trajectory and the hand three-dimensional model data includes:

[0151] The three-dimensional coordinates of the hand key points are calculated by using the hand key point coordinate data combined with the intrinsic parameters of the user-end device camera (such as focal length, principal point, etc.) and the known hand size constraints.

[0152] The method of calculating the three-dimensional coordinates of the key points of the hand includes:

[0153] Construct an intrinsic parameter matrix K based on the intrinsic parameters of the camera of the user end device;

[0154] Among them: fh is the horizontal focal length of the camera of the user end device; fz is the vertical focal length of the camera of the user end device; ch is the horizontal coordinate of the principal point of the camera of the user end device; cz is the vertical coordinate of the principal point of the camera of the user end device;

[0155] The hand key point coordinate data is used to remove the scale factor using the parameter matrix K to obtain the normalized image plane coordinates;

[0156] The formula for removing the scale factor is: Among them, hk j zk is the horizontal coordinate of the jth hand key point image plane coordinate; j is the ordinate of the image plane coordinates of the jth hand key point; uk j vk is the horizontal coordinate of the jth hand key point in the hand key point heat map; j is the ordinate of the jth hand key point in the hand key point heat map.

[0157] Given the distance constraints of the key points of the hand in three-dimensional space, such as palm width, finger length, etc., a nonlinear least squares optimization problem is constructed to calculate the vertical coordinates of each key point.

[0158] The formula for calculating the vertical coordinate of the ruler is: Among them, d jl is the known distance constraint between the jth and lth hand key points; Zh j Zh is the vertical coordinate of the jth hand key point; l is the vertical coordinate of the ruler of the lth hand key point;

[0159] And project the image plane coordinates according to the vertical coordinates of the ruler to obtain the three-dimensional coordinates of the key points of the hand;

[0160] The projection formula is: Among them, (Hk j ,Zk j ,Ck j ) is the three-dimensional coordinate of the hand key point of the jth hand key point.

[0161] Based on the three-dimensional coordinates of the key points of the hand, a hand skeleton model is constructed; the hand skeleton model consists of J skeleton chain nodes, each skeleton chain node corresponds to a key point; the skeleton chain nodes are connected by virtual bones to simulate the skeleton structure of the hand.

[0162] The method of obtaining the joint angle includes:

[0163] For each set of two adjacent bone chain nodes, obtain the two bone vectors they are connected to; use the vector dot multiplication operation to calculate the angle between the two bone vectors, that is, the joint angle; the joint angle describes the posture and movement of the hand.

[0164] The method of obtaining the hand motion trajectory includes:

[0165] For each key point in the hand skeleton model, a three-dimensional motion trajectory is obtained based on continuous image frames using a motion estimation algorithm (such as Kalman filtering, particle filtering, etc.), that is, the hand motion trajectory; the three-dimensional motion trajectory describes the movement of the hand key points in three-dimensional space; the three-dimensional motion trajectory is the change of the three-dimensional coordinates and the change of the movement speed of the three-dimensional coordinates; both are time series.

[0166] Furthermore, the preset methods of the n gesture command modes include:

[0167] Collect gesture data, which comes from real user gesture capture or is generated by motion capture equipment or computer animation modeling; gesture data includes information about the hand skeleton model, joint angles, and hand motion trajectory;

[0168] Manually label the collected gesture data and the gesture instructions corresponding to each gesture data;

[0169] The gesture data is a continuous time data sequence; the sequence composed of each gesture action is segmented from the gesture data; and the segmented sequences are aligned at the same time so that sequences of different lengths can correspond;

[0170] Cluster the sequence of each gesture action to obtain n categories; clustering can be based on features such as hand key point coordinate sequence, joint angle sequence, etc., using algorithms such as K-Means, spectral clustering, etc.;

[0171] For each category, the gesture command mode is defined including gesture command, hand skeleton model, joint angle and hand motion trajectory.

[0172] Furthermore, the method of matching the hand skeleton model, joint angles and hand motion trajectory with the gesture instruction pattern includes:

[0173] The hand skeleton model, hand motion trajectory and joint angle are converted into a multi-dimensional feature sequence as a sequence to be matched; each time step of the multi-dimensional feature sequence corresponds to a feature vector;

[0174] For each preset gesture command pattern, its corresponding hand skeleton model, hand motion trajectory and joint angle are also converted into a multi-dimensional feature sequence as a gesture template sequence;

[0175] The best matching gesture command pattern is found by calculating the time warping distance between the multidimensional feature sequence and the gesture command pattern.

[0176] The time warp distance is calculated by:

[0177] Calculate the local distance matrix between the sequence to be matched and each gesture template sequence; the size of the local distance matrix is ​​M×N, where M is the length of the sequence to be matched and N is the length of the gesture template sequence; each element D(λ,β) in the local distance matrix represents the distance between the sequence to be matched at time step λ and the gesture template sequence at time step β, which is usually calculated using Euclidean distance or other distance metrics;

[0178] Based on the local distance matrix, a cumulative distance matrix is ​​calculated. Each element D'(λ,β) in the cumulative distance matrix represents the minimum cumulative distance from time step 0 to the sequence to be matched at time step λ and the gesture template sequence at time step β.

[0179] D'(λ,β)=D(λ,β)+min(D'(λ-1,β),D'(λ,β-1),D'(λ-1,β-1));

[0180] Among them, D'(λ-1,β) represents the minimum cumulative distance from time step 0 to the sequence to be matched at time step λ-1 and the gesture template sequence at time step β; D'(λ,β-1) represents the minimum cumulative distance from time step 0 to the sequence to be matched at time step λ and the gesture template sequence at time step β-1; D'(λ-1,β-1) represents the minimum cumulative distance from time step 0 to the sequence to be matched at time step λ-1 and the gesture template sequence at time step β-1.

[0181] This process achieves nonlinear alignment on the time axis, allowing for a certain amount of time warping between sequences.

[0182] Starting from the lower right corner element of the cumulative distance matrix, backtrack along the direction of the minimum cumulative distance to find an optimal path from time step 0 to the lower right corner element; this optimal path corresponds to the best time alignment between the sequence to be matched and the gesture template sequence;

[0183] The sum of the minimum cumulative distances of all elements on the optimal path is obtained, which is the dynamic time warping distance between the sequence to be matched and the gesture template sequence; the smaller the dynamic time warping distance, the higher the similarity between the two sequences.

[0184] Calculate the dynamic time warping distance between the sequence to be matched and all gesture template sequences; select the gesture instruction pattern corresponding to the gesture template sequence with the smallest dynamic time warping distance as the best matching gesture instruction pattern; and complete the matching.

[0185] Preferably, if all dynamic time warping distances exceed a preset distance threshold, it is determined to be an unknown gesture; and the unknown gesture is added to the gesture command mode.

[0186] It should be noted that the digital twin virtual scene is a combination of digital twin and Web technology to develop an application based on the user browser; it can be accessed and interacted through the Internet without installing special client software.

[0187] In this embodiment, the present invention realizes natural and convenient human-computer interaction; adopts a generative adversarial network to realize high-precision hand key point detection, and improves the accuracy and robustness of gesture recognition; can adapt to different gestures and complex application scenarios; by adaptively adjusting the hand area frame and combining the ROI adjustment algorithm, the accuracy of hand detection and key point positioning is improved; the recognition difficulty caused by hand movement is reduced; a gesture recognition method based on a hand skeleton model and time warp distance is proposed, and the time continuity analysis of gestures is introduced; by performing nonlinear alignment on the time axis, the dynamic changes of gestures can be better characterized, and the accuracy of gesture classification can be improved; a variety of gesture command modes are preset, and dynamic updates are supported; personalized and scalable gesture interaction capabilities are provided, enriching the human-computer interaction mode; gesture interaction is combined with digital twin WEB applications to expand the application scenarios of gesture interaction.

[0188] Example 2

[0189] See also Figure 2 As shown, the part not described in detail in this embodiment is described in Example 1, which provides a digital twin WEB application air gesture interaction system, including:

[0190] The data acquisition and processing module is used to acquire the real-time video stream of the camera of the user terminal device and pre-process the real-time video stream to obtain X hand area frames;

[0191] A key point detection module is used to input X hand region frames into a preset hand key point detection model based on a generative adversarial network to obtain hand key point coordinate data;

[0192] The motion detection module obtains the hand skeleton model based on the coordinate data of the key points of the hand; obtains the joint angles and hand motion trajectory according to the hand skeleton model;

[0193] The best matching module is used to map the hand skeleton model to the digital twin virtual scene, preset n gesture command modes in the digital twin virtual scene, and match the hand skeleton model, joint angles and hand motion trajectory with the gesture command mode; each module is connected by wired and / or wireless means to realize data transmission between modules.

[0194] Example 3

[0195] This embodiment discloses an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the operation mode of the digital twin WEB application air gesture interaction method provided above is implemented.

[0196] Since the electronic device introduced in this embodiment is an electronic device used to implement a digital twin WEB application air gesture interaction method in the embodiment of this application, based on the digital twin WEB application air gesture interaction method introduced in the embodiment of this application, the technical personnel of this field can understand the specific implementation method of the electronic device of this embodiment and its various variations, so how the electronic device implements the method in the embodiment of this application is not introduced in detail here. As long as the technical personnel of this field implement the electronic device used by the digital twin WEB application air gesture interaction method in the embodiment of this application, it belongs to the scope of protection of this application.

[0197] The above formulas are all dimensionless and numerical calculations. The formula is a formula for the most recent real situation obtained by collecting a large amount of data and performing software simulation. The preset parameters and thresholds in the formula are set by technicians in this field according to actual conditions.

[0198] The above is only a preferred embodiment of the present invention, and the protection scope of the present invention is not limited to the above embodiments. All technical solutions under the concept of the present invention belong to the protection scope of the present invention. It should be pointed out that for ordinary technical users in this technical field, some improvements and modifications without departing from the principle of the present invention should also be regarded as the protection scope of the present invention.

Claims

1. A digital twin WEB application air gesture interaction method, characterized in that: include: S1, collect the real-time video stream of the camera of the user terminal device, and pre-process the real-time video stream; Get X hand area frames; S2, inputting X hand region frames into a preset hand key point detection model based on a generative adversarial network to obtain hand key point coordinate data; S3, obtaining a hand skeleton model based on the hand key point coordinate data; obtaining joint angles and hand motion trajectory according to the hand skeleton model; S4. Map the hand skeleton model to the digital twin virtual scene, preset n gesture command modes in the digital twin virtual scene, and match the hand skeleton model, joint angles, and hand motion trajectories with the gesture command modes.

2. According to claim 1, a digital twin WEB application air gesture interaction method is characterized in that: The method of preprocessing the real-time video stream includes: Extract image frames frame by frame from the real-time video stream; input the image frames into the pre-trained hand detection model to obtain the bounding box coordinates of the hand; crop the hand region box containing only the hand from the original image frame according to the bounding box coordinates; and use the ROI adjustment algorithm to adjust the size and position of the hand region box in real time; The training method of the hand detection model includes: Initially define the basic framework of the hand detection model, which includes the backbone network, feature pyramid network, detection head network and anchor box; The ResNet-101 network is used as the backbone network to output feature maps. The ResNet-101 network consists of b stages, each of which consists of m residual blocks. A feature pyramid network is constructed based on the feature maps output by the backbone network. Based on the feature pyramid network, an independent detection head sub-network is designed for each level of the pyramid feature map; the detection head includes 4 convolutional layers and 2 parallel fully connected layers; the 2 parallel fully connected layers include a classification fully connected layer and a regression fully connected layer; On each level of the pyramid feature map, 9 predefined anchor boxes are set for each pixel position; the size of the anchor box is defined as {32 2 , 64 2 , 128 2 , 256 2 , 512 2 } pixels; and the aspect ratio is {1:2, 1:1, 2:1}; The loss function of the hand detection model is defined as a classification loss function and a regression loss function. Classification loss function FL(p c )=-(1-p c ) γ ·log(p c ), where γ is the regulatory factor; Regression loss function Collect k images of hands to form an image dataset; annotate the images in the image dataset and generate labels, where the labels are the bounding box coordinates of the hand in each image, i.e., the annotation box; The image dataset is divided into U batches; the batches are input into the hand detection model, and the values ​​of the classification loss function and the regression loss function after each batch is input are counted, and the parameters in the hand detection model are continuously adjusted; the values ​​of the classification loss function and the regression loss function no longer decrease in Y consecutive batches, and the training is stopped; a trained hand detection model is obtained.

3. According to claim 2, a digital twin WEB application air gesture interaction method is characterized in that: The formula of the output feature map is: y = F(x, W i )+x; where y is the feature map output by the residual block; x is the input image frame; F(x,W i ) is the residual function; W i is the learnable parameter of the i-th residual block; F(x,W i )=ReLU(Conv(x,W i ));Wherein, Conv represents the convolution operation; ReLU is the rectified linear unit activation function; The construction method of the feature pyramid network includes: By constructing horizontally connected and vertically connected paths, a pyramid feature map with 7 levels is generated; The formula for lateral connection is: P b =Conv(C b ,256)+Upsample(P b+1 ), where Pb is the b-th level pyramid feature map; C b The feature map output by the b-th stage of the backbone network; b is a positive integer greater than or equal to 1 and less than or equal to 7; Conv(C b ,256) means to perform 1×1 convolution operation on Cb and change the number of channels to 256; Upsample(P b+1 ) represents the upsampling operation, Pb +1 The size is upsampled to the same size as P b Same size; P b+1 is the b+1th level pyramid feature map; The formula for vertical connection is: Among them, P6 is the sixth level pyramid feature map; P7 is the seventh level pyramid feature map; the feature map output by the fifth stage of the C5 backbone network; Maxpool (P6) means the maximum pooling operation on P6; The classification fully connected layer uses a Softmax activation function to calculate the probability that each anchor box belongs to the hand or the background; The probability is calculated as: in, is the weight parameter W of the classification fully connected layer c The transpose of is the feature vector from the feature pyramid network; B c is the bias parameter of the classification fully connected layer; p c The probability of each predicted anchor box belonging to the hand or the background; The regression fully connected layer has no activation function and is used to obtain the offset fu from the predicted anchor box to the annotation box.

4. According to claim 3, a digital twin WEB application air gesture interaction method is characterized in that: The method of adjusting the size and position of the hand area frame in real time includes: Using Kalman filtering or motion estimation algorithm, the position of the hand in the continuous image frames is tracked; the displacement and movement speed of the hand in the coordinate system of the image frame are obtained; the displacement includes vertical displacement and horizontal displacement; the movement speed includes horizontal movement speed and vertical movement speed; Set the size of the initial hand region box to W0×H0. The size of the initial hand region box surrounds the detected hand bounding box and leaves a fixed margin. Adjust the size of the hand area box according to the movement speed; The formula for adjusting the size of the hand area box is: Among them, α is the adjustment coefficient; W t H is the width of the hand region frame that changes according to the time frame t; t is the height of the hand region box that changes according to the time frame t; v h is the horizontal movement speed of the hand in the coordinate system of the image frame; v z is the vertical movement speed of the hand in the coordinate system of the image frame; Adjust the center coordinates of the hand area frame according to the displacement; The formula for adjusting the center coordinates of the hand area box is: Among them, h t z is the horizontal coordinate of the center coordinate of the hand area frame according to the time frame t; t is the ordinate of the center coordinate of the hand region frame according to the change of time frame t; h0 is the abscissa of the hand region frame of the previous time frame; z0 is the ordinate of the hand region frame of the previous time frame; Δh is the horizontal displacement; Δz is the vertical displacement; Set the maximum size threshold and the minimum size threshold of the hand area frame; if the size of the adjusted hand area frame exceeds the range of the maximum size threshold and the minimum size threshold, it will be cropped to the maximum size threshold or the minimum size threshold; Use the adjusted hand region box size (W t ,H t ) and the center coordinates (h t ,z t ) as the initial hand region frame of the next image frame.

5. According to claim 4, a digital twin WEB application air gesture interaction method is characterized in that: The preset method of the hand key point detection model includes: The basic structure of the hand key point detection model is defined to include the generator network structure and the discriminator network structure; and adversarial training is performed on the generator network structure and the discriminator network structure; The generator network structure adopts an encoder-decoder structure. The input is the image of the hand region frame, and the output is the generated hand key point heat map y_fake. The encoder consists of a convolutional layer and a downsampling layer to extract image features. The decoder consists of an upsampling layer and a convolutional layer to generate a hand key point heat map. The goal of the generator network structure is to minimize the difference LC between the generated hand key point heat map y_fake and the real heat map y_real. The difference is calculated using the L1 or L2 loss at the pixel level. The input of the discriminator network structure is the hand key point heat map y_fake or the real heat map y_real generated by the generator, and the output is the probability value of the hand key point heat map y_fake or the real heat map y_real being real or generated; The discriminator network structure adopts a convolutional neural network structure to extract features and classify the input hand key point heat map y_fake or real heat map y_real; The goal of the discriminator network structure is to maximize the probability that the real heat map is judged as real, and the probability that the generated hand key point heat map is judged as generated; that is, the binary loss function LD that defines the discriminator network structure; in, is the negative logarithmic expectation of the probability that the real heat map y_real is correctly judged as real; D y_real Indicates the probability that the discriminator network structure believes that y_real is the real heat map; is the negative logarithmic expectation of the probability that the generated hand key point heat map y_fake is correctly judged as generated; D y_fake Indicates the probability that the discriminator thinks y_fake is a real heat map; Initialize the parameters of the generator network structure and the discriminator network structure; collect a training data set containing hand images and corresponding real heat maps; for the training data set, shuffle the training data set and divide it into R batches; sample by batch; for each batch, sample the real heat map from the batch; input the current generator network structure into the hand image to generate y_fake; input the real heat map and y_fake into the discriminator network structure; and calculate the binary loss function LD of the discriminator network structure; Sample hand images from the batch; input the hand images into the generator network structure to generate y_fake; input y_fake into the current discriminator network structure; calculate the LC of the generator network structure; iteratively update the parameters of the generator network structure and the discriminator network structure and meet the training objectives; that is, complete the preset of the hand key point detection model.

6. According to claim 5, a digital twin WEB application air gesture interaction method is characterized in that: The method of performing adversarial training includes: The training goal is to find a Nash equilibrium point, and update the parameters of the generator network structure and the discriminator network structure; the Nash equilibrium point is to minimize the difference between the generated hand key point heat map y_fake and the real heat map y_real while maximizing LD; during training, the generator network structure and the discriminator network structure are alternately optimized to update their respective network parameters; The method of acquiring the hand motion trajectory and the hand three-dimensional model data includes: The three-dimensional coordinates of the key points of the hand are calculated by combining the coordinate data of the key points of the hand with the intrinsic parameters of the camera of the user end device and the known hand size constraints; Based on the three-dimensional coordinates of the key points of the hand, a hand skeleton model is constructed; the hand skeleton model consists of J skeleton chain nodes, each skeleton chain node corresponds to a key point; the skeleton chain nodes are connected by virtual bones; The method of obtaining the joint angle includes: For each set of two adjacent bone chain nodes, obtain the two bone vectors they are connected to; use vector dot multiplication to calculate the angle between the two bone vectors, i.e. the joint angle; the joint angle describes the posture and movement of the hand; The method of obtaining the hand motion trajectory includes: For each key point in the hand skeleton model, a motion estimation algorithm is used based on continuous image frames to obtain a three-dimensional motion trajectory, namely, the hand motion trajectory.

7. A digital twin WEB application air gesture interaction method according to claim 6, characterized in that: The method of calculating the three-dimensional coordinates of the key points of the hand includes: Construct an intrinsic parameter matrix K based on the intrinsic parameters of the camera of the user end device; Among them: fh is the horizontal focal length of the camera of the user end device; fz is the vertical focal length of the camera of the user end device; ch is the horizontal coordinate of the principal point of the camera of the user end device; cz is the vertical coordinate of the principal point of the camera of the user end device; The hand key point coordinate data is used to remove the scale factor using the parameter matrix K to obtain the normalized image plane coordinates; The formula for removing the scale factor is: Among them, hk j zk is the horizontal coordinate of the jth hand key point image plane coordinate; j is the ordinate of the jth hand key point image plane coordinate; uk j vk is the horizontal coordinate of the jth hand key point in the hand key point heat map; j is the ordinate of the jth hand key point in the hand key point heat map; Given the distance constraints of the key points of the hand in three-dimensional space, a nonlinear least squares optimization problem is constructed to calculate the vertical coordinates of each key point; The formula for calculating the vertical coordinate of the ruler is: Among them, d jl is the known distance constraint between the jth and lth hand key points; Zh j Zh is the vertical coordinate of the jth hand key point; l is the vertical coordinate of the ruler of the lth hand key point; And project the image plane coordinates according to the vertical coordinates of the ruler to obtain the three-dimensional coordinates of the key points of the hand; The projection formula is: Among them, (Hk j ,Zk j ,Ck j ) is the three-dimensional coordinate of the hand key point of the jth hand key point.

8. According to claim 7, a digital twin WEB application air gesture interaction method is characterized in that: The preset methods of the n gesture command modes include: Collect gesture data, which comes from real user gesture capture or is generated by motion capture equipment or computer animation modeling; gesture data includes information about the hand skeleton model, joint angles, and hand motion trajectory; Manually annotate the collected gesture data, and identify the gesture instructions corresponding to each gesture data; the gesture data is a continuous time data sequence; separate the sequence composed of each gesture action from the gesture data; and align the separated sequences at the same time; The sequence of each gesture action is clustered to obtain n categories; for each category, a gesture instruction pattern is defined including gesture instructions, hand skeleton model, joint angles and hand motion trajectory.

9. A digital twin WEB application air gesture interaction method according to claim 8, characterized in that: The method of matching the hand skeleton model, joint angles and hand motion trajectory with the gesture instruction pattern includes: The hand skeleton model, hand motion trajectory and joint angle are converted into a multi-dimensional feature sequence as a sequence to be matched; each time step of the multi-dimensional feature sequence corresponds to a feature vector; For each preset gesture command pattern, its corresponding hand skeleton model, hand motion trajectory and joint angle are also converted into a multi-dimensional feature sequence as a gesture template sequence; By calculating the time warping distance between the multi-dimensional feature sequence and the gesture command pattern, the best matching gesture command pattern is found; The time warp distance is calculated by: Calculate the local distance matrix between the sequence to be matched and each gesture template sequence; the size of the local distance matrix is ​​M×N, where M is the length of the sequence to be matched and N is the length of the gesture template sequence; each element D(λ,β) in the local distance matrix represents the distance between the sequence to be matched at time step λ and the gesture template sequence at time step β; Based on the local distance matrix, a cumulative distance matrix is ​​calculated. Each element D'(λ,β) in the cumulative distance matrix represents the minimum cumulative distance from time step 0 to the sequence to be matched at time step λ and the gesture template sequence at time step β. D'(λ,β)=D(λ,β)+min(D'(λ-1,β),D'(λ,β-1),D'(λ-1,β-1)); Among them, D'(λ-1,β) represents the minimum cumulative distance from time step 0 to the sequence to be matched at time step λ-1 and the gesture template sequence at time step β; D'(λ,β-1) represents the minimum cumulative distance from time step 0 to the sequence to be matched at time step λ and the gesture template sequence at time step β-1; D'(λ-1,β-1) represents the minimum cumulative distance from time step 0 to the sequence to be matched at time step λ-1 and the gesture template sequence at time step β-1; Starting from the lower right corner element of the cumulative distance matrix, backtrack along the direction of the minimum cumulative distance to find an optimal path from time step 0 to the lower right corner element; The sum of the minimum cumulative distances of all elements on the optimal path is obtained, which is the dynamic time warping distance between the sequence to be matched and the gesture template sequence; the dynamic time warping distance between the sequence to be matched and all gesture template sequences is calculated; the gesture instruction pattern corresponding to the gesture template sequence with the smallest dynamic time warping distance is selected as the best matching gesture instruction pattern; the matching is completed.

10. A digital twin WEB application air gesture interaction system, used to implement a digital twin WEB application air gesture interaction method according to claims 1 to 9, characterized in that: include: The data acquisition and processing module is used to collect the real-time video stream of the camera of the user terminal device and pre-process the real-time video stream; Get X hand area frames; A key point detection module is used to input X hand region frames into a preset hand key point detection model based on a generative adversarial network to obtain hand key point coordinate data; The motion detection module obtains the hand skeleton model based on the coordinate data of the key points of the hand; obtains the joint angles and hand motion trajectory according to the hand skeleton model; The best matching module is used to map the hand skeleton model to the digital twin virtual scene, preset n gesture command modes in the digital twin virtual scene, and match the hand skeleton model, joint angles and hand motion trajectory with the gesture command mode; each module is connected by wired and / or wireless means.

Images