Digital twin web application air gesture interaction method and system

Abstract

The application belongs to the technical field of human-computer interaction, and discloses a digital twin WEB application air gesture interaction method and system; the method comprises the following steps: collecting a real-time video stream of a camera of a user terminal device, and preprocessing the real-time video stream; obtaining X hand region frames; inputting the X hand region frames into a preset hand key point detection model based on a generative adversarial network to obtain hand key point coordinate data; obtaining a hand skeleton model based on the hand key point coordinate data; obtaining joint angles and hand motion trajectories according to the hand skeleton model; mapping the hand skeleton model to a digital twin virtual scene, presetting n gesture instruction modes in the digital twin virtual scene, and matching the hand skeleton model, the joint angles and the hand motion trajectories with the gesture instruction modes; the method provides personalized and expandable gesture interaction capabilities, and enriches the mode of human-computer interaction.

Description

Technical Field

[0001] This invention relates to the field of human-computer interaction technology, and more specifically, to a method and system for air gesture interaction in digital twin web applications. Background Technology

[0002] Patent application CN112394811A discloses an interactive method and electronic device for air gestures, relating to the fields of terminal technology and AI technology, such as AI image recognition. This method can improve the user experience when interacting with electronic devices air-to-air and reduce the error rate of air gestures. The method includes: the electronic device displaying a first display interface of a first application; the electronic device detecting a first air gesture input by the user via a camera; in response to the first air gesture, the electronic device moving an indicator icon in the first display interface along a trajectory of the first air gesture in a preset direction; and in response to the first air gesture, the electronic device displaying a second display interface of the first application.

[0003] However, traditional human-computer interaction methods, such as keyboards, mice, and touchscreens, lack naturalness and convenience, failing to provide an interactive experience anytime, anywhere. Users need to directly touch hardware devices, resulting in a limited interaction method and causing numerous inconveniences. Existing gesture recognition technologies suffer from low accuracy and poor robustness, making them difficult to adapt to complex application scenarios. The accuracy of hand key point detection and gesture classification in gesture recognition needs improvement. Furthermore, there are certain difficulties in accurately detecting the position of hand key points and distinguishing similar gestures, affecting the overall recognition effect. Existing gesture recognition technologies lack analysis of the temporal continuity of dynamic gesture changes, failing to effectively represent and capture the dynamic process of gestures, thus limiting recognition capabilities. They also lack personalization and scalability, offering a limited interaction method and failing to meet the diverse needs of different users and application scenarios.

[0004] In view of this, the present invention proposes a digital twin web application air gesture interaction method and system to solve the above problems. Summary of the Invention

[0005] In order to overcome the above-mentioned defects of the prior art and to achieve the above objectives, the present invention provides the following technical solution: a digital twin WEB application air gesture interaction method, comprising: S1, acquiring real-time video stream from the user terminal device camera and preprocessing the real-time video stream; obtaining X hand region boxes;

[0006] S2. Input X hand region boxes into a preset hand keypoint detection model based on generative adversarial network to obtain hand keypoint coordinate data;

[0007] S3, obtaining a hand skeleton model based on the hand key point coordinate data; obtaining joint angles and hand motion trajectories according to the hand skeleton model;

[0008] S4, mapping the hand skeleton model into a digital twin virtual scene, presetting n gesture instruction modes in the digital twin virtual scene, and matching the hand skeleton model, the joint angles and the hand motion trajectories with the gesture instruction modes.

[0009] Further, the manner of preprocessing the real-time video stream comprises:

[0010] extracting image frames from the real-time video stream frame by frame; inputting the image frames into a pre-trained hand detection model to obtain hand bounding box coordinates; cropping a hand region box containing only the hand from the original image frame according to the bounding box coordinates; and adjusting the size and position of the hand region box in real time by using an ROI adjustment algorithm;

[0011] The training manner of the hand detection model comprises:

[0012] initially defining a basic framework of the hand detection model, the basic framework comprising a backbone network, a feature pyramid network, a detection head subnetwork and anchor boxes;

[0013] using a ResNet-101 network as the backbone network for outputting feature maps; the ResNet-101 network comprises b stages, each stage comprising m residual blocks; and the feature pyramid network is constructed on the basis of the feature maps output by the backbone network;

[0014] designing an independent detection head subnetwork for each level of pyramid feature map based on the feature pyramid network; the detection head comprises four convolutional layers and two parallel fully connected layers; the two parallel fully connected layers comprise a classification fully connected layer and a regression fully connected layer;

[0015] setting nine predefined anchor boxes for each pixel position on each level of pyramid feature map; the size of the anchor boxes is defined as {32 2 , 64 2 , 128 2 , 256 2 , 512 2} pixels; and the aspect ratio is {1:2, 1:1, 2:1};

[0016] defining a loss function of the hand detection model, which comprises a classification loss function and a regression loss function;

[0017] the classification loss function FL(p c ) = -(1-p c ) γ ·log(p c ); wherein γ is an adjustment factor;

[0018] Regression loss function

[0019] Collecting images containing k hands to form an image dataset; labeling the images in the image dataset to generate labels, which are the bounding box coordinates of the hands in each image, i.e., the labeled boxes;

[0020] Dividing the image dataset into U batches; inputting the batches into the hand detection model, and counting the values of the classification loss function and the regression loss function after the input of each batch ends, and constantly adjusting the parameters in the hand detection model; causing the values of the classification loss function and the regression loss function to no longer decrease in the next Y batches, stopping training; obtaining a trained hand detection model.

[0021] Further, the formula of the output feature map is: y=F(x,W i )+x; wherein y is the feature map output by the residual block; x is the input image frame; F(x,W i ) is a residual function; W i is the learnable parameter of the i-th residual block;

[0022] F(x,W i )=ReLU(Conv(x,W i )); wherein Conv represents convolution operation; ReLU is a rectified linear unit activation function;

[0023] The feature pyramid network is constructed in the following manner:

[0024] By constructing a path of horizontal connection and vertical connection; a pyramid feature map with 7 levels is generated;

[0025] The formula of the horizontal connection is: P b =Conv(C b ,256)+Upsample(P b+1 ); wherein P b is the b-th level pyramid feature map; C b is the feature map output by the b-th stage of the backbone network; b is a positive integer greater than or equal to 1 and less than or equal to 7; Conv(C b ,256) represents performing 1×1 convolution operation on C b to change the channel number to 256; Upsample(P b+1 ) represents up-sampling operation, which up-samples the size of P b+1 to the same size as P b ; P b+1 is the b+1-th level pyramid feature map;

[0026] The formula of vertical connection is: Wherein, P6 is the 6th level pyramid feature map; P7 is the 7th level pyramid feature map; C5 is the output feature map of the 5th stage of the backbone network; Maxpool(P6) represents the maximum pooling operation on P6;

[0027] The classification full connection layer uses a Softmax activation function to calculate the probability of each anchor box belonging to a hand or background;

[0028] The formula of the probability is: Wherein, W c T is the transpose of the weight parameter W c of the classification full connection layer; is the feature vector from the feature pyramid network; B c is the bias parameter of the classification full connection layer; p c is the predicted probability of each anchor box belonging to a hand or background;

[0029] The regression full connection layer has no activation function and is used to obtain the offset fu of the predicted anchor box to the labeled box.

[0030] Further, the manner of adjusting the size and position of the hand region box in real time comprises:

[0031] The position of the hand in the continuous image frames is tracked by using Kalman filtering or motion estimation algorithm; the displacement and motion speed of the hand in the coordinate system in which the image frames are located are obtained; the displacement comprises vertical displacement and horizontal displacement; the motion speed comprises horizontal motion speed and vertical motion speed;

[0032] The size of the initial hand region box is set as W0xH0, and the size of the initial hand region box is around the detected hand bounding box and has a fixed edge allowance;

[0033] The size of the hand region box is adjusted according to the motion speed;

[0034] The formula of adjusting the size of the hand region box is: Wherein, a is an adjustment coefficient; W t is the width of the hand region box varying with the time frame t; H t is the height of the hand region box varying with the time frame t; v h is the horizontal motion speed of the hand in the coordinate system in which the image frames are located; v z is the vertical motion speed of the hand in the coordinate system in which the image frames are located;

[0035] The center coordinates of the hand region box are adjusted according to the displacement;

[0036] The formula of adjusting the center coordinates of the hand region box is: wherein h t is the horizontal coordinate of the center coordinate of the hand region box varying with the time frame t; z t is the vertical coordinate of the center coordinate of the hand region box varying with the time frame t; h0 is the horizontal coordinate of the hand region box of the previous time frame; z0 is the vertical coordinate of the hand region box of the previous time frame; Δh is the horizontal displacement; Δz is the vertical displacement;

[0037] a maximum size threshold and a minimum size threshold of the hand region box are set; if the size of the adjusted hand region box exceeds the range of the maximum size threshold and the minimum size threshold, it is cropped to the maximum size threshold or the minimum size threshold;

[0038] the size (W t ,H t ) and the center coordinate (h t ,z t ) of the adjusted hand region box are used as the initial hand region box of the next image frame.

[0039] Further, the preset manner of the hand key point detection model comprises:

[0040] defining the basic structure of the hand key point detection model comprises generating a generator network structure and a discriminator network structure; and performing adversarial training on the generator network structure and the discriminator network structure;

[0041] the generator network structure adopts an encoder-decoder structure, the input is an image of a hand region box, and the output is a generated hand key point heat map y_fake; the encoder is composed of a convolution layer and a down-sampling layer, and is used for extracting image features; the decoder is composed of an up-sampling layer and a convolution layer, and is used for generating a hand key point heat map;

[0042] the target of the generator network structure is to minimize the difference LC between the generated hand key point heat map y_fake and the real heat map y_real, and the calculation of the difference is implemented by using pixel-level L1 or L2 loss;

[0043] the input of the discriminator network structure is the hand key point heat map y_fake generated by the generator or the real heat map y_real, and the output is the probability value that the hand key point heat map y_fake or the real heat map y_real is real or generated;

[0044] the discriminator network structure adopts a convolutional neural network structure, and performs feature extraction and classification on the input hand key point heat map y_fake or real heat map y_real;

[0045] The objective of the discriminator network structure is to maximize the probability that the real heat map is judged to be real and the probability that the generated hand key point heat map is judged to be generated; that is, a binary loss function LD of the discriminator network structure is defined;

[0046] wherein, is the negative logarithmic expectation of the probability that the real heat map y real is correctly judged to be real; D y_real represents the probability that the discriminator network structure considers y real to be a real heat map;

[0047] is the negative logarithmic expectation of the probability that the generated hand key point heat map y fake is correctly judged to be generated; D y_fake represents the probability that the discriminator considers y fake to be a real heat map;

[0048] The parameters of the generator network structure and the discriminator network structure are initialized; a training data set containing hand images and corresponding real heat maps is collected; for the training data set, the training data set is shuffled and divided into R batches; sampling is performed according to the batches; for each batch, real heat maps are sampled from the batch; the current generator network structure is input into the hand image to generate y fake; the real heat map and y fake are input into the discriminator network structure; and the binary loss function LD of the discriminator network structure is calculated;

[0049] The hand image is sampled from the batch; the hand image is input into the generator network structure to generate y fake; y fake is input into the current discriminator network structure; the LC of the generator network structure is calculated; the parameters of the generator network structure and the discriminator network structure are iteratively updated and meet the training objective; that is, the preset of the hand key point detection model is completed.

[0050] Further, the manner of performing adversarial training includes:

[0051] The training objective is to find a Nash equilibrium point to update the parameters of the generator network structure and the discriminator network structure; the Nash equilibrium point is to minimize the difference between the generated hand key point heat map y fake and the real heat map y real while maximizing LD; in the training, the generator network structure and the discriminator network structure are alternately optimized, and the network parameters of each are updated;

[0052] The hand motion trajectory and hand three-dimensional model data acquisition method includes:

[0053] The three-dimensional coordinates of the hand key points are calculated by using the hand key point coordinate data in combination with the internal parameters of the camera of the user terminal device and the known hand size constraint;

[0054] construct a hand skeleton model based on the three-dimensional coordinates of the hand key points; the hand skeleton model is composed of J skeleton chain nodes, each of which corresponds to a key point; the skeleton chain nodes are connected through virtual skeletons;

[0055] The joint angle acquisition method comprises:

[0056] For each set of two adjacent skeleton chain nodes, two bone vectors connected by the two skeleton chain nodes are acquired; the included angle of the two bone vectors, i.e. the joint angle, is calculated by using vector dot multiplication operation; the joint angle describes the posture and action of the hand;

[0057] The hand motion trajectory acquisition method comprises:

[0058] In the hand skeleton model, for each key point, a three-dimensional motion trajectory, i.e. a hand motion trajectory, is obtained by using a motion estimation algorithm based on consecutive image frames.

[0059] Further, the method for calculating the three-dimensional coordinates of the hand key points comprises:

[0060] An intrinsic parameter matrix K is constructed based on the intrinsic parameters of the camera of the user terminal device;

[0061] wherein fh is the horizontal focal length of the camera of the user terminal device; fz is the vertical focal length of the camera of the user terminal device; ch is the horizontal coordinate of the principal point of the camera of the user terminal device; cz is the vertical coordinate of the principal point of the camera of the user terminal device;

[0062] The hand key point coordinate data is removed from the scale factor by using the parameter matrix K to obtain normalized image plane coordinates;

[0063] The formula for removing the scale factor is: wherein hk j is the horizontal coordinate of the image plane coordinates of the jth hand key point; zk j is the vertical coordinate of the image plane coordinates of the jth hand key point; uk j is the horizontal coordinate of the jth hand key point in the hand key point heat map; vk j is the vertical coordinate of the jth hand key point in the hand key point heat map;

[0064] A nonlinear least squares optimization problem is constructed based on the distance constraint of the hand key points in the three-dimensional space to calculate the scale vertical coordinates of each key point;

[0065] The calculation formula of the scale vertical coordinates is: wherein d jl is the known distance constraint between the jth and lth hand key points; Zh jZh is the ruler vertical coordinate of the jth hand key point; l Zh is the ruler vertical coordinate of the lth hand key point;

[0066] Project the image plane coordinates according to the ruler vertical coordinates to obtain the three-dimensional coordinates of the hand key points;

[0067] The projection formula is: wherein, (Hk j ,Zk j ,Ck j ) is the three-dimensional coordinates of the jth hand key point.

[0068] Further, the preset manner of the n gesture instruction modes includes:

[0069] Collect gesture data, the gesture data is from the gesture capture of a real user, or is generated by a motion capture device or computer animation modeling; the gesture data includes information of a hand skeleton model, joint angles and a hand motion trajectory;

[0070] Manually annotate the collected gesture data, each gesture data corresponds to a gesture instruction; the gesture data is a continuous time data sequence; cut out a sequence composed of each gesture action from the gesture data; and align the cut-out sequence;

[0071] Cluster the sequence composed of each gesture action to obtain n categories; for each category, define a gesture instruction mode including a gesture instruction, a hand skeleton model, joint angles and a hand motion trajectory.

[0072] Further, the manner of matching the hand skeleton model, joint angles and hand motion trajectory with the gesture instruction mode includes:

[0073] Convert the hand skeleton model, hand motion trajectory and joint angles into a multi-dimensional feature sequence as a to-be-matched sequence; each time step of the multi-dimensional feature sequence corresponds to a feature vector;

[0074] For each preset gesture instruction mode, convert the corresponding hand skeleton model, hand motion trajectory and joint angles into a multi-dimensional feature sequence as a gesture template sequence;

[0075] Find the best matching gesture instruction mode by calculating the time warping distance between the multi-dimensional feature sequence and the gesture instruction mode;

[0076] The calculation manner of the time warping distance includes:

[0077] calculating a local distance matrix between the to-be-matched sequence and each gesture template sequence; the local distance matrix has a size of MxN, where M is the length of the to-be-matched sequence, and N is the length of the gesture template sequence; each element D(λ,β) in the local distance matrix represents a distance between the to-be-matched sequence at time step λ and the gesture template sequence at time step β;

[0078] Based on the local distance matrix, a cumulative distance matrix is calculated, and each element D'(λ,β) in the cumulative distance matrix represents the minimum cumulative distance from 0 time step to the to-be-matched sequence at time step λ and the gesture template sequence at time step β.

[0079] D'(λ,β)=D(λ,β)+min(D'(λ-1,β),D'(λ,β-1),D'(λ-1,β-1));

[0080] Wherein, D'(λ-1,β) represents the minimum cumulative distance from 0 time step to the to-be-matched sequence at time step λ-1 and the gesture template sequence at time step β; D'(λ,β-1) represents the minimum cumulative distance from 0 time step to the to-be-matched sequence at time step λ and the gesture template sequence at time step β-1; D'(λ-1,β-1) represents the minimum cumulative distance from 0 time step to the to-be-matched sequence at time step λ-1 and the gesture template sequence at time step β-1.

[0081] Starting from the right lower corner element of the cumulative distance matrix, the optimal path from 0 time step to the right lower corner element is found by backtracking along the direction of the minimum cumulative distance.

[0082] The sum of the minimum cumulative distances of all elements on the optimal path is obtained, which is the dynamic time warping distance between the to-be-matched sequence and the gesture template sequence; the dynamic time warping distances between the to-be-matched sequence and all gesture template sequences are calculated; the gesture instruction mode corresponding to the gesture template sequence with the minimum dynamic time warping distance is selected as the best matching gesture instruction mode; that is, the matching is completed.

[0083] A digital twin WEB application air gesture interaction system for implementing the digital twin WEB application air gesture interaction method, comprising: a data acquisition and processing module for acquiring real-time video stream of a user terminal device camera and preprocessing the real-time video stream; obtaining X hand region frames;

[0084] A key point detection module for inputting the X hand region frames into a preset hand key point detection model based on a generative adversarial network to obtain hand key point coordinate data;

[0085] A motion detection module for obtaining a hand skeleton model based on the hand key point coordinate data; obtaining joint angles and hand motion trajectories according to the hand skeleton model;

[0086] The best matching module is used for mapping the hand skeleton model into the digital twin virtual scene, presetting n gesture instruction modes in the digital twin virtual scene, and matching the hand skeleton model, joint angle and hand motion trajectory with the gesture instruction mode.

[0087] The technical effects and advantages of the digital twin WEB application air gesture interaction method and system are as follows:

[0088] The natural and convenient human-computer interaction is realized, the high-precision hand key point detection is realized by using the generative adversarial network, the accuracy and robustness of gesture recognition are improved, different gestures and complex application scenarios can be adapted, the precision of hand detection and key point positioning is improved by adaptively adjusting the hand region frame combined with the ROI adjustment algorithm, the recognition difficulty caused by hand movement is reduced, the gesture recognition method based on the hand skeleton model and the time twist distance is proposed, the time continuity analysis of the gesture is introduced, the dynamic change of the gesture is better represented by nonlinear alignment on the time axis, the accuracy of gesture classification is improved, multiple gesture instruction modes are preset and dynamic update is supported, personalized and extensible gesture interaction capability is provided, the human-computer interaction mode is enriched, the gesture interaction is combined with the digital twin WEB application, and the application scenario of gesture interaction is expanded. BRIEF DESCRIPTION OF DRAWINGS

[0089] Figure 1 A digital twin WEB application air gesture interaction method of the present application is shown in the figure;

[0090] Figure 2 A digital twin WEB application air gesture interaction system of the present application is shown in the figure. DETAILED DESCRIPTION

[0091] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0092] Embodiment 1

[0093] Please refer to Figure 1 The digital twin WEB application air gesture interaction method described in the embodiment includes the following steps:

[0094] S1, collect the real-time video stream of the camera of the user terminal device, and pre-process the real-time video stream; X hand region frames are obtained;

[0095] S2, inputting the X hand region frames into a preset hand key point detection model based on a generative adversarial network to obtain hand key point coordinate data;

[0096] S3, obtaining a hand skeleton model based on the hand key point coordinate data; and obtaining joint angles and a hand motion trajectory according to the hand skeleton model;

[0097] S4, mapping the hand skeleton model to a digital twin virtual scene, presetting n gesture instruction modes in the digital twin virtual scene, and matching the hand skeleton model, the joint angles and the hand motion trajectory with the gesture instruction modes.

[0098] Further, the manner of preprocessing the real-time video stream comprises:

[0099] extracting image frames from the real-time video stream frame by frame; inputting the image frames into a pre-trained hand detection model to obtain hand bounding box coordinates; and cropping a hand region frame containing only a hand from the original image frame according to the bounding box coordinates;

[0100] and using an ROI adjustment algorithm to adjust the size and position of the hand region frame in real time; when the hand moves, the ROI adjustment algorithm can adaptively enlarge or reduce the hand region frame to ensure that the hand is completely contained.

[0101] It should be explained that the hand region frame is cropped from the original image frame and is also essentially an image.

[0102] The training manner of the hand detection model comprises:

[0103] initially defining a basic framework of the hand detection model, the basic framework comprising a backbone network, a feature pyramid network, a detection head subnetwork and anchor boxes;

[0104] using a ResNet-101 network as the backbone network for outputting feature maps; the ResNet-101 network comprises b stages, and each stage comprises m residual blocks.

[0105] The formula of the output feature map is: y=F(x,W i )+x; wherein y is a feature map output by the residual block; x is an input image frame; F(x,W i ) is a residual function; W i is a learnable parameter of the i-th residual block; F(x,W i )=ReLU(Conv(x,W i)); in which, Conv represents a convolution operation; ReLU is a rectified linear unit activation function; specifically, the residual function adds the result of x after the convolution operation to x, and then passes it through ReLU;

[0106] a feature pyramid network is constructed based on the feature map output by the backbone network;

[0107] The feature pyramid network is constructed in the following manner:

[0108] By constructing paths of horizontal connection and vertical connection, feature maps of different scales are fused; pyramid feature maps with 7 levels are generated;

[0109] The formula of horizontal connection is: P b = Conv(C b , 256) + Upsample(P b+1 ); wherein, P b is the bth level pyramid feature map; C b is the feature map output by the bth stage of the backbone network; b is a positive integer greater than or equal to 1 and less than or equal to 7; Conv(C b , 256) represents performing a 1×1 convolution operation on C b to change the number of channels to 256; Upsample(P b+1 ) represents an up-sampling operation, which up-samples the size of P b+1 to the same size as P b ; P b+1 is the b+1th level pyramid feature map;

[0110] The formula of vertical connection is: in which, P6 is the 6th level pyramid feature map; P7 is the 7th level pyramid feature map; C5 is the feature map output by the 5th stage of the backbone network; Maxpool(P6) represents performing a maximum pooling operation on P6; constructing vertical connection starting from P6 and P7 can make the model better handle small size targets, while larger targets can be processed in horizontal connection through lower level features; it can more comprehensively cover targets of various scales and improve the performance of the detection model.

[0111] Based on the feature pyramid network, an independent detection head sub-network is designed for each level of pyramid feature map; the detection head includes 4 convolution layers and 2 parallel fully connected layers; the 2 parallel fully connected layers include a classification fully connected layer and a regression fully connected layer;

[0112] The classification fully connected layer uses a Softmax activation function to calculate the probability of each anchor box belonging to a hand or a background;

[0113] The formula for calculating the probability is: wherein, W c T is the weight parameter of the classification fully connected layer; c ; is the feature vector from the feature pyramid network; B c is the bias parameter of the classification fully connected layer; p c is the probability of each anchor box belonging to hand or background;

[0114] The regression fully connected layer has no activation function, and is used to obtain the offset fuof the predicted anchor box to the labeled box.

[0115] On each level of the pyramid feature map, 9 predefined anchor boxes are set for each pixel position; the size of the anchor box is defined as {32 2 , 64 2 , 128 2 , 256 2 , 512 2} pixels; and the aspect ratio is {1:2, 1:1, 2:1};

[0116] The loss function of the hand detection model includes a classification loss function and a regression loss function;

[0117] The classification loss function FL(p c ) = -(1-p c ) γ ·log(p c ); wherein, γ is an adjustment factor; used to balance the weights of positive and negative samples; since in most cases, the number of negative samples (anchor boxes not containing hands) is much larger than that of positive samples (anchor boxes containing hands), directly calculating the classification loss may cause the model to be biased towards predicting negative samples, thereby affecting the accuracy of hand detection; by setting a larger value of the adjustment factor (usually between 0.5-0.9), the weight of positive samples can be increased, so that the model pays more attention to positive samples and improves the performance of hand detection.

[0118] The regression loss function

[0119] An image dataset is collected, which contains k images of hands, ensuring that k is large enough and the images of hands are diverse; the images in the image dataset are labeled to generate labels, which are the bounding box coordinates of the hands in each image, i.e., labeled boxes;

[0120] The image dataset is divided into U batches; the batches are input into the hand detection model, and the values of the classification loss function and the regression loss function after each batch input is counted, and the parameters in the hand detection model are continuously adjusted, such as using an optimizer, SGD or Adam, etc.; so that the values of the classification loss function and the regression loss function no longer decrease in the next Y batches, and the training is stopped; and a trained hand detection model is obtained.

[0121] The manner of adjusting the size and position of the hand region frame in real time comprises:

[0122] The position of the hand in the continuous image frames is tracked by using Kalman filtering or motion estimation algorithm; the displacement and motion speed of the hand in the coordinate system of the image frame are obtained; the displacement comprises vertical displacement and horizontal displacement; the motion speed comprises horizontal motion speed and vertical motion speed.

[0123] The size of the initial hand region frame is set as W0xH0, and the size of the initial hand region frame is around the detected hand bounding box and has a fixed edge margin;

[0124] The size of the hand region frame is adjusted according to the motion speed;

[0125] The formula for adjusting the size of the hand region frame is: Wherein, a is an adjustment coefficient, used to control the change rate of the size of the hand region frame; W t is the width of the hand region frame varying with the time frame t; H t is the height of the hand region frame varying with the time frame t; v h is the horizontal motion speed of the hand in the coordinate system of the image frame; v z is the vertical motion speed of the hand in the coordinate system of the image frame.

[0126] The center coordinates of the hand region frame are adjusted according to the displacement;

[0127] The formula for adjusting the center coordinates of the hand region frame is: Wherein, h t is the horizontal coordinate of the center coordinates of the hand region frame varying with the time frame t; z t is the vertical coordinate of the center coordinates of the hand region frame varying with the time frame t; h0 is the horizontal coordinate of the hand region frame of the previous time frame; z0 is the vertical coordinate of the hand region frame of the previous time frame; Δh is the horizontal displacement; Δz is the vertical displacement.

[0128] The maximum size threshold and the minimum size threshold of the hand region frame are set; if the size of the adjusted hand region frame exceeds the range of the maximum size threshold and the minimum size threshold, it is cropped to the maximum size threshold or the minimum size threshold.

[0129] It should be noted that the setting of the maximum size threshold and the minimum size threshold is by collecting a large amount of hand image data, and statistically analyzing the size distribution of the hand region frame; according to the statistical result, a suitable percentile is selected as the maximum size threshold and the minimum size threshold, for example, 95% and 5%.

[0130] The size (W t ,H t ) and center coordinates (h t ,z t ) of the adjusted hand region frame are used as the initial hand region frame of the next image frame; the above process is repeated on the new image frame to achieve adaptive adjustment of the hand region frame.

[0131] The hand region frame is scaled to a fixed resolution to meet the input requirements of the hand key point detection model, and reasonable scaling helps to retain sufficient detail information while reducing computational complexity.

[0132] The hand region frame after resolution scaling is converted into the input format required by the hand key point detection model; this conversion helps to speed up the model inference process.

[0133] Further, the preset manner of the hand key point detection model includes:

[0134] Defining the basic structure of the hand key point detection model includes generating a generator network structure and a discriminator network structure; and performing adversarial training on the generator network structure and the discriminator network structure.

[0135] The generator network structure adopts an encoder-decoder structure, the input is an image of a hand region frame, and the output is a generated hand key point heat map y_fake; the encoder is composed of convolutional layers and down-sampling layers, which are used to extract image features; the decoder is composed of up-sampling layers and convolutional layers, which are used to generate hand key point heat maps;

[0136] The goal of the generator network structure is to minimize the difference LC between the generated hand key point heat map y_fake and the real heat map y_real, and the difference is calculated using pixel-level L1 or L2 loss.

[0137] The input of the discriminator network structure is the hand key point heat map y_fake generated by the generator or the real heat map y_real, and the output is the probability value that the hand key point heat map y_fake or the real heat map y_real is real or generated.

[0138] The discriminator network structure adopts a convolutional neural network structure to extract features and classify the input hand key point heat map y_fake or real heat map y_real.

[0139] The objective of the discriminator network structure is to maximize the probability that the real heat map is judged to be real and the probability that the generated hand key point heat map is judged to be generated; that is, the binary loss function LD of the discriminator network structure is defined;

[0140] wherein, is the negative logarithmic expectation of the probability that the real heat map y real is correctly judged to be real; D y_real represents the output of the discriminator network structure to the real heat map y real, which is a value between 0 and 1, indicating the probability that the discriminator network structure considers y real to be a real heat map;

[0141] is the negative logarithmic expectation of the probability that the generated hand key point heat map y fake is correctly judged to be generated; D y_fake represents the output of the discriminator network structure to the generated hand key point heat map y fake, which is a value between 0 and 1, indicating the probability that the discriminator considers y fake to be a real heat map.

[0142] The manner of performing adversarial training includes:

[0143] The training objective is to find a Nash equilibrium point, and the generator network structure and the discriminator network structure are updated by parameters so that the hand key point heat map generated by the network structure cannot be distinguished by the discriminator network structure;

[0144] The Nash equilibrium point is to minimize the difference between the generated hand key point heat map y fake and the real heat map y real while maximizing LD; in training, the generator network structure and the discriminator network structure are alternately optimized, and the network parameters are updated.

[0145] Initialize the parameters of the generator network structure and the discriminator network structure; usually use random initialization;

[0146] Collect a training data set containing hand images and corresponding real heat maps; for the training data set, shuffle the training data set and divide it into R batches; sample by batch; for each batch, sample the real heat map from the batch; input the current generator network structure into the hand image to generate y fake; input the real heat map and y fake into the discriminator network structure; and calculate the binary loss function LD of the discriminator network structure;

[0147] Sample hand images from the batch; input the hand images into the generator network structure to generate y_fake; input y_fake into the current discriminator network structure; calculate the LC of the generator network structure; iteratively update the parameters of the generator network structure and the discriminator network structure and satisfy the training target; that is, complete the preset of the hand key point detection model.

[0148] Extract the coordinates of all hand key points in the hand key point heat map generated by the final generator network structure to form hand key point coordinate data.

[0149] During the training process, the generator network structure and the discriminator network structure are mutually antagonistic, the generator tries to generate more and more realistic hand key point heat maps to deceive the discriminator, and the discriminator tries to distinguish the real and generated heat maps; through such adversarial training, a high-quality generator model can be finally obtained for hand key point coordinate detection.

[0150] Further, the hand motion trajectory and hand three-dimensional model data acquisition method comprises:

[0151] The three-dimensional coordinates of the hand key points are calculated by using the hand key point coordinate data in combination with the internal parameters (such as focal length, principal point, etc.) of the camera of the user terminal device and the known hand size constraint.

[0152] The method for calculating the three-dimensional coordinates of the hand key points comprises:

[0153] An internal parameter matrix K is constructed based on the internal parameters of the camera of the user terminal device;

[0154] wherein fh is the horizontal focal length of the camera of the user terminal device; fz is the vertical focal length of the camera of the user terminal device; ch is the horizontal coordinate of the principal point of the camera of the user terminal device; and cz is the vertical coordinate of the principal point of the camera of the user terminal device.

[0155] The hand key point coordinate data is removed from the scale factor by using the parameter matrix K to obtain normalized image plane coordinates;

[0156] The formula for removing the scale factor is: wherein hk j is the horizontal coordinate of the jth hand key point image plane coordinate; zk j is the vertical coordinate of the jth hand key point image plane coordinate; uk j is the horizontal coordinate of the jth hand key point in the hand key point heat map; and vk j is the vertical coordinate of the jth hand key point in the hand key point heat map.

[0157] The distance constraints of the hand key points in the three-dimensional space, such as palm width, finger length, etc., are known, a nonlinear least squares optimization problem is constructed, and the scale vertical coordinates of each key point are calculated;

[0158] The calculation formula of the scale vertical coordinates is: Wherein, d jl is the known distance constraint between the jth and the lth hand key points; Zh j is the scale vertical coordinate of the jth hand key point; Zh l is the scale vertical coordinate of the lth hand key point;

[0159] And the three-dimensional coordinates of the hand key points are obtained by projecting the image plane coordinates according to the scale vertical coordinates;

[0160] The formula for projection is: Wherein, (Hk j ,Zk j ,Ck j ) is the three-dimensional coordinates of the jth hand key point.

[0161] Based on the three-dimensional coordinates of the hand key points, a hand skeleton model is constructed; the hand skeleton model is composed of J skeleton chain nodes, each skeleton chain node corresponds to a key point; the skeleton chain nodes are connected by virtual bones, which are used to simulate the skeletal structure of the hand.

[0162] The acquisition method of the joint angle includes:

[0163] For each group of two adjacent skeleton chain nodes, the two bone vectors connected by them are obtained; the included angle of the two bone vectors, i.e. the joint angle, is calculated by using vector dot multiplication operation; the joint angle describes the posture and action of the hand.

[0164] The acquisition method of the hand motion trajectory includes:

[0165] In the hand skeleton model, for each key point, the three-dimensional motion trajectory, i.e. the hand motion trajectory, is obtained by using motion estimation algorithm (such as Kalman filter, particle filter, etc.) based on continuous image frames; the three-dimensional motion trajectory describes the motion of the hand key points in the three-dimensional space; the three-dimensional motion trajectory is the change of three-dimensional coordinates and the change of three-dimensional motion velocity; both are time series.

[0166] Further, the preset method of the n gesture instruction modes includes:

[0167] Collect gesture data, the gesture data comes from the gesture capture of real users, or is generated by motion capture device or computer animation modeling; the gesture data includes the information of hand skeleton model, joint angle and hand motion trajectory;

[0168] manually annotating the collected gesture data, each gesture data corresponding to a gesture instruction;

[0169] The gesture data is a continuous time data sequence; a sequence composed of each gesture action is segmented from the gesture data; and the segmented sequences are aligned to enable correspondence between sequences of different lengths.

[0170] The sequences composed of each gesture action are clustered to obtain n categories; the clustering can be based on features such as hand key point coordinate sequences, joint angle sequences, and the like, and algorithms such as K-Means, spectral clustering, and the like are used.

[0171] For each category, a gesture instruction mode is defined to include a gesture instruction, a hand skeleton model, a joint angle, and a hand motion trajectory.

[0172] Further, the manner of matching the hand skeleton model, the joint angle, and the hand motion trajectory with the gesture instruction mode includes:

[0173] The hand skeleton model, the hand motion trajectory, and the joint angle are converted into a multi-dimensional feature sequence as a to-be-matched sequence; each time step of the multi-dimensional feature sequence corresponds to a feature vector.

[0174] For each preset gesture instruction mode, the corresponding hand skeleton model, hand motion trajectory, and joint angle are also converted into a multi-dimensional feature sequence as a gesture template sequence.

[0175] The best matching gesture instruction mode is found by calculating the time warping distance between the multi-dimensional feature sequence and the gesture instruction mode.

[0176] The calculation manner of the time warping distance includes:

[0177] A local distance matrix between the to-be-matched sequence and each gesture template sequence is calculated; the local distance matrix has a size of MxN, where M is the length of the to-be-matched sequence, and N is the length of the gesture template sequence; each element D(λ,β) in the local distance matrix represents the distance between the to-be-matched sequence at time step λ and the gesture template sequence at time step β, which is usually calculated using Euclidean distance or other distance metrics.

[0178] Based on the local distance matrix, a cumulative distance matrix is calculated, and each element D'(λ,β) in the cumulative distance matrix represents the minimum cumulative distance from time step 0 to the to-be-matched sequence at time step λ and the gesture template sequence at time step β.

[0179] D'(λ,β) = D(λ,β) + min(D'(λ-1,β),D'(λ,β-1),D'(λ-1,β-1));

[0180] D'(λ-1, β) represents the minimum cumulative distance from 0 time step to the time step λ-1 of the to-be-matched sequence and the time step β of the gesture template sequence; D'(λ, β-1) represents the minimum cumulative distance from 0 time step to the time step λ of the to-be-matched sequence and the time step β-1 of the gesture template sequence; D'(λ-1, β-1) represents the minimum cumulative distance from 0 time step to the time step λ-1 of the to-be-matched sequence and the time step β-1 of the gesture template sequence.

[0181] This process realizes nonlinear alignment on the time axis, allowing a certain time distortion between sequences.

[0182] Starting from the right lower corner element of the cumulative distance matrix, backtrack along the direction of the minimum cumulative distance to find an optimal path from 0 time step to the right lower corner element; the optimal path corresponds to the best time alignment between the to-be-matched sequence and the gesture template sequence;

[0183] The sum of the minimum cumulative distances of all elements on the optimal path is the dynamic time warping distance between the to-be-matched sequence and the gesture template sequence; the smaller the dynamic time warping distance, the higher the similarity between the two sequences.

[0184] Calculate the dynamic time warping distance between the to-be-matched sequence and all gesture template sequences; select the gesture instruction mode corresponding to the gesture template sequence with the smallest dynamic time warping distance as the best matching gesture instruction mode; that is, complete matching.

[0185] Preferably, if all dynamic time warping distances exceed the preset distance threshold, it is determined as an unknown gesture; the unknown gesture is added to the gesture instruction mode.

[0186] It should be noted that the digital twin virtual scene is to combine digital twin and Web technology to develop an application based on a user's browser; it can be accessed and interacted through the Internet without installing special client software.

[0187] The embodiment of the present application realizes natural and convenient human-computer interaction, adopts a generative adversarial network to realize high-precision hand key point detection, improves the accuracy and robustness of gesture recognition, can adapt to different gestures and complex application scenarios, improves the precision of hand detection and key point positioning by adaptively adjusting the hand region frame and combining an ROI adjustment algorithm, reduces the recognition difficulty caused by hand movement, proposes a gesture recognition method based on a hand skeleton model and time twist distance, introduces time continuity analysis of gestures, can better represent the dynamic change of gestures by nonlinear alignment on a time axis, improves the accuracy of gesture classification, presets multiple gesture instruction modes and supports dynamic updating, provides personalized and extensible gesture interaction capability, enriches the mode of human-computer interaction, and combines gesture interaction with digital twin WEB application to expand the application scenario of gesture interaction.

[0188] Embodiment 2

[0189] Please refer to Figure 2 The embodiment of the present application provides a digital twin WEB application mid-air gesture interaction system, which comprises:

[0190] A data acquisition and processing module is configured to acquire real-time video streams of a camera of a user terminal device, and pre-process the real-time video streams; and obtain X hand region frames.

[0191] A key point detection module is configured to input the X hand region frames into a preset hand key point detection model based on a generative adversarial network, and obtain hand key point coordinate data.

[0192] A motion detection module is configured to obtain a hand skeleton model based on the hand key point coordinate data, and obtain joint angles and a hand motion trajectory based on the hand skeleton model.

[0193] A best matching module is configured to map the hand skeleton model into a digital twin virtual scene, preset n gesture instruction modes in the digital twin virtual scene, and match the hand skeleton model, the joint angles and the hand motion trajectory with the gesture instruction modes; the modules are connected through wired and / or wireless modes to realize data transmission between the modules.

[0194] Embodiment 3

[0195] The embodiment of the present application discloses an electronic device, which comprises a memory, a processor and a computer program stored in the memory and capable of running on the processor, and the processor implements the running mode of the digital twin WEB application mid-air gesture interaction method provided by the above-mentioned computer program.

[0196] Since the electronic device introduced in the embodiment is the electronic device used in the implementation of the method for gesture interaction in the digital twin WEB application, based on the method for gesture interaction in the digital twin WEB application introduced in the embodiment, those skilled in the art can understand the specific implementation of the electronic device of the embodiment and its various forms, so the implementation of the method in the embodiment by the electronic device is not described in detail. As long as the electronic device used in the method for gesture interaction in the digital twin WEB application is implemented by those skilled in the art, it belongs to the scope of protection of the present application.

[0197] The above formulas are dimensionless values, and the formulas are obtained by collecting a large amount of data to simulate the most recent real situation. The preset parameters and threshold values in the formula are set by those skilled in the art according to the actual situation.

[0198] The above is only the preferred embodiment of the present application, and the protection scope of the present application is not limited to the above-mentioned embodiments. Any technical solution falling within the scope of the present application is within the protection scope of the present application. It should be noted that, for ordinary technical users in the technical field, some improvements and refinements without departing from the principles of the present application are also considered to be within the protection scope of the present application.

Claims

1. A digital twin WEB application air gesture interaction method, characterized in that, The method comprises the following steps: S1, collecting a real-time video stream of a camera of a user terminal device, and preprocessing the real-time video stream; X hand region frames are obtained; The preprocessing manner of the real-time video stream comprises: extracting image frames from the real-time video stream frame by frame; inputting the image frames into a pre-trained hand detection model to obtain the coordinates of the bounding box of the hand; cropping the hand region frame containing only the hand from the original image frame according to the coordinates of the bounding box; and adjusting the size and position of the hand region frame in real time by using an ROI adjustment algorithm; The manner of adjusting the size and position of the hand region frame in real time comprises: tracking the position of the hand in the continuous image frames by using Kalman filtering or motion estimation algorithm; obtaining the displacement and motion speed of the hand in the coordinate system of the image frame; the displacement comprises vertical displacement and horizontal displacement; the motion speed comprises horizontal motion speed and vertical motion speed; The size of the initial hand region box is set to , the size of the initial hand region box is around the detected hand bounding box, and a fixed edge allowance is left; adjusting the size of the hand region frame according to the motion speed; The formula for adjusting the size of the hand area frame is: ;in, This is the adjustment coefficient; To be based on time frames The width of the changing hand area frame; To be based on time frames The height of the changing hand area frame; The horizontal velocity of the hand within the coordinate system of the image frame; The vertical velocity of the hand within the coordinate system of the image frame; adjusting the center coordinates of the hand region frame according to the displacement; The formula for adjusting the center coordinates of the hand region frame is: ; wherein, is the horizontal coordinate of the center coordinates of the hand region frame that changes according to the time frame ; and is the vertical coordinate of the center coordinates of the hand region frame that changes according to the time frame ; and is the horizontal coordinate of the hand region frame of the previous time frame; is the vertical coordinate of the hand region frame of the previous time frame; is the horizontal displacement; is the vertical displacement; setting the maximum size threshold and the minimum size threshold of the hand region frame; if the size of the adjusted hand region frame exceeds the range of the maximum size threshold and the minimum size threshold, the hand region frame is cropped to the maximum size threshold or the minimum size threshold; using the adjusted size of the hand region box and the center coordinates as the initial hand region box for the next image frame S2, inputting the X hand region frames into a preset hand key point detection model based on a generative adversarial network to obtain hand key point coordinate data; S3, obtaining a hand skeleton model based on the hand key point coordinate data; obtaining joint angles and hand motion trajectories according to the hand skeleton model; S4, mapping the hand skeleton model into a digital twin virtual scene; presetting n gesture instruction modes in the digital twin virtual scene; matching the hand skeleton model, the joint angles and the hand motion trajectories with the gesture instruction modes.

2. The digital twin WEB application mid-air gesture interaction method of claim 1, wherein, The training manner of the hand detection model comprises: initially defining a basic framework of the hand detection model, the basic framework comprising a backbone network, a feature pyramid network, a detection head sub-network and anchor boxes; The ResNet-101 network is used as a backbone network to output a feature map; the ResNet-101 network comprises phases, each phase comprising m residual blocks; and a feature pyramid network is constructed on the basis of the feature map output by the backbone network. designing an independent detection head sub-network for each level of pyramid feature map based on the feature pyramid network; the detection head comprises four convolution layers and two parallel fully connected layers; the two parallel fully connected layers comprise a classification fully connected layer and a regression fully connected layer; On each level of the pyramid feature map, 9 predefined anchor boxes are set for each pixel position; the size of the defined anchor box is {32 2 , 64 2 , 128 2 , 256 2 , 512 2} pixels; and the aspect ratio is {1:2, 1:1, 2:1}; defining a loss function of the hand detection model, which comprises a classification loss function and a regression loss function; Classification loss function ; wherein, is a tuning factor; Regression loss function ; collecting images containing k hands to form an image dataset; labeling the images in the image dataset to generate labels, the labels being the bounding box coordinates of the hands in each image, i.e. the annotation boxes; dividing the image dataset into U batches; inputting the batches into the hand detection model, and counting the values of the classification loss function and the regression loss function after each batch is inputted, and constantly adjusting the parameters in the hand detection model; so that the values of the classification loss function and the regression loss function no longer decrease in continuous Y batches, and the training is stopped; a trained hand detection model is obtained.

3. The digital twin WEB application mid-air gesture interaction method of claim 2, wherein, The formula of the output feature map is: ; wherein, is the feature map output by the residual block; is the input image frame; is the residual function; is the learnable parameter of the th residual block; ; where, denotes a convolution operation; is a rectified linear unit activation function; The construction manner of the feature pyramid network comprises: generating pyramid feature maps with 7 levels by constructing horizontal and vertical connection paths; The formula for horizontal connection is: ;in, No. Each level of pyramid feature map; Backbone Network Feature maps output at each stage; It is a positive integer greater than or equal to 1 and less than or equal to 7; Indicates will Perform a 1×1 convolution operation to change the number of channels to 256; Indicates an upsampling operation, Size upsampling to Same size; For the first Each level of pyramid feature map; The formula of vertical connection is: ; wherein, is the 6th level pyramid feature map; is the 7th level pyramid feature map; is the feature map output by the 5th stage of the backbone network; represents that the maximum pooling operation is performed on ; The classification full connection layer uses a Softmax activation function to calculate the probability of each anchor box belonging to a hand or background; The calculation formula of the probability is: ; wherein, is a transpose of a weight parameter of the classification fully connected layer ; wherein, is a feature vector from the feature pyramid network; is a bias parameter of the classification fully connected layer; is a probability that each anchor box of the prediction belongs to a hand or a background. The regression full connection layer has no activation function, and is used for obtaining a predicted offset of an anchor box to a label box .

4. The digital twin WEB application mid-air gesture interaction method of claim 3, wherein, The preset manner of the hand key point detection model includes: Defining the basic structure of the hand key point detection model includes generator network structure and discriminator network structure; and the generator network structure and the discriminator network structure are subjected to adversarial training; The generator network structure adopts an encoder-decoder structure, an input is an image of a hand region frame, and an output is a generated hand key point heat map ; the encoder is composed of a convolution layer and a down-sampling layer, and is used for extracting image features; and the decoder is composed of an up-sampling layer and a convolution layer, and is used for generating a hand key point heat map; The objective of the generator network architecture is to minimize the difference between the generated hand keypoint heatmaps and the real heatmaps The difference is computed using pixel-wise L1 or L2 loss ​ The input of the discriminator network structure is the hand key point heat map generated by the generator or the real heat map , and the output is the hand key point heat map or the real heat map is the real or generated probability value; The discriminator network structure adopts a convolutional neural network structure to perform feature extraction and classification on the input hand key point heat map or real heat map ​ The objective of the discriminator network structure is to maximize the probability that the real heat map is judged as real and the probability that the generated hand key point heat map is judged as generated; that is, a binary loss function of the discriminator network structure is defined ; ; wherein, is a real heat map the negative log expectation of the probability of being correctly judged as real; denotes the probability that a discriminator network structure considers is a real heat map is a generated hand keypoint heat map is the negative log expectation of the probability of being correctly judged as generated; represents the probability that the discriminator believes is the probability that the heat map is real. initializing parameters of the generator network structure and the discriminator network structure; collecting a training dataset containing hand images and corresponding real heat maps; for the training dataset, shuffling the training dataset and dividing into R batches; sampling by batch; for each batch, sampling real heat maps from the batch; inputting the current generator network structure into the hand image to generate ; inputting the real heat map and into the discriminator network structure; and calculating a binary loss function of the discriminator network structure; Sampling hand images from the batch; inputting the hand images into a generator network structure to generate ; inputting the generated images into a current discriminator network structure; calculating the loss of the generator network structure ; inputting the generated images into a current discriminator network structure; calculating the loss of the generator network structure ; iteratively updating the parameters of the generator network structure and the discriminator network structure and meeting the training target; that is, completing the preset of the hand key point detection model.

5. The digital twin WEB application mid-air gesture interaction method of claim 4, wherein, The adversarial training manner includes: The definition of the training target is to find a Nash equilibrium point to update the parameters of the generator network structure and the discriminator network structure; the Nash equilibrium point is to minimize the generated hand key point heat map while maximizing the difference between the real heat map and the generated heat map During training, the generator network structure and the discriminator network structure are alternately optimized, and the network parameters of each are updated; The hand motion trajectory and the hand three-dimensional model data acquisition manner includes: The three-dimensional coordinates of the hand key points are calculated by using the hand key point coordinate data in combination with the internal parameters of the camera of the user terminal device and the known hand size constraint; Based on the three-dimensional coordinates of the hand key points, a hand skeleton model is constructed; the hand skeleton model is composed of J skeleton chain nodes, and each skeleton chain node corresponds to a key point; the skeleton chain nodes are connected through virtual bones; The joint angle acquisition manner includes: For each group of two adjacent skeleton chain nodes, two bone vectors connected by the two skeleton chain nodes are acquired; the included angle of the two bone vectors, i.e., the joint angle, is calculated by using vector dot multiplication operation; the joint angle describes the posture and action of the hand; The hand motion trajectory acquisition manner includes: In the hand skeleton model, for each key point, a three-dimensional motion trajectory, i.e., a hand motion trajectory, is obtained by using a motion estimation algorithm based on continuous image frames.

6. The digital twin WEB application mid-air gesture interaction method of claim 5, wherein, The three-dimensional coordinates of the hand key points are calculated by using the hand key point coordinate data in combination with the internal parameters of the camera of the user terminal device and the known hand size constraint; Constructing an intrinsic parameter matrix based on intrinsic parameters of a camera of a user equipment ; ; wherein; is the horizontal focal length of the user equipment camera; is the vertical focal length of the user equipment camera; is the horizontal principal point coordinate of the user equipment camera; is the vertical principal point coordinate of the user equipment camera; Utilizing parameter matrix to process hand key point coordinate data Removing the scale factor to obtain normalized image plane coordinates; The formula for removing the scale factor is: ;in, For the first The abscissa of the planar coordinates of each key hand point image; For the first The vertical coordinate of the planar coordinates of each key hand point image; For the first The horizontal axis of each key hand point in the hand key point heatmap; For the first The vertical coordinates of each key hand point in the hand key point heatmap; Based on the three-dimensional coordinates of the hand key points, a hand skeleton model is constructed; the hand skeleton model is composed of J skeleton chain nodes, and each skeleton chain node corresponds to a key point; the skeleton chain nodes are connected through virtual bones; The formula for calculating the ruler vertical coordinate is: ; wherein, is a known distance constraint between the th and the th hand key point; is the ruler vertical coordinate of the th hand key point; is the ruler vertical coordinate of the th hand key point; The joint angle acquisition manner includes: The formula of the projection is: ; wherein, is the three-dimensional coordinate of the hand key point of the th hand key point.

7. The digital twin WEB application mid-air gesture interaction method of claim 6, wherein, For each group of two adjacent skeleton chain nodes, two bone vectors connected by the two skeleton chain nodes are acquired; the included angle of the two bone vectors, i.e., the joint angle, is calculated by using vector dot multiplication operation; the joint angle describes the posture and action of the hand; The hand motion trajectory acquisition manner includes: In the hand skeleton model, for each key point, a three-dimensional motion trajectory, i.e., a hand motion trajectory, is obtained by using a motion estimation algorithm based on continuous image frames. The three-dimensional coordinates of the hand key points are calculated by using the hand key point coordinate data in combination with the internal parameters of the camera of the user terminal device and the known hand size constraint; 8. The digital twin WEB application mid-air gesture interaction method of claim 7, wherein, Based on the three-dimensional coordinates of the hand key points, a hand skeleton model is constructed; the hand skeleton model is composed of J skeleton chain nodes, and each skeleton chain node corresponds to a key point; the skeleton chain nodes are connected through virtual bones; The joint angle acquisition manner includes: For each group of two adjacent skeleton chain nodes, two bone vectors connected by the two skeleton chain nodes are acquired; the included angle of the two bone vectors, i.e., the joint angle, is calculated by using vector dot multiplication operation; the joint angle describes the posture and action of the hand; The hand motion trajectory acquisition manner includes: In the hand skeleton model, for each key point, a three-dimensional motion trajectory, i.e., a hand motion trajectory, is obtained by using a motion estimation algorithm based on continuous image frames. The three-dimensional coordinates of the hand key points are calculated by using the hand key point coordinate data in combination with the internal parameters of the camera of the user terminal device and the known hand size constraint; Based on the three-dimensional coordinates of the hand key points, a hand skeleton model is constructed; the hand skeleton model is composed of J skeleton chain nodes, and each skeleton chain node corresponds to a key point; the skeleton chain nodes are connected through virtual bones; The joint angle acquisition manner includes: For each group of two adjacent skeleton chain nodes, two bone vectors connected by the two skeleton chain nodes are acquired; the included angle of the two bone vectors, i.e., the joint angle, is calculated by using vector dot multiplication operation; the joint angle describes the posture and action of the hand; The hand motion trajectory acquisition manner includes: In the hand skeleton model, for each key point, a three-dimensional motion trajectory, i.e., a hand motion trajectory, is obtained by using a motion estimation algorithm based on continuous image frames. The three-dimensional coordinates of the hand key points are calculated by using the hand key point coordinate data in combination with the internal parameters of the camera of the user terminal device and the known hand size constraint; Based on the three-dimensional coordinates of the hand key points, a hand skeleton model is constructed; the hand skeleton model is composed of J skeleton chain nodes, and each skeleton chain node corresponds to a key point; the skeleton chain nodes are connected through virtual bones; The joint angle acquisition manner includes: For each group of two adjacent skeleton chain nodes, two bone vectors connected by the two skeleton chain nodes are acquired; the included angle of the two bone vectors, i.e., the joint angle, is calculated by using vector dot multiplication operation; the joint angle describes the posture and action of the hand; The hand motion trajectory acquisition manner includes: In the hand skeleton model, for each key point, a three-dimensional motion trajectory, i.e., a hand motion trajectory, is obtained by using a motion estimation algorithm based on continuous image frames. The three-dimensional coordinates of the hand key points are calculated by using the hand key point coordinate data in combination with the internal parameters of the camera of the user terminal device and the known hand size constraint; Based on the three-dimensional coordinates of the hand key points, a hand skeleton model is constructed; the hand skeleton model is composed of J skeleton chain nodes, and each skeleton chain node corresponds to a key point; the skeleton chain nodes are connected through virtual bones; The joint angle acquisition manner includes: For each group of two adjacent skeleton chain nodes, two bone vectors connected by the two skeleton chain nodes are acquired; the included angle of the two bone vectors, i.e., the joint angle, is calculated by using vector dot multiplication operation; the joint angle describes the posture and action of the hand; The hand motion trajectory acquisition manner includes: In the hand skeleton model, for each key point, a three-dimensional motion trajectory, i.e., a hand motion trajectory, is obtained by using a motion estimation algorithm based on continuous image frames. The three-dimensional coordinates of the hand key points are calculated by using the hand key point coordinate data in combination with the internal parameters of the camera of the user terminal device and the known hand size constraint; Based on the three-dimensional coordinates of the hand key points, a hand skeleton model is constructed; the hand skeleton model is composed of J skeleton chain nodes, and each skeleton chain node corresponds to a key point; the skeleton chain nodes are connected through virtual bones; The joint angle acquisition manner includes: For each group of two adjacent skeleton chain nodes, two bone vectors connected by the two skeleton chain nodes are acquired; the included angle of the two bone vectors, i.e., the joint angle, is calculated by using vector dot multiplication operation; the joint angle describes the posture and action of the hand; The hand motion trajectory acquisition manner includes: In the hand skeleton model, for each key point, a three-dimensional motion trajectory, i.e., a hand motion trajectory, is obtained by using a motion estimation algorithm based on continuous image frames. The three-dimensional coordinates of the hand key points are calculated by using the hand key point coordinate data in combination with the internal parameters of the camera of the user terminal device and the known hand size constraint; Based on the three-dimensional coordinates of the hand key points, a hand skeleton model is constructed; the hand skeleton model is composed of J skeleton chain nodes, and each skeleton chain node corresponds to a key point; the skeleton chain nodes are connected through virtual bones; The joint angle acquisition manner includes: For each group of two adjacent skeleton chain nodes, two bone vectors connected by the two skeleton chain nodes are acquired; the included angle of the two bone vectors, i.e., the joint angle, is calculated by using vector dot multiplication operation; the joint angle describes the posture and action of the hand; The hand motion trajectory acquisition manner includes: In the hand skeleton model, for each key point, a three-dimensional motion trajectory, i.e., a hand motion trajectory, is obtained by using a motion estimation algorithm based on continuous image frames. The three-dimensional coordinates of the hand key points are calculated by using the hand key point coordinate data in combination with the internal parameters of the camera of the user terminal device and the known hand size constraint; Based on the three-dimensional coordinates of the hand key points, a hand skeleton model is constructed; the hand skeleton model is composed of J skeleton chain nodes, and each skeleton chain node corresponds to a key point; the skeleton chain nodes are connected through virtual bones; The joint angle acquisition manner includes: For each group of two adjacent skeleton chain nodes, two bone vectors connected by the two skeleton chain nodes are acquired; the included angle of the two bone vectors, i.e., the joint angle, is calculated by using vector dot multiplication operation; the joint angle describes the posture and action of the hand; The hand motion trajectory acquisition manner includes: In the hand skeleton model, for each key point, a three-dimensional motion trajectory, i.e., a hand motion trajectory, is obtained by using a motion estimation algorithm based on continuous image frames. The three-dimensional coordinates of the hand key points are calculated by using the hand key point coordinate data in combination with the internal parameters of the camera of the user terminal device and the known hand size constraint; Based on the three-dimensional coordinates of the hand key points, a hand skeleton model is constructed; the hand skeleton model is composed of J skeleton chain nodes, and each skeleton chain node corresponds to a key point; the skeleton chain nodes are connected through virtual bones; The joint angle acquisition calculating a local distance matrix between the sequence to be matched and each gesture template sequence; the local distance matrix has a size of wherein is the length of the sequence to be matched, is the length of the gesture template sequence; each element of the local distance matrix denotes the distance between the sequence to be matched at time step and the gesture template sequence at time step ; Based on the local distance matrix, a cumulative distance matrix is computed, where each element represents the minimum cumulative distance from time step 0 to the time step of the sequence to be matched; and the time step of the gesture template sequence. ;in, This indicates the time step from time step 0 to the time step of the sequence to be matched. With gesture template sequence at time step The minimum cumulative distance; This indicates the time step from time step 0 to the time step of the sequence to be matched. With gesture template sequence at time step The minimum cumulative distance; This indicates the time step from time step 0 to the time step of the sequence to be matched. With gesture template sequence at time step The minimum cumulative distance; Starting from the bottom right element of the accumulated distance matrix, backtracking along the direction of the minimum accumulated distance, find an optimal path from the 0 time step to the bottom right element; Obtain the sum of the minimum accumulated distances of all elements on the optimal path, which is the dynamic time warping distance between the to-be-matched sequence and the gesture template sequence; calculate the dynamic time warping distance between the to-be-matched sequence and all gesture template sequences; select the gesture instruction mode corresponding to the gesture template sequence with the minimum dynamic time warping distance as the best matching gesture instruction mode; that is, the matching is completed.

9. A digital twin WEB application mid-air gesture interaction system for implementing the digital twin WEB application mid-air gesture interaction method of claims 1-8, characterized in that, It comprises: A data acquisition and processing module is configured to acquire real-time video stream of a camera of a user terminal device and pre-process the real-time video stream; X hand region frames are obtained; A key point detection module is configured to input the X hand region frames into a preset hand key point detection model based on a generative adversarial network to obtain hand key point coordinate data; A motion detection module is configured to obtain a hand skeleton model based on the hand key point coordinate data, and obtain joint angles and a hand motion trajectory based on the hand skeleton model; An optimal matching module is configured to map the hand skeleton model into a digital twin virtual scene, predefine n gesture instruction modes in the digital twin virtual scene, and match the hand skeleton model, the joint angles and the hand motion trajectory with the gesture instruction modes; the modules are connected through wired and / or wireless means.

Images