A handwriting interaction system and method in a vision-based virtual reality scene

Abstract

The present invention discloses a hand-pen interaction system in a vision-based virtual reality scene, the system comprising: a hand 3D reconstruction module, a hand-pen interaction mapping module and an interaction presentation module; a hand-pen interaction method in a vision-based virtual reality scene, the method comprising: S1, acquiring a hand image from a user's perspective through a visual sensor; S2, inputting the hand image into a constructed graph neural network to perform 3D reconstruction of both hands to obtain a model; S3, presetting a grasping posture, and distinguishing between actual postures and preset postures to distinguish various minor gesture changes; S4, designing several easily distinguishable minor gestures, mapping them to various functions of a painting system, and forming a hand-pen interaction painting system. The present invention allows users to create intuitively in a virtual environment without the need for additional hardware, effectively reducing cognitive load and operational thresholds, and providing users with an immersive painting experience in conjunction with an audio-visual feedback strategy.

Description

Technical Field

[0001] The present invention relates to the fields of virtual reality and human-computer interaction, and in particular to a handwriting interaction system and method in a vision-based virtual reality scene. Background Art

[0002] With the rapid development of virtual reality technology, VR painting and sketching applications are increasingly popular among artists and designers. These applications provide users with an immersive creative environment and intuitive spatial interaction, transforming traditional creative methods. Hand-drawn sketches play a crucial role in the early stages of design, enabling the rapid exploration and communication of design concepts.

[0003] Existing VR drawing systems primarily rely on hardware-based interaction methods, such as six-degree-of-freedom handheld controllers, styluses, and VR sketchpads. These controller-based interactions require users to simultaneously monitor the on-screen cursor while simultaneously assessing the movement of devices outside their field of view. This creates a disconnect between visual input and physical action, increasing the user's cognitive load. This requires users to focus their attention to coordinate the two, hindering the most natural and intuitive drawing experience.

[0004] To address these issues, some researchers have begun exploring bare-hand interaction as an alternative to VR drawing. Dudley et al. explored the possibility of bare-hand 3D drawing in an AR environment. Vartak et al. proposed a virtual gesture drawing system that uses hand gestures to draw lines on the screen. Rajail et al. developed a machine learning method to enable virtual line drawing on the screen using hand gestures. These studies demonstrate the potential of bare-hand interaction to enhance the naturalness and expressiveness of VR drawing.

[0005] However, existing bare-hand VR painting systems often lack well-designed interaction paradigms to accurately recognize user intent and support complex interactions. Professional VR painting applications require users to use a variety of tools and controls, but existing systems have limited gesture recognition capabilities and struggle to provide precise painting control. This lack of natural, flexible, and precise hand interaction severely impacts the VR painting experience.

[0006] Furthermore, natural hand-drawing often involves frequent changes in brush strokes and adjustments to painting parameters, such as changing brush size and switching colors. Existing gesture interaction solutions struggle to smoothly and quickly support such fine-grained operations, resulting in frequent interruptions in the painting process. Furthermore, the instability of the user's hand gestures while gesturing in mid-air also poses challenges for gesture recognition.

[0007] Therefore, in order to achieve natural bare-hand interaction in VR painting, an advanced gesture interaction paradigm is urgently needed that can accurately identify user intentions, provide rich painting function mappings, and combine intelligent drawing assistance and feedback mechanisms to ultimately bring users a seamless, smooth, and fine-grained creative experience and unleash the creativity of VR painting.

[0008] Recent research has explored bare-hand interaction as an alternative to VR painting. However, existing bare-hand VR painting systems often lack well-designed interaction paradigms to accurately identify user intent and support complex interactive operations. The lack of natural, flexible, and precise hand interaction seriously hinders the VR painting experience. Summary of the Invention

[0009] The present invention provides a visual-based handwriting interaction system and method in a virtual reality scene. The present invention can be widely applied to scenes such as virtual reality painting and sketching, allowing users to create intuitively in a virtual environment without the need for additional hardware, effectively reducing cognitive load and operational barriers, and combining audio-visual feedback strategies to provide users with an immersive painting experience.

[0010] The specific technical solutions are as follows:

[0011] A hand-pen interaction system in a vision-based virtual reality scene, comprising: a hand 3D reconstruction module, a hand-pen interaction mapping module, and an interaction presentation module;

[0012] The hand three-dimensional reconstruction module is connected to the hand-pen interaction mapping module, and the hand-pen interaction mapping module is connected to the interaction presentation module.

[0013] Preferably, the hand 3D reconstruction module is used to collect RGB image sequences of the user's hands in real time through the RGB camera on the head-mounted display device, input them into the perturbation graph contrast learning network DGCLNet, extract the graph feature representation of the gesture through the graph U-Net, and optimize the graph features through the cosine similarity loss function to achieve accurate recognition of the gesture, and then map the gesture graph features to the vertex coordinates and joint rotation angles of the MANO hand model through the graph convolution upsampling network and the spatial transformation network to reconstruct the 3D surface shape and joint posture of the hand.

[0014] Preferably, the hand-pen interaction mapping module is used to map the recognized gestures with the interactive operations of the virtual brush, and realize the virtual pen's grabbing, menu calling, and undo operations through pinching and multi-finger combination gestures, thereby realizing natural interaction between bare hands and the virtual pen.

[0015] Preferably, the interactive presentation module is used to control the virtual brush to paint in the three-dimensional virtual scene according to the results of the hand-pen interaction, and at the same time present the three-dimensional model of the hand, the status of the virtual pen and the painting results in the virtual scene in real time, providing visual feedback, and giving audio prompts according to the interaction results to enhance the user's immersion and operation feedback.

[0016] Preferably, a method for handwriting interaction in a vision-based virtual reality scene includes:

[0017] S1, obtain the hand image from the user's perspective through the visual sensor;

[0018] S2. Input the hand image into the constructed graph neural network to perform three-dimensional reconstruction of both hands to obtain a model, wherein the model includes a backbone network, a neck network, and a prediction head;

[0019] S3. Create a virtual pen in a virtual reality scene and classify four pen-holding gestures by identifying subtle changes in each finger when holding the pen;

[0020] S4. Based on the four pen-holding gestures, map the basic functions of the system in the virtual reality scene to form a hand-pen interaction paradigm.

[0021] Preferably, S2 includes the following sub-steps:

[0022] Sub-step S21, obtaining images of the user's hands;

[0023] Sub-step S22: input the images of both hands into the backbone network to extract image features, and input the image features into two independent multi-layer perceptrons (MLPs) for left and right hands respectively for feature conversion;

[0024] Sub-step S23: constructing the features output by the multi-layer perceptron into graph features using the Laplacian matrix;

[0025] Sub-step S24: input the graph features into the perturbation graph contrast learning network DGCLNet, and regenerate the graph features through the graph U-Net to form feature pairs.

[0026] Sub-step S25: The feature pairs are passed through a shared graph convolutional network (GCN) to maximize the similarity of the two output features.

[0027] Sub-step S26: Input the graph features of the original branch into the upsampling layer, and obtain the three-dimensional coordinates of 778 vertices of both hands in a MANO manner;

[0028] Sub-step S27: introducing a similarity distinction method to accurately calculate the similarity of feature pairs from the feature-pose pool to accelerate model convergence;

[0029] Sub-step S28: designing multiple loss functions to balance the relationship between hand posture, visual scale and viewpoint position to improve the quality of 3D reconstruction;

[0030] Sub-step S29, iteratively optimize the model until convergence to obtain the final two-hand three-dimensional reconstruction graph neural network model.

[0031] Preferably, S3 includes the following sub-steps:

[0032] Sub-step S31: creating a virtual body of a pen in a virtual reality environment, and grabbing the virtual pen by presetting a grabbing gesture on the virtual pen;

[0033] Sub-step S32: when the hand interacts with the virtual pen, measuring the error between the virtual hand and each finger joint of the preset gesture, and calculating the average position error of each joint;

[0034] Sub-step S33: determining whether the finger has reached the preset gesture position by setting an error threshold for each finger joint position, and accurately identifying each finger gesture;

[0035] Sub-step S34, setting four finger gestures corresponding to four functions respectively, the four finger gestures include pinching the thumb and index finger, the thumb, index finger and middle finger reaching the preset gesture finger position at the same time, the thumb, index finger and ring finger bending position at the same time, and all fingers reaching the preset position at the same time.

[0036] Preferably, S4 includes the following sub-steps:

[0037] Sub-step S41, the position of the thumb and index finger pinching corresponds to the function of moving the virtual pen. The virtual pen is selected by pinching the index finger and thumb, and the selection is canceled by releasing the thumb and index finger, which is used to control the position of the virtual pen in three-dimensional space;

[0038] Sub-step S42: The thumb, index finger, and middle finger simultaneously reach the preset gesture finger positions corresponding to calling out the menu function;

[0039] Sub-step S43: The thumb, index finger and ring finger are bent at the same time to the corresponding position to cancel the previous operation function;

[0040] Sub-step S44: When all fingers reach the preset position at the same time, it corresponds to the drawing operation.

[0041] The beneficial effects of the handwriting interaction system and method in a visual virtual reality scene of the present invention are as follows:

[0042] 1. The present invention integrates the main operations in the drawing process into a single-handed compound hand-pen interaction paradigm, simplifying the drawing operations in three-dimensional space.

[0043] 2. This hand-pen interaction system can be widely used in virtual reality painting, sketching and other scenarios, allowing users to create intuitively in a virtual environment without the need for additional hardware, effectively reducing cognitive load and operational barriers, and combined with audio-visual feedback strategies to provide users with an immersive painting experience.

[0044] 3. The present invention has broad application prospects in the field of virtual reality interaction. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 It is the overall flow chart of the present invention.

[0046] Figure 2 It is a schematic diagram of the overall architecture of the graph neural network of the present invention.

[0047] Figure 3 It is a schematic diagram of handwriting interaction of the present invention.

[0048] Figure 4 This is a structural diagram of the handwriting interaction paradigm of the present invention.

[0049] Figure 5 It is a schematic diagram of preventing the pen tip from penetrating the canvas according to the present invention.

[0050] Figure 6 Schematic diagram of visual feedback of the present invention.

[0051] Figure 7 2 is a schematic diagram of auditory feedback of the present invention.

[0052] Figure 8 It is a schematic diagram of the results created by the users of the present invention in this system. DETAILED DESCRIPTION

[0053] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0054] The present invention provides a visual-based handwriting interaction method in a virtual reality scene, comprising:

[0055] S1, obtain the hand image from the user's perspective through the visual sensor;

[0056] S2. Input the hand image into the constructed graph neural network to perform 3D reconstruction of both hands to obtain a model, where the model includes a backbone network, a neck network, and a prediction head;

[0057] S3. Create a virtual pen in a virtual reality scene and classify four pen-holding gestures by identifying subtle changes in each finger when holding the pen;

[0058] S4. Based on the four pen-holding gestures, map the basic functions of the system in the virtual reality scene to form a hand-pen interaction paradigm.

[0059] S2 of this embodiment includes:

[0060] S21, obtaining images of the user's hands;

[0061] S22, input the images of both hands into the backbone network to extract image features, and then input the features into two independent multi-layer perceptrons (MLPs) for feature conversion;

[0062] S23. Use the Laplacian matrix to construct the features output by the MLP into graph features;

[0063] S24. Input the graph features into the perturbation graph contrastive learning network DGCLNet, regenerate the graph features through the graph U-Net to form feature pairs, and pass the feature pairs through the shared graph convolutional network GCN to maximize the similarity of the two output features;

[0064] S25. Input the graph features of the original branch into the upsampling layer and obtain the three-dimensional coordinates of 778 vertices of both hands in a MANO manner;

[0065] S26. Introducing a similarity distinction method to accurately calculate the similarity of feature pairs from the feature-pose pool and accelerate model convergence;

[0066] S27. Design multiple loss functions to balance the relationship between hand posture, visual scale, and viewpoint position to improve the quality of 3D reconstruction. The loss functions include:

[0067] (1) The main training target losses include: 3D hand joint loss:

[0068]

[0069] 3D hand mesh vertex loss:

[0070]

[0071] 2D hand joint loss:

[0072]

[0073] (2) Auxiliary training target losses include:

[0074] Length loss:

[0075]

[0076] Consistency loss:

[0077]

[0078] Conversion loss:

[0079]

[0080] (3) Comparative training target losses include:

[0081] Cross-pose contrast loss:

[0082]

[0083] Self-pose contrast loss:

[0084]

[0085] in, Represent the true coordinates of the i-th 3D joint point, 3D vertex and 2D joint point respectively, is the corresponding predicted coordinate, ε is the edge set of the hand skeleton topology. R, T are the predicted hand rotation matrix and translation vector, R canonical , is the true value, f(·) is the graph feature mapping function learned by DGCLNet, H, H′, H + , H - denote the features of the original image, perturbation image, positive sample image, and negative sample image respectively. τ is the temperature hyperparameter.

[0086] S28. Iteratively optimize the model until convergence to obtain the final two-hand 3D reconstruction graph neural network model.

[0087] S22 of this embodiment includes:

[0088] S221. Use the pre-trained ResNet-50 as the backbone network, input the RGB images of both hands into the backbone network, and extract high-dimensional image features through convolution and pooling operations;

[0089] S222, the high-dimensional features output by the backbone network are input into two independent multi-layer perceptrons (MLPs) corresponding to the left and right hands for nonlinear feature transformation;

[0090] S223, Multi-layer Perceptron (MLP) consists of three fully connected layers. After each fully connected layer, ReLU activation function and Dropout regularization are used to convert the image features extracted by the backbone network into a compact feature representation. Let F be the image features extracted by the backbone network, and MLP can be expressed as:

[0091] H=σ(W3·σ(W2·σ(W1·F+b1)+b2)+b3) (9)

[0092] Among them, W i and b i are the weight matrix and bias vector of the i-th fully connected layer, and σ is the ReLU activation function.

[0093] S224, the output dimension of the last fully connected layer is set to twice the number of target key points, and the left and right hand MLPs output the two-dimensional coordinates of the key points corresponding to the left and right hands respectively:

[0094]

[0095] in, and are the parameters of the last fully connected layer of the left and right hands, K left ∈R 2n and K right ∈R 2n are the two-dimensional coordinates of N key points of the left and right hands.

[0096] S23 of this embodiment includes:

[0097] S231. Based on the skeleton topology of the hand key points, define an undirected graph G = (V, E), where V is the set of key points and E is the set of edges between the key points;

[0098] S232, use the two-dimensional coordinates of the key points as the initial features of the node, construct the adjacency matrix A, if there is an edge between two key points, then A ij =1, otherwise A ij =0;

[0099] S233, calculate the degree matrix D of the graph, D is a diagonal matrix, the diagonal elements D ii is equal to the degree of node i, that is, the number of edges connected to node i;

[0100] S234. Calculate the Laplacian matrix L = DA of the graph. The Laplacian matrix is ​​a symmetric matrix that contains the topological structure information of the graph.

[0101] S235. Use the Laplacian matrix L to perform graph convolution on the node feature X output by the MLP to obtain a new node feature H:

[0102]

[0103] in, To add a self-loop adjacency matrix, for W is the degree matrix of graph convolution, and H captures the node features and graph structure information.

[0104] S24 of this embodiment includes:

[0105] S241. Input the graph feature H obtained in S23 into the perturbation graph contrastive learning network DGCLNet. DGCLNet consists of a graph U-Net and a graph convolutional network GCN.

[0106] S242 and Graph U-Net use an encoder-decoder structure. The encoder consists of multiple graph convolutional layers and pooling layers, which downsample the graph features into multi-scale graph representations; the decoder restores the resolution of the graph features through upsampling and skip connections:

[0107]

[0108] Among them, H (l) is the output feature of the l-th layer decoder, is the operation of the l-th layer decoder, including upsampling and skip connection, H (l-1) is the output of the Ll-th layer of the encoder, which is concatenated with the decoder features through a skip connection.

[0109] S243. Graph U-Net outputs a graph representation H′ with the same feature scale as the original graph, and (H, H′) is used as a positive sample pair. At the same time, the original graph H is perturbed by data enhancement methods such as node permutation to obtain a negative sample H-, and (H, H-) is used as a negative sample pair.

[0110] S244, the positive and negative sample pairs (H, H′) and (H, H-) are respectively input into two parameter-sharing GCNs. GCN aggregates the neighborhood information of the node in each graph convolution layer and extracts high-level semantic features:

[0111]

[0112] Among them, H (l) is the output feature of the l-th layer decoder, W (l) is the weight matrix of the lth layer of graph convolution, H (l-1) is the output of the Llth layer of the encoder, which is concatenated with the decoder features through a jump connection. To add a self-loop adjacency matrix, for The degree matrix of .

[0113] The node features output by the last layer of S245 and GCN are globally pooled to obtain graph-level representations f(H), f(H′), f(H - ), through the cosine similarity s(f(H), f(H′)) and s(f(H), f(H -))Calculate the similarity between positive and negative sample pairs:

[0114]

[0115] Among them, τ is the temperature hyperparameter, k is the number of negative samples, and by minimizing the contrast loss, DGCLNet can learn robust graph representations.

[0116] S246. Maximize the similarity s(f(H), f(H′)) of the positive sample pairs and minimize the similarity s(f(H), f(H-)) of the negative sample pairs. By comparing the learning objective function, we optimize the parameters of the graph U-Net and GCN, so that DGCLNet can learn a robust graph representation.

[0117] S25 of this embodiment includes:

[0118] S251, the graph feature H learned by the original branch of DGCLNet mano Input into the multi-layer graph convolution upsampling network, and gradually restore the resolution of the graph features through the transposed convolution operation;

[0119] S252, the graph convolution upsampling network outputs high-resolution graph features that are consistent with the number of vertices of the MANO hand model template mesh, and propagates the features to each vertex of the MANO mesh through graph convolution:

[0120]

[0121] in, The operation of the upsampling layer of the lth layer includes graph transposed convolution and graph convolution.

[0122] The S253 and MANO template meshes contain 778 vertices for the left and right hands, respectively. The three-dimensional coordinates of the vertices define the three-dimensional shape of the hand surface;

[0123] S254. Vertex features are input into the spatial transformation network to predict the 3D offset of each vertex relative to the hand root joint and the 3D rotation angle of each joint:

[0124] ΔV=f V (H mano ), θ=f θ (H mano ) (16)

[0125] Among them, f V and f θ Prediction networks for vertex offsets and joint rotation angles, respectively.

[0126] S255. Based on the predicted vertex offsets and joint rotation angles, the MANO template mesh is deformed using the linear blend skinning (LBS) algorithm to obtain the 3D surface shape and joint posture of the hand:

[0127] V=LBS(T,θ,ΔV) (17)

[0128] Where T is the template mesh of the MANO hand model, LBS is the linear blending skinning function, θ is the parameter that controls the deformation of the hand model, and ΔV is the three-dimensional rotation angle of each joint.

[0129] S256, finally output the three-dimensional coordinates of 778 vertices on the left and right hands (V left , V right ), reconstructing a 3D hand model corresponding to the input image. The parameterized representation of MANO makes it possible to control the hand model with low-dimensional posture and shape parameters to generate realistic 3D hand shapes.

[0130] Creating a virtual pen to recognize subtle gesture changes in S3 of this embodiment includes:

[0131] S31. creating a virtual pen in a virtual reality environment, and grabbing the virtual pen by presetting a grabbing gesture on the pen;

[0132] S32, when the hand interacts with the pen, calculating the error between each finger joint of the virtual hand and the preset gesture, and calculating the average position error of each joint;

[0133] S33. By setting an error threshold for each finger, determining whether each finger has reached a preset gesture position, four different pen-holding finger gestures are classified, and additional constraints are introduced to reduce misjudgment caused by occlusion;

[0134] S34. Divide the pen-holding finger gestures into four types corresponding to four functions, including pinching the thumb and index finger together, the thumb, index finger and middle finger reaching the preset gesture finger positions at the same time, the thumb, index finger and ring finger bending at the same time, and all fingers reaching the preset positions at the same time.

[0135] S31 of this embodiment includes:

[0136] S311. Simply construct a three-dimensional virtual pen model in a virtual reality scene;

[0137] S312, recording the hand posture during grasping and combining it with the virtual pen to form a new three-dimensional object;

[0138] S313. Pinching the index finger and thumb together is a necessary condition for grabbing the virtual pen. When the hand approaches the virtual pen and performs a pinching action, the recorded gesture will cause the virtual pen to automatically be adsorbed onto the user's virtual hand.

[0139] S32 of this embodiment includes:

[0140] S321. When the user grabs the virtual pen, except for the index finger and thumb which are in the preset gesture positions, the remaining fingers can be bent / straightened at will;

[0141] S322: Calculate the distance error between the middle finger, ring finger, and little finger of the virtual hand and the knuckles of the preset gesture using the following formula:

[0142] E ij =||p i -p j ||2, i, j∈{1, 2,...,N}, i≠j(18)

[0143] Assume that the set of hand key points is P = {p i |i=1,2,...,N},where p i Represents the three-dimensional coordinates of the i-th key point.

[0144] S33 of this embodiment includes:

[0145] S331, when the average joint error between the virtual hand fingers and the actual hand fingers is less than a certain error threshold (for example: 0.05cm), that is:

[0146]

[0147] If the finger reaches the preset gesture position, it is considered that the finger has reached the preset gesture position, otherwise it has not reached the preset position. By setting different threshold combination conditions, a variety of grasping micro-gestures can be defined, such as:

[0148] Pinch with thumb and index finger:

[0149]

[0150] The thumb, index finger, and middle finger reach the preset position at the same time:

[0151]

[0152] The thumb, index finger, and ring finger all reach the preset position at the same time:

[0153]

[0154] All fingers reach the preset posture:

[0155]

[0156] S332 of this embodiment includes:

[0157] First, we define the hub state i(Hub) as the default state for the entire interaction and the entry point for each subtask. In this state, the user can move the virtual pen naturally without triggering any special actions. When the system detects that the user has switched to another gesture state, it enters the corresponding subtask mode.

[0158] When identifying each spoke, we introduced additional constraints to reduce misjudgments caused by occlusion:

[0159] C1 state iv: requires all five fingers to simultaneously reach or exceed the preset gesture position (which can be a fist), and has the highest priority. This full-grip gesture is easier to recognize and has a low misjudgment rate.

[0160] C2 state iii: The little finger must be in an extended state. In addition to the three required fingers being bent, the middle finger will still maintain its original state when it is contracted together, which can effectively distinguish between iii and iv.

[0161] C3 state ii: The ring finger and little finger must be extended, while the middle finger is bent. In this state, only the bending state of one finger needs to be judged, and the recognition accuracy is relatively high.

[0162] C4 Delay Protection: A certain delay is set for switching between various states. This can effectively prevent the transition from the radial state to the central state and then immediately returning to a radial state, giving users enough time to determine the next interaction to be performed.

[0163] The above method divides four basic pen-holding finger posture interaction paradigms, which can be applied to most hand-object interaction systems.

[0164] S4 of this embodiment includes:

[0165] S41, the pinching position of the thumb and index finger corresponds to the function of moving the virtual pen. The virtual pen is selected by pinching the index finger and thumb, and the selection is canceled by releasing the index finger and thumb. It is used to control the position of the virtual pen in three-dimensional space;

[0166] S42: The thumb, index finger and middle finger simultaneously reach the preset gesture position corresponding to the call-out menu function;

[0167] S43: The position where the thumb, index finger and ring finger are bent at the same time corresponds to the function of undoing the last operation;

[0168] S44. When all fingers reach the preset gesture position at the same time, a drawing operation is performed.

[0169] S42 of this embodiment includes:

[0170] S421, a visual menu panel that can be clicked by the user's index finger;

[0171] S422, a color adjustment button, which expands a hexadecimal color palette after being clicked, and the user can obtain the desired color by clicking the palette;

[0172] S423. Copy, modify handwriting style, thickness, delete, combine and free shape operation buttons.

[0173] The present invention provides a handwriting interaction system in a vision-based virtual reality scene, the system comprising:

[0174] The hand image acquisition module is used to collect RGB image sequences of the user's hands in real time through the RGB camera on the head-mounted display device as input for subsequent gesture recognition and 3D reconstruction;

[0175] The gesture recognition module is used to input the acquired RGB images of both hands into the perturbation graph contrastive learning network (DGCLNet), extract the graph feature representation of the gesture through the graph U-Net, and optimize the graph features through the cosine similarity loss function to achieve accurate recognition of the gesture;

[0176] The gesture recognition module includes a 3D reconstruction module, which uses the MANO branch in DGCLNet to map the gesture graph features to the vertex coordinates and joint rotation angles of the MANO hand model through a graph convolutional upsampling network and a spatial transformation network, thereby reconstructing the 3D surface shape and joint posture of the hand.

[0177] The hand-pen interaction mapping module is used to map recognized gestures to interactive operations of the virtual brush. Through gestures such as pinching and multi-finger combination, virtual pen operations such as grabbing, menu calling, and undoing are realized, achieving natural interaction between bare hands and the virtual pen;

[0178] The hand-pen interaction mapping module includes a painting control module, which is used to control the virtual brush to paint in the 3D virtual scene based on the results of hand-pen interaction, including controlling the movement of the brush, drawing lines, controlling the size and position of the drawn object, adjusting colors, canvas interaction, etc., to realize bare-hand painting control;

[0179] The presentation module is used to present the three-dimensional model of the hand, the status of the virtual pen, and the drawing results in real time in the virtual scene, providing visual feedback, and giving audio prompts based on the interaction results to enhance the user's immersion and operation feedback.

[0180] Each module interacts with each other through data and control signals, working together to realize a vision-based virtual reality bare-hand interactive drawing system. Users can wear a VR headset and use their hands to interact naturally with the virtual pen, controlling it through gestures to create drawings in a 3D virtual space.

[0181] The hand three-dimensional reconstruction module is connected to the hand-pen interaction mapping module, and the hand-pen interaction mapping module is connected to the interaction presentation module.

[0182] like Figure 1 As shown, the embodiment of the present invention provides a hand-pen interaction method in a virtual reality scene based on vision. By creating two hands in real time, the average finger joint error between the preset grasping gesture and the created gesture is calculated, and four different pen-holding finger postures are distinguished to form a hand-pen interaction paradigm. The four pen-holding finger postures are mapped to four basic painting system functions in a virtual reality three-dimensional scene and integrated into the hand-pen interaction system. The specific network details and hand-pen interaction paradigm are shown in FIG. Figure 2 、 Figure 3 .

[0183] like Figure 2 The following diagram shows the network architecture for virtual hand reconstruction. Given an RGB image, our model uses ResNet-50 for feature extraction and then classifies the features into left and right hands through position embedding. Features of both hands are further processed through two separate MLP layers, and graph features are constructed using knowledge of the Laplacian matrix. Next, our DGCLNet perturbs the graph features by regenerating them through a graph u-net. Pairs of graph features are passed through the same GCN to maximize their similarity. Finally, the graph features from the original branch are fed into an upsampling layer to obtain the positions of 778 vertices using the standard MANO approach.

[0184] like Figure 3 The design concept for the pen-based interaction paradigm combines common drawing operations in 3D space (moving, calling up menus, undoing, drawing) with the pen-based interaction paradigm. Using a virtual pen as a proxy for visually identifying changes in finger gestures, users can create intuitively and freely in 3D space without additional hardware input.

[0185] like Figure 4 Figure 1: The framework of the hand-pen interaction paradigm. (Left 1) The gesture column represents the basic gestures of our interaction system. (Left 2) Finger gestures are used to control the drawing process of the virtual pen. (Right 2) Actions represent some interactive actions that users can perform using these basic gestures. (Right 1) Referring to the research of Li et al., we set up several types of pen grips for interaction with the pen: TFE, TRE, QRE, Pinch, and Overhand, to meet the pen grip needs of different users.

[0186] like Figure 5 As shown in the figure, to prevent the pen tip from penetrating the canvas, we add ray detection at the pen tip. When the pen approaches the canvas, it triggers a detection to see if the pen tip is in front of or behind the canvas, and calculates the distance and difference vector from the pen tip to the plane.

[0187] First, we calculate the projection point of the pen tip on the canvas plane using the vector Indicates the direction vector of the pen tip pointing to the pen handle It can be expressed as:

[0188]

[0189] in, is the position vector of any point on the plane, is the normal vector of the drawing board plane, P pen is the coordinate of the center of the pen. The signed distance d from the pen tip coordinate P to the canvas plane can be calculated using the point-to-plane distance formula:

[0190]

[0191] If the pen tip is at the back of the canvas and the distance is greater than the threshold, the pen is corrected to the front of the canvas and kept at a certain distance from the canvas plane to simulate the feeling of a pen hitting paper in real life. At this time, the pen position P′ pen Set to:

[0192]

[0193] Here, ∈ is a small positive number that is used to maintain a certain distance between the pen tip and the canvas to avoid overlap. This prevents the pen from penetrating the canvas and improves the realism of the interaction.

[0194] like Figure 6 The figure shows the visual feedback of the bare-hand pen interaction system in a vision-based virtual reality scenario: a) The virtual pen will not penetrate the blackboard. When the real hand goes deep into the blackboard, the virtual pen can sense the user's strength and increase the width of the handwriting; b) When adjusting the color, the virtual pen will change with the color set on the palette; c)-e) are the color changes when the user grabs an object. The purpose is to show the user the currently hovered / selected object, c) is the original color of the object, c) is the original color of the object, d) is the hovering color, and e) is the color of the selected object.

[0195] like Figure 7 Figure 2 shows auditory feedback from a bare-hand pen interaction system in a vision-based virtual reality scenario: the virtual pen emits different tones when switching states, prompting the user to change states. (a) When calling out a menu, b) when releasing the virtual pen, and c) when starting to draw.

[0196] like Figure 8 Shown is a collection of works freely created using a bare-hand pen interaction system in a vision-based virtual reality scene.

[0197] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A handwriting interaction method in a vision-based virtual reality scene, characterized in that: The method comprises: S1, obtain the hand image from the user's perspective through the visual sensor; S2. Input the hand image into the constructed graph neural network to perform three-dimensional reconstruction of both hands to obtain a model, wherein the model includes a backbone network, a neck network, and a prediction head; S3. Create a virtual pen in a virtual reality scene and classify four pen-holding gestures by identifying subtle changes in each finger when holding the pen; S4. Based on the four pen-holding gestures, map the basic functions of the system in the virtual reality scene to form a hand-pen interaction paradigm; To prevent the pen tip from penetrating the canvas, a ray detection function is added to the pen tip. When the pen approaches the canvas, it will trigger a detection to see if the pen tip is in front of or behind the canvas, and calculate the distance and vector from the pen tip to the plane. First, calculate the projection point of the pen tip on the canvas plane, using vector Indicates the direction vector of the pen tip pointing to the pen handle , which can be expressed as: in, is the position vector of any point on the plane, is the normal vector of the canvas plane, is the position coordinate of the middle of the pen, the pen tip coordinate Signed distance to the canvas plane The distance from a point to a plane can be calculated using the formula: ； If the pen tip is at the back of the canvas and the distance is greater than the threshold, the pen is corrected to the front of the canvas and kept at a certain distance from the canvas plane to simulate the feeling of a pen hitting paper in real life. At this time, the pen position Set to: in, is a positive number.

2. The visual-based handwriting interaction method in a virtual reality scene according to claim 1, characterized in that: The S2 includes the following sub-steps: Sub-step S21, obtaining images of the user's hands; Sub-step S22: input the images of both hands into the backbone network to extract image features, and input the image features into two independent multi-layer perceptrons (MLPs) for left and right hands respectively for feature conversion; Sub-step S23: constructing the features output by the multi-layer perceptron into graph features using the Laplacian matrix; Sub-step S24: input the graph features into the perturbation graph contrast learning network DGCLNet, and regenerate the graph features through the graph U-Net to form feature pairs. Sub-step S25: The feature pairs are passed through a shared graph convolutional network (GCN) to maximize the similarity of the two output features. Sub-step S26: Input the graph features of the original branch into the upsampling layer, and obtain the three-dimensional coordinates of 778 vertices of both hands in a MANO manner; Sub-step S27: introducing a similarity distinction method to accurately calculate the similarity of feature pairs from the feature-pose pool to accelerate model convergence; Sub-step S28: designing multiple loss functions to balance the relationship between hand posture, visual scale and viewpoint position to improve the quality of 3D reconstruction; Sub-step S29, iteratively optimize the model until convergence to obtain the final two-hand three-dimensional reconstruction graph neural network model.

3. The method for handwriting interaction in a vision-based virtual reality scene according to claim 1, characterized in that: The S3 includes the following sub-steps: Sub-step S31: creating a virtual body of a pen in a virtual reality environment, and grabbing the virtual pen by presetting a grabbing gesture on the virtual pen; Sub-step S32: when the hand interacts with the virtual pen, measuring the error between the virtual hand and each finger joint of the preset gesture, and calculating the average position error of each joint; Sub-step S33: determining whether the finger has reached the preset gesture position by setting an error threshold for each finger joint position, and accurately identifying each finger gesture; Sub-step S34, setting four finger gestures corresponding to four functions respectively, the four finger gestures include pinching the thumb and index finger, the thumb, index finger and middle finger reaching the preset gesture finger position at the same time, the thumb, index finger and ring finger bending position at the same time, and all fingers reaching the preset position at the same time.

4. The method for handwriting interaction in a vision-based virtual reality scene according to claim 1, characterized in that: The S4 includes the following sub-steps: Sub-step S41, the thumb and index finger pinch position corresponds to the mobile virtual pen function, by the index finger and thumb pinch to select the virtual pen, release to cancel the selection, used to control the virtual pen in three-dimensional space posture; Sub-step S42, the thumb, index finger and middle finger simultaneously reach the preset gesture finger position corresponding to the call-out menu function; Sub-step S43, the thumb, index finger and ring finger are bent at the same time to the corresponding position to undo the previous operation function; Sub-step S44: When all fingers reach the preset position at the same time, it corresponds to the drawing operation.

5. A handwriting interaction system in a vision-based virtual reality scene, characterized in that: The method according to any one of claims 1 to 4 is performed, wherein the system comprises: a hand three-dimensional reconstruction module, a hand-pen interaction mapping module, and an interaction presentation module; The hand 3D reconstruction module is connected to the hand gesture interaction mapping module, and the hand gesture interaction mapping module is connected to the interaction presentation module. The 3D hand reconstruction module is used to collect RGB image sequences of the user's hands in real time through the RGB camera on the head-mounted display device, input them into the perturbation graph contrast learning network DGCLNet, extract the graph feature representation of the gesture through the graph U-Net, and optimize the graph features through the cosine similarity loss function to achieve accurate recognition of the gesture. Then, through the graph convolution upsampling network and spatial transformation network, the gesture graph features are mapped to the vertex coordinates and joint rotation angles of the MANO hand model to reconstruct the 3D surface shape and joint posture of the hand. The hand-pen interaction mapping module is used to map the recognized gestures with the interactive operations of the virtual brush, and realize the virtual pen grabbing, menu calling, and undo operations through pinching and multi-finger combination gestures, so as to achieve natural interaction between bare hands and the virtual pen. The interactive presentation module is used to control the virtual brush to paint in the three-dimensional virtual scene based on the results of the hand-pen interaction, and at the same time present the three-dimensional model of the hand, the status of the virtual pen, and the painting results in the virtual scene in real time, providing visual feedback and giving audio prompts based on the interaction results, thereby enhancing the user's immersion and operation feedback.

Images