AR space sand table interaction method based on gesture control

Abstract

The invention relates to an AR space sand table interaction method based on gesture control. The method comprises the following steps: S1, displaying an AR digital sand table in a seen space; s2, identifying gesture finger joints of the user; s3, according to the corresponding relation between the gesture of the user and the sand table object, recognizing the holding and placing actions; s4, dragging an object or an icon of the digital sand table; and S5, simulating drilling according to the dragging action of the user. According to the method, the control on the virtual object is realized by identifying the gesture action of the user, and the intuition and naturalness of interaction are improved. In addition, according to the method, personalized setting can be carried out according to habits and preferences of the users, and the requirements of different users are met.

Description

Technical Field

[0001] The present invention relates to the field of augmented reality technology, and in particular to an AR space sandbox interaction method and system based on gesture control. Background Art

[0002] Augmented reality technology is a technology that combines virtual information with the real world. It can control virtual objects by recognizing user gestures. In the military field, AR space sandboxes are often used for simulation exercises to improve the efficiency and accuracy of combat command. However, traditional AR space sandbox interaction methods usually require operation through devices such as mice and touch screens. The operation process is relatively cumbersome and the user experience needs to be improved. Summary of the invention

[0003] The present invention aims to provide an AR space sandbox interaction method and system based on gesture control, and the technical problems to be solved include at least how to improve the intuitiveness and naturalness of the interaction and enhance the user experience.

[0004] In order to achieve the above object, the present invention provides an AR space sandbox interaction method based on gesture control, comprising the following steps:

[0005] S1. Display the AR digital sandbox in the visible space;

[0006] S2, identifying the user's gesture finger joints;

[0007] S3, identifying the grasping and placing actions according to the correspondence between the user's gestures and the sand table objects;

[0008] S4. Drag the objects or icons on the digital sandbox;

[0009] S5. Perform simulation drill based on the user's dragging action.

[0010] Preferably, in step S1, the digital sandbox is presented in front of the user through AR technology, so that the user can intuitively see the virtual sandbox scene.

[0011] Preferably, in step S2, the user's hand motion images or video data are collected by a camera, and the hand motion state transfer modeling method based on the LSTM neural network is combined with a recursive least squares adaptive algorithm with a forgetting factor to identify the user's finger joint positions and motion trajectories.

[0012] Preferably, in step S3, by identifying the movement trajectory of the user's finger joints, it is determined whether the user intends to grasp or place the virtual object.

[0013] Preferably, in step S3, when the user brings the finger joints close to the virtual object, it is recognized as a pinching action.

[0014] Preferably, in step S3, when the user moves the finger joint close to the placement position of the virtual object, it is recognized as a placement action.

[0015] Preferably, in step S4, according to the recognized user intention and the movement trajectory of the finger joints, the virtual object is dragged in the AR space sandbox through AR technology.

[0016] Preferably, in step S4, the operation result is fed back to the user through real-time feedback technology, so that the user can intuitively see the effect of his or her operation.

[0017] Preferably, in step S5, the drill process is simulated according to the user's dragging action and operation result to improve the convenience of sandbox operation.

[0018] Preferably, in step S5, when the user drags the virtual object to the target position, it is determined according to a preset rule whether the target is successfully occupied or a predetermined event is triggered.

[0019] The present invention also provides an AR space sandbox interactive system based on gesture control, including a display module, a gesture finger joint recognition module, an action recognition module, a drag module and a simulation drill module;

[0020] The display module is used to display the AR digital sand table in the visible space, and present the digital sand table in front of the user through AR technology, so that the user can intuitively see the virtual sand table scene;

[0021] The gesture finger joint recognition module is used to collect the user's hand movement images or video data through the camera, and recognize the user's finger joint position and movement trajectory based on the hand movement state transfer modeling method of the LSTM neural network and the recursive least squares adaptive algorithm with a forgetting factor;

[0022] The action recognition module is used to identify the grasping and placing actions according to the correspondence between the user's gestures and the sandbox objects, and to determine whether the user intends to grasp or place the virtual object by identifying the movement trajectory of the user's finger joints;

[0023] The drag module is used to drag objects or icons in the digital sandbox, and according to the recognized user intention and the movement trajectory of the finger joints, the virtual objects are dragged in the AR space sandbox through AR technology.

[0024] The simulation drill module is used to simulate the drill according to the user's dragging action, and simulate the drill process according to the user's dragging action and operation results, thereby improving the convenience of sandbox operation.

[0025] Compared with the prior art, the present invention has the following beneficial effects:

[0026] The AR space sandbox interaction method based on gesture control of the present invention realizes the control of virtual objects by recognizing the user's gesture actions, thereby improving the intuitiveness and naturalness of the interaction. In addition, the method can also be personalized according to the user's habits and preferences to meet the needs of different users.

[0027] The AR spatial sandbox interaction method based on gesture control described in the present invention has the advantages of being intuitive, natural, and efficient, and provides new ideas and methods for the development of AR technology. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] The accompanying drawings are used to provide a further understanding of the technical solution of the present invention and constitute a part of the specification. Together with the specific implementation methods of the present application, they are used to explain the technical solution of the present invention and do not constitute a limitation on the technical solution of the present invention.

[0029] Figure 1 It is a flow chart of the AR space sandbox interaction method based on gesture control described in the present invention. DETAILED DESCRIPTION

[0030] The present invention is described in more detail hereinafter to facilitate understanding of the present invention.

[0031] like Figure 1 As shown, the AR space sandbox interaction method based on gesture control of the present invention includes the following steps:

[0032] S1. Displaying the AR digital sand table in the visible space: The digital sand table is presented in front of the user through AR technology, so that the user can intuitively see the virtual sand table scene;

[0033] S2. Identify the user's hand gestures and finger joints: Collect the user's hand motion images or video data through a camera or other device, and use the hand motion state transition modeling method based on the LSTM neural network, combined with the recursive least squares adaptive algorithm with a forgetting factor, to identify the user's finger joint positions and motion trajectories. The specific design of the algorithm is as follows:

[0034] The present invention models the dynamic changes of hand motion as changes in joint position states, and defines the state transition formula as:

[0035]

[0036] Among them, x(k+1) represents the trajectory change of the hand joint at the next M moments k+1, k+2, ..., k+M, and M is the prediction range. Assuming p(k) is the joint position at moment k, x(k+1) can be expressed as the future M joint positions, i.e., p(k+1), p(k+2), ..., p(k+M). * (k) represents the joint position of the past N moments k, k-1, ..., k-N+1, that is, p(k), p(k-1), ..., p(k-N+1). ω k is zero-mean Gaussian white noise. Function f * It reflects the state transition of human motion, with the input being the historical trajectory and the output being the future joint position.

[0037] The present invention adopts a neural network model to model hand behavior. Since hand movement changes over time, the present invention proposes an adaptive algorithm for online adjustment of f * Parameters.

[0038] (1) Neural network model structure design

[0039] In order to reduce the dimension and adjust the parameters online, the present invention adds a fully connected layer after the two LSTM layers. The activation function of this layer is ReLU, and the formula is:

[0040] f(x)=max(0,x)#(2)

[0041] By constructing an LSTM model, the present invention models the state transition of the hand joint position. The trained neural network model can be used as an effective feature extractor, and the weights of the model output layer are adjusted online by fixing the features learned by the LSTM layer. Based on the above model, the state transition model of the human joint position is:

[0042]

[0043] In formula (3), x * (k)∈R 3N is the input vector; g represents the LSTM neural network, whose weight is U; ε(x * (k))∈R 3M is the function reconstruction error, which approaches zero after sufficient training; W∈R h×3M is the weight of the output layer, and h is the number of neurons in the fully connected layer.

[0044] (2) Adaptive model design

[0045] In order to adapt to the time-varying nature of human behavior, it is very important to update the weight parameters online. The present invention uses a recursive least squares parameter adaptation algorithm with a forgetting factor to perform asymptotic adaptation on the parameters in the neural network.

[0046] To minimize The expected error from the true value is:

[0047]

[0048] Superimpose all column vectors of W in equation (3) to obtain a time-varying vector, set as θ∈R 3Mh Among them, θ k Indicates the value at time k. To extract the features of the neural network model, define a new data matrix θ k ∈R 3M×3Mh , indicating M max(0,g(U,x * (k))) diagonal splicing. Substituting into equation (1) and equation (3), we get:

[0049] x(k+1)=Φ k θ k +ω k #(5)

[0050] Will as the parameter estimate at time k, and is the parameter estimation error, the prior estimate and the estimation error are defined as:

[0051]

[0052] The core of the algorithm lies in iteratively updating parameters And predict x(k+1), the parameter update rule is:

[0053]

[0054] Among them, F k It is called learning gain, and the update rule is:

[0055]

[0056] Where 0≤λ 1 (k)≤1,0≤λ 2 (k)≤1, the value is: 0<λ 1 (k)<1 and λ 2 (k) = 1, where λ1 and λ2 are forgetting factors. The least squares gain with forgetting factors is expressed as:

[0057] S3. Identify the grasping and placing actions based on the correspondence between the user's gestures and the sandbox objects: By identifying the movement trajectory of the user's finger joints, determine whether the user intends to grasp or place the virtual object. For example, when the user brings the finger joints close to the virtual object, the system will identify it as a grasping action; when the user brings the finger joints close to the placement position of the virtual object, the system will identify it as a placement action.

[0058] S4. Drag the objects or icons in the digital sandbox: Based on the recognized user intention and the movement trajectory of the finger joints, the virtual objects are dragged in the AR space sandbox through AR technology. At the same time, the operation results can be fed back to the user through real-time feedback technology, so that the user can intuitively see the effect of his operation.

[0059] S5. Simulate drills based on the user's dragging actions: Simulate drills based on the user's dragging actions and operation results to improve the convenience of sandbox operations. For example, when the user drags a virtual object to the target location, the system will determine whether the target is successfully occupied or other events are triggered based on preset rules.

[0060] The present invention also provides an AR space sandbox interactive system based on gesture control, including a display module, a gesture finger joint recognition module, an action recognition module, a drag module and a simulation drill module;

[0061] The display module is used to display the AR digital sand table in the visible space, and present the digital sand table in front of the user through AR technology, so that the user can intuitively see the virtual sand table scene;

[0062] The gesture finger joint recognition module is used to collect the user's hand movement images or video data through the camera, and recognize the user's finger joint position and movement trajectory based on the hand movement state transfer modeling method of the LSTM neural network and the recursive least squares adaptive algorithm with a forgetting factor;

[0063] The action recognition module is used to identify the grasping and placing actions according to the correspondence between the user's gestures and the sandbox objects, and to determine whether the user intends to grasp or place the virtual object by identifying the movement trajectory of the user's finger joints;

[0064] The drag module is used to drag objects or icons in the digital sandbox, and according to the recognized user intention and the movement trajectory of the finger joints, the virtual objects are dragged in the AR space sandbox through AR technology.

[0065] The simulation drill module is used to simulate the drill according to the user's dragging action, and simulate the drill process according to the user's dragging action and operation results, thereby improving the convenience of sandbox operation.

[0066] The AR space sandbox interaction method based on gesture control described in the present invention realizes the control of virtual objects in the AR space sandbox by recognizing the user's gesture actions, thereby improving the intuitiveness and naturalness of the interaction and enhancing the user experience.

[0067] The above describes the preferred embodiments of the present invention, but it is not intended to limit the present invention. Those skilled in the art may make improvements and changes to the embodiments disclosed herein without departing from the scope and spirit of the present invention.

Claims

1. An AR space sandbox interaction method based on gesture control, characterized in that: The AR space sandbox interaction method based on gesture control includes the following steps: S1. Display the AR digital sandbox in the visible space; S2, identifying the user's gesture finger joints; S3, identifying the grasping and placing actions according to the correspondence between the user's gestures and the sand table objects; S4. Drag the objects or icons on the digital sandbox; S5. Perform simulation drill based on the user's dragging action.

2. The AR space sandbox interaction method based on gesture control according to claim 1, characterized in that: In step S1, the digital sandbox is presented to the user through AR technology, so that the user can intuitively see the virtual sandbox scene.

3. The AR space sandbox interaction method based on gesture control according to claim 1, characterized in that: In step S2, the user's hand movement images or video data are collected through a camera, and the hand movement state transfer modeling method based on the LSTM neural network is combined with the recursive least squares adaptive algorithm with a forgetting factor to identify the user's finger joint positions and movement trajectories.

4. The AR space sandbox interaction method based on gesture control according to claim 1, characterized in that: In step S3, by identifying the movement trajectory of the user's finger joints, it is determined whether the user intends to grasp or place the virtual object.

5. The AR space sandbox interaction method based on gesture control according to claim 1, characterized in that: In step S3, when the user brings the finger joints close to the virtual object, it is recognized as a pinching action.

6. The AR space sandbox interaction method based on gesture control according to claim 1, characterized in that: In step S3, when the user moves the finger joint close to the placement position of the virtual object, it is recognized as a placement action.

7. The AR space sandbox interaction method based on gesture control according to claim 1, characterized in that: In step S4, according to the recognized user intention and the movement trajectory of the finger joints, the virtual object is dragged in the AR space sandbox through AR technology.

8. The AR space sandbox interaction method based on gesture control according to claim 1, characterized in that: In step S4, the operation result is fed back to the user through real-time feedback technology, so that the user can intuitively see the effect of his or her operation.

9. The AR space sandbox interaction method based on gesture control according to claim 1, characterized in that: In step S5, the drill process is simulated according to the user's dragging action and operation results to improve the convenience of sandbox operation.

10. The AR space sandbox interaction method based on gesture control according to claim 1, characterized in that: In step S5, when the user drags the virtual object to the target position, it is determined according to a preset rule whether the target is successfully occupied or a predetermined event is triggered.

Images