A Virtual Reality Teaching Gesture Recognition Method Based on Dual-Camera Multi-Branch Network

Abstract

This application discloses a virtual reality teaching gesture recognition method based on a dual-camera multi-branch network, comprising: acquiring 3D point cloud images and depth images of the gesture changes of the object to be recognized at different times; performing hand region detection on the 3D point cloud images and depth images respectively, and segmenting the hand region images from them; inputting the hand region images at different times into a trained gesture recognition model to generate gesture recognition results of the object to be recognized at different times; and forming operation instructions of the object to be recognized at different times based on the gesture recognition results and the preset mapping relationship between standard gestures and instructions. This invention integrates data from two modalities, depth images and 3D point cloud images, for hand recognition. The data from the two modalities are complementary, resulting in more robust recognition capabilities and improved recognition accuracy. Based on the recognition results, virtual scene teaching can be implemented, allowing students to focus more on learning in the classroom.

Description

Technical Field

[0001] This application relates to the field of pattern recognition and information processing technology, and more specifically, to a virtual reality teaching gesture recognition method, system, and computer device based on a dual-camera multi-branch network. Background Technology

[0002] Education is an indispensable human cultural activity in every era. Introducing VR (virtual reality) technology into the education industry and exploring new teaching methods is crucial for students to acquire various skills more efficiently and proactively. Gesture recognition can be applied in multiple scenarios such as VR, AR, and assisted teaching, and research on gesture recognition plays a vital role. Teaching is precisely an excellent application scenario for virtual reality, suitable for use in universities, primary and secondary schools, and science popularization education in museums and science centers. Currently, most teaching remains at the level of traditional image-based instruction, and a small portion uses virtual reality technology but suffers from poor interactivity and over-reliance on game controllers.

[0003] In recent years, significant progress has been made in 3D gesture recognition methods. However, compared to 2D image recognition, 3D gestures often suffer from occlusion issues, resulting in insufficient recognition accuracy for practical applications, thus leaving considerable room for improvement. Furthermore, for educational applications, there is a need for a virtual reality gesture recognition method and system designed for instructional purposes. Summary of the Invention

[0004] To address at least one deficiency or improvement need in the existing technology, this invention provides a virtual reality teaching gesture recognition method, system, and computer device based on a dual-camera multi-branch network. It integrates data from two modalities, depth images and 3D point cloud images, for hand recognition. The data from the two modalities are complementary, resulting in more robust recognition capabilities and thus improving recognition accuracy.

[0005] To achieve the above objectives, according to a first aspect of the present invention, a virtual reality teaching gesture recognition method based on a dual-camera multi-branch network is provided, the method comprising the following steps:

[0006] Acquire 3D point cloud images and depth images of the gesture changes of the object to be identified at different times;

[0007] Hand region detection is performed on the 3D point cloud image and depth image respectively, and the hand region image is segmented from them;

[0008] The hand region images corresponding to the 3D point cloud images and depth images at different times are input into the trained gesture recognition model. The feature vectors of the depth image and the feature vectors of the point cloud image of the hand region are extracted and fused. The gesture recognition results of the object to be recognized at different times are generated based on the fused feature vector.

[0009] Based on the gesture recognition results and the preset mapping relationship between standard gestures and instructions, operation instructions for the object to be recognized at different times are generated.

[0010] Furthermore, the aforementioned virtual reality teaching gesture recognition method also includes:

[0011] Geometric transformations are performed on the 3D point cloud image, including in-plane rotation, 3D scaling, and 3D translation;

[0012] The NewDropout function is used to enhance the depth image.

[0013] Furthermore, in the above-mentioned virtual reality teaching gesture recognition method, the gesture recognition model includes:

[0014] The first image processing network extracts features from the hand region image in the depth image to obtain the depth image feature vector of the hand region;

[0015] The second image processing network extracts features from the hand region image in the 3D point cloud image to obtain the feature vector of the hand region point cloud image.

[0016] The fusion module fuses the depth image feature vector with the point cloud image feature vector to generate a fused feature vector;

[0017] A gesture recognition network generates gesture recognition results of the object to be recognized at different times based on the fused feature vector.

[0018] Furthermore, in the above-mentioned virtual reality teaching gesture recognition method, the first image processing network includes:

[0019] The hourglass network extracts features from the hand region of a depth image to obtain hand feature data.

[0020] The depth image space branch calculates a supervised attention map for each joint based on the hand features output by the hourglass network;

[0021] The attention enhancement branch calculates an unsupervised attention map for each joint based on the hand features output by the hourglass network;

[0022] The depth extraction branch generates a dense depth feature map containing depth feature vectors for each joint based on the hand feature quantities output by the hourglass network.

[0023] The attention fusion network fuses the supervised attention map with the unsupervised attention map to obtain a fused attention map, and identifies the true depth value of each joint based on the fused attention map and the dense depth feature map to obtain the depth image feature vector of the hand region.

[0024] Furthermore, in the above-mentioned virtual reality teaching gesture recognition method, the calculation method of the supervised attention map is as follows:

[0025] Calculate the attention map for each joint based on the hand features output by the hourglass network;

[0026] The attention map is normalized to obtain a heatmap representing the probability of each pixel appearing at a joint position;

[0027] Based on the heatmap, the recognition coordinates of each joint in the depth image space are calculated through integral operations;

[0028] The attention map is supervised to obtain a supervised attention map by minimizing the average distance between the recognized coordinates and the actual coordinates of each joint in the depth image space.

[0029] Furthermore, in the above-mentioned virtual reality teaching gesture recognition method, the second image processing network includes:

[0030] The downsampling module downsamples the input 3D point cloud image and outputs several three-dimensional data points.

[0031] The normalization module is used to obtain the coordinate values ​​and surface normal vectors of each of the three-dimensional data points as initial feature inputs, and to normalize them in the oriented bounding box to obtain normalized coordinates;

[0032] The layered extraction module takes the normalized coordinates and three-dimensional surface normals of each three-dimensional data point as input, divides all three-dimensional data points into multiple local regions, and obtains the point cloud image feature vector of the hand region by extracting features from each local region and mapping them into a three-dimensional vector.

[0033] Furthermore, in the above-mentioned virtual reality teaching gesture recognition method, the gesture recognition network includes:

[0034] A multi-layer transformer model takes the fused feature vector output by the fusion module as input to generate offsets and prior information; the offsets are added to the three-dimensional average joint positions of the predefined template hand mesh to obtain coarse predicted coordinates;

[0035] An autoregressor performs multiple iterations of regression on the coarse predicted coordinates based on the prior information to obtain the fine predicted coordinates, which is the gesture recognition result.

[0036] According to a second aspect of the present invention, a virtual reality teaching gesture recognition system based on a dual-camera multi-branch network is provided, the system comprising:

[0037] The data acquisition module is used to acquire 3D point cloud images and depth images of the gesture changes of the object to be identified at different times;

[0038] The preprocessing module is used to perform hand region detection on the 3D point cloud image and depth image respectively, and segment out the hand region image from them;

[0039] The recognition module is used to input the hand region images corresponding to the 3D point cloud images and depth images at different times into the trained gesture recognition model. The model extracts the depth image feature vector and the point cloud image feature vector of the hand region and fuses them. Based on the fused feature vector, the gesture recognition results of the object to be recognized at different times are generated.

[0040] The response module is used to generate operation instructions for the object to be recognized at different times based on the gesture recognition results and the preset mapping relationship between standard gestures and instructions.

[0041] Furthermore, in the aforementioned virtual reality teaching gesture recognition system, an event camera is used to capture 3D point cloud images of the gesture changes of the object to be recognized at different times;

[0042] The Leap Motion controller is used to record depth images of the gesture changes of the object to be recognized at different times.

[0043] According to a third aspect of the present invention, a computer device is also provided, comprising at least one processing unit and at least one storage unit, wherein the storage unit stores a computer program that, when executed by the processing unit, causes the processing unit to perform the steps of any of the above-described virtual reality teaching gesture recognition methods.

[0044] In summary, compared with the prior art, the above-described technical solutions conceived by this invention can achieve the following beneficial effects:

[0045] (1) This invention uses two modal data, depth image and 3D point cloud image, for gesture recognition. The data of the two modalities can complement each other, making the method more robust in recognition.

[0046] (2) This invention utilizes an innovative model to improve the accuracy of gesture recognition in depth images. It decomposes three-dimensional hand gesture recognition into the recognition of two-dimensional joint positions in the depth image space, and identifies their corresponding depths through two complementary attention maps. This allows for the extraction of joint features and skeleton features, thereby improving recognition accuracy.

[0047] (3) This invention introduces an event camera to acquire the gesture image data of the object. The event camera can sensitively capture moving objects. Compared with traditional cameras, it has the characteristics of low latency and can capture pixel changes within a very short time interval. At the same time, the image data acquired by the event camera can directly determine whether an action has occurred. Attached Figure Description

[0048] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0049] Figure 1 This is a flowchart illustrating the virtual reality teaching gesture recognition method provided in this embodiment;

[0050] Figure 2 This is a schematic diagram of the operation flow of the virtual reality teaching gesture recognition method based on a dual-camera multi-branch network provided in this embodiment;

[0051] Figure 3 This is a schematic diagram of the data acquisition scenario provided in this embodiment;

[0052] Figure 4 This is a schematic diagram of the network structure of the hand recognition model provided in an embodiment of this application.

[0053] Figure 5 The standard gesture design and its corresponding operation instructions are provided for the embodiments of this application. Detailed Implementation

[0054] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0055] The terms "first," "second," "third," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or apparatuses.

[0056] Furthermore, to avoid obscuring the understanding of the invention by those skilled in the art, well-known or widely used techniques, elements, structures, and processes may not be described or shown in detail. Although the accompanying drawings illustrate exemplary embodiments of the invention, the drawings are not necessarily drawn to scale, and specific features may be enlarged or omitted to better illustrate and explain the invention.

[0057] Figure 1 This is a flowchart illustrating the virtual reality teaching gesture recognition method provided in this embodiment. Figure 2 This is a schematic diagram illustrating the operation flow of the virtual reality teaching gesture recognition method based on a dual-camera multi-branch network provided in this embodiment; please refer to... Figure 1 , 2 The method mainly includes the following steps:

[0058] S1 acquires 3D point cloud images and depth images of the gesture changes of the object to be identified at different times;

[0059] In this embodiment, the object to be identified is a teacher who wears virtual reality devices to conduct teaching activities; Figure 3 This is a schematic diagram of the data acquisition scenario provided in this embodiment, such as... Figure 3 As shown, data is acquired using an event camera fixed in front of the teacher and a LeapMotion controller. Specifically, the event camera records 3D point cloud images of the teacher's hand gestures at different times. p Using a Leap Motion controller, depth images of the teacher's hand gestures at different times were recorded. d .

[0060] Because event cameras output a large number of events by capturing changes in light intensity, they can only capture objects that are moving relative to the camera. When an object is stationary relative to the camera, they basically do not output any information. Therefore, when a hand appears in the image recorded by the camera, it is assumed that the teacher user is issuing a gesture command.

[0061] S2 performs hand region detection on the 3D point cloud image and depth image respectively, and segments out the hand region image from them;

[0062] In this embodiment, before performing hand region detection, the acquired 3D point cloud image and depth image are preprocessed; specifically, the 3D point cloud image is subjected to geometric transformation, including in-plane rotation ([-180,180] degrees), 3D scaling ([0.9,1,1]), and 3D translation ([-8,8] mm); the depth image is enhanced using the NewDropout function.

[0063] The data augmentation function NewDropout proposed in this embodiment is defined as function Tα:R W×H →R W×H The parameter α ∈ [0,1] controls the intensity of the enhancement. In the first step, we uniformly sample a probability γ from the range [0, α]. In the second step, we uniformly sample a set of pixels Q from the hand surface. The selection probability of each pixel is γ. The enhanced depth image is calculated by deleting selected pixels. As shown below:

[0064]

[0065] Where p represents any pixel in the depth image; C represents a constant value assigned to the background (non-hand) pixel.

[0066] Then, hand region detection is performed on the preprocessed 3D point cloud image and depth image, and the hand region image Img is segmented from them. p 1mg d In a specific example, the YOLOv7 model is used to detect objects in both hands. The corresponding hand regions are cropped from the 3D point cloud image and depth image and adjusted to a fixed size of 128×128.

[0067] S3 inputs the hand region images corresponding to the 3D point cloud images and depth images at different times into the trained gesture recognition model, extracts the feature vectors of the depth image and the feature vectors of the point cloud image of the hand region respectively and fuses them, and generates the gesture recognition results of the object to be recognized at different times based on the fused feature vectors.

[0068] The hand pose at a given moment is determined based on the hand region image data at the same moment. By inputting the image data at the current moment k into the trained gesture recognition model, the gesture recognition result P at the current moment k can be obtained. k .

[0069] Figure 4 This is a schematic diagram of the network structure of the hand recognition model provided in an embodiment of this application. In this embodiment, the hand recognition model includes a first image processing network, a second image processing network, a fusion module, and a hand recognition network; wherein,

[0070] The first image processing network is used to extract features from the hand region image in the depth image to obtain the depth image feature vector of the hand region;

[0071] Please see Figure 4 In one optional implementation, the first image processing network includes an hourglass network, a depth image space branch, an attention enhancement branch, a depth extraction branch, and an attention fusion network.

[0072] The hourglass network is used to extract features from the hand region of a depth image, obtaining hand features. The hourglass network is used as the encoder, defined as the output feature quantity f:R from the input depth image. W×H →R c×h×w The nonlinear mapping is defined by h, w, and c, where h, w, and c represent the height, width, and number of channels of the output feature quantity, respectively. To maximize accuracy, the encoder network should have a high ability to extract and fuse features at different scales. An hourglass network is used as the encoder. It uses bridging connections and repeated bottom-up and top-down processing to extract and consolidate features at different scales, processes the input depth image, extracts and combines low-level and high-level features of the hand, outputs a high-resolution feature quantity, and passes the feature quantity to the depth image space branch, the attention enhancement branch, and the depth extraction branch.

[0073] The depth image spatial branch is used to calculate a supervised attention map for each joint based on the hand features output by the hourglass network; wherein, the supervised attention map is calculated as follows:

[0074] Calculate the attention map for each joint based on the hand features output by the hourglass network;

[0075] The attention map is normalized to obtain a heatmap representing the probability of each pixel appearing at a joint position;

[0076] Based on the heatmap, the recognition coordinates of each joint in the depth image space are calculated through integral operations;

[0077] The attention map is supervised to obtain a supervised attention map by minimizing the average distance between the recognized coordinates and the actual coordinates of each joint in the depth image space.

[0078] Specifically, the depth image spatial branch takes the encoder's output features as input and computes the attention map Att for each joint. uv ∈R i×h×w We use Let Att represent the attention map corresponding to the i-th joint. To ensure that each attention map focuses on the pixels appearing at the corresponding joint, 2D supervision of the heatmap will be utilized. Therefore, the attention map Att is first... uvNormalization of the spatial softmax layer is performed to obtain the corresponding heatmap. as follows:

[0079]

[0080] Where σ represents the spatial softmax layer; heatmap Ω represents the probability of the i-th joint appearing at each pixel position; Ω represents the attention map. The spatial domain; the two-dimensional position of the i-th joint is calculated through integration, as shown below:

[0081]

[0082] Represents the recognition coordinates of the i-th joint in the depth image space; supervision is applied to all attention maps. By minimizing the average distance L1, it is defined as:

[0083]

[0084] Among them, (U) i V i ) represents the actual coordinates of the i-th joint in the depth image space.

[0085] By using 2D supervision of the heatmap, we can ensure that each attention map focuses on the pixel where the corresponding joint appears.

[0086] The attention enhancement branch is used to compute an unsupervised attention map for each joint based on the hand features output by the hourglass network;

[0087] This branch aims to compute a more flexible attention map to enhance the attention map Att computed by the depth image space branch. uv This facilitates the recognition of joint depth values. Specifically, the attention enhancement branch takes the encoder's output features as input and computes an attention map Att for each joint. mix ∈R i×h×w . This represents the attention map corresponding to the i-th joint. (And Att) uv In comparison, this attention map Without external constraints (supervision), it is allowed to freely learn which hand pixels are most relevant, resulting in a more flexible attention map that enhances the attention map generated by the depth image spatial branch, making it easier to identify the depth values ​​of each joint in different scenes.

[0088] The depth extraction branch is used to generate a dense depth feature map containing the depth feature vector of each joint based on the hand feature quantities output by the hourglass network.

[0089] Unlike typical methods that compute pixel-level depth offsets or predictions, this model computes dense pixel-level depth feature vectors. Specifically, the depth extraction branch takes the encoder's output features as input to generate a dense depth feature map D∈R. D×h×w Where D represents the dimension of the depth feature vector, which is set to D = 64 in this embodiment. The depth feature vector at a specific position (x, y) in the depth feature map D is represented as D(x, y) ∈ R. D This includes information about the joint depth values ​​collected from the input depth image pixels within the receptive domain of a specific location (x, y) in the depth feature volume. The final feature vector used to identify the depth value of each joint is obtained using a weighted average of all depth feature vectors, where the weights of each depth feature vector are assigned using the corresponding fusion attention map.

[0090] Attention fusion network is used to fuse supervised attention maps with unsupervised attention maps to obtain a fused attention map, and to identify the true depth value of each joint based on the fused attention map and dense depth feature map to obtain the depth image feature vector of the hand region.

[0091] In this embodiment, the fusion principle is as follows:

[0092] Two attention maps and They are complementary. The network attention is shifted to the pixel where the i-th joint occurs, and The network is helped to notice non-joint pixels, which may contain useful information for identifying the depth value of the i-th joint. The two attention maps are then fused as follows:

[0093]

[0094] Here, β i ∈[0,1] represents the learned parameters, controlling the fused attention graph used for each attention graph pair. The contribution of the model; the model uses As a guide, features are pooled from the pixels containing information most relevant to the depth of the i-th joint; this fused attention map is used to guide feature pooling for subsequent depth value recognition.

[0095] The principle of depth value recognition is as follows:

[0096] The depth value of the i-th joint is identified by the feature vector obtained through feature pooling of the pixels containing the most relevant depth information. The guidance is as follows:

[0097]

[0098] in, Let be the set feature vector of the i-th joint; let be the recognition depth value of the i-th joint, expressed as . This indicates that a linear layer estimate shared by all relevant nodes is used, as shown below:

[0099]

[0100] Where W∈R D , b∈R represents the weights of the linear layer.

[0101] The identification of node depth values ​​is supervised by the following loss term:

[0102]

[0103] Among them, Z i This represents the actual depth value of the i-th joint.

[0104] The second image processing network is used to extract features from the hand region image in the 3D point cloud image to obtain the feature vector of the hand region point cloud image;

[0105] In one optional implementation, the second image processing network includes a downsampling module, a normalization module, and a hierarchical extraction module;

[0106] The downsampling module is used to downsample the input 3D point cloud image and output several three-dimensional data points;

[0107] The normalization module is used to obtain the coordinate values ​​and surface normal vectors of each of the three-dimensional data points as initial feature inputs, and to normalize them in the oriented bounding box to obtain normalized coordinates;

[0108] Specifically: the input point cloud image data is downsampled into N points p. i ∈R 3 (i = 1, 2, ..., N). The coordinates and surface normal vector of each point are used as initial feature inputs and normalized within the oriented bounding box. The oriented bounding box is a bounding box that tightly fits the input point cloud. The orientation of the oriented bounding box is determined by principal component analysis (PCA) of the 3D coordinates of the input points. The x, y, and z axes of the oriented bounding box are aligned with the eigenvectors of the input point covariance matrix, corresponding to eigenvalues ​​from largest to smallest. The original points in the camera coordinate system are first converted to points in the oriented bounding box coordinate system, then these points are moved to have a mean of zero and scaled to a unit size. The formula for this method is as follows:

[0109]

[0110]

[0111] Where, p obb p represents the normalized coordinates of the oriented bounding box. cam Let p be the coordinates of point p in the camera coordinate system. It is a 3D rotation matrix. Let L be the centroid coordinates. obb p is the maximum side length of the bounding box. nor For the final normalized coordinates.

[0112] The layered extraction module takes the normalized coordinates and three-dimensional surface normals of each three-dimensional data point as input, divides all three-dimensional data points into multiple local regions, and obtains the point cloud image feature vector of the hand region by extracting features from each local region and mapping them into three-dimensional vectors.

[0113] In this embodiment, the hierarchical extraction module uses a multi-layer point cloud classification and segmentation network to extract normalized points. The input is fed into a point cloud classification and segmentation network, where... Let the coordinates of the i-th point be normalized. Let be the normal to the 3D surface corresponding to the i-th point;

[0114] This point cloud classification and segmentation network has three levels of point set abstraction. The first two layers group the input points into two local regions, N1 = 512 and N2 = 128, respectively, with each local region containing k = 64 points. These two layers extract C1 = 128 and C2 = 256-dimensional feature vectors for each local region, respectively. The last layer extracts a 1024-dimensional global feature vector, which is then mapped to a 3D output vector through three fully connected layers.

[0115] During the training phase of the model, the true 3D joint positions in the camera coordinate system are also transformed using formulas (9) and (10) to obtain the normalized 3D joint positions in the oriented bounding box coordinate system.

[0116] By performing principal component analysis on the ground-value 3D joint positions in the training dataset, we can obtain... 3D projection α t The formula is as follows:

[0117]

[0118] Where E is the principal component and m is the empirical mean.

[0119] Set the loss function:

[0120]

[0121] Where w represents the network parameters, F represents the hierarchical PinotNet network, and λ represents the regularization strength;

[0122] During the testing phase, feature vectors are obtained by reconstructing the estimated 3D joint positions based on the network output:

[0123]

[0124] The fusion module fuses the feature vectors of the depth image with the feature vectors of the point cloud image to generate a fused feature vector;

[0125] After obtaining the eigenvectors respectively and eigenvectors Fuse the two feature vectors:

[0126]

[0127] Where α and β are the eigenvectors. and eigenvectors The weight.

[0128] The fusion of feature vectors is supervised by the following loss term:

[0129]

[0130] The proposed model is end-to-end differentiable and is trained by minimizing a loss function, which includes the three loss terms discussed earlier, and its expression is:

[0131] L = L uv +λ1L d +λ2L AB (16)

[0132] Where λ1 and λ2 are equilibrium L uv L d和 L AB Weighting factors.

[0133] The gesture recognition network generates gesture recognition results of the object to be recognized at different times based on the fused feature vectors; in an optional implementation, the gesture recognition network includes a multi-layer transformer model and an autoregressor.

[0134] The multi-layer transformer model takes the fused feature vector output by the fusion module as input to generate offsets and prior information; the offsets are added to the three-dimensional average joint position of the predefined template hand mesh to obtain coarse predicted coordinates;

[0135] The autoregressor performs multiple iterations of regression on the coarse predicted coordinates based on the prior information to obtain the fine predicted coordinates, which is the gesture recognition result.

[0136] After obtaining the feature vectors of data fusion Then, the offset is first obtained through multiple transformer layers. and prior information X;

[0137] Pose length regularization is performed using the Jacobian matrix. The expression for the Jacobian matrix is ​​as follows:

[0138]

[0139] The pose length regularization expression is as follows:

[0140]

[0141] Where 'a' represents the moving average of the Jacobian matrix.

[0142] Then, in a predefined template, hand mesh Add offset to the three-dimensional average joint position Obtain coarse predicted coordinates Its expression is as follows:

[0143]

[0144] The detailed predicted coordinates are regressed using an autoregressive method. First, make:

[0145]

[0146] Iterate using the following formula:

[0147]

[0148] Where X represents prior information and R represents a readjustment factor; generally, the more iterations, the better the results.

[0149] S4 generates operation instructions for the object to be recognized at different times based on the gesture recognition results and the preset mapping relationship between standard gestures and instructions.

[0150] This embodiment provides a set of gesture designs suitable for virtual reality teaching scenarios, and matches each gesture with an operation instruction, forming a system gesture library; please refer to Figure 5 The system's gesture library contains 14 standard gestures: left hand rotates left, left hand rotates right, right hand rotates left, right hand rotates right, both hands close, both hands open, one hand zooms in, one hand zooms out, both hands zooms in, both hands zooms out, left hand moves right, right hand moves left, forward, and backward.

[0151] The model outputs the teacher's gesture recognition results at different times and matches them with the standard gestures in the system's gesture library. If the gesture recognition result does not exist in the system's gesture library, the gesture is invalid and the system does not react. If the match is successful, it is a valid gesture. The system queries and executes the operation command corresponding to the matched standard gesture, and changes the virtual reality screen accordingly to realize non-contact human-computer interaction teaching.

[0152] This application allows teachers to operate virtual reality devices without physical contact, using these devices to assist in teaching and leading students in an immersive learning experience. The system's interactivity facilitates teaching, while its engaging nature helps students focus more on learning in the classroom.

[0153] It should be noted that although the operations of the methods of the embodiments of this specification are described in a specific order in the above embodiments, this does not require or imply that these operations must be performed in that specific order, or that all the operations shown must be performed to achieve the desired result. On the contrary, the steps depicted in the flowcharts may be executed in a different order. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into one step, and / or one step may be broken down into multiple steps.

[0154] This embodiment provides a virtual reality teaching gesture recognition system based on a dual-camera multi-branch network. This system can be implemented using software and / or hardware and can be integrated into an electronic device. Specifically, the system includes:

[0155] The data acquisition module is used to acquire 3D point cloud images and depth images of the gesture changes of the object to be identified at different times;

[0156] The preprocessing module is used to perform hand region detection on the 3D point cloud image and depth image respectively, and segment out the hand region image from them;

[0157] The recognition module is used to input the hand region images corresponding to the 3D point cloud images and depth images at different times into the trained gesture recognition model. The model extracts the depth image feature vector and the point cloud image feature vector of the hand region and fuses them. Based on the fused feature vector, the gesture recognition results of the object to be recognized at different times are generated.

[0158] The response module is used to generate operation instructions for the object to be recognized at different times based on the gesture recognition results and the preset mapping relationship between standard gestures and instructions.

[0159] In an optional example, the system also includes an event camera and a Leap Motion controller; the event camera is used to acquire 3D point cloud images of the gesture changes of the object to be identified at different times; and the Leap Motion controller is used to record depth images of the gesture changes of the object to be identified at different times.

[0160] Specific limitations regarding the virtual reality teaching gesture recognition system can be found in the limitations of the virtual reality teaching gesture recognition method described above, and will not be repeated here. Each module in the aforementioned virtual reality teaching gesture recognition system can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in the processor of the computer device in hardware form or independent of it, or stored in the memory of the computer device in software form, so that the processor can call and execute the corresponding operations of each module.

[0161] This embodiment also provides a computer device, which includes at least one processor and at least one memory. The memory stores a computer program. When the computer program is executed by the processor, it causes the processor to perform the steps of a virtual reality teaching gesture recognition method. The specific steps are described above and will not be repeated here. In this embodiment, the types of processor and memory are not specifically limited. For example, the processor can be a microprocessor, a digital information processor, an on-chip programmable logic system, etc.; the memory can be volatile memory, non-volatile memory, or a combination thereof.

[0162] The electronic device can also communicate with one or more external devices (such as a keyboard, pointing terminal, display, etc.), one or more terminals that enable users to interact with the electronic device, and / or any terminal that enables the electronic device to communicate with one or more other computing terminals (such as a network card, modem, etc.). This communication can be performed via an input / output (I / O) interface. Furthermore, the electronic device can also communicate with one or more networks (such as a Local Area Network (LAN), a Wide Area Network (WAN), and / or a public network, such as the Internet) via a network adapter.

[0163] This application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the above-described method. The computer-readable storage medium may include, but is not limited to, any type of disk, including floppy disks, optical disks, DVDs, CD-ROMs, microdrives, as well as magneto-optical disks, ROMs, RAMs, EPROMs, EEPROMs, DRAMs, VRAMs, flash memory devices, magnetic cards or optical cards, nanosystems (including molecular memory ICs), or any type of medium or device suitable for storing instructions and / or data.

[0164] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0165] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0166] In the several embodiments provided in this application, it should be understood that the disclosed apparatus can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some service interface; the indirect coupling or communication connection between devices or units may be electrical or other forms.

[0167] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0168] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0169] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage device (CMD). Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a memory and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned memory includes various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0170] The foregoing description is merely an exemplary embodiment of this disclosure and should not be construed as limiting the scope of this disclosure. Any equivalent changes and modifications made in accordance with the teachings of this disclosure shall still fall within the scope of this disclosure. Those skilled in the art will readily conceive of embodiments of this disclosure upon considering the specification and practicing the disclosure herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not described herein. The specification and embodiments are to be considered exemplary only, and the scope and spirit of this disclosure are defined by the claims.

[0171] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A virtual reality teaching gesture recognition method based on a dual-camera multi-branch network, characterized in that, include: Acquire 3D point cloud images and depth images of the gesture changes of the object to be identified at different times; Hand region detection is performed on the 3D point cloud image and depth image respectively, and the hand region image is segmented from them; Hand region images corresponding to 3D point cloud images and depth images at different times are input into a trained gesture recognition model, wherein the gesture recognition model includes: The first image processing network extracts features from the hand region image in the depth image to obtain the depth image feature vector of the hand region; The second image processing network extracts features from the hand region image in the 3D point cloud image to obtain the feature vector of the hand region point cloud image. The fusion module fuses the depth image feature vector with the point cloud image feature vector to generate a fused feature vector; A gesture recognition network generates gesture recognition results of the object to be recognized at different times based on the fused feature vector; Based on the gesture recognition results and the preset mapping relationship between standard gestures and instructions, operation instructions for the object to be recognized at different times are formed. The first image processing network includes: The hourglass network extracts features from the hand region of a depth image to obtain hand feature data. The depth image spatial branch calculates a supervised attention map for each joint based on the hand features output by the hourglass network; the supervised attention map is calculated as follows: Calculate the attention map for each joint based on the hand features output by the hourglass network; The attention map is normalized to obtain a heatmap representing the probability of each pixel appearing at a joint position; Based on the heatmap, the recognition coordinates of each joint in the depth image space are calculated by integral operation; the attention map is supervised to obtain a supervised attention map by minimizing the average distance between the recognition coordinates and the actual coordinates of each joint in the depth image space. The attention enhancement branch calculates an unsupervised attention map for each joint based on the hand features output by the hourglass network; The depth extraction branch generates a dense depth feature map containing depth feature vectors for each joint based on the hand feature quantities output by the hourglass network. The attention fusion network fuses the supervised attention map with the unsupervised attention map to obtain a fused attention map, and identifies the true depth value of each joint based on the fused attention map and the dense depth feature map to obtain the depth image feature vector of the hand region.

2. The virtual reality teaching gesture recognition method as described in claim 1, characterized in that, Also includes: Geometric transformations are performed on the 3D point cloud image, including in-plane rotation, 3D scaling, and 3D translation; The NewDropout function is used to enhance the depth image.

3. The virtual reality teaching gesture recognition method as described in claim 1, characterized in that, The second image processing network includes: The downsampling module downsamples the input 3D point cloud image and outputs several three-dimensional data points. The normalization module is used to obtain the coordinate values ​​and surface normal vectors of each of the three-dimensional data points as initial feature inputs, and to normalize them in the oriented bounding box to obtain normalized coordinates; The layered extraction module takes the normalized coordinates and three-dimensional surface normals of each three-dimensional data point as input, divides all three-dimensional data points into multiple local regions, and obtains the point cloud image feature vector of the hand region by extracting features from each local region and mapping them into a three-dimensional vector.

4. The virtual reality teaching gesture recognition method as described in claim 1, characterized in that, The gesture recognition network includes: A multi-layer transformer model takes the fused feature vector output by the fusion module as input to generate offsets and prior information; the offsets are added to the three-dimensional average joint position of the predefined mesh hand template to obtain coarse predicted coordinates; An autoregressor performs multiple iterations of regression on the coarse predicted coordinates based on the prior information to obtain the fine predicted coordinates, which is the gesture recognition result.

5. A virtual reality teaching gesture recognition system based on a dual-camera multi-branch network, characterized in that, include: The data acquisition module is used to acquire 3D point cloud images and depth images of the gesture changes of the object to be identified at different times; The preprocessing module is used to perform hand region detection on the 3D point cloud image and depth image respectively, and segment out the hand region image from them; The recognition module is used to input the hand region images corresponding to the 3D point cloud images and depth images at different times into the trained gesture recognition model, wherein the gesture recognition model includes: The first image processing network extracts features from the hand region image in the depth image to obtain the depth image feature vector of the hand region; The second image processing network extracts features from the hand region image in the 3D point cloud image to obtain the feature vector of the hand region point cloud image. The fusion module fuses the depth image feature vector with the point cloud image feature vector to generate a fused feature vector; A gesture recognition network generates gesture recognition results of the object to be recognized at different times based on the fused feature vector; The response module is used to generate operation instructions for the object to be recognized at different times based on the gesture recognition results and the preset mapping relationship between standard gestures and instructions. The first image processing network includes: The hourglass network extracts features from the hand region of a depth image to obtain hand feature data. The depth image spatial branch calculates a supervised attention map for each joint based on the hand features output by the hourglass network; the supervised attention map is calculated as follows: Calculate the attention map for each joint based on the hand features output by the hourglass network; The attention map is normalized to obtain a heatmap representing the probability of each pixel appearing at a joint position; Based on the heatmap, the recognition coordinates of each joint in the depth image space are calculated by integral operation; the attention map is supervised to obtain a supervised attention map by minimizing the average distance between the recognition coordinates and the actual coordinates of each joint in the depth image space. The attention enhancement branch calculates an unsupervised attention map for each joint based on the hand features output by the hourglass network; The depth extraction branch generates a dense depth feature map containing depth feature vectors for each joint based on the hand feature quantities output by the hourglass network. The attention fusion network fuses the supervised attention map with the unsupervised attention map to obtain a fused attention map, and identifies the true depth value of each joint based on the fused attention map and the dense depth feature map to obtain the depth image feature vector of the hand region.

6. The virtual reality teaching gesture recognition system as described in claim 5, characterized in that, Use an event camera to capture 3D point cloud images of the gesture changes of the object to be identified at different times; The Leap Motion controller is used to record depth images of the gesture changes of the object to be recognized at different times.

7. A computer device, characterized in that, It includes at least one processing unit and at least one storage unit, wherein the storage unit stores a computer program, and when the computer program is executed by the processing unit, the processing unit performs the steps of the virtual reality teaching gesture recognition method according to any one of claims 1 to 4.

Images