Real-time dynamic action recognition method and device based on convolutional neural network

Abstract

The invention relates to the field of augmented reality, and provides a real-time dynamic action recognition method and device based on a convolutional neural network, and the method comprises the steps: obtaining a plurality of groups of action data of a part where a user participates in a target action in a preset time window, and each group of action data comprises the three-dimensional information of a preset key point; and inputting the multiple groups of action data into a preset three-dimensional convolutional neural network, so that the three-dimensional convolutional neural network extracts spatial-temporal characteristics of the preset key point moving in a three-dimensional space according to the multiple groups of action data, and determines an action recognition result based on the extracted spatial-temporal characteristics. According to the method, interference of unnecessary information can be eliminated, limitation of the occupied range and the appearing position of a user operation part in a video picture is avoided, computing resources can be occupied as few as possible under the condition that the observation range is not lost, and therefore dynamic action recognition is achieved.

Description

Technical Field

[0001] The present application relates to the field of extended reality, and in particular to a real-time dynamic action recognition method and device based on convolutional neural networks. Background Art

[0002] Understanding user dynamic posture is an important requirement in XR (Extended Reality) devices.

[0003] Methods based on computer vision to recognize user dynamic postures are often easily affected by environmental factors such as lighting, background, and occlusion, which leads to low posture recognition accuracy.

[0004] Methods based on additional sensors to recognize user dynamic postures, such as wearing gloves with sensor devices to recognize gestures, although they overcome the influence of the above-mentioned environmental factors, wearing additional sensor devices will affect the user's operating experience.

[0005] Methods for recognizing dynamic user gestures based on deep learning algorithms can overcome these challenges. By training and learning from gesture images or video data, they extract gesture feature representations, thereby achieving high-precision gesture recognition and classification. Deep learning models, such as convolutional neural networks (CNNs), can achieve effective recognition by training and learning from large amounts of gesture images or video data. However, building deep learning models using gesture images or video data requires a large amount of training data (i.e., images and videos) and high computing resources. The training process is complex and faces the challenge of adjusting numerous parameters (including convolution kernel size, number, stride, and number of network layers). Parameter selection significantly impacts model performance, and finding the optimal parameter combination requires extensive, time-consuming and labor-intensive experimentation and tuning. Furthermore, the models suffer from poor interpretability, making it difficult to understand the basis for their decisions.

[0006] In summary, there is an urgent need for a solution that can recognize user dynamic postures with less computing resources and strong interpretability. Summary of the Invention

[0007] This application is used to solve the problems of existing methods of recognizing user dynamic postures based on convolutional neural networks based on images / videos as input information, such as the large amount of data required for convolution operations, the complex convolution process, the high computing resources occupied, and the poor interpretability of the features obtained by convolution.

[0008] In order to solve the above technical problems, the present application provides a real-time dynamic action recognition method based on convolutional neural network, comprising:

[0009] Acquire multiple sets of motion data of the user's parts participating in the target motion within a preset time window, each set of motion data including three-dimensional information of preset key points;

[0010] The multiple sets of motion data are input into a preset three-dimensional convolutional neural network, so that the three-dimensional convolutional neural network extracts the spatiotemporal features of the preset key points moving in three-dimensional space according to the multiple sets of motion data, and determines the motion recognition results based on the extracted spatiotemporal features.

[0011] As a further embodiment of the present application, the three-dimensional information of the key point includes three-dimensional information of at least one of the following parameters: position, rotation, acceleration and angular velocity.

[0012] As a further embodiment of the present application, the three-dimensional convolutional neural network includes: an input layer, a convolution layer, a pooling layer, and a fully connected layer;

[0013] The input layer is used to receive multiple sets of action data and construct preparation data for each channel;

[0014] The convolution layer has one or more convolution kernels, and the prepared data of each channel is convolved by the convolution kernel to obtain the spatiotemporal features of the key points;

[0015] The pooling layer is optional and is used to downsample the convolutional spatiotemporal features;

[0016] The fully connected layer is used to flatten the spatiotemporal features of the pooling layer or the convolutional layer into a one-dimensional vector, and perform classification based on the one-dimensional vector to determine an action recognition result.

[0017] As a further embodiment of the present application, the channel includes at least one of a first channel, a second channel, a third channel and a key point channel;

[0018] The first channel is used to construct spatiotemporal feature data of each set of action data;

[0019] The second channel is used to construct spatiotemporal variation feature data of adjacent groups of motion data at the same key point in the multiple groups of motion data;

[0020] The third channel is used to construct temporal and spatial relative change feature data of adjacent groups of action data at the same key point in the multiple groups of action data;

[0021] The key point channel is used to construct spatiotemporal feature data of important key points in each set of action data relative to the reference point.

[0022] As a further embodiment of the present application, the process of determining the convolution kernel of each channel includes:

[0023] Randomly determine the convolution kernel of each channel in the three-dimensional convolutional neural network model; and / or

[0024] Based on prior knowledge, determine a set of key points involved in the target action and their type information; and

[0025] According to the key points involved in the target action and their type information, the convolution kernel of each channel that conforms to the spatiotemporal characteristics of the user's target action is determined.

[0026] As a further embodiment of the present application, the convolution kernel of each channel that conforms to the spatiotemporal characteristics of the user's target action is determined based on the key points involved in the target action and their type information, including:

[0027] Obtain 3D information of key points in the process of the user completing the target action;

[0028] According to the key points and their type information, the three-dimensional information of the key points, and the functions of each channel, the convolution kernel of each channel is determined to meet the spatiotemporal characteristics of the user's target action.

[0029] As a further embodiment of the present application, obtaining three-dimensional information of key points in the process of the user completing the target action includes:

[0030] Controlling the extended reality device to display a guidance screen for the target action;

[0031] The user controls the key points involved in the target action according to the guidance screen to perform standard actions, and collects three-dimensional data of the key points.

[0032] As a further embodiment of the present application, for each channel, the process of determining the number of channel convolutions includes:

[0033] When the size of any spatial dimension in the convolution result of the channel is smaller than a preset threshold, or the convolution result of the time dimension of the channel is 1, the convolution is stopped.

[0034] The second aspect of the present application provides a real-time dynamic action recognition device based on a convolutional neural network, comprising:

[0035] a data acquisition unit, configured to acquire multiple sets of motion data of the user's parts participating in a target motion within a preset time window, each set of motion data including three-dimensional data of preset key points;

[0036] A prediction unit is used to input the multiple sets of motion data into a preset three-dimensional convolutional neural network, so that the three-dimensional convolutional neural network can extract the spatiotemporal features of the preset key points moving in three-dimensional space based on the multiple sets of motion data, and determine the motion recognition result based on the extracted spatiotemporal features.

[0037] A third aspect of the present application provides an extended reality device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method described in any of the aforementioned embodiments when executing the computer program.

[0038] A fourth aspect of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor of an augmented reality device, the method described in any of the aforementioned embodiments is implemented.

[0039] A fifth aspect of the present application provides a computer program product, which includes a computer program. When the computer program is executed by a processor of an extended reality device, the method described in any of the above embodiments is implemented.

[0040] The real-time dynamic action recognition method and device based on convolutional neural networks provided in this application innovate the existing method of image or video-based action recognition into a method of action recognition based on multiple sets of action data including three-dimensional information of preset key points. It can eliminate the interference of unnecessary information, is not restricted by the range and position of the user's operating part in the video screen, and can achieve the recognition of dynamic actions while occupying as few computing resources as possible without losing the observation range.

[0041] Specifically, a 3D convolutional neural network has one or more channels, each with a convolution kernel. The kernel is determined based on the spatiotemporal characteristics of the target action, improving recognition accuracy and reducing computing resource requirements. Furthermore, the kernel directly reflects the spatiotemporal characteristics of the target action, enhancing the interpretability of the recognition results.

[0042] In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0044] Figure 1 The structure diagram of the real-time dynamic action recognition system based on convolutional neural network in an embodiment of the present application is shown;

[0045] Figure 2A A schematic diagram showing key points of a hand according to an embodiment of the present application is shown;

[0046] Figure 2B A schematic diagram showing key points of the face according to an embodiment of the present application is shown;

[0047] Figure 3A flow chart of a real-time dynamic action recognition method based on a convolutional neural network according to an embodiment of the present application is shown;

[0048] Figure 4 A schematic diagram of a three-dimensional convolutional neural network according to an embodiment of the present application is shown;

[0049] Figure 5 Shown is a schematic diagram of the channel characteristics of this application;

[0050] Figure 6 A flowchart of a convolution kernel determination process according to an embodiment of the present application is shown;

[0051] Figure 7 The structure diagram of the real-time dynamic action recognition device based on convolutional neural network in an embodiment of the present application is shown;

[0052] Figure 8 A structural diagram of an extended reality device according to an embodiment of the present application is shown;

[0053] Figure 9A and Figure 9B A schematic diagram showing a pinching action according to an embodiment of the present application is shown;

[0054] Figure 10 A schematic diagram of the convolution process is shown.

[0055] Description of the accompanying symbols:

[0056] 101. Motion data collection equipment;

[0057] 102. Processing equipment;

[0058] 401, input layer;

[0059] 402, convolutional layer;

[0060] 403, pooling layer;

[0061] 404, fully connected layer;

[0062] 701, data acquisition unit;

[0063] 702, prediction unit;

[0064] 802. Extended reality devices;

[0065] 804, processor;

[0066] 806, memory;

[0067] 808, driving mechanism;

[0068] 810, input / output module;

[0069] 812. Input devices;

[0070] 814. Output device;

[0071] 816. Presentation equipment;

[0072] 818. Graphical User Interface;

[0073] 820, network interface;

[0074] 822, communication link;

[0075] 824. Communication bus. DETAILED DESCRIPTION

[0076] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0077] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, device, product or equipment comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or equipment.

[0078] This specification provides method operation steps as described in the embodiments or flowcharts, but more or fewer operation steps may be included based on routine or non-creative work. The order of steps listed in the embodiments is only one way of executing the steps among many orderings and does not represent the only execution order. When a system or device product is actually executed, the method can be executed in the order shown in the embodiments or the drawings or in parallel.

[0079] It should be noted that the real-time dynamic action recognition method and device based on convolutional neural network of the present application are applicable to extended reality devices and can also be applied to other interactive scenarios.

[0080] It should be noted that the data involved in this application (including but not limited to data used for analysis, stored data, displayed data, etc.) are all information and data authorized by the user or fully authorized by all parties, and the acquisition, transmission, storage, use and processing of relevant data comply with the relevant laws, regulations and standards of relevant countries and regions.

[0081] It should be noted that in the embodiments of the present application, certain software, components, models and other existing solutions in the industry may be mentioned. They should be regarded as exemplary. Their purpose is only to illustrate the feasibility of implementing the technical solution of the present application, but it does not mean that the applicant has or will necessarily use the solution.

[0082] In existing methods for recognizing user gestures using convolutional neural networks (CNNs) as input, the size of the input layer's observation range directly affects the complexity of the convolution operation, necessitating a balance between computational complexity and the range of information acquired by the neurons. To keep computational complexity within the capabilities of computing resources, some input size must be sacrificed—that is, the original input frame is cropped to reduce the size of the "observation range." This results in nearby, larger objects that conform to facial features and are outside the "observation range" being missed. In summary, existing methods for recognizing user gestures using CNNs as input suffer from the large amount of data required for the convolution operation, the complex convolution process, the high computational resource consumption, and the poor interpretability of the convolutional features. Furthermore, there is a problem of loss of observation range, which leads to low recognition accuracy. The user gestures described in this application include, but are not limited to, gestures, expressions, and lip gestures, and are suitable for gesture recognition, lip reading, and expression recognition scenarios.

[0083] In order to solve the above technical problems, this application provides a real-time dynamic action recognition system based on convolutional neural network, such as Figure 1 As shown, it includes: a motion data acquisition device 101 and a processing device 102.

[0084] The motion data acquisition device 101 is used to collect motion data of the user's parts participating in the target motion. Each set of motion data includes three-dimensional information of preset key points, and the motion data is sent to the processing device 102.

[0085] The processing device 102 is used to receive the motion data sent by the motion data acquisition device 101, and input the multiple sets of motion data of the parts of the user participating in the target motion within a preset time window into a preset convolutional neural network, so that the convolutional neural network can extract the spatiotemporal features of the movement of the preset key points in the three-dimensional space based on the multiple sets of motion data, and determine the motion recognition result based on the extracted spatiotemporal features.

[0086] In detail, the target actions and parts described in this application can be determined according to the device usage scenario. For example, in a hand interaction scenario, the part is the hand, and the target action is, for example, fist clenching, pinching, etc. For another example, in an eye interaction scenario, the part is the eye, and the target action is, for example, eye movement, blinking, etc. The preset key points can reflect the action information of the part, and when implemented, they can be set according to the characteristics of the recognized action, such as Figure 2A As shown in , the hand key points include 21 hand joint points, such as Figure 2B As shown in FIG, the facial key points include 68 key points, of which 20 key points representing the lip area can be used to identify lip movements such as the upward movement of the mouth corners. In specific implementations, the key points of the hand and lip areas can also be set in other ways.

[0087] The three-dimensional information of key points includes three-dimensional information of at least one of the following parameters: position, rotation, acceleration, and angular velocity. In some embodiments, the three-dimensional information of key points can be acquired visually, for example, by using cameras or depth cameras arranged at different angles. In some real-time methods, the three-dimensional information of key points can also be captured by deploying sensors at the target motion site, such as data gloves. This application does not specifically limit the method for capturing the three-dimensional information of key points.

[0088] The preset time window can be a preset value, and can also be set by the user according to the recognition accuracy or scene requirements. This application does not limit its specific value. The multiple groups of action data can be action data sampled from all information collected within the preset time window. The number F of multiple groups of action data can be changed according to computing power or specific needs. This application does not limit this. For example, if the number F of multiple groups of action data is 5, and 41 groups of data are sampled within the preset time window, then the multiple groups of action data include the 1st group, the 11th group, the 21st group, the 31st group, and the 41st group of action data. During specific implementation, multiple sets of action data may also include the currently collected action data (denoted as F_0), the action data sampled 10 times before the current action data (denoted as F_-10), the action data sampled 20 times before the current action data (denoted as F_-20), the action data sampled 10 times after the current action data (denoted as F_10), and the action data sampled 20 times after the current action data (denoted as F_20). Multiple sets of action data are expressed as (F_-20, F_-10, F_0, F_10, F_20), and the data volume of each set of action data is K*(3*x), where K is the number of key points and x is the number of parameters, such as position and rotation.

[0089] A 3D convolutional neural network consists of at least one or more channels and a fully connected layer. Each channel constructs a type of feature data based on the raw data. Each channel has a convolution kernel, determined based on the spatiotemporal characteristics of the target action. Each kernel is used to extract a spatiotemporal feature of the movement of a preset key point in 3D space. The fully connected layer integrates the spatiotemporal features of all channels to determine the action recognition result.

[0090] In some embodiments, when a 3D convolutional neural network is designed to recognize only one type of motion for a certain part of the body (e.g., a pinching motion of the hand), the output of the fully connected layer can be classified into two categories: yes or no motion; or three categories: yes or no motion, or unrecognizable. Multiple convolutional networks can be designed to recognize different types of motion for a certain part of the body.

[0091] In some embodiments, when the three-dimensional convolutional neural network is designed to simultaneously recognize multiple actions of a certain part of the body (such as pinching vs. grabbing vs. making an OK gesture of the hand), it can be divided into three categories based on the output results of the fully connected layer: the first action (pinching), the second action (grabbing), or the third action (making an OK gesture); or it can be divided into four categories: the first action (pinching), the second action (grabbing), the third action (making an OK gesture), or it cannot be recognized.

[0092] This application takes into account that when making a target action, the three-dimensional information of the action part (such as three-dimensional position information, three-dimensional rotation information, etc.) will maintain a characteristic relationship. Based on this, this application innovates the existing method of image or video-based action recognition into a method of action recognition based on multiple sets of action data including three-dimensional information of preset key points. It can eliminate the interference of unnecessary information and is not limited by the range and position of the user's operating part in the video screen. It solves the problems of exceeding boundaries and occlusion that the existing image or video-based action recognition methods cannot solve. It can occupy as few computing resources as possible without losing the observation range to achieve the recognition of dynamic actions. If the original computing conditions allow, more sets of action data can be selected to make the recognition results more accurate.

[0093] Specifically, a three-dimensional convolutional neural network has one or more channels, each with a convolution kernel. The convolution kernel is determined based on the spatiotemporal characteristics of the target action, which can improve the accuracy of the recognition results and eliminate the process of obtaining convolution parameters (convolution kernel) through model training, reducing the demand for computing resources. In addition, the convolution kernel directly reflects the spatiotemporal characteristics of the target action, rather than relying solely on visual features. Based on information such as position, rotation, acceleration, and angular velocity, it can make the recognition results more stable and improve the interpretability of the recognition results.

[0094] In one embodiment of the present application, a real-time dynamic action recognition method based on a convolutional neural network is also provided. The real-time dynamic action recognition method based on a convolutional neural network is applied to the terminal side or the network side. The terminal side, that is, the program related to the real-time dynamic action recognition method based on a convolutional neural network runs in the extended reality device, and the network side, that is, the program related to the real-time dynamic action recognition method based on a convolutional neural network runs in the server.

[0095] Specifically, such as Figure 3 As shown in FIG, the real-time dynamic action recognition method based on convolutional neural network includes:

[0096] Step 301 : Acquire multiple sets of motion data of the user's parts participating in a target motion within a preset time window, each set of motion data including three-dimensional information of preset key points.

[0097] The three-dimensional structure of multiple sets of motion data can be expressed as F*[K*(3*x)], where K is the number of key points, x is the number of parameters, and F is the number of sets of motion data. In some real-time methods, the three-dimensional information of the key points includes three-dimensional information of at least one of the following parameters: position, rotation, acceleration, and angular velocity.

[0098] In step 302, the multiple sets of motion data are input into a preset three-dimensional convolutional neural network, so that the three-dimensional convolutional neural network extracts the spatiotemporal features of the preset key points moving in the three-dimensional space according to the multiple sets of motion data, and determines the motion recognition result based on the extracted spatiotemporal features.

[0099] like Figure 4 As shown, the three-dimensional convolutional neural network includes: an input layer 401, a convolution layer 402, a pooling layer 403, and a fully connected layer 404.

[0100] The input layer 401 is used to receive multiple sets of motion data and construct the preparation data for each channel. The three-dimensional structure of the motion data received by the input layer 401 is expressed as F*[K*(3*x)], where K is the number of key points, x is the number of parameters, 3*x is the spatial data volume of the parameters, and F is the number of groups of motion data. At the same time, the input layer 401 is also used to pre-process the received motion data and construct the following Figure 5 The prepared data for each channel is shown on the right.

[0101] Convolutional layer 402 has one or more convolution kernels, which convolve the prepared data of each channel with the convolution kernel to obtain the spatiotemporal characteristics of the key points. When each channel has a convolution kernel, the convolution kernel of each channel is determined based on the spatiotemporal characteristics of the target action part in each channel when performing the target action.

[0102] For example, pinching actions such as Figure 9A and Figure 9BAs shown in the figure, a spatial feature convolution kernel representing the index finger and thumb pinching action is constructed to represent the spatial positions of the thumb and index fingertips and the wrist. For example, the spatial position of the thumb is Vector3(-0.08, 0.98, 0.23); the spatial position of the index finger is Vector3(-0.08, 0.98, 0.24); and the spatial position of the wrist is Vector3(-0.17, 0.91, 0.2). This spatial feature indicates that the two fingertips are spatially within a very small range, meaning that they are almost touching.

[0103] Correspondingly, the convolution kernel of the spatial feature is: each column represents the spatial position information of the key node, as shown in Table 1 below:

[0104] Table 1

[0105] -0.08 -0.08 -0.17 0.98 0.98 0.91 0.23 0.24 0.20

[0106] The movement process of the convolution kernel is as follows Figure 10 shown.

[0107] In one embodiment, the channel includes at least one of a first channel, a second channel, a third channel, and a keypoint channel group. The following describes the data constructed by each channel, taking the example of spatial information for each keypoint including three spatial coordinates and three rotational information. During implementation, F frames are selected, each corresponding to spatial information for K keypoints.

[0108] The first channel is used to construct the spatiotemporal (including position and rotation) feature data of each set of action data. The three-dimensional structure of the feature data constructed by the first channel is F@(K*6).

[0109] The second channel is used to construct the spatiotemporal change (including position and rotation) feature data of adjacent groups of action data at the same key points in the multiple groups of action data, that is, the difference between adjacent groups of action data at the same key points (that is, the change of position and angle in the three directions of x, y, and z). F frames consist of (F-1) frame pairs, and the three-dimensional structure of the feature data constructed by the second channel is (F-1)@(K*6).

[0110] The third channel is used to construct the feature data of the relative spatiotemporal changes (including position and rotation) of adjacent groups of action data at the same key points in the multiple groups of action data, that is, the absolute value of the difference between adjacent groups of action data at the same key points (that is, the size of the change in position and angle in the three directions of x, y, and z). F frames consist of (F-1) frame pairs, and the three-dimensional structure of the feature data constructed by the third channel is (F-1)@(K*6).

[0111] The keypoint channel is used to construct spatiotemporal feature data for important keypoints in each set of motion data relative to reference points. There are K_check groups of keypoint channels, each with F frames, where K_check represents the number of important keypoints. The reference points in each keypoint channel are keypoints other than the keypoints in that channel, meaning there are K-1 reference points, where K represents the number of keypoints. The data for each keypoint channel represents the difference between the important keypoints in each set of motion data and the reference points (i.e., the difference in position and angle in the x, y, and z directions). The three-dimensional structure of the feature data composed of F frames for each keypoint channel is F@[(K-1)*6].

[0112] Specifically, important key points are key points with a large range of key point changes in the target action area, that is, important key points that can distinguish the target action. Taking the 21 key points of the hand as an example, that is, K = 21, the important key points for pinching (pinch) are the index finger and the thumb tip, that is, K_check is 2, there are two groups of key point channels, and the reference point is the key point other than the important key point among the K key points, that is, K-1 key points, where 1 represents the important key point itself. The pinch action only involves the relative position relationship of the two important key points of the index finger and thumb, and does not need to pay attention to the relative position relationship of the little finger, ring finger, etc. The scissors (V) action only requires the relative position relationship of important key points such as the index finger, middle finger, and ring finger.

[0113] During specific implementation, the channel can be determined based on the recognition accuracy and action. In addition to the number of channels exemplified in this application, other numbers of channels can also be set.

[0114] In specific implementation, in order to further reduce the amount of computation, if the characteristics of the action are more obvious at certain key points, the three-dimensional convolutional network can be designed to include only the channels and convolution kernels of these key points.

[0115] The convolution of each channel can extract one aspect of the features of the original data at a time. For the recognition of dynamic objects, it is necessary to make a comprehensive judgment on multiple input features. For example, the position and rotation changes of each joint on the x, y, and z axes between the selected adjacent motion data. In the convolutional neural network, each layer uses multiple filters to form multiple channels, and each channel extracts abstract features of different aspects of the previous layer. Therefore, the present invention preprocesses the input data to establish multiple convolution channels. Taking the prediction of hand movements of 21 key points as an example, each set of data is a matrix of 21 columns * (3 spatial information + 3 rotation information) rows. Each column represents the spatial information of a key point, that is, the spatial position of the point and its rotation direction. The same columns of the two sets of data represent that the same key point is in different spatial positions and directions due to movement at different times.

[0116] like Figure 5 As shown, Figure 5 Each frame of data is a set of action data, and F frames of data include F sets of action data. The channels include the first channel, the second channel, the third channel, and the key point channel. The feature data constructed by the channels includes: F@(K*6), (F-1)@(K*6), (F-1)@(K*6), K_check, and F@[(K-1)*6] feature information. If (K*6) and (K-1)*6 are regarded as a feature map, the 3D convolutional neural network input layer can construct F+K_check*F+(F–1)*2 feature maps (FeatureMap).

[0117] In a specific embodiment, for the first channel, the second channel and the third channel, the convolution kernel size is G*3*2, where G*3 is the spatial feature, 3 is 3 adjacent rows, and the three-dimensional information combination of the key points (i.e., spatial position or rotation or a combination thereof) is recorded. G is the adjacent G columns, that is, the adjacent G key points, and G is the number of a group of connected key points. For example, the five fingers on the hand can be divided into 5 groups, and each group has 4 key points from the fingertip to the last knuckle, so G is 4. In this way, the spatial features of a group of key points connected (associated) by bones can be obtained. When this application is implemented, the value range of G is not limited. In different usage scenarios, G can be different values. For example, in the lip reading recognition scenario, G can be 5 key points representing the right upper lip, and G is 5. The last 2 is the time feature, that is, two adjacent groups of action data. (The time feature is also not limited to 2, and 3 groups, 4 groups, etc. can be operated simultaneously. 2 is the minimum value).

[0118] For the important keypoint channel, a convolution kernel of 2 (columns) * 2 (rows) * 2 (groups) is used. 2 (columns) * 2 (rows) represent the spatial relationship between adjacent keypoints. 2 (groups) represents the convolution kernel in the temporal dimension, and 3 groups can also be used, depending on the number of action data groups F selected. For complex actions, requiring more keyframes, multiple groups can be selected by sliding over time.

[0119] The pooling layer 403 is optional and is used to downsample the convolutional spatiotemporal features. In practice, a pooling operation may be performed after each convolution to reduce the data dimension.

[0120] The fully connected layer 404 is used to flatten the spatiotemporal features of the pooling layer or convolutional layer into a one-dimensional vector, and then classify the one-dimensional vector to determine the action recognition result. The fully connected layer 404 can use existing fully connected methods, such as downsampling and linear full connection. The output of the fully connected layer can be whether a certain action was performed, or an action classification label, such as 1, 2, etc., where each classification label represents a type of action.

[0121] In an example of the present application, for each channel of the convolution layer, the process of determining the number of convolutions of the channel includes:

[0122] When the size of any spatial dimension in the convolution result of the channel is smaller than a preset threshold, or the convolution result of the time dimension of the channel is 1, the convolution is stopped.

[0123] The convolution kernel used in each convolution in each channel is the same.

[0124] In one embodiment of the present application, the process of determining the convolution kernel of each channel includes:

[0125] Method 1: Randomly determine the convolution kernel of each channel in the 3D convolutional neural network model.

[0126] Method 2: Based on prior knowledge, a set of key points involved in the target action and their type information are determined. Based on this information, convolution kernels for each channel are determined that match the spatiotemporal characteristics of the user's target action. Key point types include important key points and reference points. The definitions of important key points and reference points are similar to those in the previous embodiment.

[0127] In specific implementation, the convolution kernels of each channel that conform to the spatiotemporal characteristics of the user's target action are determined based on the key points involved in the target action and their type information, including: obtaining the three-dimensional information of the key points in the process of the user completing the target action; and determining the convolution kernels of each channel that conform to the spatiotemporal characteristics of the user's target action based on the key points and their type information, the three-dimensional information of the key points, and the functions of each channel.

[0128] In some implementations, obtaining three-dimensional information of key points in the process of the user completing the target action includes:

[0129] Controlling the extended reality device to display a guidance screen for the target action;

[0130] The user controls the key points involved in the target action according to the guidance screen to perform standard actions, and collects three-dimensional data of the key points and reference key points.

[0131] This embodiment can quickly and accurately obtain the three-dimensional information of the key points that match the user, thereby improving the accuracy of the subsequent determination of the convolution kernel. In specific implementation, other methods can also be used to obtain standard action sample data of the user's operating parts.

[0132] Considering that the method of randomly determining the convolution kernel has the problem of low recognition accuracy, such as Figure 6 As shown, the convolution kernel of each channel can also be determined by the following method three.

[0133] Method three includes:

[0134] Step 601: randomly determine the convolution kernel of each channel in the three-dimensional convolutional neural network model.

[0135] Step 602: Acquire a plurality of pre-collected standard action sample data and action label data of the user's operating parts.

[0136] Step 603: input the standard action sample data into the three-dimensional convolutional neural network model to predict the action recognition result.

[0137] Step 604: When the predicted action recognition result is different from the action label data of the action sample data, the convolution kernel of each channel in the three-dimensional convolutional neural network model is adjusted.

[0138] Step 605: When the action recognition results obtained by multiple consecutive predictions are the same as the action recognition results of the action sample data, the final convolution kernel of each channel in the three-dimensional convolutional neural network model is obtained.

[0139] This embodiment adjusts the convolution kernel through user standard action sample data, which can improve the recognition accuracy of user actions, and can reduce the complexity and resource requirements of training by fine-tuning the convolution kernel.

[0140] Based on the same inventive concept, the present application also provides a real-time dynamic action recognition device based on a convolutional neural network, as described in the following embodiments. Since the principle of solving the problem of the real-time dynamic action recognition device based on a convolutional neural network is similar to that of the real-time dynamic action recognition method based on a convolutional neural network, the implementation of the real-time dynamic action recognition device based on a convolutional neural network can refer to the real-time dynamic action recognition method based on a convolutional neural network, and the repeated parts will not be repeated.

[0141] Specifically, such as Figure 7 As shown, the real-time dynamic action recognition device based on convolutional neural network includes:

[0142] A data acquisition unit 701 is configured to acquire multiple sets of motion data of the user's parts participating in a target motion within a preset time window, each set of motion data including three-dimensional data of preset key points;

[0143] The prediction unit 702 is used to input the multiple sets of action data into a preset convolutional neural network, so that the convolutional neural network extracts the spatiotemporal features of the movement of the preset key points in three-dimensional space based on the multiple sets of action data, and determines the action recognition result based on the extracted spatiotemporal features.

[0144] This embodiment innovates the existing method of recognizing actions based on images or videos into a method of recognizing actions based on multiple sets of action data including three-dimensional information of preset key points. It can eliminate the interference of unnecessary information, is not restricted by the range and position of the user's operating parts in the video screen, and can achieve the recognition of dynamic actions while occupying as few computing resources as possible without losing the observation range.

[0145] Specifically, a three-dimensional convolutional neural network has one or more channels, each with a convolution kernel. The kernel is determined based on the spatiotemporal characteristics of the target action, improving recognition accuracy while eliminating model training and reducing computing resource requirements. Furthermore, the kernel directly reflects the spatiotemporal characteristics of the target action, enhancing the interpretability of the recognition results.

[0146] In one embodiment of the present application, an extended reality device is provided, such as Figure 8 As shown, 802 may include one or more processors 804, such as one or more central processing units (CPUs), each of which may implement one or more hardware threads. The extended reality device 802 may also include any memory 806 for storing any type of information, such as code, settings, data, and the like. For example, and without limitation, the memory 806 may include any one or more combinations of the following: any type of RAM, any type of ROM, a flash memory device, a hard disk, an optical disk, and the like. More generally, any memory may use any technology to store information. Furthermore, any memory may provide volatile or non-volatile retention of information. Furthermore, any memory may represent a fixed or removable component of the extended reality device 802. In one embodiment, when the processor 804 executes associated instructions stored in any memory or combination of memories, the extended reality device 802 may perform any operation of the associated instructions. The extended reality device 802 also includes one or more drive mechanisms 808, such as a hard disk drive mechanism, an optical disk drive mechanism, and the like, for interacting with any memory.

[0147] The extended reality device 802 may also include an input / output module 810 (I / O) for receiving various inputs (via input device 812) and for providing various outputs (via output device 814). A specific output mechanism may include a presentation device 816 and an associated graphical user interface 818 (GUI). In other embodiments, the input / output module 810 (I / O), input device 812, and output device 814 may not be included, and the device may simply serve as an extended reality device in a network. The extended reality device 802 may also include one or more network interfaces 820 for exchanging data with other devices via one or more communication links 822. One or more communication buses 824 couple the components described above together.

[0148] The communication link 822 may be implemented in any manner, for example, via a local area network, a wide area network (e.g., the Internet), a point-to-point connection, etc., or any combination thereof. The communication link 822 may include any combination of hardwired links, wireless links, routers, gateway functions, name servers, etc., governed by any protocol or combination of protocols.

[0149] An embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above method are executed.

[0150] An embodiment of the present application also provides a computer-readable instruction, wherein when a processor executes the instruction, the program therein causes the processor to execute the method shown in any of the aforementioned embodiments.

[0151] It should be understood that in the various embodiments of the present application, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0152] It should also be understood that in the embodiments of this application, the term "and / or" is merely a description of the relationship between associated objects, indicating that three possible relationships exist. For example, A and / or B can represent: A exists alone, A and B exist simultaneously, or B exists alone. Furthermore, the character " / " in this application generally indicates that the associated objects are in an "or" relationship.

[0153] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed in this application can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0154] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0155] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, or can be electrical, mechanical or other forms of connection.

[0156] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the embodiments of the present application.

[0157] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0158] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or partly contributed to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0159] Specific embodiments are used in this application to illustrate the principles and implementation methods of this application. The description of the above embodiments is only used to help understand the method and core ideas of this application. At the same time, for those skilled in the art, according to the ideas of this application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting this application.

Claims

1. A real-time dynamic action recognition method based on convolutional neural network, characterized in that: include: Acquire multiple sets of motion data of the user's parts participating in the target motion within a preset time window, each set of motion data including three-dimensional information of preset key points; The multiple sets of motion data are input into a preset three-dimensional convolutional neural network, so that the three-dimensional convolutional neural network extracts the spatiotemporal features of the preset key points moving in three-dimensional space according to the multiple sets of motion data, and determines the motion recognition results based on the extracted spatiotemporal features.

2. The method according to claim 1, wherein The three-dimensional information of the key point includes three-dimensional information of at least one of the following parameters: position, rotation, acceleration, and angular velocity.

3. The method according to claim 1, wherein The three-dimensional convolutional neural network includes: an input layer, a convolution layer, a pooling layer, and a fully connected layer; The input layer is used to receive multiple sets of action data and construct preparation data for each channel; The convolution layer has one or more convolution kernels, and the prepared data of each channel is convolved by the convolution kernel to obtain the spatiotemporal features of the key points; The pooling layer is optional and is used to downsample the convolutional spatiotemporal features; The fully connected layer is used to flatten the spatiotemporal features of the pooling layer or the convolutional layer into a one-dimensional vector, and perform classification based on the one-dimensional vector to determine an action recognition result.

4. The method according to claim 3, wherein The channel includes at least one channel among a first channel, a second channel, a third channel and a key point channel; The first channel is used to construct spatiotemporal feature data of each set of action data; The second channel is used to construct spatiotemporal variation feature data of adjacent groups of motion data at the same key point in the multiple groups of motion data; The third channel is used to construct temporal and spatial relative change feature data of adjacent groups of action data at the same key point in the multiple groups of action data; The key point channel is used to construct spatiotemporal feature data of important key points in each set of action data relative to the reference point.

5. The method according to claim 4, wherein The process of determining the convolution kernel of each channel includes: Randomly determine the convolution kernel of each channel in the three-dimensional convolutional neural network model; and / or Based on prior knowledge, determine a set of key points involved in the target action and their type information; and According to the key points involved in the target action and their type information, the convolution kernel of each channel that conforms to the spatiotemporal characteristics of the user's target action is determined.

6. The method according to claim 5, wherein According to the key points involved in the target action and their type information, the convolution kernel of each channel that conforms to the spatiotemporal characteristics of the user's target action is determined, including: Obtain 3D information of key points in the process of the user completing the target action; According to the key points and their type information, the three-dimensional information of the key points, and the functions of each channel, the convolution kernel of each channel is determined to meet the spatiotemporal characteristics of the user's target action.

7. The method according to claim 6, wherein The acquisition of three-dimensional information of key points in the process of the user completing the target action includes: Controlling the extended reality device to display a guidance screen for the target action; The user controls the key points involved in the target action according to the guidance screen to perform standard actions, and collects three-dimensional data of the key points.

8. The method according to claim 3, wherein For each channel, the process of determining the number of convolutions for the channel includes: When the size of any spatial dimension in the convolution result of the channel is smaller than a preset threshold, or the convolution result of the time dimension of the channel is 1, the convolution is stopped.

9. A real-time dynamic action recognition device based on convolutional neural network, characterized in that: include: a data acquisition unit, configured to acquire multiple sets of motion data of the user's parts participating in a target motion within a preset time window, each set of motion data including three-dimensional data of preset key points; A prediction unit is used to input the multiple sets of motion data into a preset three-dimensional convolutional neural network, so that the three-dimensional convolutional neural network can extract the spatiotemporal features of the preset key points moving in three-dimensional space based on the multiple sets of motion data, and determine the motion recognition result based on the extracted spatiotemporal features.

10. An extended reality device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the method according to any one of claims 1 to 8 is implemented.

11. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor of an augmented reality device, the method according to any one of claims 1 to 8 is implemented.

12. A computer program product, comprising a computer program, characterized in that: When the computer program is executed by a processor of an augmented reality device, the method according to any one of claims 1 to 8 is implemented.

Images