A data processing method and related products

Abstract

The embodiment of the present application discloses a data processing method and related products, the data processing method comprising: obtaining target motion data to be processed, the target motion data is obtained by collecting the motion of the object using a micro-electric inertial navigation sensor; parsing the target motion data to determine the type of motion, the type includes a first type and a second type, the motion amplitude of the first type is greater than the motion amplitude of the second type; if the type of motion is the first type, the target motion data is identified and processed in the time domain dimension; if the type of motion is the second type, the target motion data is identified and processed in the frequency domain dimension. By adopting the embodiment of the present application, it is possible to ensure the recognition accuracy in the process of motion recognition based on camera-free, reduce the computing power requirements for recognition processing, improve recognition efficiency, thereby reducing the delay and cost of recognition processing, and improving the recognition experience.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a data processing method, a data processing device, a computer equipment, a computer-readable storage medium, and a computer program product. Background Art

[0002] With the development of artificial intelligence technology, various interactive devices, based on their intelligent and automated characteristics, bring many conveniences and interesting experiences to people's production and life. At present, some wearable devices, virtual interactive devices, etc. can capture and recognize the actions made by people in reality (such as gestures and human body movements), and apply the recognition results to virtual control operations, such as using the human body movements in reality to drive virtual characters to perform the same actions, etc. The recognition used in the virtual reality interaction solution can be based on camera image recognition. This recognition method has a high camera hardware cost, and is limited by the camera's shooting range, image acquisition effect, etc., and the recognition result is not accurate enough. At present, virtual reality interaction solutions are more based on non-camera action recognition, for example: motion capture through sensors, and then recognition through recognition algorithms. This method has relatively higher precision and accuracy than image recognition. However, practice has found that the current non-camera action recognition method has problems such as high latency, high cost, and poor experience. Summary of the invention

[0003] The embodiments of the present application provide a data processing method and related products, which can ensure recognition accuracy in the process of camera-free motion recognition while reducing the computing power requirements for recognition processing and improving recognition efficiency, thereby reducing the delay and cost of recognition processing and improving the recognition experience.

[0004] On the one hand, an embodiment of the present application provides a data processing method, including:

[0005] Acquire the target motion data to be processed, where the target motion data is acquired by using a micro-electric inertial navigation sensor to collect the motion of the object;

[0006] Parsing the target motion data to determine the type of motion, the type including a first type and a second type, the motion amplitude of the first type being greater than the motion amplitude of the second type;

[0007] If the type of the action is the first type, the target action data is identified and processed in the time domain dimension;

[0008] If the type of the action is the second type, the target action data is identified and processed in the frequency domain dimension.

[0009] On the one hand, an embodiment of the present application provides a data processing device, including:

[0010] An acquisition module is used to acquire target motion data to be processed, where the target motion data is acquired by using a micro-electric inertial navigation sensor to collect the motion of the object;

[0011] A parsing module, used for parsing the target motion data to determine the type of motion, the type including a first type and a second type, the motion amplitude of the first type being greater than the motion amplitude of the second type;

[0012] an identification and processing module, configured to perform identification and processing on target action data in a time domain dimension if the action type is the first type;

[0013] The recognition and processing module is further used to perform recognition processing on the target action data in the frequency domain dimension if the action type is the second type.

[0014] In one embodiment, the parsing module is specifically used to: calculate the energy of the target action data; if the energy of the target action data is greater than or equal to a preset threshold, determine the type of action to be the first type; if the energy of the target action data is less than the preset threshold, determine the type of action to be the second type.

[0015] In one embodiment, the energy of the target action data includes the average energy of the target action data and the maximum variance of the target action data; the average energy of the target action data is used to reflect the intensity of the movement of the action indicated by the target action data; the maximum variance of the target action data is used to reflect the energy change of the action indicated by the target action data; the preset threshold includes a preset energy threshold and a preset variance threshold; the energy of the target action data is greater than or equal to the preset threshold, which means: the average energy of the target action data is greater than or equal to the preset energy threshold, and the maximum variance of the target action data is greater than or equal to the preset variance threshold; the energy of the target action data is less than the preset threshold, which means: the average energy of the target action data is less than the preset energy threshold; or the maximum variance of the target action data is less than the preset variance threshold; or the average energy of the target action data is less than the preset energy threshold, and the maximum variance of the target action data is less than the preset variance threshold.

[0016] In one embodiment, the acquisition module is further used to: acquire the action timing signal sequence obtained by the micro-electric inertial navigation sensor collecting the action of the object; the preprocessing module is further used to: preprocess the action timing signal sequence to determine the execution starting point of the action.

[0017] In one embodiment, the preprocessing module is specifically used to: locate the starting sampling point of the first time window T1 to the first action timing signal in the action timing signal sequence; detect whether the energy of the action timing signal corresponding to the starting sampling point of the first time window T1 meets the preset condition; if the energy of the action timing signal corresponding to the starting sampling point of the first time window T1 meets the preset condition, then take the starting sampling point of the first time window T1 as a reference, and perform sequential sampling in the action timing signal sequence according to the first time window T1 to obtain multiple sampling points in the first time window T1; detect whether the energy mean of the action timing signal corresponding to all sampling points in the first time window T1 meets the preset condition; if the energy mean of the action timing signal corresponding to all sampling points in the first time window T1 meets the preset condition, then determine the starting sampling point of the first time window T1 as the execution starting point of the action.

[0018] In one embodiment, the preprocessing module is specifically used to: if the energy of the action timing signal corresponding to the starting sampling point of the first time window T1 does not meet the preset condition, or if the energy mean of the action timing signal corresponding to all sampling points in the first time window T1 does not meet the preset condition, then the first time window T1 is sequentially shifted in the action timing signal sequence to reposition the starting sampling point of the first time window T1.

[0019] In one embodiment, an action timing signal includes an acceleration signal and an angular velocity signal, and the energy of the action timing signal corresponding to the starting sampling point of the first time window T1 satisfies the preset condition, which means: the angular velocity energy of the action timing signal corresponding to the starting sampling point of the first time window T1 is greater than or equal to the preset angular velocity energy threshold, and the acceleration energy of the action timing signal corresponding to the starting sampling point of the first time window T1 is greater than or equal to the preset acceleration energy threshold; the energy mean of the action timing signal corresponding to all sampling points in the first time window T1 satisfies the preset condition, which means: the angular velocity average energy of the action timing signal corresponding to all sampling points in the first time window T1 is greater than or equal to the preset angular velocity energy threshold, and the acceleration average energy of the action timing signal corresponding to all sampling points in the first time window T1 is greater than or equal to the preset acceleration energy threshold.

[0020] In one embodiment, the acquisition module is specifically used to: take the execution starting point of the action as a reference, sequentially sample the action timing signal sequence according to the time window, and obtain action data corresponding to multiple time windows respectively; wherein, the starting sampling point of the first time window T1 is the execution starting point of the action; the target action data to be processed refers to the action data corresponding to the i-th time window Ti, and the action data corresponding to the i-th time window Ti includes the action timing signals corresponding to each sampling point in the i-th time window Ti; the i-th time window Ti is any one of the multiple time windows, and i is a positive integer.

[0021] In one embodiment, the recognition and processing module is specifically used to: if the type of action is the first type, extract the time domain features of the target action data; call the action recognition model to perform recognition and processing on the time domain features of the target action data to obtain a first recognition and processing result, and the first recognition and processing result is used to indicate the action category to which the action indicated by the target action data belongs.

[0022] In one embodiment, the action recognition model includes a time domain signal recognition network; a recognition processing module, specifically used to: call the time domain signal recognition network in the action recognition model to perform recognition processing on the time domain features of the target action data to obtain a first recognition processing result; wherein the time domain signal recognition network includes any one of the following: a single-dimensional convolutional neural network, a multi-dimensional convolutional neural network.

[0023] In one embodiment, the recognition processing module is specifically used to: if the type of action is the second type, perform time domain transformation processing on the target action data to obtain frequency domain information corresponding to the target action data; perform feature extraction processing on the frequency domain information to obtain frequency domain features of the target action data; call the action recognition model to perform recognition processing on the frequency domain features of the target action data to obtain a second recognition processing result, and the second recognition processing result is used to indicate the action category to which the action indicated by the target action data belongs.

[0024] In one embodiment, the action recognition model includes a frequency domain signal recognition network and a recognition processing module, which is specifically used to: call the frequency domain signal recognition network in the action recognition model to perform recognition and processing on the frequency domain features of the target action data to obtain a second recognition and processing result; wherein the frequency domain signal recognition network includes any one of the following: a long short-term memory network, a recurrent neural network, and a gated recurrent neural network.

[0025] In one embodiment, the target action data includes gesture data, and the recognition processing results obtained by recognizing and processing the target action data include any one or more of the following: drawing a circle clockwise in a vertical plane, drawing a circle counterclockwise in a vertical plane, drawing a circle clockwise in a horizontal plane, drawing a circle counterclockwise in a horizontal plane, sliding to the left, sliding to the right, sliding up, and sliding down.

[0026] In one embodiment, a micro-electric inertial navigation sensor is deployed in a virtual reality interaction device, and a result of the recognition process is used to indicate an action category to which an action belongs; a control module is used to: in response to the result of the recognition process, control the virtual reality interaction device to perform an interactive operation matching the action; or control the virtual reality interaction device to display an interactive interface matching the action.

[0027] In one embodiment, a micro-electric inertial navigation sensor is deployed in a cloud gaming interaction device, and the result of the recognition processing is used to indicate the action category to which the action belongs; a control module is used to: in response to the result of the recognition processing, control the cloud gaming interaction device to execute game instructions corresponding to the action category; or, control the cloud gaming interaction device to update and display a cloud gaming screen that matches the game instructions corresponding to the action category.

[0028] Accordingly, an embodiment of the present application provides a computer device, comprising: a processor, a memory, and a network interface; the processor is connected to the memory and the network interface, wherein the network interface is used to provide network communication functions, the memory is used to store program codes, and the processor is used to call program codes to execute the data processing method in the embodiment of the present application.

[0029] Accordingly, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. The computer program includes program instructions. When the program instructions are executed by a processor, the data processing method in the embodiment of the present application is executed.

[0030] Accordingly, an embodiment of the present application provides a computer program product, which includes a computer program or computer instructions. When the computer program or computer instructions are executed by a processor, the data processing method of the embodiment of the present application is implemented.

[0031] In an embodiment of the present application, the target motion data obtained by the micro-electric inertial navigation sensor from collecting the motion of the object can be obtained, and the target motion data to be processed can be analyzed and processed to obtain the type of the object's motion; the target motion data can be identified and processed in the time domain dimension or the frequency domain dimension based on the type of motion. Specifically, for the first type of motion with a relatively large motion amplitude, the target motion data can be identified and processed in the time domain dimension; for the second type of motion with a relatively small motion amplitude, the target motion data can be identified and processed in the frequency domain dimension. Since the target motion data is usually motion data in the time domain, for the motion with a relatively large motion amplitude, the target motion data can be directly identified and processed in the time domain dimension without being transformed into motion data in the frequency domain, which can greatly reduce the demand for computing power and improve the recognition speed, thereby reducing the delay of the recognition processing; and for the motion with a relatively small motion amplitude, the recognition and processing of the target motion data in the frequency domain dimension is supported. Since the motion data captured in the time domain dimension changes rapidly, the target motion data of the small-amplitude motion can be analyzed more easily and intuitively from the frequency domain dimension, thereby ensuring the accuracy of the recognition processing. In summary, by distinguishing the types of actions and performing recognition and processing in the time domain or frequency domain based on the size of the action amplitude, the computing power requirements for recognition and processing can be reduced in the process of camera-free action recognition, the recognition efficiency can be improved, the recognition accuracy can be ensured, and the delay and cost of recognition processing can be reduced while improving the recognition experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1a is a functional schematic diagram of a micro-electric inertial navigation sensor provided by an exemplary embodiment of the present application;

[0033] Figure 1b is an architecture diagram of a data processing system provided by an exemplary embodiment of the present application;

[0034] Figure 2 It is a flowchart of a data processing method provided by an exemplary embodiment of the present application;

[0035] Figure 3 is a flowchart of another data processing method provided by an exemplary embodiment of the present application;

[0036] Figure 4a is a schematic diagram of an action timing signal sequence provided by an exemplary embodiment of the present application;

[0037] Figure 4b It is a flowchart of an execution starting point detection of an action provided by an exemplary embodiment of the present application;

[0038] Figure 4cIt is a schematic diagram of the result of sequentially sampling an action timing signal sequence provided by an exemplary embodiment of the present application;

[0039] Figure 5 is a flowchart of another data processing method provided by an exemplary embodiment of the present application;

[0040] Figure 6a is a flowchart of processing target action data provided by an exemplary embodiment of the present application;

[0041] Figure 6b It is a schematic diagram of motion trajectories of various types of motion gestures provided by an exemplary embodiment of the present application;

[0042] Figure 6c is a schematic diagram of an identification processing flow provided by an exemplary embodiment of the present application;

[0043] Figure 6d It is a schematic diagram of a calculation principle of a GRU operator in a gated recurrent network provided by an exemplary embodiment of the present application;

[0044] Figure 6e It is a schematic diagram of a dual-model algorithm framework provided by an exemplary embodiment of the present application;

[0045] Figure 6f It is a schematic diagram comparing the recognition time and accuracy of a single gesture provided by an exemplary embodiment of the present application;

[0046] Figure 7 is a structural schematic diagram of a data processing device provided in an embodiment of the present application;

[0047] Figure 8 It is a structural diagram of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0048] First, a brief introduction to the related technologies involved in the embodiments of the present application is given:

[0049] 1. Artificial Intelligence (AI)

[0050] Artificial intelligence is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making.

[0051] Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. The basic technologies of artificial intelligence generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics and other technologies. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0052] 2. Computer Vision (CV)

[0053] Computer vision is a science that studies how to make machines "see". To put it more specifically, it refers to machine vision such as using cameras and computers to replace human eyes to identify and measure targets, and further perform graphic processing to make computer processing into images that are more suitable for human eye observation or transmission to instruments for detection. As a scientific discipline, computer vision studies related theories and technologies, and attempts to establish an artificial intelligence system that can obtain information from images or multi-dimensional data. Computer vision technology usually includes image processing, image recognition, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, video content / behavior recognition, three-dimensional object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous positioning and mapping, and other technologies, as well as common biometric recognition technologies such as face recognition and fingerprint recognition.

[0054] 3. Machine Learning (ML)

[0055] Machine learning is a multi-disciplinary subject that involves probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and other disciplines. It specializes in studying how computers simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications are spread across all areas of artificial intelligence. Machine learning and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and self-learning.

[0056] 4. Cloud Gaming

[0057] Cloud gaming is an online gaming technology based on cloud computing technology. In a cloud gaming scenario, the game is not run on the game client, but on the cloud server. The game client can transmit game instructions to the cloud server to control the cloud server to render the cloud game screen and update the display in the game client.

[0058] 5. MEMS-IMU (Micro-Electro-Mechanical System-Inertial Measurement Unit, micro-electromechanical system inertial measurement unit, referred to as micro-electric inertial sensor)

[0059] Micro-electric inertial navigation sensors are mainly used to detect and measure acceleration and rotational motion. Their principle is to use the law of inertia. The most basic micro-electric inertial navigation sensors include a 3-axis gyroscope (also called an angular velocity meter) and a 3-axis accelerometer, such as Figure 1a As shown in (1). They can respond to physical movements, such as linear displacement or angular rotation, and convert this response into electrical signals, which are amplified and processed by electronic circuits to obtain timing signals. Among them, the accelerometer is a sensor that senses axial acceleration and converts it into a usable output signal, and the gyroscope is a sensor that can sense the angular velocity of a moving body relative to the inertial space. The accelerometer can be used to detect the acceleration signal when an action is performed (such as the acceleration signal of a gesture action), which can be recorded as an A signal, and the gyroscope can detect the angular velocity signal when an action is performed, which can be recorded as a G signal. Specifically, the accelerometer collects acceleration signals based on the coordinate system formed by the x, y, and z axes, and the gyroscope collects angular velocity signals based on the coordinate system formed by the R (x), R (y), and R (z) axes. The coordinate systems are as follows: Figure 1a As shown in (2), Axz is the angle between the Rz axis and a ray in the Rxz plane (the plane formed by the R(x) and R(z) axes), and Ayz is the angle between the Rz axis and a ray in the Ryz plane (the plane formed by the R(y) and R(z) axes).

[0060] The micro-electric inertial navigation sensor can continuously collect the motion data generated by the object's actions to obtain a sequence of motion timing signals, which includes multiple motion timing signals arranged in chronological order. An motion timing signal includes an angular velocity signal and an acceleration signal, that is, an A+G timing signal collected by the accelerometer and gyroscope integrated in the micro-electric inertial navigation sensor. By processing and analyzing the collected motion timing signals, the motion category to which the object's motion belongs can be identified, such as Figure 1a The motion trajectories of some actions shown in FIG. 1 may correspond to actions of different categories, such as a left sliding gesture action.

[0061] In order to ensure the recognition accuracy and reduce the computing power requirement of recognition processing in the process of motion recognition based on no camera; The embodiment of the present application provides a data processing scheme, which first determines whether the type of action is a large action (a first type of action with a relatively large motion amplitude) or a micro action (i.e., a second type of action with a relatively small motion amplitude) before the motion data collected by MEMS-IMU is recognized and processed; Then, according to the recognition and processing strategies of different types and dimensions, the action is recognized and processed, specifically: If the type of action is determined to be a first type of action with a relatively large motion amplitude, the target action data can be recognized and processed from the time domain dimension, because the motion data collected by MEMS-IMU is itself a signal under the time domain dimension, the motion data is directly recognized and processed in the time domain dimension, which can avoid additional transformation and calculation, can greatly reduce the demand for computing power, and improve the recognition speed while ensuring the recognition accuracy. If the type of action is determined to be a second type of action with a relatively small motion amplitude, the target action data can be recognized and processed from the frequency domain dimension, that is, the motion data of the time domain dimension collected by MEMS-IMU is transformed into the frequency domain for recognition and processing, so that the accuracy of recognition can be guaranteed. It can be seen that the embodiments of the present application can reduce the computing power requirements for recognition processing and improve recognition efficiency by distinguishing the types of actions and performing recognition processing in the time domain or frequency domain based on the size of the action amplitude, thereby ensuring recognition accuracy while reducing the delay and cost of recognition processing and improving the recognition experience.

[0062] Secondly, the data processing solution provided in the embodiment of the present application can be applied to the following application scenarios:

[0063] 1. AR / VR scenarios

[0064] In an AR (Augmented Reality) scene or a VR (Virtual Reality) interactive scene, an object can wear an AR / VR intelligent detection device to detect and identify the object's movements, and then present the content controlled by the action category in the corresponding virtual interactive interface based on the identified action category. For example, when a left-swiping gesture is detected, the virtual furniture displayed in the virtual interactive interface can be controlled to move and be placed in the corresponding position, so that the effects of different virtual furniture placements can be intuitively seen, which facilitates object selection.

[0065] The data processing method provided in the embodiment of the present application can be applied to virtual reality interaction solutions. In the virtual reality interaction solution, the data processing method provided in the present application can be used to identify and process the target action data, and quickly and accurately obtain the recognition and processing results of the object's action, thereby reducing the delay of the virtual interaction process, reducing the cost of the entire set of virtual reality interaction equipment, improving the product experience, and facilitating the promotion of the product.

[0066] 2. Cloud Gaming Scenario

[0067] The object can use a cloud game interactive device (an interactive device for controlling the operation of cloud games) to detect and identify the action performed by the object through the cloud game interactive device, and control the operation of the cloud game through game instructions matching the action. For example, when the object makes an OK gesture, the game instruction can be a confirmation instruction, and the content to be confirmed in the cloud game can be automatically confirmed through the confirmation instruction. For another example, if the object makes a gesture of drawing a cross trajectory, the cancellation operation can be performed through the cancellation instruction, so as not to confirm the content to be confirmed in the cloud game.

[0068] By applying the data processing solution provided in the present application to the cloud gaming scenario, cloud games can be controlled touchlessly by directly recognizing the player's actions without a keyboard or mouse. Since the recognition processing process is relatively efficient, the action recognition results can be quickly obtained to perform game operations, thereby reducing the delay of game operations and improving the player's gaming experience.

[0069] 3. Virtual Social Scenarios

[0070] The virtual social scene can be a live broadcast scene or a virtual conversation scene. In the live broadcast scene, the actions of the host object can be identified, so as to control the virtual image corresponding to the host object to perform the corresponding actions. For example, if the host object makes a waving motion, the AI ​​virtual host (i.e., the virtual image) displayed in the live broadcast interface can also make a waving motion, or if the host object makes a clockwise circle gesture, the AI ​​virtual host can do a backflip. In the virtual conversation scene, the actions of the characters can also be identified through this solution, so as to control the virtual image to perform the same actions in the social conversation. For example, if the character makes a greeting gesture, the corresponding virtual image in the social conversation can also perform a greeting gesture.

[0071] By applying the data processing solution provided in this application to action recognition in virtual social scenarios, it is possible to quickly and accurately identify the actions of characters, flexibly and in real time control virtual images, improve the efficiency, real-time and fun of virtual social interaction, and enhance the virtual social experience.

[0072] 4. Intelligent Driving Scenarios

[0073] In the intelligent driving scenario, interaction through actions can avoid the driver from performing touch operations to a certain extent. The driver does not need to know where the buttons in the car are. As long as the driver makes a specific action, a series of instructions can be transmitted after the action is recognized to control the vehicle navigation, infotainment system, etc. For example, if the driver's waving action is recognized, the song can be automatically switched, and if the driver's upward swipe gesture is recognized, the volume can be automatically adjusted.

[0074] By applying the data processing solution provided in this application to intelligent driving scenarios, the actions taken by the driver can be accurately and quickly identified, and the identification results of the driver's movements can be used as input to appropriately output response results, such as unlocking the device, starting an application, etc., thereby assisting the driver to control the relevant systems in the vehicle without touching the device, achieving very convenient operation and reducing equipment wear.

[0075] It should be noted that the application scenarios of the data processing solution provided in this application are not limited to the four application scenarios introduced above, but can also be applied to other application scenarios, such as medical, education and other related scenarios.

[0076] The data processing scheme of the embodiment of the present application can also be combined with artificial intelligence. For example, an action recognition model is used in the recognition process, which can be various artificial neural networks, such as convolutional neural networks, gated recurrent neural networks, and long short-term memory networks. The target action data to be processed is recognized and processed to determine the action type to which the action belongs. These networks can be lightweight recognition networks with fewer model parameters, which can not only ensure accurate recognition, but also obtain recognition processing results more efficiently.

[0077] The data processing system of the embodiment of the present application is introduced below. Figure 1b 1 is an architecture diagram of a data processing system provided by an exemplary embodiment of the present application. The data processing system includes a data acquisition device 100 and a data processing device 101, and a communication connection can be established between the data acquisition device 100 and the data processing device 101 in a wired or wireless manner.

[0078] The data acquisition device 100 is used to collect motion data generated by an object performing an action, such as collecting data on gesture actions, and the motion data may be an original motion timing signal sequence. The data acquisition device 100 may send the motion data collected by the micro-electric inertial navigation sensor to the data processing device 101, and the data processing device 101 may obtain the target motion data to be processed from the original motion data, and perform recognition processing on the target motion data to obtain the recognition processing result. The micro-electric inertial navigation sensor MEMS-IMU may be deployed in the data acquisition device 100, so the data acquisition device 100 may be an intelligent wearable device equipped with a MEMS-IMU sensor, such as Figure 1b The smart glove device shown in the figure is equipped with a micro-electric inertial navigation sensor, and the motion data collected by the data acquisition device 100 can be an A+G timing signal. It should be noted that the above-mentioned data acquisition device 100 can be a terminal device, and the terminal device includes but is not limited to: smart phones, tablet computers, smart wearable devices, smart voice interaction devices, smart home appliances, personal computers, vehicle-mounted terminals, etc., and this application does not limit this. This application does not limit the number of data acquisition devices 100.

[0079] The data processing device 101 is used to receive the action data collected and sent by the data acquisition device 100 through the micro-electric inertial navigation sensor on the object, and the action data can be the original action timing signal sequence. For example, the A+G timing signal collected for the gesture action. The data processing device 101 can obtain the target action data to be processed based on the action timing signal sequence, and can determine the type of action by parsing the target action data, and then when the type of action is the first type, the target action data is identified and processed in the time domain dimension, and when the type of action is the second type, the target action data is identified and processed in the frequency domain dimension. The target action data can be identified to obtain an identification processing result, and the identification processing result can be used to indicate the action category to which the action belongs. For example, by identifying the gesture data, the category of the gesture action can be known, such as a gesture action of drawing a circle clockwise with a relatively large action amplitude. In one implementation, the identification processing result obtained by the data processing device 101 for identifying the target action data can be visually displayed to the object, or a corresponding operation can be performed based on the identification processing result. For example, the target action data is gesture data, and the data processing device 101 can obtain a gesture category by performing gesture recognition on the gesture data. The data processing device can respond to the gesture category and control the interactive device to complete interactive operations such as movement control, selection confirmation, etc. that match the gesture category.

[0080] It should be noted that the data processing device 101 is a computer device, such as a server. The server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Deliver Network), and big data and artificial intelligence platforms, but is not limited to this. This application does not limit the number of servers.

[0081] It is understandable that the above-mentioned data processing device 101 and data acquisition device 100 are only an exemplary data processing system, and the data processing system may also include other interactive devices, such as a data display device for displaying the execution result of the interactive operation matching the recognition processing result to the object. The data display device, the data processing device and the data acquisition device may constitute a complete set of virtual reality interactive devices. Here, the data processing system is not limited.

[0082] It can be seen that the data processing system provided by the embodiment of the present application can obtain motion data by collecting the motion of the object through the micro-electric inertial navigation sensor in the data acquisition device. The data processing device can obtain the target motion data to be processed from the collected motion data, and determine the type of motion by analyzing the target motion data. When the type of the object's motion is the first type with a relatively large motion amplitude, the target motion data can be directly identified and processed from the time domain dimension without converting the time domain signal to the frequency domain for processing, thereby greatly reducing the demand for computing power and improving the recognition speed while ensuring the accuracy of recognition; when the type of the object's motion is the second type with a relatively small motion amplitude, the target motion data can be identified and processed from the frequency domain dimension, thereby accurately identifying the motion category to which the motion belongs and ensuring the accuracy of recognition.

[0083] See also Figure 2 , Figure 2 is a flow chart of a data processing method provided in an embodiment of the present application. The data processing method can be performed by a computer device (eg Figure 1b The data processing method may include the following steps S201 to S204:

[0084] S201, obtaining target action data to be processed.

[0085] The target action data is obtained by collecting the action of the object using a micro-electric inertial navigation sensor. Among them, the micro-electric inertial navigation sensor (i.e., MEMS-IMU) is a data acquisition module without a camera. The object can wear an interactive device equipped with a micro-electric inertial navigation sensor. When the object performs the corresponding action, the action data corresponding to the action performed by the object can be collected by the micro-electric inertial navigation sensor. The action data is a time domain signal. The specific action data can be a sequence of action timing signals, such as an A signal (i.e., an acceleration signal) collected by an accelerometer and a G signal (i.e., an angular velocity signal) collected by a gyroscope. Among them, the action of the object can be a gesture action, a limb action (such as an arm movement), or a human body action (the amplitude of the action is greater than the amplitude of the limb action), etc. The target action data to be processed is used to describe the execution of the object's action, and can specifically be part or all of the action data collected. The target action data corresponds to the action of the object, for example, the target action data can be gesture data corresponding to a gesture action.

[0086] S202, parsing the target action data to determine the type of action.

[0087] The type includes a first type and a second type, and the motion amplitude of the first type is greater than the motion amplitude of the second type. The first type of motion is a motion with a relatively large motion amplitude (for example, the motion amplitude is greater than a preset amplitude threshold), which can be referred to as a large motion, and the second type of motion is a motion with a relatively small motion amplitude (for example, the motion amplitude is less than a preset amplitude threshold), which can be referred to as a micro motion. The motion amplitude refers to the range of movement from the beginning of the motion (i.e., the execution starting point of the motion) to the end of the motion (i.e., the execution end point of the motion). The larger the motion amplitude, the larger the motion range, and conversely, the smaller the motion amplitude, the smaller the motion range. In some optional embodiments, there is a relationship between the motion amplitude and the duration of the motion, for example: the larger the motion amplitude, the longer the motion may last, and the smaller the motion amplitude, the shorter the motion may last. Different recognition processing methods can be used for different types of motions, so as to achieve accurate and efficient recognition of the motion by adapting to the type of the corresponding motion.

[0088] The analysis of the target action data may be to calculate relevant data based on the target action data, such as calculating energy, calculating energy mean, etc. Through these relevant data, the type of the object's action can be determined, and then identification processing is performed in the time domain or frequency domain according to the object's action type. The determined type of the object's action may be the first type or the second type mentioned above. When the object's action is of the first type, identification processing may be performed according to the content described in S203 below. When the object's action is of the second type, identification processing may be performed according to the content described in S204 below.

[0089] S203: If the action type is the first type, identification processing is performed on the target action data in the time domain dimension.

[0090] If the type of action is the first type, it means that the action of the object is a relatively large action, and the action with a relatively large action amplitude may last for a relatively long time, and the target action data information obtained is richer. Therefore, the target action data can be recognized and processed in the time domain dimension, without the need to transform the target action data in the time domain into the frequency domain, but directly use the time domain information of the target action data. In this way, the demand for computing power can be greatly reduced while maintaining the same recognition accuracy, and the recognition speed can also be improved. In one implementation, the recognition and processing of the target action data in the time domain dimension can be implemented using an action recognition model (such as a convolutional neural network).

[0091] S204: If the action type is the second type, identification processing is performed on the target action data in the frequency domain dimension.

[0092] If the type of action is the second type, it means that the action of the object is an action with a relatively small amplitude, and the duration of an action with a relatively small amplitude may be relatively short. The information captured for this type of action may be relatively small, resulting in the action data of the micro-action being unable to support the accuracy required by the business in the time domain dimension. Therefore, the target action data can be identified and processed in the frequency domain dimension. Specifically, the target action data can be converted into spectral domain information, such as transforming the target action data into the frequency domain through Fourier transform. Since the frequency domain dimension can support the processing of extremely complex signal changes, the target action data in the time domain is converted into a frequency domain component with a larger amount of information than the original time domain waveform for identification processing to obtain the result of the identification processing. The result of the identification processing can be used to indicate the action category to which the action belongs. In this way, accurate identification of the category to which the action belongs can also be achieved in the frequency domain dimension. In one implementation method, the identification processing of the target action data in the frequency domain dimension can also be implemented using an action recognition model (such as a recurrent neural network).

[0093] The data processing method provided in the embodiment of the present application can obtain the target motion data obtained by the micro-electric inertial navigation sensor from the motion collection of the object, and parse and process the target motion data to be processed to obtain the type of the object's motion; based on the type of motion, the target motion data is identified and processed in the time domain dimension or the frequency domain dimension. Specifically, for the motion with a relatively large motion amplitude, the target motion data can be identified and processed in the time domain dimension, and for the motion with a relatively small motion amplitude, the target motion data can be identified and processed in the frequency domain dimension. Since the target motion data is usually motion data in the time domain, for the motion with a relatively large motion amplitude, the target motion data can be directly identified and processed in the time domain dimension without being transformed into motion data in the frequency domain, which can greatly reduce the demand for computing power and improve the recognition speed, thereby reducing the delay of the recognition processing; and for the motion with a relatively small motion amplitude, the recognition processing of the target motion data in the frequency domain dimension is supported, because the motion data captured in the time domain dimension changes rapidly, and the target motion data of the small amplitude motion can be analyzed more easily and intuitively from the frequency domain dimension, thereby ensuring the accuracy of the recognition processing. In summary, by distinguishing the types of actions and performing recognition processing in the time domain or frequency domain based on the size of the action amplitude, the computing power requirements for recognition processing can be reduced, recognition efficiency can be improved, recognition accuracy can be ensured, and at the same time, the latency and cost of recognition processing can be reduced, thereby improving the recognition experience.

[0094] See also Figure 3 , Figure 3 is a flowchart of another data processing method provided by an exemplary embodiment of the present application. The data processing method can be performed by a computer device (eg Figure 1b The data processing device 101 shown in the figure is executed. This embodiment mainly introduces the content executed before obtaining the target action data to be processed, specifically the detection scheme of the execution starting point of the object's action. The data processing method may include the following steps S301 to S306:

[0095] S301, obtaining a sequence of action timing signals obtained by collecting the action of an object by a micro-electric inertial navigation sensor.

[0096] The action timing signal sequence includes at least one action timing signal arranged in chronological order, and these action timing signals are time domain signals obtained by collecting the action of the object in the time domain. Each action timing signal is obtained by collecting the action of the object at different time points by the micro-electric inertial navigation sensor. In one implementation, an action timing signal may include an angular velocity signal and an acceleration signal. For example, Figure 4a The action timing signal sequence shown is a continuous time domain signal, with the horizontal axis being time and the vertical axis being the acceleration signal.

[0097] S302: pre-process the action timing signal sequence to determine the execution starting point of the action.

[0098] Before the target action data is identified and processed, the action timing signal sequence can be preprocessed in real time. The execution starting point of the object's action can be determined through preprocessing, so that effective target action data can be obtained based on the execution starting point of the action, which is convenient for accurately identifying the action category to which the action belongs. The embodiment of the present application relates to the logic of continuously sampling the action timing signal sequence according to multiple time windows. The multiple time windows can be expressed as: the first time window T1, the second time window T2, the third time window T3... and so on in the sampling order. The first time window T1 represents the time window used first, and the second time window T2 represents the next time window of the first time window T1, and so on. The embodiment of the present application uses the first time window T1 to determine the execution starting point of the action.

[0099] In some optional embodiments, the specific implementation of S302 may include the following steps:

[0100] (1) The starting sampling point of the first time window T1 is located at the first action timing signal in the action timing signal sequence.

[0101] The first time window T1 is a sliding time window with a time length of T. Since the first action timing signal in the action timing signal sequence is the starting point for data collection of the object's action, but not necessarily the starting point for the object's action to be executed. The starting sampling point of the first time window T1 is positioned to the first action timing signal, and sequential detection can be performed from the first action timing signal in the action timing signal sequence through the time window to determine the execution starting point of the action.

[0102] (2) Detect whether the energy of the action timing signal corresponding to the starting sampling point of the first time window T1 meets the preset conditions.

[0103] In one implementation, an action timing signal includes an acceleration signal and an angular velocity signal. The energy of the action timing signal corresponding to the starting sampling point of the first time window T1 satisfies the preset condition, which means that the angular velocity energy of the action timing signal corresponding to the starting sampling point of the first time window T1 is greater than or equal to the preset angular velocity energy threshold, and the acceleration energy of the action timing signal corresponding to the starting sampling point of the first time window T1 is greater than or equal to the preset acceleration energy threshold.

[0104] The angular velocity energy of the action timing signal corresponding to the starting sampling point of the first time window can be determined based on the angular velocity signal of the starting sampling point, and the acceleration energy of the action timing signal corresponding to the starting sampling point of the first time window can be determined based on the acceleration signal of the starting sampling point. For example, if the acceleration signal in the action timing signal corresponding to the starting sampling point is (a xi , a yi , a zi ), where a xi , a yi , a zi Represent the acceleration signal values ​​of the starting sampling point of the action on the x-axis, y-axis, and z-axis respectively. Then the acceleration energy can be calculated as: The angular velocity signal can be expressed as (w xi , w yi , w zi ), where w xi , w yi , w zi Represent the angular velocity signal values ​​of the starting sampling point of the action on the x-axis, y-axis, and z-axis respectively. Then the angular velocity energy can be calculated as:

[0105] The preset angular velocity energy threshold refers to the angular velocity energy threshold Tw1 of the starting sampling point, and the preset acceleration energy threshold refers to the acceleration energy threshold Ta1 of the starting sampling point. These two energy thresholds can be used to determine whether the energy meets the sampling conditions. The energy of the action timing signal corresponding to the starting sampling point of the first time window T1 includes angular velocity energy and acceleration energy. If the angular velocity energy is greater than or equal to the preset angular velocity energy threshold Tw1, and the acceleration energy is greater than or equal to the preset acceleration energy threshold Ta1, it means that the intensity of the action meets the requirements and may be the starting point of the action. Therefore, it can be determined that the energy of the action timing signal corresponding to the starting sampling point of the first time window T1 meets the preset conditions, and the content in the following step (3) is executed.

[0106] (3) If the energy of the action timing signal corresponding to the starting sampling point of the first time window T1 meets the preset condition, the starting sampling point of the first time window T1 is used as a reference, and sequential sampling is performed in the action timing signal sequence according to the first time window T1 to obtain multiple sampling points within the first time window T1.

[0107] If the energy of the action timing signal corresponding to the starting sampling point of the first time window T1 meets the preset conditions, it means that the action timing signal corresponding to the starting sampling point of the first time window T1 may be the starting point of the action. The starting sampling point of the first time window T1 can be used as a reference to sequentially sample the action timing signals in the first time window T1 in the action timing signal sequence, thereby obtaining multiple sampling points in the first time window T1, and each sampling point can correspond to an action timing signal in the action timing signal sequence.

[0108] On the contrary, if the energy of the action timing signal corresponding to the starting sampling point of the first time window T1 does not meet the preset condition, the first time window T1 is sequentially shifted in the action timing signal sequence, and the starting sampling point of the first time window T1 is relocated. In other words, when the energy of the action timing signal corresponding to the starting sampling point of the first time window T1 is less than the energy threshold, it can be determined that the first action timing signal located by the starting sampling point of the current first time window T1 is not the execution starting point of the action, and the starting sampling point of the first time window T1 is sequentially moved to the second action timing signal in the action timing signal sequence, and the second action timing signal is used as the starting sampling point of the relocated first time window.

[0109] In one implementation, an action timing signal includes an acceleration signal and an angular velocity signal. The energy of the action timing signal corresponding to the starting sampling point of the first time window T1 does not meet the preset condition, which means that the angular velocity energy of the action timing signal corresponding to the starting sampling point of the first time window T1 is less than the preset angular velocity energy threshold, or the acceleration energy of the action timing signal corresponding to the starting sampling point of the first time window T1 is less than the preset acceleration energy threshold. For the introduction of angular velocity energy, acceleration energy, preset angular velocity energy threshold and preset acceleration energy threshold, please refer to the above content, which will not be repeated here.

[0110] (4) Check whether the energy mean of the action timing signal corresponding to all sampling points in the first time window T1 meets the preset conditions.

[0111] In one implementation, an action timing signal includes an acceleration signal and an angular velocity signal. The energy mean of the action timing signal corresponding to all sampling points in the first time window T1 includes an acceleration average energy and an angular velocity average energy. The first time window T1 corresponds to an angular velocity average energy and an acceleration average energy. The angular velocity average energy is determined based on the angular velocity energy of the action timing signal corresponding to all sampling points in the first time window T1. The acceleration average energy is determined based on the acceleration energy of the action timing signal corresponding to all sampling points in the first time window T1. Exemplarily, the first time window T1 includes 10 sampling points, and the angular velocity energy calculated from the action timing signal corresponding to each sampling point is recorded as E wi (i=0,1……,9), the acceleration energy calculated from the action timing signal corresponding to each sampling point is recorded as E ai (i=0,1……,9). The calculation method of angular velocity average energy can be: The calculation method of the average energy of acceleration can be:

[0112] The energy mean of the action timing signal corresponding to all sampling points in the first time window T1 satisfies the preset condition, which means that the angular velocity average energy of the action timing signal corresponding to all sampling points in the first time window T1 is greater than or equal to the preset angular velocity energy threshold, and the acceleration average energy of the action timing signal corresponding to all sampling points in the first time window T1 is greater than or equal to the preset acceleration energy threshold.

[0113] The preset angular velocity energy threshold here is the preset angular velocity threshold Tw1 for judging angular velocity energy, and the preset acceleration energy threshold is the preset acceleration threshold Ta1 for judging acceleration energy. The average angular velocity energy corresponding to the first time window T1 is greater than or equal to the preset angular velocity energy threshold, and the acceleration energy threshold is greater than or equal to the preset acceleration energy threshold, indicating that the intensity of the movement in the first time window meets the preset requirements, and it can be determined that the energy mean corresponding to the first time window meets the preset conditions. Further, the content shown in (5) below can be executed.

[0114] (5) If the energy mean of the action timing signal corresponding to all sampling points in the first time window T1 meets the preset condition, the starting sampling point of the first time window T1 is determined as the execution starting point of the action.

[0115] Since the energy of the starting sampling point can reflect the intensity of the object's action at that moment, and the energy mean corresponding to the first time window can reflect the intensity of the object's action in a certain time length. On the premise that the energy mean of the action timing signal corresponding to all sampling points in the first time window T1 satisfies the preset conditions, if the energy mean of the action timing signal corresponding to all sampling points in the first time window T1 also satisfies the preset conditions, it means that the intensity of the action at a certain sampling point and within a period of time meets the requirements, and at this time, it can be regarded that the object's action starts to be executed at the starting sampling point and continues to move in the first time window.

[0116] On the contrary, in one implementation, if the energy mean of the action timing signal corresponding to all sampling points in the first time window T1 does not meet the preset condition, the first time window T1 is sequentially shifted in the action timing signal sequence, and the starting sampling point of the first time window T1 is relocated. At this time, the energy of the action timing signal corresponding to the starting sampling point of the first time window T1 meets the preset condition, while the energy mean of the action timing signal corresponding to all sampling points in the first time window T1 does not meet the preset condition, indicating that the intensity of the action within a period of time has not reached the characteristics of the start of the action, thereby determining that the first action timing signal located at the starting sampling point of the first time window T1 is not the execution starting point of the action, so the first time window T1 can be sequentially shifted to the second action timing signal in the action timing signal sequence, and the second action timing signal is used as the starting sampling point of the first time window T1.

[0117] It should be noted that, according to the above process (1)-(5) to detect the execution starting point of the action, before the execution starting point of the action is determined, the first time window T1 can be continuously shifted in sequence in the action timing signal sequence, and the starting sampling point of the first time window T1 is continuously relocated until a sampling point that meets the conditions is found in the action timing signal and is determined as the execution starting point of the action. In order to more intuitively illustrate the above process (1)-(5), please refer to Figure 4b The figure shows a flow chart of an exemplary scheme for detecting the execution starting point of an action.

[0118] S303: Obtain target action data to be processed.

[0119] In one embodiment, based on S301-S302, the specific implementation of S303 may include: taking the execution starting point of the action as a reference, sequentially sampling and processing the action timing signal sequence according to the time window, and obtaining the action data corresponding to the multiple time windows. Among them, the starting sampling point of the first time window T1 is the execution starting point of the action; the target action data to be processed refers to the action data corresponding to the i-th time window Ti, and the action data corresponding to the i-th time window Ti includes the action timing signals corresponding to each sampling point in the i-th time window Ti; the i-th time window Ti is any one of the multiple time windows, and i is a positive integer.

[0120] That is to say, after determining the execution starting point of the action, the action timing signal sequence can be divided according to the time window to obtain the action data corresponding to multiple time windows. Each time window is arranged in time order, and the time length of each time window can be the same. The action data corresponding to each time window includes the action timing signal corresponding to each sampling point in the corresponding time window. For example, see Figure 4c The result schematic diagram of sequential sampling of the action timing signal sequence shown includes target action data corresponding to multiple time windows.

[0121] The target action data to be processed may be the action timing signals of each sampling point in the first time window, and the starting sampling point in the first time window is located at the execution starting point of the action. Figure 2 After the process in the illustrated embodiment is processed, the target action data to be processed may be the action data corresponding to the second time window, and the action timing signals of each sampling point in the second time window are processed. The time length of the second time window is the same as that of the first time window, and the time corresponding to the sampling points in the second time window is greater than the time corresponding to the sampling points in the first time window. The target action data to be processed may also be the action data corresponding to the third time window, the action data corresponding to the fourth time window, and so on. On this basis, the action timing signal sequence is divided by time windows to obtain action data corresponding to multiple time windows. The action data of any time window can be used as the target data to be processed, thereby obtaining a waveform formed by framing. By framing, the signal that changes with time can keep the characteristics relatively stable for a period of time, which is conducive to the recognition and processing of the target action data.

[0122] S304, parsing the target action data to determine the type of action.

[0123] S305: If the action type is the first type, identification processing is performed on the target action data in the time domain dimension.

[0124] S306: If the action type is the second type, identification processing is performed on the target action data in the frequency domain dimension.

[0125] The above S304-S306 can be found in Figure 2 The introduction of S202 - S204 or the introduction of the corresponding steps in the following embodiments will not be repeated here.

[0126] The data processing method provided in the embodiment of the present application can pre-process the action timing signal in the action timing signal sequence collected by the micro-electric inertial navigation sensor through the time window before obtaining the target action data to be processed, and comprehensively judge the energy of the starting sampling point of the time window and the energy mean of the time window, so as to accurately determine the execution starting point of the action, thereby facilitating the acquisition of effective target action data, using resources for effective identification and processing, and avoiding wasting resources for identification and processing of useless action data. By parsing the target data to be processed and distinguishing the types of actions into the first type and the second type, accurate and efficient identification of actions can be achieved from the time domain dimension or the frequency domain dimension.

[0127] See also Figure 5 , Figure 5 is a flowchart of another data processing method provided by an exemplary embodiment of the present application. The data processing method can be performed by a computer device (eg Figure 1b The data processing method may include the following steps S501 to S506:

[0128] S501, obtaining target action data to be processed.

[0129] S502, calculating the energy of the target action data.

[0130] In one embodiment, the energy of the target action data includes the average energy of the target action data and the maximum variance of the target action data. The average energy of the target action data is used to reflect the intensity of the movement of the action indicated by the target action data; the maximum variance of the target action data is used to reflect the energy change of the action indicated by the target action data. The action indicated by the target action data may refer to the action performed by the object in a time window. The larger the average energy, the more intense the movement of the action indicated by the target action data, the larger the maximum variance, the greater the energy fluctuation of the action indicated by the target action data, further indicating that the amplitude of the action is larger; the action belongs to a large action; the smaller the average energy, the softer the movement of the action indicated by the target action data, the smaller the maximum variance, the smoother the energy change of the action indicated by the target action data, further indicating that the amplitude of the action is smaller, and the action belongs to a micro action. It can be seen that the amplitude of the object's action can be judged by the average energy and the maximum variance, and then the type of action can be determined.

[0131] In one implementation, the target action data includes an action timing signal corresponding to each sampling point in a time window, and the action timing signal corresponding to a sampling point includes an angular velocity signal and an acceleration signal. The average energy of the target action data may specifically include an angular velocity average energy and an acceleration average energy. The angular velocity average energy here is calculated based on the angular velocity energy corresponding to each sampling point in a time window, and the acceleration average energy is calculated based on the acceleration energy corresponding to each sampling point in a time window. Correspondingly, the maximum variance of the target action data may include: a maximum angular velocity variance calculated based on the angular velocity average energy and the angular velocity energy corresponding to each sampling point, and a maximum acceleration variance calculated based on the acceleration average energy and the acceleration energy corresponding to each sampling point.

[0132] It should be noted that the target motion data can be a motion timing signal of each sampling point in a time window, and each motion timing signal includes an angular velocity signal and an acceleration signal. The angular velocity signal and the angular velocity signal can be used to accurately identify the motion using multi-dimensional data.

[0133] S503: If the energy of the target action data is greater than or equal to a preset threshold, determine that the type of the action is the first type.

[0134] In one implementation, the preset threshold includes a preset energy threshold and a preset variance threshold, and the energy of the target action data is greater than or equal to the preset threshold means that: the average energy of the target action data is greater than or equal to the preset energy threshold, and the maximum variance of the target action data is greater than or equal to the preset variance threshold. If the above conditions are met, it means that the intensity of the movement of the action indicated by the target action data and the energy change fluctuations over a period of time meet the conditions required for the action amplitude of the large action, and thus, it can be determined that the type of action indicated by the target action is the first type. The first type of action is an action with a relatively large action amplitude. Optionally, if the average energy of the target action data includes the average energy of the angular velocity and the average energy of the acceleration, and the maximum variance includes the maximum angular velocity variance and the maximum acceleration variance, then the preset threshold here can set different energy thresholds for the average energy of the angular velocity and the average energy of the acceleration, or use the same preset energy threshold, and the preset variance threshold is the same. If the same preset energy threshold and preset variance threshold are used, the average energy greater than or equal to the preset energy threshold includes that the angular velocity average energy and the acceleration average energy are both greater than or equal to the preset energy threshold, and the maximum variance greater than or equal to the preset variance threshold includes the maximum angular velocity variance and the maximum acceleration variance.

[0135] S504: If the energy of the target action data is less than a preset threshold, determine that the type of the action is the second type.

[0136] In one implementation, the energy of the target action data is less than a preset threshold value means that the average energy of the target action data is less than a preset energy threshold value; or means that the maximum variance of the target action data is less than a preset variance threshold value; or means that the average energy of the target action data is less than the preset energy threshold value, and the maximum variance of the target action data is less than the preset variance threshold value. In other words, if the average energy and / or the maximum variance of the target action data do not meet the conditions required for the action amplitude of the large action, it indicates that the action indicated by the target action data is a micro-action of a smaller amplitude, and the action can be determined to be the second type. The second type of action is an action with a relatively small action amplitude (for example, the action amplitude is less than the preset amplitude threshold value).

[0137] S505: If the action type is the first type, identification processing is performed on the target action data in the time domain dimension.

[0138] In one embodiment, the specific implementation method of S505 includes: if the type of action is the first type, extracting the time domain features of the target action data; calling the action recognition model to perform recognition processing on the time domain features of the target action data to obtain a first recognition processing result, and the first recognition processing result is used to indicate the action category to which the action indicated by the target action data belongs.

[0139] If the type of action is the first type, it means that the action amplitude is relatively large (for example, the action amplitude is greater than the preset amplitude threshold). The time domain signal collected by the micro-electric inertial navigation sensor can fully represent the action information. Therefore, the time domain information of the target action data can be directly used to simply filter the target action data to obtain the time domain feature. The time domain feature can be a feature vector of the time domain. In this way, there is no need for transformation processing (such as Fourier transform), and the action recognition model can be directly called to perform recognition processing on the extracted time domain features, thereby obtaining a first recognition processing result for indicating the action category to which the action indicated by the target action data belongs.

[0140] Furthermore, the action recognition model includes a time domain signal recognition network, and calling the action recognition model to perform recognition processing on the time domain features of the target action data to obtain a first recognition processing result includes: calling the time domain signal recognition network in the action recognition model to perform recognition processing on the time domain features of the target action data to obtain a first recognition processing result. Among them, the time domain signal recognition network is a recognition network for processing time domain features, and the time domain signal recognition network includes any of the following: a single-dimensional convolutional neural network and a multi-dimensional convolutional neural network. A single-dimensional convolutional neural network refers to a 1-dimensional (1 layer) convolutional neural network (CNN), and a multi-dimensional convolutional neural network refers to a convolutional neural network of at least two dimensions. It can be seen that the time domain signal recognition network is a lightweight recognition network with relatively few network parameters and small computing power requirements, which can realize quick processing of time domain features.

[0141] When the action type is the first type and the time domain signal recognition network adopts a single-dimensional convolutional neural network, the processing flow of the target action data can be as follows: Figure 6a As shown. When the action type is the first type (such as an obvious large gesture Obvious Gesture), the time-domain signal (Time-domain Signal), that is, the time-domain target action data, can be directly used, and then the time-domain signal is subjected to feature extraction processing (Feature extraction) to obtain the time-domain features of the target action data, and then input into a single-dimensional convolutional neural network for recognition processing, output the target features, and then pass through a fully connected layer (Dense Layer) to obtain a first recognition result. Here, the single-dimensional convolutional neural network includes multiple convolutional neural networks CNN in a series relationship.

[0142] If the target action data is gesture data, the action recognition model can specifically be a model for gesture recognition, the action is specifically a gesture action, the first type of action is a large action gesture, and the categories to which the gesture action belongs include but are not limited to: drawing a circle clockwise (CWV) in a vertical plane, drawing a circle counterclockwise (CCWV) in a vertical plane, drawing a circle clockwise (CWH) in a horizontal plane, drawing a circle counterclockwise (CCWH) in a horizontal plane, sliding left (L), sliding right (R), sliding up (U) and sliding down (D). For the action trajectory diagram of the above categories of action gestures, please refer to Figure 6b. The gesture of drawing a circle clockwise (or counterclockwise) on a vertical plane refers to a gesture that moves clockwise (or counterclockwise) along a circular trajectory in the plane formed by the y-axis and the z-axis (the plane is perpendicular to the x-axis). The gesture of drawing a circle clockwise (or counterclockwise) on a horizontal plane refers to a gesture that moves clockwise (or counterclockwise) along a circular trajectory in the plane formed by the x-axis and the y-axis (the plane is perpendicular to the z-axis). The gesture of sliding left (right) refers to sliding left (right) along the z-axis, and the trajectory from the starting point to the end point of the gesture is from the right (left) to the left (right), and then back to the right (left). Similarly, the gesture of sliding upward (downward) refers to sliding upward (downward) along the z-axis, and the trajectory from the starting point to the end point of the gesture is from bottom (up) to top (down), and then from top (down) to bottom (up). The first recognition result obtained by performing recognition processing on the gesture data includes one or more categories to which the above gesture actions belong. For example, the first recognition result is a gesture of drawing a circle clockwise on a vertical plane.

[0143] The time domain signal recognition network in the action recognition model is a lightweight recognition network. In the actual processing implementation, the time domain signal recognition network shows excellent performance in terms of recognition time consumption, number of network parameters and accuracy. Taking gesture recognition as an example, the lightweight solution combining the time domain information provided by this solution with CNN and the commonly used solutions combining the spectrum domain information and LSTM (Long Short-Term Memory, referred to as long short-term memory network) and the spectrum domain information and RNN (Recurrent Neural Network, referred to as recurrent neural network) in the industry are compared in terms of recognition effect, recognition efficiency and computing power requirements. The specific comparison results are shown in Table 1 below.

[0144] Table 1 Comparison of large motion gesture recognition

[0145]

[0146]

[0147] As can be seen from Table 1, the [time domain information + CNN] scheme has obvious advantages in recognition efficiency (reflected by recognition time consumption: the smaller the recognition time consumption, the higher the recognition efficiency) and computing power consumption (reflected by the number of network parameters: the more network parameters, the greater the computing power consumption). In terms of recognition accuracy, the [time domain information + 2D CNN] scheme and the [spectral domain information + LSTM] scheme are the best, the [time domain information + 1D CNN] scheme is second best, and the [spectral domain information + bidirectional RNN] scheme is slightly worse. Considering the recognition effect, recognition efficiency, and computing power requirements, the [time domain information + 1D CNN] recognition scheme can be selected as the algorithm for simple large motion gesture recognition. In actual applications, it is found that for large motion gestures (91%), the time domain information of the A+G signal can be directly used without Fourier transform. Using a single-dimensional CNN (a lightweight time domain signal recognition network) for calculation and recognition can reduce the time consumption by 53% and the model parameters by 87% while maintaining the same algorithm accuracy. While improving the recognition speed, it greatly reduces the demand for computing power and improves the recognition efficiency.

[0148] S506: If the action type is the second type, identification processing is performed on the target action data in the frequency domain dimension.

[0149] In one embodiment, the specific implementation method of S506 includes: if the type of action is the second type, performing time domain transformation processing on the target action data to obtain frequency domain information corresponding to the target action data; performing feature extraction processing on the frequency domain information to obtain frequency domain features of the target action data; calling the action recognition model to perform recognition processing on the frequency domain features of the target action data to obtain a second recognition processing result, and the second recognition processing result is used to indicate the action category to which the action indicated by the target action data belongs.

[0150] If the type of action is the second type, it means that the action is a micro-action with a small amplitude. Since the time domain information of the micro-action may not last long, the recognition of the action may not be very accurate if only the time domain information is used. Therefore, the target action data can be transformed in the time domain, and the target action data can be transformed from the time domain to the frequency domain to obtain the frequency domain information corresponding to the target action data, such as performing a fast Fourier transform (Fast Fourier Transform, FFT) on the A+G signal to convert it into a frequency domain signal (Spectrum-domain Signal), that is, frequency domain information. Then, feature extraction processing (Feature extraction) is performed on the frequency domain information to obtain frequency domain features, such as obtaining the frequency domain features of the target action data by filtering the frequency domain information, and then inputting the frequency domain features into the action recognition model for recognition processing to obtain a second recognition processing result for indicating the action category to which the action indicated by the target action data belongs.

[0151] Furthermore, the action recognition model includes a frequency domain signal recognition network, and calling the action recognition model to perform recognition processing on the frequency domain features of the target action data to obtain a second recognition processing result includes: calling the frequency domain signal recognition network in the action recognition model to perform recognition processing on the frequency domain features of the target action data to obtain a second recognition processing result. The frequency domain signal recognition network is a recognition network for processing frequency domain features, and can be a multidimensional neural network. In one implementation, the frequency domain signal recognition network includes any one of the following: a long short-term memory network, a recurrent neural network, and a gated recurrent neural network. Among them, the gated recurrent neural network is a neural network composed of gated recurrent units (GRU).

[0152] When the frequency domain signal recognition network adopts a gated recurrent neural network, the recognition processing flow using the gated neural recurrent network can be as follows: Figure 6c As shown, for the second type of action (such as a Subtle Gesture), the frequency domain signal is obtained by Fourier transforming the target action data, and the frequency domain feature is obtained by extracting the feature of the frequency domain signal, and is simultaneously input into the first layer (1 st layer) and the second layer (2 nd The target feature (specifically, a feature vector) is output after the recognition processing by the gated recurrent neural network. The output target feature is then predicted by the fully connected layer to obtain the predicted probability that the target feature belongs to each action category. The action category with the highest predicted probability can be used as the second recognition result. If applied to gesture recognition, the second recognition result can be as follows: Figure 6b One or more of the various categories of gestures shown.

[0153] Among them, the gated recurrent neural network is a multi-dimensional recurrent neural network, and the recurrent neural network of each dimension includes a series of gated recurrent units GRU, and an output of the gated recurrent unit of the first dimension is used as an input of the gated recurrent unit of the second dimension, such as Figure 6a The first layer (1 st The first gated recurrent unit of layer) will output h(t-1) as the second layer (2 nd The calculation principle of each GRU operator in the gated recurrent network is as follows: Figure 6d shown.

[0154] The input of GRU includes a current input x tand the hidden state h passed down from the previous node t-1 , the hidden state contains the relevant information of the previous node. Combined with x t and h t-1 , GRU can get the output y of the current hidden node t And the hidden state h passed to the next node t Specifically, first, we can use x t and h t-1 Two gate states r and z can be obtained, where r indicates the gate that controls reset (reset gate) and z indicates the gate that controls update (updategate). For the method of obtaining r and z, see the following expression:

[0155]

[0156] Among them, σ is the sigmoid function, through which the data can be transformed into a value in the range of 0-1, thus acting as a gating signal.

[0157] After that, the reset data h can be obtained by resetting the gate t-1′ =h t-1 ⊙r, ⊙ represents the multiplication of the corresponding elements in the operation matrix. t-1′ and x t After concatenation, a tanh activation function is used to scale the data to the range of -1 to 1, that is:

[0158]

[0159] Since h′ mainly contains the current input x t Data, adding h′ to the current hidden state is equivalent to memorizing the current state. After that, you can use the update gate z to forget and select memory at the same time. The update expression is as follows:

[0160] h t =(1-z)⊙h t-1 +z⊙h′

[0161] Among them, (1-z)⊙h t-1 Indicates the selective forgetting of the original hidden state, that is, forgetting h t-1 Some unimportant information in the dimension, z⊙h′ represents the selective memory of h′ containing the current node information.

[0162] Compared with recurrent neural networks (RNNs) and long short-term memory (LSTMs), gated recurrent neural networks contain fewer network parameters, which can save computing power to a certain extent in complex micro-motion recognition algorithms, thereby saving computing time costs while taking into account the computing power of the hardware. In terms of training efficiency, GRU is also better than RNN and LSTM.

[0163] In the actual processing implementation, taking gesture recognition as an example, for micro-motion gestures (9%), the aforementioned lightweight solution of calling the time domain signal recognition network in the action recognition model for recognition processing may not be able to support the accuracy required by the business. Therefore, the A+G signal can be Fourier transformed and converted into spectral domain information. In the use of the frequency domain signal recognition network in the action recognition model, the industry-wide LSTM network or RNN network can be used, or a set of GRU networks can be retrained. The GRU network is preferred in this application. Using different frequency domain signal recognition networks to recognize micro-motion gestures has different recognition effects, recognition efficiency, and computing power requirements. Therefore, in actual applications, the recognition effects, recognition efficiency, and computing power requirements of micro-motion gesture recognition in different networks (LSTM, RNN, GRU) are compared, as shown in Table 2 below.

[0164] Table 2 Comparison of micro-motion gesture recognition

[0165]

[0166]

[0167] As can be seen from Table 2, the overall effect of GRU after parameter adjustment is significantly better than that of bidirectional RNN. Compared with bidirectional LSTM, GRU has obvious advantages in recognition efficiency and computing power consumption. Both recognition accuracy can be guaranteed to be >99%, meeting business needs. Among them, the recognition accuracy of GRU is slightly lower than that of bidirectional LSTM, but the gap is very small. Considering the gains in efficiency and computing power, GRU can be preferred as the algorithm model for complex micro-motion gesture recognition. For micro-motion gestures (9%), the lightweight solution introduced above may not be able to support the accuracy required by the business. In this step, the A+G signal is Fourier transformed into spectral domain information. A retrained set of GRU models can be preferably used in the model. The model parameters of the GRU model are far less than the former, and computing power is also saved to a certain extent in the complex micro-motion recognition algorithm.

[0168] As can be seen from the above, the action recognition model includes a time domain signal recognition network and a frequency domain signal recognition network. The action recognition model is a dual model (i.e., a dual network model). Different recognition networks in the action recognition model are specifically called for the recognition processing of different types of actions. Combined with the above introduction of the preprocessing of the action timing signal sequence collected by the micro-electric inertial navigation sensor to obtain the execution starting point of the action, the following can be provided: Figure 6e The algorithm framework of the dual model is shown in Figure 1, which is a framework of the dual model hierarchical algorithm. Figure 6e As shown in FIG. 1 , after pre-detection, the type of the action can be determined to be the first type or the second type by the average energy (avg_energy) and the maximum variance (Max_variance) of the target action data. If the type of the action is the first type, then according to the Figure 6a The processing flow shown in the figure is used for identification processing. If the action type is the second type, the above Figure 6c The identification process is performed according to the processing flow shown.

[0169] Tests in actual applications show that compared with the commonly used algorithm in the industry (i.e., frequency domain signal + bidirectional LSTM), the dual-model hierarchical algorithm used in this application can ensure better recognition accuracy while greatly improving computing power and speed. Figure 6f The comparison of the recognition time and accuracy of a single gesture is shown in the figure. In terms of recognition time, the frequency domain signal + bidirectional LSTM algorithm takes 33 milliseconds (unit: ms) to recognize, while the dual-model hierarchical algorithm used in this application can directly reduce the recognition time to 18 milliseconds, greatly reducing the recognition time; in terms of accuracy, the accuracy of the frequency domain signal + bidirectional LSTM algorithm is 99.5%, while the accuracy of the dual-model hierarchical algorithm used in this application is 99.3%, with a difference of only 0.2%. Further, in actual tests, the results of gesture recognition using this solution are shown in Table 3 below.

[0170] Table 3 Gesture recognition results

[0171]

[0172] It can be seen that the above Table 3 adopts the data processing solution of this application, and the recognition accuracy of each group of sample gestures obtained on the real test set is maintained at about 99.3%, which is very impressive. It can be seen that the hierarchical algorithm using dual models can greatly improve the computing efficiency and maintain a high accuracy. That is, the overall solution can meet the online needs of services (such as gesture recognition services) while reducing costs and improving experience.

[0173] It should be noted that the serial numbers of steps such as S501 and S502 in the embodiment of the present application do not limit the execution order of the steps, for example, S505 can be executed after S503, and S506 can be executed after S504. The same is true for other embodiments.

[0174] The embodiments of the present application can be specifically applied in various virtual reality interaction scenarios. Through the data processing solution provided by the present application, the recognition processing results can be quickly obtained, so as to quickly respond to the recognition processing results and perform related processing. The virtual reality interaction scenarios are briefly introduced below.

[0175] In one implementation, a micro-electric inertial navigation sensor is deployed in a virtual reality interaction device, and the result of the recognition processing is used to indicate the action category to which the action belongs. The virtual reality interaction device can be a VR (Virtual Reality) device or an AR (Augmented Reality) device, etc. The micro-electric inertial navigation sensor MEMS-IMU can be deployed in any of the above virtual reality interaction devices, and the action data (a time domain signal) is collected through the MEMS-IMU sensor. The recognition processing module in the virtual reality interaction device obtains the target action data from the action data collected by the MEMS-IMU sensor, and parses the target action data to obtain the type of action, and then selects to directly recognize and process the target action data in the time domain dimension based on the action type, or converts the time domain target action data into a frequency domain signal through Fourier transform in the frequency domain dimension, and then calculates through the action recognition model to obtain the result of the recognition processing. In one implementation, the target action gesture data includes gesture data, and the recognition processing results obtained by performing recognition processing on the target action data include any one or more of the following: drawing a circle clockwise on a vertical plane, drawing a circle counterclockwise on a vertical plane, drawing a circle clockwise on a horizontal plane, drawing a circle counterclockwise on a horizontal plane, sliding left, sliding right, sliding up, and sliding down. For details, please refer to the aforementioned Figure 6b The content introduced will not be repeated here.

[0176] Furthermore, the result of the recognition processing can also be applied to the following scenarios: in response to the result of the recognition processing, the virtual reality interaction device is controlled to perform an interactive operation matching the action; or, the virtual reality interaction device is controlled to display an interactive interface matching the action.

[0177] The computer device controls the virtual reality interactive device to perform interactive operations such as movement control operations, selection operations, confirmation operations, page sliding operations, etc. in response to the result of the recognition process. Actions and interactive operations have a corresponding relationship, and one action can correspond to one interactive operation. Exemplarily, the result of the recognition process is used to indicate that the gesture action is a downward sliding gesture, and the computer device can control the virtual reality interactive device to perform a certain operation in response to the result of the recognition process. Alternatively, the virtual reality interactive device is controlled to display a new interactive interface by turning pages.

[0178] In another implementation, the micro-electric inertial navigation sensor is deployed in the cloud game interaction device, and the result of the recognition process is used to indicate the action category to which the action belongs. The cloud game interaction device is an interactive device used to control the operation of cloud games. Similar to the processing principle in the virtual interactive device, after obtaining the result of the recognition process, it is also possible to: respond to the result of the recognition process, control the cloud game interaction device to execute the game instructions corresponding to the action category; or control the cloud game interaction device to update and display the cloud game screen that matches the game instructions corresponding to the action category.

[0179] An action performed by the object can correspond to a game instruction. For example, a left-slide gesture can control the cloud game interactive device to execute a left-moving game instruction, and then control the virtual image in the cloud game to move left based on the game instruction. Alternatively, the cloud game interactive device can be controlled to update the cloud game screen. For example, a circle gesture on a vertical plane can control the cloud game device to update the original cloud game screen to another cloud game screen with different content.

[0180] Since the result of the recognition process can be determined quickly with low latency, the delay of the entire control process can be reduced in the above scenarios, the real-time nature of the interaction is high, and the interaction can be carried out efficiently to improve the interactive experience. In this way, the entire set of virtual reality interactive equipment or cloud game interactive equipment can obtain accurate recognition results without requiring very powerful computing power during the recognition process. While reducing costs and improving the experience, it can also maintain accuracy and meet the needs of business launch.

[0181] The data processing scheme provided in the embodiment of the present application includes an efficient motion recognition algorithm. To solve the problems of high transaction cost and poor experience of MEMS-IMU gesture recognition, a dual-model hierarchical motion recognition algorithm can be implemented based on the CNN model and the GRU model, which can be specifically used for gesture recognition. For the first type of action, the time domain information of the target action data can be used directly without additional transformation and calculation. A lightweight time domain signal recognition network is used for calculation and recognition. Under the premise of maintaining the same algorithm accuracy, the time consumption can be greatly shortened and the model parameters can be reduced. While improving the recognition speed, the demand for computing power is greatly reduced; for the second type of action, in order to further ensure the accuracy required by the business, the target action data can be transformed into the frequency domain, and a frequency domain signal recognition network with fewer model parameters is used for calculation and recognition. For complex second-type actions, computing power can be saved to a certain extent. In summary, the overall solution can reduce the recognition cost, improve the recognition experience through efficient recognition, and ensure accuracy.

[0182] See also Figure 7 , Figure 7 is a structural diagram of a data processing device provided in an embodiment of the present application. The above-mentioned data processing device may be a computer program (including program code) running in a computer device, for example, the data processing device is an application software; the data processing device may be used to execute the corresponding steps in the method provided in an embodiment of the present application. Figure 7 As shown, the data processing device 700 may include at least one of the following: an acquisition module 701 , a parsing module 702 , an identification processing module 703 , a pre-processing module 704 and a control module 705 .

[0183] An acquisition module 701 is used to acquire target motion data to be processed, where the target motion data is acquired by using a micro-electric inertial navigation sensor to collect the motion of the object;

[0184] A parsing module 702, configured to parse the target motion data to determine the type of motion, the type comprising a first type and a second type, the motion amplitude of the first type being greater than the motion amplitude of the second type;

[0185] The recognition processing module 703 is used to perform recognition processing on the target action data in the time domain dimension if the action type is the first type;

[0186] The recognition processing module 703 is further used to perform recognition processing on the target action data in the frequency domain dimension if the action type is the second type.

[0187] In one embodiment, the analysis module 702 is specifically used to: calculate the energy of the target action data; if the energy of the target action data is greater than or equal to a preset threshold, determine the type of action to be the first type; if the energy of the target action data is less than the preset threshold, determine the type of action to be the second type.

[0188] In one embodiment, the energy of the target action data includes the average energy of the target action data and the maximum variance of the target action data; the average energy of the target action data is used to reflect the intensity of the movement of the action indicated by the target action data; the maximum variance of the target action data is used to reflect the energy change of the action indicated by the target action data; the preset threshold includes a preset energy threshold and a preset variance threshold; the energy of the target action data is greater than or equal to the preset threshold, which means: the average energy of the target action data is greater than or equal to the preset energy threshold, and the maximum variance of the target action data is greater than or equal to the preset variance threshold; the energy of the target action data is less than the preset threshold, which means: the average energy of the target action data is less than the preset energy threshold; or the maximum variance of the target action data is less than the preset variance threshold; or the average energy of the target action data is less than the preset energy threshold, and the maximum variance of the target action data is less than the preset variance threshold.

[0189] In one embodiment, the acquisition module 701 is further used to: acquire the action timing signal sequence obtained by the micro-electric inertial navigation sensor collecting the action of the object; the preprocessing module 704 is further used to: preprocess the action timing signal sequence to determine the execution starting point of the action.

[0190] In one embodiment, the preprocessing module 704 is specifically used to: locate the starting sampling point of the first time window T1 to the first action timing signal in the action timing signal sequence; detect whether the energy of the action timing signal corresponding to the starting sampling point of the first time window T1 meets the preset condition; if the energy of the action timing signal corresponding to the starting sampling point of the first time window T1 meets the preset condition, then take the starting sampling point of the first time window T1 as a reference, and perform sequential sampling in the action timing signal sequence according to the first time window T1 to obtain multiple sampling points in the first time window T1; detect whether the energy mean of the action timing signal corresponding to all sampling points in the first time window T1 meets the preset condition; if the energy mean of the action timing signal corresponding to all sampling points in the first time window T1 meets the preset condition, then determine the starting sampling point of the first time window T1 as the execution starting point of the action.

[0191] In one embodiment, the preprocessing module 704 is specifically used to: if the energy of the action timing signal corresponding to the starting sampling point of the first time window T1 does not meet the preset condition, or if the energy mean of the action timing signal corresponding to all sampling points in the first time window T1 does not meet the preset condition, then the first time window T1 is sequentially shifted in the action timing signal sequence to reposition the starting sampling point of the first time window T1.

[0192] In one embodiment, an action timing signal includes an acceleration signal and an angular velocity signal, and the energy of the action timing signal corresponding to the starting sampling point of the first time window T1 satisfies the preset condition, which means: the angular velocity energy of the action timing signal corresponding to the starting sampling point of the first time window T1 is greater than or equal to the preset angular velocity energy threshold, and the acceleration energy of the action timing signal corresponding to the starting sampling point of the first time window T1 is greater than or equal to the preset acceleration energy threshold; the energy mean of the action timing signal corresponding to all sampling points in the first time window T1 satisfies the preset condition, which means: the angular velocity average energy of the action timing signal corresponding to all sampling points in the first time window T1 is greater than or equal to the preset angular velocity energy threshold, and the acceleration average energy of the action timing signal corresponding to all sampling points in the first time window T1 is greater than or equal to the preset acceleration energy threshold.

[0193] In one embodiment, the acquisition module 701 is specifically used to: take the execution starting point of the action as a reference, sequentially sample the action timing signal sequence according to the time window, and obtain action data corresponding to multiple time windows respectively; wherein, the starting sampling point of the first time window T1 is the execution starting point of the action; the target action data to be processed refers to the action data corresponding to the i-th time window Ti, and the action data corresponding to the i-th time window Ti includes the action timing signals corresponding to each sampling point in the i-th time window Ti; the i-th time window Ti is any one of the multiple time windows, and i is a positive integer.

[0194] In one embodiment, the recognition processing module 703 is specifically used to: if the type of action is the first type, extract the time domain features of the target action data; call the action recognition model to perform recognition processing on the time domain features of the target action data to obtain a first recognition processing result, and the first recognition processing result is used to indicate the action category to which the action indicated by the target action data belongs.

[0195] In one embodiment, the action recognition model includes a time domain signal recognition network; the recognition processing module 703 is specifically used to: call the time domain signal recognition network in the action recognition model to perform recognition processing on the time domain features of the target action data to obtain a first recognition processing result; wherein the time domain signal recognition network includes any one of the following: a single-dimensional convolutional neural network, a multi-dimensional convolutional neural network.

[0196] In one embodiment, the recognition processing module 703 is specifically used to: if the type of action is the second type, perform time domain transformation processing on the target action data to obtain frequency domain information corresponding to the target action data; perform feature extraction processing on the frequency domain information to obtain frequency domain features of the target action data; call the action recognition model to perform recognition processing on the frequency domain features of the target action data to obtain a second recognition processing result, and the second recognition processing result is used to indicate the action category to which the action indicated by the target action data belongs.

[0197] In one embodiment, the action recognition model includes a frequency domain signal recognition network, and the recognition processing module 703 is specifically used to: call the frequency domain signal recognition network in the action recognition model to recognize and process the frequency domain characteristics of the target action data to obtain a second recognition processing result; wherein the frequency domain signal recognition network includes any one of the following: long short-term memory network, recurrent neural network, gated recurrent neural network.

[0198] In one embodiment, the target action data includes gesture data, and the recognition processing results obtained by recognizing and processing the target action data include any one or more of the following: drawing a circle clockwise in a vertical plane, drawing a circle counterclockwise in a vertical plane, drawing a circle clockwise in a horizontal plane, drawing a circle counterclockwise in a horizontal plane, sliding to the left, sliding to the right, sliding up, and sliding down.

[0199] In one embodiment, the micro-electric inertial navigation sensor is deployed in a virtual reality interaction device, and the result of the recognition processing is used to indicate the action category to which the action belongs; the control module 705 is used to: in response to the result of the recognition processing, control the virtual reality interaction device to perform an interactive operation matching the action; or control the virtual reality interaction device to display an interactive interface matching the action.

[0200] In one embodiment, a micro-electric inertial navigation sensor is deployed in a cloud gaming interaction device, and the result of the recognition processing is used to indicate the action category to which the action belongs; the control module 705 is used to: in response to the result of the recognition processing, control the cloud gaming interaction device to execute the game instructions corresponding to the action category; or, control the cloud gaming interaction device to update and display the cloud gaming screen that matches the game instructions corresponding to the action category.

[0201] It is understandable that the functions of the functional modules of the data processing device described in the embodiment of the present application can be specifically implemented according to the method in the above method embodiment, and the specific implementation process can refer to the relevant description of the above method embodiment, which will not be repeated here. In addition, the description of the beneficial effects of using the same method will not be repeated.

[0202] See also Figure 8 , Figure 8800 is a schematic diagram of the structure of a computer device provided in an embodiment of the present application. The computer device 800 may include an independent device (e.g., one or more of a server, a node, a terminal, etc.), or may include components inside an independent device (e.g., a chip, a software module, or a hardware module, etc.). The computer device 800 may include at least one processor 801 and a communication interface 802. Further optionally, the computer device 800 may also include at least one memory 803 and a bus 804. The processor 801, the communication interface 802, and the memory 803 are connected via a bus 804.

[0203] Among them, the processor 801 is a module that performs arithmetic operations and / or logical operations, and can specifically be a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MPU), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a complex programmable logic device (CPLD), a coprocessor (assisting the central processing unit to complete corresponding processing and applications), a microcontroller unit (MCU) and other processing modules, or a combination of multiple of them.

[0204] The communication interface 802 may be used to provide information input or output for at least one processor. And / or, the communication interface 802 may be used to receive data sent externally and / or send data externally, and may be a wired link interface including an Ethernet cable, etc., or a wireless link (Wi-Fi, Bluetooth, general wireless transmission, vehicle-mounted short-range communication technology, and other short-range wireless communication technologies, etc.) interface. The communication interface 802 may be used as a network interface.

[0205] The memory 803 is used to provide a storage space in which data such as an operating system and a computer program can be stored. The memory 803 can be a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), or a portable read-only memory (CD-ROM), etc., or a combination of multiple thereof.

[0206] At least one processor 801 in the computer device 800 is used to call a computer program stored in at least one memory 803 to execute the data processing method described in the embodiment shown in this application.

[0207] In one possible implementation, the processor 801 in the computer device 800 is used to call a computer program stored in at least one memory 803 to perform the following operations: obtain target motion data to be processed, where the target motion data is obtained by collecting the motion of the object using a micro-electric inertial navigation sensor; parse the target motion data to determine the type of motion, where the types include a first type and a second type, and the motion amplitude of the first type is greater than the motion amplitude of the second type; if the type of motion is the first type, identify and process the target motion data in the time domain dimension; if the type of motion is the second type, identify and process the target motion data in the frequency domain dimension.

[0208] The processor 801 is further configured to, if the type of the action is the second type, perform recognition processing on the target action data in the frequency domain dimension.

[0209] In one embodiment, the processor 801 is specifically used to: calculate the energy of the target action data; if the energy of the target action data is greater than or equal to a preset threshold, determine the type of action to be the first type; if the energy of the target action data is less than the preset threshold, determine the type of action to be the second type.

[0210] In one embodiment, the energy of the target action data includes the average energy of the target action data and the maximum variance of the target action data; the average energy of the target action data is used to reflect the intensity of the movement of the action indicated by the target action data; the maximum variance of the target action data is used to reflect the energy change of the action indicated by the target action data; the preset threshold includes a preset energy threshold and a preset variance threshold; the energy of the target action data is greater than or equal to the preset threshold, which means: the average energy of the target action data is greater than or equal to the preset energy threshold, and the maximum variance of the target action data is greater than or equal to the preset variance threshold; the energy of the target action data is less than the preset threshold, which means: the average energy of the target action data is less than the preset energy threshold; or the maximum variance of the target action data is less than the preset variance threshold; or the average energy of the target action data is less than the preset energy threshold, and the maximum variance of the target action data is less than the preset variance threshold.

[0211] In one embodiment, the processor 801 is further used to: obtain a motion timing signal sequence obtained by the micro-electric inertial navigation sensor collecting the motion of the object; the processor 801 is further used to: pre-process the motion timing signal sequence to determine the execution starting point of the action.

[0212] In one embodiment, the processor 801 is specifically used to: locate the starting sampling point of the first time window T1 to the first action timing signal in the action timing signal sequence; detect whether the energy of the action timing signal corresponding to the starting sampling point of the first time window T1 meets the preset condition; if the energy of the action timing signal corresponding to the starting sampling point of the first time window T1 meets the preset condition, then take the starting sampling point of the first time window T1 as a reference, and perform sequential sampling in the action timing signal sequence according to the first time window T1 to obtain multiple sampling points in the first time window T1; detect whether the energy mean of the action timing signal corresponding to all sampling points in the first time window T1 meets the preset condition; if the energy mean of the action timing signal corresponding to all sampling points in the first time window T1 meets the preset condition, then determine the starting sampling point of the first time window T1 as the execution starting point of the action.

[0213] In one embodiment, the processor 801 is specifically used to: if the energy of the action timing signal corresponding to the starting sampling point of the first time window T1 does not meet the preset condition, or if the energy mean of the action timing signal corresponding to all sampling points in the first time window T1 does not meet the preset condition, then the first time window T1 is sequentially shifted in the action timing signal sequence to reposition the starting sampling point of the first time window T1.

[0214] In one embodiment, an action timing signal includes an acceleration signal and an angular velocity signal, and the energy of the action timing signal corresponding to the starting sampling point of the first time window T1 satisfies the preset condition, which means: the angular velocity energy of the action timing signal corresponding to the starting sampling point of the first time window T1 is greater than or equal to the preset angular velocity energy threshold, and the acceleration energy of the action timing signal corresponding to the starting sampling point of the first time window T1 is greater than or equal to the preset acceleration energy threshold; the energy mean of the action timing signal corresponding to all sampling points in the first time window T1 satisfies the preset condition, which means: the angular velocity average energy of the action timing signal corresponding to all sampling points in the first time window T1 is greater than or equal to the preset angular velocity energy threshold, and the acceleration average energy of the action timing signal corresponding to all sampling points in the first time window T1 is greater than or equal to the preset acceleration energy threshold.

[0215] In one embodiment, the processor 801 is specifically used to: take the execution starting point of the action as a reference, sequentially sample and process the action timing signal sequence according to the time window, and obtain action data corresponding to multiple time windows respectively; wherein, the starting sampling point of the first time window T1 is the execution starting point of the action; the target action data to be processed refers to the action data corresponding to the i-th time window Ti, and the action data corresponding to the i-th time window Ti includes the action timing signals corresponding to each sampling point in the i-th time window Ti; the i-th time window Ti is any one of the multiple time windows, and i is a positive integer.

[0216] In one embodiment, the processor 801 is specifically used to: if the type of action is the first type, extract the time domain features of the target action data; call the action recognition model to perform recognition processing on the time domain features of the target action data to obtain a first recognition processing result, and the first recognition processing result is used to indicate the action category to which the action indicated by the target action data belongs.

[0217] In one embodiment, the action recognition model includes a time domain signal recognition network; the processor 801 is specifically used to: call the time domain signal recognition network in the action recognition model to identify and process the time domain features of the target action data to obtain a first recognition processing result; wherein the time domain signal recognition network includes any one of the following: a single-dimensional convolutional neural network, a multi-dimensional convolutional neural network.

[0218] In one embodiment, the processor 801 is specifically used to: if the type of action is the second type, perform time domain transformation processing on the target action data to obtain frequency domain information corresponding to the target action data; perform feature extraction processing on the frequency domain information to obtain frequency domain features of the target action data; call the action recognition model to perform recognition processing on the frequency domain features of the target action data to obtain a second recognition processing result, and the second recognition processing result is used to indicate the action category to which the action indicated by the target action data belongs.

[0219] In one embodiment, the action recognition model includes a frequency domain signal recognition network, and the processor 801 is specifically used to: call the frequency domain signal recognition network in the action recognition model to identify and process the frequency domain features of the target action data to obtain a second recognition processing result; wherein the frequency domain signal recognition network includes any one of the following: a long short-term memory network, a recurrent neural network, and a gated recurrent neural network.

[0220] In one embodiment, the target action data includes gesture data, and the recognition processing results obtained by recognizing and processing the target action data include any one or more of the following: drawing a circle clockwise in a vertical plane, drawing a circle counterclockwise in a vertical plane, drawing a circle clockwise in a horizontal plane, drawing a circle counterclockwise in a horizontal plane, sliding to the left, sliding to the right, sliding up, and sliding down.

[0221] In one embodiment, a micro-electric inertial navigation sensor is deployed in a virtual reality interaction device, and a result of the recognition process is used to indicate an action category to which the action belongs; the processor 801 is used to: in response to the result of the recognition process, control the virtual reality interaction device to perform an interactive operation matching the action; or, control the virtual reality interaction device to display an interactive interface matching the action.

[0222] In one embodiment, a micro-electric inertial navigation sensor is deployed in a cloud gaming interaction device, and the result of the recognition processing is used to indicate the action category to which the action belongs; the processor 801 is used to: in response to the result of the recognition processing, control the cloud gaming interaction device to execute the game instructions corresponding to the action category; or, control the cloud gaming interaction device to update and display the cloud gaming screen that matches the game instructions corresponding to the action category.

[0223] It should be understood that the computer device 800 described in the embodiment of the present application can execute the description of the data processing method in the above corresponding embodiment, and can also execute the above Figure 7 The description of the data processing device 700 in the corresponding embodiment will not be repeated here. In addition, the description of the beneficial effects of adopting the same method will not be repeated here either.

[0224] In addition, it should be noted that an exemplary embodiment of the present application also provides a storage medium, in which a computer program of the aforementioned data processing method is stored, and the computer program includes program instructions. When one or more processors load and execute the program instructions, the description of the data processing method in the embodiment can be implemented, which will not be repeated here, and the description of the beneficial effects of adopting the same method will not be repeated here. It is understandable that the program instructions can be deployed on one or multiple computer devices that can communicate with each other for execution.

[0225] The computer-readable storage medium may be the data processing device provided in any of the aforementioned embodiments or the internal storage unit of the computer device, such as the hard disk or memory of the computer device. The computer-readable storage medium may also be an external storage device of the computer device, such as a plug-in hard disk, a smart memory card (smart media card, SMC), a secure digital (secure digital, SD) card, a flash card (flash card), etc. equipped on the computer device. Further, the computer-readable storage medium may also include both the internal storage unit of the computer device and an external storage device. The computer-readable storage medium is used to store the computer program and other programs and data required by the computer device. The computer-readable storage medium may also be used to temporarily store data that has been output or is to be output.

[0226] In one aspect of the present application, a computer program product or a computer program is provided, the computer program product or the computer program includes computer instructions, the computer instructions are stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the method provided in one aspect of the embodiments of the present application.

[0227] In one aspect of the present application, another computer program product is provided, which includes a computer program or computer instructions, and when the computer program or computer instructions are executed by a processor, the steps of the data processing method provided in the embodiment of the present application are implemented.

[0228] The steps in the method of the embodiment of the present application can be adjusted in order, combined and deleted according to actual needs.

[0229] The modules in the device of the embodiment of the present application can be merged, divided and deleted according to actual needs.

[0230] The above disclosure is only part of the embodiments of the present application, which certainly cannot be used to limit the scope of rights of the present application. Ordinary technicians in this field can understand that all or part of the processes of implementing the above embodiments and making equivalent changes according to the claims of this application are still within the scope of the invention.

Claims

1. A data processing method, characterized in that: include: Acquire target motion data to be processed, wherein the target motion data is obtained by collecting the motion of the object using a micro-electric inertial navigation sensor; Parsing the target action data to determine the type of the action, comprising: calculating the energy of the target action data; if the energy of the target action data is greater than or equal to a preset threshold, determining the type of the action to be a first type; if the energy of the target action data is less than the preset threshold, determining the type of the action to be a second type; wherein the action amplitude of the first type is greater than the action amplitude of the second type; If the type of the action is the first type, extracting the time domain features of the target action data; calling the action recognition model to perform recognition processing on the time domain features of the target action data to obtain a first recognition processing result, wherein the first recognition processing result is used to indicate the action category to which the action indicated by the target action data belongs; If the type of the action is the second type, the target action data is subjected to time domain transformation processing to obtain frequency domain information corresponding to the target action data; the frequency domain information is subjected to feature extraction processing to obtain frequency domain features of the target action data; the action recognition model is called to perform recognition processing on the frequency domain features of the target action data to obtain a second recognition processing result, and the second recognition processing result is used to indicate the action category to which the action indicated by the target action data belongs.

2. The method according to claim 1, characterized in that The energy of the target motion data includes the average energy of the target motion data and the maximum variance of the target motion data; the average energy of the target motion data is used to reflect the intensity of the motion indicated by the target motion data; The maximum variance of the target action data is used to reflect the energy change of the action indicated by the target action data; the preset threshold includes a preset energy threshold and a preset variance threshold; The energy of the target action data is greater than or equal to a preset threshold value means that: the average energy of the target action data is greater than or equal to the preset energy threshold value, and the maximum variance of the target action data is greater than or equal to the preset variance threshold value; The energy of the target action data is less than a preset threshold value, which means: the average energy of the target action data is less than the preset energy threshold value; or the maximum variance of the target action data is less than the preset variance threshold value; or the average energy of the target action data is less than the preset energy threshold value, and the maximum variance of the target action data is less than the preset variance threshold value.

3. The method according to claim 1, characterized in that Before obtaining the target action data to be processed, the method further includes: Acquire a sequence of action timing signals obtained by the micro-electric inertial navigation sensor collecting the action of the object; The action timing signal sequence is preprocessed to determine the execution starting point of the action.

4. The method according to claim 3, characterized in that The preprocessing of the action timing signal sequence to determine the execution starting point of the action includes: Positioning the starting sampling point of the first time window T1 to the first action timing signal in the action timing signal sequence; Detecting whether the energy of the action timing signal corresponding to the starting sampling point of the first time window T1 meets a preset condition; If the energy of the action timing signal corresponding to the starting sampling point of the first time window T1 meets the preset condition, taking the starting sampling point of the first time window T1 as a reference, sequentially sampling is performed in the action timing signal sequence according to the first time window T1 to obtain a plurality of sampling points within the first time window T1; Detecting whether the energy mean of the action timing signal corresponding to all sampling points in the first time window T1 meets the preset condition; If the energy mean values ​​of the action timing signals corresponding to all sampling points in the first time window T1 meet the preset condition, the starting sampling point of the first time window T1 is determined as the execution starting point of the action.

5. The method according to claim 4, characterized in that The preprocessing of the action timing signal sequence to determine the execution starting point of the action further includes: If the energy of the action timing signal corresponding to the starting sampling point of the first time window T1 does not meet the preset condition, or if the energy mean of the action timing signal corresponding to all sampling points in the first time window T1 does not meet the preset condition, the first time window T1 is sequentially shifted in the action timing signal sequence to reposition the starting sampling point of the first time window T1.

6. The method according to claim 4, characterized in that An action timing signal includes an acceleration signal and an angular velocity signal, and the energy of the action timing signal corresponding to the starting sampling point of the first time window T1 satisfies the preset condition, which means that the angular velocity energy of the action timing signal corresponding to the starting sampling point of the first time window T1 is greater than or equal to the preset angular velocity energy threshold, and the acceleration energy of the action timing signal corresponding to the starting sampling point of the first time window T1 is greater than or equal to the preset acceleration energy threshold; The energy mean of the action timing signal corresponding to all sampling points in the first time window T1 satisfies the preset condition, which means that the angular velocity average energy of the action timing signal corresponding to all sampling points in the first time window T1 is greater than or equal to the preset angular velocity energy threshold, and the acceleration average energy of the action timing signal corresponding to all sampling points in the first time window T1 is greater than or equal to the preset acceleration energy threshold.

7. The method according to claim 3, characterized in that The step of obtaining the target action data to be processed includes: Taking the execution starting point of the action as a reference, sequentially sampling the action timing signal sequence according to the time window to obtain action data corresponding to multiple time windows respectively; Among them, the starting sampling point of the first time window T1 is the starting point for executing the action; the target action data to be processed refers to the action data corresponding to the i-th time window Ti, and the action data corresponding to the i-th time window Ti includes the action timing signal corresponding to each sampling point in the i-th time window Ti; the i-th time window Ti is any one of the multiple time windows, and i is a positive integer.

8. The method according to claim 1, characterized in that The action recognition model includes a time domain signal recognition network; the calling action recognition model performs recognition processing on the time domain features of the target action data to obtain a first recognition processing result, including: Calling the time domain signal recognition network in the action recognition model to perform recognition processing on the time domain features of the target action data to obtain a first recognition processing result; Among them, the time domain signal recognition network includes any one of the following: a single-dimensional convolutional neural network, a multi-dimensional convolutional neural network.

9. The method according to claim 1, characterized in that The action recognition model includes a frequency domain signal recognition network, and the calling action recognition model performs recognition processing on the frequency domain features of the target action data to obtain a second recognition processing result, including: Calling the frequency domain signal recognition network in the action recognition model to perform recognition processing on the frequency domain features of the target action data to obtain a second recognition processing result; Among them, the frequency domain signal recognition network includes any one of the following: long short-term memory network, recurrent neural network, gated recurrent neural network.

10. The method according to claim 1, characterized in that The target action data includes gesture data, and the recognition processing result obtained by performing recognition processing on the target action data includes any one or more of the following: Gestures of drawing a circle clockwise on a vertical plane, drawing a circle counterclockwise on a vertical plane, drawing a circle clockwise on a horizontal plane, drawing a circle counterclockwise on a horizontal plane, swiping left, swiping right, swiping up, and swiping down.

11. The method according to claim 1, characterized in that The micro-electric inertial navigation sensor is deployed in a virtual reality interactive device, and the result of the recognition process is used to indicate the action category to which the action belongs; the method also includes: In response to the result of the recognition processing, the virtual reality interaction device is controlled to perform an interactive operation matching the action; or, the virtual reality interaction device is controlled to display an interactive interface matching the action.

12. The method according to claim 1, characterized in that The micro-electric inertial navigation sensor is deployed in a cloud game interaction device, and the result of the recognition process is used to indicate the action category to which the action belongs; The method further comprises: In response to the result of the identification processing, the cloud game interaction device is controlled to execute the game instructions corresponding to the action category; or, the cloud game interaction device is controlled to update and display the cloud game screen that matches the game instructions corresponding to the action category.

13. A data processing device, characterized in that: include: An acquisition module is used to acquire target motion data to be processed, wherein the target motion data is acquired by collecting the motion of the object using a micro-electric inertial navigation sensor; A parsing module, used for parsing the target action data to determine the type of the action, including: calculating the energy of the target action data; if the energy of the target action data is greater than or equal to a preset threshold, determining the type of the action is a first type; if the energy of the target action data is less than the preset threshold, determining the type of the action is a second type; wherein the action amplitude of the first type is greater than the action amplitude of the second type; an identification processing module, configured to: if the type of the action is the first type, extract the time domain features of the target action data; call the action recognition model to perform identification processing on the time domain features of the target action data to obtain a first identification processing result, wherein the first identification processing result is used to indicate the action category to which the action indicated by the target action data belongs; The recognition and processing module is also used to: if the type of the action is the second type, perform time domain transformation processing on the target action data to obtain frequency domain information corresponding to the target action data; perform feature extraction processing on the frequency domain information to obtain frequency domain features of the target action data; call the action recognition model to perform recognition processing on the frequency domain features of the target action data to obtain a second recognition processing result, and the second recognition processing result is used to indicate the action category to which the action indicated by the target action data belongs.

14. A computer device, characterized in that: include: Processor, memory, and network interface; The processor is connected to the memory and the network interface, wherein the network interface is used to provide a network communication function, the memory is used to store program code, and the processor is used to call the program code to execute the data processing method according to any one of claims 1 to 12.

15. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a processor, the data processing method according to any one of claims 1 to 12 is executed.

16. A computer program product, characterized in that The computer program product comprises a computer program or a computer instruction, and when the computer program or the computer instruction is executed by a processor, the data processing method according to any one of claims 1 to 12 is implemented.

Images