Virtual reality device interaction recognition system based on arm tapping and control device

Abstract

The application discloses a virtual reality device interaction recognition system and control device based on arm knocking. The virtual reality device can recognize the action that a user uses a virtual reality handle to knock the forearm of the other hand without additional equipment. Based on the recognition of the knocking signal, the application provides a new interaction mode for the virtual reality device. Meanwhile, in order to further enrich the interaction form of the knocking action, the application also provides a method for recognizing the knocking force and the knocking position, which can be used for further enriching the control instruction types that can be issued by the application.

Description

Technical Field

[0001] This invention relates to a design scheme for an interactive technology, applicable to the interaction between users and virtual reality devices, in order to enrich the way of human-computer interaction and improve the user experience. Background Technology

[0002] Virtual Reality (VR) is a computer simulation system that can create and experience virtual worlds. It uses computers to generate a simulated environment, which is an interactive, three-dimensional dynamic visual and physical behavior simulation system that integrates multi-source information, thereby immersing the user in that environment.

[0003] VR controllers are an indispensable component of most modern VR systems. A common way to use VR controllers is to hold them in each hand, allowing users to mimic real-world behaviors and improve the performance of two-handed controller interaction. VR controllers provide significant support for interactive control, but in current technology, the control buttons, function buttons, and joysticks on interactive controllers are randomly and scattered, making it difficult for users to control them flexibly. This not only increases the difficulty of use but also fails to provide a smooth user experience.

[0004] For existing VR devices, researchers aim to streamline essential components for easier portability and use, while simultaneously increasing user interaction methods to simplify operation. In other words, they hope VR devices can achieve rich interactive functions with only a few components. Currently, commonly used VR controllers typically consist of a joystick, control buttons, and function buttons, with their functionality limited by the number of buttons. More interaction usually relies on the controller emitting light to select components within the VR scene. Selection requires the user to move the controller to the desired position, which is time-consuming, inconsistent, and inconvenient. Summary of the Invention

[0005] Purpose of the invention: In order to enrich the methods and types of human-computer interaction in VR devices, this invention designs a novel VR interaction technology based on arm tapping, which increases the ways of human-computer interaction without the need for additional hardware.

[0006] To achieve the above-mentioned objectives, the technical solution proposed by this invention is as follows:

[0007] In a first aspect, the present invention provides a virtual reality device interaction recognition system based on arm tapping, for recognizing tapping action information intended by a user through the controller of a virtual reality device, comprising:

[0008] The knock interaction data acquisition module is configured to acquire sensor time series data from a sensor built in a handle of the virtual reality device in real time during a user performing an arm knock interaction action, the arm knock interaction action being that, in a state where the user wears the virtual reality device and holds left and right handles with both hands respectively, one hand knocks the forearm of the other hand while holding the handle, and the sensor time series data including handle position coordinates, handle speed, handle acceleration, handle angular speed and handle angular acceleration of the two handles respectively.

[0009] The action feature extraction module is configured to extract the sensor time series data acquired by the knock interaction data acquisition module in real time through a sliding window, and the sensor time series data in each sliding window constituting an action feature set.

[0010] The knock action recognition module is configured to input the latest action feature set in the action feature extraction module into a trained machine learning model, and determine whether the latest action feature set has a left-hand knock action or a right-hand knock action, if not, continue to determine the next action feature set, and if so, continue to identify the knock intensity and the knock position of the knock action.

[0011] The knock intensity recognition module is configured to extract the maximum amplitude of acceleration of the handle acceleration time series data from the latest action feature set when the knock action recognition module detects that one hand has a knock action, then determine the interval range of the maximum amplitude of acceleration in a first threshold matching interval corresponding to the hand performing the knock action, and determine the knock intensity of the knock action according to the mapping relationship between the interval range in the first threshold matching interval and the knock intensity.

[0012] The knock position recognition module is configured to, when the knock action recognition module detects that one hand has a knock action, convert the handle position coordinate time series data of the two handles in the latest action feature set to obtain spatial distance time series data of the two handles, then extract the minimum valley value of the spatial distance in the spatial distance time series data, determine the interval range of the minimum valley value of the spatial distance in a second threshold matching interval corresponding to the hand performing the knock action, and determine the knock position of the knock action on the knocked arm according to the mapping relationship between the interval range in the second threshold matching interval and the knock position of the knock point on the knocked arm.

[0013] As a preferred embodiment of the first aspect, the first threshold matching interval corresponding to each hand contains N interval ranges, the N interval ranges being divided by N-1 first threshold points from the upper and lower boundary ranges of the maximum amplitude of acceleration corresponding to the hand, and each interval range corresponding to a knock intensity; N is an integer not less than 2.

[0014] As a preferred embodiment of the first aspect, the first threshold matching interval preferably comprises two or three interval ranges.

[0015] As a preferred embodiment of the first aspect, the first threshold matching interval is generated in the initialization enabling stage of the virtual reality device, and is generated in the following manner:

[0016] The user is prompted to repeatedly perform the tapping action with the specified hand at different tapping forces, and sensor time series data is acquired in real time from the built-in sensor of the handle of the virtual reality device during the execution of the tapping action. Then, the action feature extraction module and the tapping action recognition module are used to determine each tapping action and the corresponding action feature set that occurs during the repeated execution of the tapping action. The maximum amplitude of acceleration of the handle acceleration time series data is extracted from the action feature set corresponding to each tapping action. The maximum amplitude of acceleration corresponding to all tapping actions is then counted to determine the upper and lower bounds of the maximum amplitude of acceleration. Finally, the maximum amplitude of acceleration under the same tapping force is averaged, and the average of the maximum amplitude of acceleration of adjacent two tapping forces is taken as a first threshold point that distinguishes the two tapping forces. All first threshold points are used to divide the upper and lower bounds of the maximum amplitude of acceleration to form a plurality of interval ranges that have a mapping relationship with the tapping force, thereby generating the first threshold matching interval corresponding to the specified hand.

[0017] As a preferred embodiment of the first aspect, the second threshold matching interval corresponding to each hand comprises M interval ranges, which are divided by M-1 second threshold points from the upper and lower bounds of the spatial distance minimum valley value corresponding to the hand, and each interval range corresponds to a tapping position. M is an integer not less than 2.

[0018] As a preferred embodiment of the first aspect, the second threshold matching interval preferably comprises two or three interval ranges.

[0019] As a preferred embodiment of the first aspect, the second threshold matching interval is generated in the initialization enabling stage of the virtual reality device, and is generated in the following manner:

[0020] The user is prompted to repeatedly perform a tapping action with a specified hand at different tapping positions on the other hand, and sensor timing data is acquired in real time from the built-in sensors of the virtual reality device handles during the tapping action; then, the action feature extraction module and the tapping action recognition module obtain an action feature set corresponding to each tapping action, extract handle position coordinate timing data of the two handles from the action feature set corresponding to each tapping action, and convert the handle position coordinate timing data into spatial distance timing data of the two handles, and then extract the minimum spatial distance valley value in the spatial distance timing data; then, the minimum spatial distance valley values corresponding to all tapping actions are counted to determine the upper and lower boundary ranges of the minimum spatial distance valley value; finally, the minimum spatial distance valley values under the same tapping position are averaged, and the average of the minimum spatial distance valley values of adjacent two tapping positions is taken as a second threshold point for distinguishing the two tapping positions, and all second threshold points are used to divide the upper and lower boundary ranges of the minimum spatial distance valley value to form a plurality of interval ranges having a mapping relationship with the tapping positions, so as to generate a second threshold matching interval corresponding to the specified hand.

[0021] As a preferred embodiment of the first aspect, the machine learning model is a support vector machine model.

[0022] As a preferred embodiment of the first aspect, the support vector machine model is trained in advance using labeled sample data, each label sample contains the sensor timing data in a sliding window and a label indicating whether there is a tapping action in the sliding window, and the label is divided into three categories: left hand tapping action, right hand tapping action and no tapping action.

[0023] In a second aspect, the present application provides a virtual reality device interaction control device based on arm tapping, characterized in that it comprises the virtual reality device interaction recognition system based on arm tapping according to any one of the first aspect, and a control instruction output module.

[0024] The control instruction output module is configured to receive the tapping action information recognized by the virtual reality device interaction recognition system in real time, the tapping action information including one or more of the tapping arm, the tapping force and the tapping position corresponding to the tapping action, and outputting a control instruction representing the current control intention of the user according to the mapping relationship between the tapping action information and the control instruction.

[0025] Compared with the prior art, the present application has the following advantages:

[0026] 1) The tapping interaction method proposed by the present application has multiple variants, which are formed by combinations of tapping information such as tapping force, tapping forearm position and tapping arm. These types of tapping actions are easy to distinguish, greatly enriching the current virtual reality interaction method.

[0027] 2) The tapping interaction method proposed in this invention has good time performance. Each tapping action can be completed within two seconds, which improves the efficiency of use compared to using a controller to emit light to select buttons.

[0028] 3) The tapping interaction method proposed in this invention requires very simple actions, so users do not have the burden of learning and the user experience is consistent.

[0029] 4) The software design of this invention is simple, requires no additional hardware support, does not contain complex algorithms, is easy to implement and port, and therefore has a wide range of applications.

[0030] 5) The tapping interaction method proposed in this invention is compatible with existing VR device interaction technologies, and using this technology will not affect the existing interaction methods.

[0031] 6) This invention adopts a new feature extraction and analysis scheme, which greatly improves the accuracy of action recognition. Attached Figure Description

[0032] Figure 1 This is a diagram illustrating the action of tapping with the arm.

[0033] Figure 2 This shows a comparison of the handle acceleration characteristics using two force examples.

[0034] Figure 3 This section compares the relative positional features of the handle in three different examples of striking the arm. Detailed Implementation

[0035] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Many specific details are set forth in the following description to provide a thorough understanding of the present invention. However, the present invention can be practiced in many other ways different from those described herein, and those skilled in the art can make similar modifications without departing from the spirit of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below. Technical features in the various embodiments of the present invention can be combined accordingly without mutual conflict.

[0036] In the description of this invention, it should be understood that when an element is considered to be "connected" to another element, it can be a direct connection to the other element or an indirect connection, i.e., there is an intermediate element. Conversely, when an element is said to be "directly" connected to another element, there is no intermediate element.

[0037] In the description of the present application, it should be understood that the terms "first", "second" are only used for the purpose of distinguishing description, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the technical features indicated. Therefore, the features defined with "first", "second" can explicitly or implicitly include at least one of the features.

[0038] In life, due to the different physical conditions of each person, the height and arm length are different, and the habits of exercise are also different, and many personal habits and characteristics will also appear when using VR equipment. The goal of the present application is to realize the recognition function of the arm knocking action by analyzing the common characteristics of all people in the knocking process, and further combining the knocking characteristics of each person to pre-collect data on this basis, realizing the accurate positioning of the knocking action, including the position of the knocking arm and the force of the knocking arm. The knocking action can be used to output the related control command of the virtual reality scene according to the control intention of the user. The present application mainly improves the following three aspects:

[0039] (1) The present application proposes a new virtual reality double-hand handle interaction technology, and the interaction action is to hold the handle with one hand and knock the forearm of the other hand (the hand also needs to hold the handle). The technology proposed in the present application only needs to use the position sensor and gyroscope built in the virtual reality device handle to detect the knocking event, thereby increasing the way of human-computer interaction.

[0040] (2) The present application proposes a new feature analysis method, which analyzes the sensor data features of the knocking action, and effectively improves the recognition accuracy. The present application organizes the feature data of the sensor in the form of a sliding window as a new action feature set. For the processed action feature set, a machine learning method is used for knocking action detection.

[0041] (3) The present application proposes a knocking force and position detection method based on sensor data. On the basis of realizing the detection of the knocking action of the double hands, the relative position of the handle, the acceleration change amplitude and other characteristics are further analyzed, the knocking force and position characteristic matching interval is divided, and the further qualitative of the knocking action is realized.

[0042] It should be noted that the specific definition of the user's knocking action in the present application is that the user wears virtual reality glasses, and holds the corresponding sensor handle with both hands. The state of holding the handle is as shown in Figure 1 Before the knocking occurs, there should be a certain distance between the arms of the two hands, and then the handle of the side that will knock is driven by the arm to the front side of the other arm for knocking. During the knocking process, the hands cannot leave the handle, the knocking action is a one-time action, and after the impact occurs, the hands should be separated immediately. During this knocking process, the swinging hand is the knocking arm, and the other hand is the knocked arm.

[0043] In a preferred embodiment of the present application, a virtual reality device interaction recognition system based on arm knocking is provided for recognizing the knocking action information intended to be issued by a user through the handle of a virtual reality device, which comprises the following functional modules.

[0044] A knocking interaction data acquisition module is configured to acquire sensor time series data from the built-in sensors of the handles of the virtual reality device in real time during the execution of an arm knocking interaction action by the user. The arm knocking interaction action is knocking the forearm of one hand by holding the handle of the other hand in the state that the user wears the virtual reality device and holds the left and right handles with both hands. The sensor time series data includes the handle position coordinates, handle speed, handle acceleration, handle angular speed, and handle angular acceleration of the two handles respectively.

[0045] It should be further noted that the virtual reality device and the corresponding handles will generate different sensor data during the knocking process, and these sensor data may be redundant, so these sensor data can be reduced in dimension to retain the core data dimensions of the handle position coordinates, handle speed, handle acceleration, handle angular speed, and handle angular acceleration, thereby synthesizing the original scattered data as the knocking recognition features.

[0046] An action feature extraction module is configured to extract the sensor time series data acquired by the knocking interaction data acquisition module in real time through a sliding window, and the sensor time series data in each sliding window constitutes an action feature set.

[0047] A knocking action recognition module is configured to input the current latest action feature set in the action feature extraction module into a trained machine learning model to determine whether there is a left-hand knocking action or a right-hand knocking action in the current latest action feature set. If there is no knocking action, the next action feature set is continuously determined, and if there is a knocking action, the knocking force and the knocking position of the knocking action are continuously recognized subsequently.

[0048] The present application needs to further locate the tapping position after determining the tapping action: when the user uses the function for the first time, the device will require the user to tap with different tapping forces and different tapping positions respectively, which are recorded as the user tapping characteristics in the device. Then, according to the user tapping characteristics, the handle relative position characteristic data of different tapping positions is analyzed to determine the characteristic matching interval, so as to determine the specific position of the user's tapping arm according to the landing point of the handle relative position characteristic of each tapping. In addition, when identifying the tapping force, the acceleration change characteristics of the handle in the sliding window are analyzed according to the user tapping characteristics, and the median value of the acceleration change characteristics between adjacent two forces is used as the threshold to divide into multiple change amplitude stages to obtain the acceleration change amplitude threshold corresponding to different forces, and then the specific force of the user's tapping is determined according to the threshold. The handle relative position characteristics and acceleration change characteristics specifically selected by the present application are the minimum valley value of the spatial distance and the maximum amplitude value of the acceleration, and the following will specifically describe the tapping force identification module and the tapping position identification module for realizing the identification of tapping force and tapping position.

[0049] The tapping force identification module is used to extract the maximum amplitude value of the handle acceleration time series data from the current latest action characteristic set when the tapping action identification module detects that one hand has a tapping action, then judge the interval range of the maximum amplitude value in the first threshold matching interval corresponding to the hand performing the tapping action, and determine the tapping force of the tapping action according to the mapping relationship between the interval range in the first threshold matching interval and the tapping force.

[0050] The tapping position identification module is used to convert the hand position coordinate time series data of the two handles in the current latest action characteristic set to obtain the spatial distance time series data of the two handles when the tapping action identification module detects that one hand has a tapping action, then extract the minimum valley value of the spatial distance in the spatial distance time series data, judge the interval range of the minimum valley value in the second threshold matching interval corresponding to the hand performing the tapping action, and determine the tapping position of the tapping action on the tapped arm according to the mapping relationship between the interval range in the second threshold matching interval and the tapping position of the tapping point on the tapped arm.

[0051] It should be noted that the present application determines the tapping force and the tapping position of the tapping action by using two different information, i.e., the maximum amplitude value of the acceleration is used to determine the tapping force, and the minimum valley value of the spatial distance is used to determine the tapping position. The spatial distance of the two handles can be calculated based on the handle position coordinates of the two handles.

[0052] The maximum amplitude of acceleration refers to the maximum value of each amplitude in the handle acceleration time series data in the sliding window. The minimum valley value of spatial distance refers to the minimum value of each valley value in the spatial distance time series data in the sliding window. The determination of the above-mentioned tapping strength and tapping position needs to be realized by combining the first threshold matching interval and the second threshold matching interval respectively. The specific form and generation method of the first threshold matching interval and the second threshold matching interval will be described below. Since the maximum amplitude of acceleration and the minimum valley value of spatial distance generated by the left and right hands when performing the tapping action are different, the first threshold matching interval and the second threshold matching interval also need to be constructed for each hand respectively.

[0053] The first threshold matching interval corresponding to each hand contains N interval ranges, which are divided by N-1 first threshold points from the upper and lower boundary ranges of the maximum amplitude of acceleration corresponding to the hand, and each interval range corresponds to a tapping strength; N is an integer not less than 2. However, considering that too many tapping strength levels are easy to produce misrecognition, the first threshold matching interval preferably contains two or three interval ranges, and further preferably two interval ranges corresponding to two tapping strengths.

[0054] The above-mentioned first threshold matching interval is generated in the initialization and enabling link of the virtual reality device, and its generation method is as follows:

[0055] The user is prompted to repeatedly perform the tapping action according to different tapping strengths by using the specified hand, and sensor time series data is acquired in real time from the built-in sensor of the handle of the virtual reality device during the execution of the tapping action. Then, each tapping action and the corresponding action feature set occurring during the repeated execution of the tapping action are determined by the action feature extraction module and the tapping action recognition module, the maximum amplitude of acceleration of the handle acceleration time series data is extracted from the action feature set corresponding to each tapping action, the upper and lower boundary ranges of the maximum amplitude of acceleration are determined by counting the maximum amplitudes of acceleration corresponding to all tapping actions, and finally, the maximum amplitudes of acceleration under the same tapping strength are averaged, the maximum amplitudes of acceleration of adjacent two tapping strengths are averaged as a first threshold point distinguishing the two tapping strengths, and the upper and lower boundary ranges of the maximum amplitude of acceleration are divided by all first threshold points to form multiple interval ranges having a mapping relationship with the tapping strength, thereby generating the first threshold matching interval corresponding to the current specified hand.

[0056] In addition, the second threshold matching interval corresponding to each hand contains M interval ranges, which are divided by M-1 second threshold points from the upper and lower boundary ranges of the minimum valley value of spatial distance corresponding to the hand, and each interval range corresponds to a tapping position; M is an integer not less than 2. However, considering that setting too many tapping positions is easy to produce misrecognition, the second threshold matching interval preferably contains two or three interval ranges.

[0057] The second threshold matching interval is generated in the initialization enabling link of the virtual reality device, and the generation manner is as follows:

[0058] The user is prompted to repeatedly perform the knocking action with the specified hand at different knocking positions on the other hand, and sensor time sequence data is acquired in real time from the built-in sensor of the handle of the virtual reality device during the execution of the knocking action; then the action feature set corresponding to each knocking action is obtained through the action feature extraction module and the knocking action recognition module, the handle position coordinate time sequence data of the two handles is extracted from the action feature set corresponding to each knocking action, and the spatial distance time sequence data of the two handles is converted to obtain the minimum spatial distance valley value in the spatial distance time sequence data; then the minimum spatial distance valley values corresponding to all the knocking actions are counted to determine the upper and lower boundary ranges of the minimum spatial distance valley value; finally, the minimum spatial distance valley values under the same knocking position are averaged, and the minimum spatial distance valley values of adjacent two knocking positions are averaged as a second threshold point for distinguishing the two knocking positions, and all the second threshold points are used to divide the upper and lower boundary ranges of the minimum spatial distance valley value to form a plurality of interval ranges which have a mapping relationship with the knocking positions, so that the second threshold matching interval corresponding to the current specified hand is generated.

[0059] In the initialization enabling link of the virtual reality device, the user can be prompted to perform the corresponding knocking action through a visual image or sound.

[0060] The machine learning model used for judging whether the knocking action exists in the application can be a trainable classification model, and a support vector machine model can be used in the application. The support vector machine model is trained in advance by using labeled sample data, each labeled sample contains the sensor time sequence data in a sliding window and a label indicating whether the knocking action exists in the sliding window, and the label is divided into three categories: the left hand performs the knocking action, the right hand performs the knocking action and there is no knocking action.

[0061] It should be noted that the modules in the virtual reality device interaction recognition system based on arm knocking described above can be program modules executable on a computer device, and therefore essentially execute a data processing process. It can be clearly understood by those skilled in the art that, for the convenience and brevity of description, the specific working process of the system described above can refer to the corresponding process in the foregoing method embodiments, which will not be described herein. In each embodiment provided in the application, the division of steps or modules in the method and system is only a logical functional division, and another division mode can be used in actual implementation, for example, multiple modules or steps can be combined or integrated together, or a module or a step can be split.

[0062] The virtual reality device interaction recognition system based on arm knocking can recognize different action information from the knocking interaction action of the user, mainly the knocking arm, the knocking force and the knocking position corresponding to the knocking action. These action information can form different combination modes, and each combination action can correspond to a control instruction. Therefore, on the basis of the virtual reality device interaction recognition system based on arm knocking, another embodiment of the present application further provides a virtual reality device interaction control device based on arm knocking, which comprises the virtual reality device interaction recognition system based on arm knocking and a control instruction output module. The control instruction output module is used to receive the knocking action information recognized by the virtual reality device interaction recognition system in real time, and output the control instruction representing the current control intention of the user according to the mapping relationship between the knocking action information and the control instruction.

[0063] However, it should be noted that the knocking action information mapped to the control instruction can include one or more of the knocking arm, the knocking force and the knocking position corresponding to the knocking action. For example, for a control mode including only one kind of knocking action information, only the knocking arm can be used to output the control instruction, and the user can use the left hand to knock and the right hand to knock to generate two different control instructions respectively; for a control mode including multiple kinds of knocking action information, two or three of the knocking arm, the knocking force and the knocking position can be combined to form different control instructions.

[0064] The specific construction and implementation of the virtual reality device interaction recognition system based on arm knocking will be described below through a specific embodiment.

[0065] Embodiment

[0066] In this embodiment, in order to realize the detection of the knocking action, a plurality of testers are randomly selected to collect their knocking data. During the experiment, each tester will perform a plurality of tests, and a certain number of knocking actions will be randomly selected for each test, and each kind of knocking action will be repeatedly knocked 5 times. These knocking actions will be randomly selected from the action set composed of different knocking arms (left hand or right hand), different knocking forces (two kinds of heavy and light) and different knocking positions (knocking on the forearm of the knocked arm near the wrist position, knocking on the forearm of the knocked arm at the middle position, and knocking on the forearm of the knocked arm near the elbow position).

[0067] Before the experiment, the experimenter will naturally place the two hands in front of the handle, ensure that there is a certain distance between the two handles, generally the width of the body. Then according to the experimental indication, the required action of knocking. The VR device will collect the handle sensor data of the knocking process at a fixed frequency through the written software, and the main collected data are handle position coordinates, handle speed, handle acceleration, handle angular velocity and handle angular acceleration.

[0068] The main implementation of the present application is that: during the occurrence of the knocking, the relative position change of the handle is representative, and at the same time when the impact occurs, the acceleration and angular acceleration of the two handles will also have corresponding mutations, which can be used as characteristics for distinguishing whether it is a knocking action, as shown in the following Figure 2 、 3 .

[0069] The specific details of the feature extraction scheme are as follows: the knocking action is defined as a process, including two time periods before and after the occurrence of the knocking point. For each set of real-time data collected, the original form is very complex and redundant, such as handle speed information which will be divided into x, y and z three directions for description. In order to simplify the original multi-dimensional sensor data, first, through feature commonality analysis, the feature is integrated and reduced in dimension, and the data with the same description is merged into the same class of data features, for example, the same class of handle position data in the VR head-mounted device and the handle is merged. Then, by analyzing the acceleration and the change of the handle position, the time from the impact mutation of the acceleration to the restoration of the knocking action is summarized, which is the restoration time after the impact. Taking the knocking time as the midpoint, a certain range of time before and after the knocking is taken, the knocking time window is obtained, and the time window data is used as the basic data to construct the positive samples of the training machine learning model. The knocking time window can be set to 100ms in this embodiment. Of course, in addition to the positive samples, the training machine learning model also needs to include negative samples, which do not contain knocking actions, and the length of the knocking time window of the negative samples is the same as that of the positive samples. As shown in the right box of Figure 2 、 3 , the knocking time window is the knocking time window, and the window width is different because the sampling density of the horizontal coordinate is different. The knocking time window is in the form of a sliding window, which continuously slides on the real-time generated sensor data sequence to form a new action feature set data, which is sent to the machine learning model for training. The selected model is a support vector machine model.

[0070] After training, the support vector machine model can achieve detection of the tapping action with high accuracy. To achieve more accurate tapping action positioning, the embodiment will require the user of the function to collect tapping features in advance, specifically: the device instructs the user to tap with different forces, saves the time window centered on the tapping point detected by the model, analyzes the handle acceleration amplitude of the two forces, and takes the median value of the maximum amplitude of acceleration between the adjacent two tapping forces as the threshold to divide into a plurality of maximum amplitude of acceleration interval ranges, i.e. the first threshold matching interval. Subsequent each tapping action is determined by judging the interval range of the maximum amplitude of acceleration of the tapping time window, thereby determining the specific tapping force. For example Figure 2 As shown in the acceleration feature example, the right box area of the figure is the time window where the tapping point is located. It can be found that the maximum amplitude of acceleration of the two tapping forces has obvious difference, so the maximum amplitude of acceleration interval range can be divided based on this characteristic to determine the actual tapping force.

[0071] In addition, the device will also instruct the user to tap at different positions of the forearm, divided into tapping the forearm near the wrist position, tapping the forearm middle position, and tapping the forearm near the elbow position. By analyzing the minimum valley value of the spatial distance between the handles of different tapping positions, the minimum valley value interval range of the spatial distance corresponding to different tapping positions is determined, i.e. the second threshold matching interval, so as to determine the specific position of the tapping arm according to the spatial distance minimum valley interval of each tapping. As shown in Figure 3 According to the data change of the spatial distance between the handles in the tapping time window extracted by the model, the red box area of the right figure is the common tapping time window, and the three example tapping positions in it have different distances from the wrist. It can be found that the peak value of the handle relative position feature (dashed line of three colors) also changes with the change of the distance, so it can be used for accurate positioning of the tapping position in actual detection.

[0072] Therefore, by using the support vector machine model, the first threshold matching interval, and the second threshold matching interval obtained above, the tapping action recognition module, the tapping force recognition module, and the tapping position recognition module of the virtual reality device interaction recognition system can be constructed. Through the device interface, the tapping interaction data acquisition module is set, and the action feature extraction module is constructed by designing the data extraction processing program, so that the virtual reality device interaction recognition system is obtained, which is used to recognize the tapping action information intended by the user through the handle of the virtual reality device.

[0073] The above-described embodiments are only the preferred ones of the present application, and are not intended to limit the present application. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the present application. Therefore, any technical solutions obtained by equivalent replacement or equivalent transformation shall fall within the protection scope of the present application.

Claims

1. A virtual reality device interaction recognition system based on arm tapping, used to recognize tapping action information intentionally emitted by a user through the controller of a virtual reality device, characterized in that, include: The tapping interaction data acquisition module is used to acquire real-time sensor timing data from the built-in sensors of the virtual reality device's controllers during the user's arm tapping interaction action; the arm tapping interaction action is when the user is wearing the virtual reality device and holding the left and right controllers respectively, and taps the forearm of the other hand with one hand holding the controller; the sensor timing data includes the controller position coordinates, controller velocity, controller acceleration, controller angular velocity, and controller angular acceleration of each of the two controllers. The motion feature extraction module is used to extract the sensor time-series data acquired in real time by the tapping interaction data acquisition module through a sliding window. The sensor time-series data in each sliding window constitutes a motion feature set. The tapping action recognition module is used to input the latest action feature set from the action feature extraction module into the trained machine learning model to determine whether there is a left-hand tapping action or a right-hand tapping action in the latest action feature set. If there is no tapping action, the next action feature set is judged. If there is a tapping action, the tapping force and tapping position of the tapping action are identified. The striking force recognition module is used to extract the maximum acceleration value of the handle acceleration time sequence data from the latest action feature set when the striking action recognition module detects that a hand is striking. Then, it determines the range of the maximum acceleration value in the first threshold matching interval corresponding to the hand performing the striking action, and determines the striking force of the striking action based on the mapping relationship between the range of the first threshold matching interval and the striking force. The tapping position recognition module is used to convert the spatial distance time series data of the two handles based on the latest action feature set of the handle position coordinate time series data of the two handles when the tapping action recognition module detects that a hand is tapping. Then, it extracts the minimum valley value of the spatial distance time series data, determines the range of the minimum valley value of the spatial distance in the second threshold matching interval corresponding to the hand that performed the tapping action, and determines the tapping position of the tapping action on the arm being tapped based on the mapping relationship between the range of the second threshold matching interval and the tapping position of the tapping point on the arm being tapped.

2. The virtual reality device interaction recognition system based on arm tapping as described in claim 1, characterized in that, The first threshold matching interval for each hand contains N interval ranges. These N interval ranges are divided by the upper and lower bounds of the maximum amplitude of the acceleration corresponding to this hand through N-1 first threshold points. Each interval range corresponds to a tapping force; N is an integer not less than 2.

3. The virtual reality device interaction recognition system based on arm tapping as described in claim 2, characterized in that, The first threshold matching interval preferably includes two or three interval ranges.

4. The virtual reality device interaction recognition system based on arm tapping as described in claim 2, characterized in that, The first threshold matching interval is generated during the initialization and activation phase of the virtual reality device, and its generation method is as follows: The system prompts the user to repeatedly perform a tapping action with a specified hand at different tapping forces. During the tapping action, the system acquires real-time sensor timing data from the built-in sensors of the virtual reality device's controller. Then, the system uses a motion feature extraction module and a tapping action recognition module to determine each tapping action and its corresponding motion feature set during the repeated tapping action. The system then extracts the maximum acceleration value of the controller's acceleration timing data from the motion feature set corresponding to each tapping action. Next, the maximum acceleration values ​​corresponding to all striking actions are statistically analyzed to determine the upper and lower bounds of the maximum acceleration values. Finally, the maximum acceleration values ​​under the same striking force are averaged, and the average of the maximum acceleration values ​​of two adjacent striking forces is used as a first threshold point to distinguish between the two striking forces. All first threshold points are used to divide the upper and lower bounds of the maximum acceleration values, forming multiple intervals that are mapped to the striking force, thereby generating the first threshold matching interval corresponding to the currently specified hand.

5. The virtual reality device interaction recognition system based on arm tapping as described in claim 1, characterized in that, The second threshold matching interval for each hand contains M interval ranges. These M interval ranges are divided by the upper and lower bounds of the minimum valley value of the spatial distance corresponding to this hand through M-1 second threshold points. Each interval range corresponds to a tapping position; M is an integer not less than 2.

6. The virtual reality device interaction recognition system based on arm tapping as described in claim 5, characterized in that, The second threshold matching interval preferably includes two or three interval ranges.

7. The virtual reality device interaction recognition system based on arm tapping as described in claim 5, characterized in that, The second threshold matching interval is generated during the initialization and activation phase of the virtual reality device, and its generation method is as follows: The system prompts the user to repeatedly perform tapping actions using a designated hand at different tapping positions on the other hand. During these actions, real-time temporal data is acquired from the built-in sensors of the virtual reality device's controllers. Then, a motion feature extraction module and a tapping action recognition module obtain the motion feature set corresponding to each tapping action. From this set, the temporal data of the controller position coordinates for both controllers are extracted and converted to spatial distance temporal data. The minimum spatial distance valley value is then extracted from this data. The minimum spatial distance valley values ​​for all tapping actions are statistically analyzed to determine their upper and lower bounds. Finally, the minimum spatial distance valley values ​​at the same tapping position are averaged. The average of the minimum spatial distance valley values ​​for two adjacent tapping positions is used as a second threshold point to distinguish between these two positions. All second threshold points are used to divide the upper and lower bounds of the minimum spatial distance valley values, forming multiple intervals that map to the tapping positions, thus generating the second threshold matching interval corresponding to the currently designated hand.

8. The virtual reality device interaction recognition system based on arm tapping as described in claim 1, characterized in that, The machine learning model is a support vector machine model.

9. The virtual reality device interaction recognition system based on arm tapping as described in claim 8, characterized in that, The support vector machine model is pre-trained using labeled sample data. Each labeled sample contains the sensor time series data within a sliding window and a label indicating whether a tapping action exists within the sliding window. The labels are divided into three categories: tapping action performed by the left hand, tapping action performed by the right hand, and no tapping action.

10. A virtual reality device interaction control device based on arm tapping, characterized in that, It includes a virtual reality device interaction recognition system based on arm tapping as described in any one of claims 1 to 9, and a control command output module; The control command output module is used to receive tapping action information identified by the virtual reality device interaction recognition system in real time. The tapping action information includes one or more of the tapping arm, tapping force, and tapping position corresponding to the tapping action. Based on the mapping relationship between the tapping action information and the control command, the module outputs a control command representing the user's current control intention.

Images