An interactive system based on lower limb movement

Abstract

The application discloses an interactive system based on lower limb movement, comprising a flexible insole, a motion analysis end, a mapping end and a driving end; the motion analysis end receives lower limb movement data from the motion insole, and identifies lower limb movement information containing primary movement and secondary movement from the lower limb movement data by using a neural network based on machine learning, and sends the lower limb movement information to the mapping end; the mapping end receives the lower limb movement information, and generates mapping movement instructions suitable for the driving end according to the lower limb movement information; and the driving end receives the mapping movement instructions from the mapping end, and performs interactive operation on a control target object. By using the application, the flexible insole has a large activity range of lower limb movement and small binding requirement, and can easily meet the requirements of various sites; the accuracy of lower limb movement recognition is improved by using a machine algorithm, and immersive interaction is realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of somatosensory interaction technology, and particularly relates to an interaction system based on lower limb movement. BACKGROUND

[0002] Controlling the walk, run, jump, squat and other lower limb related movements of a virtual image is a very common interaction mode. At present, the mainstream way to control the movement of a virtual image is through a mouse, keyboard or handle, by pressing the corresponding keys to perform different directions and different types of lower limb movements. This way has the advantages of simplicity, speed, low implementation cost, etc., but it lacks immersive and natural experience to control the movement by hand. Compared with hand control, omnidirectional treadmills, hollow spherical space simulation systems, real motion capture devices and other devices can correspond to the actual movement of the lower limbs, thereby providing a more natural and immersive experience.

[0003] However, current foot action recognition based on foot action recognition is usually applied to human gait phase and robot action mode recognition, and does not involve action direction changes, making it difficult to apply to virtual game action scenarios. The standing action of small range movement of the foot has small differentiation, and the recognition difficulty is large, and the existing interaction device is difficult to accurately recognize the user's action and thus control the virtual image. SUMMARY

[0004] The present application provides an interaction system based on lower limb movement, simulating the input device of a host computer, issuing corresponding action instructions according to the lower limb movement of the user, and replacing the keyboard, mouse and handle to realize input control of the host computer, with a wider application range.

[0005] To solve the above technical problems, the present application provides an interaction system based on lower limb movement, comprising: a flexible insole, an action analysis end, a mapping end and a driving end;

[0006] The flexible insole comprises a flexible pressure sensor, an acceleration sensor, a data acquisition module and a wireless communication module; the flexible insole senses the lower limb movement through the flexible pressure sensor and the acceleration sensor, acquires the lower limb movement data of the flexible pressure sensor and the acceleration sensor through the data acquisition module, and sends the lower limb movement data to the action analysis end through the wireless communication module;

[0007] The action analysis end receives the lower limb movement data, and identifies the lower limb action information containing a primary action and a secondary action from the lower limb movement data by using a neural network based on machine learning, and sends the lower limb action information to the mapping end; the primary action includes sprinting, squatting, fast moving, jumping and standing, and the secondary action includes forward, backward, left, right and in-place;

[0008] The mapping end receives the lower limb action information and generates a mapping action instruction suitable for the driving end according to the lower limb action information;

[0009] The driving end receives the mapping action instruction from the mapping end and interacts with the control target object.

[0010] In a possible implementation, the lower limb action information containing the primary action and the secondary action is identified from the lower limb movement data by using a neural network based on machine learning, and specifically includes:

[0011] Action features are extracted from the lower limb movement data; the action features include mean standard deviation, combined acceleration, foot average pressure, and acceleration DCT processing value;

[0012] The lower limb movement data is identified by using the trained neural network in combination with the action feature extraction result, to obtain the lower limb action information containing the primary action and the secondary action.

[0013] In a possible implementation, the trained neural network is an LSTM recurrent neural network, and the training process is as follows:

[0014] Multiple users perform the primary action and the secondary action multiple times, and the corresponding flexible pressure sensor data and acceleration sensor data of each time when the multiple users perform the primary action and the secondary action are collected as a standardized data set;

[0015] Each group of standardized data is filtered, and the flexible pressure sensor data is converted into a numerical value to obtain pressure value data;

[0016] All standardized data sets and action features contained in each standardized data set are used as training samples to train the LSTM recurrent neural network, and model parameters of the LSTM recurrent neural network are iteratively adjusted according to a training result.

[0017] In a possible implementation, the trained neural network is an SVM classification network, and the training process is as follows:

[0018] Multiple users perform the primary action and the secondary action multiple times, and the corresponding flexible pressure sensor data and acceleration sensor data of each time when the multiple users perform the primary action and the secondary action are collected as a standardized data set;

[0019] Each group of standardized data is filtered, and the flexible pressure sensor data is converted into a numerical value to obtain pressure value data;

[0020] A hierarchical vector machine, a one-to-one SVM classifier, and a one-to-many SVM classifier are established;

[0021] The training sample is the normalized data set about the static and motion postures, the motion features contained in the corresponding normalized data set, the normalized data set about the slow motion and the severe motion, and the motion features contained in the corresponding normalized data set.

[0022] The training sample is the normalized data set, and the motion features contained in each normalized data set.

[0023] The SVM classification network is obtained by fusing the hierarchical vector machine, the one-to-one SVM classifier, and the one-to-many SVM classifier.

[0024] In a possible implementation, the trained neural network is a BP neural network, and the training process is as follows:

[0025] The first-level action and the second-level action are performed by multiple users multiple times, and the corresponding flexible pressure sensor data and acceleration sensor data of each user when performing the first-level action are collected as a normalized data set.

[0026] The normalized data sets are filtered, and the numerical value of the flexible pressure sensor data is converted to obtain pressure value data.

[0027] The training sample is the normalized data set, and the motion features contained in each normalized data set.

[0028] In a possible implementation, the interaction operation on the control target object specifically includes:

[0029] If the drive end is connected to the external host through a network, after receiving the mapping action instruction from the mapping end, the drive end converts the mapping action instruction according to the simulated host input device type, to ensure that the information format of the mapping action instruction is consistent with the signal format of the host input device type.

[0030] In a possible implementation, the interaction operation on the control target object specifically includes:

[0031] If the drive end is connected to the control module of the external host, after receiving the mapping action instruction from the mapping end, the information format of the mapping action instruction is converted into a signal format allowed by the firmware of the external host.

[0032] In a possible implementation, the flexible insole further comprises a master chip and a power supply and charging module.

[0033] The acceleration sensor, the master chip, the data acquisition module, the wireless communication module and the power supply and charging module are integrated into the same box body, and the box body is placed at a heel recess of an elastic insole layer of the flexible insole.

[0034] In a possible implementation, the flexible pressure sensor is used to acquire plantar pressure and comprises at least 8 sampling channels; and the flexible pressure sensor is a sensor fitting layer with the same size as the insole and is located between a breathable layer of the insole directly contacting the skin and the elastic insole layer.

[0035] In a possible implementation, the motion analysis end can be integrated into the box body.

[0036] Compared with the prior art, the application has the following beneficial effects:

[0037] By defining an interactive mode of a small-range standing lower limb movement, the first action type and the second action type are defined by changing the center of gravity, keeping the foot still or moving in a small range, and a plurality of combination types formed by the first action type and the second action type ensure accurate division of the lower limb movement. In the system, when the user makes a corresponding lower limb movement, the flexible pressure insole can detect the pressure signal change of different channels under different movements, identify the action type through data analysis, and control the virtual character to make lower limb movements in different directions and different types through the driving end. Compared with the pressure blanket, the pressure insole has only 10-20 channels, which is much smaller than the thousands of detection points of the flexible array sensor, and has lower requirements for the hardware circuit, which can save costs. Since the pressure insole can be worn, it is not limited by the use position and can be moved more flexibly; since the activity range of the specified action is small, there is no specific site requirement;

[0038] The established neural network based on machine learning is also trained, which improves the accuracy of action recognition and realizes immersive interaction.

[0039] In addition to being applied to the motion capture game, the application can also be applied to non-motion capture games, and the driving end issues corresponding instructions (such as up, down, left and right) according to the lower limb movement of the user, to replace the keyboard, mouse and handle to control the virtual character. BRIEF DESCRIPTION OF DRAWINGS

[0040] Figure 1 FIG. 1 shows a structure schematic diagram of an interaction system based on lower limb movement according to an embodiment of the application;

[0041] Figure 2A structural schematic diagram of a flexible insole is shown in an embodiment of the present application.

[0042] Figure 3 A motion schematic diagram of a user of the insole performing a standing forward motion is shown in an embodiment of the present application.

[0043] Figure 4 A motion schematic diagram of a user of the insole performing a forward jumping motion is shown in an embodiment of the present application.

[0044] Figure 5 A motion schematic diagram of a user performing a forward squatting motion is shown in an embodiment of the present application.

[0045] Figure 6 A motion schematic diagram of a user performing a forward fast moving motion is shown in an embodiment of the present application.

[0046] Figure 7 A motion schematic diagram of a user performing a sprinting motion is shown in an embodiment of the present application.

[0047] Figure 8 A lower limb motion data curve diagram of a user standing in place is shown in an embodiment of the present application. DETAILED DESCRIPTION

[0048] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of protection of the present application.

[0049] Please refer to Figure 1 , Figure 1 An interactive system based on lower limb motion is provided in an embodiment of the present application, which comprises a flexible insole 1, a motion analysis end 2, a mapping end 3 and a driving end 4.

[0050] The flexible insole 1 comprises a flexible pressure sensor, an acceleration sensor, a data acquisition module and a wireless communication module. The flexible insole 1 senses lower limb motion through the flexible pressure sensor and the acceleration sensor, acquires lower limb motion data of the flexible pressure sensor and the acceleration sensor through the data acquisition module, and sends the lower limb motion data to the motion analysis end 2 through the wireless communication module.

[0051] The action analysis end 2 receives the lower limb movement data, and identifies lower limb action information containing primary actions and secondary actions from the lower limb movement data by using a neural network based on machine learning, and sends the lower limb action information to the mapping end 3; the primary actions include sprinting, squatting, fast moving, jumping and standing, and the secondary actions include moving forward, moving backward, moving left, moving right and staying in place.

[0052] The mapping end 3 receives the lower limb action information, and generates a mapping action instruction suitable for the driving end 4 according to the lower limb action information.

[0053] The driving end 4 receives the mapping action instruction from the mapping end 3, and interacts with the control target object.

[0054] In the process of preparing the flexible insole 1, the flexible pressure sensor used can be a piezoresistive type, a piezocapacitive type, etc. static sensor, preferably a microstructure type flexible pressure sensor, which is used to collect foot pressure and can be set to 10-20 channels, and at least 8 sampling channels are required (such as shown in the figure). When the action changes, the center of gravity moves, which will cause the foot pressure distribution to be different, and can be used to identify the action of the user. The acceleration sensor selects LISZDH12 chip, which is used to collect the signal of acceleration change, and can define positive and negative to distinguish the direction of the user's action. Figure 2

[0055] In particular, the data acquisition module is used to acquire the signal of the sensor, including a sensor signal acquisition circuit, which acquires the ADC value of multiple channels at a time, and transmits the acquired signal to the master control chip. Preferably, the signal acquisition circuit is an 8-channel ADC or less; the wireless communication module adopts an integrated wireless chip, the main body is Bluetooth, WIFI, NFC or other modules with wireless communication function, which is used to transmit sensor data to the receiving end; the master control chip is a master control chip microprocessor (nRF52832), which is used to accept the data collected by the data acquisition module, process the collected signal, and control the communication module to communicate with external devices; the power and charging module includes a battery, a battery voltage acquisition circuit and a battery charging circuit, the battery selects an (ME62107) LDO voltage conversion chip, the battery voltage acquisition circuit adopts a (HX7013) 12-bit ADC acquisition chip, and the battery charging circuit adopts a TP4057 chip.

[0056] ​In the mapping end 3, the mapping of the specified lower limb action to generate game control signals includes 21 kinds, and the virtual character is controlled to perform corresponding movements, which are divided into five categories: standing, jumping, fast moving, squatting and sprinting, and five directions: front, back, left and right (sprinting only forward), and 21 kinds of actions: standing in place, standing forward, standing backward, standing left, standing right, jumping in place, jumping forward, jumping backward, jumping left, jumping right, fast moving in place, fast moving forward, fast moving backward, fast moving left, fast moving right, squatting in place, squatting forward, squatting backward, squatting left, squatting right, and sprinting forward.

[0057] Generally, the driving end 4 can be installed on the game end, and is used to define the lower limb action signals (obtained by calling software data, cloud downloading or wireless transmission) as operations corresponding to input devices such as handles or keyboards and mice, generate game control signals, and transmit the game control signals to the game end to control the movement of the virtual character in the game.

[0058] The external host includes but is not limited to a computer, a mobile phone, a tablet, a television or other display screens with game functions.

[0059] After the required devices of the system are installed, the user wears the flexible insole 1, the flexible pressure sensor and the acceleration sensor arranged in the flexible insole 1 generate signal changes, the signals are collected by the data acquisition module and processed by the main control chip, transmitted to the receiving end through the wireless communication module, the signal is interpreted by the action analysis end 2, the lower limb action of the user is recognized, the driving end 4 simulates the input devices such as handles or keyboards, generates corresponding game control signals from the lower limb action signals, and transmits the game control signals to the game display end to control the movement direction of the virtual character.

[0060] The embodiment of the application prepares a pressure insole with plantar pressure sensing function based on a flexible pressure sensor, and specifies a set of interactive methods of standing type lower limb movement with small range movement, and defines the action type by changing the center of gravity of the user, keeping the foot still or moving in a small range. In the system, when the user makes corresponding lower limb actions, the pressure insole can detect the pressure signal change of different paths under different actions, identify the action type through data analysis, and control the virtual character to perform lower limb movements in different directions and different types through the feedback device. Compared with the pressure blanket, the pressure insole has only 10-20 channels, which is much smaller than the thousands of detection points on the flexible array sensor, and has lower demand for hardware circuit, which can save cost. Since the pressure insole can be worn and is not limited by the use position, it can be moved more flexibly; since the activity range of the specified action is required to be small, there is no specific site requirement; the accuracy of action recognition is improved through the algorithm, and immersive interaction with the external host is realized.

[0061] Exemplarily, the lower limb movement data is identified to contain first-level action and second-level action information by using a neural network based on machine learning, specifically including:

[0062] Action feature extraction is performed on the lower limb movement data; the action features include mean standard deviation, combined acceleration, foot average pressure, and acceleration DCT processing value;

[0063] In combination with the action feature extraction result, the lower limb movement data is identified by using the trained neural network to obtain lower limb action information containing first-level action and second-level action.

[0064] The trained neural network is preset in the action analysis end 2. Optionally, the neural network includes a long short term (LSTM) recurrent neural network, a support vector machine (SVM) classifier model, and a back-propagation (BP) neural network model.

[0065] The data feature extraction adopts a long sequence segment division algorithm, taking foot lifting and foot landing as the start and end of single-class feature action during human body movement, and dividing the entire action sequence into different small segments according to the data features of the foot average pressure.

[0066] Feature data acquisition: The original data is converted to extract action features, aiming to strengthen the embodiment and output of single-class action features. The embodiment of the present application requires the support of multiple action features, especially the following four:

[0067] Action feature one: mean standard deviation, used to identify static action and dynamic action. According to the mean standard deviation data waveform of the accelerometer presented by the human body when doing static action and dynamic action, it is used as the identification basis of the two types of action.

[0068] Action feature two: combined acceleration, used to identify slow motion and fast motion. The combined acceleration of the human body is an important feature of the human body movement mode, which can effectively reflect the intensity of human body movement.

[0069] Action feature three: foot average pressure, which can be used to identify small foot pressure and normal pressure action, such as sitting and standing (the currently collected actions are all standing type actions, so this feature is used as a backup)

[0070] Action feature four: the data after the discrete cosine transform (DCT) processing of the sliced acceleration has energy concentration characteristics. Many signals in life have energy concentrated in the low frequency part after DCT, so DCT is widely used in real signal processing. The embodiment of the application takes this part of the feature as one of the recognitions, which can improve the recognition effect.

[0071] As a preferred neural network, the trained neural network is an LSTM recurrent neural network, and the training process is as follows:

[0072] Let multiple users perform the first-level action and the second-level action multiple times, and collect the corresponding flexible pressure sensor data and acceleration sensor data of each time when the multiple users perform the first-level action and the second-level action as a regular data set;

[0073] Filter each group of regular data, and perform numerical conversion on the flexible pressure sensor data to obtain pressure value data;

[0074] Take all the regular data sets and the action features contained in each regular data set as training samples to train the LSTM recurrent neural network, and iteratively adjust the model parameters of the LSTM recurrent neural network according to the training result.

[0075] Let multiple users perform the first-level action and the second-level action multiple times and collect lower limb movement data at this time, for example, let the user perform a forward movement as shown in Figure 3 , which includes the first-level action of standing and the second-level action of moving forward; let the user perform a forward jump as shown in Figure 4 , which includes the first-level action of jumping and the second-level action of moving forward; let the user perform a forward squat as shown in Figure 5 , which includes the first-level action of squatting and the second-level action of moving forward; let the user perform a forward rapid movement as shown in Figure 6 , which includes the first-level action of rapid movement and the second-level action of moving forward; let the user perform a sprint as shown in Figure 7 , which includes the first-level action of sprinting. Figure 3 The action shown in -7 includes two action periods, which go through the process of standing in place - designated action - resetting - designated action - standing in place, and mainly focuses on moving the feet and changing the center of gravity.

[0076] Taking the forward movement as an example (attached Figure 2 -7):

[0077] (1) Stand forward, take a small step with the left foot, tilt forward to change the center of gravity, and repeat the action;

[0078] (2) Squat forward, squat down a little, take a small step with your left foot, lean forward to change your center of gravity, and repeat the action;

[0079] (3) Move forward quickly, step out with your left foot at a faster speed, tilt your body to change your center of gravity, and then return to a standing position. Repeat the movement.

[0080] (4) Jump forward, step your left foot forward slightly as you jump, then shift your weight and return to a standing position. Repeat the movement.

[0081] (5) Sprint, take a big step with your left foot, lean your body to change your center of gravity, and then stand up again. Repeat the action.

[0082] After collecting motion data from multiple users performing actions repeatedly, the data needs to be organized to obtain primary motion data and secondary motion data.

[0083] Taking Level 1 motion data as an example, the collected insole data is categorized and integrated, as shown in Table 1:

[0084] Table 1 Classification of Level 1 Motion Data

[0085]

[0086] Sprinting only includes one type of forward sprinting action; crouching, quick movement, jumping, and standing each include five secondary actions: forward, backward, left, right, and standing.

[0087] Data for each set of normalized data was extracted starting from 1500 frames after the initial export of the raw lower limb motion data. The aim was to avoid some preparatory movements that precede the corresponding actions.

[0088] The exported raw data is filtered to remove noise and artifacts. For example... Figure 8 As shown, Figure 8 The original lower limb motion data for standing postures before filtering is displayed.

[0089] Then, the filtered data values ​​are transformed. The ADC values ​​of the original data are converted into pressure values.

[0090] When transforming the filtered data, the following points should be noted:

[0091] (1) The ADC data output by the pressure sensor used in this study has the following characteristics: the greater the pressure, the smaller the ADC value; the smaller the pressure, the larger the ADC value. Therefore, the data needs to be reversed. Optionally, the selected value for this part can be "4096".

[0092] (2) The output raw data ADC value is converted into a specific formula. At the same time, it is approximately believed that The pressure value is proportional to the pressure value Pa. Thus, the pressure value of each sensor is derived.

[0093] Note: R f R is the resistance value in the sensor.

[0094] (3) Note the size of each pressure sensor in the insole and the average area value of the entire insole.

[0095] A large amount of sensor data specifying lower limb movements is continuously accumulated to form a lower limb movement library. Then, an LSTM neural network model improved by RNN (Recurrent Neural Networks) neural network is constructed, through continuous forgetting, remembering and updating of the lower limb movement library information, the effective information in the input sequence data can be effectively extracted and trained. The feature vector of the sensor is extracted, through the training of a large number of samples in the lower limb movement library, the effective feature information is extracted and the corresponding rule is derived, so that the input sensor information can recognize the lower limb movement.

[0096] It should be noted that (1) sequence data refers to data sequences composed of each frame of data recorded and collected after data preprocessing. The sequence length is the frame amount, and the width is the feature of each dimension.

[0097] (2) Information is effective or not: LSTM updates or forgets various data information in the recorded pressure data and accelerometer data. That is, the features with high frequency in the recorded data sequence are forgotten, and the features with low frequency are forgotten. This is achieved through the input gate, output gate and forget gate in the LSTM network. At the same time, through selective memory or forgetting of the information in the data, the utilization rate of the data information can be maximized.

[0098] (3) Feature vector refers to multiple data features in each frame of data, such as mean standard deviation, total acceleration, foot average pressure, and data after discrete cosine transform (DCT) of acceleration data.

[0099] (4) Derivation of the corresponding rule refers to multiple training of the model through continuous input data, and continuously iterating out the optimal parameters to realize the model with the highest output recognition accuracy.

[0100] As a preferred neural network, the trained neural network is an SVM classification network.

[0101] It should be noted that the support vector machine (SVM) is a binary classification model, and its basic model is a linear classifier with maximum interval defined in a feature space, which is different from the perceptron; the SVM also includes a kernel trick, which makes it a substantial nonlinear classifier. The learning strategy of the SVM is to maximize the interval, which can be formalized as a convex quadratic programming problem, and is equivalent to the minimization problem of the regularized hinge loss function. The learning algorithm of the SVM is an optimization algorithm for solving the convex quadratic programming.

[0102] The training process of the SVM classification network is as follows:

[0103] Let multiple users perform the primary action and the secondary action multiple times, and collect the corresponding flexible pressure sensor data and acceleration sensor data of each time when the multiple users perform the primary action and the secondary action as a normalized data set;

[0104] Filter each group of normalized data, and perform numerical conversion on the flexible pressure sensor data to obtain pressure value data;

[0105] Establish a hierarchical support vector machine, a one-to-one SVM classifier, and a one-to-many SVM classifier;

[0106] Take the normalized data set about the static and dynamic postures and the action features contained in the corresponding normalized data set, and the normalized data set about the slow motion and the intense motion and the action features contained in the corresponding normalized data set as training samples, train the hierarchical support vector machine, and iteratively adjust the model parameters of the hierarchical support vector machine according to the training result;

[0107] Take all the normalized data sets and the action features contained in each normalized data set as training samples, train the one-to-one SVM classifier and the one-to-many SVM classifier, and iteratively adjust the model parameters of the one-to-one SVM classifier and the one-to-many SVM classifier according to the training result;

[0108] Fuse the hierarchical support vector machine, the one-to-one SVM classifier, and the one-to-many SVM classifier to obtain an SVM classification network.

[0109] In this embodiment, a hierarchical support vector machine (H-SVM), a one-to-one SVM classifier, and a one-to-many SVM classifier are used. The feature data output by the flexible insole 1 is trained and recognized.

[0110] In the hierarchical support vector machine (H-SVM), the static and dynamic postures and the slow motion and the intense motion are recognized and classified. The one-to-one SVM classifier and the one-to-many SVM classifier train and recognize all the lower limb actions recorded. Finally, the three types of SVM classifiers are fused to maximize the recognition accuracy.

[0111] As a preferred neural network, the trained neural network is a BP neural network, and the training process is as follows:

[0112] Let multiple users perform the first-level action and the second-level action multiple times, and collect the corresponding flexible pressure sensor data and acceleration sensor data of each user when performing the first-level action as a regular data set;

[0113] Filter each group of regular data, and perform numerical conversion on the flexible pressure sensor data to obtain pressure value data;

[0114] Take all regular data sets and the action features contained in each regular data set as training samples to train the BP neural network, and adjust the neuron parameters of the hidden layer according to the mean square error of the network after each training.

[0115] The BP neural network has the characteristics of simple model structure and strong nonlinear mapping ability, and the human body static posture sequence has the characteristic of single sequence feature, so the BP network model has good recognition performance in static posture recognition. We will build a 3-layer BP neural network model to recognize the five actions of sprinting, squatting, fast moving, jumping and standing.

[0116] The algorithm steps are as follows:

[0117] 1) Input the training sample, which includes: sample sequence data composed of features such as average standard deviation of acceleration, combined acceleration, and average pressure value.

[0118] 2) Calculate the input value A of each neuron in the hidden layer i , the calculation formula (1) is as follows,

[0119]

[0120] where w ij is the connection weight value between hidden layer neuron i and input layer neuron j. δ i is the threshold value of the hidden layer neuron.

[0121] 3) Substitute the corresponding input A of the hidden layer i into formula (2) to calculate the output value B of each neuron in the hidden layer i , and calculate the output y of a single neuron through formula (3),

[0122]

[0123]

[0124] where C iConnection weights of hidden layer neurons and output layer neurons.

[0125] 4) Calculate the mean square error E(w) of the network, and the calculation formula (4) is as follows,

[0126]

[0127] Wherein, y n is the expected output value, is the actual output value, and s is the number of training samples.

[0128] 5) Train the BP network model, and input the corresponding test sequence to verify the recognition result.

[0129] Exemplarily, the interactive operation on the control target object specifically comprises:

[0130] If the driving end 4 is connected with the external host through a network, after receiving the mapping action instruction from the mapping end 3, the driving end 4 converts the mapping action instruction according to the simulated host input device type, so as to ensure that the information format of the mapping action instruction is consistent with the signal format of the host input device type.

[0131] In the embodiment, the driving end 4 is independent of the external host, and helps the external host to realize the somatosensory interaction function by simulating the external host input device signal.

[0132] Exemplarily, the interactive operation on the control target object specifically comprises:

[0133] If the driving end 4 is connected with the control module of the external host, after receiving the mapping action instruction from the mapping end 3, the information format of the mapping action instruction is converted into a signal format allowed by the firmware of the external host.

[0134] In the embodiment, the driving end 4 is built in the external host, and the interactive system provided by the external host in the embodiment is a complete somatosensory interaction device.

[0135] Exemplarily, the flexible insole 1 further comprises a master control chip and a power supply and charging module.

[0136] The acceleration sensor, the master control chip, the data acquisition module, the wireless communication module and the power supply and charging module are integrated into the same box body, and the box body is placed in the heel recess of the elastic insole layer of the flexible insole 1.

[0137] Exemplarily, the flexible pressure sensor is used to collect foot bottom pressure, and includes at least 8 sampling channels; the flexible pressure sensor is a sensor fitting layer with the same size as the insole, and is located between the breathable layer of the insole directly contacting the skin and the elastic insole layer.

[0138] Exemplarily, the motion analysis end 2 can be integrated in the box body.

[0139] The motion analysis end 2 is a signal processing software (PC end or can be integrated into the insole), which is used for receiving and interpreting signals collected by the master control chip, and identifying lower limb motions of the user based on a feature algorithm of machine learning.

[0140] The embodiment of the application defines a set of interactive modes of standing type lower limb motions with small range movement, defines first motion types and second motion types through center of gravity change, foot in-place movement or small range movement, and guarantees accurate division of lower limb motions through various combination types formed by the first motion types and the second motion types. In the system, when the user makes corresponding lower limb motions, the flexible pressure insole can detect pressure signal change conditions of different channels under different motions, identify motion types through data analysis, and control the virtual character to make lower limb motions in different directions and different types through the driving end 4. Compared with the pressure blanket, the pressure insole has only 10-20 channels, which is much smaller than the thousands of detection points of the flexible array sensor, and has lower requirements for the hardware circuit, thereby saving costs. Since the pressure insole can be worn and is not limited by the use position, it can be moved more flexibly; since the activity range of the specified motion is required to be small, there is no specific site requirement;

[0141] The established neural network based on machine learning is also trained, thereby improving the accuracy of motion recognition and realizing immersive interaction.

[0142] The embodiment of the application can be applied to non-motion sensing games in addition to motion sensing games, and the driving end 4 issues corresponding instructions (such as up, down, left and right) according to lower limb motions of the user, thereby replacing the keyboard, mouse and handle to control the virtual character.

[0143] The above-described specific embodiments further specifically describe the purposes, technical solutions and beneficial effects of the application, and it should be understood that the above-described specific embodiments are only specific embodiments of the application and are not used to limit the protection scope of the application. It is particularly pointed out that any modification, equivalent replacement, improvement, etc. made by those skilled in the art within the spirit and principles of the application should be included in the protection scope of the application.

Claims

1. An interactive system based on lower limb movement, characterized in that, include: Flexible insole, motion analysis terminal, mapping terminal, and driving terminal; The flexible insole includes a flexible pressure sensor, an acceleration sensor, a data acquisition module, and a wireless communication module; The flexible insole senses lower limb movement through the flexible pressure sensor and the accelerometer, collects lower limb movement data from the flexible pressure sensor and the accelerometer through the data acquisition module, and sends the lower limb movement data to the motion analysis terminal through the wireless communication module. The motion analysis terminal receives the lower limb motion data and uses a machine learning-based neural network to identify lower limb motion information, including primary and secondary movements, from the lower limb motion data, and sends the lower limb motion information to the mapping terminal; the primary movements include sprinting, squatting, fast movement, jumping, and standing, and the secondary movements include forward, backward, left, right, and standing still; The mapping terminal receives the lower limb movement information and generates a mapping movement command suitable for the driving terminal based on the lower limb movement information; After receiving the mapping action instruction from the mapping end, the driving end performs interactive operations on the controlled target object; The machine learning-based neural network is an SVM classification network, and the training process is as follows: Multiple users are allowed to perform primary and secondary actions multiple times, and flexible pressure sensor data and acceleration sensor data are collected from each user's primary and secondary actions as a regularized data set. The data from each set of regular data is filtered, and the data from the flexible pressure sensor is converted into numerical values ​​to obtain pressure data. Establish hierarchical vector machines, one-to-one SVM classifiers, and one-to-many SVM classifiers; The hierarchical vector machine is trained using regularized data sets on static and dynamic postures and corresponding motion features, as well as regularized data sets on slow and vigorous motion and corresponding motion features, as training samples. The model parameters of the hierarchical vector machine are then iteratively adjusted based on the training results. The entire regularized data set and the action features contained in each regularized data set are used as training samples to train the one-to-one SVM classifier and the one-to-many SVM classifier. The model parameters of the one-to-one SVM classifier and the one-to-many SVM classifier are iteratively adjusted according to the training results. The hierarchical vector machine, the one-to-one SVM classifier, and the one-to-many SVM classifier are fused to obtain an SVM classification network.

2. The interactive system based on lower limb movement as described in claim 1, characterized in that, The process of identifying lower limb movement information, including primary and secondary movements, from the lower limb movement data using a machine learning-based neural network specifically includes: Motion features are extracted from the lower limb motion data; the motion features include mean standard deviation, resultant acceleration, mean plantar pressure, and DCT-processed acceleration values. By combining the results of motion feature extraction, the trained neural network is used to identify the lower limb motion data, and lower limb motion information including primary and secondary movements is obtained.

3. The interactive system based on lower limb movement as described in claim 2, characterized in that, The trained neural network is an LSTM recurrent neural network, and the training process is as follows: Multiple users are allowed to perform primary and secondary actions multiple times, and flexible pressure sensor data and acceleration sensor data are collected from each user's primary and secondary actions as a regularized data set. The data from each set of regular data is filtered, and the data from the flexible pressure sensor is converted into numerical values ​​to obtain pressure data. The LSTM recurrent neural network is trained using all regularized data sets and the action features contained in each regularized data set as training samples, and the model parameters of the LSTM recurrent neural network are iteratively adjusted based on the training results.

4. The interactive system based on lower limb movement as described in claim 2, characterized in that, The trained neural network is a backpropagation (BP) neural network, and the training process is as follows: Multiple users are allowed to perform primary and secondary actions multiple times, and flexible pressure sensor data and acceleration sensor data are collected from each user when performing the primary action as a regularized data set. The data from each set of regular data is filtered, and the data from the flexible pressure sensor is converted into numerical values ​​to obtain pressure data. The entire set of regularized data and the action features contained in each set of regularized data were used as training samples to train the BP neural network. After each training session, the neuron parameters of the hidden layer were adjusted according to the mean square error of the network.

5. The interactive system based on lower limb movement as described in claim 1, characterized in that, The interactive operation on the controlled target object specifically includes: If the driver is connected to an external host via a network, after receiving the mapping action command from the mapping terminal, the driver converts the format of the mapping action command according to the simulated host input device type to ensure that the information format of the mapping action command is consistent with the signal format of the host input device type.

6. The interactive system based on lower limb movement as described in claim 1, characterized in that, The interactive operation on the controlled target object specifically includes: If the driver is connected to the control module of an external host, after receiving the mapping action command from the mapping terminal, it converts the information format of the mapping action command into a signal format allowed by the external host firmware.

7. The interactive system based on lower limb movement as described in claim 1, characterized in that, The flexible insole also includes a main control chip and a power supply and charging module; The accelerometer, the main control chip, the data acquisition module, the wireless communication module, and the power supply and charging module are integrated into the same box, which is placed in the heel recess of the elastic insole layer of the flexible insole.

8. The interactive system based on lower limb movement as described in claim 1, characterized in that, The flexible pressure sensor is used to collect foot pressure and includes at least 8 sampling channels; the flexible pressure sensor is a sensor bonding layer of the same size as the insole, located between the breathable layer of the insole and the elastic insole layer that are in direct contact with the skin.

9. The interactive system based on lower limb movement as described in claim 7, characterized in that, The motion analysis terminal is integrated into the housing.

Images