Identity recognition method and system based on multi-modal multi-task learning

Abstract

The invention provides an identity recognition method and system based on multi-mode and multi-task learning, and belongs to the technical field of identity recognition. The method comprises the following steps: acquiring motion data of a tester under different motion behaviors; preprocessing the motion data, and constructing a gait event detection task label and an identity recognition task label; recombining the constructed task label and the motion data to obtain a data set; inputting data in the data set into a trained CNN-LSTM network model containing an attention mechanism, and performing weight learning on time and space dimensions through the attention mechanism so as to obtain feature representation with weight; learning is carried out through a CNN layer and an LSTM network layer, and feature representation after dimension change is output; and outputting the gait characteristics of the testee and the corresponding identity information by using the two full-connection networks. The problem that the performance is poor due to the fact that only single modal information is utilized in the prior art is solved. Compared with a traditional method for realizing feature extraction depending on manpower, the workload is greatly reduced.

Description

Technical Field

[0001] The present invention belongs to the technical field of identity recognition, and in particular relates to an identity recognition method and system based on multi-modal multi-task learning. Background Art

[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

[0003] Identity recognition is one of the most popular research directions in recent years, among which fingerprint recognition, face recognition and other technologies have been widely used in various fields. In addition, there are many studies on identity recognition using other physiological characteristics, and gait characteristics are one of the important branches.

[0004] At present, most of the research on identity recognition using gait features is based on image information. However, image information has disadvantages such as environmental limitations, high computational cost, and poor real-time performance. With the maturity of micro-electromechanical system (MEMS) technology, inertial sensors have attracted widespread attention due to their advantages such as portability and wearability and freedom from environmental constraints. At present, most of the research based on inertial sensor data relies on manual feature extraction, and then the appropriate threshold is designed based on statistical methods for learning, which makes its generalization performance and robustness poor. With the superior performance of deep learning technology in various fields, some research is also based on deep learning methods. However, whether based on image information or inertial sensors, only single modality information (kinematic information) of personnel can be obtained, which reduces the performance of the task to a certain extent. Summary of the invention

[0005] In order to overcome the shortcomings of the above-mentioned prior art, the present invention provides an identity recognition method and system based on multimodal multi-task learning, wherein the plantar pressure sensor and the inertial sensor are respectively fixed on the shoe, the two modal information obtained are preprocessed, and then multi-task learning is performed based on the time series model, specifically referring to gait event detection and identity recognition tasks. On the one hand, it reduces the tediousness of relying on manual feature extraction and provides an end-to-end model; on the other hand, through the combination of multimodal and multi-task learning, the accuracy and robustness of identity recognition are greatly improved.

[0006] To achieve the above objectives, one or more embodiments of the present invention provide the following technical solutions:

[0007] The first aspect of the present invention provides an identity recognition method based on multimodal multi-task learning;

[0008] An identity recognition method based on multimodal multi-task learning, comprising:

[0009] Obtain the exercise data of the tester under different exercise behaviors;

[0010] Preprocessing the motion data to construct gait event detection task labels and identity recognition task labels;

[0011] Recombine the constructed task labels and motion data to obtain the data set;

[0012] Input the data in the dataset into the trained CNN-LSTM network model with attention mechanism, and output the gait characteristics and corresponding identity information of the tester;

[0013] The CNN-LSTM network model with attention mechanism uses the attention mechanism to learn the weights of time and space dimensions to obtain weighted feature representations; then the weighted feature representations are learned through the CNN layer to capture higher-level semantic information; after two layers of LSTM networks, long-term information is learned to further obtain high-level semantic information; two fully connected networks are used to map the learned feature representations into outputs of different dimensions, respectively outputting the tester's gait characteristics and corresponding identity information.

[0014] As a further technical solution, the movement behaviors include five kinds of movement behaviors: normal walking on flat ground, slow walking on flat ground, fast walking on flat ground, running on flat ground, and going up and down stairs.

[0015] As a further technical solution, obtaining the motion data of the tester under different motion behaviors includes:

[0016] Use inertial sensors to obtain 3D acceleration, 3D angular velocity and 3D magnetic field data under different motion behaviors;

[0017] Use pressure sensors to obtain pressure data under different sports behaviors;

[0018] Using three-dimensional acceleration, three-dimensional angular velocity and three-dimensional magnetic field data, posture solution is performed based on quaternion error Kalman filtering to obtain the three-dimensional posture information of the tester; the three-dimensional posture information is spliced ​​with the three-dimensional acceleration, three-dimensional angular velocity, three-dimensional magnetic field data and pressure data to obtain the motion data of the tester under different motion behaviors.

[0019] As a further technical solution, the process of preprocessing the motion data includes:

[0020] The motion data is segmented using a window of preset length to construct a motion data sample set;

[0021] The samples in the motion data sample set are labeled to construct gait event detection task labels and identity recognition task labels; the gait event detection task label is a one-hot encoding vector with a dimension of 5, and the position of element 1 corresponds to the category; the identity recognition task label is a one-hot vector with a dimension of N, and the position of element 1 corresponds to the index position of the tester.

[0022] As a further technical solution, the constructed task labels and motion data are reorganized, and the process of obtaining the data set also includes:

[0023] The preprocessed data is randomly disrupted in its original order and then packaged according to the batch size; the final data set is divided into a training set and a test set according to a preset ratio; the training set is further divided into a training set and a validation set according to a preset ratio.

[0024] As a further technical solution, the CNN-LSTM network model with attention mechanism performs weight learning on time and space dimensions through the attention mechanism, so as to obtain the process of feature representation with weights, including:

[0025] Change the dimension of the input data;

[0026] Input the dimensionally changed data into the first self-attention mechanism, obtain the feature weights on the spatial dimension, and iterate according to the batch size;

[0027] Use one-dimensional convolution to reduce the dimension of each output to obtain the reduced dimension features;

[0028] The reduced-dimensional features are concatenated and input into the second self-attention mechanism to obtain the feature weights in the time dimension, and finally a feature representation with weights in the time and space dimensions is obtained.

[0029] A second aspect of the present invention provides an identity recognition system based on multimodal multi-task learning.

[0030] An identity recognition system based on multimodal multi-task learning, comprising:

[0031] The data acquisition module is configured to: acquire the motion data of the tester under different motion behaviors;

[0032] A preprocessing module is configured to: preprocess the motion data to construct a gait event detection task label and an identity recognition task label;

[0033] The data set acquisition module is configured to: reorganize the constructed task labels and motion data to obtain the data set;

[0034] The gait feature and identity information recognition module is configured to: input the data in the dataset into the trained CNN-LSTM network model with attention mechanism, and output the gait features and corresponding identity information of the tester.

[0035] The third aspect of the present invention provides a computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the steps in an identity recognition method based on multimodal multi-task learning as described in the first aspect of the present invention.

[0036] The fourth aspect of the present invention provides an electronic device, comprising a memory, a processor, and a program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of an identity recognition method based on multimodal multi-task learning as described in the first aspect of the present invention are implemented.

[0037] One or more of the above technical solutions have the following beneficial effects:

[0038] (1) Based on inertial sensors and plantar pressure sensors, the present invention integrates two modal data and designs an attention mechanism that integrates spatial and temporal information. Different weights are assigned to features in the spatial and temporal dimensions, thereby obtaining higher-level semantic information and ultimately achieving gait event detection tasks and identity recognition tasks. In the above process, kinematic information and dynamic information are fully utilized. This solves the problem of poor performance caused by the prior art that only uses single modal information.

[0039] (2) The present invention combines the gait event detection task with the identity recognition task and proposes a multi-task learning framework that utilizes the correlation between different tasks to further improve the performance of each task. Compared with the traditional feature extraction method that relies on manual implementation, the present invention provides an end-to-end model that greatly reduces the workload.

[0040] Advantages of additional aspects of the present invention will be given in part in the following description, and in part will become obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0042] Figure 1 This is a flow chart of the method of the first embodiment.

[0043] Figure 2 It is a schematic diagram of the single-sided arrangement of the inertial sensor and the pressure sensor in the first embodiment.

[0044] Figure 3 Schematic diagram of the structure of the CNN-LSTM network model with attention mechanism in the first embodiment.

[0045] Figure 4 It is a system structure diagram of the second embodiment. DETAILED DESCRIPTION

[0046] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.

[0047] It should be noted that the terms used herein are for describing specific embodiments only and are not intended to be limiting of exemplary embodiments according to the present invention.

[0048] In the absence of conflict, the embodiments of the present invention and the features of the embodiments may be combined with each other.

[0049] Embodiment 1

[0050] This embodiment discloses an identity recognition method based on multi-modal multi-task learning;

[0051] like Figure 1 As shown, an identity recognition method based on multimodal multi-task learning includes:

[0052] Step S1, obtaining the exercise data of the tester under different exercise behaviors;

[0053] Combination Figure 2 The tester wears a device consisting of 6 inertial sensors and 2 plantar pressure sensors. The 6 inertial sensors are installed on the heels, insteps and toes of the left and right feet. Each sensor contains a three-axis accelerometer, a three-axis gyroscope and a three-axis magnetometer; the two plantar pressure sensors act as insoles, and each plantar pressure sensor contains 8 measurement points, including 3 on the heel and 5 on the forefoot.

[0054] The test subjects were asked to complete five behaviors: normal walking on flat ground, slow walking on flat ground, fast walking on flat ground, running on flat ground, and going up and down stairs. Each behavior lasted for 3 minutes and was repeated 20 times.

[0055] The data obtained for each behavior include 6 sets of 3D acceleration, 3D angular velocity and 3D magnetic field, totaling 54-dimensional data, and 2 sets of plantar pressure sensor data, totaling 16-dimensional data. By splicing, a set of data with 70 dimensions can be obtained.

[0056] Using three-dimensional acceleration, three-dimensional angular velocity and three-dimensional magnetic field, attitude solution is performed based on quaternion error Kalman filtering. The specific steps are as follows:

[0057] (1) Attitude angle initialization

[0058] At the initial moment, the pitch angle φ and roll angle θ are calculated using acceleration data, and the heading angle is calculated using the magnetometer In this embodiment, the northeast sky is used as the reference coordinate system, and the rotation order is specified as ZYX. When stationary, the acceleration only has a component in the vertical direction, so

[0059]

[0060] where a x Accelerometer X-axis data, the subscript is the direction of the axis. is the rotation matrix from the sensor coordinate system to the reference coordinate system, and g is the value of gravity acceleration. Further, we get:

[0061] φ=arctan(a y ,a z )(2)

[0062]

[0063] There is the following relationship between the geomagnetic distribution and magnetometer data:

[0064]

[0065] Where m is the magnetometer reading, the subscript is the coordinate axis, and b is the geomagnetic component.

[0066] Further we can get:

[0067]

[0068]

[0069] Simplifying, we get:

[0070]

[0071] What is required is the declination angle relative to the magnetic north. There is a magnetic declination angle with the geographic north. (North-west is positive, the opposite is negative). Compensating it, the heading angle is:

[0072]

[0073] Convert the above Euler angles into quaternion representation:

[0074]

[0075] (2) Kalman Update Process

[0076] The entire Kalman process can be divided into a prediction phase and an update phase. Taking the error quaternion as the state quantity, the prediction phase can be expressed as follows:

[0077]

[0078] in, is the state quantity at time t, the negative superscript indicates the prior, the positive superscript indicates the posterior, Φ is the state transfer matrix, P is the state error covariance matrix, and Q is the process noise covariance matrix.

[0079] The update phase first calculates the Kalman gain:

[0080]

[0081] Among them, H is the observation matrix, R is the observation noise covariance matrix;

[0082]

[0083] in, is the measured value.

[0084] After obtaining the error quaternion, update the original quaternion:

[0085]

[0086] After attitude solution based on quaternion error Kalman filtering, 6 sets of 3D attitude information with a total of 18-dimensional data are obtained. This data is spliced ​​with the 70-dimensional data obtained in the previous step, and finally the model input data with a dimension of 88 at each moment is obtained, x = {x1, x2, …, x 88},x∈R 88 .

[0087] Step S2, preprocessing the motion data to construct gait event detection task labels and identity recognition task labels;

[0088] The data is segmented using a window of size 100, with a step size of 25, to construct a sample set, and each sample is X = {x1, x2, ..., x 100},X∈R 100×88 .

[0089] The segmented data is labeled. First, the labels are constructed. For the gait event detection task, the corresponding label relationship is {“heel touchdown”: 1, “toe touchdown”: 2, “heel off”: 3, “toe off”: 4, “other”: 5}; for the identity recognition task, if there are N testers, the corresponding labels are from 1 to N, and all sample data collected by the same person correspond to the same label. Finally, the label of each sample in the gait event detection task is a one-hot encoding vector with a dimension of 5. The position of element 1 corresponds to the category, that is, the corresponding vector of “heel touchdown” is y={1 0 0 0 0}, y∈R 5 ; Similarly, the label corresponding to the identity recognition task is a one-hot vector with a dimension of N, and the position of element 1 corresponds to the person index position.

[0090] Step S3, recombining the constructed task labels and motion data to obtain a data set;

[0091] The sample data and the label together constitute a group of data. To facilitate network training, multiple groups of data are grouped into a batch. For model training, the present invention uses a batch size of 32, that is, batch_size=32. For the gait event detection task, one batch corresponds to 32 samples, and each sample has a corresponding label; that is, the input data dimension is 32×100×88, and the output (label) dimension is 32×5; for the identity recognition task, one batch contains 32 samples, but only corresponds to one label. That is, the input data dimension is 32×100×88, and the output (label) dimension is 1×N. Therefore, when constructing a data set, it is necessary to ensure that the data in the same batch belongs to the same person.

[0092] To improve the robustness of the model, for the data collected by the same person, after data segmentation and labeling, the original order is randomly disrupted, and then packaged according to the batch size. The final data set is divided into a training set and a test set in a 4:1 ratio, and the training set is further divided into a 9:1 ratio to obtain a training set and a validation set.

[0093] Step S4, input the data in the data set into the trained CNN-LSTM network model with attention mechanism, and output the gait characteristics and corresponding identity information of the tester;

[0094] Combination Figure 3For the gait event detection task, different features of the same data contribute differently to the gait event detection. For example, for the heel-landing event, the pressure data at the heel position is obviously more important, while for the toe-landing event, the pressure data at the sole position plays a more critical role. Similarly, the same set of data contributes differently to the result at different times. Based on this, in this embodiment, the input data is first dimensionalized, from B×L×88 to B×L×88×1, where B is the batch size and L is the length. L×88×1 is sent to the first self-attention mechanism to obtain an output of L×88×d1, where d1 is the output dimension size, and iterates B times. Different weights can be assigned to features in different spatial dimensions through the first self-attention mechanism. Then, one-dimensional convolution is used to reduce the dimension of the feature with a dimension of L×88×d1, and a reduced dimension feature with an output of L×d2 is obtained, where d2 is the output dimension of the one-dimensional convolution. The results of B iterations are spliced ​​to obtain B×L×d2, which is used as the input of the second self-attention mechanism. The weight on the time dimension can be effectively obtained through the second self-attention mechanism. Therefore, through two self-attention mechanisms, we finally get the feature representation B×L×d with time and space dimension weights, where d is the output dimension size.

[0095] The CNN-LSTM network layer contains one CNN layer and two LSTM layers. The weighted feature representation B×L×d obtained in the previous step is used as the input of this layer. The CNN layer learns the features of the data, and the convolutional neural network learns the local features better to capture higher-level semantic information. Then, the two LSTM networks learn long-term information to further obtain high-level semantic information. The final output dimension is B×L×h, where h is the output dimension of the model.

[0096] For different tasks, two fully connected networks are used to map them into outputs of dimensions B×5 and 1×N, corresponding to the gait event detection task and identity recognition task, respectively. Then, the loss is calculated with the label and feedback training is performed. Through multiple iterations, the model reaches the optimal value.

[0097] Among them, the loss function is calculated as follows:

[0098] L=αL1+βL2(17)

[0099]

[0100] Where L1 and L2 are the losses calculated for the gait event detection task and the identity recognition task, respectively, and α and β are the corresponding weights. The loss function calculation method uses the cross entropy loss function, as shown in equations (18) and (19). i Corresponding to the true label of the i-th category, p i is the predicted probability of the i-th category.

[0101] Embodiment 2

[0102] This embodiment discloses an identity recognition system based on multi-modal multi-task learning;

[0103] like Figure 4 As shown, an identity recognition system based on multimodal multi-task learning includes:

[0104] The data acquisition module is configured to: acquire the motion data of the tester under different motion behaviors;

[0105] A preprocessing module is configured to: preprocess the motion data to construct a gait event detection task label and an identity recognition task label;

[0106] The data set acquisition module is configured to: reorganize the constructed task labels and motion data to obtain the data set;

[0107] The gait feature and identity information recognition module is configured to: input the data in the dataset into the trained CNN-LSTM network model with attention mechanism, and output the gait features and corresponding identity information of the tester.

[0108] Embodiment 3

[0109] The purpose of this embodiment is to provide a computer-readable storage medium.

[0110] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps in an identity recognition method based on multimodal multi-task learning as described in Example 1.

[0111] Embodiment 4

[0112] The purpose of this embodiment is to provide an electronic device.

[0113] An electronic device comprises a memory, a processor and a program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps in the identity recognition method based on multimodal multi-task learning as described in Example 1 are implemented.

[0114] The steps involved in the apparatuses of the above embodiments 2, 3 and 4 correspond to the method embodiment 1, and the specific implementation methods can refer to the relevant description part of embodiment 1. The term "computer-readable storage medium" should be understood as a single medium or multiple media including one or more instruction sets; it should also be understood to include any medium that can store, encode or carry an instruction set for execution by a processor and enable the processor to execute any method in the present invention.

[0115] Those skilled in the art should understand that the modules or steps of the present invention described above can be implemented by a general-purpose computer device, or alternatively, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. The present invention is not limited to any specific combination of hardware and software.

[0116] Although the above describes the specific implementation mode of the present invention in conjunction with the accompanying drawings, it is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art on the basis of the technical solution of the present invention without creative work are still within the scope of protection of the present invention.

Claims

1. An identity recognition method based on multimodal multi-task learning, characterized in that: include: Obtain the exercise data of the tester under different exercise behaviors; Preprocessing the motion data to construct gait event detection task labels and identity recognition task labels; Recombine the constructed task labels and motion data to obtain the data set; Input the data in the dataset into the trained CNN-LSTM network model with attention mechanism, and output the gait characteristics and corresponding identity information of the tester; The CNN-LSTM network model with attention mechanism uses the attention mechanism to learn the weights of time and space dimensions to obtain weighted feature representations; then the weighted feature representations are learned through the CNN layer to capture higher-level semantic information; after two layers of LSTM networks, long-term information is learned to further obtain high-level semantic information; finally, two fully connected networks are used to map the learned feature representations into outputs of different dimensions, respectively outputting the tester's gait characteristics and corresponding identity information.

2. The identity recognition method based on multimodal multi-task learning as claimed in claim 1, characterized in that: The movement behaviors include normal walking on flat ground, slow walking on flat ground, fast walking on flat ground, running on flat ground, and going up and down stairs.

3. The identity recognition method based on multimodal multi-task learning as claimed in claim 1, characterized in that: The exercise data obtained under different exercise behaviors of the tester include: Use inertial sensors to obtain 3D acceleration, 3D angular velocity and 3D magnetic field data under different motion behaviors; Use pressure sensors to obtain pressure data under different sports behaviors; Using three-dimensional acceleration, three-dimensional angular velocity and three-dimensional magnetic field data, posture solution is performed based on quaternion error Kalman filtering to obtain the three-dimensional posture information of the tester; the three-dimensional posture information is spliced ​​with the three-dimensional acceleration, three-dimensional angular velocity, three-dimensional magnetic field data and pressure data to obtain the motion data of the tester under different motion behaviors.

4. The identity recognition method based on multimodal multi-task learning as claimed in claim 1, characterized in that: The process of preprocessing the motion data includes: The motion data is segmented using a window of preset length to construct a motion data sample set; The samples in the motion data sample set are labeled to construct gait event detection task labels and identity recognition task labels; the gait event detection task label is a one-hot encoding vector with a dimension of 5, and the position of element 1 corresponds to the category; the identity recognition task label is a one-hot vector with a dimension of N, and the position of element 1 corresponds to the index position of the tester.

5. The identity recognition method based on multimodal multi-task learning as claimed in claim 1, characterized in that: The constructed task labels and motion data are reorganized, and the process of obtaining the data set also includes: The preprocessed data is randomly disrupted in its original order and then packaged according to the batch size; the final data set is divided into a training set and a test set according to a preset ratio; the training set is further divided into a training set and a validation set according to a preset ratio.

6. The identity recognition method based on multimodal multi-task learning as claimed in claim 1, characterized in that: The CNN-LSTM network model with attention mechanism performs weight learning on the time and space dimensions through the attention mechanism, so as to obtain the feature representation with weights, including the following process: Change the dimension of the input data; Input the dimensionally changed data into the first self-attention mechanism, obtain the feature weights on the spatial dimension, and iterate according to the batch size; Use one-dimensional convolution to reduce the dimension of each output to obtain the reduced dimension features; The reduced-dimensional features are concatenated and input into the second self-attention mechanism to obtain the feature weights in the time dimension, and finally a feature representation with weights in the time and space dimensions is obtained.

7. An identity recognition system based on multimodal multi-task learning, characterized in that: include: The data acquisition module is configured to: acquire the motion data of the tester under different motion behaviors; A preprocessing module is configured to: preprocess the motion data to construct a gait event detection task label and an identity recognition task label; The data set acquisition module is configured to: reorganize the constructed task labels and motion data to obtain the data set; The gait feature and identity information recognition module is configured to: input the data in the dataset into the trained CNN-LSTM network model with attention mechanism, and output the gait features and corresponding identity information of the tester.

8. The identity recognition system based on multi-modal multi-task learning as claimed in claim 7, characterized in that: The movement behaviors include normal walking on flat ground, slow walking on flat ground, fast walking on flat ground, running on flat ground, and going up and down stairs.

9. A computer-readable storage medium having a program stored thereon, characterized in that: When the program is executed by a processor, the steps in the identity recognition method based on multimodal multi-task learning as described in any one of claims 1 to 6 are implemented.

10. An electronic device comprising a memory, a processor, and a program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps in the identity recognition method based on multimodal multi-task learning as described in any one of claims 1 to 6 are implemented.

Images