Gesture recognition method and system based on multi-modal feature fusion and small sample learning

Abstract

The present application relates to a kind of gesture recognition method and system based on multi-modal feature fusion and small sample learning.The method comprises: obtaining source domain data and target domain data of gesture recognition;Using the source domain data from different links to train first feature extractor and second feature extractor respectively and obtain first feature and second feature, and the first feature and the second feature are input into multi-modal feature fusion to carry out feature fusion to obtain multi-modal fusion feature;Based on the training feature of the target domain data obtained based on the multi-modal feature fusion fusion, train classifier based on the training feature, and then carry out gesture recognition based on the trained classifier.The present application can carry out feature fusion to the data received by multiple pairs of transceiver devices using multi-modal feature fusion, thereby enhancing features, so using the method of small sample learning, only a small amount of target domain data needs to be collected to realize high-accuracy cross-domain gesture recognition.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of gesture recognition, and more particularly, to a gesture recognition method and system based on multi-modal feature fusion and small sample learning. BACKGROUND

[0002] As one of the core technologies of human-computer interaction, gesture recognition has a wide range of applications in the field of human-computer interaction such as smart home and VR. Traditional gesture recognition research methods mainly include vision-based gesture recognition technology and sensor-based gesture recognition technology. Although these two methods can achieve high estimation accuracy, they have their own shortcomings.

[0003] Vision-based gesture recognition technology does not require users to carry additional devices, so it can collect the required data without disturbing the user. Therefore, from the perspective of gesture sample collection, this method is more convenient than the sensor-based method and is more conducive to long-term data collection. However, when the environment is in non-line-of-sight conditions and has similar backgrounds, the data collection will be greatly disturbed. At the same time, visual recognition of gestures may infringe on the privacy of users and is not suitable for some special occasions (such as bathrooms or bedrooms).

[0004] Sensor-based gesture recognition requires the target to carry a device equipped with a sensor at all times to collect data, and its characteristics are small amount of data collected, fast transmission speed. The collected data contains information such as finger position, motion direction, and wrist bending degree, so it can recognize various hand movements in real time. In addition, this method does not rely on environmental conditions such as lighting when collecting data, and has very good anti-interference performance. However, this method requires users to wear related devices at all times, which is very inconvenient for users and is not realistic. At the same time, such devices are usually expensive and difficult to deploy on a large scale.

[0005] Compared with the above two methods, WiFi-based gesture recognition technology has low equipment cost, and because almost every household has WiFi equipment today, it does not require users to carry collection devices at all times, causing inconvenience in use. At the same time, unlike cameras, WiFi signals are not affected by factors such as line of sight when working, and do not cause privacy leaks.

[0006] The data form used by the gesture recognition based on WiFi mainly includes received signal strength indication (RSSI) and channel state information (CSI). The RSSI represents the strength information of the received signal in a period of time, and reflects the total amplitude of the multipath superposition, and belongs to coarse-grained information. Compared with the RSSI, the CSI presents the amplitude and phase of the multipath propagation at different frequencies (corresponding to different subcarriers), so that the channel with frequency selective fading characteristics can be more accurately represented, and belongs to fine-grained information. Therefore, for the judgment of gesture recognition, the CSI can achieve higher accuracy than the RSSI.

[0007] At present, the gesture recognition technology based on WiFi signals can be divided into a model-based method and a method based on a machine learning algorithm. The former needs to construct a model for gesture data to extract features, and then uses a classification algorithm (such as a nearest neighbor algorithm, a support vector machine, a logistic regression, a random forest, etc.) to recognize and classify the features. Although this method does not require a large number of samples, it needs to calculate parameters such as the arrival of angle (AoA), the time of flight (ToF) or the Doppler frequency shift (DFS). However, such parameters are usually difficult to calculate and have insufficient accuracy. The latter uses a deep learning neural network to complete feature extraction and recognition of gestures. However, this method needs to collect a large number of samples for model training, and at the same time, due to the change of the experimental environment (such as changing the experimental site, the experimental personnel, the position of the receiving and transmitting equipment, the experimental site and the direction of the personnel, etc.) for the same gesture, the corresponding CSI measurement value change mode may be different, thereby causing the model cross-domain recognition accuracy to rapidly decrease. SUMMARY

[0008] The purpose of the present application is to provide a multi-modal feature fusion and small sample learning gesture recognition method and system, which aims to solve the problem that the accuracy of gesture recognition will greatly decrease and the model robustness is poor when the direction and position of the gesture performed by the experimental personnel are different from the samples during the training model, and a large number of samples are needed to complete the feature extraction work of deep learning.

[0009] According to a first aspect of the present application, a multi-modal feature fusion and small sample learning gesture recognition method is provided, comprising:

[0010] S1, obtaining source domain data and target domain data for gesture recognition;

[0011] S2, training a first feature extractor and a second feature extractor respectively by using the source domain data from different links and obtaining a first feature and a second feature, and inputting the first feature and the second feature into a multi-modal feature fusioner for feature fusion to obtain a multi-modal fusion feature;

[0012] S3, training a classifier based on the training feature of the target domain data fused by the multi-modal feature fusioner, and performing gesture recognition based on the trained classifier.

[0013] In a preferred embodiment of the present application, the step S2 further comprises:

[0014] S21, training the first feature extractor by using the source domain data from the first link as a first training set to obtain the first feature, and training the second feature extractor by using the source domain data from the second link as a second training set to obtain the second feature, wherein the first feature extractor and the second feature extractor are trained independently;

[0015] S22, inputting the first feature and the second feature into the multi-modal feature fusioner for feature fusion to obtain a multi-modal fusion feature, wherein the multi-modal fusion feature is an element product of the first feature and the second feature.

[0016] In a preferred embodiment of the present application, the first feature extractor and the second feature extractor respectively comprise one one-dimensional CNN layer and three two-dimensional CNN layers and two fully connected layers.

[0017] In a preferred embodiment of the present application, the one-dimensional CNN layer comprises 256 convolution kernels with a size of 2, a standardization layer, a ReLU activation function and a pooling layer;

[0018] The first two-dimensional CNN layer comprises 256 convolution kernels with a size of 3x3, a batch normalization layer and a ReLU activation function, and a pooling layer; the second two-dimensional CNN layer comprises 128 convolution kernels with a size of 2x2, a batch normalization layer and a ReLU activation function, and a pooling layer; and the third two-dimensional CNN layer comprises 64 convolution kernels with a size of 2x2, a batch normalization layer and a ReLU activation function, and a pooling layer.

[0019] In a preferred embodiment of the present application, the multi-modal feature fusioner comprises a first flattening layer, a second flattening layer, a first linear layer, a second linear layer and an element product layer, the first flattening layer and the second flattening layer are respectively connected to the output of the second two-dimensional CNN layer of the first feature extractor and the second feature extractor to obtain and fuse the first feature and the second feature;

[0020] the multi-modal fusion feature is represented as ° represents Hadmard product, is a bilinear model output, a and respectively represent the first feature and the second feature.

[0021] In a preferred embodiment of the present application, the multi-modal feature fusioner further comprises a sum-pooling layer and a normalization layer, and the multi-modal fusion feature is processed by the sum-pooling layer to obtain a converged feature wherein the function sumpooling(X, P) represents that the sum-pooling on X is implemented using a 1-dimensional non-overlapping window with a size of P, and k represents the matrix dimension of the source domain data and the target domain data.

[0022] In a preferred embodiment of the present application, the step S1 further comprises: S11, collecting gesture recognition signals at different positions and in different directions by using at least two signal receivers; S12, pre-processing the gesture recognition signals; and S13, dividing the gesture recognition signals into the source domain data and the target domain data based on the position information of the gesture recognition signals.

[0023] In a preferred embodiment of the present application, the gesture recognition signal is CSI data, and the pre-processing comprises normalization processing, static part removal processing, slice PCA processing, reconstruction processing and smoothing processing.

[0024] In a preferred embodiment of the present application, the normalization processing comprises normalizing the CSI data stream in the subcarrier dimension:

[0025]

[0026] wherein represents the normalized CSI data stream, x max and x min are the maximum value and the minimum value of the 56 subcarriers in the CSI data collected at each time point, and are the minimum value and the maximum value of the normalized data, which are set to -1 and 1 respectively.

[0027] According to a first aspect of the present application, a gesture recognition system based on multi-modal feature fusion and small sample learning is provided, comprising:

[0028] a data acquisition module, configured to acquire source domain data and target domain data for gesture recognition;

[0029] The multi-modal fusion module is configured to train a first feature extractor and a second feature extractor respectively by using the source domain data from different links, and obtain a first feature and a second feature, and input the first feature and the second feature into a multi-modal feature fusion device for feature fusion to obtain a multi-modal fusion feature.

[0030] The gesture recognition module is configured to train a classifier based on the training feature of the target domain data obtained by the multi-modal feature fusion device, and perform gesture recognition based on the trained classifier.

[0031] The gesture recognition method based on multi-modal feature fusion and small sample learning provided by the application can enhance features by fusing the data received by multiple pairs of transceiving devices, and can realize high-accuracy cross-domain gesture recognition by using the small sample learning method only by collecting a small amount of target domain data, thereby solving the problem that the accuracy of gesture recognition is affected by the change of orientation caused by the position direction dependence of the gesture recognition technology based on WiFi, and reducing the demand for a large number of samples for model training in a new environment and improving the robustness of the model. BRIEF DESCRIPTION OF DRAWINGS

[0032] The application will be further described below with reference to the accompanying drawings and embodiments. In the drawings:

[0033] Figure 1 is a flowchart of the gesture recognition method based on multi-modal feature fusion and small sample learning of the preferred embodiment of the application;

[0034] Figure 2 is a schematic diagram of the acquisition of source domain data according to the preferred embodiment of the application;

[0035] Figure 3 is a schematic diagram of the acquisition of target domain data according to the preferred embodiment of the application;

[0036] Figure 4 is a logic flowchart of the gesture recognition method based on multi-modal feature fusion and small sample learning of another preferred embodiment of the application;

[0037] Figure 5 is Figure 4 is a flowchart of the data preprocessing of the gesture recognition method based on multi-modal feature fusion and small sample learning shown in the figure;

[0038] Figure 6 is a flowchart of the multi-modal fusion feature acquisition step according to the preferred embodiment of the application;

[0039] Figure 7 is a model schematic diagram of the feature extractor according to the preferred embodiment of the application;

[0040] Figure 8 This is a schematic diagram of a multimodal feature fusion device according to a preferred embodiment of the present invention;

[0041] Figure 9 This is a schematic diagram illustrating the recognition accuracy of a gesture recognition method based on multimodal feature fusion and few-shot learning at the same location in different directions, using a preferred embodiment of the present invention.

[0042] Figure 10 This is a schematic diagram showing the classification results of a single gesture at different locations using a gesture recognition method based on multimodal feature fusion and few-shot learning according to a preferred embodiment of the present invention.

[0043] Figure 11A -B is a schematic diagram showing the classification results of a single gesture by different experimenters at the same position using a gesture recognition method based on multimodal feature fusion and few-shot learning according to a preferred embodiment of the present invention.

[0044] Figure 12 This is a block diagram illustrating the principle of a gesture recognition system based on multimodal feature fusion and few-shot learning, according to a preferred embodiment of the present invention. Detailed Implementation

[0045] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0046] The application designs a gesture recognition method based on multi-modal feature fusion and small sample learning. At present, the dependence of the gesture recognition technology based on Wi-Fi on the gesture execution position and the execution person direction is a key problem to be solved, that is, when the execution gesture position and direction of the experimental personnel are different from the samples during the training model, the accuracy of gesture recognition will be greatly reduced. We divide the data collected in different positions and different directions of gesture execution into source domain data and target domain data. The source domain data is the data in four directions of any position. At the same time, in order to enhance the features and improve the recognition accuracy, we use the multi-modal feature fusion method to fuse the data of different transceiving devices. The features extracted by this method contain more rich information, so the detection accuracy of the subsequent model is significantly improved, thereby solving the problem that the accuracy of gesture recognition will be greatly reduced when the direction and position of the gesture execution of the experimental personnel are different from the samples during the training model, resulting in poor model robustness. Moreover, in order to reduce the sample collection work, the application adopts the idea of small sample learning, so that the training of the whole model only needs a small amount of samples. For the recognition of the target domain (gesture executed in different positions and directions), only a small amount of training samples (1 sample for each direction and each gesture type) is needed to achieve high-accuracy recognition effect of the target domain.

[0047] The gesture recognition method based on multi-modal feature fusion and small sample learning of the application can fuse the features of the data received by multiple pairs of transceiving devices, thereby enhancing the features. Then, by using the small sample learning method, a small amount of target domain data can be collected to achieve high-accuracy cross-domain gesture recognition, so as to solve the problem that the change of the direction caused by the dependence of the gesture recognition technology based on Wi-Fi on the position and direction affects the accuracy of gesture recognition, reduce the demand for a large number of samples for model training in a new environment, and improve the robustness of the model.

[0048] The gesture recognition method based on multi-modal feature fusion and small sample learning of the application will be described below in combination with specific embodiments. Figure 1 The figure is a flowchart of the gesture recognition method based on multi-modal feature fusion and small sample learning of the preferred embodiment of the application. It should be understood that the serial numbers of the steps in the embodiment do not mean the order of execution, the execution order of the processes should be determined according to its function and inherent logic, and should not constitute any limitation on the implementation process of the embodiment of the application.

[0049] As shown in the figure, in step S1, source domain data and target domain data for gesture recognition are acquired. Figure 1 As shown in the figure, in step S1, source domain data and target domain data for gesture recognition are acquired. Figure 2 The figure is a schematic diagram of the acquisition of the source domain data according to the preferred embodiment of the application. Figure 3is a schematic diagram of target domain data acquisition according to a preferred embodiment of the present application. As shown in Figure 2 The experimental scenario is a conference room, and we use receiver 1 and receiver 2 to collect data of 6 gestures in 4 directions (up, down, left, and right) at any position in the conference room as source domain data, respectively, as shown in Figure 3 The experimental scenario is the same conference room, and we use receiver 1 and receiver 2 to collect data of 4 directions at another position in the conference room as target domain data, respectively. In this way, we can divide the source domain data (receiver 1 and receiver 2 data when the experimental personnel are at position 1) and the target domain data (receiver 1 and receiver 2 data when the experimental personnel are at other positions) according to different gesture execution positions and different gesture execution directions. The source domain data will be used to train our feature fusioner to perform feature fusion on the source domain data through the feature fusioner, so as to extract features containing more rich information, so as to improve the detection accuracy of the subsequent model. Then the target domain data will be input into the feature fusioner to obtain gesture features for training a classifier. At this time, the training classifier only needs to use a small amount of samples (1 sample in different directions for each action category).

[0050] In step S2, the first feature extractor and the second feature extractor are trained respectively using the source domain data from different links, and the first feature and the second feature are obtained, and the first feature and the second feature are input into the multi-modal feature fusioner for feature fusion to obtain multi-modal fusion features.

[0051] In a preferred embodiment of the present application, the source domain data from different links can be source domain data from different transceiving devices, such as source domain data from receiver 1 and its corresponding transmitter can be source domain data from a first link, and source domain data from receiver 2 and its corresponding transmitter can be source domain data from a second link.

[0052] In this application, we first use the idea of multi-modal feature fusion, and our first stage goal is to train a multi-modal feature fusioner f θ The multi-modal feature fusioner can fuse the CSI information collected from two different sources (two acquisition links) under the same gesture into a feature. The reason for this fusion is that the same gesture has different effects on the CSI information under different links, but there is also information intersection and complementary phenomenon between them. Therefore, the multi-modal fusion feature is better than the single feature and contains more feature information, which is beneficial to the subsequent classification work.

[0053] In step S3, the training features of the target domain data are fused based on the multi-modal feature fusioner, a classifier is trained based on the training features, and gesture recognition is performed based on the trained classifier.

[0054] In this step, the purpose is to use the idea of knowledge transfer of meta-learning to train a superior multi-modal feature fusioner f θ After that, only 1 sample of each gesture in different directions needs to be collected from the target domain to achieve superior classification performance.

[0055] In this application, we further consider that general machine learning algorithm training model needs a large number of samples, and gesture sample collection needs a lot of time, so we use the way of meta-learning, train the feature extractor in the source domain, and combine it into a multi-modal feature fusioner, the parameters of the model will not be changed, and then use the feature fusioner to complete the feature extraction and fusion of the target domain data in the target domain, then use the integrated training method to learn and train four kinds of classifiers such as support vector machine (SVM), logistic regression algorithm (LR), random forest algorithm (RF) and K-nearest neighbor algorithm (KNN) independently, and make prediction, then combine these predictions into combined prediction to complete the gesture classification work. After the classifier is trained, subsequent gesture recognition is performed based on it, which is obvious to those skilled in the art, so it will not be repeated.

[0056] The gesture recognition method based on multi-modal feature fusion and small sample learning of the application can enhance the features by fusing the data received by multiple pairs of transceiver devices, and can achieve high-accuracy cross-domain gesture recognition by using the small sample learning method only by collecting a small amount of target domain data, thereby solving the problem of the change of the direction of the gesture recognition technology based on WiFi caused by the position direction dependence affecting the accuracy of the gesture recognition, reducing the demand for a large number of samples for model training in a new environment, and improving the robustness of the model.

[0057] Figure 4 The logic flow chart of the gesture recognition method based on multi-modal feature fusion and small sample learning of another preferred embodiment of the application is shown in FIG. 4. The gesture recognition method based on multi-modal feature fusion and small sample learning of the preferred embodiment of the application will be described below. Figure 4 The gesture recognition method based on multi-modal feature fusion and small sample learning of the preferred embodiment of the application will be described below.

[0058] First, we can use ordinary commercial Wi-Fi to complete the data collection work. Compared with radar, commercial Wi-Fi is low in cost and easy to popularize and use in a large area. As shown in Figures 2-3 At least two signal receivers can be used to collect gesture recognition signals in different positions and different directions. In the present application, the gesture recognition signal obtains the channel state information in the scene to be detected through a line, and performs gesture detection according to the channel state information. It should be noted that the channel state information (CSI, Channel State Information) is the attribute of the communication link, which reflects the change of the wireless channel with the environment, such as scattering, fading, etc. The number of people in the environment, the action of the human body, etc. will affect the wireless channel, thereby causing the change of the channel state information, so the change of the gesture can be detected by monitoring the change of the channel state information in the present application. That is, in the present application, it can be understood that the gesture recognition signal is the CSI data.

[0059] In the preferred embodiment of the present application, the gesture recognition signal, that is, the CSI data, can be preprocessed. Figure 5 is Figure 4 The flowchart of data preprocessing of the gesture recognition method based on multi-modal feature fusion and small sample learning shown in

[0060] The channel state information represents the coefficient of the wireless channel. In the Wi-Fi system modulated by the orthogonal frequency division multiplexing (OFDM) technology, the CSI of each subcarrier is a complex number. We use Atheros CSI Tool as the acquisition software. Therefore, the original CSI data collected by us is a three-dimensional complex matrix Wherein, M is the number of receiving antennas, N is the number of transmitting antennas, N sc is the number of subcarriers (56 subcarriers under 20MHz channel bandwidth, 114 subcarriers under 40MHz). For the collected CSI data, we perform subsequent preprocessing on the amplitude.

[0061] The specific processing flow is shown in Figure 5 We sort the received CSI data in time, that is, we form a CSI data stream T is the time dimension) from the original CSI data. Then, the CSI data stream is preprocessed to remove a large amount of noise in the data.

[0062] I. Normalization processing:

[0063] We normalize the original CSI data stream in the subcarrier dimension. This operation is expressed in equation (1):

[0064]

[0065] where, where is the normalized CSI stream, x max and x min are the maximum and minimum values of the 56 subcarriers in the CSI data collected at each time point, and are the minimum and maximum values of the normalized data. In our measurement, we set them to -1 and 1.

[0066] II. Static part removal

[0067] CSI data can be divided into two parts, one part is caused by the movement or other actions of people in the environment, the other part is caused by the static objects in the environment. In fact, the influence of the static part on CSI will be much greater than the relatively slight behavior caused by the execution of gestures on CSI. Therefore, it is necessary to remove the static part first, and then process the dynamic part alone. In our method, we eliminate the influence of the static part by subtracting the corresponding coefficient The size of this coefficient is the average CSI amplitude per 0.1 second.

[0068] III. Piecewise PCA processing

[0069] After removing the static part, we cut the CSI data into a new matrix Because the size of the cut window is 0.1s and the transmission frequency is 100Hz, t is 10. K is 56, representing the number of subcarriers collected. Then calculate the covariance matrix with dimensions k x k. In the next step, we should calculate the eigenvector matrix of the covariance matrix Then we need to find the first largest eigenvector ε K×i = {e1, …, e i} i∈[1, k], which can represent more than 90% of the cumulative contribution in the eigenvector matrix. We can get the reduced dimension data, i.e. the main feature component of the CSI dynamic part, by matrix multiplication

[0070] IV. Reconstruction processing

[0071] By using PCA dimension reduction to extract the main feature component of the CSI dynamic part, we can use it to reconstruct the CSI. The steps are as follows, we first use the equation ​The static part is then recovered by adding the corresponding coefficients removed in the previous step .

[0072] IV. Smoothing

[0073] Finally, we apply a low-pass filter to smooth the data. The parameters of the filter are as follows: passband frequency is 10 Hz, stopband frequency is 100 Hz, passband ripple is 1 dB, and stopband attenuation is 20 dB.

[0074] In the preferred embodiment, first, the amplitude signal is normalized from the subcarrier dimension (cannot be normalized from the time dimension), and then the remaining most noise is removed using the piecewise principal component analysis method with a 90% contribution reservation, and then data smoothing is performed using a low-pass filter, so that the features are more recognizable;

[0075] After the pre-processing is completed, we divide the gesture recognition signal into the source domain data and the target domain data based on the position information of the gesture recognition signal. Specifically, after the data is pre-processed, we label each sample and save the sample as (x i ,y i ), where is the CSI amplitude matrix, and y i is the label. Since the gesture recognition technology is dependent on direction and position, we need to consider different directions or different positions as different domains.

[0076] As shown in Figures 2-3 , we respectively use receiver 1 and receiver 2 to collect data of 6 gestures in 4 directions (up, down, left, and right) at any position in the conference room as source domain data. We use receiver 1 and receiver 2 to collect data of 4 directions at another position in the conference room as target domain data.

[0077] The data collected for different positions, different experimenters, and different actions will be divided into different training sets and test sets. For example, the following experiment can be used:

[0078] Change the experimenter: only the data of experimenter No. 1 in 4 directions at position 1 is used as source domain data, and the gesture data performed by the remaining experimenters in 4 directions at position 2 is used as target domain data.

[0079] Change the source domain data: use the data of experimenter No. 2 in 4 directions at position 1 as source domain data, and the gesture data performed by the remaining 3 experimenters in 4 directions at position 2 as target domain data.

[0080] See Figure 4It can be seen that after obtaining the source domain data and the target domain data, we use the source domain data to train our multi-modal feature fusioner; then we input the data collected in the target domain into the multi-modal feature fusioner to obtain new gesture features, and then use the features to train a classifier. Among them, only a small amount of samples (1 sample in different directions for each action category) is needed to train the classifier. We randomly select one sample from each category of gesture in each of the four directions of the dataset as the data set for training the classifier. In order to reduce the influence of randomly selected samples on the final accuracy, the final result comes from the average of ten experiments.

[0081] Specifically, we first use the idea of multi-modal feature fusion. Our first stage goal is to train a multi-modal feature fusioner f θ The multi-modal feature fusioner can fuse the CSI information of the same gesture collected from two different sources (two acquisition links) into a feature. The reason for this fusion is that under different links, the same gesture has different effects on the CSI information, but there is also information intersection and complementary phenomenon between them. Therefore, the multi-module fusion feature is better than the single feature and contains more feature information, which is beneficial to the subsequent classification work. Our second stage goal is to use the idea of knowledge transfer of meta-learning. After we train a multi-modal feature fusioner f θ with superior performance from the source domain, we only need to collect 1 sample of each gesture in different directions from the target domain to achieve superior classification performance.

[0082] First of all, in this application, our dataset is classified as follows:

[0083] Training dataset: The training dataset is divided into two categories. One is the source domain training dataset, which contains source domain data from two acquisition links of the source domain data for multi-modal feature fusioner. Among them, and respectively refer to the source domain data collected by acquisition link 1 and acquisition link 2, and respectively refer to the corresponding labels; the other is the cross-training dataset It has very few labeled target domain samples for training the classifier.

[0084] Test dataset: The test dataset is also divided into two categories. One is the validation dataset, which contains a small amount of double-link data from the source domain data The other is the query dataset contains unlabeled target domain samples, which are used to verify the function of the classifier in cross-domain gesture recognition.

[0085] Figure 6 is a flow chart of the multi-modal fusion feature acquisition step according to the preferred embodiment of the present application. The specific steps are explained as follows. In step S21, the source domain data from the first link is used as a first training set to train the first feature extractor to acquire the first feature, and the source domain data from the second link is used as a second training set to train the second feature extractor to acquire the second feature.

[0086] In the preferred embodiment of the present application, the first feature extractor and the second feature extractor can adopt any suitable feature extractor construction, such as Convolutional Neural Networks (CNN), deep learning models (LSTM (long short-term memory), GRU (Gate Recurrent Unit), etc.

[0087] In the preferred embodiment of the present application, the first feature extractor and the second feature extractor adopt a CNN model. Figure 7 is a model schematic diagram of the feature extractor according to the preferred embodiment of the present application. As shown in Figure 7 the first feature extractor and the second feature extractor each include one one-dimensional CNN layer and three two-dimensional CNN layers, as well as two fully connected layers. The one-dimensional CNN layer contains 256 convolution kernels with a size of 2, a standardization layer, a ReLU activation function, and a pooling layer; the first two-dimensional CNN layer includes 256 convolution kernels with a size of 3x3, a batch normalization layer, and a Relu activation function, as well as a pooling layer; the second two-dimensional CNN layer includes 128 convolution kernels with a size of 2x2, a batch normalization layer, and a Relu activation function, as well as a pooling layer; and the third two-dimensional CNN layer contains 64 convolution kernels with a size of 2x2, a batch normalization layer, and a Relu activation function, as well as a pooling layer.

[0088] The purpose of the source domain is to train the feature extractor i.e. wherein, is a cross-entropy loss function, denotes a training set. In the present application, since the first feature extractor and the second feature extractor are trained by the source domain data of the two links respectively, the first feature extractor is used to extract the amplitude feature of the CSI data of link 1, and the second feature extractor This is used to extract the amplitude features of the CSI data from link 2. Both features are trained independently. During the extraction phase, we will train two independent first and second feature extractors using the datasets from the two links in the source domain, and the parameters in these two feature extractors will not be changed afterward. Then we proceed to the fusion step.

[0089] Of course, in other preferred embodiments of the present invention, the number of layers of the CNN can be changed or other models can be used, such as deep learning models (LSTM (long short-term memory), GRU (Gate Recurrent Unit), etc.).

[0090] In step S22, the first feature and the second feature are input into the multimodal feature fusion unit to perform feature fusion to obtain a multimodal fused feature, wherein the multimodal fused feature is the element-wise product of the first feature and the second feature.

[0091] Figure 8 This is a schematic diagram of a multimodal feature fusion processor according to a preferred embodiment of the present invention. Figure 8 As shown, the multimodal feature fusion processor is a multimodal bilinear pooling model, comprising a first flattening layer, a second flattening layer, a first linear layer, a second linear layer, an element-wise multiplication layer, a summation pooling layer, and a normalization layer. The first flattening layer and the second flattening layer are respectively connected to the outputs of the second two-dimensional CNN layer of the first feature extractor and the second feature extractor. The first linear layer and the second linear layer are respectively connected to the first flattening layer and the second flattening layer. The first linear layer and the second linear layer are simultaneously connected to the element-wise multiplication layer, entering the element-wise fusion stage.

[0092] In the fusion phase, our goal is to fuse the features extracted from the two links separately to construct new features with richer information than before. The previous phase transformed the two raw link data into unique features, which are then fused using element-wise multiplication. Specifically, we assume two feature representations from different modalities, namely the CSI features received by different transceiver devices—the first feature and the second feature—and represent them as feature vectors... and Exported and Therefore, the simplest multimodal bilinear modality, i.e., the aforementioned multimodal fusion feature, can be given as follows: ° refers to the Hadmard product. The output of the bilinear model is the multimodal fusion feature, a and These represent the first feature and the second feature, respectively.

[0093] To converge the classification algorithm and overcome overfitting, we connect a sum-pooling layer after the element-wise product layer, using the sum-pooling function sumpooling(X, P) defined as

[0094]

[0095] where the function sumpooling(X, P) denotes the sum-pooling on X using a 1-D non-overlapping window of size P. i.e., using a 1-D non-overlapping window of size k of the matrix dimension of the source domain data and the target domain data to implement the sum-pooling on To overcome overfitting, we also add a normalization layer after the sum-pooling layer.

[0096] In the preferred embodiment, a multi-modal factorized bilinear pooling (MFB) model based on CNN is proposed as a CSI signal feature fusioner, two separately trained models are set up to extract the amplitude information of CSI from two transceiving links respectively, and then the information is fused into new features. The features extracted by the MFB feature fusioner proposed in the application contain more abundant information, and the detection accuracy of the subsequent model is improved.

[0097] Referring back to Figure 4 After the feature extractor and the multi-modal feature fusioner are trained, the training features of the target domain data are obtained based on the multi-modal feature fusioner, a classifier is trained based on the training features, and gesture recognition is performed based on the trained classifier.

[0098] In one preferred embodiment of the application, the first feature extractor and / or the second feature extractor can be used to extract features from the target domain data, and then the training features of the target domain data are obtained based on the multi-modal feature fusioner. Then, a classifier is trained based on the training features, and the trained classifier is used for gesture recognition.

[0099] Considering that a general machine learning algorithm needs a large number of samples to train a model, and the sample collection of gestures needs a large amount of time, the meta-learning method is adopted, after the feature extractor and the multi-modal feature fusioner are trained in the source domain, the parameters of the whole model will no longer be changed, and then the data in the target domain is used to complete the feature extraction and fusion work by using the model, and then the integrated training method is used, four kinds of classifiers such as support vector machine (SVM), logistic regression algorithm (LR), random forest algorithm (RF) and K-nearest neighbor algorithm (KNN) are independently learned and trained, and prediction is made, and then the combined prediction is combined to complete the gesture classification work.

[0100] In a further preferred embodiment of the application, a generative adversarial network (GAN) module is added before small sample learning, the samples in the target domain are put into the GAN model for data enhancement, and then small sample learning is performed on the enhanced data. The application proposes the idea of small sample learning, which only needs to collect one sample in different positions and directions to train a model with high cross-domain gesture recognition rate, which can solve the robustness problem of the performance decline of the deep learning model in cross-domain recognition to a certain extent, and also reduces the number of sample collection and the training cost of the model.

[0101] In the application, the data collected in different positions and directions of gesture execution are divided into source domain data and target domain data. The source domain data is the data in four directions of any position. Meanwhile, in order to enhance the features and improve the recognition accuracy, the multi-modal bilinear matrix decomposition pooling method is adopted to fuse the features of the data collected by different receiving and transmitting devices. In order to reduce the sample collection work, the application adopts the idea of small sample learning, so that the training of the whole model only needs a small number of samples. For the recognition of the target domain (gestures performed in different positions and directions), only a small number of training samples (one sample for each direction and each gesture type) are needed to achieve high accuracy recognition effect. Experimental results show that when the test set of the target domain uses one sample for each direction and each gesture type, the average recognition accuracy can reach more than 98%.

[0102] The specific process and beneficial effects of the gesture recognition method based on multi-modal feature fusion and small sample learning of the application will be described below by using specific experiments and data.

[0103] We collected samples of six types of gestures, including 1) push-pull gestures, 2) swipe gestures, 3) wave gestures, 4) Z-drawing gestures, 5) circle-drawing gestures, and 6) clapping gestures, from multiple experimenters at different positions in one scene. As described above, in the source domain, we first collected CSI samples from four directions of the experimenter using two pairs of transceiver devices, and preprocessed the collected CSI amplitudes (normalization processing, static part removal processing, slice PCA processing, reconstruction processing, and smoothing processing). Subsequently, we trained two convolutional neural networks using the processed data of the two pairs of transceiver devices to extract the features of the CSI amplitudes, and then input the multi-modal feature fusioner to perform feature fusion to obtain multi-modal fusion features. In the target domain, we input a small amount of labeled samples (one sample per direction per type of gesture) into the multi-modal feature fusioner to perform feature fusion to obtain training features, which are then used to train a classifier for the target domain to classify samples in the target domain. Experimental verification shows that, in the case of a change in the position and direction of the experimenter, only one sample in different directions needs to be collected after the position is changed, and a good network model can be quickly trained. The experimental results of the dataset in four directions of the experimenter at position 1 as the test set in the same scene show that, in the case of one data acquisition, the recognition accuracy for the six types of gestures in directions 1, 2, 3, and 4 can reach an average precision of 100%, 99.04%, 100%, and 99.82%, respectively. Figure 9

[0104] Due to the direction and position dependence of gesture recognition technology, we need to consider different directions as different domains. Therefore, we randomly selected one sample per type of gesture from each of the four directions of the dataset as the data set for training the classifier. To reduce the impact of randomly selected samples on the final accuracy, the final result is the average of ten experiments. Figure 10 The classification results of a single gesture of the same person at position 2 are shown. We can see that the gesture recognition method based on multi-modal feature fusion and small sample learning of the present application can achieve an average precision of 98.38% at different positions.

[0105] At the same time, we also considered the impact of other experimenters on the robustness of the system. We collected data from four directions of two other experimenters at position 2, as shown in Figure 11A and 11B The classification results of a single gesture of different experimenters at position 2 are shown, with an average accuracy of 98.55% and 99.71%, respectively. Therefore, even if the experimenter changes, we can still achieve good accuracy.

[0106] ​Therefore, the application designs a gesture recognition method based on multi-modal feature fusion and small sample learning. The multi-modal feature fusion can fuse the features of the data received by multiple pairs of transceiving devices, thereby enhancing the features. The small sample learning method can realize high-accuracy cross-domain gesture recognition only by collecting a small amount of target domain data. The method can solve the position and direction dependence problem of the gesture recognition technology based on WiFi, reduce the demand for a large number of samples for model training in a new environment, and improve the robustness of the model. It is verified through experiments that, in the case of position and direction transformation of the experimental personnel, only one sample in different directions after the transformation position is collected, and a network model with good effect can be quickly trained. The experimental results show that the application can obtain a cross-domain gesture recognition accuracy of up to 98% or more. The application realizes position-independent and direction-independent gesture recognition, and solves the influence of different execution directions of the same gesture and different positions of the experimental personnel on the recognition accuracy. In addition, the data preprocessing method adopted can effectively denoise the original data, which lays a good foundation for the subsequent model training.

[0107] Figure 12 is a principle block diagram of a gesture recognition system based on multi-modal feature fusion and small sample learning of a preferred embodiment of the application. As shown in Figure 12 the gesture recognition system based on multi-modal feature fusion and small sample learning includes a data acquisition module 100, a multi-modal fusion module 200 and a gesture recognition module 300. The data acquisition module 100 is used to acquire source domain data and target domain data for gesture recognition. As described above, the data acquisition module 100 can include multiple sets of transceiving devices, such as receiver 1 and its corresponding transmitter, receiver 2 and its corresponding transmitter, which can be used to acquire source domain data and target domain data for gesture recognition. The acquisition process of the source domain data and the target domain data for gesture recognition can refer to the prior art, and will not be repeated here.

[0108] The multi-modal fusion module 200 can be used to train a first feature extractor and a second feature extractor respectively using the source domain data from different links and acquire first features and second features, and input the first features and the second features into a multi-modal feature fusion device for feature fusion to obtain multi-modal fusion features.

[0109] The gesture recognition module 300 is used to train a classifier based on the training features of the target domain data fused by the multi-modal feature fusion device, and perform gesture recognition based on the trained classifier.

[0110] Here, the data acquisition module 100, the multi-modal fusion module 200 and the gesture recognition module 300 can refer to Figures 1-11BThe gesture recognition method based on multi-modal feature fusion and small sample learning shown in the method structure can refer to the foregoing method structure for specific implementation manners and beneficial effects, and will not be repeated here.

[0111] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the division of the functional units and modules is exemplified, and in actual application, the functions can be completed by different functional units and modules according to needs, that is, the internal structure of the system is divided into different functional units or modules to complete all or part of the functions described above. Each functional unit or module in the embodiment can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit, and the integrated unit can be realized in the form of hardware or in the form of a software functional unit. In addition, the specific names of each functional unit or module are only for convenient distinction, and do not limit the protection scope of the present application. The specific working process of the unit or module in the system can refer to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0112] In the embodiments, the description of each embodiment has its own focus, and the parts not described or recorded in a certain embodiment can refer to the related description of other embodiments. Those skilled in the art can realize that the units and algorithm steps of each example described in combination with the embodiments disclosed in the present text can be realized by electronic hardware or a combination of computer software and electronic hardware. Whether the functions are executed in hardware or software depends on the specific application and design constraints of the technical solution. Professional technicians can use different methods for each specific application to realize the described functions, but such implementation should not be considered beyond the scope of the present application.

[0113] The integrated module / unit, if implemented in the form of a software function unit and sold or used as an independent product, can be stored in a computer readable storage medium. Based on such understanding, all or part of the processes in the embodiment methods of the present application can also be completed by a computer program instructing related hardware, and the computer program can be stored in a computer readable storage medium. The computer program can implement the steps of the various method embodiments when executed by a processor. The computer program includes computer program code, which can be in the form of source code, object code, executable files or some intermediate forms of the computer program code. The computer readable medium can include any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the computer readable storage medium contains contents which can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction.

[0114] Although the present application is described by specific embodiments, those skilled in the art should understand that various modifications and equivalent replacements can be made to the present application without departing from the scope of the present application. In addition, various modifications can be made to the present application for specific situations or materials without departing from the scope of the present application. Therefore, the present application is not limited to the disclosed specific embodiments, but should include all embodiments falling within the scope of the claims of the present application.

[0115] The above is only the preferred embodiment of the present application, and is not intended to limit the present application. Any modification, equivalent replacement and improvement made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A gesture recognition method based on multi-modal feature fusion and small sample learning, characterized in that, The method comprises the following steps: S1, acquiring source domain data and target domain data of gesture recognition; S2, training a first feature extractor and a second feature extractor respectively using the source domain data from different links, acquiring first features and second features, and inputting the first features and the second features into a multi-modal feature fusion device for feature fusion to obtain multi-modal fusion features; S3, training a classifier based on training features of the target domain data obtained by the multi-modal feature fusion device, and performing gesture recognition based on the trained classifier; The step S2 further comprises: S21, training the first feature extractor using the source domain data from a first link as a first training set to obtain the first features, and training the second feature extractor using the source domain data from a second link as a second training set to obtain the second features, wherein the first feature extractor and the second feature extractor are trained independently; S22, inputting the first features and the second features into the multi-modal feature fusion device for feature fusion to obtain multi-modal fusion features, wherein the multi-modal fusion features are element products of the first features and the second features; The first feature extractor and the second feature extractor each comprise one one-dimensional CNN layer, three two-dimensional CNN layers, and two fully connected layers; the one-dimensional CNN layer comprises 256 convolution kernels with a size of 2, a standardization layer, a ReLU activation function, and a pooling layer; The first two-dimensional CNN layer comprises 256 convolution kernels with a size of 3*3, a batch normalization layer, a ReLU activation function, and a pooling layer; the second two-dimensional CNN layer comprises 128 convolution kernels with a size of 2*2, a batch normalization layer, a ReLU activation function, and a pooling layer; and the third two-dimensional CNN layer comprises 64 convolution kernels with a size of 2*2, a batch normalization layer, a ReLU activation function, and a pooling layer; The multi-modal feature fusion device comprises a first flattening layer, a second flattening layer, a first linear layer, a second linear layer, and an element product layer, the first flattening layer and the second flattening layer are connected to outputs of the second two-dimensional CNN layers of the first feature extractor and the second feature extractor respectively to acquire and fuse the first features and the second features; The multi-modal fusion feature is denoted as ; denotes a Hadmard product, is a bilinear model output, and denote the first feature and the second feature, respectively.

2. The gesture recognition method based on multi-modal feature fusion and small sample learning according to claim 1, characterized in that, The multi-modal feature fusioner further comprises a sum-pooling layer and a normalization layer, and the multi-modal fusion features are processed by the sum-pooling layer to obtain convergent features ; wherein the function represents that the sum-pooling on X is implemented using a 1-dimensional non-overlapping window with a size of P, and k represents the matrix dimension of the source domain data and the target domain data. 3.The gesture recognition method based on multi-modal feature fusion and small sample learning according to claim 1, characterized in that, The step S1 further comprises: S11, acquiring gesture recognition signals in different positions and different directions using at least two signal receivers; S12, pre-processing the gesture recognition signals; S13, dividing the gesture recognition signals into the source domain data and the target domain data based on position information of the gesture recognition signals.

4. The gesture recognition method based on multi-modal feature fusion and small sample learning according to claim 3, characterized in that, The gesture recognition signals are CSI data, and the pre-processing comprises normalization processing, static part removal processing, slice PCA processing, reconstruction processing, and smoothing processing.

5. The gesture recognition method based on multi-modal feature fusion and small sample learning according to claim 4, characterized in that, The normalization processing comprises normalizing CSI data streams in a subcarrier dimension: wherein denotes the normalized CSI data stream, and are the maximum and minimum values of the 56 subcarriers of the CSI data acquired at each time point, and denote the minimum and maximum values of the normalized data, which are set to -1 and 1, respectively.

6. A gesture recognition system based on multi-modal feature fusion and small sample learning, characterized in that, The method is used to implement the gesture recognition method based on multi-modal feature fusion and small sample learning according to any one of claims 1-5. The gesture recognition system based on multi-modal feature fusion and small sample learning comprises: a data acquisition module configured to acquire source domain data and target domain data for gesture recognition; a multi-modal fusion module configured to train a first feature extractor and a second feature extractor respectively by using the source domain data from different links, acquire first features and second features, and input the first features and the second features into a multi-modal feature fusion device for feature fusion to obtain multi-modal fusion features; a gesture recognition module configured to acquire training features of the target domain data based on the multi-modal feature fusion device, train a classifier based on the training features, and perform gesture recognition based on the trained classifier.

Images