Posture recognition method based on wireless sensing

Abstract

The invention discloses a posture recognition method based on wireless sensing, and relates to the field of computer vision. The method comprises the following steps: receiving BVP sample data; determining a first posture recognition result according to the BVP sample data by using a posture recognition model; wherein the posture recognition model is used for determining a first posture recognition result by fusing second posture recognition results output by at least two base learners according to the BVP sample data based on an integrated learning strategy, and the base learners are obtained by training based on an early stop method. Compared with the prior art, the method is low in recognition cost, high in safety and high in recognition precision.

Description

Technical Field

[0001] The present invention relates to the fields of wireless sensing technology and computer vision technology, and more specifically, to a gesture recognition method based on wireless sensing. Background Art

[0002] Early human posture recognition was mainly based on wearable sensors. By selecting suitable sensors, such as accelerometers and gyroscopes, and placing them on the human body, human activity data was collected. Currently, mainstream human posture recognition mainly relies on cameras and other devices for image capture and analysis. Computer vision technology can be used to complete tasks such as face recognition and traffic violation recognition.

[0003] Although the above recognition technologies can effectively capture postures, they have disadvantages such as relatively high sensor cost, poor user comfort, privacy leakage, and strong environmental dependence. Summary of the invention

[0004] In order to overcome the defects of the prior art such as high cost and poor safety, the present invention provides a gesture recognition method based on wireless sensing.

[0005] In order to solve the above technical problems, the technical solution of the present invention is as follows: In a first aspect, a gesture recognition method based on wireless sensing includes: Receive BVP sample data; A first posture recognition result is determined based on the BVP sample data using a posture recognition model; wherein the posture recognition model determines the first posture recognition result by fusing the second posture recognition results output by at least two base learners based on the BVP sample data based on an integrated learning strategy, and the base learners are trained based on the early stopping method.

[0006] In a second aspect, a computer program product comprises a computer program or computer executable instructions, wherein when the computer program or computer executable instructions are executed by a processor, the method described in the first aspect is implemented.

[0007] Compared with the prior art, the technical solution of the present invention has the following beneficial effects: The present application discloses a posture recognition method based on wireless sensing, which adopts BVP sample data based on wireless WiFi signals for posture recognition, reduces the recognition cost, avoids the privacy issues of traditional methods, improves security, and has strong adaptability; in addition, through the integration of learning strategies, multiple base learners trained based on early stopping method are integrated to improve the accuracy of posture recognition results while saving computing resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Figure 1This is a flow chart of a gesture recognition method based on wireless sensing in Example 1 of the present application; Figure 2 Schematic diagram of the process of featureConv operation in Example 1 of the present application; Figure 3 This is a flow chart comparing the reshape operation and the featureConv operation in Example 1 of the present application; Figure 4 This is a schematic diagram of the accuracy of each model on the training set at a training-to-test ratio of 0.8:0.2 in Example 2 of the present application; Figure 5 This is a schematic diagram of the accuracy of each model on the validation set at a training-to-test ratio of 0.8:0.2 in Example 2 of the present application; Figure 6 Schematic diagram of the loss value of each model on the training set at a training-to-test ratio of 0.8:0.2 in Example 2 of the present application; Figure 7 Schematic diagram of the loss value of each model on the validation set at a training-to-test ratio of 0.8:0.2 in Example 2 of the present application; Figure 8 This is a schematic diagram of the accuracy of each model on the training set at a training-to-test ratio of 0.9:0.1 in Example 2 of the present application; Fig. 9 This is a schematic diagram of the accuracy of each model on the validation set at a training-to-test ratio of 0.9:0.1 in Example 2 of the present application; Fig.10 Schematic diagram of the loss value of each model on the training set at a training-to-test ratio of 0.9:0.1 in Example 2 of the present application; Fig.11 This is a schematic diagram of the loss values ​​of each model on the validation set at a training-to-test ratio of 0.9:0.1 in Example 2 of the present application. DETAILED DESCRIPTION

[0009] The terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the terms used in this way can be interchanged in appropriate cases, which is only to describe the distinction mode adopted by the objects of the same attributes in the embodiments of the present application when describing. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, so that the process, method, system, product or equipment containing a series of units need not be limited to those units, but may include other units that are not clearly listed or inherent to these processes, methods, products or equipment. The term "determine" widely covers various actions, and may include acquisition, calculation, calculation, processing, derivation, investigation, search (for example, search in a table, database or other data structure), ascertainment, and similar actions, and may also include reception (for example, receiving information), access (for example, accessing data in a memory) and similar actions, and may also include generation, creation, establishment and similar actions, as well as parsing, selection, selection and similar actions, etc. The relevant definitions of other terms will be given in the following description.

[0010] It should be noted that when an element is considered to be "connected" to another element, it can be directly connected to the other element, or connected to the other element through an intermediate element. In addition, the "connection" in the following embodiments should be understood as "electrical connection", "communication connection", etc. if there is transmission of electrical signals or data between the connected objects.

[0011] The drawings are for illustrative purposes only and should not be construed as limiting the present patent; In order to better illustrate the present embodiment, some parts in the drawings may be omitted, enlarged or reduced, and do not represent the size of the actual product; It is understandable to those skilled in the art that some well-known structures and their descriptions may be omitted in the drawings.

[0012] The technical solution of the present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0013] Example 1 As mentioned in the background technology section, although existing gesture recognition technologies can effectively capture gestures, they have disadvantages such as relatively high sensor costs, poor user comfort, privacy leakage, and strong environmental dependence. In view of this, this embodiment provides a posture recognition method based on wireless sensing, which can detect various postures and gestures of the human body without image capture, thereby solving the above-mentioned deficiencies.

[0014] First of all, it should be noted that the BVP (Body-coordinate Velocity Profile) sample data involved in this embodiment is extracted from CSI (Channel State Information). CSI refers to information used to describe the status of the communication channel; in a wireless communication system, a channel refers to the medium through which a signal is transmitted from a transmitter to a receiver, and its status is affected by many factors, such as multipath fading, interference, attenuation, etc. Channel state information includes a description of channel characteristics, such as channel fading, delay characteristics, phase changes, etc.

[0015] Existing research has found that neural network models can be used to extract various information from CSI data, but they have defects such as low information extraction rate and low recognition accuracy. In different environments, the interference of background factors may cause the same gesture to have different CSI features, which will bring greater challenges to model recognition. In addition, due to the complexity of gestures, there may be hundreds of dynamic contours for a single gesture, and it is difficult to achieve good prediction accuracy using only ordinary CSI data; however, in the human body coordinate system, each gesture has its own unique speed distribution. BVP data based on the human body coordinate system enables it to achieve environment-independent and cross-domain gesture recognition.

[0016] It should also be noted that BVP sample data is similar to a picture sequence. Each BVP sample is a A tensor of is the number of snapshots of BVP, It is the dimension of the feature map.

[0017] Referring to FIG. 1 , a gesture recognition method based on wireless sensing is provided in this embodiment, including: Receive BVP sample data; A first posture recognition result is determined based on the BVP sample data using a posture recognition model; wherein the posture recognition model determines the first posture recognition result by fusing the second posture recognition results output by at least two base learners based on the BVP sample data based on an integrated learning strategy, and the base learners are trained based on the early stopping method.

[0018] The method described in this embodiment realizes posture recognition through BVP sample data based on wireless signals, avoids the privacy issues of traditional methods, improves security, and has strong adaptability; in addition, multiple base learners are fused through an integrated learning strategy to improve the accuracy of posture recognition results; at the same time, the early stopping method is used to train the base learners, saving computing resources.

[0019] As a non-limiting example, the at least two base learners may be neural network models with different model structures, or neural network models with the same model structure but different model parameters, or neural network models with the same model structure but different training methods and / or training data.

[0020] In some preferred embodiments, the base learner includes at least one of an MLP model, an LSTM model, a BiLSTM model, a ResNet18 model, a ResNet50 model, and a ViT model.

[0021] In some optional embodiments, when the base learner is a ResNet18 model, before the BVP sample data is input into the ResNet18 model, a reshape operation is performed on the BVP sample data, including: The BVP sample data is sequentially passed through the first deconvolution layer and the second deconvolution layer; wherein, The first deconvolution layer includes a first number of convolution kernels with a first step length and a first size, a padding size of a first padding amount, and uses a ReLU activation function as an activation function; and, The second deconvolution layer includes a second number of convolution kernels with a second step size and a second size, a padding size of a second padding amount, and uses a ReLU activation function as an activation function.

[0022] It should be noted that ResNet18 was originally a neural network used to solve image classification. The input of the original ResNet18 network is a The tensor is a three-channel sample; therefore, in order to use ResNet18 for the posture recognition problem in this embodiment, it is necessary to perform a reshape operation on the sample before the sample enters the ResNet18 network.

[0023] In some specific implementations, the 22×20×20 BVP sample data set provided by Widar3.0 is used. The data set contains 22 types of human gestures, namely: Push&Pull, Sweep, Clap, Slide, Draw-N(H), Draw-O(H), Draw-Rectangle(H), Draw-Triangle(H), Draw-Zigzag(H), Draw-Zigzag(V), Draw-N(V), Draw-O(V), Draw-1, Draw-2, Draw-3, Draw-4, Draw-5, Draw-6,Draw-7, Draw-8, Draw-9, Draw-0. The processing process of BVP sample data based on ResNet18 is as follows: S11, reshape operation: The number of input channels is 22. First, a convolution kernel with 3 convolution kernels and a size of , a deconvolution layer with a step size of 2 and a padding of 3, and then a ReLU activation function; then a convolution kernel with a convolution kernel size of 3 and a convolution kernel size of , a deconvolution layer with a step size of 2 and a padding of 3, and finally a ReLU activation function. The number of output channels is 3. After this operation, the data is The tensor form becomes The tensor form can be used to directly input into the ResNet18 architecture.

[0024] S12, the first convolution layer conv1: the number of input channels is 3. First, it passes through a convolution kernel with a size of 64. , a convolutional layer with a stride of 2, then a Batch Normalization layer, and finally a ReLU activation function. The number of output channels is 64.

[0025] S13, the first residual block conv2_x: the number of input channels is 64. First, it undergoes a maximum downsampling pooling operation with a convolution kernel size of 3×3 and a stride of 2; then it stacks 2 Each layer of the residual block of the structure passes through a Batch Normalization layer and a ReLU activation function. The number of output channels is 64.

[0026] S14, the second residual block conv3_x: the number of input channels is 64. This layer is stacked 2 Each layer of the residual block of the structure passes through a Batch Normalization layer and a ReLU activation function. The number of output channels is 128.

[0027] S15, the third residual block conv4_x: the number of input channels is 128. This layer is stacked 2 Each layer of the residual block of the structure passes through a Batch Normalization layer and a ReLU activation function. The number of output channels is 256.

[0028] S16, the fourth residual block conv5_x: the number of input channels is 256. This layer is stacked 2 Each layer of the residual block of the structure passes through a Batch Normalization layer and a ReLU activation function. The number of output channels is 512.

[0029] S17, average pooling layer and fully connected layer average pool&fc: Finally, it passes through an average pooling layer and a fully connected layer. The number of input channels is 512, and the number of output channels is 22, which is the number of categories corresponding to the human gesture classification in the Widar3.0 dataset.

[0030] The method based on ResNet18 combined with the reshape operation in the above embodiment is recorded as ResNet18 (1.0). It can be seen that before the sample data enters the first convolution layer conv1 of ResNet18, the sample will first be reshaped, and then input into the original ResNet18 network after the reshape operation. After testing, ResNet18 (1.0) achieved a prediction accuracy of 71.70% on the test set, exceeding the prediction accuracy of 71.00% of the conventional ResNet18 model without the reshape operation.

[0031] Although the reshape operation can make the number of sample channels correspond to the network architecture of ResNet18, it will also cause the input samples to lose their order information, making it difficult to extract the time features in the samples and limiting the improvement of the model's recognition accuracy.

[0032] To this end, in some optional embodiments, when the base learner is a ResNet18 model, before the BVP sample data is input into the ResNet18 model, a featureConv operation is performed on the BVP sample data, referring to FIG. 2 , including: Performing a dimension permutation on the BVP sample data, and subjecting the BVP sample data after the dimension permutation to convolution operations through a first convolution layer and a second convolution layer in sequence, to obtain a first intermediate tensor; After performing a second dimension permutation on the first intermediate tensor, the first intermediate tensor is again subjected to a convolution operation through the first convolution layer and the second convolution layer in sequence to obtain a second intermediate tensor; After performing three-dimensional permutation on the second intermediate tensor, a third intermediate tensor is obtained as the input of the ResNet18 model, and the number of input channels of the ResNet18 model is adapted to the dimension of the third intermediate tensor.

[0033] Further, the first convolution layer includes a third number of convolution kernels with a third step size and a third size, a padding size of a third padding amount, and uses a ReLU activation function as an activation function; and, The second convolution layer includes a fourth number of convolution kernels with a fourth step size and a fourth size, a padding size of a fourth padding amount, and uses a ReLU activation function as an activation function.

[0034] Furthermore, the number of input channels of the first convolutional layer is 20, the third number is 11, the third step length is 1, the third size is 7*7, and the third padding amount is 3; The number of output channels of the second convolutional layer is 5, the fourth number is 5, the fourth step size is 1, the fourth size is 5*5, and the fourth padding amount is 2.

[0035] In some specific implementations, for the 22×20×20 BVP sample data set provided by Widar3.0, the featureConv operation includes: S21. (Right now ) is first replaced by (Right now ).

[0036] S22. Perform the following convolution operation on the replaced samples: the number of input channels is 20. First, pass through a convolution layer with 11 convolution kernels, 7 convolution kernel size, 1 step size, and 3 padding, then pass through a ReLU activation function; then pass through a convolution layer with 5 convolution kernels, 5 convolution kernel size, 1 step size, and 2 padding, and finally pass through a ReLU activation function. The number of output channels is 5.

[0037] S23, after the above convolution operation, the (Right now ) (i.e., the first intermediate tensor) is then permuted into (Right now ), and then repeat the operation of step S22.

[0038] S24, after the above two dimension permutation and convolution operations, we get (Right now ) size (i.e., the second intermediate tensor), and then replace the sample to its original form, i.e. ( ), and then input the sample (i.e., the third intermediate tensor) into the original ResNet18 network architecture, and change the number of input channels of the conv1 part in ResNet18 to 22.

[0039] The network architecture after the above-mentioned featureConv operation is ResNet18 (2.0). The comparison between the featureConv operation and the reshape operation in ResNet18 (1.0) is shown in Figure 3.

[0040] After testing, ResNet18(2.0) after using featureConv operation achieved a prediction accuracy of 72.98% on the test set, exceeding the recognition accuracy of 71.70% of ResNet18(1.0), indicating that the featureConv operation does play a role.

[0041] It should be noted that due to the number of channels in the BVP sample data (22) It is more about expressing the order of actions rather than different performances of the same action. Therefore, when considering convolution, featureConv embeds the two information dimensions of a single image into the time dimension through a series of alternating transposed convolutions (i.e., the dimension permutation and convolution operations mentioned above). In. While retaining the order information of the action, the features of a frame of action are extracted, so that the available information is richer, thereby improving the recognition accuracy. In contrast, ResNet18 (1.0) combined with the reshape operation only performs a normal convolution operation and changes the shape of the input tensor, and does not extract the feature information of each time point and combine it with the time information.

[0042] It should be emphasized that even ResNet18(1.0) has higher recognition accuracy than the existing technology.

[0043] In some preferred embodiments, the gesture recognition model uses soft voting for integrated learning, including: receiving a second posture recognition result output by each of the base learners; Calculate the prediction probability corresponding to each second posture recognition result using softmax; Grouping is performed according to whether the second posture recognition results are the same, and calculating the mean of the predicted probabilities corresponding to the second posture recognition results as the grouping probability; The second posture recognition result corresponding to the largest grouping probability is selected as the first posture recognition result.

[0044] In some optional embodiments, the process of selecting the base learner in the gesture recognition model includes: Receiving the second posture recognition result output by each candidate base learner on the test set; wherein the test set includes the BVP sample data for testing and the corresponding true category label; Determining the prediction accuracy of each of the candidate base learners on the test set according to the second posture recognition result and the true category label; A preset number of the candidate base learners are selected according to the prediction accuracy and combined into the posture recognition model.

[0045] It should be understood that the candidate base learners refer to base learners that are used for training for screening as part of the integrated model (ie, the gesture recognition model).

[0046] In some specific implementation processes, based on the Widar3.0 dataset, the pseudo code for the soft voting algorithm implementation and accuracy verification of multiple candidate base learners is shown in Table 1.

[0047] Table 1 Soft voting pseudo code

[0048] In the specific implementation process, the most intuitive model selection strategy is adopted. According to the performance of a single model on the test set, the six models with the best performance are selected: ResNet18(1.0), ResNet18(2.0), ResNet50, MLP, BiLSTM, and ViT are combined respectively to obtain five different model combinations (i.e., posture recognition models after integrating multiple base learners), which are respectively recorded as Ensemble1, Ensemble2, Ensemble3, Ensemble4, and Ensemble5. Table 2 shows the performance of these five different model combinations on the test set.

[0049] Table 2 Model ensemble prediction accuracy

[0050] Ensemble Learning is introduced in the above embodiment to significantly improve the prediction accuracy of the model on the existing basis, and even reach a prediction accuracy exceeding 80%. Specifically, a soft voting method is used to count the prediction category probabilities of all models (i.e., base learners), and these prediction probabilities are summed and averaged. The class with the largest probability in the final probability mean (i.e., the second posture recognition result) is the final prediction class (i.e., the first posture recognition result).

[0051] Those skilled in the art should understand that, based on the method described in this embodiment, the ensemble learning strategy may also be implemented by using an averaging method, a weighted averaging method, a hard voting method, or the like.

[0052] Further, training the candidate base learner based on the early stopping method includes: Establishing the initial candidate base learner, initializing the counter, the optimal loss value, the loss value change threshold and the patience value; Iteratively train the candidate base learners on the training set to update the model parameters, and After every T training rounds: Determining a validation loss value of the candidate base learner on a validation set; According to whether the optimal loss value is an initialization value or whether the difference between the optimal loss value and the verification loss value satisfies the loss value change threshold, updating the optimal loss value based on the verification loss value and reinitializing the counter; When the difference between the optimal loss value and the verification loss value does not meet the loss value change threshold, the counter is incremented to end the current round, and, Whether to terminate the training is determined according to whether the value of the counter satisfies the patience value; when the value of the counter satisfies the patience value, the training is stopped and the model parameters corresponding to the candidate base learner of the current round are saved.

[0053] It should be understood that the patience value is a preset threshold set for the counter.

[0054] Furthermore, different base learners may be trained on the training set with the same training-to-test ratio or with different training-to-test ratios, which may be set by technicians in this field according to actual conditions.

[0055] In order to further alleviate the overfitting phenomenon in the model, this embodiment also discusses the setting of the value of the model training round epoch, and each model needs to find the training round that best suits it.

[0056] The simplest method is to set multiple epoch values ​​that may meet the conditions, perform multiple trainings under these epoch values ​​and compare them, and select the epoch value with the highest accuracy on the test set as the final training round. However, this method is not only time-consuming, but the epoch value obtained in the end may not be the optimal training round.

[0057] Therefore, this embodiment adopts the early stopping method to solve this problem, which can save time and find the training rounds that are most suitable for the base learner at one time.

[0058] In some specific implementations, the pseudo code for implementing the early stopping method used in the candidate base learner training process is shown in Table 3.

[0059] Table 3 Early stopping pseudo code

[0060] It should be noted that the early stopping method in the above embodiment solves the overfitting problem and enhances the generalization ability of the model. When training on large deep learning models or large-scale data sets, each additional training cycle may consume a large amount of GPU hours, energy, and storage space. The early stopping rule ensures that only training that is beneficial to improving the generalization ability of the model is performed, avoiding invalid calculations in the overfitting stage, which is very beneficial for accelerating experimental iterations and optimizing resource utilization.

[0061] It should be emphasized that the computing resources (including CPU, GPU time and memory, etc.) required for ensemble learning are usually large. This embodiment uses the early stopping method to further save computing resources while ensuring accuracy and significantly reduce time costs.

[0062] This embodiment also provides a computer-readable storage medium, on which is stored at least one instruction, at least one program, code set or instruction set, and the at least one instruction, at least one program, code set or instruction set is loaded and executed by a processor, so that the processor performs some or all of the steps of the method provided in the embodiment of the present application.

[0063] It is understood that the storage medium may be transient or non-transient. Exemplarily, the storage medium includes, but is not limited to, a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and other media that can store program codes.

[0064] Exemplarily, the processor may be a central processing unit (CPU), a microprocessor (MPU), a digital signal processor (DSP), or a field programmable gate array (FPGA).

[0065] Exemplarily, the read-only memory includes but is not limited to MASK ROM, PROM, EPROM, EEPROM, Flash, etc.

[0066] Exemplarily, the random access memory includes but is not limited to DRAM, SRAM, SDRAM, DDR SDRAM, etc.

[0067] In some examples, a computer program product is provided, which can be implemented in hardware, software or a combination thereof. As a non-limiting example, the computer program product can be embodied as the storage medium, or as a software product, such as an SDK (Software Development Kit).

[0068] As a non-limiting example, a computer program product is provided, the computer program product includes a computer program or a computer executable instruction, the computer program or the computer executable instruction is stored in a computer-readable storage medium. A processor of an electronic device reads the computer program or the computer executable instruction from the computer-readable storage medium, and the processor executes the computer executable instruction, so that the electronic device performs some or all steps of the method described in the embodiment of the present application.

[0069] In some examples, a computer program is provided, comprising a computer-readable code, and when the computer-readable code is run in a computer device, a processor in the computer device executes a part or all of the steps in the method.

[0070] This embodiment also proposes an electronic device, including a memory and a processor, wherein the memory stores at least one instruction, at least one program, a code set or an instruction set, and when the processor executes the at least one instruction, at least one program, a code set or an instruction set, it implements part or all of the steps of the method described in the embodiment of the present application.

[0071] In some examples, a hardware entity of the electronic device is provided, including: a processor, a memory and a communication interface; wherein the processor generally controls the overall operation of the electronic device; the communication interface is used to enable the electronic device to communicate with other terminals or servers through a network; the memory is configured to store instructions and applications executable by the processor, and can also cache data to be processed or processed by the processor and various modules in the electronic device (including but not limited to image data, audio data, voice communication data and video communication data), which can be implemented by flash memory (FLASH), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM) or random access memory (RAM).

[0072] The processor may include one or more processing elements. Thus, the processor may include one or more integrated circuits (ICs) configured to perform the functions of the processor. In addition, each integrated circuit may include circuits (e.g., a first circuit, a second circuit, and other circuits, etc.) configured to perform the functions of the processor.

[0073] Furthermore, data may be transmitted between the processor, the communication interface and the memory via a bus, which may include any number of interconnected buses and bridges, and the bus connects various circuits of one or more processors and memories together.

[0074] Example 2 In order to prove the technical effect of the method described in this application, this embodiment also carried out a comparative experiment on the Widar3.0 dataset for the method proposed in this application and the method based on the conventional network model. The deep learning environment used in the experiment is shown in Table 4.

[0075] Table 4 Deep learning environment

[0076] First, the conventional method (fixed 100 Epochs) is used to train multiple deep neural network models as base learners, and their prediction accuracy is shown in Table 5: Table 5 Prediction accuracy of conventional scheme

[0077] Next, multiple models were trained and tested based on the early stopping method, including MLP, LSTM, BiLSTM, ResNet18 (1.0), ResNet18 (2.0), ResNet50, and ViT, a total of 7 models. Figures 4 to 7 show the accuracy and loss values ​​of the training process of these 7 models under the condition of a training-to-test ratio of 0.8:0.2, and Figures 8 to 11 show the accuracy and loss values ​​of the training process of these 7 models under the condition of a training-to-test ratio of 0.9:0.1.

[0078] Referring to Table 6, the experimental results show that under the condition of a training-test ratio of 0.9:0.1, ResNet18(2.0) achieved a prediction accuracy of 74.74%, surpassing the highest prediction accuracy of 72.98% achieved by ResNet18(2.0) on this dataset before the introduction of the early stopping method, and achieved a new high in accuracy on the current test set. This shows that the introduction of the early stopping method has indeed alleviated the overfitting phenomenon in model training to a certain extent.

[0079] Table 6 Accuracy of each model on the test set (training-test ratio 0.9:0.1)

[0080] In addition, the model training curves in Figures 4-11 show the training time of each model. The training time of the most time-consuming model is just over 1 hour, while the shortest one only takes 15 minutes. Compared with the conventional model training strategy (such as training each base learner for 100 epochs), the model training time is shortened from the shortest half a day to less than an hour, and some models can even be fitted in just a few epochs. The early stopping method terminates unnecessary training iterations in advance, significantly reducing the time and computing resources required for model training.

[0081] Furthermore, combining Table 5 and Table 6, it can be found that the prediction accuracy of ResNet50 has increased from 68.56% to 71.40%, which is the largest improvement among the seven models. The prediction accuracy of ResNet50 after applying the early stopping method also exceeds the target accuracy of 71% of the subject. For deep neural network models, the more parameters there are, the stronger the expressive power of the model is, and it can learn more complex functional relationships to adapt to the details in the training data. High-complexity models have greater flexibility and can accurately fit various specific features including noise during training. When the number of model parameters is too large, the model may have enough degrees of freedom to "memorize" individual samples or noise in the training set, rather than learning the general rules behind the data. This over-adaptation to the training data increases the risk that the model will not be able to generalize well when faced with new data, that is, overfitting occurs. Table 7 gives the parameter values ​​of the above 7 models. ResNet50 has 23.58M parameters, which is the most complex model with the most parameters. Therefore, ResNet50 has a higher probability of overfitting. When the early stopping method is applied to ResNet50, the overfitting phenomenon is alleviated more obviously, and the model is improved more.

[0082] Table 7 Model parameters

[0083] Finally, a comparison between the method provided in the above embodiment and the conventional method is given, as shown in Table 8. The base learner combinations corresponding to different ensembles can be found in Table 2.

[0084] Table 8 Comparison with conventional methods

[0085] It can be found that the method provided in this application has achieved an improvement over conventional methods, with the highest recognition accuracy reaching 81.46%. On the basis of the improved accuracy of ResNet18 (2.0) that implements the featureConv operation, the accuracy of the integrated model (i.e., the posture recognition model) based on the integrated learning strategy has been greatly improved.

[0086] It can be understood that the options in the above-mentioned embodiment 1 are also applicable to this embodiment, so they will not be described again here.

[0087] The same or similar reference numerals correspond to the same or similar components; The terms used to describe the positional relationship in the drawings are only used for illustrative purposes and should not be construed as limiting the present application; It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application may be combined with each other.

[0088] In different specific implementations, the method or system described in the present application can be implemented in software, hardware or a combination thereof. In addition, the order of the steps of the method can be changed, and various elements can be added, reordered, combined, omitted, modified, etc.

[0089] Obviously, the above-mentioned embodiments of the present application are merely examples for clearly illustrating the present application, and are not intended to limit the implementation methods of the present application, and are not intended to limit the present application. For those of ordinary skill in the art, other different forms of changes or modifications can be made based on the above description. Each discrete structure / function module or unit can be integrated together to form an independent part, or each module can exist separately, or two or more modules can be integrated to form an independent part, and the structure and function of the discrete components can be implemented as a combined structure or component. It is not necessary and impossible to enumerate all the implementation methods here. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application should be included in the scope of protection of the claims of the present application.

Claims

1. A gesture recognition method based on wireless sensing, characterized in that: include: Receive BVP sample data; A first posture recognition result is determined based on the BVP sample data using a posture recognition model; wherein the posture recognition model determines the first posture recognition result by fusing the second posture recognition results output by at least two base learners based on the BVP sample data based on an integrated learning strategy, and the base learners are trained based on the early stopping method.

2. A gesture recognition method based on wireless sensing according to claim 1, characterized in that: The base learner includes at least one of an MLP model, an LSTM model, a BiLSTM model, a ResNet18 model, a ResNet50 model and a ViT model.

3. A gesture recognition method based on wireless sensing according to claim 2, characterized in that: When the base learner is a ResNet18 model, before inputting the BVP sample data into the ResNet18 model, a reshape operation is performed on the BVP sample data, including: The BVP sample data is sequentially passed through the first deconvolution layer and the second deconvolution layer; wherein, The first deconvolution layer includes a first number of convolution kernels with a first step length and a first size, a padding size of a first padding amount, and uses a ReLU activation function as an activation function; and, The second deconvolution layer includes a second number of convolution kernels with a second step size and a second size, a padding size of a second padding amount, and uses a ReLU activation function as an activation function.

4. The method for gesture recognition based on wireless sensing according to claim 2, characterized in that: When the base learner is a ResNet18 model, before the BVP sample data is input into the ResNet18 model, a featureConv operation is performed on the BVP sample data, including: Performing a dimension permutation on the BVP sample data, and subjecting the BVP sample data after the dimension permutation to convolution operations through a first convolution layer and a second convolution layer in sequence, to obtain a first intermediate tensor; After performing a second dimension permutation on the first intermediate tensor, the first intermediate tensor is again subjected to a convolution operation through the first convolution layer and the second convolution layer in sequence to obtain a second intermediate tensor; After performing three-dimensional permutation on the second intermediate tensor, a third intermediate tensor is obtained as the input of the ResNet18 model, and the number of input channels of the ResNet18 model is adapted to the dimension of the third intermediate tensor.

5. A gesture recognition method based on wireless sensing according to claim 4, characterized in that: The first convolution layer includes a third number of convolution kernels with a third step size and a third size, a padding size of a third padding amount, and uses a ReLU activation function as an activation function; and, The second convolution layer includes a fourth number of convolution kernels with a fourth step size and a fourth size, a padding size of a fourth padding amount, and uses a ReLU activation function as an activation function.

6. A gesture recognition method based on wireless sensing according to claim 5, characterized in that: The number of input channels of the first convolutional layer is 20, the third number is 11, the third step length is 1, the third size is 7*7, and the third padding amount is 3; The number of output channels of the second convolutional layer is 5, the fourth number is 5, the fourth step size is 1, the fourth size is 5*5, and the fourth padding amount is 2.

7. A gesture recognition method based on wireless sensing according to any one of claims 1 to 6, characterized in that: The posture recognition model adopts soft voting for integrated learning, including: receiving a second posture recognition result output by each of the base learners; Calculate the prediction probability corresponding to each second posture recognition result using softmax; Grouping is performed according to whether the second posture recognition results are the same, and calculating the mean of the predicted probabilities corresponding to the second posture recognition results as the grouping probability; The second posture recognition result corresponding to the largest grouping probability is selected as the first posture recognition result.

8. The method for gesture recognition based on wireless sensing according to claim 7, characterized in that: The selection process of the base learner in the posture recognition model includes: Receiving the second posture recognition result output by each candidate base learner on the test set; wherein the test set includes the BVP sample data for testing and the corresponding true category label; Determining the prediction accuracy of each of the candidate base learners on the test set according to the second posture recognition result and the true category label; A preset number of the candidate base learners are selected according to the prediction accuracy and combined into the posture recognition model.

9. The method for gesture recognition based on wireless sensing according to claim 8, characterized in that: Training the candidate base learner based on the early stopping method includes: Establishing the initial candidate base learner, initializing the counter, the optimal loss value, the loss value change threshold and the patience value; Iteratively train the candidate base learners on the training set to update the model parameters, and After every T training rounds: Determining a validation loss value of the candidate base learner on a validation set; According to whether the optimal loss value is an initialization value or whether the difference between the optimal loss value and the verification loss value satisfies the loss value change threshold, updating the optimal loss value based on the verification loss value and reinitializing the counter; When the difference between the optimal loss value and the verification loss value does not meet the loss value change threshold, the counter is incremented to end the current round, and, Whether to terminate the training is determined according to whether the value of the counter satisfies the patience value; when the value of the counter satisfies the patience value, the training is stopped and the model parameters corresponding to the candidate base learner of the current round are saved.

10. A computer program product comprising a computer program or computer executable instructions, characterized in that: When the computer program or computer executable instructions are executed by a processor, the method according to any one of claims 1 to 9 is implemented.

Images