Human body behavior recognition method and system based on space-time decoupling multi-scale feature fusion

Abstract

The invention discloses a human body behavior recognition method and system based on space-time decoupling multi-scale feature fusion. The system comprises a local space-time multi-scale feature fusion network MDCNN and a space-time decoupling network DTST. Extracting different scale features of multichannel sensor data by using a multi-scale convolutional neural network, extracting shallow layer features and deep layer features of signals at the same time, and extracting features representing action information by fully utilizing complementary information among different level features; the method comprises the following steps of: decoupling a network into a time decoupling structure and a space decoupling structure through a time-space decoupling network based on Transform, so as to accurately extract time-space characteristics in an inertial signal; and finally, using a Softmax classifier to carry out action classification and recognition. According to the method, consistency and complementarity among multi-sensor data can be efficiently utilized, multi-scale spatial-temporal characteristics are fused, changes in time dimensions and relevant information in spatial positions are fully considered, different attention degrees are kept for signals of different sensing channels, and the recognition precision in human behavior recognition is improved.

Description

Background Art

[0002] With the continuous improvement and advancement of modern technology, behavior recognition has been expanded to the use of wearable sensors. Sensors can collect multimodal data of users. Scholars at home and abroad have carried out many studies related to this aspect.

[0003] However, traditional behavior recognition methods often rely on manual feature extraction, and the models rely on statistical features and distribution features, which greatly increases the labor cost and limits the scope of application of human behavior recognition methods. With the rapid progress of deep learning technology, more and more researchers have begun to explore the use of deep learning models to process heterogeneous time series data from sensors. Deep learning technology has shown remarkable results in the field of behavior recognition based on wearable sensors. It greatly improves recognition accuracy and efficiency by automatically extracting features from complex data. However, although deep learning models have made progress in dealing with such problems, there are still some blind spots in current research. Many research works do not fully consider the inherent spatiotemporal feature information of one-dimensional inertial signals, that is, the changes in the signal in the time dimension and the related information in the spatial position. In addition, the signals of different sensor channels may have different effects and contributions to the final action classification results, which is often overlooked in current research. Summary of the invention

[0004] The technical problem to be solved by the present invention is to provide a human behavior recognition method and system based on spatiotemporal decoupling and multi-scale feature fusion in response to the above-mentioned deficiencies in the prior art, so as to solve the technical problems that the existing sensor-based behavior recognition methods do not give sufficient consideration to the changes in the time dimension and the related information in the spatial position, and always maintain the same attention to the signals of different sensor channels.

[0005] The present invention adopts the following technical solutions: A human behavior recognition method based on spatiotemporal decoupling multi-scale feature fusion includes the following steps: S1. A linear local difference embedding algorithm is used to obtain a low-dimensional dynamic feature set, which is then input into a multi-scale convolutional neural network to extract multi-scale deep features from inertial signals, and multi-scale fusion features are output by splicing feature matrices; S2, setting the relevant parameters of the multi-scale convolutional neural network, formulating the Transformer spatiotemporal decoupling module stacking strategy, inputting the multi-scale fusion features output from step S1 into the Transformer-based spatiotemporal decoupling network model, and extracting the spatiotemporal features; S3. Input the spatiotemporal feature matrix obtained in step S2 into the softmax classifier to complete the training of the human behavior recognition model and realize human behavior recognition.

[0006] Preferably, step S1 specifically includes: S101, extracting the time-frequency domain features of the data set using short-time Fourier transform to obtain the original high-dimensional action feature set; S102, using a nonlinear dimensionality reduction method to reduce dimensionality and remove redundant information; S103, inputting the feature set after dimension reduction into a multi-scale convolutional neural network, and outputting multi-scale fusion features.

[0007] Preferably, the multi-scale convolutional neural network includes three convolutional layers, and the three convolutional layers select convolution kernels of different sizes; different branches include three convolutional layers; the number of convolution kernels in different convolutional layers in each branch is 16, 32, and 64 respectively; branch one uses a convolution kernel size of 1*3, branch two uses a convolution kernel size of 5*5, and branch three uses a convolution kernel size of 7*7; Batch Normalization, Max-Pool and Dropout layers are introduced at the same time.

[0008] Preferably, the sensor data of different parts of the human body are divided into multiple independent channels as the input of the multi-scale convolutional neural network. Assuming that the sensor signal contains M independent channels, the output of channel i at time t is recorded as , then the output of multiple wearable sensors at time t is .

[0009] Preferably, step S2 specifically includes: Suppose the fusion feature matrix processed by the multi-scale feature fusion network is recorded as ,Will Divide into The grouped fusion features are used as the input of the spatial decoupling module. The multi-scale fusion features are input into the spatiotemporal decoupling network model, and multiple attention mechanisms are used for parallel calculation to obtain the final output. ; Use a convolutional feedforward network to get the output , the input of the convolutional feedforward network and output After sequence addition and layer normalization, the final features extracted by the spatiotemporal decoupling encoder module are obtained. .

[0010] Preferably, the spatiotemporal decoupling network model introduces N alternately stacked spatiotemporal Transformer encoder modules to extract the spatiotemporal dependency information in the fused feature sequence; each encoder module includes a multi-head self-attention mechanism with a residual structure and a convolutional feed-forward layer.

[0011] Preferably, multiple attention mechanisms are calculated in parallel as follows: Suppose there are h attention heads; First, for each attention head, a linear transformation is performed using different weight matrices to obtain the query matrix Q, key matrix K, and value matrix V; Then use the attention mechanism to calculate the output of each attention head, concatenate the output of each attention head and perform a linear transformation to get the final output , as follows:

[0012] in, is the output of the attention head, Concatenate the output vectors calculated by each attention head to form a long vector. is the weight matrix of the linear transformation.

[0013] Preferably, the feature Specifically:

[0014] in, is the input of the multi-head self-attention module, is the output of the convolutional feed-forward network.

[0015] Preferably, the input of the multi-head self-attention module for:

[0016] Output of a convolutional feedforward network Specifically:

[0017] in, is the weight parameter of the convolution kernel, is the weight parameter of the convolution kernel, is the output result of the first layer, It is a one-dimensional convolution operation, which convolves the input data with the convolution kernel. For batch processing, each batch of data is normalized.

[0018] In a second aspect, an embodiment of the present invention provides a human behavior recognition system based on spatiotemporal decoupling multi-scale feature fusion, including: The splicing module uses a linear local difference embedding algorithm to obtain a low-dimensional dynamic feature set, which is then input into a multi-scale convolutional neural network to extract multi-scale deep features from inertial signals and output multi-scale fusion features by splicing feature matrices; The extraction module sets the relevant parameters of the multi-scale convolutional neural network, formulates the Transformer spatiotemporal decoupling module stacking strategy, inputs the output multi-scale fusion features into the Transformer-based spatiotemporal decoupling network model, and extracts the spatiotemporal features; The recognition module inputs the spatiotemporal feature matrix into the softmax classifier to complete the training of the human behavior recognition model and realize human behavior recognition.

[0019] In a third aspect, a computer device comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the steps of the above-mentioned method for human behavior recognition based on spatiotemporal decoupling and multi-scale feature fusion are implemented.

[0020] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, including a computer program, which, when executed by a processor, implements the steps of the above-mentioned human behavior recognition method based on spatiotemporal decoupling and multi-scale feature fusion.

[0021] Compared with the prior art, the present invention has at least the following beneficial effects: A human behavior recognition method based on spatiotemporal decoupling multi-scale feature fusion is proposed. First, the multi-scale feature fusion network MDCNN is used to extract feature information of different scales in multi-channel inertial signals to obtain a fusion feature matrix. Then, DTST decoupling is used to extract spatiotemporal features in the fusion feature matrix to accurately describe its spatiotemporal dependency. This method efficiently utilizes the correlation between multi-sensor data, and solves the problem that existing sensor-based behavior recognition methods do not consider changes in the time dimension and related information in the spatial position sufficiently, and always pay the same attention to signals from different sensor channels.

[0022] Furthermore, the setting of step S1 refines and optimizes the input features layer by layer through three steps: time-frequency domain feature extraction, nonlinear dimensionality reduction, and multi-scale convolutional neural network feature extraction, providing the model with high-quality multi-scale fusion features. First, short-time Fourier transform is used to extract time-frequency domain features. The extracted time-frequency domain features contain the dynamic change information of behavioral actions and can reflect the characteristic frequencies of different actions in the time dimension. Then, nonlinear dimensionality reduction is used to remove redundant information, thereby improving the efficiency and generalization ability of the model. Finally, multi-scale convolutional neural network is used to extract multi-scale deep features, which can find short-term and long-term dependencies of behavioral actions in low-dimensional feature sets, fuse multiple scale features, and provide rich input information for the subsequent Transformer spatiotemporal decoupling model. This method can improve the accuracy of human behavior recognition, while effectively reducing the computational cost, so that the model has strong robustness when processing diversified actions.

[0023] Furthermore, the multi-scale convolutional neural network is set up to extract features from different scales through convolution kernels of different sizes, and enhance the generalization and robustness of the model through regularization and pooling operations. Through the combination of different convolution kernel sizes, branch structures, batch normalization, maximum pooling and Dropout, the computational efficiency, generalization ability and noise resistance of the model are ensured while extracting multi-scale features. This design enables the network to more accurately capture the multi-level features of human behavior and improve the accuracy and robustness of behavior recognition.

[0024] Furthermore, the sensor data of different parts of the human body are divided into multiple independent channels as the input of the multi-scale convolutional neural network in order to better capture the dynamic information of different parts of the human body during behavior and enhance the representation ability of features. Dividing the sensor data of different parts of the human body into independent channels and aggregating them at each time step can effectively improve the sensitivity of the multi-scale convolutional network to features of different parts and scales, providing a clearer and richer feature expression for subsequent behavior recognition. This structural design helps to improve the recognition accuracy and robustness of the model.

[0025] Furthermore, the setting of step S2 aims to effectively extract the spatiotemporal features in human behavior through spatiotemporal decoupling and attention mechanism, thereby improving the recognition accuracy and robustness of the model for complex behaviors. First, the division and regional grouping of multi-scale fusion features are used. This division method helps the model to identify and locate the action features of specific spatial regions, especially in different body parts or action patterns, so that the decoupled features can be more focused on local action features and detail changes, and improve the recognition effect of complex actions. Next, through spatial decoupling, the multi-scale features are input into the spatiotemporal decoupling network model based on Transformer, and the spatiotemporal features are calculated in parallel using the attention mechanism to enhance the model's perception of different time steps and spatial regions, and improve the model's accuracy and generalization ability. Then, the fused spatiotemporal features are processed by the convolutional feedforward network to obtain rich deep features, which are added to the original input sequence to enhance the spatial continuity of the features while retaining local information. After adding the input and output sequences of the network, it helps to retain the important content in the input information and avoid information loss. At the same time, the convolution operation helps to smooth and denoise the spatiotemporal features, and the features finally extracted are more robust and stable. Layer normalization is performed after the convolutional feedforward network to stabilize feature distribution and improve the training efficiency and convergence speed of the model.

[0026] Furthermore, the introduction of N alternately stacked spatiotemporal Transformer encoder modules is to fully capture the complex spatiotemporal dependency information in human behavior. The multi-head self-attention mechanism focuses on a variety of spatiotemporal features through different heads, avoiding the model from focusing too much on local features and ignoring the overall dynamics, and strengthening the model's ability to understand details and global features. In addition, the introduction of the residual structure can ensure the stability of the attention mechanism, avoid gradient vanishing, and improve the training effect. At the same time, the convolutional feedforward layer processes features through convolution operations to ensure that local key information is retained during the spatiotemporal feature transmission process. This local information is particularly important for human behavior recognition, because behavior recognition usually relies on subtle local changes. The residual structure of the convolutional feedforward layer can also effectively reduce information loss, making feature transmission smoother. . This module design realizes the hierarchical decoupling and focused processing of complex behaviors, so that the human behavior recognition system shows higher accuracy and robustness in dynamic scenes.

[0027] Furthermore, in order to improve the model's ability to understand human behavior characteristics from multiple angles, multiple attention mechanisms are parallelly calculated through a multi-head self-attention mechanism, which captures different spatiotemporal relationships through multiple independent attention heads, thereby extracting richer and more diverse feature information during the recognition process. This mechanism can enrich the expressiveness of features and maintain information integrity through splicing and linear transformation, ultimately improving the robustness and accuracy of the model in human behavior recognition tasks.

[0028] It can be understood that the beneficial effects of the second aspect mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here.

[0029] In summary, the present invention can efficiently utilize the consistency and complementarity between multi-sensor data, integrate multi-scale spatiotemporal features, fully consider the changes in the time dimension and the related information in the spatial position, maintain different levels of attention to the signals of different parts of the body and different sensor channels, and improve the recognition accuracy in human behavior recognition.

[0030] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings to be used in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0032] Figure 1 It is a multi-scale convolutional neural network; Figure 2 Decoupled networks for time and space; Figure 3 A schematic diagram of a computer device provided by an embodiment of the present invention; Figure 4 The present invention is a block diagram of an electronic device provided according to an embodiment. DETAILED DESCRIPTION

[0033] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0034] In the description of the present invention, it should be understood that the terms “include” and “comprises” indicate the presence of described features, wholes, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or collections thereof.

[0035] It should also be understood that the terms used in the present specification are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the present specification and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0036] It should be further understood that the term "and / or" used in the present specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes these combinations. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in the present invention generally indicates that the associated objects are in an "or" relationship.

[0037] It should be understood that, although the terms first, second, third, etc. may be used to describe preset ranges, etc. in the embodiments of the present invention, these preset ranges should not be limited to these terms. These terms are only used to distinguish preset ranges from each other. For example, without departing from the scope of the embodiments of the present invention, the first preset range may also be referred to as the second preset range, and similarly, the second preset range may also be referred to as the first preset range.

[0038] The word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrases "if it is determined" or "if (stated condition or event) is detected" may be interpreted as "when it is determined" or "in response to determining" or "when detecting (stated condition or event)" or "in response to detecting (stated condition or event)", depending on the context.

[0039] Various structural schematic diagrams of the embodiments disclosed in the present invention are shown in the accompanying drawings. These figures are not drawn to scale, and some details are magnified and some details may be omitted for the purpose of clear expression. The shapes of various regions and layers shown in the figures and the relative sizes and positional relationships therebetween are only exemplary, and may deviate in practice due to manufacturing tolerances or technical limitations, and those skilled in the art may additionally design regions / layers with different shapes, sizes, and relative positions according to actual needs.

[0040] The present invention provides a method for human behavior recognition based on spatiotemporal decoupling multi-scale feature fusion, including a local spatiotemporal multi-scale feature fusion network MDCNN and a spatiotemporal decoupling network DTST. A multi-scale convolutional neural network (MDCNN) is used to extract different scale features of multi-channel sensor data, and at the same time, shallow features and deep features of the signal are extracted, and the complementary information between features at different levels is fully utilized to extract features that represent action information; then, through a Transformer-based spatiotemporal decoupling network (Decoupling Temporal-Spatially Transformer, DTST), the network is decoupled into a temporal decoupling and a spatial decoupling structure, so as to accurately extract spatiotemporal features in inertial signals; finally, a Softmax classifier is used for action classification and recognition. The method of the present invention can efficiently utilize the consistency and complementarity between multi-sensor data, fuse multi-scale spatiotemporal features, fully consider the changes in the time dimension and the relevant information in the spatial position, maintain different attention to the signals of different sensor channels, and improve the recognition accuracy in human behavior recognition.

[0041] The present invention provides a human behavior recognition method based on spatiotemporal decoupling multi-scale feature fusion, comprising the following steps: S1. The low-dimensional dynamic feature set obtained by the Locally Linear Embedding (LLE) algorithm is used as the input of the Multi-Dimensione Convolutional Neural Network (MDCNN) to extract multi-scale deep features from inertial signals to exploit the consistency and complementarity between multi-sensor data. Finally, the multi-scale fusion features are output by splicing the feature matrix. See also Figure 1 , the MDCNN module contains three convolutional layers, and the three convolutional layers select convolution kernels of different sizes. The structures of different branches are similar, all including three convolutional layers; the number of convolution kernels in different convolutional layers in each branch is 16, 32, and 64 respectively. Branch 1 uses a convolution kernel size of 1*3, branch 2 uses a convolution kernel size of 5*5, and branch 3 uses a convolution kernel size of 7*7; at the same time, in order to prevent overfitting and underfitting, Batch Normalization, Max-Pool, and Dropout layers are introduced into the network to reduce the complexity of the network and improve the generalization ability of the network.

[0042] S101, using short-time Fourier transform (STFT) to extract the time-frequency domain features of the data set to obtain the original high-dimensional action feature set; S102. Use the nonlinear dimensionality reduction method (Locally Linear Embedding, LLE) to reduce the dimension and remove redundant information; S103. Use the reduced feature set as the input of MDCNN.

[0043] MDCNN is used to extract feature information from inertial signals. It aims to capture channel features of inertial signals at different scales through different branches, and perform cross-scale information transmission and fusion to prevent the loss of sample information in feature extraction, which is conducive to realizing information interaction along the time and space dimensions in subsequent Transformer blocks.

[0044] In addition, considering the heterogeneity of multi-channel data, the sensor data of different parts of the human body contribute differently to the recognition results. Therefore, the sensor data of different parts of the human body are divided into multiple independent channels as the input of the MDCNN module. Assume that the sensor signal contains M independent channels, and the output of channel i at time t is recorded as , then the output of multiple wearable sensors at time t is .

[0045] S2. Set network-related parameters, formulate the Transformer spatiotemporal decoupling (TDB / SDB) module stacking strategy, input the multi-scale fusion features output from step S1 into the spatiotemporal decoupling network model (Decoupling Temporal-SpatiallyTransformer, DTST), and use the characteristics of DTST decoupling to reduce the model calculation complexity and the advantages of deep learning of spatiotemporal feature information to extract spatiotemporal features; See also Figure 2 , let the fusion feature matrix processed by the multi-scale feature fusion network be recorded as , which is divided into area, namely Each region of , There are two ways to group and fuse features: One way is to group along the time dimension, that is, , , as the input of the time decoupling module.

[0046] Another way to group along the spatial dimension is , , as the input of the spatial decoupling module.

[0047] The multi-head attention mechanism of the temporal decoupling and spatial decoupling modules is the same, and the network is input alternately according to different feature grouping methods to search different areas.

[0048] The features are input into the DTST network. The DTST network introduces N alternately stacked spatiotemporal Transformer encoder modules to further extract the spatiotemporal dependency information in the fusion feature sequence. Each encoder module includes a multi-head self-attention mechanism with a residual structure and a convolutional feedforward layer.

[0049] In order to enhance the model's ability to focus on different positions, multiple attention mechanisms are used for parallel calculation, also known as multi-head attention mechanisms. Suppose there are h attention heads. First, for each attention head, a linear transformation is performed using different weight matrices to obtain the query matrix Q, key matrix K, and value matrix V. Then, the attention mechanism is used to calculate the output of each attention head, and the output of each attention head is concatenated and linearly transformed to obtain the final output: (1) in, .

[0050] In order to alleviate the gradient vanishing problem caused by module stacking and keep the input sequence information intact, the multi-head self-attention module adopts residual connection, so The output of the multi-head self-attention is added to the input sequence and layer normalization (LN) is performed before inputting the feed-forward layer with residual connection. The input is recorded as: (2) In order to further extract local information in the sequence, the present invention uses a convolutional feedforward network (CFN) module; CFN includes two 1D convolutional layers, each followed by a BN layer and a ReLU layer. In order to prevent overfitting, a Dropout layer is added after the second convolutional layer.

[0051] The output of the entire CFN is denoted as , the calculation process is as follows: (3) in, , .

[0052] The output of the CFN module is added to the input sequence and layer normalized to obtain the final feature extracted by the spatiotemporal decoupling encoder module, denoted as , the calculation process is as follows: (4) S3. Use the spatiotemporal feature matrix output by DTST as the input of the softmax classifier to complete the recognition of human behavior.

[0053] In the Windows 10 operating system and GTX1660 graphics card environment, Tensorflow2 is used as the deep learning framework for training. Specifically, the original high-dimensional action features are reduced in dimension through LLE, and multi-scale fusion features are obtained through S1. The spatiotemporal features contained in the inertial signal in the decoupled fusion features are input into S2, and finally S3 outputs the action classification label. By comparing with the original label, the loss of the model is calculated, and then the Adam optimizer adjusts the network weights layer by layer based on error back propagation to finally obtain the best training weight file. Through this weight file, combined with S1, S2, and S3, effective behavior recognition can be achieved.

[0054] It will be appreciated by those skilled in the art that various aspects of the present invention may be implemented as systems, methods or program products. Therefore, various aspects of the present invention may be specifically implemented in the following forms, namely: complete hardware implementation, complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software implementations, which may be collectively referred to herein as "circuits", "modules" or "platforms".

[0055] In another embodiment of the present invention, a human behavior recognition system based on spatiotemporal decoupling multi-scale feature fusion is provided. The system can be used to implement the above-mentioned human behavior recognition method based on spatiotemporal decoupling multi-scale feature fusion. Specifically, the human behavior recognition system based on spatiotemporal decoupling multi-scale feature fusion includes a splicing module, an extraction module and a recognition module.

[0056] Among them, the splicing module uses a linear local difference embedding algorithm to obtain a low-dimensional dynamic feature set, which is then input into a multi-scale convolutional neural network to extract multi-scale deep features from inertial signals, and outputs multi-scale fusion features through the splicing of feature matrices; The extraction module sets the relevant parameters of the multi-scale convolutional neural network, formulates the Transformer spatiotemporal decoupling module stacking strategy, inputs the output multi-scale fusion features into the Transformer-based spatiotemporal decoupling network model, and extracts the spatiotemporal features; The recognition module inputs the spatiotemporal feature matrix into the softmax classifier to complete the training of the human behavior recognition model and realize human behavior recognition.

[0057] In another embodiment of the present invention, a terminal device is provided, which includes a processor and a memory, wherein the memory is used to store a computer program, the computer program includes program instructions, and the processor is used to execute the program instructions stored in the computer storage medium. The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. It is the computing core and control core of the terminal, which is suitable for implementing one or more instructions, and is specifically suitable for loading and executing one or more instructions to implement the corresponding method flow or corresponding function; the processor described in the embodiment of the present invention can be used for the operation of the human behavior recognition method based on spatiotemporal decoupling multi-scale feature fusion, including: A linear local difference embedding algorithm is used to obtain a low-dimensional dynamic feature set, which is then input into a multi-scale convolutional neural network to extract multi-scale deep features from inertial signals, and multi-scale fusion features are output by splicing feature matrices. The relevant parameters of the multi-scale convolutional neural network are set, and a Transformer spatiotemporal decoupling module stacking strategy is formulated. The output multi-scale fusion features are input into a Transformer-based spatiotemporal decoupling network model to extract spatiotemporal features. The obtained spatiotemporal feature matrix is ​​input into a softmax classifier to complete the training of the human behavior recognition model and realize human behavior recognition.

[0058] In another embodiment of the present invention, the present invention also provides a storage medium, specifically a computer-readable storage medium (Memory), which is a memory device in a terminal device for storing programs and data. It is understandable that the computer-readable storage medium here can include both the built-in storage medium in the terminal device and the extended storage medium supported by the terminal device, and can be any tangible medium containing or storing a program, which can be used by an instruction execution system, device or device or used in combination with it. The computer-readable storage medium provides a storage space, which stores the operating system of the terminal. In addition, one or more instructions suitable for being loaded and executed by the processor are also stored in the storage space, and these instructions can be one or more computer programs (including program codes). It should be noted that more specific examples (non-exhaustive list) of the computer-readable storage medium here include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0059] Computer readable storage media also include data signals propagated in baseband or as part of a carrier wave, which carry readable program codes. Such propagated data signals can take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The readable storage medium can also be any readable medium other than a readable storage medium, which can send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, device, or device. The program code contained on the readable storage medium can be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the above.

[0060] Program code for performing the operations of the present invention may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., through the Internet using an Internet service provider).

[0061] The processor may load and execute one or more instructions stored in a computer-readable storage medium to implement the corresponding steps of the human behavior recognition method based on spatiotemporal decoupling multi-scale feature fusion in the above embodiment; the processor may load and execute the following steps: A linear local difference embedding algorithm is used to obtain a low-dimensional dynamic feature set, which is then input into a multi-scale convolutional neural network to extract multi-scale deep features from inertial signals, and multi-scale fusion features are output by splicing feature matrices. The relevant parameters of the multi-scale convolutional neural network are set, and a Transformer spatiotemporal decoupling module stacking strategy is formulated. The output multi-scale fusion features are input into a Transformer-based spatiotemporal decoupling network model to extract spatiotemporal features. The obtained spatiotemporal feature matrix is ​​input into a softmax classifier to complete the training of the human behavior recognition model and realize human behavior recognition.

[0062] See also Figure 3 , the terminal device is a computer device, and the computer device 60 of this embodiment includes: a processor 61, a memory 62, and a computer program 63 stored in the memory 62 and executable on the processor 61. When the computer program 63 is executed by the processor 61, the human behavior recognition method based on spatiotemporal decoupling multi-scale feature fusion in the embodiment is implemented. To avoid repetition, it is not described one by one here. Alternatively, when the computer program 63 is executed by the processor 61, the functions of each model / unit in the human behavior recognition system based on spatiotemporal decoupling multi-scale feature fusion in the embodiment are implemented. To avoid repetition, it is not described one by one here.

[0063] The computer device 60 may be a computing device such as a desktop computer, a notebook, a PDA, or a cloud server. The computer device 60 may include, but is not limited to, a processor 61 and a memory 62. Those skilled in the art will appreciate that Figure 3This is only an example of the computer device 60 and does not constitute a limitation of the computer device 60. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the computer device may also include input and output devices, network access devices, buses, etc.

[0064] The processor 61 may be a central processing unit (CPU), or other general-purpose processors, central processing units, graphics processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, data processing logic based on quantum computing, discrete hardware components, etc. The general-purpose processor may be a microprocessor or the processor may be any conventional processor, etc.

[0065] The memory 62 may be an internal storage unit of the computer device 60, such as a hard disk or memory of the computer device 60. The memory 62 may also be an external storage device of the computer device 60, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc., equipped on the computer device 60.

[0066] Furthermore, the memory 62 may include both an internal storage unit of the computer device 60 and an external storage device. The memory 62 is used to store computer programs and other programs and data required by the computer device. The memory 62 may also be used to temporarily store data that has been output or is to be output.

[0067] Any reference to a memory, database or other medium used in the embodiments provided in the present application may include at least one of a non-volatile and a volatile memory. Non-volatile memory may include a read-only memory (ROM), a magnetic tape, a floppy disk, a flash memory, an optical memory, a high-density embedded non-volatile memory, a resistive random access memory (ReRAM), a magnetic random access memory (MRAM), a ferroelectric random access memory (FRAM), a phase change memory (PCM), a graphene memory, etc. Volatile memory may include a random access memory (RAM) or an external cache memory, etc. As an illustration and not limitation, RAM may be in various forms, such as a static random access memory (SRAM) or a dynamic random access memory (DRAM), etc.

[0068] The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. The non-relational database may include a distributed database based on blockchain, etc., but is not limited thereto. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but is not limited thereto.

[0069] See also Figure 4 The terminal device is an electronic device 600, and the electronic device is in the form of a general-purpose computing device. The components of the electronic device may include, but are not limited to: at least one processing unit 610, at least one storage unit 620, a bus 630 connecting different platform components (including the storage unit 620 and the processing unit 610), a display unit 640, etc.

[0070] The storage unit stores program codes, which can be executed by the processing unit 610, so that the processing unit 610 performs the steps according to various exemplary embodiments of the present invention described in the above method section of this specification. For example, the processing unit 610 can perform the following steps: Figure 1 Follow the steps shown in .

[0071] The storage unit 620 may include a readable medium in the form of a volatile storage unit, such as a random access storage unit (RAM) 6201 and / or a cache storage unit 6202 , and may further include a read-only storage unit (ROM) 6203 .

[0072] The storage unit 620 may also include a program / utility 6204 having a set (at least one) of program modules 6205, such program modules 6205 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0073] Bus 630 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0074] The electronic device 600 may also communicate with one or more external devices 700 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 600, and / or communicate with any device that enables the electronic device 600 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed via an input / output (I / O) interface 650. Furthermore, the electronic device 600 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) via a network adapter 660. The network adapter 660 may communicate with other modules of the electronic device 600 via a bus 630. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 600, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage platforms, etc.

[0075] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all of the embodiments. The components of the embodiments of the present invention described and shown in the drawings here can usually be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the claimed invention, but merely represents selected embodiments of the present invention. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0076] Example 1. Experimental Data The experimental data adopts the UCI-MHEALTH public data set with a sampling frequency of 50Hz. The sensors are placed on the chest, right wrist and left ankle of the volunteers. The data set contains three types of sensor data: accelerometer, gyroscope and magnetometer. The present invention selects 8 types of actions as shown in Table 1.

[0077] Table 1 Number and quantity of various types of actions in UCI-MHEALTH

[0078] The experiment extracts the time-frequency domain features of the sensor data. For a sensor in a body part, 69 features are extracted, including feature vectors such as standard deviation, spectral coefficient, and root mean square. The sensors used in this data set are placed in three parts of the body, so a total of 207-dimensional time-frequency domain features are extracted. The experiment of the present invention uses 8 common actions in this data set, and randomly divides the 8401 data of this data set into a training set (70%) and a test set (30%) through a program for experimentation. The number of samples in the experimental data set is shown in Table 2.

[0079] Table 2 Dataset details

[0080] 2. Experimental Results Analysis For the DTST model, the depth of the model can be increased by stacking multiple Transformer encoder modules. Deep models have more layers and nonlinear transformations, and can better capture complex relationships and patterns in input sequences. When there are too many stacked modules, the computational complexity of the model will increase, and when there are too few stacked modules, the feature learning will be incomplete, affecting the recognition effect of the model. By referring to relevant literature, the number of all stacked modules is determined to be 8, and an ablation study of the impact of different stacking strategies is conducted to evaluate their contribution to the entire model. The experimental results are shown in Table 3. "T" represents the temporal decoupling block (TDB); "S" represents the spatial decoupling block (SDB); Xn represents n repetitions. " ” indicates the change in the F1 value of the stacking strategy compared with the best result. The results of comparing different stacking strategies combining TDB and SDB are shown in Table 3.

[0081] Table 3 Stacking strategy

[0082] By comparing the first and fourth rows of Table 3, both models are composed of a TDB and an SDB stacked four times alternately, and both have good F1 values, but the model starting from SDB performs worse than the model starting from TDB. Compared with the best T1S1-X4 model, the F1 value of S1T1-X4 drops by 0.01. By comparing the second and fifth rows, and the third and sixth rows in Table 3, similar phenomena appear, that is, the F1 value of the model starting from TDB is higher than that of the model starting from SDB. This is because starting from TBD can borrow feature information from other time steps in the first Transformer block, taking advantage of the time-related characteristics of sensor data, and there is a dependency relationship between the data of adjacent time steps, which is a good starting point. However, the model starting from SDB can only search for other patches in its own module in the first layer, resulting in incomplete extracted spatial feature information.

[0083] By first extracting advanced time-related features from the time series through TDB, and then further extracting space-related features based on these features through SDB, and finally alternately stacking these two modules, the time correlation and space correlation can be fully utilized to achieve multi-level feature extraction and improve computational efficiency and recognition rate. Therefore, the present invention selects T1S1-X4 as the stacking strategy of the spatiotemporal decoupling network model.

[0084] In order to verify the effectiveness of the model proposed in the present invention for human behavior recognition, this experiment is compared with different networks in the model. In the experiment, the number and size of convolution kernels used in each branch of CNN and MDCNN are the same. The recognition results obtained from the experiment are shown in Table 4.

[0085] Table 4 Recognition effects of different branches

[0086] As shown in Table 4, MDCNN-DTST is better than CNN, DTST and CNN-DTST. In CNN, the behavior recognition accuracy is the highest when the convolution kernel size is 5*5, and in CNN-DTST, the behavior recognition accuracy is the highest when the convolution kernel size is 7*7. This is because when the convolution kernel size is too small, it cannot effectively capture higher-level features and cannot efficiently obtain action information; when the convolution kernel size is too large, it cannot effectively capture detailed features and cannot fully obtain action information. The CNN in CNN-DTST extracts features through convolution kernels of different sizes, which not only ensures the extraction of global features, but also processes detailed features in the data. The subsequent DTST further extracts spatiotemporal features, making the extracted feature information richer and improving the recognition effect of behavior recognition.

[0087] To further verify the overall action recognition effect, the UCI-MHEALTH dataset is used to compare with other network models. The accuracy of human action recognition by each network model is shown in Table 5.

[0088] Table 5 Accuracy of different networks

[0089] As can be seen from Table 5, the accuracy of deep learning methods is generally higher than that of traditional machine learning methods. Among the compared deep learning methods, the method proposed in the present invention achieves the highest accuracy compared with other methods.

[0090] In summary, the present invention provides a human behavior recognition method and system based on spatiotemporal decoupling multi-scale feature fusion, and uses the UCI-MHEALTH dataset for experimental analysis, which effectively improves the accuracy of classification and recognition and improves the effect of action recognition.

[0091] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0092] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0093] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed in the present invention can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0094] In the embodiments provided by the present invention, it should be understood that the disclosed devices / terminals and methods can be implemented in other ways. For example, the device / terminal embodiments described above are only schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0095] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0096] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0097] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0098] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0099] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0100] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0101] The above contents are only for explaining the technical idea of ​​the present invention and cannot be used to limit the protection scope of the present invention. Any changes made on the basis of the technical solution in accordance with the technical idea proposed by the present invention shall fall within the protection scope of the claims of the present invention.

Claims

1. A human behavior recognition method based on spatiotemporal decoupling multi-scale feature fusion, characterized in that: The following steps are involved: S1. A linear local difference embedding algorithm is used to obtain a low-dimensional dynamic feature set, which is then input into a multi-scale convolutional neural network to extract multi-scale deep features from inertial signals, and multi-scale fusion features are output by splicing feature matrices; S2, setting the relevant parameters of the multi-scale convolutional neural network, formulating the Transformer spatiotemporal decoupling module stacking strategy, inputting the multi-scale fusion features output from step S1 into the Transformer-based spatiotemporal decoupling network model, and extracting the spatiotemporal features; S3. Input the spatiotemporal feature matrix obtained in step S2 into the softmax classifier to complete the training of the human behavior recognition model and realize human behavior recognition.

2. The human behavior recognition method based on spatiotemporal decoupling multi-scale feature fusion according to claim 1 is characterized in that: Step S1 is specifically as follows: S101, extracting the time-frequency domain features of the data set using short-time Fourier transform to obtain the original high-dimensional action feature set; S102, using a nonlinear dimensionality reduction method to reduce dimensionality and remove redundant information; S103, inputting the feature set after dimension reduction into a multi-scale convolutional neural network, and outputting multi-scale fusion features.

3. The human behavior recognition method based on spatiotemporal decoupling and multi-scale feature fusion according to claim 2 is characterized in that: The multi-scale convolutional neural network contains three convolutional layers, and the three convolutional layers select convolution kernels of different sizes; different branches include three convolutional layers; the number of convolution kernels in different convolutional layers in each branch is 16, 32, and 64 respectively; branch one uses a convolution kernel size of 1*3, branch two uses a convolution kernel size of 5*5, and branch three uses a convolution kernel size of 7*7; at the same time, Batch Normalization, Max-Pool and Dropout layers are introduced.

4. The human behavior recognition method based on spatiotemporal decoupling multi-scale feature fusion according to claim 2 is characterized in that: The sensor data of different parts of the human body are divided into multiple independent channels as the input of the multi-scale convolutional neural network. Assume that the sensor signal contains M independent channels and the output of channel i at time t is recorded as , then the output of multiple wearable sensors at time t is .

5. The human behavior recognition method based on spatiotemporal decoupling and multi-scale feature fusion according to claim 1 is characterized in that: Step S2 is specifically as follows: Suppose the fusion feature matrix processed by the multi-scale feature fusion network is recorded as ,Will Divide into The grouped fusion features are used as the input of the spatial decoupling module. The multi-scale fusion features are input into the spatiotemporal decoupling network model, and multiple attention mechanisms are used for parallel calculation to obtain the final output. ; Use a convolutional feedforward network to get the output , the input of the convolutional feedforward network and output After sequence addition and layer normalization, the final features extracted by the spatiotemporal decoupling encoder module are obtained. .

6. The human behavior recognition method based on spatiotemporal decoupling and multi-scale feature fusion according to claim 5 is characterized in that: The spatiotemporal decoupling network model introduces N alternately stacked spatiotemporal Transformer encoder modules to extract the spatiotemporal dependency information in the fused feature sequence; each encoder module includes a multi-head self-attention mechanism with a residual structure and a convolutional feedforward layer.

7. The human behavior recognition method based on spatiotemporal decoupling and multi-scale feature fusion according to claim 5 is characterized in that: The parallel calculation of multiple attention mechanisms is as follows: Suppose there are h attention heads; First, for each attention head, a linear transformation is performed using different weight matrices to obtain the query matrix Q, key matrix K, and value matrix V; Then use the attention mechanism to calculate the output of each attention head, concatenate the output of each attention head and perform a linear transformation to get the final output , as follows: in, is the output of the attention head, Concatenate the output vectors calculated by each attention head to form a long vector. is the weight matrix of the linear transformation.

8. The human behavior recognition method based on spatiotemporal decoupling and multi-scale feature fusion according to claim 5 is characterized in that: feature Specifically: in, is the input of the multi-head self-attention module, is the output of the convolutional feed-forward network.

9. The human behavior recognition method based on spatiotemporal decoupling and multi-scale feature fusion according to claim 8 is characterized in that: Input of the multi-head self-attention module for: Output of a convolutional feedforward network Specifically: in, is the weight parameter of the convolution kernel, is the weight parameter of the convolution kernel, is the output result of the first layer, It is a one-dimensional convolution operation, which convolves the input data with the convolution kernel. For batch processing, each batch of data is normalized.

10. A human behavior recognition system based on spatiotemporal decoupling multi-scale feature fusion, characterized in that: include: The splicing module uses a linear local difference embedding algorithm to obtain a low-dimensional dynamic feature set, which is then input into a multi-scale convolutional neural network to extract multi-scale deep features from inertial signals and output multi-scale fusion features by splicing feature matrices; The extraction module sets the relevant parameters of the multi-scale convolutional neural network, formulates the Transformer spatiotemporal decoupling module stacking strategy, inputs the output multi-scale fusion features into the Transformer-based spatiotemporal decoupling network model, and extracts the spatiotemporal features; The recognition module inputs the spatiotemporal feature matrix into the softmax classifier to complete the training of the human behavior recognition model and realize human behavior recognition.

Images