Human body behavior recognition system and method based on multi-dimensional and multi-scale feature extraction

Abstract

The invention relates to the technical field of artificial intelligence, and particularly provides a human behavior recognition system and method based on multi-dimensional and multi-scale feature extraction. The system comprises a human body behavior data acquisition module, a human body behavior data transmission module, a human body behavior data storage module, a human body behavior data preprocessing module, an LGAM simple human body behavior recognition network module, an MPFEFN complex human body behavior recognition network module and a human body behavior information application module. The MPFEFN complex behavior recognition network module comprises an initialization unit, a residual error multi-parallel multi-attention feature extraction unit, a residual error multi-parallel channel enhancement and extraction unit, a feature fusion unit and a complex behavior judgment output unit, and the residual error multi-parallel multi-attention feature extraction unit and the residual error multi-parallel channel enhancement and extraction unit are connected in parallel. The recognition accuracy of the easily-confused behaviors is improved, effective recognition of complex human body behaviors is achieved, and the recognition performance of an existing model is improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a human behavior recognition system and method based on multi-dimensional and multi-scale feature extraction. Background Art

[0002] Human Activity Recognition (HAR) refers to the intelligent recognition of human behavior or activity status by collecting and analyzing the input individual or group action data. At present, human activity recognition has become an important research topic in the fields of artificial intelligence, pattern recognition and human-computer interaction. It is widely used in application scenarios such as smart home, health care, and security monitoring. It has huge commercial value and broad development prospects, and has received close attention from academia and industry. According to the type of data collected, recognition methods are mainly divided into two categories: vision-based HAR and sensor-based HAR. The former analyzes image or video data, and the latter studies time series data collected by wearable sensors and environmental sensors. Compared with vision-based HAR, sensor-based HAR has the advantages of low cost, good privacy, and strong anti-interference ability.

[0003] With the advancement and iteration of technologies in related fields, HAR algorithms and systems have entered a rapid development stage. At present, research on HAR mainly focuses on classification models based on deep neural networks. Deep neural networks can automatically extract behavioral features, thereby achieving end-to-end behavior recognition and effectively improving recognition accuracy. Convolutional Neural Network (CNN) is one of the most widely used deep neural networks. CNN is a multi-layer stacked deep feedforward network. By processing the input data layer by layer, it realizes the gradual integration of information, converts the original data into a high-level feature representation that is more closely related to the output target, and finally completes the label mapping through the classifier. Compared with CNN, Recurrent Neural Network (RNN) pays more attention to the temporal characteristics of data and can capture the correlation between temporal features. Therefore, RNN and its various variant networks such as Long Short-Term Memory (LSTM) and Gated Recurrent Unit (GRU) are widely used in HAR model construction.

[0004] At present, there are still many challenges in the field of human behavior recognition. First, in terms of feature extraction, existing models do not fully utilize the correlation between features and have limited ability to reshape low-level features into effective high-level representations. Second, in terms of information acquisition, existing frameworks cannot adaptively capture the complex dynamic information contained in the motion patterns and speed changes of human behavior. Finally, in terms of classification, most existing recognition frameworks have difficulty distinguishing confusing behaviors, and can usually only classify and recognize simple behaviors of short duration, but have difficulty recognizing long-term complex behaviors composed of multiple simple behaviors. Summary of the invention

[0005] In view of this, the present invention provides a human behavior recognition system and method based on multi-dimensional and multi-scale feature extraction, so as to better utilize the multi-dimensional features contained in the human behavior data, improve the recognition accuracy of confusing behaviors, realize the effective recognition of complex human behaviors, and enhance the recognition performance of existing models.

[0006] In a first aspect, the present invention provides a human behavior recognition system based on multi-dimensional and multi-scale feature extraction, the system comprising: Human behavior data acquisition module, human behavior data storage module, human behavior data storage module, human behavior data preprocessing module, LGAM simple human behavior recognition network module, MPFEFN complex human behavior recognition network module and human behavior information application module; The LGAM simple human behavior recognition network module includes a feature pre-extraction unit, a local and global adaptive spatiotemporal feature extraction unit, a feature fusion unit and a simple behavior discrimination output unit connected in sequence; The MPFEFN complex behavior recognition network module includes an initialization unit, a parallel-connected residual multi-parallel multi-attention feature extraction unit and a residual multi-parallel channel enhancement and extraction unit, a feature fusion unit and a complex behavior discrimination output unit.

[0007] Optionally, the human behavior data acquisition module includes several different types of motion data sensing units and physiological data sensing units, the motion data sensing unit includes an acceleration data sensing unit, an angular velocity data sensing unit and a magnetic induction data sensing unit; the physiological data sensing unit includes a heart rate data sensing unit, a blood pressure data sensing unit, a blood oxygen data sensing unit and a surface electromyography data sensing unit; The transmission mode of the human behavior data transmission module includes ultra-wideband UWB, Wi-Fi, long-distance radio Lora, ZigBee, Bluetooth, 4G, 5G and long-distance radio information transmission; The human behavior data preprocessing module includes a data denoising unit, a multimodal data merging unit, a missing value processing unit, a data normalization unit and a data sliding window segmentation unit which are connected in sequence.

[0008] Optionally, the multimodal data merging unit merges the motion data and physiological data collected by different sensors by aligning the vertical timestamps and arranging and splicing them horizontally; the missing value processing unit uses the mean interpolation method to fill in the missing information, and uses the average value of the column where the missing data is located as the missing value; the data normalization unit performs Z-Score normalization processing on data with different dimensions and value ranges, and converts them to the same value range; the data sliding window segmentation unit segments the continuous time series collected by the sensor into multiple data segments.

[0009] Optionally, the feature pre-extraction unit uses a single one-dimensional convolution module to perform preliminary extraction of behavioral features; the data after feature pre-extraction is sequentially input into the local and global adaptive feature extraction unit and the simple behavior discrimination unit; the features output by the feature fusion unit are input into the sequentially connected initialization unit, the parallel connected residual multi-parallel multi-attention feature extraction unit and the residual multi-parallel channel enhancement and feature extraction unit; the local and global adaptive feature extraction unit extracts local and global features by adaptively adjusting the convolution kernel size; the residual multi-parallel multi-attention feature extraction unit uses a residual multi-parallel structure and utilizes multiple attention mechanisms to extract multi-dimensional features contained in the human behavior data; the residual multi-parallel channel enhancement and feature pre-extraction unit uses a residual multi-parallel structure and utilizes channel enhancement technology to extract channel features contained in the human behavior data.

[0010] Optionally, the feature fusion unit uses M-reluGRU and a multi-head self-attention mechanism to effectively fuse long-term temporal features in complex behaviors; M-reluGRU removes the reset gate in the gated recurrent unit GRU, simplifies the GRU into a single-gate structure, uses the ReLU function as the activation function, and uses batch normalization (Batch Normalization, BN) as the normalization method.

[0011] Optionally, the human behavior information application module includes a human behavior visualization unit, a human behavior statistics unit, and a human behavior analysis unit; the human behavior data storage module includes a cloud database and a local database.

[0012] Optionally, the simple behavior discrimination output unit includes a fully connected layer and a Softmax function connected in sequence. After multi-dimensional feature fusion, the data is output to the fully connected layer and classified by the Softmax function to output the final simple behavior type.

[0013] In a second aspect, the present invention provides a method for human behavior recognition based on multi-dimensional and multi-scale feature extraction, the method being implemented based on a human behavior recognition system based on multi-dimensional and multi-scale feature extraction in the first aspect or any possible implementation of the first aspect; the method comprising: S1, human behavior data collection, collecting user behavior information through a multimodal sensor in a human behavior data collection module, wherein the behavior information includes motion data and physiological data; S2, human behavior data transmission, transmitting the behavior information collected by the human behavior data collection module to the local server and cloud server database of the human behavior data storage module through the human behavior data transmission module, so that the human behavior data storage module can store the behavior information; S3, human behavior data preprocessing, through the human behavior data preprocessing module, the behavior information is sequentially subjected to human behavior data denoising, multimodal human behavior data merging, human behavior data missing value processing, human behavior data normalization and human behavior data sliding window segmentation; S4, constructing the LGAM simple human behavior recognition network to perform simple human behavior recognition and classification; inputting the behavior data that has completed feature preprocessing into the LGAM simple human behavior recognition network in batches, performing feature extraction and fusion, and outputting the simple discrimination results through the simple behavior discrimination output unit; S5. Construct MPFEFN complex human behavior recognition network to perform complex human behavior recognition and classification; input the initialized behavior data into the MPFEFN complex human behavior recognition network in batches, perform feature extraction and fusion, and output the complex discrimination results through the complex behavior discrimination output unit; S6, feature fusion, the output feature data is fused through M-reluGRU and multi-head self-attention mechanism and input into the general access norm layer GAP, the GAP layer converts the feature map of each channel into a feature point, and the feature point is the average value of the entire feature map; S7, output action recognition results. After multi-dimensional spatiotemporal feature extraction and fusion, the obtained feature data is input into the fully connected layer. The number of hidden units in the fully connected layer is the number of simple behavior categories contained in the data. The data after the fully connected layer is passed through the Softmax classifier to calculate the probability of the corresponding behavior. The behavior type with the highest probability is the final judgment result. S8. Display and application of human behavior information: Through the human behavior information application module, the behavior recognition results are visualized, counted and analyzed.

[0014] Optionally, step S3 includes: S31, human behavior data denoising; the data collected by the sensor is denoised using the soft threshold wavelet denoising method based on Stein unbiased likelihood estimation; , is the original signal, is a noisy signal, is Gaussian white noise, , is the noise intensity, and the denoising process is to Lieutenant Noise Remove and get the original signal The best approximation of Discrete sampling is performed to obtain Point discrete signal , the wavelet transform coefficients are shown in formula (1): (1); in, is the wavelet coefficient, is the scale function, j is the scale parameter, k is the number of units of the scale function translation, and through the dual-scale equations (2) and (3), the recursive implementation method of equation (1) is obtained: (2); (3); Among them, the symbol * represents convolution, h and g represent low-pass and high-pass filters respectively. Represents the original signal , Represents the approximation coefficient on scale j, and the wavelet transform reconstruction formula is shown in formula (4): (4); The threshold is determined by the SURE-based soft threshold estimation method, and the likelihood estimate of the given threshold is obtained, as shown in formula (5): (5); in, represents the selected initial threshold, Indicates that it comes from the subband The wavelet coefficients of Represents the sum of the number of wavelet coefficients in each subband; minimize the likelihood function and obtain the required threshold, as shown in formula (6): (6); in, Describe the obtained threshold parameters; The wavelet transform coefficients of the behavioral data are processed using a soft threshold function. The absolute value of the data is compared with the threshold. Points less than the threshold are set to zero, and points greater than or equal to the threshold shrink toward zero and become the difference between the point value and the threshold. The soft threshold function is shown in formula (7): (7); According to formula (3), the signal is reconstructed by wavelet to obtain the denoised signal; S32, merging multimodal human behavior data: aligning the behavior data collected by the sensor vertically according to the timestamps, and merging them horizontally, and the merged data is in a two-dimensional array format; S33, missing value processing of human behavior data: use mean interpolation method to process missing values ​​of behavior data, and fill the missing data with the average value of the column where the missing data is located; S34, normalization of human behavior data: using Z-Score normalization method to normalize human behavior data to ensure that the data are in the same order of magnitude; Assume that the input data sample sequence is , the output sequence after Z-Score normalization is , the calculation method is shown in formula (8): (8); in, is the mean of the input data sample sequence, is the standard deviation of the input data sample sequence; S35, human behavior data sliding window segmentation, use a fixed-length window to segment the continuous sensor data into fixed-length data segments, and ensure that each data segment contains at least one complete action of a simple behavior during segmentation. A 50% window overlap rate is used during the window sliding process.

[0015] Optionally, step S4 includes: S41: Human behavior data is input into a simple human behavior recognition network; the preprocessed data is transformed into a shape suitable for the one-dimensional convolutional layer, and is input into the feature extraction and fusion unit in batches. The data shape is ,in is the batch size, is the number of data channels, is the window length; The feature data output by the feature pre-extraction unit is input into the local and global adaptive spatiotemporal feature extraction units and feature fusion units connected in sequence; S42: Extraction of local and global spatiotemporal features; the data of the input module is processed by the initialization unit, which contains a one-dimensional convolution module, which is composed of a one-dimensional convolution layer, a batch normalization layer, an activation layer, and a maximum pooling layer connected in sequence. The convolution kernel size of the one-dimensional convolution layer is 3, the stride size is 1, the padding method is SAME, and the nonlinear activation function selects the ReLU function; the calculation method of the one-dimensional convolution module is shown in formula (9): ; in, Represents the feature map List, represents the convolution kernel List, represents the bias term; The local and global adaptive spatiotemporal feature extraction unit constructs an adaptive time kernel consisting of a local branch and a global branch. The local branch adopts a local sensitive modeling scheme to capture short-term dynamic changes by establishing a position-sensitive importance weight. The extraction is shown in formula (10). The global branch is based on the local invariant modeling idea and obtains global information and long-term dependency information by constructing a dynamic aggregation convolution kernel. The extraction is shown in formula (11). The overall process is shown in formula (12): (10); in, represents the local branch transformation, represents the multiplication operator, represents a one-dimensional convolution operation, and Respectively represent Sigmoid activation function and ReLU activation function; (11); in represents the global branch transformation, represents the equivalent transformation function of the fully connected layer operation, Represent the weights of the two FC layers respectively; in order to realize the interaction between the information obtained by the global branch and the information obtained by the local branch, finally, the outputs of the two branches are convolved; (12); in, represents the output feature of the time adaptive module, Represents the convolution operator.

[0016] In a third aspect, an embodiment of the present invention provides a computer-readable storage medium, which includes a stored program, wherein when the program is running, the device where the computer-readable storage medium is located is controlled to execute the human behavior recognition method based on multi-dimensional and multi-scale feature extraction in the second aspect or any possible implementation of the second aspect.

[0017] In a fourth aspect, an embodiment of the present invention provides an electronic device, comprising: one or more processors; a memory; and one or more computer programs, wherein the one or more computer programs are stored in the memory, and the one or more computer programs include instructions, which, when executed by the device, enable the device to perform the human behavior recognition method based on multi-dimensional and multi-scale feature extraction in the second aspect or any possible implementation of the second aspect.

[0018] In the technical solution provided by the present invention, the system includes a human behavior data acquisition module, a human behavior data transmission module, a human behavior data storage module, a human behavior data preprocessing module, an LGAM simple human behavior recognition network module, an MPFEFN complex human behavior recognition network module and a human behavior information application module; the LGAM simple human behavior recognition network module includes a feature pre-extraction unit, a local and global adaptive spatiotemporal feature extraction unit, a feature fusion unit and a simple behavior discrimination output unit connected in sequence; the MPFEFN complex behavior recognition network module includes an initialization unit, a residual multi-parallel multi-attention feature extraction unit and a residual multi-parallel channel enhancement and extraction unit connected in parallel, a feature fusion unit and a complex behavior discrimination output unit. The system makes better use of the multi-dimensional features contained in the human behavior data, improves the recognition accuracy of easily confused behaviors, realizes the effective recognition of complex human behaviors, and improves the recognition performance of the existing model. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0020] Figure 1 A schematic diagram of the structure of a human behavior recognition system based on multi-dimensional and multi-scale feature extraction provided by an embodiment of the present invention; Figure 2 A schematic diagram of the structure of a local and global adaptive spatiotemporal feature extraction unit provided in an embodiment of the present invention; Figure 3 A schematic diagram of the structure of a residual multi-parallel multi-attention feature extraction unit provided in an embodiment of the present invention, wherein (a) is the detailed structure of FEMM, and (b) is the overall framework of the residual multi-parallel multi-attention feature extraction unit; Figure 4 A schematic diagram of the structure of a residual multi-parallel channel enhancement and extraction unit provided in an embodiment of the present invention, wherein (a) is a detailed structure of a CBEM, and (b) is an overall framework of the residual multi-parallel channel enhancement and extraction unit; Figure 5 A flowchart of a method for human behavior recognition based on multi-dimensional and multi-scale feature extraction provided by an embodiment of the present invention; Figure 6 A schematic diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0021] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0022] It should be clear that the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0023] The terms used in the embodiments of the present invention are only for the purpose of describing specific embodiments, and are not intended to limit the present invention. The singular forms "a", "said" and "the" used in the embodiments of the present invention are also intended to include plural forms, unless the context clearly indicates other meanings.

[0024] It should be understood that the term "and / or" used in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0025] The word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrases "if it is determined" or "if (stated condition or event) is detected" may be interpreted as "when it is determined" or "in response to determining" or "when detecting (stated condition or event)" or "in response to detecting (stated condition or event)", depending on the context.

[0026] The human behavior recognition system based on multi-dimensional and multi-scale feature extraction provided by the present invention is as follows: Figures 1 to 4 As shown, the system includes: Human behavior data acquisition module, human behavior data storage module, human behavior data storage module, human behavior data preprocessing module, LGAM simple human behavior recognition network module, MPFEFN complex human behavior recognition network module and human behavior information application module; The LGAM simple human behavior recognition network module includes a feature pre-extraction unit, a local and global adaptive spatiotemporal feature extraction unit, a feature fusion unit and a simple behavior discrimination output unit connected in sequence; The MPFEFN complex behavior recognition network module includes an initialization unit, a parallel-connected residual multi-parallel multi-attention feature extraction unit and a residual multi-parallel channel enhancement and extraction unit, a feature fusion unit, and a complex behavior discrimination output unit. The initialization unit uses a single one-dimensional convolution and maximum pooling to perform preliminary extraction of behavior features; the initialized data is simultaneously input into the sequentially connected parallel multi-dimensional feature extraction unit, multi-head attention unit, and feature fusion unit; In an embodiment of the present invention, the human behavior data acquisition module includes several different types of motion data sensing units and physiological data sensing units, wherein the motion data sensing units include acceleration data sensing units, angular velocity data sensing units, and magnetic induction data sensing units, and the motion data include X, Y, and Z three-axis data of the acceleration sensor, X, Y, and Z three-axis data of the angular velocity sensor, and X, Y, and Z three-axis data of the magnetometer; the physiological data sensing units include heart rate data sensing units, blood pressure data sensing units, blood oxygen data sensing units, and surface electromyography data sensing units, and the physiological data include heart rate values, blood pressure values, blood oxygen values, and surface electromyography signal values; The motion data sensing unit mainly includes a three-axis acceleration sensor, a three-axis angular velocity sensor and a three-axis magnetometer, and the physiological data sensing unit mainly includes a heart rate sensor, a blood pressure sensor and a skin electrical sensor. The sampling frequencies of different types of sensors are set to the same value according to user needs.

[0027] The transmission modes of the human behavior data transmission module include ultra-wideband UWB, Wi-Fi, long-distance radio Lora, ZigBee, Bluetooth, 4G, 5G and long-distance radio information transmission; the appropriate transmission mode is selected according to the user's application requirements.

[0028] The human behavior data preprocessing module includes a data denoising unit, a multimodal data merging unit, a missing value processing unit, a data normalization unit and a data sliding window segmentation unit which are connected in sequence, and finally obtains a smooth data segment which is not affected by dimension and is convenient for model processing.

[0029] In an embodiment of the present invention, the multimodal data merging unit merges the motion data and physiological data collected by different sensors by aligning the vertical timestamps and arranging and splicing them horizontally; the missing value processing unit uses the mean interpolation method to fill in the missing information, and uses the average value of the column where the missing data is located as the missing value; the data normalization unit performs Z-Score normalization processing on data with different dimensions and value ranges, and converts them to the same value range, that is, transforms all sensor behavior data to between [-1,1]; the data sliding window segmentation unit divides the continuous time series collected by the sensor into multiple data segments, and ensures that a complete simple behavior action data falls within a sliding window during the segmentation process.

[0030] In an embodiment of the present invention, a feature pre-extraction unit uses a single one-dimensional convolution module to perform preliminary extraction of behavioral features; data after feature pre-extraction is sequentially input into a local and global adaptive feature extraction unit and a simple behavior discrimination unit; the features output by a feature fusion unit are input into an initialization unit connected sequentially, a residual multi-parallel multi-attention feature extraction unit connected in parallel, and a residual multi-parallel channel enhancement and feature extraction unit; the local and global adaptive feature extraction unit extracts local and global features by adaptively adjusting the convolution kernel size; the residual multi-parallel multi-attention feature extraction unit uses a residual multi-parallel structure and utilizes multiple attention mechanisms to extract multidimensional features contained in human behavior data; the residual multi-parallel channel enhancement and feature pre-extraction unit uses a residual multi-parallel structure and utilizes channel enhancement technology to extract channel features contained in human behavior data.

[0031] In an embodiment of the present invention, the feature fusion unit uses M-reluGRU and a multi-head self-attention mechanism to effectively fuse long-term features in complex behaviors; M-reluGRU removes the reset gate in the gated recurrent unit GRU, simplifies the GRU into a single-gate structure, uses the ReLU function as the activation function, and uses batch normalization BN as the normalization method.

[0032] In the embodiment of the present invention, batch normalization is used to avoid numerical instability caused by the unboundedness of the ReLU activation function; M-reluGRU can obtain the output of each hidden layer at different times with less computational cost, and extract the behavioral context information of the previous and subsequent times. Compared with GRU, M-reluGRU is more suitable for time series data; the multi-head self-attention mechanism enables the network to pay attention to information from different feature subspaces, and a multi-head self-attention layer is added after M-reluGRU to give a higher weight to the single-window feature data that contributes the most to the identification of complex behaviors and has the most obvious features; finally, through global average pooling (Global Average Pooling, GAP), each channel feature of the data is averaged to achieve multi-dimensional feature fusion.

[0033] In the embodiment of the present invention, the human behavior information application module includes a human behavior visualization unit, a human behavior statistics unit, and a human behavior analysis unit; the human behavior data storage module includes a cloud database and a local database.

[0034] In an embodiment of the present invention, the simple behavior discrimination output unit includes a fully connected layer and a Softmax function connected in sequence. After multi-dimensional feature fusion, the data is output to the fully connected layer and classified by the Softmax function to output the final simple behavior type.

[0035] The recognition results of the simple behavior recognition module and the complex behavior recognition module can be transmitted to various application platforms in real time for display and statistics, so as to analyze and manage user behaviors.

[0036] The present invention provides a feasible solution for multi-complexity behavior recognition based on sensors. It aims to solve the problem that the feature extraction of existing models is not comprehensive, enhances the ability of feature representation and the adaptability of complex information capture, and improves the recognition accuracy. In view of the defects of existing models in complex behavior recognition, a recognition method for complex human behavior is proposed, which enhances the practical application performance of existing models.

[0037] The present invention can be applied to scenarios such as smart homes. Taking the recognition of daily behavioral activities in power operation application scenarios as an example, human behavior in power scenarios is highly logical and conforms to the time sequence. Complex operation behaviors are usually composed of multiple continuous simple behaviors. For example, the behavior of a person cleaning a bedroom is composed of multiple simple behaviors such as walking, bending over, and sweeping the floor. The recognition system first analyzes the short-term features of the collected actions to realize the recognition of simple behaviors, and on this basis, comprehensively analyzes the behavior context information to obtain the long-term features of the behavior and realize the recognition of complex behaviors.

[0038] Figure 5 A flow chart of a method for human behavior recognition based on multi-dimensional and multi-scale feature extraction provided by an embodiment of the present invention is shown in FIG. Figure 5 As shown, the method is implemented based on the above-mentioned human behavior recognition system based on multi-dimensional and multi-scale feature extraction; the method includes: S1, human behavior data collection, collecting user behavior information through a multimodal sensor in a human behavior data collection module, wherein the behavior information includes motion data and physiological data; S2, human behavior data transmission, transmitting the behavior information collected by the human behavior data collection module to the local server and cloud server database of the human behavior data storage module through the human behavior data transmission module, so that the human behavior data storage module can store the behavior information; S3, human behavior data preprocessing, through the human behavior data preprocessing module, the behavior information is sequentially subjected to human behavior data denoising, multimodal human behavior data merging, human behavior data missing value processing, human behavior data normalization and human behavior data sliding window segmentation; S4, constructing the LGAM simple human behavior recognition network to perform simple human behavior recognition and classification; inputting the behavior data that has completed feature preprocessing into the LGAM simple human behavior recognition network in batches, performing feature extraction and fusion, and outputting the simple discrimination results through the simple behavior discrimination output unit; S5. Construct MPFEFN complex human behavior recognition network to perform complex human behavior recognition and classification; input the initialized behavior data into the MPFEFN complex human behavior recognition network in batches, perform feature extraction and fusion, and output the complex discrimination results through the complex behavior discrimination output unit; S6, feature fusion, the output feature data is fused through M-reluGRU and multi-head self-attention mechanism and input into the general access norm layer GAP, the GAP layer converts the feature map of each channel into a feature point, and the feature point is the average value of the entire feature map; S7, output action recognition results. After multi-dimensional spatiotemporal feature extraction and fusion, the obtained feature data is input into the fully connected layer. The number of hidden units in the fully connected layer is the number of simple behavior categories contained in the data. The data after the fully connected layer is passed through the Softmax classifier to calculate the probability of the corresponding behavior. The behavior type with the highest probability is the final judgment result. S8. Display and application of human behavior information: Through the human behavior information application module, the behavior recognition results are visualized, counted and analyzed.

[0039] In the embodiment of the present invention, step S3 includes:

[0040] S31, human behavior data denoising: the soft threshold wavelet denoising method of Stein's Unbiased Risk Estimation (SURE) is used to denoise the data collected by the sensor; , is the original signal, is a noisy signal, is Gaussian white noise, , is the noise intensity, and the denoising process is to Lieutenant Noise Remove and get the original signal The best approximation of Discrete sampling is performed to obtain Point discrete signal , the wavelet transform coefficients are shown in formula (1): (1); in, is the wavelet coefficient, is the scale function, j is the scale parameter, k is the number of units of the scale function translation, and through the dual-scale equations (2) and (3), the recursive implementation method of equation (1) is obtained: (2); (3); Among them, the symbol * represents convolution, h and g represent low-pass and high-pass filters respectively. Represents the original signal , Represents the approximation coefficient on scale j, and the wavelet transform reconstruction formula is shown in formula (4): (4); The threshold is determined by the SURE-based soft threshold estimation method, and the likelihood estimate of the given threshold is obtained, as shown in formula (5): (5); in, represents the selected initial threshold, Indicates that it comes from the subband The wavelet coefficients of Represents the sum of the number of wavelet coefficients in each subband; minimize the likelihood function and obtain the required threshold, as shown in formula (6): (6); in, Describe the obtained threshold parameters; The wavelet transform coefficients of the behavioral data are processed using a soft threshold function. The absolute value of the data is compared with the threshold. Points less than the threshold are set to zero, and points greater than or equal to the threshold shrink toward zero and become the difference between the point value and the threshold. The soft threshold function is shown in formula (7): (7); According to formula (3), the signal is reconstructed by wavelet to obtain the denoised signal; S32. Merge multimodal human behavior data; align the behavior data collected by the sensor vertically according to the timestamp, and merge them horizontally. The merged data is in a two-dimensional array format; the order of horizontal splicing is the X, Y, and Z axis data of the acceleration sensor, the X, Y, and Z axis data of the angular velocity sensor, the X, Y, and Z axis data of the magnetometer, the heart rate data, the blood pressure data, the blood oxygen data, and the surface electromyography data.

[0041] S33, missing value processing of human behavior data: use mean interpolation method to process missing values ​​of behavior data, and fill the missing data with the average value of the column where the missing data is located; S34, normalization of human behavior data: use Z-Score normalization method to normalize human behavior data to ensure that the data are in the same order of magnitude and avoid the adverse effects of different dimensions and value ranges on calculations; Assume that the input data sample sequence is , the output sequence after Z-Score normalization is , the calculation method is shown in formula (8): (8); in, is the mean of the input data sample sequence, is the standard deviation of the input data sample sequence; S35, human behavior data sliding window segmentation, use a fixed-length window to segment the continuous sensor data into fixed-length data segments, and ensure that each data segment contains at least one complete action of a simple behavior during segmentation. A 50% window overlap rate is used during the window sliding process.

[0042] In the embodiment of the present invention, step S4 includes: S41: Human behavior data is input into a simple human behavior recognition network; the preprocessed data is transformed into a shape suitable for the one-dimensional convolutional layer, and is input into the feature extraction and fusion unit in batches. The data shape is ,in is the batch size, is the number of data channels, is the window length; The feature data output by the feature pre-extraction unit is input into the local and global adaptive spatiotemporal feature extraction units and feature fusion units connected in sequence; S42: Extraction of local and global spatiotemporal features; the data of the input module is processed by the initialization unit, which contains a one-dimensional convolution module, which is composed of a one-dimensional convolution layer, a batch normalization layer, an activation layer, and a maximum pooling layer connected in sequence. The convolution kernel size of the one-dimensional convolution layer is 3, the stride size is 1, the padding method is SAME, and the nonlinear activation function selects the ReLU function; the calculation method of the one-dimensional convolution module is shown in formula (9): ; in, Represents the feature map List, represents the convolution kernel List, represents the bias term; The local and global adaptive spatiotemporal feature extraction unit constructs an adaptive time kernel consisting of a local branch and a global branch. The local branch adopts a local sensitive modeling scheme to capture short-term dynamic changes by establishing a position-sensitive importance weight. The extraction is shown in formula (10). The global branch is based on the local invariant modeling idea and obtains global information and long-term dependency information by constructing a dynamic aggregation convolution kernel. The extraction is shown in formula (11). The overall process is shown in formula (12): (10); in, represents the local branch transformation, represents the multiplication operator, represents a one-dimensional convolution operation, and Respectively represent Sigmoid activation function and ReLU activation function; (11); in represents the global branch transformation, represents the equivalent transformation function of the fully connected (FC) layer operation, Represent the weights of the two FC layers respectively; in order to realize the interaction between the information obtained by the global branch and the information obtained by the local branch, finally, the outputs of the two branches are convolved; (12); in, represents the output features of the Temporal Adaptive Module (TAM), Represents the convolution operator.

[0043] In the embodiment of the present invention, step S5 includes: Complex action data temporal feature extraction; feature data is sequentially input into the initialization unit, the parallel residual multi-parallel multi-attention feature extraction unit and the residual multi-parallel channel enhancement and feature extraction unit; local features related to body parts are further obtained through the initialization layer, and the extraction is as follows: (13); in, is the feature map, is the two-dimensional convolution calculation, and are the average and maximum values ​​of the pooling operation, respectively; The residual multi-parallel multi-attention feature extraction unit adopts a special parallel structure. Instead of simply parallelizing a single branch, it reduces the output dimension through specific addition and dimensionality reduction operations while ensuring that the branches do not interfere with each other. The calculation process is shown in formula (14); each branch uses dynamic convolution based on the modal-temporal attention mechanism to help capture the modal and temporal correlations between features, improving the ability to identify key information in multimodal data; the dynamic convolution based on the modal-temporal attention mechanism (Modal and Temporal Based Dynamic Convolution, MT-Dyconv) calculation process is shown in formulas (15)-(22): (14); in, is the concatenation vector, Represents a splicing operation, represents the summation and dimensionality reduction operation, that is, compression after summing the first dimension, and It represents the operation process of Basic Convolution Block (BCB) and Feature Extraction with Multi-attention Module (FEMM); the dynamic convolution process can be expressed as: (15); in, and are the value and weight of the first core respectively, represents the input feature map, represents the output feature map; Specifically, first use the size of ( )and( ) encodes each channel along the horizontal and vertical coordinates respectively; therefore, the feature maps output after pooling are expressed as: (16); (17); in, Indicates the height Channel output, Indicates the width is Channel output; In order to make full use of modal information and temporal information, the above two feature maps are cascaded and then transformed using shared convolution, which is expressed as: (18); in, is the internal feature map of spatial information in the horizontal and vertical directions, Represents the sigmoid activation function; Indicates the sampling rate of downsampling, which is used to control the size of the module; Next, along the spatial dimension, the feature map is divided into two separate tensors and ; Using two convolutions and A size of ( ), the feature maps and are converted to the same number of channels as the input; the calculation process is as follows: (19); (20); in, and is the output feature map after channel transformation, is the sigmoid activation function; Then, the output and are expanded into attention weights, which can be expressed as: (twenty one); in, and represents the expanded attention weight, Indicates input, Represents the final output; After calculating the attention weights, two convolutional layers are set to change the module capacity and computational cost. The calculated attention scalar is: (twenty two); in, is a hyperparameter used to control the sparsity of weights; The residual multi-parallel channel enhancement and feature extraction units use the same parallel structure, reducing the output dimension through specific addition and dimensionality reduction operations while ensuring that the branches do not interfere with each other. The overall operation is as follows: (twenty three); in, is the concatenation vector, Represents a splicing operation, represents the summation and dimensionality reduction operation, that is, compression after summing the first dimension, and Represents the operation process of BCB and ChannelBoost and Extraction Module (CBEM); The Inception structure and SE module are used to extract channel features. The Inception structure inputs the input vector into four sets of parallel convolution blocks respectively, and then concatenates them on the channel dimension, which is expressed as: (twenty four); in, Features represent the enhanced features. It means concatenating the outputs of multiple two-dimensional convolutional layers. , and They represent the convolution kernel sizes respectively. , and Two-dimensional convolution operation; The SE calculation method is as follows: In the compression stage, for the enhanced features of each branch, global average pooling is used to reduce the context and capture the channel dependencies in the global information, expressed as: (25); in, For input, To capture the elements of the feature; The excitation is to make full use of the information captured in the squeezing process, which uses two FC layers connected by an activation function as a gating mechanism to learn the nonlinear interactions between multiple channels. The excitation stage is defined as: (26); in, is the attention weight, and They are activation functions ReLU and sigmoid, and Representation transformation The weight of the scale; Then, the scaling operation assigns the previously obtained attention weights to the features of each channel, expressed as: (27); in, represents the output feature map elements.

[0044] In the embodiment of the present invention, step S6 includes: The feature fusion unit can assign different weights to features at different times, using M-reluGRU and multi-head self-attention mechanism. The feature data of a single window is first input into M-reluGRU for long-term feature fusion. The calculation process of M-reluGRU is shown in equations (27) to (29): (28); (29); (30); in, Represents input data, represents the output of the update gate, Represents the output of M-reluGRU at the previous moment, represents the candidate hidden state, Represents the output of M-reluGRU at the current moment; The data output by M-reluGRU is input into the multi-head attention layer, which assigns different weights to the single-window simple behavior feature data input at different times; the feature matrix composed of several single-window feature data is ,in, Represents the simple behavior feature acquisition unit The single window feature data output at each moment, Indicates the total number of windows; Multiplying with the corresponding weight matrix to get the query matrix , the bond matrix and the value matrix , and then repeat multiple times , , Perform different linear mappings, calculate the outputs of different attention heads, and finally concatenate the outputs of multiple attention heads. The calculation method of the multi-head attention layer is shown in Equation (30)-Equation (32): (31); (32); (33); in, Representation Matrix and matrix The square root of the dimension, , and Respectively represent Second pair , , The weight matrix for linear mapping, Represents the calculation result of the first head in the multi-head attention mechanism, represents the total number of heads in the multi-head attention layer, It means concatenating the outputs of multiple heads; finally, the preliminarily fused features are input into the GAP layer, which converts the feature map of each channel into a feature point, which is the average value of the entire feature map.

[0045] The present invention realizes sensor-based human behavior classification and recognition by collecting and analyzing sensor data, which makes up for the shortcomings of vision-based behavior recognition such as high cost, susceptibility to interference, and poor privacy; through multiple attention mechanisms and residual parallel structures, the effectiveness, comprehensiveness and ability to capture complex dynamics of feature extraction are effectively improved. Compared with mainstream and latest models, it has obvious advantages in adaptability, reliability and practicality; the feature extraction and fusion module proposed in the present invention makes up for the shortcomings of existing models in complex human behavior recognition, and can effectively solve the problem that existing models can only realize simple behavior recognition.

[0046] In the technical solution provided by the present invention, the system includes a human behavior data acquisition module, a human behavior data transmission module, a human behavior data storage module, a human behavior data preprocessing module, an LGAM simple human behavior recognition network module, an MPFEFN complex human behavior recognition network module and a human behavior information application module; the LGAM simple human behavior recognition network module includes a feature pre-extraction unit, a local and global adaptive spatiotemporal feature extraction unit, a feature fusion unit and a simple behavior discrimination output unit connected in sequence; the MPFEFN complex behavior recognition network module includes an initialization unit, a residual multi-parallel multi-attention feature extraction unit and a residual multi-parallel channel enhancement and extraction unit connected in parallel, a feature fusion unit and a complex behavior discrimination output unit. The system makes better use of the multi-dimensional features contained in the human behavior data, improves the recognition accuracy of easily confused behaviors, realizes the effective recognition of complex human behaviors, and improves the recognition performance of the existing model.

[0047] Each step of the embodiment of the present invention may be performed by an electronic device, which includes but is not limited to a mobile phone, a tablet computer, a portable PC, a desktop computer, etc.

[0048] An embodiment of the present invention provides a computer-readable storage medium, which includes a stored program, wherein when the program is running, the electronic device where the computer-readable storage medium is located is controlled to execute the above-mentioned embodiment of the human behavior recognition method based on multi-dimensional and multi-scale feature extraction.

[0049] Figure 6 A schematic diagram of an electronic device provided by an embodiment of the present invention, such as Figure 6As shown, the electronic device 21 includes: a processor 211, a memory 212, and a computer program 213 stored in the memory 212 and executable on the processor 211. When the computer program 213 is executed by the processor 211, the human behavior recognition method based on multi-dimensional and multi-scale feature extraction in the embodiment is implemented. To avoid repetition, they are not described one by one here.

[0050] The electronic device 21 includes, but is not limited to, a processor 211 and a memory 212. Those skilled in the art will appreciate that Figure 6 It is only an example of the electronic device 21 and does not constitute a limitation of the electronic device 21. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the electronic device may also include input and output devices, network access devices, buses, etc.

[0051] The processor 211 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0052] The memory 212 may be an internal storage unit of the electronic device 21, such as a hard disk or memory of the electronic device 21. The memory 212 may also be an external storage device of the electronic device 21, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card (FlashCard), etc. equipped on the electronic device 21. Further, the memory 212 may also include both an internal storage unit of the electronic device 21 and an external storage device. The memory 212 is used to store computer programs and other programs and data required by network devices. The memory 212 may also be used to temporarily store data that has been output or is to be output.

[0053] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0054] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A human behavior recognition system based on multi-dimensional and multi-scale feature extraction, characterized in that: The system comprises: a human behavior data acquisition module, a human behavior data transmission module, a human behavior data storage module, a human behavior data preprocessing module, a LGAM simple human behavior recognition network module, a MPFEFN complex human behavior recognition network module and a human behavior information application module; The LGAM simple human behavior recognition network module includes a feature pre-extraction unit, a local and global adaptive spatiotemporal feature extraction unit, a feature fusion unit and a simple behavior discrimination output unit connected in sequence; The MPFEFN complex behavior recognition network module includes an initialization unit, a parallel-connected residual multi-parallel multi-attention feature extraction unit and a residual multi-parallel channel enhancement and extraction unit, a feature fusion unit and a complex behavior discrimination output unit.

2. The system according to claim 1, characterized in that The human behavior data acquisition module includes several different types of motion data sensing units and physiological data sensing units. The motion data sensing unit includes an acceleration data sensing unit, an angular velocity data sensing unit, and a magnetic induction data sensing unit; the physiological data sensing unit includes a heart rate data sensing unit, a blood pressure data sensing unit, a blood oxygen data sensing unit, and a surface electromyography data sensing unit; The transmission mode of the human behavior data transmission module includes ultra-wideband UWB, Wi-Fi, long-distance radio Lora, ZigBee, Bluetooth, 4G, 5G and long-distance radio information transmission; The human behavior data preprocessing module includes a data denoising unit, a multimodal data merging unit, a missing value processing unit, a data normalization unit and a data sliding window segmentation unit which are connected in sequence.

3. The system according to claim 2, characterized in that The multimodal data merging unit merges the motion data and physiological data collected by different sensors by aligning the vertical timestamps and arranging and splicing them horizontally; the missing value processing unit uses the mean interpolation method to fill in the missing information, and uses the average value of the column where the missing data is located as the missing value; the data normalization unit performs Z-Score normalization processing on data with different dimensions and value ranges, and converts them to the same value range; the data sliding window segmentation unit segments the continuous time series collected by the sensor into multiple data segments.

4. The system according to claim 1, characterized in that The feature pre-extraction unit uses a single one-dimensional convolution module to preliminarily extract the behavior features; The data after feature pre-extraction are sequentially input into the local and global adaptive feature extraction units and the simple behavior discrimination unit; The features output by the feature fusion unit are the initialization units connected in sequence, the residual multi-parallel multi-attention feature extraction units connected in parallel, and the residual multi-parallel channel enhancement and feature extraction units connected in parallel; The local and global adaptive feature extraction unit extracts local and global features by adaptively adjusting the convolution kernel size; the residual multi-parallel multi-attention feature extraction unit adopts a residual multi-parallel structure and utilizes multiple attention mechanisms to extract multi-dimensional features contained in human behavior data; The residual multi-parallel channel enhancement and feature pre-extraction unit adopts a residual multi-parallel structure and utilizes a channel enhancement technology to extract channel features contained in human behavior data.

5. The system according to claim 4, characterized in that The feature fusion unit adopts M-reluGRU and multi-head self-attention mechanism to effectively fuse long-term features in complex behaviors; M-reluGRU removes the reset gate in the gated recurrent unit GRU, simplifies the GRU into a single-gate structure, adopts the ReLU function as the activation function, and uses BN as the normalization method.

6. The system according to claim 1, characterized in that The human behavior information application module includes a human behavior visualization unit, a human behavior statistics unit, and a human behavior analysis unit; the human behavior data storage module includes a cloud database and a local database.

7. The system according to claim 1, characterized in that The simple behavior discrimination output unit includes a fully connected layer and a Softmax function connected in sequence. After multi-dimensional feature fusion, the data is output to the fully connected layer and classified by the Softmax function to output the final simple behavior type.

8. A human behavior recognition method based on multi-dimensional and multi-scale feature extraction, characterized in that: The method is implemented based on the human behavior recognition system based on multi-dimensional and multi-scale feature extraction according to any one of claims 1 to 7; the method comprises: S1, human behavior data collection, collecting user behavior information through a multimodal sensor in a human behavior data collection module, wherein the behavior information includes motion data and physiological data; S2, human behavior data transmission, transmitting the behavior information collected by the human behavior data collection module to the local server and cloud server database of the human behavior data storage module through the human behavior data transmission module, so that the human behavior data storage module can store the behavior information; S3, human behavior data preprocessing, through the human behavior data preprocessing module, the behavior information is sequentially subjected to human behavior data denoising, multimodal human behavior data merging, human behavior data missing value processing, human behavior data normalization and human behavior data sliding window segmentation; S4, constructing the LGAM simple human behavior recognition network to perform simple human behavior recognition and classification; inputting the behavior data that has completed feature preprocessing into the LGAM simple human behavior recognition network in batches, performing feature extraction and fusion, and outputting the simple discrimination results through the simple behavior discrimination output unit; Discrete sampling is performed to obtain a point discrete signal, and the wavelet transform coefficient is shown in formula (1): S5. Construct MPFEFN complex human behavior recognition network to perform complex human behavior recognition and classification; input the initialized behavior data into the MPFEFN complex human behavior recognition network in batches, perform feature extraction and fusion, and output the complex discrimination results through the complex behavior discrimination output unit; S6, feature fusion, the output feature data is fused through M-reluGRU and multi-head self-attention mechanism and input into the general access norm layer GAP, the GAP layer converts the feature map of each channel into a feature point, and the feature point is the average value of the entire feature map; S7, output action recognition results. After multi-dimensional spatiotemporal feature extraction and fusion, the obtained feature data is input into the fully connected layer. The number of hidden units in the fully connected layer is the number of simple behavior categories contained in the data. The data after the fully connected layer is passed through the Softmax classifier to calculate the probability of the corresponding behavior. The behavior type with the highest probability is the final judgment result. S8. Display and application of human behavior information: Through the human behavior information application module, the behavior recognition results are visualized, counted and analyzed.

9. The method according to claim 8, characterized in that The step S3 comprises: S31, human behavior data denoising; the data collected by the sensor is denoised using the soft threshold wavelet denoising method based on Stein unbiased likelihood estimation; , is the original signal, is a noisy signal, is Gaussian white noise, , is the noise intensity, and the denoising process is to Lieutenant Noise Remove and get the original signal The best approximation of Discrete sampling is performed to obtain Point discrete signal , the wavelet transform coefficients are shown in formula (1): （1）； in, is the wavelet coefficient, is the scale function, j is the scale parameter, k is the number of units of the scale function translation, and through the dual-scale equations (2) and (3), the recursive implementation method of equation (1) is obtained: （2）； （3）； Among them, the symbol * represents convolution, h and g represent low-pass and high-pass filters respectively. Represents the original signal , Represents the approximation coefficient on scale j, and the wavelet transform reconstruction formula is shown in formula (4): （4）； The threshold is determined by the SURE-based soft threshold estimation method, and the likelihood estimate of the given threshold is obtained, as shown in formula (5): （5）； in, represents the selected initial threshold, Indicates that it comes from the subband The wavelet coefficients of Represents the sum of the number of wavelet coefficients in each subband; minimize the likelihood function and obtain the required threshold, as shown in formula (6): （6）； in, Describe the obtained threshold parameters; The wavelet transform coefficients of the behavioral data are processed using a soft threshold function. The absolute value of the data is compared with the threshold. Points less than the threshold are set to zero, and points greater than or equal to the threshold shrink toward zero and become the difference between the point value and the threshold. The soft threshold function is shown in formula (7): （7）； According to formula (3), the signal is reconstructed by wavelet to obtain the denoised signal; S32, merging multimodal human behavior data: aligning the behavior data collected by the sensor vertically according to the timestamps, and merging them horizontally, and the merged data is in a two-dimensional array format; S33, missing value processing of human behavior data: use mean interpolation method to process missing values ​​of behavior data, and fill the missing data with the average value of the column where the missing data is located; S34, normalization of human behavior data: using Z-Score normalization method to normalize human behavior data to ensure that the data are in the same order of magnitude; Assume that the input data sample sequence is , the output sequence after Z-Score normalization is , the calculation method is shown in formula (8): （8）； in, is the mean of the input data sample sequence, is the standard deviation of the input data sample sequence; S35, human behavior data sliding window segmentation, use a fixed-length window to segment the continuous sensor data into fixed-length data segments, and ensure that each data segment contains at least one complete action of a simple behavior during segmentation. A 50% window overlap rate is used during the window sliding process.

10. The method according to claim 8, characterized in that The step S4 comprises: S41: Human behavior data is input into a simple human behavior recognition network; the preprocessed data is transformed into a shape suitable for the one-dimensional convolutional layer, and is input into the feature extraction and fusion unit in batches. The data shape is ,in is the batch size, is the number of data channels, is the window length; The feature data output by the feature pre-extraction unit is input into the local and global adaptive spatiotemporal feature extraction units and feature fusion units connected in sequence; S42: Extraction of local and global spatiotemporal features; the data of the input module is processed by the initialization unit, which contains a one-dimensional convolution module, which is composed of a one-dimensional convolution layer, a batch normalization layer, an activation layer, and a maximum pooling layer connected in sequence. The convolution kernel size of the one-dimensional convolution layer is 3, the stride size is 1, the padding method is SAME, and the nonlinear activation function selects the ReLU function; the calculation method of the one-dimensional convolution module is shown in formula (9): （9）； in, Represents the feature map List, represents the convolution kernel List, represents the bias term; The local and global adaptive spatiotemporal feature extraction unit constructs an adaptive time kernel consisting of a local branch and a global branch. The local branch adopts a local sensitive modeling scheme to capture short-term dynamic changes by establishing a position-sensitive importance weight. The extraction is shown in formula (10). The global branch is based on the local invariant modeling idea and obtains global information and long-term dependency information by constructing a dynamic aggregation convolution kernel. The extraction is shown in formula (11). The overall process is shown in formula (12): （10）； in, represents the local branch transformation, represents the multiplication operator, represents a one-dimensional convolution operation, and Respectively represent Sigmoid activation function and ReLU activation function; （11）； in represents the global branch transformation, represents the equivalent transformation function of the fully connected layer operation, Represent the weights of the two FC layers respectively; in order to realize the interaction between the information obtained by the global branch and the information obtained by the local branch, finally, the outputs of the two branches are convolved; （12）； in, represents the output feature of the time adaptive module, Represents the convolution operator.

Images