A behavior recognition method for small kernel convolution reindexing

Abstract

This application belongs to the field of human behavior recognition technology and provides a behavior recognition method using small kernel convolution reindexing. The method includes: acquiring multi-channel inertial sensor time-series data and labeling behavior categories; processing missing values, filtering, normalizing, and segmenting the sensor data using a sliding window to obtain time window samples, which are then divided into training, validation, and test sets; inputting the training set samples into a pre-defined TempRF-Net model, which extracts spatiotemporal features and reorders and fuses these features to map them into prediction results; calculating the loss based on the prediction results and labeled categories, iteratively updating the model parameters, and evaluating performance using the validation set to determine the optimal model parameters; loading the optimal model parameters into the model, performing behavior recognition using test set samples, and outputting the recognition results. This application can improve the accuracy and generalization ability of human behavior recognition while maintaining a lightweight model.

Description

Technical Field

[0001] This application belongs to the field of human behavior recognition technology, and in particular relates to a behavior recognition method using small kernel convolution reindexing. Background Technology

[0002] Human behavior recognition is an important research direction in the field of intelligent sensing, and it is widely used in scenarios such as health monitoring, motion analysis, and human-computer interaction. With the development of wearable devices and edge computing technology, human behavior recognition methods based on time series data from inertial sensors such as accelerometers and gyroscopes have become a research hotspot in this field due to their advantages such as low power consumption, ease of continuous data acquisition, and good privacy protection. Currently, the commonly used method is to use convolutional neural networks to model the sensor time series, extract local time features through small convolutional kernels, and combine pooling and fully connected layers to achieve behavior classification.

[0003] However, current methods have the following problems: conventional small convolutional kernels have limited receptive fields, making it difficult to fully capture long-term time-dependent and periodic motion patterns in human behavior. On the other hand, expanding the receptive field by stacking multiple layers of networks or explicitly introducing large convolutional kernels will increase the number of model parameters and computational overhead, making it difficult to balance recognition accuracy with the lightweight deployment requirements of edge devices. Summary of the Invention

[0004] This application provides a behavior recognition method using small kernel convolution reindexing, which can solve the problem that conventional small convolution kernels have limited receptive fields, making it difficult to fully capture long-term time dependence and periodic motion patterns in human behavior. On the other hand, expanding the receptive field by stacking multiple layers of networks or explicitly introducing large convolution kernels will lead to an increase in the number of model parameters and computational overhead, making it difficult to balance recognition accuracy with the lightweight deployment requirements of edge devices.

[0005] In a first aspect, embodiments of this application provide a behavior recognition method using small kernel convolution reindexing, comprising: S1, collecting multi-channel inertial sensor time-series data during human behavior and labeling the corresponding human behavior categories; S2, preprocessing the sensor time-series data, the preprocessing including missing value handling, filtering, normalization, and sliding window segmentation, obtaining time window samples of consistent length, and dividing the time window samples into training set, validation set, and test set; S3, inputting the time window samples in the training set into a preset TempRF-Net human behavior recognition model, and mapping them to prediction results after spatiotemporal feature extraction and temporal feature rearrangement and fusion; S4, calculating a loss function based on the prediction results and the behavior categories labeled in S1, and iteratively updating the parameters of the human behavior recognition model based on the loss function; using the validation set to evaluate the recognition performance of the human behavior recognition model after parameter updates, and determining the model parameters corresponding to the best validation performance as the optimal model parameters; S5, loading the optimal model parameters into the human behavior recognition model, using the time window samples in the test set to perform human behavior recognition, and outputting the corresponding human behavior recognition results.

[0006] In one possible implementation of the first aspect, the time window samples in S2 above are organized as a two-dimensional feature representation consisting of a time dimension and a sensor dimension, and the input tensor is formed through channel expansion:

[0007]

[0008] in, Let X represent the real number field, and let X represent the input tensor. This indicates the batch size, and 1 indicates the channel dimension. Indicates the length of the time window. Indicates the number of sensor axes or sensor channels.

[0009] Optionally, in another possible implementation of the first aspect, the preset TempRF-Net human behavior recognition model sequentially includes an input layer, a Stem downsampling layer, a multi-stage deep feature extraction layer, a pooling aggregation layer, a normalization layer, and a classification output layer. In step S3, the time window samples from the training set are input into the preset TempRF-Net human behavior recognition model, and after spatiotemporal feature extraction and temporal feature rearrangement and fusion, they are mapped to the prediction result, specifically including:

[0010] The time window samples are input into the input layer, tensorized, and the output model input tensor is generated.

[0011] The model input tensor is fed into the Stem downsampling layer, and downsampling is performed along the time dimension to output downsampled features.

[0012] The downsampled features are input into a multi-stage deep feature extraction layer, and multi-stage temporal feature extraction and temporal feature rearrangement and fusion are performed sequentially to output deep temporal features.

[0013] Deep temporal features are input into a pooling aggregation layer for global aggregation processing, and the aggregated feature vector is output.

[0014] The aggregated feature vector is input into the normalization layer for distribution alignment, and the normalized feature vector is output.

[0015] The normalized feature vector is input into the classification output layer, mapped to the predicted probability of each human behavior category, and the prediction result is output.

[0016] Optionally, in another possible implementation of the first aspect, the multi-stage deep feature extraction layer includes multiple stages stacked sequentially, each stage including at least one TempRF block, with TempRF blocks within the same stage connected sequentially. The downsampled features are input to the multi-stage deep feature extraction layer, and multi-stage temporal feature extraction and temporal feature rearrangement and fusion processing are performed sequentially to output deep temporal features, specifically including:

[0017] The downsampled features are input into the first TempRF block of the first stage. After temporal feature extraction and temporal feature rearrangement and fusion processing by the TempRF block, the processing result of the current TempRF block is output.

[0018] The current TempRF block processing result is input into the next TempRF block in the same phase for further processing, until all TempRF blocks in the current phase are processed, and the current phase processing result is output.

[0019] The current stage processing result is input into the inter-stage downsampling layer, downsampling is performed along the time dimension, and the downsampled stage features are output.

[0020] The downsampled stage features are input into the first TempRF block of the next stage. The TempRF block processing and inter-stage downsampling processing are repeated until all stages are processed, and the deep temporal features are output.

[0021] Optionally, in another possible implementation of the first aspect, the downsampled features are input into the first TempRF block of the first stage. After temporal feature extraction and temporal feature rearrangement and fusion processing by the TempRF block, the processing result of the current TempRF block is output, specifically including:

[0022] The downsampled features are input into the TempRF convolution module in the TempRF block, and after temporal reindexed convolution processing, the temporal reindexed convolution features are output.

[0023] The temporally reindexed convolutional features are input into the multilayer perceptron module in the TempRF block. After being processed by dimension permutation, layer normalization and multilayer perceptron in sequence, the channel-mixed features are output.

[0024] The channel blending features are restored to their original dimensions and then residually fused with the original input features of the TempRF block. After random depth regularization, the processing result of the current TempRF block is output.

[0025] Optionally, in another possible implementation of the first aspect, the downsampled features are input into the TempRF convolution module in the TempRF block, and after temporal reindexing convolution processing, the temporal reindexed convolution features are output, specifically including:

[0026] The downsampled features input to the TempRF convolution module are divided into active channel features and bypass channel features along the channel dimension;

[0027] The active channel features are input into at least one small kernel depth convolution branch for local temporal feature extraction, and the extraction results of each branch are aggregated to output the aggregated local temporal response.

[0028] Based on the target equivalent large convolution kernel size and the small convolution kernel size of the small kernel depth convolution branch, the aggregated local time response is decomposed into multiple sets of small kernel convolution responses;

[0029] Discrete temporal offsets are applied to the small kernel convolution responses of each group for temporal alignment. The aligned responses of each group are aggregated to output the equivalent large kernel convolution receptive field response.

[0030] After residual fusion of the equivalent large kernel convolution receptive field response and the active channel features, it is concatenated with the bypass channel features along the channel dimension to output the temporally reindexed convolutional features.

[0031] Optionally, in another possible implementation of the first aspect, the above-mentioned decomposition of the aggregated local temporal response into multiple sets of small-kernel convolutional responses based on the target equivalent large convolutional kernel size and the small convolutional kernel size of the small-kernel depthwise convolutional branch specifically includes:

[0032] Let the target equivalent large convolution kernel size be... The kernel size of the small kernel depthwise convolution branch is Then the number of decomposition groups of the aggregated local time response is:

[0033]

[0034] Based on the number of decomposition groups, the aggregated local time response is divided along the channel dimension into... Group small kernel convolution response:

[0035]

[0036] in, Indicates the first The response of small kernel convolution.

[0037] Optionally, in another possible implementation of the first aspect, after decomposing the aggregated local time response into multiple sets of small-kernel convolutional responses, the process further includes an expansion padding process, specifically including:

[0038] The kernel size of the mini-convolutional branch is... Then the expanded fill size P is:

[0039]

[0040] Based on the padding size P, the time dimension and sensor dimension of the feature maps containing multiple small kernel convolution responses are expanded and padded respectively. Under the condition of a stride of 1 and a dilation rate of 1, the length of the time dimension of the intermediate feature map after expansion and padding is... and sensor dimension length satisfy:

[0041]

[0042] in, and These represent the time dimension length and sensor dimension length of the feature map before expansion and padding, respectively.

[0043] Optionally, in another possible implementation of the first aspect, the above-mentioned application of discrete temporal offsets to each group of small-kernel convolutional responses for temporal alignment, and aggregation of the aligned responses to output an equivalent large-kernel convolutional receptive field response, specifically including:

[0044] Let the first The temporal alignment offset of the group little kernel convolution response is:

[0045]

[0046] Through the discrete time series reindexing operator For the Group mini-kernel convolution responses are time-aligned with offsets Temporal transfer and coordinate alignment are performed to obtain the aligned kernel-small convolutional responses for each group.

[0047] The aligned small-kernel convolution responses from each group are aggregated to obtain the equivalent large-kernel convolution receptive field response:

[0048]

[0049] in, This represents the equivalent large kernel convolution receptive field response.

[0050] Optionally, in another possible implementation of the first aspect, after residual fusion of the equivalent large-kernel convolutional receptive field response and the active channel features, the above is concatenated with the bypass channel features along the channel dimension to output temporally reindexed convolutional features, specifically including:

[0051] Multiple parallel temporal reindexing branches are employed, with different branches using different offset orders to reindex the equivalent large kernel convolution receptive field response R, resulting in the output of each branch. , , Among them, the time-series reindexing branches include two or more of the following: natural order branches, reverse order branches, and out-of-order offset branches.

[0052] Output to each branch , , Batch normalization is performed separately, and the outputs of each branch after batch normalization are fused to obtain the temporal re-indexing fusion feature. :

[0053]

[0054] in, This indicates a batch normalization operation;

[0055] Time-series reindexing fusion features Features of active channels Perform residual fusion to obtain the active channel output. :

[0056]

[0057] Output the active channel Bypass channel characteristics By concatenating along the channel dimension, the output of the temporally reindexed convolutional module is obtained. :

[0058]

[0059] Furthermore, during the model inference stage, a structural reparameterization technique is used to merge multiple small-kernel depthwise convolution branches into a single equivalent depthwise convolution branch:

[0060]

[0061] in, Indicates the first Depth convolution operation with small kernel depthwise convolution branches , This indicates the number of small kernel depthwise convolution branches configured in parallel during the training phase. This represents the equivalent depthwise convolution operation after structural reparameterization and merging. Indicates input feature placeholders ;

[0062] Among them, the merged equivalent convolution kernel weights and bias They are respectively:

[0063]

[0064] in, Indicates the training phase The kernel weights of the depthwise convolutional branches with small kernels. Indicates the training phase Bias terms for small kernel depthwise convolution branches;

[0065] Output temporally reindexed convolutional features .

[0066] Beneficial Effects: This application extracts local temporal features through small-kernel depthwise convolutions and employs temporal reindexing technology to group, offset-align, and aggregate multiple sets of small-kernel convolutional responses, effectively expanding the temporal receptive field without increasing the size of the convolutional kernel, thus significantly enhancing the modeling ability for motion trends and periodic changes over long time spans. Simultaneously, by dividing the input features into active and bypass channels, only the active channel participates in the temporal reindexing calculation, significantly reducing the overall computational cost. The use of multiple parallel reindexing branches with natural order, reverse order, and out-of-order offsets enhances the model's robustness to behavioral phase changes and boundary offsets. During the inference stage, structural reparameterization merges multi-branch convolutions into a single-path equivalent convolution, further reducing computational cost and inference latency. Furthermore, network downsampling is performed only along the temporal dimension, effectively preserving the structural relationships and complementary information between multi-axis sensors. Compared with existing methods, this application significantly improves the accuracy and generalization ability of behavior recognition while maintaining the lightweight nature of the model. It has low parameter and computational cost, making it easy to deploy in smartphones, smartwatches, smart bracelets and embedded edge devices, and has good engineering practical value. Attached Figure Description

[0067] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0068] Figure 1 This is a flowchart illustrating a behavior recognition method for small kernel convolutional reindexing provided in an embodiment of this application;

[0069] Figure 2 This is a schematic diagram of the structure of a TempRF convolution module provided in an embodiment of this application;

[0070] Figure 3 This is a schematic diagram of the training loss curve provided in an embodiment of this application. Detailed Implementation

[0071] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0072] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0073] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0074] As used in this application specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if detected [the described condition or event]" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once detected [the described condition or event]," or "in response to detection [the described condition or event]."

[0075] Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0076] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0077] The following is a detailed description of a behavior recognition method for small kernel convolutional reindexing provided in this application, with reference to the accompanying drawings.

[0078] Figure 1 The diagram shows a flowchart of a behavior recognition method for small kernel convolution reindexing provided in an embodiment of this application.

[0079] like Figure 1 As shown, the behavior recognition method of small kernel convolution reindexing includes the following steps:

[0080] S1. Collect multi-channel inertial sensor time series data during human behavior and label the corresponding human behavior categories;

[0081] S2. Preprocess the sensor time series data. The preprocessing includes missing value handling, filtering, normalization and sliding window segmentation to obtain time window samples of consistent length. The time window samples are then divided into training set, validation set and test set.

[0082] Furthermore, in this embodiment, the time window sample organization in S2 above is a two-dimensional feature representation composed of the time dimension and the sensor dimension, and the input tensor is formed through channel expansion:

[0083]

[0084] in, Let X represent the real number field, and let X represent the input tensor. This indicates the batch size, and 1 indicates the channel dimension. Indicates the length of the time window. Indicates the number of sensor axes or sensor channels.

[0085] For example, sample fragments can be divided into training, validation, and test sets in a 7:1:2 ratio. The training set is used for learning model parameters, the validation set is used to select the optimal parameter combination, and the test set is used for final performance evaluation.

[0086] In the embodiments of this application, the multi-channel sensor time series can be converted into an input tensor suitable for two-dimensional convolution processing through the above representation method, so that the model can perform hierarchical feature modeling mainly along the time dimension while preserving the sensor axis structure relationship.

[0087] S3. Input the time window samples in the training set into the preset TempRF-Net human behavior recognition model. After spatiotemporal feature extraction and temporal feature rearrangement and fusion, the results are mapped to the prediction results.

[0088] Furthermore, in this embodiment, the aforementioned preset TempRF-Net human behavior recognition model sequentially includes an input layer, a Stem downsampling layer, a multi-stage deep feature extraction layer, a pooling aggregation layer, a normalization layer, and a classification output layer; the aforementioned S3 specifically includes:

[0089] S31. Input the time window samples into the input layer, perform tensor quantization, and output the model input tensor.

[0090] S32. Input the model input tensor into the Stem downsampling layer, perform downsampling along the time dimension, and output the downsampled features;

[0091] S33. Input the downsampled features into the multi-stage deep feature extraction layer, and perform multi-stage temporal feature extraction and temporal feature rearrangement and fusion processing in sequence to output deep temporal features;

[0092] S34. Input the deep temporal features into the pooling aggregation layer, perform global aggregation processing, and output the aggregated feature vector;

[0093] S35. Input the aggregated feature vector into the normalization layer, perform distribution alignment processing, and output the normalized feature vector;

[0094] S36. Input the normalized feature vector into the classification output layer, map it to the predicted probability of each human behavior category, and output the prediction result.

[0095] In this embodiment, the Stem downsampling layer and the inter-stage downsampling layer only downsample along the time dimension and do not over-compress the sensor dimension, thereby preserving the structural relationships and complementary information between different sensor axes while extracting high-level temporal features.

[0096] Furthermore, in this embodiment, the multi-stage deep feature extraction layer includes multiple stages stacked sequentially, each stage including at least one TempRF block, and the TempRF blocks within the same stage are connected sequentially; S33 specifically includes:

[0097] S331. Input the downsampled features into the first TempRF block of the first stage. After the TempRF block performs temporal feature extraction and temporal feature rearrangement and fusion, output the processing result of the current TempRF block.

[0098] S332. Input the current TempRF block processing result into the next TempRF block in the same stage to continue processing until all TempRF blocks in the current stage are processed, and output the current stage processing result.

[0099] S333: Input the current stage processing result into the inter-stage downsampling layer, perform downsampling processing along the time dimension, and output the downsampled stage features;

[0100] S334. Input the downsampled stage features into the first TempRF block of the next stage, repeat the TempRF block processing and inter-stage downsampling processing until all stages are processed, and output the deep temporal features.

[0101] Furthermore, in the embodiments of this application, the above-mentioned S331 specifically includes:

[0102] S3311. Input the downsampled features into the TempRF convolution module in the TempRF block. After temporal reindexed convolution processing, output the temporal reindexed convolution features.

[0103] S3312. Input the temporally reindexed convolutional features into the multilayer perceptron module in the TempRF block. After sequentially undergoing dimension permutation, layer normalization, and multilayer perceptron processing, the channel-mixed features are output.

[0104] S3313. After dimensionality restoration, the channel blending features are residually fused with the original input features of the TempRF block, and then processed by random depth regularization to output the current TempRF block processing result.

[0105] In one embodiment, within each TempRF block, a TempRF convolutional module replaces the traditional large-kernel depthwise convolution to construct an equivalent large temporal receptive field. Given input features The calculation process of the TempRF block is as follows:

[0106]

[0107] in, This represents the convolutional features output by the TempRF convolutional module. This represents the TempRF convolutional module; Indicates by Channel hybrid features obtained through dimension permutation, layer normalization, and multilayer perceptron processing. This indicates that the features will be converted from NCHW format to NHWC format in order to perform layer normalization and linear transformation; Representation layer normalization; This indicates a channel mixing module composed of Linear-GELU-Linear; This indicates a channel-by-channel learnable scaling parameter; This represents element-wise multiplication; This represents a random depth regularization operation; This indicates the output characteristics of the TempRF block.

[0108] Furthermore, in the embodiments of this application, such as Figure 2 As shown, the above S3311 specifically includes:

[0109] The downsampled features input to the TempRF convolution module are divided into active channel features and bypass channel features along the channel dimension;

[0110] The active channel features are input into at least one small kernel depth convolution branch for local temporal feature extraction, and the extraction results of each branch are aggregated to output the aggregated local temporal response.

[0111] Based on the target equivalent large convolution kernel size and the small convolution kernel size of the small kernel depth convolution branch, the aggregated local time response is decomposed into multiple sets of small kernel convolution responses;

[0112] Discrete temporal offsets are applied to the small kernel convolution responses of each group for temporal alignment. The aligned responses of each group are aggregated to output the equivalent large kernel convolution receptive field response.

[0113] After residual fusion of the equivalent large kernel convolution receptive field response and the active channel features, it is concatenated with the bypass channel features along the channel dimension to output the temporally reindexed convolutional features.

[0114] In one embodiment, the downsampled features input to the TempRF convolution module are divided into active channel features and bypass channel features along the channel dimension; the active channel features are input to at least one small kernel depth convolution branch for local temporal feature extraction, and the extraction results of each branch are aggregated to output the aggregated local temporal response, as follows:

[0115] The TempRF convolutional module first divides the input features into active channel features and secondary channel features according to the channel dimension. Let the input features be:

[0116]

[0117] in, Indicates batch size, Indicates the number of feature channels. Indicates the length of the time dimension. This represents the sensor dimension length. The input features are divided into:

[0118]

[0119] in, It represents active channel features and participates in the computation of small kernel convolution and temporal reindexing; This indicates the characteristics of bypass channels, which are directly retained and participated in subsequent fusion through bypass connections. This channel partitioning method can reduce the number of channels participating in complex time-series re-indexing calculations, thereby reducing the overall computational overhead.

[0120] Active channel characteristics Local temporal responses are extracted using multi-path, low-kernel, depthwise convolutional branches. Assume there are a total of [number missing] training phases. There are 1 parallel small-kernel depthwise convolution branches, with a small convolution kernel size of 1. The aggregated response of multi-path small kernel depthwise convolution, i.e., the aggregated local time response, is:

[0121]

[0122] in, Indicates the first Multiple small-kernel depthwise convolution branches. Through multi-path small-kernel depthwise convolution, the model is able to learn multiple complementary local temporal patterns during the training phase.

[0123] Furthermore, in the embodiments of this application, the above-mentioned decomposition of the aggregated local temporal response into multiple groups of small-kernel convolutional responses based on the target equivalent large convolutional kernel size and the small-kernel depth convolutional branch small convolutional kernel size specifically includes:

[0124] Let the target equivalent large convolution kernel size be... The kernel size of the small kernel depthwise convolution branch is Then the number of decomposition groups of the aggregated local time response is:

[0125]

[0126] Based on the number of decomposition groups, the aggregated local time response is divided along the channel dimension into... Group small kernel convolution response:

[0127]

[0128] in, Indicates the first The response of small kernel convolution.

[0129] Furthermore, in the embodiments of this application, after decomposing the aggregated local time response into multiple groups of small kernel convolutional responses, the process further includes expansion padding, specifically including:

[0130] The kernel size of the mini-convolutional branch is... Then the expanded fill size P is:

[0131]

[0132] Based on the padding size P, the time dimension and sensor dimension of the feature maps containing multiple small kernel convolution responses are expanded and padded respectively. Under the condition of a stride of 1 and a dilation rate of 1, the length of the time dimension of the intermediate feature map after expansion and padding is... and sensor dimension length satisfy:

[0133]

[0134] in, and These represent the time dimension length and sensor dimension length of the feature map before expansion and padding, respectively.

[0135] Furthermore, in the embodiments of this application, the above-mentioned application of discrete temporal offsets to perform temporal alignment on each group of small kernel convolution responses, and the aggregation of the aligned responses to output an equivalent large kernel convolution receptive field response, specifically includes:

[0136] Let the first The temporal alignment offset of the group little kernel convolution response is:

[0137]

[0138] Through the discrete time series reindexing operator For the Group mini-kernel convolution responses are time-aligned with offsets Temporal transfer and coordinate alignment are performed to obtain the aligned kernel-small convolutional responses for each group.

[0139] The aligned small-kernel convolution responses from each group are aggregated to obtain the equivalent large-kernel convolution receptive field response:

[0140]

[0141] in, This represents the equivalent large kernel convolution receptive field response.

[0142] Furthermore, in this embodiment, the above-mentioned residual fusion of the equivalent large kernel convolutional receptive field response and the active channel features, followed by concatenation with the bypass channel features along the channel dimension, outputs temporally reindexed convolutional features, specifically including:

[0143] Multiple parallel temporal reindexing branches are employed, with different branches using different offset orders to reindex the equivalent large kernel convolution receptive field response R, resulting in the output of each branch. , , Among them, the time-series reindexing branches include two or more of the following: natural order branches, reverse order branches, and out-of-order offset branches.

[0144] Output to each branch , , Batch normalization is performed separately, and the outputs of each branch after batch normalization are fused to obtain the temporal re-indexing fusion feature. :

[0145]

[0146] in, This indicates a batch normalization operation;

[0147] Time-series reindexing fusion features Features of active channels Perform residual fusion to obtain the active channel output. :

[0148]

[0149] Output the active channel Bypass channel characteristics By concatenating along the channel dimension, the output of the temporally reindexed convolutional module is obtained. :

[0150]

[0151] Furthermore, during the model inference stage, a structural reparameterization technique is used to merge multiple small-kernel depthwise convolution branches into a single equivalent depthwise convolution branch:

[0152]

[0153] in, Indicates the first Depth convolution operation with small kernel depthwise convolution branches , This indicates the number of small kernel depthwise convolution branches configured in parallel during the training phase. This represents the equivalent depthwise convolution operation after structural reparameterization and merging. Indicates input feature placeholders ;

[0154] Among them, the merged equivalent convolution kernel weights and bias They are respectively:

[0155]

[0156] in, Indicates the training phase The kernel weights of the depthwise convolutional branches with small kernels. Indicates the training phase Bias terms for small kernel depthwise convolution branches;

[0157] Output temporally reindexed convolutional features .

[0158] In this embodiment, to enhance the diversity of temporal coverage and reduce branch correlation caused by a fixed reindexing order, the TempRF convolution module sets up three parallel temporal reindexing branches. Branch-level batch normalization can align the feature distributions of different reindexing branches, preventing any one branch from dominating the fusion process, thereby improving model training stability. TempRF-Net can use small-kernel depthwise convolutions as local temporal feature extraction units, and form an equivalent large temporal receptive field through temporal reindexing, three-branch fusion, and active / bypass channel fusion, thereby enhancing the model's ability to model long-term temporal dependencies and periodic human motion patterns without explicitly introducing large convolutional kernels.

[0159] S4. Calculate the loss function based on the prediction results and the behavior categories labeled in S1, and iteratively update the parameters of the human behavior recognition model based on the loss function; use the validation set to evaluate the recognition performance of the human behavior recognition model after parameter update, and determine the model parameters corresponding to the best validation performance as the optimal model parameters;

[0160] S5. Load the optimal model parameters into the human behavior recognition model, use the time window samples in the test set to perform human behavior recognition, and output the corresponding human behavior recognition results.

[0161] In one embodiment, training set samples are input into the TempRF-Net model for training. The model obtains human behavior category prediction vectors through forward propagation, and calculates a loss function using these vectors and the true category labels. During training, cross-entropy loss or focus loss can be used as the optimization objective, and model parameters are optimized through backpropagation and gradient update algorithms. Specifically, the training epochs are set to 200, the batch size to 128, the optimizer is AdamW, and the initial learning rate is set to... A warmup-cosine learning rate scheduling strategy is employed. During training, validation set accuracy, F1 score, or loss value are used as the model selection criteria. When the validation set performance reaches its current optimal level, the corresponding model parameters are saved. Figure 3As shown, the training loss curve decreases rapidly in the early stages of training and then gradually stabilizes, indicating that the model can converge effectively. This phenomenon demonstrates that the TempRF-Net model exhibits good optimization stability during training and can gradually learn temporal features used to distinguish different human behavior categories. Once the model achieves optimal recognition performance on the validation set, the trained model parameters are saved as a weight file or exported as a model file suitable for terminal inference. The model file can be deployed to edge computing devices. The terminal device collects human motion sensor data in real time and converts the continuous sensor data stream into fixed-length sample segments according to the preprocessing method in step one. These segments are then input into the trained TempRF-Net model for inference, outputting the corresponding human behavior category.

[0162] This application provides a behavior recognition method using small kernel convolution reindexing. The method collects multi-channel inertial sensor time-series data and labels behavior categories. It processes the sensor data by handling missing values, filtering, normalizing, and segmenting it using a sliding window to obtain time-window samples, which are then divided into training, validation, and test sets. The training set samples are input into a pre-defined TempRF-Net model, and after spatiotemporal feature extraction and temporal feature rearrangement and fusion, the results are mapped to predictions. The loss is calculated based on the predictions and labeled categories, and the model parameters are iteratively updated. Performance is evaluated using the validation set, and the model parameters with the best validation performance are determined as the optimal model parameters. The optimal model parameters are then loaded into the model, and behavior recognition is performed using test set samples to output the recognition results. This application extracts local temporal features through small-kernel depthwise convolutions and employs temporal reindexing technology to group, offset-align, and aggregate multiple small-kernel convolutional responses. This effectively expands the temporal receptive field without increasing the size of the convolutional kernel, significantly enhancing the modeling ability for motion trends and periodic changes over long time spans. Simultaneously, by dividing the input features into active and bypass channels, only the active channel participates in the temporal reindexing calculation, significantly reducing overall computational overhead. Multiple parallel reindexing branches with natural order, reverse order, and out-of-order offsets are fused to enhance the model's robustness to behavioral phase changes and boundary offsets. During the inference stage, structural reparameterization merges multi-branch convolutions into a single equivalent convolution, further reducing computational cost and inference latency. Furthermore, network downsampling is performed only along the temporal dimension, effectively preserving the structural relationships and complementary information between multi-axis sensors. Compared to existing methods, this application significantly improves behavior recognition accuracy and generalization ability while maintaining a lightweight model. With low parameter and computational costs, it is easy to deploy in smartphones, smartwatches, smart bracelets, and embedded edge devices, demonstrating significant engineering practical value.

[0163] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0164] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A method for recognizing behavior through small kernel convolutional reindexing, characterized in that, Includes the following steps: S1. Collect multi-channel inertial sensor time series data during human behavior and label the corresponding human behavior categories; S2. Preprocess the sensor time series data. The preprocessing includes missing value handling, filtering, normalization and sliding window segmentation to obtain time window samples of consistent length. The time window samples are then divided into training set, validation set and test set. S3. Input the time window samples in the training set into the preset TempRF-Net human behavior recognition model, and after spatiotemporal feature extraction and temporal feature rearrangement and fusion, map them into the prediction result; S4. Calculate the loss function based on the prediction result and the behavior category marked in S1, and iteratively update the parameters of the human behavior recognition model based on the loss function; use the validation set to evaluate the recognition performance of the human behavior recognition model after parameter update, and determine the model parameters corresponding to the best validation performance as the optimal model parameters; S5. Load the optimal model parameters into the human behavior recognition model, use the time window samples in the test set to perform human behavior recognition, and output the corresponding human behavior recognition results.

2. The method according to claim 1, characterized in that, The time window samples in S2 are organized as a two-dimensional feature representation consisting of the time dimension and the sensor dimension, and the input tensor is formed through channel expansion: ； in, Let X represent the real number field, and let X represent the input tensor. This indicates the batch size, and 1 indicates the channel dimension. Indicates the length of the time window. Indicates the number of sensor axes or sensor channels.

3. The method according to claim 2, characterized in that, The preset TempRF-Net human behavior recognition model sequentially includes an input layer, a Stem downsampling layer, a multi-stage deep feature extraction layer, a pooling aggregation layer, a normalization layer, and a classification output layer; step S3 involves inputting the time window samples from the training set into the preset TempRF-Net human behavior recognition model, and mapping them to the prediction result after spatiotemporal feature extraction and temporal feature rearrangement and fusion, specifically including: The time window samples are input into the input layer, tensorized, and the model input tensor is output. The model input tensor is fed into the Stem downsampling layer, and downsampling is performed along the time dimension to output downsampled features. The downsampled features are input into the multi-stage deep feature extraction layer, and multi-stage temporal feature extraction and temporal feature rearrangement and fusion processing are performed sequentially to output deep temporal features; The deep temporal features are input into the pooling aggregation layer for global aggregation processing, and the aggregated feature vector is output. The aggregated feature vector is input into the normalization layer for distribution alignment processing, and a normalized feature vector is output. The normalized feature vector is input into the classification output layer, mapped to the predicted probability of each human behavior category, and the prediction result is output.

4. The method according to claim 3, characterized in that, The multi-stage deep feature extraction layer comprises multiple stacked stages, each stage including at least one TempRF block, with TempRF blocks within the same stage connected sequentially. The process of inputting the downsampled features into the multi-stage deep feature extraction layer, performing multi-stage temporal feature extraction and temporal feature rearrangement and fusion processing sequentially, and outputting deep temporal features specifically includes: The downsampled features are input into the first TempRF block of the first stage. After the TempRF block performs temporal feature extraction and temporal feature rearrangement and fusion, the processing result of the current TempRF block is output. The current TempRF block processing result is input into the next TempRF block in the same stage for further processing until all TempRF blocks in the current stage are processed, and the current stage processing result is output. The current stage processing result is input into the inter-stage downsampling layer, downsampling is performed along the time dimension, and the downsampled stage features are output. The downsampled stage features are input into the first TempRF block of the next stage. The TempRF block processing and inter-stage downsampling processing are repeated until all stages are processed, and the deep temporal features are output.

5. The method according to claim 4, characterized in that, The process of inputting the downsampled features into the first TempRF block of the first stage, and then performing temporal feature extraction and temporal feature rearrangement and fusion processing on the TempRF block, and outputting the processing result of the current TempRF block, specifically includes: The downsampled features are input into the TempRF convolution module in the TempRF block, and after temporal reindexed convolution processing, the temporal reindexed convolution features are output. The temporally reindexed convolutional features are input into the multilayer perceptron module in the TempRF block. After sequential processing by dimension permutation, layer normalization and multilayer perceptron, channel-mixed features are output. The channel blending features are restored to their original dimensions and then residually fused with the original input features of the TempRF block. After random depth regularization, the processing result of the current TempRF block is output.

6. The method according to claim 5, characterized in that, The step of inputting the downsampled features into the TempRF convolution module in the TempRF block, and outputting the temporally reindexed convolution features after temporal reindexing convolution processing, specifically includes: The downsampled features input to the TempRF convolution module are divided into active channel features and bypass channel features along the channel dimension; The active channel features are input into at least one small kernel depth convolution branch for local temporal feature extraction, and the extraction results of each branch are aggregated to output the aggregated local temporal response. Based on the target equivalent large convolutional kernel size and the small convolutional kernel size of the small kernel depth convolutional branch, the aggregated local time response is decomposed into multiple sets of small kernel convolutional responses; Discrete temporal offsets are applied to the small kernel convolution responses of each group for temporal alignment. The aligned responses of each group are aggregated to output the equivalent large kernel convolution receptive field response. After residual fusion of the equivalent large kernel convolutional receptive field response and the active channel feature, it is concatenated with the bypass channel feature along the channel dimension to output the temporal reindexed convolutional feature.

7. The method according to claim 6, characterized in that, The step of decomposing the aggregated local temporal response into multiple sets of small-kernel convolutional responses based on the target equivalent large convolutional kernel size and the small-kernel depthwise convolutional branch size specifically includes: Let the target equivalent large convolution kernel size be... The kernel size of the small kernel depthwise convolution branch is Then the number of decomposition groups of the aggregated local time response is: ； Based on the number of decomposition groups, the aggregated local time response is divided along the channel dimension into... Group small kernel convolution response: ； in, Indicates the first The response of small kernel convolution.

8. The method according to claim 7, characterized in that, After decomposing the aggregated local time response into multiple sets of small kernel convolutional responses, the process further includes expansion padding, specifically including: The kernel size of the mini-convolutional branch is as follows: Then the expanded fill size P is: ； Based on the expansion padding size P, the time dimension and sensor dimension of the feature maps containing the multiple sets of small kernel convolution responses are expanded and padded respectively. Under the condition of a stride of 1 and an inflation rate of 1, the length of the time dimension of the intermediate feature map after expansion and padding is... and sensor dimension length satisfy: ； in, and These represent the time dimension length and sensor dimension length of the feature map before expansion and padding, respectively.

9. The method according to claim 8, characterized in that, The step of applying discrete temporal offsets to each group of small-kernel convolutional responses for temporal alignment, and then aggregating the aligned responses to output an equivalent large-kernel convolutional receptive field response, specifically includes: Let the first The temporal alignment offset of the group little kernel convolution response is: ； Through the discrete time series reindexing operator For the first The small kernel convolution response is aligned with the time-series offset. Temporal transfer and coordinate alignment are performed to obtain the aligned kernel-small convolutional responses for each group. The aligned small-kernel convolution responses from each group are aggregated to obtain the equivalent large-kernel convolution receptive field response: ； in, This represents the equivalent large kernel convolution receptive field response.

10. The method according to claim 9, characterized in that, The step of residually fusing the equivalent large-kernel convolutional receptive field response with the active channel feature and concatenating it along the channel dimension with the bypass channel feature to output the temporal reindexed convolutional feature specifically includes: Multiple parallel temporal reindexing branches are employed, with different branches using different offset orders to reindex the equivalent large kernel convolutional receptive field response R, resulting in the output of each branch. , , The time-series reindexing branch includes two or more of the following: natural order branch, reverse order branch, and out-of-order offset branch. Output to each branch , , Batch normalization is performed separately, and the outputs of each branch after batch normalization are fused to obtain the temporal re-indexing fusion feature. : ； in, This indicates a batch normalization operation; The time-series reindexing fusion feature With the active channel features Perform residual fusion to obtain the active channel output. : ； Output the active channel With the bypass channel features The output of the temporal reindexed convolutional module is obtained by concatenating along the channel dimension. : ； Furthermore, during the model inference phase, a structural reparameterization technique is used to merge multiple small-kernel depthwise convolution branches into a single equivalent depthwise convolution branch: ； in, Indicates the first Depth convolution operation with small kernel depthwise convolution branches , This indicates the number of small kernel depthwise convolution branches configured in parallel during the training phase. This represents the equivalent depthwise convolution operation after structural reparameterization and merging. Indicates input feature placeholders ; Among them, the merged equivalent convolution kernel weights and bias They are respectively: ； in, Indicates the training phase The kernel weights of the depthwise convolutional branches with small kernels. Indicates the training phase Bias terms for small kernel depthwise convolution branches; Output the temporally reindexed convolutional features .

Images