A wearable activity recognition method based on multi-modal learnable patching

Abstract

The application discloses a wearable activity recognition method based on multi-modal learnable blocks, and belongs to the technical field of wearable human activity recognition, and comprises the following steps: inputting activity data into a modal shared block network and a modal specific block network respectively to obtain modal shared block information and modal specific block information; resampling the activity data by using the modal shared block information to obtain modal shared block activity data, and resampling the modal shared block activity data by using the modal specific block information to obtain modal specific block activity data; extracting a first data representation from the modal specific block activity data, encoding the modal shared block information to guide the modal shared block activity data to extract a second data representation, fusing the two types of representations to obtain a block representation, and aggregating all the block representations to send into a classifier to output a recognition result. The application can adaptively extract complete local semantics and local subtle patterns in the activity data, and improve the recognition accuracy and robustness in a complex activity scene.

Description

Technical Field

[0001] This invention belongs to the field of wearable human activity recognition technology, specifically relating to a wearable activity recognition method based on multimodal learnable blocks. Background Technology

[0002] Wearable Human Activity Recognition (WHAR) aims to infer human activity from data collected by wearable sensors. Due to the excellent privacy protection characteristics of wearable sensor data, WHAR has been widely used in smart homes, healthcare, motion monitoring, and industrial assistance.

[0003] Feature extraction, which aims to extract discriminative information between activities from wearable sensor data, is a crucial step in wearable sensor data analysis (WHAR). However, WHAR incorporates strong inter-activity similarities and intra-activity differences, posing significant challenges to feature extraction.

[0004] To address these challenges, early research primarily employed statistical feature extraction methods, relying on constructing statistical features from activity signals and combining them with machine learning classification algorithms for activity identification. For example, manual extraction of time-domain and frequency-domain statistical information from activity signals was used as features for activity identification. The advantages of these methods are their simplicity, intuitiveness, and strong interpretability. However, their drawback lies in the fact that the feature construction process heavily depends on rich domain knowledge, resulting in poor scalability. Furthermore, because these methods can only extract shallow features, they have limitations in uncovering complex patterns.

[0005] With the development of deep learning technology, feature extraction methods based on deep neural networks have been widely used in the WHAR field. Deep neural networks can automatically capture high-level semantic information in data in an end-to-end manner, overcoming the limitations of manual feature representation. Existing methods mainly use convolutional neural networks (CNNs), recurrent neural networks (RNNs), and deep hybrid models to model the temporal dependencies of wearable sensor data. Recent works have further utilized self-attention mechanisms to model intermodal correlations in wearable sensor data, improving the richness of feature representations.

[0006] It is worth noting that, due to semantic influences, human activity data often contains multiple cyclical variations. The aforementioned deep learning-based feature extraction methods, through explicit block segmentation or using CNN convolutional operations, can preserve the local semantics of activity data. However, these methods rely on fixed block lengths for segmentation, preserving only the local semantics of activity data within a specific cycle. They struggle to perceive complex patterns with variable cycles within the data, leading to potentially incomplete extracted local semantics. This can result in the fragmentation of activity patterns spanning multiple cycles, or the inclusion of sub-activities with different semantics within a single block, thereby reducing the discriminability of the features.

[0007] Therefore, designing a feature extraction method that can adaptively capture complete local semantics is key to improving WHAR recognition performance. Summary of the Invention

[0008] In view of the above, the purpose of this invention is to provide a wearable activity recognition method based on multimodal learnable blocks. By introducing multimodal learnable blocks into the wearable activity recognition model, the method can adaptively output the block information required by each block according to the input activity data, thereby effectively extracting features with complete local semantics from the activity data, thus improving the feature extraction capability of the wearable activity recognition model. This method has broad application prospects in fields such as smart homes, healthcare, motion monitoring, and industrial assistance.

[0009] To achieve the above-mentioned objectives, the present invention provides the following technical solution: This invention provides a wearable activity recognition method based on multimodal learnable blocks, comprising the following steps: S1, The collected multimodal activity time series data are preprocessed and then used to construct a training set; S2, input each batch of training samples from the training set into the wearable activity recognition model and execute: The activity data from each batch of training samples are input into the modality sharing block network and the modality-specific block network respectively to obtain modality sharing block information and modality-specific block information respectively. The activity data is resampled using the modality sharing block information to obtain modality sharing block activity data, and the modality-specific block activity data is resampled using the modality-specific block information to obtain modality-specific block activity data. The first data representation is obtained by extracting features from the modality-specific block activity data. The second data representation is obtained by encoding the modality-shared block information and then guiding the feature extraction of the modality-shared block activity data. The first and second data representations are then fused to obtain the block representation. The aggregated block representation is the final representation, which is fed into the classifier to output the activity recognition result; S3, calculate the loss between the activity recognition results and the true labels in each batch of training samples to update the model parameters, repeat the training until convergence, and use the trained model to identify new activity data to be tested.

[0010] Preferably, the step of preprocessing the collected multimodal activity time-series data to construct a training set includes: Outlier processing and normalization are performed on multimodal activity time-series data collected by various sensors. The processed data is divided into multiple activity data samples by using a sliding window. The activity data samples in each window are paired with their corresponding real labels of activity categories to obtain training samples. All training samples are used to construct a training set.

[0011] Preferably, the modality-sharing block network performs learnable feature mapping on the activity data, and the output modality-sharing block information includes the relative block length of each block and the block center offset of each block, both in the dimension of the number of blocks. The modality-specific block network also performs learnable feature mapping on the activity data, and the output modality-specific block information includes the relative block length of each block in each modality and the block center offset of each block in each modality, both in the dimension of the number of blocks. ×number of modes The starting and ending positions of each block are determined based on the relative block length and the block center offset.

[0012] Preferably, the modality-sharing block network and the modality-specific block network have the same structure but independent parameters. Both contain multiple convolutional blocks connected in sequence and a fully connected layer. Each convolutional block includes a one-dimensional convolutional layer, a batch normalization layer and a nonlinear activation function in sequence. Finally, the fully connected layer outputs the relative block length and the block center offset.

[0013] Preferably, the step of resampling the activity data using modal sharing block information to obtain modal sharing block activity data includes: Based on the modal sharing block information, calculate the intra-block sampling points of each block shared by all modes, where the first... The sampling points within a block are the first sampling points in the modal sharing block information. The relative block length and block center offset of the block are obtained by correcting the preset sampling points within the block and the preset block center; For each mode Using a linear interpolation algorithm, based on the first... Intra-block sampling points of the block for modal The activity data was resampled to obtain the modal data. Next Block output of a block; The block outputs of all modal blocks are concatenated to obtain the modal shared block activity data.

[0014] Preferably, the step of resampling modal shared block activity data using modal-specific block information to obtain modal-specific block activity data includes: Based on the modality-specific block information, the intra-block sampling points are independently calculated for each block for each mode, where the first... The modality of the first The sampling points within a block are the first sampling points based on the modality-specific block information. The modality of the first The relative block length and block center offset of the block are obtained by correcting the preset sampling points within the block and the preset block center; For each mode Using a linear interpolation algorithm, based on the first... The modality of the first Intra-block sampling points of the block for modal The modal shared block activity data is resampled to obtain the modal data. Next Block output of a block; The block outputs of all modalities are concatenated to obtain modality-specific block activity data.

[0015] Preferably, the step of extracting features from modality-specific block activity data to obtain a first data representation includes: Multiple independent convolutional neural networks, the same number as the number of modes, are used to process the modality-specific block activity data of each mode. Each convolutional neural network contains multiple convolutional layers and the convolution operation is performed along the time dimension. The outputs of each convolutional neural network are concatenated to obtain the first data representation.

[0016] Preferably, the step of encoding the modal sharing block information and then guiding the modal sharing block activity data to extract features to obtain a second data representation includes: The relative block length and block center offset in the modality sharing block information are input into two independent coding networks, and each block is encoded to obtain the relative block length coding feature and the block center offset coding feature. The two are then concatenated to obtain the modality sharing block information representation. For each mode modality Modal shared block activity data and modality Modal shared block information representation is input to the modality. The corresponding local subtle feature extractor contains multiple dynamic convolutional layers. Each dynamic convolutional layer generates a dynamic convolutional kernel based on the current input, performs a convolutional operation, and outputs the features of that layer. The features output by the local subtle feature extractors of all modalities are concatenated to obtain the second data representation.

[0017] Preferably, the fusion of the first data representation and the second data representation to obtain the block representation includes: After concatenating the first data representation with the second data representation, the data is fused through a two-dimensional convolutional layer. The fusion result is then passed sequentially through multiple multimodal convolutional modules and a fully connected layer. Each multimodal convolutional module includes a two-dimensional convolutional layer, a batch normalization layer, and a nonlinear activation function. Finally, the fully connected layer outputs a block representation.

[0018] This invention introduces a multimodal learnable block-based mechanism and a dynamic convolutional layer guided by block information, which can capture relatively complete local semantics in activity data, effectively improving the accuracy and robustness of wearable activity recognition. Compared with the prior art, the beneficial effects of this invention include at least: (1) This invention introduces a multimodal learnable block segmentation mechanism, which can adaptively output the block information (relative block length, block center offset) required for each block according to the input activity data, instead of using a fixed block segmentation method. This mechanism enables the model to dynamically adjust the block boundaries for different activity types and different sensor modalities, extract local segments with relatively complete semantics from the activity data, avoid semantic fragmentation or redundancy caused by fixed block segmentation, thereby enhancing the model's adaptability to activity patterns at different scales and significantly improving the recognition performance in complex activity scenarios.

[0019] (2) This invention introduces a dynamic convolutional layer guided by block information, enabling subsequent feature extractors to perform targeted feature extraction based on the block information (relative block length, block center offset) of each block. The generation process of the dynamic convolutional kernel integrates block information representation with the features of the current layer, allowing the convolutional operation to focus on subtle temporal patterns within a block while also taking into account the temporal correlation between blocks. This design effectively improves the model's ability to capture detailed features such as local subtle motion patterns and brief pose changes, and compared with traditional static convolutional layers, it has stronger expressive power and task adaptability. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 This is an overall flowchart of the wearable activity recognition method based on multimodal learnable blocks provided in the embodiments of the present invention; Figure 2 This is a framework diagram of a wearable activity recognition model based on multimodal learnable blocks provided in an embodiment of the present invention; Figure 3This is a framework diagram of the block network provided in an embodiment of the present invention; Figure 4 This is a diagram of a dynamic convolutional layer framework guided by block information provided in an embodiment of the present invention. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and do not limit the scope of protection of this invention.

[0023] The inventive concept of this invention is as follows: Addressing the problems of existing technologies where fixed block segmentation methods easily lead to semantic fragmentation of activity data, difficulty in capturing complete local semantics, and the inability of traditional static convolutional layers to adaptively extract features based on block information and insufficient ability to capture subtle local patterns, this invention provides a wearable activity recognition method based on multimodal learnable block segmentation: First, wearable sensor data is preprocessed and a training set is constructed; second, the activity data is input into two independent block networks, which output block information under modality-shared and modality-specific block segmentation methods, respectively; based on the corresponding block information, activity data is resampled from the activity data to obtain modality-shared and modality-specific block activity data; the modality-specific blocks are then... Activity data is fed into a standard convolutional network to obtain modality-specific block activity data representations. Modality-shared block information is encoded to obtain modality-shared block information representations. These block information representations and modality-shared block activity data are then input into multiple dynamic convolutional layers guided by the block information for feature extraction to obtain modality-shared block activity data representations. The two block activity data representations are then fused to obtain block representations. Finally, the aggregated block representations are used as the final representations and fed into a classifier to output activity recognition results. This achieves adaptive block segmentation and accurate recognition of multimodal wearable activity data, effectively extracting relatively complete local semantics and subtle local patterns from the activity data, and improving the recognition accuracy and robustness in complex activity scenarios.

[0024] like Figure 1 As shown, the embodiment provides a wearable activity recognition method based on multimodal learnable blocks, specifically including the following steps: Step 1: Collect multimodal time-series data of users performing different activities using wearable devices, including... Data collected by various sensors.

[0025] Step 2: Outlier removal and normalization are performed on the collected multimodal time-series data. A sliding window is then used to divide the processed data into a training set. The training set consists of the following samples: , Indicates sensor Reading matrix ( Indicates the total number of sensors. Indicates the size of the sliding window. Indicates sensor The number of channels, (representing the space of all real numbers) , This represents a collection of activity category labels.

[0026] In this step, the processing of the collected data mainly includes: 1) Perform outlier processing on the collected sensor data, replacing values ​​that are outside the normal range with the maximum or minimum value within the normal range, and replacing missing values ​​with 0.

[0027] 2) Normalize the processed data using the following formula: , in, Represents the original value. This represents the minimum value of the channel containing the original value. This indicates the maximum value of the channel containing the original value. This represents the value after normalization.

[0028] 3) Set the time window size manually based on experience. The sliding window overlaps by 50%, and the normalized data is divided into multiple active data samples using the sliding window.

[0029] 4) Sample data for each activity Real tags for the corresponding activity categories Pairing, used as samples in the training set. Thus, a training set is constructed.

[0030] Step 3: Divide the training set into manually set batch sizes. The process will be carried out in batches, with a total number of batches being [number missing]. .

[0031] The batch processing of the training set can be determined based on the total number of training set samples and the batch size, as shown in the following formula: , in, This represents the total number of samples in the training set.

[0032] Step 4: Manually set the hyperparameters required for the model, including the number of blocks. Number of sampling points within the block Feature Dimension Hidden layer dimensions of block networks Dimensions of segmented information representation The number of candidate convolutional kernels in a dynamic convolutional layer guided by block information Candidate convolution kernel size .

[0033] Step 5: Select a batch of samples from the training set, and process each sample in that batch... Input as follows Figure 2 The wearable activity recognition model based on multimodal learnable blocks shown is repeated from steps 6 to 12.

[0034] Step 6, transfer the activity data The data is fed into the modal sharing segmentation network to obtain the segmentation information under this segmentation method (i.e., all modes share the same group of segmentation parameters). , These represent the relative block length and block center offset of each block under this partitioning method. This partitioning information is used to analyze the active data. Resampling was performed to obtain modal shared block activity data. .

[0035] In this step, the modality-sharing segmented network framework is as follows: Figure 3 As shown, the diagram contains three convolutional blocks (Conv Blocks) connected in sequence and a fully connected layer (Linear). Each convolutional block consists of a one-dimensional convolutional layer (Conv1D), a batch normalization layer (BatchNorm), and a non-linear activation function (ReLU). The fully connected layer ultimately outputs the relative block length and the block center offset. For simplicity, subscripts are omitted to avoid ambiguity. .

[0036] Based on the block information output by the modality-sharing block network, calculate the first... Intra-block sampling points The formula is as follows: , in, This represents a preset set of sampling points within a block. Indicates the preset first Block center, Indicates the first The relative block length of the block, Indicates the first The block center is offset. ,Right now Indicates the actual center of each block. , None of them are learnable, which is used to avoid extreme relative block lengths / block center offsets in the output of the block network, which would cause the sampling points to degenerate into single points and affect the training stability.

[0037] Subsequently, according to the Intra-block sampling points Activity data is located in modality Part Modalities are obtained by resampling Next Block output The formula is as follows: , , in, , This represents a linear interpolation algorithm. This indicates that the activity data is in a modal. Time step All channels, Indicates the first The first block Sampling points within each block, .

[0038] modality The outputs of each block are concatenated, and then the concatenation results of each mode are collected to obtain the modal shared block activity data. The formula is as follows: , , in, Representing modes Modal shared block activity data.

[0039] Step 7, transfer the activity data The data is fed into a mode-specific block network to obtain the block information under this block method (i.e., the independent block parameters for each mode). , These represent the relative block length and block center offset of each block under this partitioning method. This partitioning information is used to analyze the modal shared block activity data. Resampling is performed to obtain modality-specific block activity data. .

[0040] In this step, the modality-specific block network framework is also as follows: Figure 3 As shown, the only difference between the modality-specific block network and the modality-shared block network is the output data dimension. The modality-shared block network receives the overall activity data after concatenating all modalities and outputs data with a dimension of [missing information]. The modality-specific segmented network also receives the overall activity data after concatenating all modalities and outputs data with a dimension of [missing information]. The block information. To simplify the description of tensor broadcasting operations, if tensor superscripts appear in subsequent steps... This indicates that when performing a slicing operation on this tensor, the tensor should be preserved. All elements in the given dimension do not need to be selected within that dimension.

[0041] Block information based on the output of the modality-specific block network Calculate the mode Next Intra-block sampling points The formula is as follows: , in, This represents a preset set of sampling points within a block. Indicates the preset first Block center, Representing modes Next The relative block length of the block, Representing modes Next The block center is offset. In superscript Indicates by formula Received The sampling points are arranged in The second dimension .

[0042] Based on mode Next Intra-block sampling points and modality Modal shared block activity data Modes are obtained through resampling Next Block output The formula is as follows: , in, , Linear interpolation algorithm, definition and formula Consistent Represents the modal shared block activity data modality. Next Block 1 All channels of each point, as indicated by the superscript Indicates to Preserve the third dimension when performing slicing operations. .

[0043] modality The outputs of each block are spliced ​​together, among which The splicing results of each mode are collected to obtain mode-specific block activity data. The formula is as follows: , , in, Representing modes Modal-specific block activity data.

[0044] Step 8, transfer the modality-specific block activity data. The data is fed into a standard convolutional network to obtain a representation of modality-specific block activity data. .

[0045] In this step, using Each independent convolutional neural network processes modality-specific block activity data. Different modal data are used to obtain modality-specific block activity data representations (i.e., the first data representation). The convolutional neural network structures of each modality are the same, all including three convolutional layers, and the convolution operation is performed along the time dimension.

[0046] Step 9: Modal sharing block information The relative block length and block center offset are fed into separate coding networks. , Encode each block individually to obtain the relative block length encoding features of each block. and block center offset coding features Subsequently, the two are concatenated to obtain the modality-sharing block information representation. .

[0047] In this step, the coding network , They have the same structure but independent parameters. For simplicity, this step introduces a block attribute. A word and its encoding network This describes the encoding process relative to the block length / block center offset.

[0048] For block attributes Utilizing coding networks The block-by-block encoding is performed using two methods: multilayer perceptron and learnable Fourier feature location encoding. The formula is as follows: , in, Indicates block attributes The coding features, This represents a multilayer perceptron. The learnable Fourier feature location encoding is represented by the following formula: , in, For learnable parameters, the two superscripts are... Indicates to When performing slicing operations, retain its first and third dimensions. , .

[0049] by As Input, As The input is used to obtain the relative block length encoding features of each block. Block center offset coding features Subsequently, the two data points are concatenated and fed into a multilayer perceptron for fusion to obtain a modality-shared block information representation. .

[0050] Step 10, for each mode Modal shared block activity data under this mode Modal sharing of block information representation under this mode (The two in superscript) Indicates to (Preserve its first and third dimensions when performing slicing operations) and feed it into the first... The process is performed by a local fine feature extractor, among which... Finally, the features extracted from different modalities are spliced ​​together to obtain the modality-shared block activity data representation (i.e., the second data representation). .

[0051] In this step, the structures of each local fine feature extractor are identical, all including... The layers are dynamic convolutional layers guided by block information (dynamic convolutional layers), and the parameters between layers are not shared.

[0052] The following is from the first The local subtle feature extractor Dynamic convolutional layer (modal) No. Taking a dynamic convolutional layer as an example, the feature extraction process is explained in detail, where... , Similarly, the feature extraction process for each layer of the other modalities can be obtained. For the sake of simplicity, the modalities are omitted to avoid ambiguity. No. The superscript of the layer.

[0053] For mode No. Dynamic convolutional layers, such as Figure 4 As shown, firstly, regarding the modes Global features are extracted from modal shared block activity data and modal shared block information representation. and The calculation formula is as follows: , , in, This indicates an inter-block pooling operation. , This indicates a fully connected layer. Indicates input to modality No. Modal sharing of active data in dynamic convolutional layers, and .

[0054] Subsequently, the two global features are input into the weight network of this layer, and the output is a weighted weight of the candidate convolutional kernel. The calculation formula is as follows: , in, This represents the Softmax activation function (normalized exponential function). This represents a multilayer perceptron. This indicates a tensor splicing operation. satisfy , This refers to the number of candidate convolution kernels set in step 4.

[0055] Based on the weighted convolution kernels, the candidate convolution kernels for this layer are... The aggregation is performed using the following formula: , in, This represents the aggregated convolutional kernel. The aggregated convolutional kernel of this layer is used to process the modality-shared block activity data / features input to this layer (if...). If the input to this layer is modal shared block activity data, then the input is modal shared block activity data features. Feature extraction is performed using the following formula: , in, This represents the features output by this layer. This indicates a batch normalization operation. This represents a convolution operation performed along the time dimension within the block.

[0056] Finally, The local subtle feature extractor Features output by the dynamic convolutional layer To splice, among which The modal shared block activity data representation is obtained, and the formula is as follows: , Step 11: Fuse the two block activity data representations to obtain a block representation. .

[0057] In this step, the two block activity data representations are concatenated and then processed using a layer with... × Convolutional layers of the same kernel size are fused to obtain a fused block activity data representation, as shown in the following formula: , in, , Represents two-dimensional convolution. This indicates a tensor splicing operation.

[0058] Subsequently, it is fed into a three-layer multimodal convolutional module and a fully connected layer to fuse the information within each block, thus obtaining the block representation. The formula is as follows: , , in, Indicates the first The layer convolution module consists of two-dimensional convolution, batch normalization, and non-linear activation functions. Sequential composition, Indicates the first The output of the multimodal convolutional module has , This indicates a fully connected layer.

[0059] Step 12, Characterization of the aggregate block For the final representation The results are fed into the classifier and output as activity recognition results. .

[0060] In this step, the blocks are represented. The initial hidden state is fed into the GRU, and the output of its last hidden layer is pooled as the final representation. The final representation is then fed into a linear classifier to obtain the activity recognition result. .

[0061] Step 13: Calculate the activity recognition result for all samples in this batch. and real-life activity categories training loss And update the network parameters of the wearable activity recognition model to minimize training loss.

[0062] In this step, the training loss is the cross-entropy loss function, as shown in the following formula: , in, This represents the distribution that the training data follows. This represents the cross-entropy function.

[0063] Step 14, based on training loss The network parameters in the entire model are adjusted.

[0064] Based on the training loss obtained in step 13 For the network parameters in the entire model The updated formula is as follows: , in, This indicates a learning rate that is set manually.

[0065] Step 15: Repeat steps 5-14 until all batches of the training set have been used in model training.

[0066] Step 16: If the specified number of training iterations has been reached, the training ends; otherwise, return to step 5.

[0067] Step 17: Input the activity data to be identified into the trained model to obtain the identification result.

[0068] The specific embodiments described above illustrate the technical solution and beneficial effects of the present invention in detail. It should be understood that the above description is only the most preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, additions, and equivalent substitutions made within the scope of the principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A wearable activity recognition method based on multimodal learnable blocks, characterized in that, Includes the following steps: S1, The collected multimodal activity time series data are preprocessed and then used to construct a training set; S2, input each batch of training samples from the training set into the wearable activity recognition model and execute: The activity data from each batch of training samples are input into the modality sharing block network and the modality-specific block network respectively to obtain modality sharing block information and modality-specific block information respectively. The activity data is resampled using the modality sharing block information to obtain modality sharing block activity data, and the modality-specific block activity data is resampled using the modality-specific block information to obtain modality-specific block activity data. The first data representation is obtained by extracting features from the modality-specific block activity data. The second data representation is obtained by encoding the modality-shared block information and then guiding the feature extraction of the modality-shared block activity data. The first and second data representations are then fused to obtain the block representation. The aggregated block representation is the final representation, which is fed into the classifier to output the activity recognition result; S3, calculate the loss between the activity recognition results and the true labels in each batch of training samples to update the model parameters, repeat the training until convergence, and use the trained model to identify new activity data to be tested.

2. The wearable activity recognition method based on multimodal learnable blocks according to claim 1, characterized in that, The preprocessing of the collected multimodal activity time-series data to construct a training set includes: Outlier processing and normalization are performed on multimodal activity time-series data collected by various sensors. The processed data is divided into multiple activity data samples by using a sliding window. The activity data samples in each window are paired with their corresponding real labels of activity categories to obtain training samples. All training samples are used to construct a training set.

3. The wearable activity recognition method based on multimodal learnable blocks according to claim 1, characterized in that, The modality-sharing block network performs learnable feature mapping on the activity data, and the output modality-sharing block information includes the relative block length of each block and the block center offset of each block, both in the dimension of the number of blocks. The modality-specific block network also performs learnable feature mapping on the activity data, and the output modality-specific block information includes the relative block length of each block in each modality and the block center offset of each block in each modality, both in the dimension of the number of blocks. ×number of modes The starting and ending positions of each block are determined based on the relative block length and the block center offset.

4. The wearable activity recognition method based on multimodal learnable blocks according to claim 1, characterized in that, The modality-sharing block network and the modality-specific block network have the same structure but independent parameters. Both contain multiple convolutional blocks connected in sequence and a fully connected layer. Each convolutional block includes a one-dimensional convolutional layer, a batch normalization layer and a nonlinear activation function in sequence. Finally, the fully connected layer outputs the relative block length and the block center offset.

5. The wearable activity recognition method based on multimodal learnable blocks according to claim 1, characterized in that, The method of resampling active data using modal sharing block information to obtain modal sharing block active data includes: Based on the modal sharing block information, calculate the intra-block sampling points of each block shared by all modes, where the first... The sampling points within a block are the first sampling points in the modal sharing block information. The relative block length and block center offset of the block are obtained by correcting the preset sampling points within the block and the preset block center; For each mode Using a linear interpolation algorithm, based on the first... Intra-block sampling points of the block for modal The activity data was resampled to obtain the modal data. Next Block output of a block; The block outputs of all modal blocks are concatenated to obtain the modal shared block activity data.

6. The wearable activity recognition method based on multimodal learnable blocks according to claim 1, characterized in that, The method of resampling modal shared block activity data using modal-specific block information to obtain modal-specific block activity data includes: Based on the modality-specific block information, the sampling points within each block are calculated independently for each mode, where the first... The modality of the first The sampling points within a block are the first sampling points based on the modality-specific block information. The modality of the first The relative block length and block center offset of the block are obtained by correcting the preset sampling points within the block and the preset block center; For each mode Using a linear interpolation algorithm, based on the first... The modality of the first Intra-block sampling points of the block for modal The modal shared block activity data is resampled to obtain the modal data. Next Block output of a block; The block outputs of all modalities are concatenated to obtain modality-specific block activity data.

7. The wearable activity recognition method based on multimodal learnable blocks according to claim 1, characterized in that, The first data representation obtained by extracting features from modality-specific block activity data includes: Multiple independent convolutional neural networks, the same number as the number of modes, are used to process the modality-specific block activity data of each mode. Each convolutional neural network contains multiple convolutional layers and the convolution operation is performed along the time dimension. The outputs of each convolutional neural network are concatenated to obtain the first data representation.

8. The wearable activity recognition method based on multimodal learnable blocks according to claim 1, characterized in that, The process of encoding modal sharing block information and then guiding feature extraction of modal sharing block activity data to obtain a second data representation includes: The relative block length and block center offset in the modality sharing block information are input into two independent coding networks, and each block is encoded to obtain the relative block length coding feature and the block center offset coding feature. The two are then concatenated to obtain the modality sharing block information representation. For each mode modality Modal shared block activity data and modality Modal shared block information representation is input to the modality. The corresponding local subtle feature extractor contains multiple dynamic convolutional layers. Each dynamic convolutional layer generates a dynamic convolutional kernel based on the current input, performs a convolutional operation, and outputs the features of that layer. The features output by the local subtle feature extractors of all modalities are concatenated to obtain the second data representation.

9. The wearable activity recognition method based on multimodal learnable blocks according to claim 1, characterized in that, The fusion of the first data representation and the second data representation to obtain the block representation includes: After concatenating the first data representation with the second data representation, the data is fused through a two-dimensional convolutional layer. The fusion result is then passed sequentially through multiple multimodal convolutional modules and a fully connected layer. Each multimodal convolutional module includes a two-dimensional convolutional layer, a batch normalization layer, and a nonlinear activation function. Finally, the fully connected layer outputs a block representation.

Images