Motion recognition methods, devices, and smart dumbbells
By integrating inertial sensors into smart dumbbells and utilizing convolutional autoencoders and fully connected autoencoders for feature extraction and cross-modal feature learning, the accuracy problem of smart dumbbell motion recognition was solved, achieving accurate recognition of multiple motion types.
Patent Information
- Application Number
- CN202611264766.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-08-19
- Publication Date
- 2026-11-17
AI Technical Summary
Users often struggle to obtain accurate and reliable motion recognition and real-time guidance when training with smart dumbbells.
Multiple modal data are collected by inertial sensors, and feature extraction and cross-modal feature learning are performed using convolutional autoencoders and fully connected autoencoders. A classifier is then used to identify action types.
It improves the accuracy and precision of motion recognition, supports the recognition of various motion types of smart dumbbells, and meets users' motion recognition guidance needs.
Smart Images

Figure CN123116004A_ABST
Abstract
Description
Technical Field
[0001] This disclosure relates to the field of data analysis technology, and in particular to a motion recognition method, device, and smart dumbbell. Background Technology
[0002] Currently, fitness equipment is gradually iterating towards intelligence. Taking smart dumbbells as an example, as core equipment for home and small fitness scenarios, their convenience and practicality have led to a gradual increase in their penetration rate.
[0003] In practical applications, users still find it difficult to obtain accurate and reliable motion recognition and real-time guidance when training with smart fitness equipment such as smart dumbbells. Summary of the Invention
[0004] To address the aforementioned technical problems, this disclosure provides the following technical solutions: Firstly, an action recognition method is provided, the method comprising: Acquire multiple modal data collected by inertial sensors; Feature extraction is performed on each modal data in the multiple modal data to obtain single-modal features of each modal data; the single-modal features are the specific representations of each modal data within that modality; Cross-modal features are extracted based on the single-modal features of the various modal data to obtain cross-modal features; the cross-modal features are specific representations between at least two modal data. The action type of the inertial sensor is identified based on the cross-modal characteristics.
[0005] In this embodiment, modal features are extracted independently for each modal data, i.e., single-modal features are extracted, which can uncover specific representations within a modality; then, cross-modal features are extracted based on the single-modal features, which can fully uncover the inter-modal correlation features, i.e., cross-modal features. The action type is identified by combining single-modal features and cross-modal features, which can improve the accuracy of action recognition.
[0006] In one embodiment, feature extraction is performed on various modal data from the multiple modal data, including: The data of various modalities are segmented according to a preset time window to obtain multiple time window data segments; Each time window data segment under each modality is input into a convolutional autoencoder for feature extraction to obtain single-modal features under each modality; wherein, the convolutional autoencoder includes a bottleneck structure, which is a symmetric network structure in which the encoder channel continuously compresses the data dimension and the decoder channel gradually restores the data dimension.
[0007] In this embodiment, by setting a convolutional autoencoder with a bottleneck structure, the convolutional autoencoder can learn the modality-specific representation of modal data under a single modality, learn a compact modality-specific representation, and reduce the feature redundancy of single modality features.
[0008] In one embodiment, the method further includes a step of obtaining the length of the preset time window, specifically including: Peak detection is performed on the modal data to determine the local maximum values in the modal data, thereby obtaining a candidate peak list; The candidate peaks in the candidate peak list are sorted according to the amplitude, and the peak period of each candidate peak is determined by autocorrelation analysis. The length of the preset time window is determined based on the peak period.
[0009] In this embodiment, the action cycle is automatically identified through peak detection and autocorrelation analysis, and the time window is adaptively adjusted, which helps to improve the accuracy of the obtained single-modal features.
[0010] In one embodiment, the convolutional autoencoder includes an encoder and a decoder, the encoder including at least one convolutional layer and at least one pooling layer, and the decoder including at least one deconvolutional layer and at least one depooling layer; the encoder and the decoder are arranged in a symmetrical structure.
[0011] In this embodiment, by setting the convolutional autoencoder to a symmetrical structure, with the same number of convolutional layers in the encoder and the same number of deconvolutional layers in the decoder, as well as the same number of pooling layers in the encoder and the same number of depooling layers in the decoder, the correspondence between feature extraction and reconstruction can be guaranteed, thereby improving the expressive power of modal features.
[0012] In one embodiment, cross-modal feature extraction is performed based on the single-modal features of the various modal data to obtain cross-modal features, including: The single-modal features of each modality are flattened and then stitched together to obtain the stitched features of all single-modal features under multiple modalities; The spliced features are input into a fully connected autoencoder for cross-modal feature extraction to obtain the cross-modal features; the fully connected autoencoder is used to learn cross-modal representations that are discriminative for action type recognition tasks.
[0013] In this embodiment, all single-modal features are concatenated by a fully connected autoencoder, and cross-modal features are extracted from the concatenated features to learn discriminative cross-modal representations. This fully utilizes the complementary information between different modal data, which helps to improve the accuracy of action recognition.
[0014] In one embodiment, the fully connected autoencoder includes an encoding part and a decoding part, the encoding part including at least one fully connected layer, and the decoding part including at least one fully connected layer; the decoding part of the fully connected autoencoder is used to reconstruct a vector of the same size as the spliced feature.
[0015] In this embodiment, high-level semantic features are extracted by dimensional compression of the spliced features through the encoding part, and then a vector of the same size as the spliced features is reconstructed by the decoding part. This learns a compact cross-modal representation containing key multimodal information, thereby enhancing the generalization ability and information fidelity of the cross-modal features.
[0016] In one embodiment, identifying the action type of the inertial sensor based on the cross-modal characteristics includes: The cross-modal features are input into a classifier to obtain the action type of the inertial sensor; the classifier includes at least one fully connected layer.
[0017] In this embodiment, a classifier containing at least one fully connected layer maps the cross-modal feature space to the action type space, and the final action type is determined by taking the maximum value of the output probability distribution.
[0018] In one embodiment, the classifier is connected to the embedding layer of a fully connected autoencoder, which consists of three fully connected layers; and the first two fully connected layers each contain 128 hidden units and use the ReLU activation function as the activation function.
[0019] In this embodiment, a classifier with three fully connected layers is used to map the cross-modal feature space to the action type space, and the final action type is determined by taking the maximum value of the output probability distribution.
[0020] In one embodiment, the inertial sensor is disposed inside the dumbbell body.
[0021] In this embodiment, the sensor is integrated inside the dumbbell body to directly collect dumbbell motion data, improving the accuracy of data collection and eliminating the need for additional wearable devices.
[0022] In one embodiment, the type of movement includes at least one of the following: dumbbell bench press, dumbbell fly, dumbbell push-up, dumbbell deadlift, dumbbell curl, dumbbell squat, dumbbell lunge, dumbbell Russian twist, dumbbell crunch, and dumbbell power clean.
[0023] In this embodiment, it can support the recognition of ten types of movements of smart dumbbells, covering most resistance training programs and meeting the user's need for guidance on movement recognition.
[0024] Secondly, a motion recognition device is provided, the device comprising: The multimodal data acquisition module is used to acquire multiple modal data collected by the inertial sensor; The single-modal feature acquisition module is used to extract features from various modal data in the multiple modal data to obtain single-modal features of various modal data; the single-modal features are specific representations of various modal data within that modality; A cross-modal feature acquisition module is used to extract cross-modal features based on the single-modal features of the various modal data to obtain cross-modal features; the cross-modal features are specific representations between at least two modal data. An action type recognition module is used to identify the action type of the inertial sensor based on the cross-modal characteristics.
[0025] In this embodiment, modal features are extracted independently for each modal data, i.e., single-modal features are extracted, which can uncover specific representations within a modality; then, cross-modal features are extracted based on the single-modal features, which can fully uncover the inter-modal correlation features, i.e., cross-modal features. The action type is identified by combining single-modal features and cross-modal features, which can improve the accuracy of action recognition.
[0026] In one embodiment, the single-modal feature acquisition module includes: The time window data segment acquisition submodule is used to segment various modal data according to a preset time window to obtain multiple time window data segments; The single-modal feature acquisition submodule is used to input each time window data segment under each modality into the convolutional autoencoder for feature extraction to obtain the single-modal features under each modality; wherein, the convolutional autoencoder includes a bottleneck structure, which is a symmetric network structure in which the encoder channel continuously compresses the data dimension and the decoder channel gradually restores the data dimension.
[0027] In this embodiment, by setting a convolutional autoencoder with a bottleneck structure, the convolutional autoencoder can learn the modality-specific representation of modal data under a single modality, learn a compact modality-specific representation, and reduce the feature redundancy of single modality features.
[0028] In one embodiment, the single-modal feature acquisition module further includes a time window length acquisition module, which is used to acquire the length of the preset time window; the time window length acquisition module includes: The candidate peak list acquisition submodule is used to perform peak detection on the modal data to determine the local maximum value in the modal data and obtain a candidate peak list; The candidate peak list sorting submodule is used to sort the candidate peaks in the candidate peak list according to the amplitude; The peak period determination submodule is used to determine the peak period of each candidate peak based on autocorrelation analysis. The time window length determination submodule is used to determine the length of the preset time window based on the peak period.
[0029] In this embodiment, the action cycle is automatically identified through peak detection and autocorrelation analysis, and the time window is adaptively adjusted, which helps to improve the accuracy of the obtained single-modal features.
[0030] In one embodiment, the convolutional autoencoder includes an encoder and a decoder, the encoder including at least one convolutional layer and at least one pooling layer, and the decoder including at least one deconvolutional layer and at least one depooling layer; the encoder and the decoder are arranged in a symmetrical structure.
[0031] In this embodiment, by setting the convolutional autoencoder to a symmetrical structure, with the same number of convolutional layers in the encoder and the same number of deconvolutional layers in the decoder, as well as the same number of pooling layers in the encoder and the same number of depooling layers in the decoder, the correspondence between feature extraction and reconstruction can be guaranteed, thereby improving the expressive power of modal features.
[0032] In one embodiment, the cross-modal feature acquisition module includes: The splicing feature acquisition submodule is used to flatten and splice the single-modal features of each modality to obtain the spliced features of all single-modal features under multiple modalities; The cross-modal feature acquisition submodule is used to input the spliced features into a fully connected autoencoder for cross-modal feature extraction to obtain the cross-modal features; the fully connected autoencoder is used to learn cross-modal representations that are discriminative for action type recognition tasks.
[0033] In this embodiment, all single-modal features are concatenated by a fully connected autoencoder, and cross-modal features are extracted from the concatenated features to learn discriminative cross-modal representations. This fully utilizes the complementary information between different modal data, which helps to improve the accuracy of action recognition.
[0034] In one embodiment, the fully connected autoencoder includes an encoding part and a decoding part, the encoding part including at least one fully connected layer, and the decoding part including at least one fully connected layer; the decoding part of the fully connected autoencoder is used to reconstruct a vector of the same size as the spliced feature.
[0035] In this embodiment, high-level semantic features are extracted by dimensional compression of the spliced features through the encoding part, and then a vector of the same size as the spliced features is reconstructed by the decoding part. This learns a compact cross-modal representation containing key multimodal information, thereby enhancing the generalization ability and information fidelity of the cross-modal features.
[0036] In one embodiment, the action type recognition module includes: An action type recognition submodule is used to input the cross-modal features into a classifier to obtain the action type of the inertial sensor; the classifier includes at least one fully connected layer.
[0037] In this embodiment, a classifier containing at least one fully connected layer maps the cross-modal feature space to the action type space, and the final action type is determined by taking the maximum value of the output probability distribution.
[0038] In one embodiment, the classifier is connected to the embedding layer of a fully connected autoencoder, which consists of three fully connected layers; and the first two fully connected layers each contain 128 hidden units and use the ReLU activation function as the activation function.
[0039] In this embodiment, a classifier with three fully connected layers is used to map the cross-modal feature space to the action type space, and the final action type is determined by taking the maximum value of the output probability distribution.
[0040] In one embodiment, the inertial sensor is disposed inside the dumbbell body.
[0041] In this embodiment, the sensor is integrated inside the dumbbell body to directly collect dumbbell motion data, improving the accuracy of data collection and eliminating the need for additional wearable devices.
[0042] In one embodiment, the type of movement includes at least one of the following: dumbbell bench press, dumbbell fly, dumbbell push-up, dumbbell deadlift, dumbbell curl, dumbbell squat, dumbbell lunge, dumbbell Russian twist, dumbbell crunch, and dumbbell power clean.
[0043] In this embodiment, it can support the recognition of ten types of movements of smart dumbbells, covering most resistance training programs and meeting the user's need for guidance on movement recognition.
[0044] Thirdly, a computing device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program and performs the method as described in any of the first aspects.
[0045] Fourthly, a smart dumbbell is provided, including an inertial sensor for collecting multiple modal data, including acceleration data and angular velocity data.
[0046] Fifthly, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, performs the method as described in any of the first aspects.
[0047] In a sixth aspect, a computer program product is provided, the computer program product comprising a computer program that, when executed by a processor, performs the method as described in any of the first aspects.
[0048] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description
[0049] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings.
[0050] Figure 1 This is a flowchart of an action recognition method provided in this disclosure.
[0051] Figure 2 This is a flowchart of a method for obtaining single-modal features provided in this disclosure.
[0052] Figure 3 This is a flowchart of a method for obtaining cross-modal features provided in this disclosure.
[0053] Figure 4 This is a block diagram of an action recognition device provided in this disclosure.
[0054] Figure 5 This is a schematic diagram of the structure of a computing device provided in this disclosure. Detailed Implementation
[0055] This disclosure describes multiple technical solutions with different concepts. Each concept has one or more embodiments, and different concepts can be combined to form more embodiments. Those skilled in the art, after reading this disclosure, can combine different concepts to obtain new technical solutions, and these new technical solutions should also fall within the scope of this disclosure.
[0056] The technical solutions of these different concepts will be introduced in turn below. Some concepts may appear in multiple technical solutions of different concepts. For these concepts, this disclosure will explain them when they first appear and will not repeat them in the following text.
[0057] Considering that in related technologies, users still find it difficult to obtain accurate and reliable motion recognition and real-time guidance when training with smart fitness equipment such as smart dumbbells, this disclosure provides a motion recognition method, device, and smart dumbbell to address the aforementioned technical problems.
[0058] The aforementioned motion recognition method can be applied to motion recognition on fitness equipment such as smart dumbbells. Taking a smart dumbbell as an example, it can include an inertial sensor. This inertial sensor can be located inside the dumbbell, or a portion of the inertial sensor can be located inside the dumbbell while the other portion is fixed to the user's wrist, arm, or body. Understandably, when the other portion of the inertial sensor is fixed to the user's wrist and / or arm, this portion can be replaced by a smart bracelet or smartwatch. When the other portion of the inertial sensor is fixed to the user's body, this portion can be replaced by smart glasses, a smart helmet, or similar devices.
[0059] Understandably, based on the action recognition method provided in this disclosure, the action type corresponding to the inertial sensor can be identified by collecting multiple modal data from the inertial sensor; in other words, the user's action type can be determined. For ease of description, the above action recognition method is executed by a computing device, which can be located inside the inertial sensor; it can also be implemented using a smartphone or the cloud, without limitation.
[0060] Understandably, the aforementioned smart dumbbell and computing device can constitute a motion recognition system. In this system, the inertial sensors within the smart dumbbell collect multiple modal data. The computing device can acquire this multi-modal data and identify the motion type of the inertial sensors, i.e., the user's motion type.
[0061] In one embodiment, see Figure 1 An action recognition method, applicable to computing devices, includes steps 11-14.
[0062] In step 11, the computing device can acquire multiple modal data collected by the inertial sensor.
[0063] In this step, there can be multiple inertial sensors, each positioned at different locations, such as different ends of a smart dumbbell and / or on the user's body. Each inertial sensor can have its own coordinate system, such as oxyz. Sensor data collected by different types of inertial sensors at different locations constitutes one modal data, while sensor data collected by multiple inertial sensors at different locations and of different types constitutes multiple modal data. Alternatively, in this step, the modality is determined by the combination of sensor type and acquisition location. Sensor data collected by inertial sensors at the same location and of the same type is considered the same modal data, while sensor data collected by inertial sensors at different locations and / or of different types is considered different modal data. The computing device can communicate with each inertial sensor to obtain the multiple modal data collected by the inertial sensors.
[0064] In step 12, the computing device can extract features from each modal data in the multiple modal data to obtain single modal features of each modal data; the single modal features are the specific representations of each modal data within that modality.
[0065] In step 13, the computing device can extract cross-modal features based on the single-modal features of the various modal data to obtain cross-modal features; the cross-modal features are specific representations between at least two modal data.
[0066] In step 14, the computing device can identify the action type of the inertial sensor based on the cross-modal characteristics.
[0067] In this embodiment, modal features are first extracted independently for each modality of data, i.e., single-modal features are extracted, which can uncover specific representations within a modality. Then, cross-modal features are extracted based on the single-modal features, which can fully uncover the correlation features between different modalities, i.e., cross-modal features. Finally, the action type is identified based on the cross-modal features. In other words, this embodiment uses a combination of single-modal features and cross-modal features to identify action types, which can improve the accuracy of action type identification.
[0068] exist Figure 1 Based on the example action recognition method, the acquisition of multiple modal data collected by the inertial sensor in step 11 can be achieved in the following ways.
[0069] Inertial sensors are worn at different locations on the subject. These sensors can be placed inside or outside a smart dumbbell, in a smartwatch, smart bracelet, smart helmet, or other devices. Five different inertial sensors are worn on each subject.
[0070] In this embodiment, it can be applied to single-person exercise type recognition scenarios designed for home fitness, so each sensor data collection involves only one subject. For each subject, the task consists of three sets of ten movements, with ten repetitions per set; the selected movements are all classic resistance training exercises: dumbbell bench press, dumbbell fly, dumbbell push-up, dumbbell deadlift, dumbbell curl, dumbbell squat, dumbbell lunge, dumbbell Russian twist, dumbbell crunch, and dumbbell power clean.
[0071] To ensure participants are familiar with the movements, a demonstration of the exercises is conducted before data collection. Afterward, participants can freely choose dumbbells of appropriate weight for self-weight training, with a maximum weight of 7.5 kg. Participants mimic the movements, and no posture corrections are provided during the training. Rest periods between sets are determined by the participants themselves, and the sensors continuously collect data.
[0072] For ease of description, the following embodiments are implemented by having the subject wear a smartwatch to collect acceleration and angular velocity data. The sampling frequency for both acceleration and angular velocity is 100Hz, and the data format is shown in Table 1.
[0073] Table 1. Data format of sensor data In Table 1, N represents the length of the data, and X, Y, and Z represent the data at different speeds in each dimension.
[0074] Based on the above process, multiple modal data collected by the inertial sensor corresponding to the subject can be obtained. Considering that the subject has learned standard movements and has not been corrected during training, the above multiple modal data can accurately represent the movement of the inertial sensor, that is, the subject's movement. This is beneficial for accurately obtaining the subject's movement type in the future, so that the subject or other personnel can make corrections or records based on the above movement type, thereby improving the subject's user experience.
[0075] In one embodiment, in Figure 1 Based on the example action recognition method, see [link to example]. Figure 2 Step 12 means that the computing device can extract features from various modal data in the multiple modal data to obtain single modal features of various modal data, including steps 21 and 22.
[0076] In step 21, the computing device can segment various modal data according to a preset time window to obtain multiple time window data segments.
[0077] In this step, the computing device can obtain the length of the preset time window for each modality data, including: (a) The computing device can perform peak detection on various modal data, determine the local maxima in the modal data, and obtain a candidate peak list. Considering that multiple modal data come from inertial sensors at different locations and / or of different types, the computing device can preprocess the multiple modal data. This preprocessing can include standardization and smoothing. Standardization refers to unifying various modal data into the same value range. For example, multimodal data with different dimensions and large differences in value range (such as tens and thousands of units) can be uniformly transformed to a standard distribution with a mean of 0 and a standard deviation of 1, eliminating differences in units and scales, and making different modal features comparable. Smoothing can be implemented using a smoothing filtering algorithm such as a cubic polynomial Savitzky-Golay filter to eliminate high-frequency noise.
[0078] (b) The computing device can use the Principal Component Analysis (PCA) method to determine the direction with the largest data variance as the first principal component direction, or in other words, the direction with the largest motion deformation as the first principal component direction; and project the preprocessed modal data onto the aforementioned first principal component direction to obtain a one-dimensional signal.
[0079] (c) The computing device can obtain the peak value (i.e., the local maximum value) within the one-dimensional signal as candidate peak values by comparing the numerical values within the one-dimensional signal. After processing the entire one-dimensional signal, a list of candidate peak values can be obtained. The computing device can sort the candidate peak values in the candidate peak value list according to the amplitude and update the candidate peak value list according to the sorting.
[0080] (d) The computing device can determine the peak period of each candidate peak based on autocorrelation analysis. For example, a time window is set with each peak as the center, and the lag value between the expected duration of the minimum and maximum values within the time window is calculated; the largest autocorrelation value within the lag value range is selected as the peak period P of the candidate peak. Here, the lag value refers to the number of sampling points that are staggered between two signal segments, or the number of sampling points required for the signal to repeat once. For example, if the lag value k=0, the signal does not shift and is aligned with itself, and the autocorrelation is at its maximum value; if the lag value k=1, the signal is shifted 1 point to the right and then multiplied and summed with the original signal; if the lag value k=2, the signal is shifted 2 points to the right and then multiplied and summed with the original signal; and so on, until the autocorrelation values of all sampling points have been calculated.
[0081] (e) The computing device can determine the length of the preset time window based on the peak period and the sampling frequency of each sampling point.
[0082] Then, the computing device can segment the modal data according to a preset time window for various modal matching to obtain multiple time window data segments under that modality.
[0083] In step 22, the computing device can input each time window data segment under each modality into the convolutional autoencoder for feature extraction to obtain single-modality features under each modality; wherein, the convolutional autoencoder includes a bottleneck structure, which is a symmetric network structure in which the encoder channel continuously compresses the data dimension and the decoder channel gradually restores the data dimension.
[0084] In this step, a convolutional autoencoder can be pre-stored within the computing device to extract single-modal features from the input data. This convolutional autoencoder includes a bottleneck structure, which is a symmetric network structure where the encoder channel continuously compresses the data dimension and the decoder channel gradually restores the data dimension. By using a convolutional autoencoder with a bottleneck structure, it can learn modality-specific representations of modal data in a single modality, learn compact modality-specific representations, and reduce feature redundancy in single-modal features.
[0085] In one example, a convolutional autoencoder includes an encoder and a decoder. The encoder includes at least one convolutional layer and at least one pooling layer, and the decoder includes at least one deconvolutional layer and at least one unpooling layer. The encoder and decoder are arranged in a symmetrical structure. By setting the convolutional autoencoder to a symmetrical structure, with the same number of convolutional layers in the encoder and the same number of deconvolutional layers in the decoder, as well as the same number of pooling layers in the encoder and the same number of unpooling layers in the decoder, the correspondence between feature extraction and reconstruction can be guaranteed, improving the expressive power of modal features.
[0086] Based on the above concept, this convolutional autoencoder has three convolutional layers and a max-pooling layer, as well as three deconvolutional layers and an unpooling layer, as shown in Table 2. Specifically, it includes an input layer, a splitting layer, a first convolutional layer (conv1), a second convolutional layer (conv2), a third convolutional layer (conv3), a max-pooling layer (Maxpool), an unpooling layer (Unpool), a first deconvolutional layer (deconv1), a second convolutional layer (deconv2), and a third convolutional layer (deconv3). The parameters of each layer are shown in Table 2.
[0087] Table 2 Parameters of each layer of the convolutional autoencoder Referring to Table 1, the input layer receives time window data segments, or sensor data, with dimensions (C×H×W) of 3×1×1000. The split layer divides the input data into segments with a preset step size, or into data corresponding to each inertial sensor. A processing channel is set up for each inertial sensor. For example, if the left and right hands are wearing smartwatches respectively, four processing channels can be set up, including left-hand acceleration channel, left-hand angle channel, right-hand acceleration channel, and right-hand angle channel. Correspondingly, the output data of the split layer consists of four sets of 3×1×250 dimensional data.
[0088] For each processing channel, a first convolutional layer conv1, a second convolutional layer conv2, a third convolutional layer conv3, and a max pooling layer Maxpool are set respectively.
[0089] Taking the left-handed acceleration channel as an example, the parameters of the first convolutional layer (conv1), the second convolutional layer (conv2), the third convolutional layer (conv3), and the max-pooling layer (Maxpool) are 9×1×11 (kernel size 11×11, stride 2), 15×3×11 (kernel size 11×11, stride 2), and 24×15×11 (kernel size 11×11, stride 2), respectively. The left-handed sensor data is finally processed by the max-pooling layer and the reshape layer. The max-pooling layer and the reshape layer can be used as embedding layers in a convolutional autoencoder.
[0090] Taking the left-handed angle channel as an example, the parameters of the first convolutional layer conv1, the second convolutional layer conv2, the third convolutional layer conv3, and the max pooling layer are 9×3×3 (kernel size 3×3, stride 2), 15×9×3 (kernel size 3×3, stride 2), and 24×15×3 (kernel size 3×3, stride 2), respectively. The left-handed sensor data finally passes through the max pooling layer and the reshape layer. The max pooling layer and the reshape layer can be used as embedding layers in a convolutional autoencoder.
[0091] Taking the right-hand accelerometer channel as an example, the parameters of the first convolutional layer (conv1), the second convolutional layer (conv2), the third convolutional layer (conv3), and the max-pooling layer (Maxpool) are 9×1×11 (kernel size 11×11, stride 2), 15×3×11 (kernel size 11×11, stride 2), and 24×15×11 (kernel size 11×11, stride 2), respectively. The right-hand sensor data finally passes through the max-pooling layer and the reshape layer. The max-pooling layer and the reshape layer can be used as embedding layers in a convolutional autoencoder.
[0092] Taking the right-hand angle channel as an example, the parameters of the first convolutional layer conv1, the second convolutional layer conv2, the third convolutional layer conv3, and the max pooling layer are 9×3×3 (kernel size 3×3, stride 2), 15×9×3 (kernel size 3×3, stride 2), and 24×15×3 (kernel size 3×3, stride 2), respectively. The right-hand sensor data finally passes through the max pooling layer and the reshape layer. The max pooling layer and the reshape layer can be used as embedding layers in a convolutional autoencoder.
[0093] Understandably, the corresponding data processing channels and the number of channels can be set according to the location and type of the inertial sensor. If the single-mode characteristics of each mode can be acquired, the corresponding solution falls within the protection scope of this disclosure. Considering the possibility of inertial sensors being installed externally on the smart dumbbell, the smart dumbbell can communicate with a smartphone or the cloud. After successful communication, a configuration page can be displayed, allowing the user to specify the sensor data collected by the inertial sensor corresponding to each processing channel.
[0094] In this embodiment, by using a convolutional autoencoder with a bottleneck structure, the convolutional autoencoder can learn the modality-specific representation of modal data in a single modality, learn a compact modality-specific representation, and reduce the feature redundancy of single-modal features. Alternatively, it can extract a compact modality-specific representation from the sensor data collected by a single inertial sensor, reducing the feature redundancy of single-modal features.
[0095] In one embodiment, the convolutional autoencoder can be implemented using a separate training method, that is, the training samples of each modality in the training sample dataset are input into the convolutional autoencoder to obtain the single-modal features of each modality. Training ends if the single-modal features tend to converge or the gradient change is less than or equal to a preset gradient change threshold.
[0096] In another embodiment, the convolutional autoencoder can be implemented using a joint training method, that is, the convolutional autoencoder is jointly trained with at least one of the fully connected autoencoder and classifier in subsequent embodiments. For example, joint training of the convolutional autoencoder and the fully connected autoencoder, or joint training of the convolutional autoencoder, the fully connected autoencoder, and the classifier, etc., can be selected according to the specific scenario, and the corresponding scheme falls within the protection scope of this disclosure. Considering that the fully connected autoencoder and classifier appear in subsequent embodiments, they will not be described here.
[0097] Understandably, the training samples in the training dataset were obtained by professionals training according to standard movements. For example, inertial sensors could be placed at different locations on the professional, such as by having them wear smartwatches to collect acceleration and angular velocity data, with both sampling frequencies at 100Hz. The task consisted of three sets of ten exercises, each set with ten repetitions. The selected exercises were all classic resistance training programs: dumbbell bench press, dumbbell flyes, dumbbell push-ups, dumbbell deadlifts, dumbbell curls, dumbbell squats, dumbbell lunges, dumbbell Russian twists, dumbbell crunches, and dumbbell power cleans. During dumbbell training, the rest time between sets was determined by the subject, and the sensors continuously collected data, obtaining modal data for multiple modes. Then, professionals calibrated the modal data to identify the movement type, inertial sensor type, and location corresponding to each modality.
[0098] In one embodiment, in Figure 1 Based on the example action recognition method, see [link to example]. Figure 3 Step 13, namely, the computing device can extract cross-modal features based on the single-modal features of the various modal data to obtain cross-modal features, includes steps 31 and 32.
[0099] In step 31, the computing device can flatten and stitch together the single-modal features of each mode to obtain the stitched features of all single-modal features under multiple modes.
[0100] In this step, the computing device can flatten the single-modal features of each modality. The flattening operation compresses and merges the single-modal features into a single-dimensional feature vector from the spatial and / or channel dimensions. Then, the computing device can concatenate the single-dimensional feature vectors of all the single-modal features into a concatenated feature, forming a data format that can be input into the subsequent fully connected autoencoder.
[0101] By first flattening and unifying the dimensionality of each single-modal feature, the problem of incompatibility of tensor structures of multiple single-modal features is solved; then, all single-modal feature vectors are spliced together, i.e., lossless fusion of fine-grained features, to construct a one-dimensional joint feature containing all modal features, i.e., spliced feature.
[0102] In step 32, the computing device can input the spliced features into a fully connected autoencoder to extract cross-modal features and obtain the cross-modal features; the fully connected autoencoder is used to learn cross-modal representations that are discriminative for action type recognition tasks.
[0103] In this step, the computing device can store a fully connected autoencoder, which can perform cross-modal feature extraction on the input data. For example, concatenated features can be input into the fully connected autoencoder for cross-modal feature extraction. Since the concatenated features include information from multiple single-modal feature vectors, the feature extraction process can obtain the associated features of adjacent single-modal feature vectors, as well as the self-features of the single-modal feature vectors, ultimately yielding the cross-modal features.
[0104] In one embodiment, the fully connected autoencoder includes an encoding part and a decoding part, implemented in a stacked structure. The encoding part includes at least one fully connected layer, and the decoding part includes at least one fully connected layer for fusing features from all modalities. The decoding part of the fully connected autoencoder is used to reconstruct a vector of the same size as the concatenated features, meaning that the input and output data of the fully connected autoencoder have the same size. By setting the encoding part to compress the dimensions of the concatenated features to extract high-level semantic features, and then setting the decoding part to reconstruct a vector of the same size as the concatenated features, a compact cross-modal representation containing key multimodal information is learned, enhancing the generalization ability and information fidelity of cross-modal features.
[0105] In one embodiment, the encoding part of the fully connected autoencoder is implemented using three fully connected encoding layers, the parameters of which are shown in Table 3: the first fully connected encoding layer enc_fc1, the second fully connected encoding layer enc_fc2, and the third fully connected encoding layer enc_fc3. The decoding part of the fully connected autoencoder is implemented using three fully connected decoding layers: the first fully connected decoding layer dec_fc1, the second fully connected decoding layer dec_fc2, and the third fully connected decoding layer dec_fc3. Table 3. Layer parameters of a fully connected autoencoder Understandably, the fully connected autoencoder is pre-trained, and its input data can be the concatenated features corresponding to the output data of the convolutional autoencoder after training, or it can be pre-labeled concatenated features. Its output is a cross-modal feature.
[0106] In this embodiment, high-level semantic features are extracted by dimensional compression of the spliced features through the encoding part, and then a vector of the same size as the spliced features is reconstructed by the decoding part. This learns a compact cross-modal representation containing key multimodal information, thereby enhancing the generalization ability and information fidelity of the cross-modal features.
[0107] In one embodiment, in Figure 1 Based on the example action recognition method, step 14, which identifies the action type of the inertial sensor according to the cross-modal features, may include: inputting the cross-modal features into a classifier to obtain the action type of the inertial sensor; the classifier includes at least one fully connected layer. In this embodiment, the cross-modal feature space is mapped to the action type space by a classifier containing at least one fully connected layer, and the final action type is determined by taking the maximum value of the output probability distribution.
[0108] In this embodiment, the classifier is connected to the embedding layer of the fully connected autoencoder, which consists of three fully connected layers. The first two fully connected layers each contain 128 hidden units and use ReLU activation functions. In this embodiment, by employing a three-layer fully connected classifier, the cross-modal feature space is mapped to the action type space, and the final action type is determined by maximizing the output probability distribution.
[0109] Based on the above convolutional autoencoder, fully connected autoencoder and classifier, an action recognition model can be obtained. The input data of the action recognition model is 1×3×1000 sensor data, which is split into four groups of 1×3×250 ([1, 3, 250]) dimensional channel data after being split by the segmentation layer. These represent the input data of the acceleration and angular velocity data of the left watch and the right watch, respectively.
[0110] The acceleration data of the left-hand watch, 1×3×250, is input into the first convolutional layer L_Acc_Conv1 of the left-hand acceleration data. This first convolutional layer L_Acc_Conv1 has a kernel shape of 1×11, a stride of 2, a group number of 3, pads of (5,5) (5 vertically and 5 horizontally), and a dilation coefficient of 1. It outputs 9×1×11 dimensional acceleration data. This 9×1×11 dimensional acceleration data is then processed by a ReLU activation function and input into the second convolutional layer L_Acc_Conv2 of the left-hand acceleration data. This second convolutional layer L_Acc_Conv2 has a kernel shape of 3×11, a stride of 2, a group number of 3, pads of (5,5), and a dilation coefficient of 1. It outputs 15×3×11 dimensional acceleration data. The 15×3×11 dimensional acceleration data is activated by a ReLU function and then input into the third convolutional layer L_Acc_Conv3 for left-handed acceleration. This third convolutional layer L_Acc_Conv3 has a kernel shape of 15×11, a stride of 2, a group number of 1, pads of (5,5), and dilations of 1, outputting 24×15×11 dimensional acceleration data. This 24×15×11 dimensional acceleration data can then be activated by a ReLU function and then fed into a max-pooling layer and a reshape layer.
[0111] The 1×3×250 acceleration data from the right-hand watch is input into the first convolutional layer R_Acc_Conv1 of the right-hand acceleration. This first convolutional layer R_Acc_Conv1 has a kernel shape of 1×11, a stride of 2, a group number of 3, pads of (5,5) (5 vertically and 5 horizontally), and a dilation coefficient of 1. It outputs 9×1×11 dimensional acceleration data. This 9×1×11 dimensional acceleration data is then processed by a ReLU activation function and input into the second convolutional layer L_Acc_Conv2 of the right-hand acceleration. This second convolutional layer L_Acc_Conv2 has a kernel shape of 3×11, a stride of 2, a group number of 3, pads of (5,5), and a dilation coefficient of 1. It outputs 15×3×11 dimensional acceleration data. The 15×3×11 dimensional acceleration data is activated by a ReLU function and then input into the third convolutional layer L_Acc_Conv3 for right-handed acceleration. This third convolutional layer L_Acc_Conv3 has a kernel shape of 15×11, a stride of 2, a group number of 1, pads of (5,5), and dilations of 1, outputting 24×15×11 dimensional acceleration data. This 24×15×11 dimensional acceleration data can then be activated by a ReLU function and then fed into a max-pooling layer and a reshape layer.
[0112] The acceleration of the left-hand watch, 1×3×250, is input into the first convolutional layer L_Gyr_Conv1 for the left-hand angular velocity. This first convolutional layer L_Gyr_Conv1 has a kernel shape of 3×3, a stride of 2, a group number of 3, pads of (1,1), and dilations of 1, outputting 9×3×3 dimensional angular velocity data. This 9×3×3 dimensional angular velocity data, after passing through a ReLU activation function, is input into the second convolutional layer L_Gyr_Conv2 for the left-hand angular velocity. This second convolutional layer L_Gyr_Conv2 also has a kernel shape of 3×3, a stride of 2, a group number of 1, pads of (1,1), and dilations of 1, outputting 15×9×3 dimensional angular velocity data. The 15×9×3 angular velocity data is activated by a ReLU function and then input into the third convolutional layer L_Gyr_Conv3 for left-handed angular velocity. This third convolutional layer L_Gyr_Conv3 has a kernel shape of 3×3, a stride of 2, a group number of 1, pads of (1,), and dilations of 1, outputting 24×15×3 angular velocity data. This 24×15×3 angular velocity data can then be activated by a ReLU function and then fed into a max-pooling layer and a reshape layer.
[0113] The acceleration of the right-hand watch, 1×3×250, is input into the first convolutional layer L_Gyr_Conv1 for the right-hand angular velocity. This first convolutional layer L_Gyr_Conv1 has a kernel shape of 3×3, a stride of 2, a group number of 3, pads of (1,1), and dilations of 1, outputting 9×3×3 dimensional angular velocity data. This 9×3×3 dimensional angular velocity data is then processed by a ReLU activation function and input into the second convolutional layer L_Gyr_Conv2 for the right-hand angular velocity. This second convolutional layer L_Gyr_Conv2 also has a kernel shape of 3×3, a stride of 2, a group number of 1, pads of (1,1), and dilations of 1, outputting 15×9×3 dimensional angular velocity data. The 15×9×3 angular velocity data is activated by a ReLU function and then input into the third convolutional layer L_Gyr_Conv3 for right-handed angular velocity. This third convolutional layer L_Gyr_Conv3 has a kernel shape of 3×3, a stride of 2, a group number of 1, pads of (1,) and dilations of 1, and outputs 24×15×3 angular velocity data. This 24×15×3 angular velocity data can then be activated by a ReLU function and then fed into a max-pooling layer and a reshape layer.
[0114] Based on the four channels mentioned above, four sets of output data are obtained, which are then concatenated together by a concat layer to obtain the spliced features of the sensor data. These spliced features can be processed by a fully connected autoencoder, specifically through a 128×1536-dimensional fully connected encoding layer (enc_fc1), a ReLU activation function, a 128×128-dimensional fully connected encoding layer (enc_fc2), and a ReLU activation function; then through a 128×128-dimensional fully connected decoding layer (dec_fc1), a 128×128-dimensional fully connected decoding layer (dec_fc2), a ReLU activation function, and an 11×128-dimensional fully connected decoding layer (dec_fc3), to obtain cross-modal features.
[0115] Then, the cross-modal features are input into a three-layer fully connected layer of the classifier to obtain a 1×11 dimensional action classification result. Finally, the maximum value of the action classification result is extracted, and the index corresponding to the maximum value is the predicted action type. The action type is one of the following: dumbbell bench press, dumbbell fly, dumbbell push-up, dumbbell deadlift, dumbbell curl, dumbbell squat, dumbbell lunge, dumbbell Russian twist, dumbbell crunch, and dumbbell power clean.
[0116] The motion recognition model described above can be used to identify the types of movements of a smart dumbbell using sensor data, thereby improving the accuracy of motion recognition.
[0117] Based on the above-described action recognition method, this disclosure also provides an action recognition device, see [link to relevant documentation]. Figure 4 The device includes: The multimodal data acquisition module 41 is used to acquire multimodal data collected by the inertial sensor; The single-modal feature acquisition module 42 is used to extract features from various modal data in the multiple modal data respectively to obtain single-modal features of various modal data; the single-modal features are specific representations of various modal data within that modality; The cross-modal feature acquisition module 43 is used to extract cross-modal features based on the single-modal features of the various modal data to obtain cross-modal features; the cross-modal features are specific representations between at least two modal data. Action type recognition module 44 is used to identify the action type of the inertial sensor based on the cross-modal characteristics.
[0118] In this embodiment, modal features are extracted independently for each modal data, i.e., single-modal features are extracted, which can uncover specific representations within a modality; then, cross-modal features are extracted based on the single-modal features, which can fully uncover the inter-modal correlation features, i.e., cross-modal features. The action type is identified by combining single-modal features and cross-modal features, which can improve the accuracy of action recognition.
[0119] In one embodiment, the single-modal feature acquisition module includes: The time window data segment acquisition submodule is used to segment various modal data according to a preset time window to obtain multiple time window data segments; The single-modal feature acquisition submodule is used to input each time window data segment under each modality into the convolutional autoencoder for feature extraction to obtain the single-modal features under each modality; wherein, the convolutional autoencoder includes a bottleneck structure, which is a symmetric network structure in which the encoder channel continuously compresses the data dimension and the decoder channel gradually restores the data dimension.
[0120] In this embodiment, by setting a convolutional autoencoder with a bottleneck structure, the convolutional autoencoder can learn the modality-specific representation of modal data under a single modality, learn a compact modality-specific representation, and reduce the feature redundancy of single modality features.
[0121] In one embodiment, the single-modal feature acquisition module further includes a time window length acquisition module, which is used to acquire the length of the preset time window; the time window length acquisition module includes: The candidate peak list acquisition submodule is used to perform peak detection on the modal data to determine the local maximum value in the modal data and obtain a candidate peak list; The candidate peak list sorting submodule is used to sort the candidate peaks in the candidate peak list according to the amplitude; The peak period determination submodule is used to determine the peak period of each candidate peak based on autocorrelation analysis. The time window length determination submodule is used to determine the length of the preset time window based on the peak period.
[0122] In this embodiment, the action cycle is automatically identified through peak detection and autocorrelation analysis, and the time window is adaptively adjusted, which helps to improve the accuracy of the obtained single-modal features.
[0123] In one embodiment, the convolutional autoencoder includes an encoder and a decoder, the encoder including at least one convolutional layer and at least one pooling layer, and the decoder including at least one deconvolutional layer and at least one depooling layer; the encoder and the decoder are arranged in a symmetrical structure.
[0124] In this embodiment, by setting the convolutional autoencoder to a symmetrical structure, with the same number of convolutional layers in the encoder and the same number of deconvolutional layers in the decoder, as well as the same number of pooling layers in the encoder and the same number of depooling layers in the decoder, the correspondence between feature extraction and reconstruction can be guaranteed, thereby improving the expressive power of modal features.
[0125] In one embodiment, the cross-modal feature acquisition module includes: The splicing feature acquisition submodule is used to flatten and splice the single-modal features of each modality to obtain the spliced features of all single-modal features under multiple modalities; The cross-modal feature acquisition submodule is used to input the spliced features into a fully connected autoencoder for cross-modal feature extraction to obtain the cross-modal features; the fully connected autoencoder is used to learn cross-modal representations that are discriminative for action type recognition tasks.
[0126] In this embodiment, all single-modal features are concatenated by a fully connected autoencoder, and cross-modal features are extracted from the concatenated features to learn discriminative cross-modal representations. This fully utilizes the complementary information between different modal data, which helps to improve the accuracy of action recognition.
[0127] In one embodiment, the fully connected autoencoder includes an encoding part and a decoding part, the encoding part including at least one fully connected layer, and the decoding part including at least one fully connected layer; the decoding part of the fully connected autoencoder is used to reconstruct a vector of the same size as the spliced feature.
[0128] In this embodiment, high-level semantic features are extracted by dimensional compression of the spliced features through the encoding part, and then a vector of the same size as the spliced features is reconstructed by the decoding part. This learns a compact cross-modal representation containing key multimodal information, thereby enhancing the generalization ability and information fidelity of the cross-modal features.
[0129] In one embodiment, the action type recognition module includes: An action type recognition submodule is used to input the cross-modal features into a classifier to obtain the action type of the inertial sensor; the classifier includes at least one fully connected layer.
[0130] In this embodiment, a classifier containing at least one fully connected layer maps the cross-modal feature space to the action type space, and the final action type is determined by taking the maximum value of the output probability distribution.
[0131] In one embodiment, the classifier is connected to the embedding layer of a fully connected autoencoder, which consists of three fully connected layers; and the first two fully connected layers each contain 128 hidden units and use the ReLU activation function as the activation function.
[0132] In this embodiment, a classifier with three fully connected layers is used to map the cross-modal feature space to the action type space, and the final action type is determined by taking the maximum value of the output probability distribution.
[0133] In one embodiment, the inertial sensor is disposed inside the dumbbell body.
[0134] In this embodiment, the sensor is integrated inside the dumbbell body to directly collect dumbbell motion data, improving the accuracy of data collection and eliminating the need for additional wearable devices.
[0135] In one embodiment, the type of movement includes at least one of the following: dumbbell bench press, dumbbell fly, dumbbell push-up, dumbbell deadlift, dumbbell curl, dumbbell squat, dumbbell lunge, dumbbell Russian twist, dumbbell crunch, and dumbbell power clean.
[0136] In this embodiment, it can support the recognition of ten types of movements of smart dumbbells, covering most resistance training programs and meeting the user's need for guidance on movement recognition.
[0137] It is understood that the various embodiments of the action recognition device provided in this embodiment have been described when describing the action recognition method. The device embodiment can be referred to the content of the method embodiment, and will not be repeated here.
[0138] This disclosure also provides an intelligent dumbbell, including the aforementioned inertial sensor, which is used to collect multiple modal data, including acceleration data and angular velocity data. This multiple modal data can be transmitted to a computing device, which determines the dynamic type of the intelligent dumbbell.
[0139] This disclosure also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the methods of any of the above embodiments.
[0140] This disclosure also provides a computing device, including a memory and a processor; the memory is used to store computer instructions that can be executed on the processor, and the processor is used to implement the methods of any of the above embodiments when executing the computer instructions.
[0141] Figure 5 This is a schematic diagram of the structure of a computing device provided in this disclosure, such as... Figure 5 As shown, the computing device 5 may include, but is not limited to: processor 51, memory 52, and bus 53 connecting different system components (including memory 52 and processor 51).
[0142] The memory 52 stores computer instructions that can be executed by the processor 51, enabling the processor 51 to perform the action recognition method of any of the above embodiments. The memory 52 may include a random access memory (RAM) 521, a cache memory 522, and / or a read-only memory (ROM) 523. The memory 52 may also include a program tool 525 having a set of program modules 524, including but not limited to: an operating system, one or more application programs, other program modules, and program data. One or more combinations of these program modules may include an implementation of a network environment.
[0143] Bus 53 may include, for example, a data bus, an address bus, and a control bus. The computing device 5 can also communicate with external devices 55 via I / O interface 54, such as a keyboard or a Bluetooth device. The computing device 5 can also communicate with one or more networks via network adapter 56, such as a local area network (LAN), a wide area network (WAN), or a public network. As shown in the figure, the network adapter 56 can also communicate with other modules of the computing device 5 via bus 53.
[0144] This disclosure also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer is able to implement the methods described above.
[0145] Furthermore, although the operations of the methods disclosed herein are described in a specific order in the accompanying drawings, this does not require or imply that these operations must be performed in that specific order, or that all of the operations shown must be performed to achieve the desired result. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into one step, and / or one step may be broken down into multiple steps.
[0146] While the spirit and principles of this disclosure have been described with reference to several specific embodiments, it should be noted that this disclosure is not limited to the disclosed specific embodiments, and the division of aspects does not imply that features in these aspects cannot be combined for benefit; such division is merely for convenience of expression. This disclosure is intended to cover various modifications and equivalent arrangements included within the spirit and scope of the appended claims.
Claims
1. An action recognition method, characterized in that, The method includes: Acquire multiple modal data collected by inertial sensors; Feature extraction is performed on each modal data in the multiple modal data to obtain single-modal features of each modal data; the single-modal features are the specific representations of each modal data within that modality; Cross-modal features are extracted based on the single-modal features of the various modal data to obtain cross-modal features; the cross-modal features are specific representations between at least two modal data. The action type of the inertial sensor is identified based on the cross-modal characteristics.
2. The method according to claim 1, characterized in that, Feature extraction is performed on various modal data from the multiple modal data, including: The data of various modalities are segmented according to a preset time window to obtain multiple time window data segments; Each time window data segment under each modality is input into a convolutional autoencoder for feature extraction to obtain single-modal features under each modality; wherein, the convolutional autoencoder includes a bottleneck structure, which is a symmetric network structure in which the encoder channel continuously compresses the data dimension and the decoder channel gradually restores the data dimension.
3. The method according to claim 2, characterized in that, The method further includes a step of obtaining the length of the preset time window, specifically including: Peak detection is performed on the modal data to determine the local maximum values in the modal data, thereby obtaining a candidate peak list; The candidate peaks in the candidate peak list are sorted according to the amplitude, and the peak period of each candidate peak is determined by autocorrelation analysis. The length of the preset time window is determined based on the peak period.
4. The method according to claim 2, characterized in that, The convolutional autoencoder includes an encoder and a decoder. The encoder includes at least one convolutional layer and at least one pooling layer, and the decoder includes at least one deconvolutional layer and at least one depooling layer. The encoder and the decoder are arranged in a symmetrical structure.
5. The method according to any one of claims 1-4, characterized in that, Cross-modal features are extracted based on the single-modal features of the various modal data to obtain cross-modal features, including: The single-modal features of each modality are flattened and then stitched together to obtain the stitched features of all single-modal features under multiple modalities; The spliced features are input into a fully connected autoencoder for cross-modal feature extraction to obtain the cross-modal features; the fully connected autoencoder is used to learn cross-modal representations that are discriminative for action type recognition tasks.
6. The method according to claim 5, characterized in that, The fully connected autoencoder includes an encoding part and a decoding part. The encoding part includes at least one fully connected layer, and the decoding part includes at least one fully connected layer. The decoding part of the fully connected autoencoder is used to reconstruct a vector of the same size as the spliced feature.
7. The method according to claim 1, characterized in that, Identifying the action type of the inertial sensor based on the cross-modal characteristics includes: The cross-modal features are input into a classifier to obtain the action type of the inertial sensor; the classifier includes at least one fully connected layer.
8. The method according to claim 1, characterized in that, The inertial sensor is located inside the dumbbell body.
9. A motion recognition device, characterized in that, The device includes: The multimodal data acquisition module is used to acquire multiple modal data collected by the inertial sensor; The single-modal feature acquisition module is used to extract features from various modal data in the multiple modal data to obtain single-modal features of various modal data; the single-modal features are specific representations of various modal data within that modality; A cross-modal feature acquisition module is used to extract cross-modal features based on the single-modal features of the various modal data to obtain cross-modal features; the cross-modal features are specific representations between at least two modal data. An action type recognition module is used to identify the action type of the inertial sensor based on the cross-modal characteristics.
10. A smart dumbbell, characterized in that, It includes an inertial sensor for acquiring multiple modal data, including acceleration data and angular velocity data.
