Behavior recognition method and device and wearable equipment
By acquiring and processing audio and sensor data, and combining them with a multimodal feature recognition model, the problem of insufficient accuracy of wearable devices in recognizing complex user behaviors has been solved. This has enabled comprehensive recognition of user posture, behavior, and environmental information, while protecting user privacy.
Patent Information
- Application Number
- CN202411178292.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-26
- Publication Date
- 2026-03-03
AI Technical Summary
Existing wearable devices lack accuracy in recognizing user behavior, especially in complex and diverse scenarios, situations, and psychological states where they struggle to effectively identify user posture, behavior, and environmental information.
By acquiring audio and sensor data, and employing a pre-defined data processing strategy for standardization, the system combines target audio segments, sensor segments, and behavior recognition models to extract multimodal features to identify user posture, behavior, and environmental information, while protecting user privacy through encryption strategies.
It improves the accuracy of user behavior recognition, ensures data security and privacy, and enables more comprehensive recognition of complex user behaviors.
Smart Images
Figure CN121587708A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of wearable devices, and more particularly to a behavior recognition method, apparatus, and wearable device. Background Technology
[0002] With the continuous improvement of people's health awareness and the continuous development of wearable device technology, the use of wearable devices for body monitoring and identification is also increasing.
[0003] Currently, wearable devices mainly monitor user movement behavior data through accelerometers and / or monitor user heart rate-related data through heart rate sensors, and then identify user behavior based on user movement behavior data and / or heart rate-related data.
[0004] In real-world applications, users exhibit a wide variety of behaviors under different scenarios, situations, and psychological states. In other words, the user behaviors that need to be identified are complex and diverse, and the aforementioned behavior identification methods are not very accurate in identifying user behaviors. Summary of the Invention
[0005] This application provides a behavior recognition method, apparatus, and wearable device that can improve the accuracy of user behavior recognition.
[0006] In a first aspect, a behavior recognition method is provided, comprising: acquiring audio data and sensor data, wherein the sensor data includes at least acceleration data and heart rate data; processing the audio data and sensor data based on a preset data processing strategy to obtain a target audio segment and a target sensor segment, wherein the preset data processing strategy is used to perform normalization operations on the audio data and sensor data; and recognizing user behavior data based on the target audio segment, the target sensor segment, and a target behavior recognition model, wherein the behavior data includes posture information, behavior information, and environmental information, and the target behavior recognition model is used to extract multimodal features and recognize behavior data based on the multimodal features, wherein the multimodal features include audio features extracted from the target audio segment and sensor features extracted from the target sensor segment.
[0007] This application uses audio data as one of the data sources for behavior recognition, and identifies user behavior data based on multimodal features. The identified behavior data includes posture information, behavior information, and environmental information, thereby improving the accuracy of user behavior recognition.
[0008] In one possible implementation, the method further includes: acquiring location data; processing the location data based on a preset data processing strategy to obtain a target location segment; and identifying user behavior data based on the target audio segment, target sensor segment, and target behavior recognition model, including: inputting the target audio segment, target sensor segment, and target location segment into the target behavior recognition model to obtain behavior data.
[0009] The acquired data includes not only audio and sensor data, but also location data, improving the accuracy of identifying user behavior data, especially environmental information.
[0010] In one possible implementation, the method further includes: encrypting the audio data based on a preset first encryption strategy, wherein the preset first encryption strategy includes an encryption algorithm, or the preset first encryption strategy includes an encryption algorithm and a privacy algorithm.
[0011] Alternatively, the target audio segment can be encrypted based on a preset second encryption strategy, wherein the preset second encryption strategy includes an encryption algorithm, or the preset second encryption strategy includes an encryption algorithm and a privacy algorithm.
[0012] Encrypting audio data ensures that this private information is not leaked, thus preventing the disclosure of users' privacy information.
[0013] In one possible implementation, audio data and sensor data are processed based on a preset data processing strategy to obtain target audio segments and target sensor segments, including:
[0014] The initial audio segments for each preset time period are determined based on audio data, and the initial sensor segments for each preset time period are determined based on sensor data. The initial audio segments and initial sensor segments are then standardized using a preset standardization strategy to obtain the target audio segments and target sensor segments. The preset standardization strategy is used to adjust the lengths of the initial audio segments and initial sensor segments.
[0015] Different data collection frequencies and data types vary. Standardizing the acquired audio and sensor data improves data quality, increases consistency between data, and facilitates subsequent data analysis.
[0016] In one possible implementation, the target behavior recognition model includes a feature extraction module and a classification module; the feature extraction module is used to extract target features and combine target features into multimodal features; the classification module is used to map multimodal features to corresponding posture information, behavior information and environmental information; wherein, the target features include at least audio features and sensor features.
[0017] In one possible implementation, the feature extraction module includes at least an audio feature extraction unit and a sensor feature extraction unit; the audio feature extraction unit is used to preprocess the target audio segment based on a first preprocessing strategy and extract audio features, wherein the first preprocessing strategy is used to perform feature transformation on the target audio segment; the sensor feature extraction unit is used to preprocess the target sensor segment based on a second preprocessing strategy and extract sensor features, wherein the second preprocessing strategy is used to perform dimensionality partitioning and normalization processing on the target sensor segment.
[0018] By setting up multiple feature extraction units, features can be extracted from different data, thereby improving the accuracy of feature extraction for different data.
[0019] In one possible implementation, the classification module includes a pose classification unit, a behavior classification unit, and an environment classification unit; the pose classification unit is used to map multimodal features to corresponding pose information; the behavior classification unit is used to map multimodal features to corresponding behavior information; and the environment classification unit is used to map multimodal features to corresponding environment information.
[0020] The classification module includes a posture classification unit, a behavior classification unit, and an environment classification unit, which can identify the posture information, behavior information, and environment information contained in the multimodal features obtained above.
[0021] In one possible implementation, the method further includes: acquiring an initial sample dataset, which includes audio sample data and sensor sample data representing various human activities; determining multiple target sample datasets based on the initial sample dataset, wherein the target sample datasets are subsets of the sample datasets; training the initial behavior recognition model based on each target sample dataset to obtain multiple candidate behavior recognition models; and determining the target behavior recognition model from the multiple candidate behavior recognition models based on a preset prediction strategy.
[0022] The target behavior recognition model is determined from multiple candidate behavior recognition models trained, thereby improving the accuracy of the target behavior recognition model in recognizing user behavior data.
[0023] Secondly, a behavior recognition device is provided, including an acquisition module, a data processing module, and a recognition module.
[0024] The acquisition module is used to acquire audio data and sensor data, and the sensor data includes at least acceleration data and heart rate data.
[0025] The data processing module is used to process audio data and sensor data based on a preset data processing strategy to obtain target audio segments and target sensor segments. The preset data processing strategy is used to perform normalization operations on the audio data and sensor data.
[0026] The recognition module is used to identify user behavior data based on target audio segments, target sensor segments, and target behavior recognition models. The behavior data includes posture information, behavior information, and environmental information. The target behavior recognition model is used to extract multimodal features and identify behavior data based on the multimodal features. The multimodal features include audio features extracted from target audio segments and sensor features extracted from target sensor segments.
[0027] Thirdly, a behavior recognition device is provided, comprising a processor and a memory, wherein the memory stores a computer program, and the processor retrieves and runs the computer program from the memory, causing the behavior recognition device to perform any of the behavior recognition methods in the first aspect.
[0028] Fourthly, a wearable device is provided, which includes at least an audio input module, a sensor module, and a processing module, wherein the processing module is communicatively connected to the audio input module and the sensor module; the processing module is configured to execute any of the behavior recognition methods in the first aspect.
[0029] Fifthly, a computer-readable storage medium is provided, which stores a computer program that, when executed by a behavior recognition device, causes the behavior recognition device to perform any of the behavior recognition methods in the first aspect.
[0030] In a sixth aspect, a computer program product is provided, comprising: a computer program that, when run by a behavior recognition device, causes the behavior recognition device to perform any of the behavior recognition methods in the first aspect. Attached Figure Description
[0031] Figure 1 This is a flowchart of the behavior recognition method provided in the embodiments of this application;
[0032] Figure 2 This is a flowchart of another behavior recognition method provided in the embodiments of this application;
[0033] Figure 3 This is a flowchart of the data processing in the behavior recognition method provided in the embodiments of this application;
[0034] Figure 4 This is a flowchart of another behavior recognition method provided in the embodiments of this application;
[0035] Figure 5This is a schematic diagram illustrating the data processing of the target behavior recognition model provided in the embodiments of this application;
[0036] Figure 6 This is a structural diagram of the target behavior recognition model provided in the embodiments of this application;
[0037] Figure 7 This is a structural diagram of another target behavior recognition model provided in the embodiments of this application;
[0038] Figure 8 This is a structural diagram of another target behavior recognition model provided in the embodiments of this application;
[0039] Figure 9 This is a structural diagram of another target behavior recognition model provided in the embodiments of this application;
[0040] Figure 10 This is an example diagram of the target behavior recognition model provided in the embodiments of this application;
[0041] Figure 11 This is a flowchart illustrating the determination of the target behavior recognition model in the behavior recognition method provided in this application embodiment;
[0042] Figure 12 This is an example diagram illustrating the determination of the target behavior recognition model in the behavior recognition method provided in this application embodiment;
[0043] Figure 13 This is a flowchart of another behavior recognition method provided in the embodiments of this application;
[0044] Figure 14 This is a schematic diagram of the behavior recognition device provided in the embodiments of this application;
[0045] Figure 15 A schematic diagram of the structure of a wearable device provided in this application is shown. Detailed Implementation
[0046] The technical solutions of the embodiments of this application will be described below with reference to the accompanying drawings. In the description of the embodiments of this application, unless otherwise stated, " / " means "or," for example, A / B can mean A or B; "and / or" in this text is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Furthermore, in the description of the embodiments of this application, "multiple" refers to two or more than two.
[0047] Hereinafter, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first," "second," or "third" may explicitly or implicitly include one or more of that feature.
[0048] Specific details, such as particular system architectures and techniques, are set forth for illustrative purposes and not for limitation, to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application can be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods are omitted to avoid unnecessary detail that could obscure the description of this application.
[0049] For ease of understanding, some concepts involved in the embodiments of this application are introduced and explained below:
[0050] (1) Photoplethysmography
[0051] Photoplethysmography (PPG) is a non-invasive optical measurement technique. It measures changes in blood volume by illuminating the skin with a light source (usually a light-emitting diode) and then detecting changes in the amount of light absorbed and reflected by the blood, thereby obtaining a pulse wave signal.
[0052] PPG (Photogrammetry) is a common working principle in heart rate sensors. PPG-based heart rate sensors typically consist of a light source (such as an LED) and a photosensor. The light source emits light of a specific wavelength, which shines onto the skin tissue. The photosensor receives the reflected or transmitted light and converts it into an electrical signal, which is then processed and analyzed to obtain heart rate data.
[0053] It should be understood that heart rate sensors are not limited to using PPG. For example, there are also heart rate sensors that use bioelectricity. The core of these sensors is to use bioelectrical impedance sensors to monitor blood flow through the body's own impedance and convert it into specific heart rate, respiratory rate, and skin conductance index.
[0054] (2) FBank features
[0055] FBank features (Filter Bank features) are a commonly used feature representation in speech signal processing.
[0056] With the continuous improvement of people's health awareness and the continuous development of wearable device technology, the use of wearable devices for body monitoring and identification is also increasing.
[0057] Currently, wearable devices mainly monitor user movement behavior data through accelerometers and / or monitor user heart rate-related data through heart rate sensors. In turn, user behavior can be identified based on user movement behavior data and / or heart rate-related data.
[0058] In real-world applications, users exhibit a wide variety of behaviors depending on the scenario, context, and psychological state. The user behaviors that need to be identified are complex and diverse. However, the aforementioned behavior identification methods are not very accurate in identifying user behaviors. Therefore, there is an urgent need for a method with high accuracy in identifying user behaviors.
[0059] In view of this, embodiments of this application provide a behavior recognition method, apparatus, and wearable device. The behavior recognition method includes: first, acquiring audio data and sensor data, wherein the sensor data includes at least acceleration data and heart rate data; performing normalization processing on the audio data and sensor data based on a preset data processing strategy to obtain a target audio segment and a target sensor segment; then, recognizing user behavior data based on the target audio segment, the target sensor segment, and a target behavior recognition model, wherein the behavior data includes posture information, behavior information, and environmental information. The target behavior recognition model is used to extract multimodal features (i.e., audio features extracted from the target audio segment and sensor features extracted from the target sensor segment), and to recognize behavior data based on the multimodal features. This application uses audio data as one of the data sources for behavior recognition, improving the accuracy of user behavior recognition.
[0060] The following is combined Figures 1 to 13 The behavior recognition method provided in the embodiments of this application will be described in detail. It should be noted that if there are substantially the same result, the method of this application is not necessarily the same. Figures 1 to 13 The sequence of processes shown is limited.
[0061] Figure 1 This is a flowchart of the behavior recognition method provided in the embodiments of this application, such as... Figure 1 As shown, the behavior recognition method includes the following steps:
[0062] S110: Acquire audio data and sensor data.
[0063] It should be understood that the behavior recognition method provided in this application embodiment can be applied to wearable devices, which can be smartwatches, smart bracelets, wristbands, helmets, headbands, glasses, or other wearable devices that can be used to detect user running behavior data and / or heart rate-related data.
[0064] It should also be understood that wearable devices can also be used in medical testing or other devices that measure a user's physiological parameters.
[0065] Wearable devices include an audio input module (such as a microphone) that is used to collect audio signals, enabling the wearable device to acquire audio data.
[0066] It should be understood that audio data includes at least ambient sound data. In some application scenarios, audio data may also include voice data, such as the voice data of wearable device users (i.e., users) or the voice data of users communicating with others.
[0067] Among them, music data, animal call data, machine operation sound data, and special sound effect data (such as explosion sound, wind sound, and rain sound) can also be separated from environmental sound data.
[0068] The sensor data includes at least acceleration data and heart rate data. Acceleration data is quantitative information used to describe how fast the user's behavioral state changes; heart rate data is quantitative information used to reflect the frequency of the user's heartbeat.
[0069] In practical applications, acceleration data can be acquired using various sensors (such as accelerometers), which measure the acceleration of an object in different directions (typically three orthogonal directions, such as the X, Y, and Z axes). Heart rate data can be monitored using various heart rate sensors (such as optical sensors).
[0070] In some embodiments, sensor data may also include one or more of the following: blood oxygen saturation data, angular velocity data, blood pressure data, sleep data, body temperature data, and environmental data (such as ambient temperature, humidity, and air pressure). The above sensor data can enhance the monitoring of user behavior from multiple angles.
[0071] This application uses audio data as one of the data sources for behavior recognition, thereby improving the accuracy of user behavior recognition.
[0072] Audio data may contain sensitive user information, such as private conversations, confidential business negotiations, and medical consultations. To prevent the leakage of user privacy information, audio data can be encrypted to ensure that this private information is not disclosed. Figure 2 This is a flowchart of another behavior recognition method provided in the embodiments of this application, such as... Figure 2 As shown, after step 110, the following steps are also included:
[0073] S210. Encrypt the audio data based on the preset first encryption strategy.
[0074] The preset first encryption strategy includes an encryption algorithm, or the preset first encryption strategy includes an encryption algorithm and a privacy algorithm.
[0075] Encryption algorithms primarily focus on encoding and converting audio data, preventing unauthorized users from understanding and accessing the original audio data. Common encryption techniques, such as symmetric encryption (e.g., Advanced Encryption Standard, or AES) and asymmetric encryption (e.g., RSA), use keys to encrypt and decrypt data.
[0076] Privacy-preserving algorithms focus more on processing and analyzing data while protecting data privacy. For example, differential privacy algorithms add noise during data processing, making it difficult to infer individual data; anonymization algorithms remove or obscure personally identifiable information to prevent data from being directly associated with specific individuals.
[0077] In practical applications, encryption algorithms can be used to encrypt audio data; alternatively, encryption algorithms can be used first to ensure the confidentiality of the data during storage and transmission, and then privacy algorithms can be used to further protect the privacy of the data when processing and analyzing the encrypted data.
[0078] In other words, a preset first encryption strategy ensures the security and privacy of audio data during acquisition, transmission, and processing.
[0079] In some embodiments, audio data and sensor data can also be encrypted based on a preset first encryption strategy. Compared to audio data, sensor data contains less sensitive information about the user, but in order to prevent leakage of the user's privacy information, sensor data can also be encrypted.
[0080] like Figure 1 As shown, it also includes:
[0081] S120. Based on a preset data processing strategy, the audio data and sensor data are processed to obtain the target audio segment and the target sensor segment.
[0082] Among them, the preset data processing strategy is used to standardize audio data and sensor data.
[0083] It should be understood that different data have different acquisition frequencies and data types. For example, the acquisition frequency of accelerometers and optical sensors is 25Hz, while the sampling frequency of microphones is 8000Hz. In order to improve data quality, increase the consistency between data, and facilitate subsequent data analysis, it is necessary to standardize the acquired audio data and sensor data. Figure 3 This is a flowchart of the data processing in the behavior recognition method provided in the embodiments of this application, such as... Figure 3 As shown, step 120 may include the following steps:
[0084] S121. Determine the initial audio segment for each preset time period based on audio data, and determine the initial sensor segment for each preset time period based on sensor data.
[0085] The data acquired based on the sampling frequency is enormous. Direct processing often exceeds the computing and storage capabilities of the equipment. Therefore, the data can be grouped by preset time periods to reduce processing complexity and facilitate efficient data processing. Grouping the data by preset time periods also allows for a clearer understanding of the data's changing trends over different time periods.
[0086] For example, the preset time period is one minute, that is, the initial audio segment is one minute of audio data, and the initial sensor segment is one minute of sensor data.
[0087] It should be understood that audio data and sensor data are different. Sensor data is continuous data, while audio data may be continuous or non-continuous. Therefore, the initial sensor segment is also continuous segment data, while the initial audio segment may be continuous segment data or non-continuous segment data.
[0088] S122. Based on a preset standardization strategy, the initial audio segment and the initial sensor segment are standardized to obtain the target audio segment and the target sensor segment.
[0089] The preset standardization strategy is used to adjust the length of the initial audio segment and the initial sensor segment.
[0090] It should be understood that due to various factors (such as the performance of the sensor or audio input module, data transmission delay, etc.), the length of the initial audio segment and the initial sensor segment with a preset time period cannot be exactly equal to the product of the preset time period and the corresponding frequency. Therefore, the length of the initial audio segment and the initial sensor segment can be adjusted by a preset standardization strategy.
[0091] The standard length of each segment can be preset. For segments whose length does not meet the standard length, data can be padded into the segment, either at the beginning or the end. For segments whose length exceeds the standard length, the segment can be truncated, retaining only the beginning or the end of the segment. The process of standardizing the length of segments described above can be understood as a data preprocessing operation, called data padding or data truncation.
[0092] In some embodiments, if the initial audio segment is non-contiguous segment data, then the missing initial audio segments need to be supplemented.
[0093] For example, if there are M initial sensor segments and N initial audio segments (N less than M), indicating that the initial audio segments are non-continuous segment data, then MN additional initial audio segments are needed.
[0094] It should be understood that, ideally, the length can be determined by a preset time period and corresponding frequency, and that length can be set as the standard length.
[0095] The filling data can be the average of the existing data in the segment, or a specific value (such as 0).
[0096] It should be understood that the acquisition frequencies of audio data and sensor data are different. Therefore, the standard lengths of audio segments and sensor segments are different. For ease of description, the standard length of the target sensor segment is referred to as the first standard length, and the standard length of the target audio segment is referred to as the second standard length.
[0097] For example, firstly, the length of the target sensor segment is uniformly set to 1500 units (first standard length), and the length of the target audio segment is uniformly set to 481280 units (second standard length).
[0098] For initial sensor segments shorter than 1500 units, the tail is padded with an average value; for initial sensor segments longer than 1500 units, the tail is truncated.
[0099] For initial audio segments shorter than 481,280 units, the end is padded with the average value; for initial audio segments longer than 481,280 units, the end is truncated; and missing initial audio segments are padded with a specific value of zero.
[0100] In some embodiments, the preset data processing strategy also includes data cleaning (such as removing noise from the data), data normalization, etc.
[0101] In some embodiments, to prevent the leakage of user privacy information, the encryption process for the data can be set after step 120. That is, after step 110, the data encryption process can be temporarily suspended to prevent the leakage of user privacy information. Figure 4 This is a flowchart of another behavior recognition method provided in the embodiments of this application, such as... Figure 4 As shown, after step 120, the following steps are also included:
[0102] S220. Encrypt the target audio segment based on the preset second encryption strategy.
[0103] The preset second encryption strategy includes an encryption algorithm, or the preset second encryption strategy includes an encryption algorithm and a privacy algorithm.
[0104] In practical applications, encryption algorithms can be used to encrypt the target audio segment; alternatively, encryption algorithms can be used first to ensure the confidentiality of the segment during storage and transmission, and then privacy algorithms can be applied to further protect the privacy of the segment during processing and analysis. A pre-set second encryption strategy ensures the security and privacy of audio data during acquisition, transmission, and processing.
[0105] In some embodiments, the target audio segment and the target sensor segment can also be encrypted based on a preset second encryption strategy.
[0106] like Figure 1 As shown, it also includes:
[0107] S130: Based on target audio segments, target sensor segments, and target behavior recognition models, identify user behavior data.
[0108] Behavioral data includes posture information, behavioral information, and environmental information.
[0109] It should be understood that user behavior is complex and diverse. In order to improve the recognition of user behavior, a target behavior recognition model is used to identify behavioral data that includes user posture information, behavioral information and environmental information.
[0110] For example, in related technologies, user behavior recognition mainly focuses on single user behaviors such as running, walking, cycling, and prolonged sitting. However, in reality, user behavior is complex and diverse. For instance, sleeping while sitting, sleeping while lying down, eating / watching TV / playing on the phone / exercising while standing, eating / watching TV / playing on the phone while sitting, exercising while lying down, etc., or sleeping while lying down at home, sleeping while sitting in the office / shopping mall, exercising outdoors while moving, and exercising indoors while standing. To better identify user behavior, user behavior data can be analyzed from different angles. In other words, user behavior can be identified from multiple perspectives, including posture information, behavioral information, and environmental information.
[0111] The information includes posture information (such as lying down, standing, sitting, or moving), behavioral information (such as sleeping, walking, eating / drinking, watching TV, using a mobile phone / tablet / computer, physical activity, reading / writing), and environmental information (such as indoors, outdoors, office areas, gyms, transportation routes, forests, and grasslands). By identifying information from multiple perspectives—posture, behavior, and environment—the system improves its ability to recognize user behavior.
[0112] This application uses audio data as one of the data sources for behavior recognition, and identifies user behavior data based on multimodal features. The identified behavior data includes posture information, behavior information, and environmental information, thereby improving the accuracy of user behavior recognition.
[0113] Figure 5 This is a schematic diagram illustrating the data processing of the target behavior recognition model provided in the embodiments of this application, such as... Figure 5 As shown, for step 130, the target audio segment 511 and the target sensor segment 512 can be used as input data and input to the target behavior recognition model 500. The target behavior recognition model 500 identifies the user's behavior data 520, which includes posture information 521, behavior information 522 and environmental information 523.
[0114] Among them, the target behavior recognition model 500 is used to extract multimodal features and recognize behavior data based on multimodal features. The multimodal features include audio features extracted from target audio segments and sensor features extracted from target sensor segments.
[0115] Figure 6 This is a structural diagram of the target behavior recognition model provided in the embodiments of this application, such as... Figure 6 As shown, the target behavior recognition model 600 may include a feature extraction module 610 and a classification module 620. The feature extraction module 610 is used to extract target features and combine target features into multimodal features. The classification module 620 is used to map multimodal features to corresponding posture information, behavior information and environmental information.
[0116] The target features include at least audio features and sensor features.
[0117] In some embodiments, the target features may also include location features, which can better identify environmental information in behavioral data.
[0118] It should be understood that after the standardization process in step 122, the target sensor fragment and the target audio fragment are consistent in terms of data length. However, there is a serious inconsistency in the value range distribution between the target audio fragment and the target sensor fragment. This inconsistency will interfere with the convergence rate and convergence result of the target behavior recognition model during the training phase, and will also have a negative impact on the generalization performance of the model during the inference phase.
[0119] The sampling frequency of the target audio segment far exceeds the acquisition frequency of the target sensor segment. For example, the absolute length of the target audio segment reaches 481,280 units. Directly using it as input to the target behavior recognition model will pose challenges to the feature extraction, convergence, and inference generalization processes of the target behavior recognition model. Therefore, the segments of each modality need to undergo corresponding processing procedures.
[0120] In other words, the data input to the target behavior recognition model 600 includes at least target audio segments and target sensor segments. The data types and lengths of the target audio segments and target sensor segments are different. By setting multiple feature extraction units, features can be extracted from different data to improve the accuracy of feature extraction for different data.
[0121] Figure 7 This is a structural diagram of another target behavior recognition model provided in the embodiments of this application, such as... Figure 7 As shown, the feature extraction module 610 includes an audio feature extraction unit 611 and a sensor feature extraction unit 612.
[0122] The audio feature extraction unit 611 is used to preprocess the target audio segment based on the first preprocessing strategy and extract audio features, wherein the first preprocessing strategy is used to perform feature transformation on the target audio segment.
[0123] The target audio segment carries relatively complex features, and its length is significantly longer than that of the target sensor segment. Therefore, the first preprocessing strategy is required to convert it into a data format that is easy for the target behavior recognition model 600 to learn.
[0124] In some embodiments, the target audio segment can be converted into a first feature through a first preprocessing strategy, and then the first feature can be expanded to obtain a second feature, wherein the first feature is a single-channel feature and the second feature is a multi-channel feature; for example, the second feature is a three-channel feature.
[0125] In deep learning and machine learning technologies, especially when processing image and audio data, the concept of "channel" is frequently used. In this embodiment, the first feature can be composed of two-dimensional information, one dimension being the time frame and the other being the filter index. This two-dimensional single-channel feature (i.e., the first feature) is expanded into a three-channel feature in the channel dimension to meet the input data requirements of deep learning models, especially Convolutional Neural Networks (CNNs).
[0126] It should be understood that CNNs are designed to process multi-channel image data (such as the three channels of an RGB image). Therefore, their input data needs to be three-channel data. In other words, expanding the first feature of a single channel into a second feature of multiple channels can make the format of the second feature compatible with these models.
[0127] It should also be understood that the obtained second feature is a three-channel feature, which facilitates the audio feature extraction unit 611 in the target behavior recognition model 600 to process the second feature and extract audio features.
[0128] Specifically, the first feature can be expanded to obtain the second feature by copying the first feature three times to form three identical channels, i.e., the second feature. It should be understood that the embodiments of this application do not limit the method of expanding the first feature into the second feature.
[0129] It should also be understood that preprocessing the target audio segment using the first preprocessing strategy can involve pre-emphasis, framing, windowing, fast Fourier transform, squaring, Mel filter, and logarithmic transformation to finally obtain the corresponding FBank feature (i.e., the first feature). In the preprocessing of the target audio segment with a length of 481280, the frame length is 2048, the frame offset is 512, and the number of Mel filters is 128. To facilitate the processing of the FBank feature by the audio feature extraction unit 611 in the model, the FBank feature is copied to obtain a three-channel dimension extended feature (i.e., the second feature), with a size of 224×224.
[0130] In some embodiments, the audio feature extraction unit 611 processes the second feature, wherein the part of the audio feature being extracted may refer to the encoder.
[0131] In order to fully extract the information contained in complex audio data, embodiments of this application may use ResNet-34 (a deep convolutional neural network architecture belonging to Residual Networks, ResNet) that has been fully trained on ImageNet as an encoder for processing target audio segments.
[0132] ResNet-34 mainly consists of 34 learnable convolutional neural networks and linear networks, as well as various non-linear activation functions, pooling layers, etc., and enhances the deep learning capabilities of the model through skip connections and residual learning.
[0133] In the example above, the extended features encoded by ResNet-34 are projected into a high-dimensional feature space of 512. The resulting audio features represent the projection of the behavioral, pose, and environmental information implied by the audio modality into this feature space.
[0134] like Figure 7 As shown, the sensor feature extraction unit 612 is used to preprocess the target sensor segment based on the second preprocessing strategy and extract sensor features, wherein the second preprocessing strategy is used to perform dimensional division and normalization processing on the target sensor segment.
[0135] The features carried by the target sensor fragments are relatively simple. For example, an accelerometer may include acceleration feature values in three spatial dimensions, and a heart rate sensor may include feature values in the heart rate feature dimension.
[0136] For the example above, the target sensor fragment can be split into four feature dimensions by the second preprocessing strategy, and maximum and minimum normalization can be performed on the four feature dimensions respectively to unify the value range and eliminate the influence of dimensions. It should be understood that the embodiments of this application do not limit the above normalization method.
[0137] Among them, the minimum-maximum normalization first centers the data in each feature dimension by the minimum value, and then uses the range to scale, finally converging its value range to [0, 1].
[0138] It should be understood that the target sensor fragments processed by the sensor feature extraction unit include acceleration data, heart rate data, and may also include one or more of the following: blood oxygen saturation data, angular velocity data, blood pressure data, sleep data, body temperature data, and environmental data (such as ambient temperature, humidity, and air pressure). Therefore, the processing of the target sensor fragments through the second preprocessing strategy can involve multiple feature dimensions. Through normalization processing, the value range is unified, and the influence of dimensions is eliminated.
[0139] For example, multiple feature dimensions can be acceleration feature values in three spatial dimensions, heart rate feature values, and angular velocity feature values; or multiple feature dimensions can also be acceleration feature values in three spatial dimensions, heart rate feature values, blood oxygen saturation feature values, angular velocity feature values, body temperature feature values, and ambient temperature feature values.
[0140] In some embodiments, the sensor feature extraction unit 612 performs sensor feature extraction on the preprocessed data, wherein the part of the sensor feature extracted may refer to the encoder.
[0141] An encoder for multiple feature dimensions can consist of multiple convolutional neural networks, nonlinear activation functions, normalization layers, and max pooling layers, and is responsible for encoding the target sensor fragments preprocessed by the second preprocessing strategy into high-dimensional feature vectors (i.e., sensor features).
[0142] Among these advantages, convolutional neural networks (CNNs) have in processing time-series data because they can capture the changing characteristics of signals in local time segments and abstract them into high-dimensional features. Nonlinear activation functions introduce nonlinearity into the model, enabling it to fit and approximate any nonlinear function, thus approximating the potential representations of various user behaviors, postures, and environments in the feature space. Normalization layers can reconverge the outputs of intermediate hidden layers to [0, 1], facilitating subsequent analysis, recognition, and encoding of the model structure. Max pooling layers can compress redundant information and retain the most significant attributes in the feature vector.
[0143] For the four feature dimensions in the example above, the encoder consists of four convolutional neural networks (CNNs). Each CNN simultaneously employs a non-linear activation function (such as a rectified linear unit, ReLU) and a batch normalization (BN) layer. Max pooling layers are applied after the second and fourth CNNs to compress the feature length and extract significant sensor features, which are then flattened. Finally, the preprocessed target sensor fragment, containing three feature dimensions from the accelerometer and one feature dimension from the heart rate monitor, is encoded and unfolded into a 1792-dimensional feature vector, representing the projection of the sensor modal's behavioral, pose, and environmental information into that feature space.
[0144] In some embodiments, if the target features also include location features, since the fragments of each modality need to go through corresponding processing flows separately, a feature extraction unit for processing location features is required. Figure 8 This is a structural diagram of another target behavior recognition model provided in the embodiments of this application, such as... Figure 8 As shown, the feature extraction module 610 includes an audio feature extraction unit 611, a sensor feature extraction unit 612, and a positioning feature extraction unit 613.
[0145] The audio feature extraction unit 611 and the sensor feature extraction unit 612 can respectively execute the corresponding steps in the above method embodiments. Therefore, the audio feature extraction unit 611 and the sensor feature extraction unit 612 will not be described in detail here. Please refer to the description of the corresponding steps above for details.
[0146] The localization feature extraction unit 613 is used to preprocess the target localization segment based on the third preprocessing strategy and extract localization features. The third preprocessing strategy is used to perform feature transformation on the target localization segment.
[0147] It should be understood that the positioning feature extraction unit 613 extracts positioning features from the preprocessed data, and the part that extracts positioning features is the encoder.
[0148] In the identification of user behavior, the accuracy of identifying environmental information through location features is higher. For example, when a user is running on an indoor treadmill, the accuracy of identifying user behavior data through location features, audio features, and sensor features is higher.
[0149] It should also be understood that if the target feature includes other features, and these features are different from the audio features, sensor features and positioning features, the corresponding feature extraction module 610 also includes a corresponding feature extraction unit. By using different feature extraction units to process each modality through corresponding processes, the accuracy of different data feature extraction can be improved.
[0150] To facilitate the analysis and recognition of different features by the classification module 620, the feature extraction module 610 can also concatenate different target features to form multimodal features.
[0151] In some embodiments, the feature extraction module further includes a combination unit for combining different target features into multimodal features.
[0152] For example, if the target features include audio features and sensor features, the audio features and sensor features are spliced together to form multimodal features. The distribution of these multimodal features in the new feature space of 512+1792 dimensions corresponds to the distribution of the behavioral, posture and environmental information contained in the sensors and audio in the feature space.
[0153] In order to identify the pose information, behavioral information, and environmental information contained in the multimodal features obtained above, Figure 9 This is a structural diagram of another target behavior recognition model provided in the embodiments of this application, such as... Figure 9 As shown, the classification module 620 includes a posture classification unit 621, a behavior classification unit 622, and an environment classification unit 623.
[0154] The posture classification unit 621 is used to map multimodal features to corresponding posture information; the behavior classification unit 622 is used to map multimodal features to corresponding behavior information; and the environment classification unit 623 is used to map multimodal features to corresponding environment information.
[0155] It should be understood that the classification module 620 is set up with three units, each of which is used to map multimodal features to corresponding information.
[0156] It should be understood that, for Figure 7 or Figure 8 The target behavior recognition model 600, its classification module 620 are all as follows: Figure 9 As shown.
[0157] In some embodiments, if the user behavior data to be identified can be further divided into other categories, the classification module can be set up as more units to map multimodal features to corresponding information.
[0158] In some embodiments, the pose classification unit 621, the behavior classification unit 622, and the environment classification unit 623 can all be classification heads. Each classification head consists of a layer normalization (LN) function and a multilayer perceptron, with different parameters and weights in each classification head. The multilayer perceptron contains three linear layers and two nonlinear activation functions (Gaussian Error Linear Unit, GeLU).
[0159] For example, posture classification unit 621 maps 512+1792-dimensional multimodal features to several specific human behaviors; behavior classification unit 622 and environment classification unit 623 map 512+1792-dimensional multimodal features to different human postures and different environments.
[0160] In other words, for target audio clips and target sensor clips, the target behavior recognition model identifies user behavior data that includes posture information, behavioral information, and environmental information. For example, posture information can include whether the user is lying down, standing, sitting, or moving; behavioral information can include actions such as sleeping, walking, eating / drinking, watching TV, using a mobile phone / tablet / computer, physical activity, reading / writing; and environmental information can include whether the user is indoors, outdoors, at home, or in the office. By recognizing information from multiple perspectives—posture, behavior, and environment—the recognition of user behavior is improved.
[0161] It should be understood that the target behavior recognition model is the initial behavior recognition model that has been trained. During the training process of the initial behavior recognition model, the cross-entropy loss function guides the model to learn the potential representation of sensor data and audio data in a high-dimensional feature space, and how to map them to different categories of behavior, posture and environment.
[0162] For example, during training, a preset number of epochs will be continuously performed, and the Adam optimizer will be used to optimize the model parameters, with a weight decay factor of 1×10⁻⁶. -5 Meanwhile, the initial behavior recognition model can employ a step learning rate (StepLR) decay strategy, where the learning rate decays to 0.95 of the original learning rate in each round, with an initial learning rate of 1×10⁻⁶. -3 The training data is organized in batches, with each batch containing 128 samples.
[0163] Figure 10 This is an example diagram of the target behavior recognition model provided in the embodiments of this application. Figure 10As shown in the above example, features are extracted from the target audio segment B×1×481280 and the target sensor segment B×4×1500 respectively to obtain multimodal features of B×(512+1792) dimensions. Then, the pose information, behavior information and environment information are obtained by passing them through the pose classification unit, behavior classification unit and environment classification unit respectively.
[0164] It should also be understood that there is a serious class imbalance problem in the human daily activities obtained from the experimental subject set, because human behavior has multiple categories, such as walking, eating, and making phone calls. In the general population, some behaviors occur much more frequently than others. For example, eating, sleeping, walking, and using a mobile phone generally occur significantly more frequently than activities such as reading, writing, and making phone calls in a person's daily activities. When learning from class-imbalanced data, neural network models tend to predict the most frequent human behavior categories to avoid errors. However, biased initial behavior recognition models exhibit poor generalization in practical applications. The model cannot correctly identify categories that occur less frequently in the training data and incorrectly identifies them as high-frequency behavior categories, affecting the model's recognition accuracy and practicality. To improve the accuracy of the initial behavior recognition model in recognizing user behavior data, Figure 11 This is a flowchart illustrating the determination of the target behavior recognition model in the behavior recognition method provided in this application embodiment, as follows: Figure 11 As shown, the behavior recognition method also includes the following steps:
[0165] S310. Obtain the initial sample dataset.
[0166] The initial sample dataset includes audio sample data and sensor sample data that characterize various human activities.
[0167] It should be understood that the initial sample dataset may also include location sample data.
[0168] S320. Based on the initial sample dataset, determine multiple target sample datasets.
[0169] The target sample dataset is a subset of the sample dataset. It can be determined by randomly drawing multiple samples with replacement from the initial sample dataset. Each target sample dataset is used to train an independent model.
[0170] S330. The initial behavior recognition model is trained based on each target sample dataset to obtain multiple alternative behavior recognition models.
[0171] S340. Based on a preset prediction strategy, determine the target behavior recognition model from multiple alternative behavior recognition models.
[0172] For example, on each target sample dataset, the performance of the corresponding trained initial behavior recognition model is evaluated using the cross-validation method, and hyperparameters are tuned; all trained candidate behavior recognition models are integrated and a final prediction is made through a voting mechanism; then, the candidate behavior recognition model with the most votes is selected as the target behavior recognition model, thereby improving the recognition accuracy.
[0173] Figure 12 This is an example diagram illustrating the determination of the target behavior recognition model in the behavior recognition method provided in this application embodiment, such as... Figure 12 As shown, four target sample datasets are determined through the initial sample dataset, which can be used to train four initial behavior recognition models, namely initial behavior recognition model 1#, initial behavior recognition model 2#, initial behavior recognition model 3#, and initial behavior recognition model 4#. The target behavior recognition model is determined by voting.
[0174] It should be understood that the step 130 of inputting the target audio segment and target sensor segment into the target behavior recognition model to identify user behavior data can be implemented in a wearable device. In some embodiments, the target audio segment and target sensor segment can also be transmitted to the cloud wirelessly. In the cloud, the target audio segment and target sensor segment are input into the target behavior recognition model to identify user behavior data. The wearable device receives the user behavior data sent from the cloud, thereby realizing the identification of user behavior and improving the speed of data processing.
[0175] It should be understood that, for the embodiment of identifying user behavior data through a cloud-based target behavior recognition model, the encryption processing of the data can be performed through step 210, and the encrypted data can be transmitted to the cloud wirelessly, and the user behavior data can be identified in the cloud.
[0176] In some embodiments, to improve the accuracy of user behavior data recognition, the data acquired in step 110 includes not only audio data and sensor data, but also location data. Figure 13 This is a flowchart of another behavior recognition method provided in the embodiments of this application, such as... Figure 13 As shown, the behavior recognition method includes the following steps:
[0177] S410: Acquire audio data, sensor data, and location data.
[0178] In other words, relative to Figure 1The flowchart shown also shows that location data is acquired, which can be determined through GPS, Wi-Fi positioning, etc. on wearable devices. Among them, GPS (Global Positioning System) accurately calculates the geographical location of the device by receiving satellite signals, including information such as longitude, latitude, and altitude. This information together constitutes the location data.
[0179] Wi-Fi positioning uses the signal strength and related information of surrounding Wi-Fi hotspots to estimate the location of a device, and the result is also data about the location such as longitude and latitude, i.e., positioning data.
[0180] S420: Based on a preset data processing strategy, process the audio data, sensor data, and positioning data to obtain the target audio segment, the target sensor segment, and the target positioning segment.
[0181] In other words, relative to Figure 1 The flowchart shown also includes: processing the positioning data based on a preset data processing strategy to obtain the target positioning segment.
[0182] S430. Input the target audio segment, target sensor segment, and target positioning segment into the target behavior recognition model to obtain behavior data.
[0183] The steps have been explained in detail above, so they will not be repeated here. Please refer to the corresponding steps above for details.
[0184] It should be understood that the above are illustrative examples of application scenarios and do not limit the application scenarios of this application in any way.
[0185] It should be understood that the above examples are provided to help those skilled in the art understand the embodiments of this application, and are not intended to limit the embodiments of this application to the specific values or scenarios illustrated. Those skilled in the art can obviously make various equivalent modifications or changes based on the above examples, and such modifications or changes also fall within the scope of the embodiments of this application.
[0186] The above text combined Figures 1 to 13 The behavior recognition method of the embodiments of this application is described in detail below. Figure 14 This document describes in detail the device embodiments of this application. It should be understood that the behavior recognition device in the embodiments of this application can execute the various behavior recognition methods described in the foregoing embodiments of this application. That is, the specific working processes of the various products described below can be referred to the corresponding processes in the foregoing method embodiments.
[0187] Figure 14 This is a schematic diagram of a behavior recognition device provided in an embodiment of this application. The behavior recognition device 700 can perform... Figures 1 to 13 The behavior recognition method shown; such as Figure 14As shown, the behavior recognition device 700 includes an acquisition module 710, a data processing module 720, and a recognition module 730.
[0188] The acquisition module 710 is used to acquire audio data and sensor data, and the sensor data includes at least acceleration data and heart rate data.
[0189] The data processing module 720 is used to process audio data and sensor data based on a preset data processing strategy to obtain target audio segments and target sensor segments. The preset data processing strategy is used to perform normalization operations on the audio data and sensor data.
[0190] The recognition module 730 is used to recognize user behavior data based on target audio segments, target sensor segments, and target behavior recognition models. The behavior data includes posture information, behavior information, and environmental information. The target behavior recognition model is used to extract multimodal features and recognize behavior data based on the multimodal features. The multimodal features include audio features extracted from target audio segments and sensor features extracted from target sensor segments.
[0191] Each unit module of the behavior recognition device 700 can execute the corresponding steps in the above method embodiment, so the details of each unit module will not be elaborated here. Please refer to the description of the corresponding steps above for details.
[0192] It should be noted that the aforementioned behavior recognition device 700 is embodied in the form of a functional unit. The term "module" here can be interpreted as "unit" and can be implemented in software and / or hardware, without specific limitations.
[0193] For example, a "module" can be a software program, hardware circuit, or a combination of both that implements the above functions. Hardware circuits may include application-specific integrated circuits (ASICs), electronic circuits, processors (e.g., shared processors, proprietary processors, or group processors) and memory for executing one or more software or firmware programs, combined logic circuits, and / or other suitable components that support the described functions.
[0194] Therefore, the units of the various examples described in the embodiments of this application can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.
[0195] Figure 15 A schematic diagram of the structure of a wearable device provided in this application is shown. Figure 15 The dashed lines in the diagram indicate that the unit or module is optional. The wearable device 800 can be used to implement the behavior recognition method described in the above method embodiments.
[0196] The wearable device 800 includes at least an audio input module 810, a sensor module 820, and a processing module 830, with the processing module 830 communicatively connected to the audio input module 810 and the sensor module 820.
[0197] The device includes an audio input module (such as a microphone) for collecting audio signals, enabling the wearable device to acquire audio data. It also includes a sensor module (such as an accelerometer, optical sensor, gyroscope, temperature sensor, etc.) for collecting sensor signals, allowing the wearable device to acquire sensor data.
[0198] The frequency of the audio data acquired by the audio input module can be 100-20000Hz, and the frequency of the sensor data acquired by the sensor module can be 10-50Hz.
[0199] The processing module 830 in the wearable device 800 includes one or more processors that can support the behavior recognition method in the method embodiments of the wearable device 800. The processor can be a general-purpose processor or a special-purpose processor. For example, the processor can be a central processing unit (CPU), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, such as discrete gates, transistor logic devices, or discrete hardware components.
[0200] The processor can be used to control the wearable device 800, execute software programs, and process data from the software programs. The wearable device 800 may also include a communication module 840 for receiving and transmitting signals.
[0201] For example, wearable device 800 can be a terminal device, communication module 840 can be the transceiver of the terminal device, or communication module 840 can be the transceiver circuit of the terminal device.
[0202] The storage module 850 in the wearable device 800 may include one or more memories storing a program 860. The program 860 can be run by the processing module 830 to generate instructions 870, causing the processor to execute the behavior recognition method described in the above method embodiments according to the instructions 870.
[0203] Optionally, the memory may also store data. Optionally, the processor may also read data stored in the memory, which may be stored at the same memory address as program 860, or the data may be stored at a different memory address than program 860.
[0204] Processors and memory can be configured separately or integrated together; for example, integrated on a system-on-chip (SOC) in a terminal device.
[0205] For example, the memory can be used to store the relevant program 860 of the behavior recognition method provided in the embodiments of this application, and the processor can be used to call the relevant program 860 of the behavior recognition method stored in the memory when performing behavior data recognition on the terminal device, and execute the behavior recognition method of the embodiments of this application.
[0206] This application also provides a computer program product that, when executed by a processor, implements the behavior recognition method of any method embodiment in this application.
[0207] The computer program product can be stored in memory, such as program 860. Program 860 is eventually converted into an executable object file that can be executed by the processor after processes such as preprocessing, compilation, assembly and linking.
[0208] This application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a computer, implements the behavior recognition method of any method embodiment in this application. The computer program may be a high-level language program or an executable object program.
[0209] The computer-readable storage medium is, for example, memory. Memory can be volatile or non-volatile, or it can include both volatile and non-volatile memory. Non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory can be random access memory (RAM), which serves as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous linked dynamic random access memory (SLDRAM), and direct rambus RAM (DR RAM).
[0210] In this application, "at least one" means one or more, and "more than one" means two or more. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or multiple items. For example, at least one of a, b, or c can mean: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.
[0211] It should be understood that in the various embodiments of this application, the order of the above-mentioned processes does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.
[0212] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.
[0213] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.
[0214] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for example, the division of units is merely a logical functional division, and there may be other division methods in actual implementation; for example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, and the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.
[0215] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.
[0216] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.
[0217] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.
Claims
1. A behavior recognition method, characterized in that, include: Acquire audio data and sensor data, wherein the sensor data includes at least acceleration data and heart rate data; The audio data and the sensor data are processed based on a preset data processing strategy to obtain a target audio segment and a target sensor segment. The preset data processing strategy is used to perform normalization operations on the audio data and the sensor data. Based on the target audio segment, the target sensor segment, and the target behavior recognition model, user behavior data is identified. The behavior data includes posture information, behavior information, and environmental information. The target behavior recognition model is used to extract multimodal features and identify the behavior data based on the multimodal features. The multimodal features include audio features extracted from the target audio segment and sensor features extracted from the target sensor segment.
2. The behavior recognition method according to claim 1, characterized in that, Also includes: Obtain location data; The positioning data is processed based on a preset data processing strategy to obtain the target positioning segment; The process of identifying user behavior data based on the target audio segment, the target sensor segment, and the target behavior recognition model includes: The target audio segment, the target sensor segment, and the target positioning segment are input into the target behavior recognition model to obtain the behavior data.
3. The behavior recognition method according to claim 1, characterized in that, Also includes: The audio data is encrypted based on a preset first encryption strategy, wherein the preset first encryption strategy includes an encryption algorithm, or the preset first encryption strategy includes the encryption algorithm and a privacy algorithm. or, The target audio segment is encrypted based on a preset second encryption strategy, wherein the preset second encryption strategy includes an encryption algorithm, or the preset second encryption strategy includes the encryption algorithm and a privacy algorithm.
4. The behavior recognition method according to claim 1, characterized in that, The process of processing the audio data and sensor data based on a preset data processing strategy to obtain target audio segments and target sensor segments includes: The initial audio segments for each preset time period are determined based on the audio data, and the initial sensor segments for each preset time period are determined based on the sensor data. The initial audio segment and the initial sensor segment are standardized based on a preset standardization strategy to obtain the target audio segment and the target sensor segment. The preset standardization strategy is used to adjust the length of the initial audio segment and the initial sensor segment.
5. The behavior recognition method according to any one of claims 1 to 4, characterized in that, The target behavior recognition model includes a feature extraction module and a classification module; The feature extraction module is used to extract target features and combine the target features into the multimodal features; The classification module is used to map the multimodal features to the corresponding pose information, behavior information, and environmental information; wherein the target features include at least audio features and sensor features.
6. The behavior recognition method according to claim 5, characterized in that, The feature extraction module includes at least an audio feature extraction unit and a sensor feature extraction unit; The audio feature extraction unit is used to preprocess the target audio segment based on a first preprocessing strategy and extract the audio features, wherein the first preprocessing strategy is used to perform feature transformation on the target audio segment; The sensor feature extraction unit is used to preprocess the target sensor fragment based on the second preprocessing strategy and extract the sensor features, wherein the second preprocessing strategy is used to perform dimensional division and normalization processing on the target sensor fragment.
7. The behavior recognition method according to claim 5, characterized in that, The classification module includes a posture classification unit, a behavior classification unit, and an environment classification unit; The pose classification unit is used to map the multimodal features to corresponding pose information; The behavior classification unit is used to map the multimodal features to corresponding behavior information; The environment classification unit is used to map the multimodal features to corresponding environmental information.
8. The behavior recognition method according to any one of claims 1 to 4, characterized in that, Also includes: Obtain an initial sample dataset, which includes audio sample data and sensor sample data characterizing various human activities; Based on the initial sample dataset, multiple target sample datasets are determined, wherein the target sample datasets are subsets of the sample datasets; The initial behavior recognition model is trained based on each of the target sample datasets to obtain multiple alternative behavior recognition models; Based on a preset prediction strategy, a target behavior recognition model is determined from multiple candidate behavior recognition models.
9. A behavior recognition device, characterized in that, The behavior recognition device includes a processor and a memory, the memory being used to store a computer program, and the processor being used to call and run the computer program from the memory, causing the behavior recognition device to perform the method of any one of claims 1 to 8.
10. A wearable device, characterized in that, The wearable device includes at least an audio input module, a sensor module, and a processing module, wherein the processing module is communicatively connected to the audio input module and the sensor module; The processing module is configured to perform the method of any one of claims 1 to 8.