Training methods and related devices for human activity type recognition models

Abstract

This application relates to the field of information processing technology, and discloses a training method and related apparatus for a human activity type recognition model. The method involves acquiring a training dataset during human activity, concatenating acceleration information from this dataset to obtain sample data from multiple sensors within the same time period, normalizing each sample data to obtain several input samples, and training the human activity type recognition model based on these input samples until the cross-entropy loss function converges. This application enables the concatenated data from multiple sensors to maintain relative consistency in the time dimension, and by using the input samples to train the human activity type recognition model, it can improve the accuracy of the model in identifying human activity types.

Description

Technical Field

[0001] This application relates to the field of information processing technology, and in particular to a training method and related apparatus for a human activity type recognition model. Background Technology

[0002] With the continuous advancement of building a strong sports nation and the ongoing promotion of sports health monitoring services, people are increasingly inclined to use smart terminal devices to record their physical activity and obtain corresponding quantitative analysis results. In other words, they evaluate sports health by identifying the types of human activities and statistically analyzing indicators such as duration.

[0003] Currently, the following method is commonly used to identify the type of human activity: Acceleration information during human activity is collected. The acceleration information can include acceleration data, angular velocity data, or gravitational acceleration data in three directions of the spatial coordinate system. Then, based on the acceleration data in different time periods, data feature information is calculated, including the variance and standard deviation of the acceleration data. Finally, the acceleration information and the corresponding data feature information are used as input to a neural network model to identify the type of human activity.

[0004] In the process of developing this application, the inventors discovered at least the following problems in the prior art:

[0005] Directly using acceleration information collected from a single device as input to the neural network model, without splicing acceleration information collected from different devices, affects the model's recognition performance. Summary of the Invention

[0006] The embodiments of this application aim to provide a training method and related apparatus for a human activity type recognition model, so as to improve the accuracy of the human activity type recognition model in recognizing human activity types.

[0007] The embodiments of this application provide the following technical solutions:

[0008] In a first aspect, embodiments of this application provide a training method for a human activity type recognition model, the training method comprising:

[0009] A training dataset is obtained during human activity. The training dataset includes an acceleration dataset consisting of multiple acceleration information and human activity type labels corresponding to the acceleration dataset. The acceleration information is collected by each sensor. The acceleration dataset is a collection of acceleration information collected by multiple sensors within the same preset collection time period.

[0010] Acceleration information is stitched together to obtain sample data from multiple sensors within the same time period;

[0011] Normalize each sample data to obtain several input samples;

[0012] Based on the human activity type recognition model, feature extraction is performed on the input samples to obtain a multi-class probability array corresponding to each input sample. The human activity type corresponding to the largest probability value in the multi-class probability array is the human activity type corresponding to the input sample.

[0013] Construct a cross-entropy loss function and train the human activity type recognition model based on the input samples until the cross-entropy loss function converges.

[0014] Secondly, embodiments of this application provide a training device for a human activity type recognition model. This training device is applied to an electronic device, which includes multiple sensors for collecting acceleration information. The training device includes:

[0015] The acquisition unit is used to acquire a training dataset during human activity. The training dataset includes an acceleration dataset consisting of multiple acceleration information and human activity type labels corresponding to the acceleration dataset. The acceleration information is collected by each sensor. The acceleration dataset is a collection of acceleration information collected by multiple sensors within the same preset acquisition time period.

[0016] The data stitching unit is used to stitch together acceleration information to obtain sample data from multiple sensors within the same time period.

[0017] The normalization unit is used to normalize each sample data to obtain several input samples;

[0018] The feature extraction unit is used to extract features from the input sample based on the human activity type recognition model to obtain a multi-class probability array corresponding to each input sample. The human activity type corresponding to the largest probability value in the multi-class probability array is the human activity type corresponding to the input sample.

[0019] The training unit is used to construct the cross-entropy loss function and train the human activity type recognition model based on the input samples until the cross-entropy loss function converges.

[0020] Thirdly, embodiments of this application provide a method for identifying types of human activity, including:

[0021] Acquire target human activity data;

[0022] The target human activity data is input into the human activity type recognition model to obtain the human activity type corresponding to the target human activity data. The human activity type recognition model is trained based on the training method of the human activity type recognition model in the first aspect.

[0023] Fourthly, embodiments of this application provide a device for identifying human activity types, including:

[0024] The data acquisition unit is used to acquire target human activity data.

[0025] The human activity type recognition unit is used to input target human activity data into the human activity type recognition model to obtain the human activity type corresponding to the target human activity data. The human activity type recognition model is trained based on a third-party human activity type recognition method.

[0026] Fifthly, embodiments of this application provide an electronic device, including:

[0027] At least one processor, and

[0028] A memory that is communicatively connected to at least one processor, wherein,

[0029] The memory stores instructions that can be executed by at least one processor, such that the at least one processor can execute either a training method for a human activity type recognition model of the first aspect or a human activity type recognition method of the third aspect.

[0030] In a sixth aspect, embodiments of this application provide a non-volatile computer-readable storage medium storing computer-executable instructions for causing an electronic device to execute a training method for a human activity type recognition model in the first aspect or a human activity type recognition method in the third aspect.

[0031] The beneficial effects of this application's embodiments are as follows: Unlike existing technologies, this application provides a training method for a human activity type recognition model. This method includes: acquiring a training dataset during human activity, wherein the training dataset includes an acceleration dataset composed of multiple acceleration information points and corresponding human activity type labels; the acceleration information is collected by each sensor; the acceleration dataset is a collection of acceleration information collected by multiple sensors within the same preset acquisition time period; concatenating the acceleration information to obtain sample data corresponding to multiple sensors within the same time period; normalizing each sample data to obtain several input samples; extracting features from the input samples based on the human activity type recognition model to obtain a multi-class probability array corresponding to each input sample, wherein the human activity type corresponding to the largest probability value in the multi-class probability array is the human activity type corresponding to the input sample; constructing a cross-entropy loss function, and training the human activity type recognition model based on the input samples until the cross-entropy loss function converges.

[0032] By acquiring a training dataset during human activity and stitching together the acceleration information in the training dataset, sample data from multiple sensors within the same time period are obtained. This application enables the data from multiple sensors to maintain relative consistency in the time dimension after stitching together. Furthermore, by using the input samples to train the human activity type recognition model, the accuracy of the human activity type recognition model in recognizing human activity types can be improved. Attached Figure Description

[0033] One or more embodiments are illustrated by way of example with reference numerals in the accompanying drawings. These illustrations do not constitute a limitation on the embodiments. Elements with the same reference numerals in the drawings are denoted as similar elements. Unless otherwise stated, the figures in the drawings are not to be limited by scale.

[0034] Figure 1 This is a schematic diagram of an application environment provided in an embodiment of this application;

[0035] Figure 2 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application;

[0036] Figure 3 This is a flowchart illustrating a training method for a human activity type recognition model provided in an embodiment of this application.

[0037] Figure 4 yes Figure 3 A detailed flowchart of step S302 in the process;

[0038] Figure 5 yes Figure 4 A detailed flowchart of step S321 in the process;

[0039] Figure 6 This is a detailed flowchart illustrating how, based on a minimum value and a position index, a row number in a second data file matches a selected row in a first data file.

[0040] Figure 7 yes Figure 4 A detailed flowchart of step S322 in the process;

[0041] Figure 8 yes Figure 4 A detailed flowchart of step S323 in the process;

[0042] Figure 9 This is a schematic diagram of the structure of a deep backbone network module provided in an embodiment of this application;

[0043] Figure 10 This is a schematic diagram of the structure of a first residual network module provided in an embodiment of this application;

[0044] Figure 11 This is a schematic diagram of the structure of a second residual network module provided in an embodiment of this application;

[0045] Figure 12 yes Figure 3 A detailed flowchart of step S305 in the process;

[0046] Figure 13 This is a flowchart illustrating a method for identifying human activity types provided in an embodiment of this application;

[0047] Figure 14 This is a schematic diagram of the structure of a training device for a human activity type recognition model provided in an embodiment of this application;

[0048] Figure 15 This is a schematic diagram of the structure of a human activity type recognition device provided in an embodiment of this application. Detailed Implementation

[0049] The present application will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present application, but do not limit the present application in any way. It should be noted that those skilled in the art can make several modifications and improvements without departing from the concept of the present application. These all fall within the protection scope of the present application.

[0050] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0051] It should be noted that, unless there is a conflict, the various features in the embodiments of this application can be combined with each other, all of which are within the protection scope of this application. Furthermore, although functional modules are divided in the device schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a different order than the module division in the device or the order in the flowchart. In addition, the terms "first," "second," and "third" used herein do not limit the data or execution order, but only distinguish identical or similar items with essentially the same function and effect.

[0052] Unless otherwise defined, all technical and scientific terms used in this specification have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the scope of this application. The term "and / or" as used in this specification includes any and all combinations of one or more of the associated listed items.

[0053] Furthermore, the technical features involved in the various embodiments of this application described below can be combined with each other as long as they do not conflict with each other.

[0054] Before introducing the embodiments of this application, a brief introduction will be given to the human activity type identification method known to the inventors of this application, so as to facilitate the understanding of the embodiments of this application later.

[0055] In some methods for identifying human activity types, acceleration information during human activity is collected using sensors such as gyroscopes and accelerometers. This acceleration information can include acceleration data, angular velocity data, and gravitational acceleration data in three directions of the spatial coordinate system. Then, based on the acceleration data over different time periods, data feature information is calculated, including the variance and standard deviation of the acceleration data. Finally, the acceleration information and the corresponding data feature information are used as input to a neural network model to identify the type of human activity.

[0056] However, traditional technologies have the following drawbacks:

[0057] Traditional technical solutions directly use acceleration information collected by a single device as input to a neural network model without splicing acceleration information collected by different devices. This results in a large error in the model's recognition of human activity types. For example, if a single device is worn on the subject's left foot, but the subject's left foot has a smaller range of motion while the right foot has a larger range of motion and more obvious features during certain movements, this will lead to an increase in the model's error in recognizing human activity types.

[0058] Based on this, in order to improve the accuracy of the human activity type recognition model in recognizing human activity types, this application provides a training method for the human activity type recognition model.

[0059] In the embodiments of this application, the training method of the human activity type recognition model and the human activity type recognition method can be executed by an electronic device with computing power. The following describes an exemplary application of the electronic device provided in the embodiments of this application for training the human activity type recognition model or for recognizing human activity types. It is understood that the electronic device can both train the human activity type recognition model and use the human activity type recognition model to recognize human activity types.

[0060] In this embodiment, the electronic device can be a server, such as a server deployed in the cloud. When the server is used to train a human activity type recognition model, it constructs the model based on training datasets of human activity processes provided by other devices or those skilled in the art, and iteratively trains the model based on input samples until the cross-entropy loss function converges to determine the final model parameters. When the server is used to identify human activity types, it calls the built-in human activity type recognition model to process the target human activity data provided by other devices or users to obtain the human activity type corresponding to the target human activity data.

[0061] In this embodiment, the electronic device can also be various types of terminals such as laptops, desktop computers, or mobile devices. When the terminal is used to train a human activity type recognition model, the sensor uploads the collected acceleration information to the terminal. Those skilled in the art design a human activity type recognition model on the terminal and construct a cross-entropy loss function, enabling the terminal to iteratively train the human activity type recognition model based on input samples until the cross-entropy loss function converges to determine the final model parameters. When the terminal is used to identify human activity types, it calls the built-in human activity type recognition model to process the target human activity data uploaded by the sensor, obtaining the human activity type corresponding to the target human activity data; or, it synchronously uploads the target human activity data uploaded by the sensor to a big data platform, enabling the big data platform to call the built-in human activity type recognition model to process the target human activity data uploaded by the terminal, obtaining the human activity type corresponding to the target human activity data, and then sending the human activity type to the terminal.

[0062] Before providing a detailed description of this application, the nouns and terms used in the embodiments of this application are explained, and the nouns and terms used in the embodiments of this application shall be interpreted as follows:

[0063] (1) Human activity type identification: This is a time series classification task that automatically identifies the types of human behavior activities through a computer system. It aims to use artificial intelligence models to reason and analyze the key features in the process of human activity, and then determine the type of human activity. Specific types include standing, walking, running, skipping rope, going up stairs, going down stairs, riding a bicycle, etc.

[0064] (2) Neural networks, also known simply as neural networks (NNs) or connection models, are mathematical models of algorithms that mimic the behavioral characteristics of animal neural networks to perform distributed parallel information processing. Specifically, a neural network can be composed of neural units, which can be understood as a neural network with input layers, hidden layers, and output layers. Generally, the first layer is the input layer, the last layer is the output layer, and the layers in between are hidden layers. Among them, a neural network with many hidden layers is called a deep neural network (DNN). The work of each layer in a neural network can be described by the mathematical expression y = a((W·x+b)). From a physical perspective, the work of each layer in a neural network can be understood as completing the transformation from the input space to the output space (i.e., from the row space to the column space of a matrix) through five operations on the input space (the set of input vectors). These five operations include: 1. Dimensional increase / decrease; 2. Magnification / reduction; 3. Rotation; 4. Translation; 5. "Bending". Operations 1, 2, and 3 are performed by "W·x", operation 4 by "+b", and operation 5 by "a()". The term "space" is used here because the objects being classified are not individual things, but a class of things; space refers to the set of all individuals within that class. W is the weight matrix of each layer in the neural network, where each value represents the weight of a neuron in that layer. This matrix W determines the spatial transformation from the input space to the output space, meaning that the W of each layer in the neural network controls how the space is transformed. The purpose of training the neural network is to ultimately obtain the weight matrices of all layers in the trained neural network. Therefore, the training process of a neural network is essentially learning how to control spatial transformation, more specifically, learning the weight matrix.

[0065] It should be noted that in the embodiments of this application, the models used for machine learning tasks are essentially neural networks. Common components in neural networks include convolutional layers, activation function layers, pooling layers, and fully connected layers. By assembling these common components in neural networks, a model is designed. When the model parameters (weight matrices of each layer) are determined such that the model error meets a preset condition or the number of model parameters is adjusted to reach a preset threshold, the model converges.

[0066] The convolutional layer is configured with multiple convolutional kernels, each with a corresponding stride, to perform convolution operations on the image. The purpose of convolution operations is to extract different features from the input image.

[0067] Activation function layers are used to allow each neuron in a neural network to accept the output value of the previous layer as its input value and pass the processing result to the next layer. Commonly used activation functions include, but are not limited to, the Rectified Linear Unit (ReLU) function, the Softmax function, and the Parametric Rectified Linear Unit (PReLU) function.

[0068] Pooling layers mimic the human visual system, allowing for dimensionality reduction or representing images with higher-level features. Common pooling operations include max pooling, mean pooling, random pooling, median pooling, and combined pooling. Typically, pooling layers are periodically inserted between convolutional layers in neural networks to achieve dimensionality reduction.

[0069] A fully connected layer (FC) is a layer in which each node is connected to all nodes in the previous layer. It is used to combine the features extracted from the previous layers, that is, to map the learned feature representations to the label space of the samples. It plays the role of a "classifier" in the entire convolutional neural network.

[0070] The technical solution of this application will be described in detail below with reference to the accompanying drawings.

[0071] Please see Figure 1 , Figure 1 This is a schematic diagram of an application environment provided in an embodiment of this application;

[0072] like Figure 1 As shown, the application environment 100 includes: a sensor 10, an electronic device 20, and a big data platform 30. The sensor 10 is connected to the electronic device 20 via network communication, and the electronic device 20 is connected to the big data platform 30 via network communication. This network includes wired and / or wireless networks. It is understood that the network includes wireless networks such as 2G, 3G, 4G, 5G, Wi-Fi, and Bluetooth, and may also include wired networks such as serial cables and Ethernet cables.

[0073] In this embodiment, the sensor 10 is communicatively connected to the electronic device 20, for example, via a Bluetooth wireless network. The sensor 10 collects acceleration information during human activity and transmits it to the electronic device 20 via Bluetooth. The sensor includes an accelerometer, such as a triaxial accelerometer. Multiple sensors can be used, and they are arranged in pairs on the human body. Specifically, at least two sensors are symmetrically worn on different parts of the same person's body, and at least two sensors face the same direction. For example, pairs of sensors are worn on the uppers of the left and right shoes, respectively, with both pairs facing the same direction. The sensors in this embodiment include, but are not limited to, inertial sensors.

[0074] In this embodiment, the electronic device 20 is communicatively connected to the sensor 10 and the big data platform 30. It receives acceleration information sent by the sensor 10 and trains a human activity type recognition model based on an acceleration dataset composed of multiple acceleration information points and corresponding human activity type labels. Alternatively, it acquires target human activity data, inputs the target human activity data into the human activity type recognition model, obtains the human activity type corresponding to the human activity data, and displays the human activity type on its own display interface to inform the user. The electronic device in this embodiment includes a terminal or a server. The terminal includes, but is not limited to, various terminals with computing capabilities such as smart bracelets, laptops, desktop computers, or mobile devices. The server includes, but is not limited to, tower servers, rack servers, blade servers, and cloud servers.

[0075] In some embodiments, the electronic device 20 can also be used to receive acceleration information sent by multiple sensors 10 and synchronously transmit multiple acceleration information to the big data platform 30, or receive the human activity type corresponding to the target human activity data identified by the human activity type recognition model sent by the big data platform 30.

[0076] Accordingly, the big data platform 30, connected to the electronic device 20, can be used to train a human activity type recognition model, or to acquire target human activity data collected by the sensor 10 and sent by the electronic device 20, and to call the built-in human activity type recognition model to process the acquired target human activity data to obtain the human activity type corresponding to the target human activity data, and then send the human activity type to the electronic device 20. The big data platform 30 includes several servers. The servers in this embodiment include, but are not limited to: tower servers, rack servers, blade servers, and cloud servers. Preferably, the server is a cloud server (Elastic Compute Service, ECS).

[0077] It is understood that, in some embodiments, the electronic device 20 may also be used to display the type of human activity on its own display interface after receiving the type of human activity sent by the big data platform 30, so as to inform the user.

[0078] Example 1

[0079] Please see Figure 2 , Figure 2 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application;

[0080] like Figure 2 As shown, the electronic device 20 includes one or more processors 21 and a memory 22. Wherein, Figure 2 Take a processor 21 as an example.

[0081] Processor 21 and memory 22 can be connected via a bus or other means. Figure 2 Taking the example of a connection between China and Israel via a bus.

[0082] The processor 21 is configured to provide computing and control capabilities to control the electronic device 20 to perform corresponding tasks, such as controlling the electronic device 20 to perform the training method of the human activity type recognition model or the human activity type recognition method in any of the following method embodiments.

[0083] Processor 21 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), a hardware chip, or any combination thereof; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The aforementioned PLD can be a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0084] The memory 22, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs, non-transitory computer-executable programs, and modules, such as the program instructions / modules corresponding to the training method of the human activity type recognition model or the human activity type recognition method in the embodiments of this application. The processor 21 can implement the training method of the human activity type recognition model or the human activity type recognition method in any of the following method embodiments by running the non-transitory software programs, instructions, and modules stored in the memory 22. Specifically, the memory 22 may include volatile memory (VM), such as random access memory (RAM); the memory 22 may also include non-volatile memory (NVM), such as read-only memory (ROM), flash memory, hard disk drive (HDD), solid-state drive (SSD), or other non-transitory solid-state storage devices; the memory 22 may also include combinations of the above types of memory.

[0085] Memory 22 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some embodiments, memory 22 may optionally include memory remotely located relative to processor 21, which can be connected to processor 21 via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0086] One or more modules are stored in memory 22. When executed by one or more processors 21, they perform the training method of the human activity type recognition model or the human activity type recognition method in any of the following method embodiments. For example, they perform the following descriptions. Figure 3 The steps shown, or perform the following descriptions. Figure 13 The steps shown.

[0087] In this embodiment, the electronic device 20 may also have wired or wireless network interfaces, input / output interfaces, and other components for input and output. The electronic device 20 may also include other components for implementing device functions, which will not be elaborated here.

[0088] The following describes the training method of the human activity type recognition model provided in the embodiments of this application.

[0089] Please see Figure 3 , Figure 3This is a flowchart illustrating a training method for a human activity type recognition model provided in an embodiment of this application.

[0090] The training method for the human activity type recognition model is applied to an electronic device. Specifically, the execution subject of the training method for the human activity type recognition model is one or at least two processors in the electronic device.

[0091] like Figure 3 As shown, the training method for this human activity type recognition model includes:

[0092] Step S301: Obtain the training dataset during human activity;

[0093] Specifically, the training dataset includes an acceleration dataset composed of multiple acceleration information points and corresponding human activity type labels. The acceleration information is collected by each sensor, and the acceleration dataset is a collection of acceleration information collected by multiple sensors within the same preset acquisition time period. The acceleration dataset includes acceleration information corresponding to various human activity types, with each type of human activity corresponding to multiple pairs of acceleration information points. The number of acceleration information points included in the acceleration dataset and the preset acquisition time period can be determined by those skilled in the art based on the actual situation.

[0094] In this process, during the acquisition of training datasets by electronic devices, a single person can wear multiple sensors. Each pair of acceleration information includes motion data collected by two sensors within the same preset acquisition time period. At least two sensors are symmetrically worn on different parts of the same person's body, and at least two sensors are oriented in the same direction. For example, a pair of sensors consists of two inertial sensors equipped with triaxial accelerometers. The two inertial sensors are respectively positioned above the uppers of the left and right shoes, and their orientations are consistent to maintain the alignment of their corresponding coordinate systems. The triaxial accelerometer is used to measure the acceleration data of the human body in three-dimensional space, such as measuring the three-dimensional acceleration data of the human body in the x, y, and z axes indicated by the inertial sensors. The data sampling frequency of each pair of sensors is the same, for example, 5–100 Hz, meaning 5–100 frames of data can be acquired per second. Each frame of data contains the acceleration of the foot wearing the inertial sensor in the x, y, and z axes in three-dimensional space at a certain moment.

[0095] Each electronic device also includes a Bluetooth module, which is used to establish a data upload channel between the inertial sensor and the electronic device. The Bluetooth communication range between the inertial sensor and the electronic device is a circular area, for example, a circular area with a radius of approximately 20 meters. Understandably, if the Bluetooth connection between the inertial sensor and the electronic device is lost, the inertial sensor will locally cache the collected data. When the inertial sensor re-enters the Bluetooth connection range of the electronic device, it will upload the data to the electronic device.

[0096] In this embodiment, the electronic device includes a start button, which instructs the electronic device to search for and bind the corresponding inertial sensors. After two inertial sensors equipped with triaxial accelerometers are worn on the uppers of the left and right shoes of a person, respectively, and facing the same direction, the electronic device searches for and binds the two inertial sensors via Bluetooth in response to the start button's activation. It is understood that the electronic device has an application app or host computer, and the device can bind and configure parameters within the application app or host computer. For example, the application app or host computer can set the frequency (i.e., data sampling frequency) and data acquisition method for the sensors, including automatic and manual acquisition. When the data acquisition method is manual, triggering the start and end buttons on the electronic device's display interface allows the sensor to collect data for a certain period and upload it to the electronic device. When the data acquisition method is automatic, the sensor automatically collects and caches data, and uploads the real-time collected data to the electronic device.

[0097] Furthermore, after the parameters are configured, data acquisition can be performed. The following describes the specific method of obtaining the training dataset during human activity, using manual acquisition as an example: Select a number of male and female participants (more than the preset number), and collect data on each person's activities during different types of human activity using electronic devices and sensors to obtain the training dataset during human activity.

[0098] For example, data was collected from more than 15 male participants and more than 15 female participants on various human activity types, including standing, walking, running, skipping rope, climbing stairs, descending stairs, and cycling. Each participant completed the data collection for each activity type sequentially, with a total individual collection time exceeding 3 minutes for each activity type. During the data collection for each activity type, the participant manually triggered the start and end buttons on the electronic device's app or host computer at the beginning and end of the activity. After each collection session, triggering the button generated a data file containing a corresponding human activity type label. This data file included multiple acceleration data points from multiple sensors collected simultaneously from the same person for a specific human activity type.

[0099] Furthermore, multiple data files corresponding to the same type of human activity can be combined to form an acceleration dataset, and multiple acceleration datasets can form a training dataset.

[0100] It is understood that human activity types include various human behavioral activities, such as standing, walking, running, skipping rope, climbing stairs, descending stairs, and cycling. Those skilled in the art can determine the types of human activities that can be included in the human activity type dataset based on actual needs, for example, including human activity type data corresponding to 10 different human activity types. Thus, the trained human activity type recognition model can identify these 10 human activity types.

[0101] Compared to existing solutions that use RGB cameras to capture video streams of human movement in fixed scenes, which have limited applicability to outdoor scenes due to the limited detection area, this application uses triaxial acceleration collected by left and right foot sensors, making it more applicable to both indoor and outdoor scenes.

[0102] Step S302: Perform data stitching on the acceleration information to obtain sample data corresponding to multiple sensors within the same time period;

[0103] Specifically, at least two acceleration information items stored in each data file of the training dataset are concatenated to obtain sample data corresponding to multiple sensors within the same time period. Each sample data item includes at least two acceleration information items after horizontal concatenation.

[0104] Understandably, since data collected by multiple inertial sensors worn by the same person is uploaded to the host computer of the same electronic device, the acceleration information collected by different inertial sensors will be mixed and stored in the same data file. A data file consists of several rows and columns of data, with each column representing: row number, data generation time, sensor device identification number, X-axis acceleration, Y-axis acceleration, and Z-axis acceleration.

[0105] Furthermore, due to the inherent differences in sensor hardware and the issue of data frame loss during data upload, the data lengths of acceleration information collected by different sensors within the same time period are inconsistent. Therefore, it is necessary to stitch the acceleration information together to obtain sample data corresponding to multiple sensors within the same time period. For example, the data dimensions of the sample data include: data generation time, left foot sensor X-axis acceleration, left foot sensor Y-axis acceleration, left foot sensor Z-axis acceleration, right foot sensor X-axis acceleration, right foot sensor Y-axis acceleration, and right foot sensor Z-axis acceleration.

[0106] Please see Figure 4 , Figure 4 yes Figure 3 A detailed flowchart of step S302 in the process;

[0107] like Figure 4 As shown, step S302: Acceleration information is stitched together to obtain sample data from multiple sensors within the same time period, including:

[0108] Step S321: Perform horizontal stitching on each pair of acceleration information to obtain stitched data;

[0109] Specifically, each pair of acceleration information in each data file is horizontally stitched together to obtain one-to-one stitched data. The stitched data includes several rows and columns of data. Each row of data includes a combination of motion data collected by two sensors at the same time point, i.e., a combination of acceleration data. Each column of data includes several pieces of information belonging to the same data dimension. For example, a column of data may contain information corresponding to the data row number, or a column of data may contain acceleration information corresponding to the left foot X-axis acceleration.

[0110] Please see Figure 5 , Figure 5 yes Figure 4 A detailed flowchart of step S321 in the process;

[0111] like Figure 5 As shown, step S321: Each pair of acceleration information is horizontally stitched together to obtain stitched data, including:

[0112] Step S3201: Obtain the raw mixed data file;

[0113] Specifically, the raw mixed data file is the data file mentioned above. Because this data file contains unprocessed data from two sensors, it is also called the raw mixed data file. Each pair of acceleration information corresponds to one raw mixed data file.

[0114] Please refer to Table 1, which is a schematic diagram of acceleration information in a raw mixed data file provided in an embodiment of this application;

[0115] Table 1

[0116]

[0117] As shown in Table 1, a raw mixed data file includes acceleration information from two sensors. The data collected by the two inertial sensors are arranged vertically in a mixed manner, and each data dimension includes a timestamp (also known as the data generation time), sensor device identification number, X-axis acceleration, Y-axis acceleration, and Z-axis acceleration. Among them, WT901 BLE01 (D03E7DA483B0) is the left foot sensor, and WT901 BLE02 (D03E7DA48C29) is the right foot sensor.

[0118] Step S3202: Determine whether the data lengths of the motion data collected by the two sensors in the original mixed data file are the same;

[0119] Specifically, since the X-axis acceleration, Y-axis acceleration, and Z-axis acceleration data of the same sensor have the same length, it is possible to determine whether the motion data collected by two sensors are the same by comparing the X-axis acceleration data, Y-axis acceleration data, or Z-axis acceleration data collected by two sensors in the same time period in a raw mixed data file.

[0120] Further, if the motion data collected by the two sensors in the original mixed data file has the same data length, proceed to step S3203; if the motion data collected by the two sensors in the original mixed data file has different data lengths, proceed to step S3204.

[0121] Step S3203: Combine two acceleration data points with similar timestamps from different sensors into the same row to obtain the combined data;

[0122] Specifically, the acceleration data from two different sensors with the closest timestamps are concatenated into the same row. For example, each row of concatenated data includes the left foot sensor's X-axis acceleration, left foot sensor's Y-axis acceleration, left foot sensor's Z-axis acceleration, right foot sensor's X-axis acceleration, right foot sensor's Y-axis acceleration, and right foot sensor's Z-axis acceleration.

[0123] Step S3204: Determine the sensor corresponding to the motion data with a larger data length as the first sensor, and the sensor corresponding to the motion data with a smaller data length as the second sensor, and store the motion data of the second sensor in the original mixed data file into the first data file;

[0124] Specifically, if the motion data collected by the two sensors in the original mixed data file has different data lengths, the sensor corresponding to the motion data with the larger data length is determined to be the first sensor, and the sensor corresponding to the motion data with the smaller data length is determined to be the second sensor. The motion data of the second sensor in the original mixed data file is then stored in the first data file.

[0125] Step S3205: Traverse each line in the first data file and obtain the mixed data in the original mixed data file according to the line number of each line of data in the original mixed data file;

[0126] Specifically, the i-th row of the first data file is selected for traversal. Based on the row number (denoted as j) of the i-th row in the original mixed data file, rows j+1 to j+m of the data are obtained from the original mixed data file. Here, rows j+1 to j+m are mixed data, m is a positive integer and m≥2. m can be set by those skilled in the art according to the actual situation. In this embodiment, there is no restriction. Optionally, m is 20.

[0127] Step S3206: Obtain the motion data collected by the first sensor in the mixed data, and delete the data that has been successfully matched by the second sensor in the motion data collected by the first sensor to obtain the second data file;

[0128] Specifically, based on the sensor device identification number of the first sensor, motion data collected by the first sensor is obtained from the mixed data. Motion data collected by the first sensor that was successfully matched with the second sensor in step S3209 is deleted from the mixed data. The resulting data is then saved to a second data file. The second data file includes motion data collected by the first sensor that was not successfully matched with the second sensor from the mixed data. It can be understood that each time a line in the first data file is traversed, m lines of mixed data are obtained, along with a corresponding second data file.

[0129] Step S3207: Determine whether the data length in the second data file is zero;

[0130] Specifically, if the data length in the second data file is zero, it indicates that the i-th row of data in the first data file corresponding to the second sensor has not been matched with a row of data with the same or closest timestamp from the first sensor data, and the process returns to step S3205 to traverse the next row in the first data file; if the data length in the second data file is not zero, it indicates that a row of data with the closest timestamp to the i-th row of data in the first data file can be found in the second data file, and the process proceeds to step S3208.

[0131] Step S3208: Calculate the absolute value of the difference between the timestamp corresponding to each row of data in the second data file and the timestamp corresponding to the selected row of data in the first data file in turn, to obtain an absolute time difference sequence, and determine the minimum value and the corresponding position index in the absolute time difference sequence;

[0132] Specifically, if the length of the data in the second data file is not zero, the absolute value of the difference between the timestamp corresponding to each row of data in the second data file and the timestamp corresponding to the selected row of data (the i-th row of data) in the first data file is calculated sequentially to obtain an absolute time difference sequence (denoted as T = [t_1, t_2, ...]). The minimum value in the absolute time difference sequence and its corresponding position index are then determined. Here, t_1 is the absolute value of the difference between the timestamp corresponding to the first row of data in the second data file and the timestamp corresponding to the i-th row of data in the first data file, t_2 is the absolute value of the difference between the timestamp corresponding to the second row of data in the second data file and the timestamp corresponding to the i-th row of data in the first data file, and the position index is the row number of the row of data in the second data file corresponding to the minimum value in the absolute time difference sequence. For example, if the position index is k, it means the k-th row.

[0133] Step S3209: Determine the row number in the second data file that matches the selected row data in the first data file based on the minimum value and the position index;

[0134] Specifically, the row number in the second data file that matches the selected row data in the first data file, that is, the row number in the second data file corresponding to the timestamp of the i-th row data in the first data file that is closest to the timestamp of the i-th row data in the first data file.

[0135] Please see Figure 6 , Figure 6 This is a detailed flowchart illustrating how, based on a minimum value and a position index, a row number in a second data file matches a selected row in a first data file.

[0136] like Figure 6As shown, the process of determining the row number in the second data file that matches the selected row data in the first data file based on the minimum value and the position index includes:

[0137] Step S601: Obtain the minimum value and its position index;

[0138] Specifically, the minimum value and position index are obtained according to step S3208.

[0139] Step S602: Determine whether the minimum value is greater than the first threshold;

[0140] The first threshold is used to determine whether the i-th row of data in the first data file matches the row of data with the same or closest timestamp in the second data file, that is, to determine whether it is necessary to traverse the next row of data in the first data file. The first threshold can be set by those skilled in the art according to the actual situation, and is not limited in this embodiment. Optionally, the first threshold is 100 milliseconds.

[0141] Specifically, if the minimum value is greater than the first threshold, proceed to step S603; if the minimum value is less than or equal to the first threshold, proceed to step S604.

[0142] Step S603: Traverse the next line of data in the first data file;

[0143] For example: iterate through the (i+1)th row of data in the first data file.

[0144] Step S604: Determine whether the position index is greater than 1, the minimum value is greater than the second threshold, and the first value in the absolute time difference sequence is less than the third threshold;

[0145] The second threshold and the third threshold are used to determine whether the row in the second data file that matches the selected row data in the first data file is the first row. The second threshold is less than the first threshold. The second threshold and the third threshold can be set by those skilled in the art according to the actual situation. In this embodiment, there is no limitation. Optionally, the second threshold is 10 milliseconds and the third threshold is 20 milliseconds.

[0146] Specifically, if the position index is greater than 1, the minimum value is greater than the second threshold, and the first value in the absolute time difference sequence is less than the third threshold, then proceed to step S605; if the position index is less than or equal to 1, or the minimum value is less than or equal to the second threshold, or the first value in the absolute time difference sequence is greater than or equal to the third threshold, then proceed to step S606.

[0147] Step S605: Determine the first row as the row in the second data file that matches the selected row data in the first data file;

[0148] Specifically, if the minimum value is less than or equal to the first threshold, the position index is greater than 1, the minimum value is greater than the second threshold, and the first value in the absolute time difference sequence is less than the third threshold, then the row in the second data file that matches the selected row data in the first data file is determined to be the first row, and the row number corresponding to the first row data in the second data file in the original mixed data file is added to the set of successfully matched row numbers. The set of successfully matched row numbers is used to store the row numbers of the motion data collected by the first sensor that is successfully matched by the second sensor, so as to obtain the second data file in step S3206.

[0149] Step S606: Determine whether the position index is greater than 1 and whether the difference between two adjacent values ​​in the absolute time difference sequence is less than the fourth threshold;

[0150] Wherein, two adjacent values ​​are the (k-1)th value and the kth value, and the difference between two adjacent values ​​is the kth value minus the (k-1)th value. The fourth threshold is used to determine whether the row in the second data file that matches the selected row data in the first data file is the (k-1)th row. The fourth threshold can be set by those skilled in the art according to the actual situation, and is not limited in this embodiment. Optionally, the fourth threshold is 10 milliseconds.

[0151] Specifically, if the position index is greater than 1 and the difference between two adjacent values ​​in the absolute time difference sequence is less than the fourth threshold, then proceed to step S607; if the position index is less than or equal to 1, or the difference between two adjacent values ​​in the absolute time difference sequence is greater than or equal to the fourth threshold, then proceed to step S608.

[0152] Step S607: Determine the (k-1)th row in the second data file that matches the selected row data in the first data file;

[0153] Specifically, if the minimum value is less than or equal to the second threshold, or the first value in the absolute time difference sequence is greater than or equal to the third threshold, then when the position index is greater than 1 and the difference between two adjacent values ​​in the absolute time difference sequence is less than the fourth threshold, the row in the second data file that matches the selected row data in the first data file is determined to be the (k-1)th row, and the row number corresponding to the (k-1)th row data in the second data file is added to the set of successfully matched row numbers in the original mixed data file.

[0154] Step S608: Determine the k-th row in the second data file that matches the selected row data in the first data file.

[0155] Specifically, if the position index is less than or equal to 1, or the difference between two adjacent values ​​in the absolute time difference sequence is greater than or equal to the fourth threshold, then the row in the second data file that matches the selected row data in the first data file is determined to be the k-th row, and the row number corresponding to the k-th row data in the second data file is added to the set of successfully matched row numbers in the original mixed data file.

[0156] Step S3210: Obtain triaxial acceleration data from the selected row data in the first data file and the row data corresponding to the row number in the second data file respectively, and stitch them together to obtain a stitched row data;

[0157] Specifically, the triaxial acceleration data includes X-axis acceleration, Y-axis acceleration, and Z-axis acceleration. These are obtained from the i-th row of data in the first data file and the row of data in the second data file that matches the i-th row of data in the first data file. Based on the predefined correspondence between sensor device numbers and sensors worn on different body parts, the corresponding sensor in the first and second data files is determined. Then, the triaxial acceleration data from the two sensors are concatenated in the order of continuous concatenation of triaxial acceleration data corresponding to the same sensor to obtain a concatenated row of data. This concatenated row of data is then stored in the third data file.

[0158] For example: Based on the predefined correspondence between sensor device numbers and left and right foot sensors, determine which foot sensor the triaxial acceleration data in the first and second data files corresponds to, i.e., whether it corresponds to the left or right foot sensor. Then, concatenate the data in the following order: left foot sensor X-axis acceleration, left foot sensor Y-axis acceleration, left foot sensor Z-axis acceleration, right foot sensor X-axis acceleration, right foot sensor Y-axis acceleration, and right foot sensor Z-axis acceleration to obtain a concatenated row of data. Store this concatenated row of data in the third data file.

[0159] Furthermore, each line in the first data file is traversed to obtain a concatenated line of data. After traversing all lines in the first data file, several concatenated lines of data are obtained. These several concatenated lines of data form the concatenated data and are stored in the third data file. The third data file is used to store the concatenated data, and each original mixed data file corresponds to one concatenated data file and one third data file.

[0160] Please refer to Table 2, which is a schematic diagram of spliced ​​data in a third data file provided in an embodiment of this application;

[0161] Table 2

[0162]

[0163] As shown in Table 2, the data dimensions of the spliced ​​data include timestamp (also known as data generation time), left foot sensor X-axis acceleration, left foot sensor Y-axis acceleration, left foot sensor Z-axis acceleration, right foot sensor X-axis acceleration, right foot sensor Y-axis acceleration, and right foot sensor Z-axis acceleration. The data in the same row is a combination of acceleration data collected by two sensors at the same (same or closest) time point.

[0164] Step S3211: Determine whether all rows in the first data file have been traversed.

[0165] If all rows in the first data file have been traversed, the process ends; otherwise, return to step S3205 and traverse the next row in the first data file.

[0166] Furthermore, after completing step S321, that is, after horizontally stitching the acceleration information collected by two sensors in an original mixed data file to obtain stitched data, if the original mixed data file also includes acceleration information collected by other sensors, when the number of other sensors is even, step S321 is repeated to horizontally stitch the acceleration information collected by every two sensors in the original mixed data file that did not participate in the previous stitching process, to obtain several stitched data. Then, according to the timestamp in each stitched data, the acceleration data of each row with the closest timestamps in the several stitched data are stitched to the same row to obtain the final stitched data corresponding to the original mixed data file. Each row of data in the final stitched data corresponding to the original mixed data file is a combination of acceleration data collected by multiple sensors at the same (same or closest) time point.

[0167] If the original mixed data file also includes acceleration information collected by other sensors, when the number of other sensors is odd, then repeat step S321 to horizontally stitch together the acceleration information collected by every two sensors in the original mixed data file that did not participate in the previous stitching process, to obtain several stitched data, until only one sensor that did not participate in the previous stitching process remains in the original mixed data file to collect acceleration information. Then, horizontally stitch together the several stitched data with the acceleration information in the original mixed data file that did not participate in the previous stitching process to obtain the final stitched data corresponding to the original mixed data file.

[0168] For example, when the original mixed data file includes three acceleration information collected by three sensors, the two acceleration information collected by two sensors are first horizontally spliced ​​to obtain spliced ​​data. For example, the two acceleration information collected by two sensors in the original mixed data file are determined according to the sensor device identification number. The two acceleration information are horizontally spliced ​​through step S321 to obtain spliced ​​data, and the spliced ​​data is stored in the third data file.

[0169] Then, based on the sensor device identification number, the acceleration information corresponding to the two sensors that have undergone the stitching process in the original mixed data file is deleted, so that the original mixed data file only saves the acceleration information collected by one sensor.

[0170] Select a row of data in the original mixed data file, and calculate the absolute value of the difference between the timestamp corresponding to each row of data in the third data file and the timestamp corresponding to the selected row of data in the original mixed data file. Obtain an absolute time difference sequence and determine the minimum value and the corresponding position index in the absolute time difference sequence. Based on the minimum value and the position index, determine the row number in the third data file that matches the selected row of data in the original mixed data file. Obtain triaxial acceleration data from the selected row of data in the original mixed data file and concatenate the triaxial acceleration data to the end of the row of data corresponding to the row number in the third data file to obtain a concatenated row of data. The specific implementation of the above steps is similar to that of steps S3208-S3210, and will not be described in detail here.

[0171] Repeat the above steps until all rows of data in the original mixed data file are concatenated to the end of the corresponding rows of data in the third data file, thus obtaining the final concatenated data corresponding to the original mixed data file. Each row of data in the final concatenated data corresponding to the original mixed data file is a combination of acceleration data collected by the three sensors at the same (same or closest) time points.

[0172] If, after completing the horizontal stitching of acceleration information collected by at least two sensors in an original mixed data file to obtain stitched data, the original mixed data file does not include acceleration information collected by other sensors, then step S321 is repeated to perform horizontal stitching of acceleration information in another original mixed data file until each data file (original mixed data file) in the training dataset obtains its final corresponding stitched data.

[0173] In this embodiment, compared to existing solutions where data from two devices differ in timestamps and frame drop rates, directly splicing the data can lead to inconsistencies in the merged data, such as different timestamps and accelerations corresponding to the same row of data, ultimately affecting the model's recognition performance. This application obtains spliced ​​data by horizontally splicing each pair of acceleration information. This application can fuse gait data from both feet and ensure that the spliced ​​data from the two sensors maintains relative consistency in the time dimension, reducing the inherent differences in sensor hardware and resolving the issue of different frame drop rates during data upload. This allows for training the human activity type recognition model with more accurate and discriminative input samples, thereby improving the model's recognition accuracy.

[0174] Step S322: Fill in the blanks in the spliced ​​data to obtain the first data;

[0175] Specifically, each piece of data includes several rows and columns of data. In cases where the sampling frequencies of the two sensors are inconsistent, or the sensors malfunction during data acquisition, or the data is lost during the upload to the electronic device, the piece of data may contain null values.

[0176] If there are empty values ​​in the concatenated data, the empty values ​​in the concatenated data are filled to obtain the first data. The first data is the concatenated data after filling in the empty values, and each data in the first data is a non-empty value.

[0177] Please see Figure 7 , Figure 7 yes Figure 4 A detailed flowchart of step S322 in the process;

[0178] like Figure 7 As shown, step S322: Fill in the blanks in the concatenated data to obtain the first data, including:

[0179] Step S3221: Obtain the splicing data;

[0180] Step S3222: Determine whether there are any empty values ​​in each column of the concatenated data;

[0181] Specifically, if a column of the concatenated data contains empty values ​​(i.e., null values), proceed to step S3223; if no column of the concatenated data contains empty values, proceed to step S3225.

[0182] Step S3223: Determine the median value of the column data;

[0183] Specifically, if there are empty values ​​in a column of data, the median value of that column is determined. The median value is the median of all the values ​​in a column.

[0184] Step S3224: Replace the empty data with the median data to obtain the first data;

[0185] Specifically, if a column of data contains empty values, then those empty values ​​are replaced with the median value of that column. Step S3222 is repeated to check if the next column contains empty values, until all empty values ​​are replaced with the median value, resulting in the first set of data.

[0186] Step S3225: Determine the spliced ​​data as the first data.

[0187] Specifically, if there are no empty values ​​in any column of the concatenated data, then the concatenated data does not need to be filled with empty values ​​and can be directly identified as the first data.

[0188] Furthermore, after completing step S322 once, that is, obtaining the first data corresponding to one piece of spliced ​​data, step S322 is repeated to obtain the first data corresponding to another piece of spliced ​​data, until step S322 is performed once for each of the several pieces of spliced ​​data obtained in step S321 to obtain several pieces of first data.

[0189] Step S323: Perform outlier processing on the first data to obtain sample data from multiple sensors within the same time period.

[0190] Specifically, if there are abnormally high or low values ​​in a certain column of the first data, the data in that column is truncated according to the upper or lower threshold of that column to obtain sample data, and the sample data is stored for use in step S303.

[0191] Please see Figure 8 , Figure 8 yes Figure 4 A detailed flowchart of step S323 in the process;

[0192] like Figure 8 As shown, step S323: outlier processing is performed on the first data to obtain sample data from multiple sensors within the same time period, including:

[0193] Step S3231: Obtain the first data;

[0194] Step S3232: Determine whether there are any data in the column data of the first data that have a value greater than the upper limit threshold;

[0195] Specifically, each column of data corresponds to an upper threshold and a lower threshold. It is determined whether there are any data in each column of data in the first data that have a value greater than the upper threshold. The upper threshold and the lower threshold can be set by those skilled in the art according to actual needs, and are not limited in this embodiment.

[0196] Furthermore, if there is no data in a column of the first data that has a value greater than the upper limit threshold, then proceed to step S3233; if there is no data in a column of the first data that has a value greater than the upper limit threshold, then proceed to step S3234.

[0197] Step S3233: Replace the numerical values ​​of the data with the upper limit threshold;

[0198] Specifically, if any column in the first set of data contains a value greater than the upper threshold, then that value is replaced with the upper threshold. Steps S3232-S3233 are repeated until no data in any column of the first set contains a value greater than the upper threshold.

[0199] Step S3234: Determine whether there are any data in the column data of the first data that have a value less than the lower threshold;

[0200] Specifically, if there is a value less than the lower threshold in a certain column of the first data, proceed to step S3235; if there is no value less than the lower threshold in a column of the first data, proceed to step S3236.

[0201] Step S3235: Replace values ​​less than the lower threshold with the lower threshold value;

[0202] Specifically, if any column in the first data contains a value lower than the lower threshold, that value is replaced with the lower threshold. Steps S3234-S3235 are repeated until no value lower than the lower threshold exists in any column of the first data, thus obtaining sample data. The sample data is the first data after outlier processing, and each column in the sample data is greater than the lower threshold of that column and less than the upper threshold of that column.

[0203] Step S3236: Obtain sample data.

[0204] Specifically, if there is no data with a value less than the lower threshold in the column data of the first data in step S3233, then the first data at this time is determined to be sample data; or, all column data when there is no data with a value less than the lower threshold in step S3234 are taken as sample data.

[0205] Further, repeat step S323 until step S323 is performed once for each of the first data obtained in step S322, resulting in a number of sample data.

[0206] In this embodiment of the application, after obtaining sample data from multiple sensors within the same time period, the training method for the human activity type recognition model further includes:

[0207] The sample data is divided into several sub-samples corresponding to the same type of human activity.

[0208] Understandably, the purpose of sample partitioning is to divide data of the same activity type over a long period of time into several samples of a specified time length. Specifically, each sample data corresponds to one type of human activity. The sample data is partitioned to obtain several sub-samples corresponding to the same type of human activity, specifically including:

[0209] The sample data is divided into samples according to a preset time window and a preset sliding step size to obtain several samples corresponding to the same type of human activity; a portion of the samples corresponding to each type of human activity are selected, and each selected sample is taken as a subsample.

[0210] The preset time window is the duration of one sample. If the duration of one sample is t seconds, then the data dimension corresponding to one sample is r rows and 6 columns, where r is the number of rows, and the number of rows = data acquisition frequency * duration of one sample. The 6 columns represent: left foot sensor X-axis acceleration, left foot sensor Y-axis acceleration, left foot sensor Z-axis acceleration, right foot sensor X-axis acceleration, right foot sensor Y-axis acceleration, and right foot sensor Z-axis acceleration, respectively. The preset sliding step size is the step size for sliding samples within the same sample data. The preset sliding step size = first parameter * number of rows corresponding to each sample data. The value of the first parameter c determines the overlap between adjacent sub-samples within the same sample data, where c ∈ (0, 1]. The larger the value of c, the lower the overlap. The duration of one sample, data acquisition frequency, and first parameter can be set by those skilled in the art according to actual needs, and are not limited in this embodiment.

[0211] Specifically, a sample data point is divided into several samples based on a preset time window and a preset sliding step size, resulting in several samples corresponding to the same type of human activity. Further, each sample data point obtained in step S302 is divided again, resulting in several samples corresponding to each sample data point, thus obtaining several samples corresponding to the same type of human activity. Further, a portion of the samples corresponding to each type of human activity is selected for training the human activity type recognition model, where each selected sample is a sub-sample. Another portion of the samples corresponding to each type of human activity is selected to verify the effectiveness of the human activity type recognition model. For example, 75% of the samples are selected for training the human activity type recognition model, and the remaining 25% are used to verify the effectiveness of the human activity type recognition model.

[0212] Step S303: Normalize each sample data to obtain several input samples;

[0213] Understandably, due to differences in individual physical indicators, the data distribution obtained from collecting data on the same type of human activity will also vary among different individuals. To enable faster convergence during model training and improve the model's robustness, it is necessary to normalize the subsamples corresponding to each sample data, that is, to map each column of data corresponding to each subsample to a fixed interval range.

[0214] Specifically, each sample data is normalized to obtain several input samples, including substituting each column of data in the subsample into the mapping formula to obtain the input sample corresponding to each subsample.

[0215] The mapping formula includes:

[0216]

[0217] Where feature_i′ represents the i-th column of the input sample, and feature_i represents the i-th column of the subsample. min feature_i represents the minimum value of the i-th column of the subsample. _max This represents the maximum value of the i-th column of the subsample.

[0218] The input sample and the subsample have the same number of columns, which is 6 columns. For example, the input sample is: [feature_1′, feature_2′, ..., feature_6′].

[0219] Step S304: Based on the human activity type recognition model, extract features from the input samples to obtain a multi-class probability array corresponding to each input sample;

[0220] Specifically, the human activity type recognition model includes a deep backbone network module. This module is a lightweight backbone neural network that takes the input sample obtained in step S303 as input and a multi-class probability array as output, and is composed of multiple residual network modules with skip structures stacked together. Each input sample corresponds to a multi-class probability array, which represents the predicted probability value for each human activity type corresponding to the input sample. The human activity type corresponding to the highest probability value in the multi-class probability array is the human activity type corresponding to the input sample.

[0221] It is understandable that as the sampling frequency of sensor data increases, the length of sample data within a specified time window also increases. Using only shallow convolutional networks cannot obtain complete features and quickly reduce the data dimensionality. Therefore, it is necessary to appropriately increase the network depth (add convolutional layers) to extract richer features. However, simply increasing the number of network layers can easily lead to degradation problems during model training. That is, the network structure becomes more complex as the number of layers increases, which increases the possibility that the model will obtain a local optimum. As a result, the accuracy of the model on the training set and test set will saturate or even decrease during training.

[0222] To avoid degradation during training of neural network models composed of multiple convolutional layers and to improve the accuracy of human activity type recognition models, this application introduces a residual network (also known as an identity mapping) module to construct a deep backbone network module. In order to improve the computing power and speed of the human activity type recognition model, this application cross-stacks several residual network modules and fully connected layers to form a deep backbone network module, thereby achieving model lightweighting.

[0223] Please see Figure 9 , Figure 9 This is a schematic diagram of the structure of a deep backbone network module provided in an embodiment of this application;

[0224] like Figure 9 As shown, the deep backbone network module includes a first convolutional layer, an activation function layer, a max pooling layer, two first residual network modules, three sets of second residual network modules and first residual network modules connected in sequence, a global average pooling layer, a first fully connected layer, an exit layer, a second fully connected layer, and a normalization layer.

[0225] The input samples of the deep backbone network module include three data dimensions (1, r, 6). The first dimension, 1, is the number of dimensions added after converting the two-dimensional (r, 6) subsamples into three-dimensional (1, r, 6) input samples, which can be regarded as the number of rows of the image data. The second dimension, r, is the length of the input sample (time-series sample data). For example, if the data acquisition frequency of the inertial sensor equipped with a three-axis accelerometer is set to 20Hz and the subsample duration t is 3 seconds, then the length of the input sample is r = 60, which can be regarded as the number of columns of the image data. The third dimension, 6, is the data dimension corresponding to the stitching of the data from two inertial sensors, including the 3-axis acceleration data of the left foot inertial sensor and the 3-axis acceleration data of the right foot inertial sensor, which can be regarded as the number of channels of the image data.

[0226] The first convolutional layer has a kernel size of 1*5, resulting in 16 channels in the data after convolution, and a kernel stride of 1. The activation function layer uses the ReLU function. The max pooling layer has a kernel size of 1*3, and a kernel stride of 2. Both first residual network modules have 16 channels in the data after convolution.

[0227] In this embodiment, the first convolutional layer uses a 1*5 kernel to perform one-dimensional convolution on the input sample, and the first two first residual network modules do not reduce the dimensionality of the input sample. That is, the dimensions of the data processed by the max pooling layer and the data output by the second first residual network module are both (1, r / 2, 16). This application enables the human activity type recognition model to learn the basic features such as fluctuation period and amplitude in time series data.

[0228] The first residual network module differs from the second residual network module. Both the first and second residual network modules are skip stacked structures. The first residual network module is used to perform identity mapping without adjusting the dimension of the input data, while the second residual network module is used to adjust the dimension of the input data and perform identity mapping. The three sets of second residual network modules and first residual network modules connected in sequence are used to learn deep features of different types of human activities.

[0229] The three sets of second residual network modules and first residual network modules connected in sequence are: a second residual network module with 32 channels after convolution, a first residual network module with 32 channels after convolution, a second residual network module with 64 channels after convolution, a first residual network module with 64 channels after convolution, a second residual network module with 128 channels after convolution, and a first residual network module with 128 channels after convolution.

[0230] In this embodiment of the application, through three sets of second residual network modules and first residual network modules connected in sequence, this application can learn deep features such as the acceleration data distribution of different types of human activities, and gradually reduce the second dimension of the input sample to less than 10, and gradually increase the number of channels corresponding to the third dimension of the input sample to 128.

[0231] The global average pooling layer is used to reduce the dimension of the data output from the previous layer from (1, q, 128) to (1, 128), where q is the length of the data output after processing by the last first residual network module, and q is less than 10.

[0232] The first fully connected layer and the second fully connected layer are used to decode the data feature information. Specifically, the first fully connected layer is used for feature weighting, that is, to reduce the dimensionality of the data output by the global average pooling layer, for example, to reduce the data feature length to 64. The exit layer is used to randomly disconnect the connection between the first fully connected layer and the second fully connected layer according to the probability value p. The second fully connected layer is used to map the sample feature space to the multi-class label space corresponding to the human activity type. For example, when the number of human activity types is k, the multi-class label space is a k-class label space.

[0233] In this embodiment of the application, the robustness of the human activity type recognition model can be enhanced by adding an exit layer between two fully connected layers.

[0234] The normalization layer outputs a multi-class probability array for each input sample. The human activity type corresponding to the highest probability value in the multi-class probability array is the human activity type corresponding to the input sample. For example, after the i-th input sample is processed by the deep backbone network module, the output multi-class probability array (k-class probability array) is p. i =[p i1 p i2 , ..., p ik ],and Where p ij This represents the predicted probability value that the i-th input sample is identified as the j-th type of human activity.

[0235] Please refer to the following: Figure 10 , Figure 10 This is a schematic diagram of the structure of a first residual network module provided in an embodiment of this application;

[0236] like Figure 10 As shown, the first residual network module includes a second convolutional layer, a first activation function layer, a third convolutional layer, and a second activation function layer connected in sequence. The input data of the first residual network module is added to the output data of the third convolutional layer and then used as the input data of the second activation function layer.

[0237] In this layer, the kernel size of both the second and third convolutional layers is 1*3, and the stride of the kernel movement is 1. The activation function used in both the first and second activation function layers is the ReLU function.

[0238] It is understandable that, since the first residual network module does not adjust the data in each dimension, the input data of the first residual network module is directly added to the output data of the third convolutional layer and used as the input data of the second activation function layer.

[0239] Please see Figure 11 , Figure 11 This is a schematic diagram of the structure of a second residual network module provided in an embodiment of this application;

[0240] like Figure 11 As shown, the second residual network module includes a fourth convolutional layer, a third activation function layer, a fifth convolutional layer, and a fourth activation function layer connected in sequence. The second residual network module also includes a skip layer. The input data of the skip layer is the same as the input data of the fourth convolutional layer. The output data of the skip layer is added to the output data of the fifth convolutional layer and used as the input data of the fourth activation function layer.

[0241] The fourth convolutional layer has a 1x3 kernel with a stride of 2, used to reduce the dimensionality of the input data and increase the dimensionality of the number of channels. The third and fourth activation function layers both use the ReLU activation function. The fifth convolutional layer has a 1x3 kernel with a stride of 1, and does not adjust the dimensions of the data. The skip layer has a 1x1 kernel with a stride of 2. It performs a 1x1 convolution on the input data to reduce the dimensionality of the input data and increase the dimensionality of the number of channels, so that the output of the skip layer is consistent with the output of the fifth convolutional layer, thus achieving superposition. The summation result is then input into the fourth activation function layer.

[0242] In this embodiment, by employing a first residual network module and a second residual network module with a skip structure, this application can achieve identity mapping, enabling the model to focus on learning the part of the information where the input and output of the residual network module differ, thus simplifying the goal and difficulty of feature learning and avoiding the problems of feature information loss and model training degradation when the network depth increases.

[0243] Compared to existing solutions that use human keypoint extraction models and multiple support vector machine classifier models as deep learning models, which are not lightweight enough in terms of model deployment, this application uses a human activity type recognition model that includes a deep backbone network module. As a lightweight neural network, the deep backbone network module has the advantages of small parameter and computational cost, which can effectively improve the accuracy of the human activity type recognition model and reduce its size and computational cost. It can be applied to lightweight sensor chips and can achieve millisecond-level inference output results, that is, achieve the effect of real-time return of motion type.

[0244] Step S305: Construct the cross-entropy loss function and train the human activity type recognition model based on the input samples until the cross-entropy loss function converges.

[0245] Specifically, a cross-entropy loss function is constructed. This cross-entropy loss function is used to adjust the parameters of the human activity type recognition model during the training process until the cross-entropy loss function converges. The human activity type recognition model at this point is taken as the trained human activity type recognition model. It can be understood that convergence means that, under a certain model parameter, the sum of the differences between the label encoding array corresponding to each data file in the human activity type dataset and the multi-class probability array predicted based on the human activity type recognition model is less than a preset threshold or fluctuates within a certain range.

[0246] Please see Figure 12 , Figure 12 yes Figure 3 A detailed flowchart of step S305 in the process;

[0247] like Figure 12 As shown, step S305: Constructing the cross-entropy loss function, training the human activity type recognition model based on the input samples until the cross-entropy loss function converges, including:

[0248] Step S3051: Encode the human activity type label corresponding to each input sample to obtain a label encoding array;

[0249] Specifically, each input sample corresponds to a data file, and the human activity type label of each input sample is the same as that of its corresponding data file. One-hot encoding is performed on the human activity type label corresponding to each input sample to obtain a label encoding array for each input sample. This label encoding array includes multiple elements, with only one element having a value of 1 and the rest having values ​​of 0. One-hot encoding is a current technique and will not be elaborated upon here. It can be understood that when acquiring the human activity type dataset, the human activity type label corresponding to each data file has already been recorded.

[0250] For example, if the human activity type label of an input sample corresponds to the first type of human activity, then the human activity type label of the input sample is hot-coded into an array of length k [1, 0, ..., 0]. That is, the first element in the array of length k has a value of 1, and the other elements have a value of 0. Similarly, if the human activity type label of the input sample corresponds to the j-th (1≤j≤k) type of human activity, then after the human activity type label of the input sample is hot-coded, the j-th element in the corresponding label encoding array of length k has a value of 1, and the other elements have a value of 0.

[0251] Step S3052: During the forward propagation process, obtain the predicted multi-class probability array;

[0252] Specifically, the forward propagation process is the forward computation process in a neural network. In the forward propagation process, the input sample passes through each layer of the deep backbone network module, and after the calculation of weights and activation functions, it is passed layer by layer to the output layer, thereby obtaining the predicted multi-class probability array. The predicted multi-class probability array includes the predicted probability value of each type of human activity corresponding to the input sample.

[0253] For example, when there are k types of human activities, the multi-class probability array corresponding to the i-th input sample is a k-class probability array, which can be represented as: p i =[p i1 p i2 , ..., p ik ], where p i p represents a multi-class probability array. i1 p represents the predicted probability value when the input sample i is the first type of human activity. i2 p represents the predicted probability value when input sample i is the second type of human activity, ..., p ik This represents the predicted probability value when the input sample i is the kth type of human activity.

[0254] Step S3053: During the backpropagation process, calculate the value of the cross-entropy loss function based on the predicted probability value and the label encoding array;

[0255] Specifically, backpropagation is the process of propagating errors backward in a neural network. During backpropagation, the predicted probability values ​​and each element of the corresponding label encoding array are substituted into the formula for calculating the cross-entropy loss function to obtain its value. The formula for calculating the cross-entropy loss function is as follows:

[0256]

[0257] Where Loss represents the cross-entropy loss function, N represents the number of samples, k represents the number of categories of human activity types, i represents the sample index, j represents the category index, and y represents the cross-entropy loss function. ij p represents the value of the j-th element in the label encoding array corresponding to the i-th sample. ij This represents the predicted probability value that the i-th sample is the j-th type of human activity.

[0258] It is understandable that if the human activity type label corresponding to the i-th sample is the j-th type, then y ij The value of y is 1, otherwise y ij The value is 0.

[0259] Step S3054: Calculate the gradient information based on the value of the cross-entropy loss function;

[0260] Specifically, during the backpropagation process, gradient information is calculated using the chain rule based on the value of the cross-entropy loss function. The gradient information is the contribution of the weights of each layer of the deep backbone network module to the cross-entropy loss function.

[0261] Step S3055: Control the gradient information to be passed backward from the output layer to the input layer of the deep backbone network module until the cross-entropy loss function converges.

[0262] Specifically, the gradient information is passed back from the output layer to the input layer of the deep backbone network module to update and optimize the parameters in the deep backbone network module (e.g., the weights corresponding to each layer of the deep backbone network module). The updated model parameters are then used for the next training iteration until the cross-entropy loss function is less than the loss threshold, that is, the cross-entropy loss function converges. The loss threshold is set according to specific needs and is not limited here.

[0263] If the cross-entropy loss function converges, it indicates that the human activity type recognition model has been trained successfully. The model parameters of the human activity type recognition model at this point are then output, thus obtaining the trained human activity type recognition model.

[0264] Understandably, after training is completed, a trained human activity type recognition model is obtained. At this point, the human activity type corresponding to the collected human activity data can be identified by calling this trained human activity type recognition model.

[0265] After training the human activity type recognition model using the training method provided in the embodiments of this application, the human activity type recognition model can be used to identify the human activity type corresponding to the human activity data.

[0266] In this embodiment, a training method for a human activity type recognition model is provided. This method includes: acquiring a training dataset during human activity, wherein the training dataset includes an acceleration dataset composed of multiple acceleration information points and corresponding human activity type labels; the acceleration information is collected by each sensor; the acceleration dataset is a collection of acceleration information collected by multiple sensors within the same preset acquisition time period; concatenating the acceleration information to obtain sample data corresponding to multiple sensors within the same time period; normalizing each sample data to obtain several input samples; extracting features from the input samples based on the human activity type recognition model to obtain a multi-class probability array corresponding to each input sample, wherein the human activity type corresponding to the largest probability value in the multi-class probability array is the human activity type corresponding to the input sample; constructing a cross-entropy loss function, and training the human activity type recognition model based on the input samples until the cross-entropy loss function converges.

[0267] By acquiring a training dataset during human activity and stitching together the acceleration information in the training dataset, sample data from multiple sensors within the same time period are obtained. This application enables the data from multiple sensors to maintain relative consistency in the time dimension after stitching together. Furthermore, by using the input samples to train the human activity type recognition model, the accuracy of the human activity type recognition model in recognizing human activity types can be improved.

[0268] Example 2

[0269] After training the human activity type recognition model using the method provided in the above embodiments, a trained human activity type recognition model is obtained, which can be used to recognize human activity data.

[0270] It is understood that the above embodiment one is the training stage of the human activity type recognition model, and embodiment two of this application is the recognition stage of the human activity type recognition model. The recognition stage requires the human activity type recognition model to identify human activity data in order to obtain the human activity type corresponding to the human activity data.

[0271] Please see Figure 13 , Figure 13 This is a flowchart illustrating a method for identifying human activity types provided in an embodiment of this application;

[0272] like Figure 13 As shown, the method for identifying this type of human activity includes:

[0273] Step S1301: Acquire target human activity data;

[0274] Specifically, the electronic device acquires target human activity data collected and uploaded by at least two sensors. For example, it acquires target human activity data collected and uploaded by inertial sensors worn on the uppers of the left and right shoes. Understandably, the data acquisition method of the inertial sensor equipped with a three-axis accelerometer can be set to automatic acquisition, and the inertial sensor will upload the real-time acquired human activity data to the electronic device.

[0275] In this embodiment, by setting the data acquisition method of the inertial sensor to automatic acquisition, compared with existing human activity type recognition devices, such as smart bracelets, which require manual triggering of corresponding function buttons before body movement occurs to achieve data monitoring and analysis, making it difficult to support real-time automatic recognition of human activity types, the electronic device in this application can automatically identify human activity types after acquiring target human activity data uploaded by the sensor, thereby improving the portability of automatic data recording and analysis in human movement scenarios.

[0276] In this embodiment, by acquiring human activity data collected by inertial sensors configured on the uppers of the left and right shoes, compared to using a wristband with sensors to monitor user movement data, this application can obtain more accurate and recognizable human activity data, thereby improving the recognition accuracy of the human activity type recognition model.

[0277] Step S1302: Input the target human activity data into the human activity type recognition model to obtain the human activity type corresponding to the target human activity data.

[0278] Specifically, after acquiring target human activity data, the electronic device inputs the target human activity data into a trained human activity type recognition model to obtain the recognition result output by the human activity type recognition model, which includes the human activity type corresponding to the human activity data.

[0279] It is understood that the human activity type recognition model is trained using the training method of the human activity type recognition model in the above embodiments, and has the same structure and function as the human activity type recognition model in the above embodiments, which will not be described in detail here.

[0280] Specifically, the target human activity data includes at least one pair of acceleration information. After being input into the human activity type recognition model, the target human activity data is first concatenated to obtain corresponding sample data, then normalized to obtain input samples. The human activity type recognition model then outputs the predicted probability value for each human activity type corresponding to the target human activity data. The human activity type with the highest probability value is determined as the human activity type corresponding to the data. The specific implementation of data concatenation is similar to step S302 in Embodiment 1 and will not be repeated here. The specific implementation of normalization is similar to step S303 in Embodiment 1 and will not be repeated here.

[0281] Specifically, the higher the predicted probability value, the closer the human activity type of the human activity data is to that human activity type. The human activity type with the highest predicted probability value is determined as the human activity type corresponding to the human activity data.

[0282] Understandably, the human activity type recognition model can also output the motion time corresponding to the human activity data. Specifically, each frame of human activity data recorded by the inertial sensor contains a corresponding data generation time (timestamp). The human activity data input into the human activity type recognition model contains multiple frames of data. The human activity type recognition model records the timestamp corresponding to the first frame of each human activity data as the motion time corresponding to that human activity data. Therefore, for each human activity data recognition result output, the human activity type recognition model will also output the motion time corresponding to that recognition result.

[0283] In some embodiments, the electronic device can upload target human activity data to a big data platform, so that the big data platform can input the target human activity data into a trained human activity type recognition model to obtain the recognition result output by the human activity type recognition model, which includes the human activity type corresponding to the target human activity data.

[0284] Furthermore, after the big data platform sends the human activity data, corresponding to the activity type and duration, to the electronic device, the electronic device calculates the corresponding exercise duration based on the exercise time at the start and end of the exercise, and displays the activity type and duration on the interface to inform the user. The exercise duration is calculated as: Exercise duration = Exercise time at the end of the exercise - Exercise time at the start of the exercise.

[0285] Compared to using a bidirectional recurrent neural network module to identify human activity data, the model's computation and inference speed is slow, especially when the sensor sampling frequency is high. This can easily lead to the inability to return the human movement type in real time, causing raw data backlog and eventually system crashes. The human activity type recognition model in this application can achieve millisecond-level inference and output recognition results, thus achieving the effect of returning the human activity type in real time.

[0286] In this embodiment, by inputting the target human activity data into the trained human activity type recognition model, the human activity type corresponding to the target human activity data is obtained. The human activity type with the highest probability value is determined as the human activity type corresponding to the human activity data, which can more accurately identify the target human activity data and determine its corresponding human activity type.

[0287] Example 3

[0288] Please see Figure 14 , Figure 14 This is a schematic diagram of the structure of a training device for a human activity type recognition model provided in an embodiment of this application;

[0289] The training device for the human activity type recognition model is applied to an electronic device. Specifically, the training device for the human activity type recognition model is applied to one or at least two processors of the electronic device.

[0290] like Figure 14 As shown, the training device 140 for the human activity type recognition model includes:

[0291] The acquisition unit 141 is used to acquire a training dataset during human activity. The training dataset includes an acceleration dataset consisting of multiple acceleration information and human activity type labels corresponding to the acceleration dataset. The acceleration information is collected by each sensor. The acceleration dataset is a collection of acceleration information collected by multiple sensors within the same preset acquisition time period.

[0292] The data stitching unit 142 is used to stitch together acceleration information to obtain sample data from multiple sensors within the same time period.

[0293] Normalization unit 143 is used to normalize each sample data to obtain several input samples;

[0294] The feature extraction unit 144 is used to extract features from the input sample based on the human activity type recognition model to obtain a multi-class probability array corresponding to each input sample, wherein the human activity type corresponding to the largest probability value in the multi-class probability array is the human activity type corresponding to the input sample.

[0295] Training unit 145 is used to construct the cross-entropy loss function and train the human activity type recognition model based on the input samples until the cross-entropy loss function converges.

[0296] In the embodiments of this application, the training device for the human activity type recognition model can also be built from hardware devices. For example, the training device for the human activity type recognition model can be built from one or more chips, and the chips can work together to complete the training method for the human activity type recognition model described in the above embodiments. As another example, the training device for the human activity type recognition model can also be built from various logic devices, such as general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), microcontrollers, ARM processors (Advanced RISC Machines, ARM) or other programmable logic devices, discrete gate or transistor logic, discrete hardware components, or any combination of these components.

[0297] The training device for the human activity type recognition model in this application embodiment can be a device with an operating system. This operating system can be Android, iOS, or other possible operating systems; this application embodiment does not specifically limit the specific operating system used.

[0298] The training device for the human activity type recognition model provided in this application embodiment can achieve... Figure 3 To avoid repetition, the various processes involved will not be described in detail here.

[0299] It should be noted that the training device for the human activity type recognition model described above can execute the training method for the human activity type recognition model provided in the above embodiments, and has the corresponding functional modules and beneficial effects of the execution method. Technical details not described in detail in the embodiments of the training device for the human activity type recognition model can be found in the training method for the human activity type recognition model provided in the above embodiments.

[0300] In this embodiment, a training device for a human activity type recognition model is provided, comprising: an acquisition unit for acquiring a training dataset during human activity, wherein the training dataset includes an acceleration dataset composed of multiple acceleration information and human activity type labels corresponding to the acceleration dataset; the acceleration information is collected by each sensor; the acceleration dataset is a collection of acceleration information collected by multiple sensors within the same preset acquisition time period; a splicing unit for splicing the acceleration information to obtain sample data corresponding to multiple sensors within the same time period; a normalization unit for normalizing each sample data to obtain several input samples; a feature extraction unit for extracting features from the input samples based on the human activity type recognition model to obtain a multi-class probability array corresponding to each input sample, wherein the human activity type corresponding to the largest probability value in the multi-class probability array is the human activity type corresponding to the input sample; and a training unit for constructing a cross-entropy loss function and training the human activity type recognition model based on the input samples until the cross-entropy loss function converges.

[0301] The application uses an acquisition unit to acquire training datasets during human activity, and a data stitching unit to stitch acceleration information together to obtain sample data from multiple sensors within the same time period. This application enables the stitched data from multiple sensors to maintain relative consistency in the time dimension. Furthermore, by training the human activity type recognition model based on the input samples through the training unit, the accuracy of the human activity type recognition model in recognizing human activity types can be improved.

[0302] Example 4

[0303] Please see Figure 15 , Figure 15 This is a schematic diagram of the structure of a human activity type recognition device provided in an embodiment of this application;

[0304] like Figure 15 As shown, the human activity type recognition device 150 includes:

[0305] Data acquisition unit 151 is used to acquire target human activity data;

[0306] The human activity type recognition unit 152 is used to input the target human activity data into the human activity type recognition model to obtain the human activity type corresponding to the target human activity data. The human activity type recognition model is trained based on the training method of the human activity type recognition model provided in the above embodiment.

[0307] In this embodiment, the human activity type recognition device can also be constructed from hardware components. For example, the human activity type recognition device can be constructed from one or more chips, and the chips can work in coordination to complete the human activity type recognition method described in the above embodiments. Furthermore, the human activity type recognition device can also be constructed from various logic devices, such as general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), microcontrollers, ARM (Acorn RISC Machine) or other programmable logic devices, discrete gate or transistor logic, discrete hardware components, or any combination of these components.

[0308] The human activity type recognition device in this application embodiment can be a device, or a component, integrated circuit, or chip in a terminal. The device can be a mobile electronic device or a non-mobile electronic device. For example, mobile electronic devices can be mobile phones, tablets, laptops, PDAs, in-vehicle electronic devices, wearable devices, ultra-mobile personal computers (UMPCs), netbooks, or personal digital assistants (PDAs), etc., while non-mobile electronic devices can be servers, network attached storage (NAS), personal computers (PCs), televisions (TVs), ATMs, or self-service machines, etc. This application embodiment does not impose specific limitations.

[0309] The human activity type recognition device in this application embodiment can be a device with an operating system. This operating system can be Android, iOS, or other possible operating systems; this application embodiment does not specifically limit it.

[0310] The human activity type recognition device provided in this application embodiment can achieve... Figure 13 To avoid repetition, the various processes involved will not be described in detail here.

[0311] It should be noted that the above-described human activity type identification device can execute the human activity type identification method provided in the above embodiments, and has the corresponding functional modules and beneficial effects of the method. Technical details not described in detail in the embodiments of the human activity type identification device can be found in the human activity type identification method provided in the above embodiments.

[0312] In this embodiment, a human activity type identification device is provided, comprising: a data acquisition unit for acquiring target human activity data; and a human activity type identification unit for inputting the target human activity data into a human activity type identification model to obtain the human activity type corresponding to the target human activity data, wherein the human activity type identification model is trained based on the training method of the human activity type identification model provided in the above embodiment.

[0313] This application enables data from two sensors to maintain relative consistency in the time dimension after being stitched together, reducing the differences in sensor hardware and solving the problem of different frame loss rates during data upload; it also ensures the integrity of feature information of different human activity types during transmission, simplifies the goal and difficulty of feature learning, avoids the problem of feature information loss and model training degradation when network depth increases, and thus accurately identifies the human activity type corresponding to the human activity type data.

[0314] Example 5

[0315] This application embodiment also provides a non-volatile computer-readable storage medium storing computer-executable instructions. These instructions are used to cause an electronic device to execute the training method for the human activity type recognition model provided in the above embodiments, for example... Figure 3 The training method for the human activity type recognition model shown, or the human activity type recognition method provided in the above embodiments, for example, such as... Figure 13 The method for identifying types of human activity is shown.

[0316] In the embodiments of this application, the storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or it may be a device including one or any combination of the above-mentioned memories.

[0317] In the embodiments of this application, the executable instructions may take the form of a program, software, software module, script or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including being deployed as a standalone program or as a module, component, subroutine or other unit suitable for use in a computing environment.

[0318] As an example, executable instructions may, but do not necessarily, correspond to files in the file system. They may be stored as part of a file that holds other programs or data, for example, in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple co-located files (e.g., a file that stores one or more modules, subroutines, or code sections).

[0319] As an example, executable instructions can be deployed to execute on a single computing device (including devices such as smart terminals and servers), or on multiple computing devices located in one location, or on multiple computing devices distributed across multiple locations and interconnected via a communication network.

[0320] This application also provides a non-volatile computer-readable storage medium storing a computer program, which includes program instructions. When executed by an electronic device, the program instructions cause the electronic device to perform the training method for the human activity type recognition model or the human activity type recognition method as described in the above embodiments.

[0321] This application also provides a computer program product comprising one or more lines of program code stored in a non-volatile computer-readable storage medium. The processor of an electronic device reads the program code from the non-volatile computer-readable storage medium and executes the program code to complete the method steps of the training method for the human activity type recognition model or the method for recognizing human activity types provided in the above embodiments.

[0322] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware, or by a program or program code related to hardware. The program can be stored in a non-volatile computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0323] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented using software and a general-purpose hardware platform, or of course, using hardware. Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0324] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and not to limit them; under the concept of this application, the technical features of the above embodiments or different embodiments can also be combined, the steps can be implemented in any order, and there are many other variations of different aspects of this application as described above. For the sake of brevity, they are not provided in detail; although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or make equivalent substitutions for some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

Claims

1. A training method for a human activity type recognition model, characterized in that, Applied to an electronic device, the electronic device including multiple sensors for acquiring acceleration information, the method includes: A training dataset is obtained during human activity, wherein the training dataset includes an acceleration dataset consisting of multiple acceleration information and human activity type labels corresponding to the acceleration dataset; the acceleration information is collected by each of the sensors; the acceleration dataset is a collection of acceleration information collected by multiple sensors within the same preset collection time period; The acceleration information is stitched together to obtain sample data from multiple sensors within the same time period; Each of the sample data is normalized to obtain several input samples; Based on the human activity type recognition model, feature extraction is performed on the input sample to obtain a multi-class probability array corresponding to each input sample, wherein the human activity type corresponding to the largest probability value in the multi-class probability array is the human activity type corresponding to the input sample. Construct a cross-entropy loss function, and train the human activity type recognition model based on the input samples until the cross-entropy loss function converges; The step of stitching together the acceleration information to obtain sample data from multiple sensors within the same time period includes: Each pair of acceleration information is horizontally stitched together to obtain stitched data, wherein the stitched data includes several rows and columns of data, and each row of data includes a combination of motion data collected by two sensors at the same time point; The spliced ​​data is filled with null values ​​to obtain the first data, wherein each data in the first data is a non-null value; Outlier processing is performed on the first data to obtain sample data corresponding to multiple sensors within the same time period; The step of filling null values ​​in the spliced ​​data to obtain the first data includes: If there are empty values ​​in the column data, then the median value of the column data is determined; Replace the empty data with the median data to obtain the first data; The step of performing outlier processing on the first data to obtain sample data includes: If there is a value in a column of the first data that is greater than the upper limit threshold, then the value of the data is replaced with the upper limit threshold, wherein each column of data corresponds to an upper limit threshold; If there are values ​​in the columns of the first data that are less than the lower threshold, then the values ​​less than the lower threshold are replaced with the lower threshold to obtain sample data, wherein each column of data corresponds to a lower threshold.

2. The method according to claim 1, characterized in that, Each pair of acceleration information includes motion data collected by two sensors within the same preset acquisition time period. At least two of the sensors are symmetrically worn on different parts of the same person's body, and at least two of the sensors are oriented in the same direction. Each pair of acceleration information corresponds to an original mixed data file, which stores acceleration information from both sensors. The process of horizontally stitching together each pair of acceleration information to obtain stitched data includes: If the motion data collected by the two sensors in the original mixed data file have different data lengths, then the sensor corresponding to the motion data with the larger data length is determined to be the first sensor, and the sensor corresponding to the motion data with the smaller data length is determined to be the second sensor, and the motion data of the second sensor in the original mixed data file is stored in the first data file; Traverse each line in the first data file and obtain the mixed data in the original mixed data file according to the line number of each line of data in the original mixed data file; The motion data collected by the first sensor in the mixed data is obtained, and the data that has been successfully matched by the second sensor is deleted from the motion data collected by the first sensor to obtain the second data file; If the length of the data in the second data file is not zero, then calculate the absolute value of the difference between the timestamp corresponding to each line of data in the second data file and the timestamp corresponding to the selected line of data in the first data file in turn to obtain an absolute time difference sequence, and determine the minimum value and the corresponding position index in the absolute time difference sequence. Based on the minimum value and the position index, determine the row number in the second data file that matches the selected row data in the first data file; The triaxial acceleration data is obtained from the selected row data in the first data file and the row data corresponding to the row number in the second data file, and then spliced ​​together to obtain a spliced ​​row data. This process is repeated until all rows in the first data file are traversed to obtain spliced ​​data, wherein the spliced ​​data includes several spliced ​​row data.

3. The method according to claim 2, characterized in that, The step of determining the row number in the second data file that matches the selected row data in the first data file based on the minimum value and the position index includes: If the minimum value is greater than the first threshold, then traverse the next line of data in the first data file; If the minimum value is less than or equal to the first threshold, the position index is greater than 1, the minimum value is greater than the second threshold, and the first value in the absolute time difference sequence is less than the third threshold, then the row in the second data file that matches the selected row data in the first data file is determined to be the first row, wherein the first threshold is greater than the second threshold; If the minimum value is less than or equal to the second threshold, or if the first value in the absolute time difference sequence is greater than or equal to the third threshold, then when the position index is greater than 1 and the difference between two adjacent values ​​in the absolute time difference sequence is less than the fourth threshold, the row in the second data file that matches the selected row data in the first data file is determined to be the (k-1)th row, where the two adjacent values ​​are the (k-1)th value and the kth value; If the position index is less than or equal to 1, or the difference between two adjacent values ​​in the absolute time difference sequence is greater than or equal to the fourth threshold, then the row in the second data file that matches the selected row data in the first data file is determined to be the k-th row.

4. The method according to claim 1, characterized in that, Each sample data point corresponds to a type of human activity, and the method further includes: The sample data is divided into several sub-samples corresponding to the same type of human activity, specifically including: The sample data is divided into samples according to a preset time window and a preset sliding step size to obtain several samples corresponding to the same type of human activity, wherein the preset sliding step size = the first parameter. The number of rows for each sample data point, where the number of rows equals the data collection frequency. The duration of a sample; Select a subset of samples from a number of samples corresponding to each type of human activity, and treat each selected sample as a subsample.

5. The method according to claim 4, characterized in that, Each of the sample data is normalized to obtain several input samples, including: Substitute each column of data in the subsample into the mapping formula to obtain the input sample corresponding to each subsample, wherein the number of columns in the input sample is the same as that in the subsample; The mapping formula includes: in, Represents the first input sample Column data, Represents the first in the subsample Column data, Indicates the subsample number The minimum value of the column data, Indicates the subsample number The maximum value of the column data.

6. The method according to claim 1, characterized in that, The construction of the cross-entropy loss function, and the training of the human activity type recognition model based on the input samples until the cross-entropy loss function converges, includes: Encode the human activity type label corresponding to each input sample to obtain a label encoding array; During the forward propagation process, a predicted multi-class probability array is obtained, wherein the predicted multi-class probability array includes the predicted probability value of each type of human activity corresponding to the input sample; During the backpropagation process, the value of the cross-entropy loss function is calculated based on the predicted probability value and the label encoding array; The gradient information is calculated based on the value of the cross-entropy loss function; The gradient information is controlled to be propagated backward from the output layer to the input layer of the deep backbone network module until the cross-entropy loss function converges.

7. The method according to claim 6, characterized in that, The cross-entropy loss function includes: in, Represents the cross-entropy loss function. Indicates the number of samples. This represents the number of categories of human activity types, where i represents the sample number and j represents the category number. This represents the value of the j-th element in the true multi-class probability array corresponding to the i-th sample. This represents the predicted probability value that the i-th sample is the j-th type of human activity.

8. The method according to claim 6, characterized in that, The human activity type recognition model includes: The deep backbone network module includes a first convolutional layer, an activation function layer, a max pooling layer, two first residual network modules, three sets of second residual network modules and first residual network modules connected in sequence, a global average pooling layer, a first fully connected layer, an exit layer, a second fully connected layer, and a normalization layer. The first residual network modules are different from the second residual network modules, and both the first residual network modules and the second residual network modules are skip stacked structures.

9. A training device for a human activity type recognition model, characterized in that, Applied to an electronic device, the electronic device including multiple sensors for acquiring acceleration information, the training device includes: An acquisition unit is used to acquire a training dataset during human activity. The training dataset includes an acceleration dataset consisting of multiple acceleration information and human activity type labels corresponding to the acceleration dataset. The acceleration information is acquired by each sensor. The acceleration dataset is a collection of acceleration information acquired by multiple sensors within the same preset acquisition time period. The data stitching unit is used to stitch together the acceleration information to obtain sample data corresponding to multiple sensors within the same time period. A normalization unit is used to normalize each of the sample data to obtain several input samples; The feature extraction unit is used to extract features from the input sample based on the human activity type recognition model to obtain a multi-class probability array corresponding to each input sample, wherein the human activity type corresponding to the largest probability value in the multi-class probability array is the human activity type corresponding to the input sample. The training unit is used to construct the cross-entropy loss function and train the human activity type recognition model based on the input samples until the cross-entropy loss function converges. The data stitching unit is specifically used to horizontally stitch together each pair of acceleration information to obtain stitched data. The stitched data includes several rows and columns of data, and each row of data includes a combination of motion data collected by two sensors at the same time point. The stitched data is then filled with null values ​​to obtain first data, where each data in the first data is a non-null value. Outlier processing is then performed on the first data to obtain sample data corresponding to multiple sensors within the same time period. The data splicing unit is specifically used to determine the median data of the column data if there is data with empty values ​​in the column data; and to replace the data with empty values ​​with the median data to obtain the first data. The data splicing unit is specifically used to replace the value of the data with the upper limit threshold if there is data in the column data of the first data that has a value greater than the upper limit threshold, wherein each column data corresponds to an upper limit threshold; and to replace the value of the data with the lower limit threshold if there is data in the column data of the first data that has a value less than the lower limit threshold, thereby obtaining sample data, wherein each column data corresponds to a lower limit threshold.

10. A method for identifying types of human activity, characterized in that, The method includes: Acquire target human activity data; The target human activity data is input into the human activity type recognition model to obtain the human activity type corresponding to the target human activity data, wherein the human activity type recognition model is trained based on the method described in any one of claims 1-8.

11. A device for identifying types of human activity, characterized in that, The identification device includes: The data acquisition unit is used to acquire target human activity data. A human activity type recognition unit is used to input the target human activity data into a human activity type recognition model to obtain the human activity type corresponding to the target human activity data, wherein the human activity type recognition model is trained based on the method described in any one of claims 1-8.

12. An electronic device, characterized in that, include: At least one processor, and The memory communicatively connected to the at least one processor, wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1-8 or 10.

13. A non-volatile computer-readable storage medium, characterized in that, The non-volatile computer-readable storage medium stores computer-executable instructions that, when executed by a processor, cause the processor to perform the method as described in any one of claims 1-8 or 10.

Images