Human behavior recognition method and device, mobile terminal and storage medium

Abstract

The application discloses a human behavior recognition method and device, a mobile terminal and a storage medium, and relates to the technical field of artificial intelligence. The method is applied to a mobile terminal connected with a wearable device, and comprises the following steps: collecting motion data of a human body as first motion data, and acquiring motion data of corresponding parts of the human body collected by the wearable device as second motion data; inputting the first motion data and the second motion data into a trained human behavior recognition model; and acquiring the behavior of the human body output by the trained human behavior recognition model. The application detects the motion data of the human body through the mobile terminal and the wearable device, so as to recognize the human behavior, and can improve the accuracy and feasibility of human behavior recognition.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and more particularly, to a human behavior recognition method and device, a mobile terminal, and a storage medium. BACKGROUND

[0002] With the development of science and technology, mobile terminals are increasingly widely used and have more and more functions, and have become one of the necessities in people's daily life. Human behavior recognition (HAR) can be performed through a mobile terminal. However, the accuracy of human behavior recognition through a mobile terminal is currently low. SUMMARY

[0003] In view of the above problems, the present application provides a human behavior recognition method and device, a mobile terminal, and a storage medium to solve the above problems.

[0004] In a first aspect, the embodiments of the present application provide a human behavior recognition method applied to a mobile terminal, wherein the mobile terminal is connected with a wearable device, and the method comprises: collecting motion data of a human body as first motion data, and collecting motion data of a corresponding part of the human body collected by the wearable device as second motion data; inputting the first motion data and the second motion data into a trained human behavior recognition model; and obtaining a behavior of the human body output by the trained human behavior recognition model.

[0005] In a second aspect, the embodiments of the present application provide a human behavior recognition device applied to a mobile terminal, wherein the mobile terminal is connected with a wearable device, and the device comprises: a motion data acquisition module configured to collect motion data of a human body as first motion data, and collect motion data of a corresponding part of the human body collected by the wearable device as second motion data; a motion data input module configured to input the first motion data and the second motion data into a trained human behavior recognition model; and a human behavior obtaining module configured to obtain a behavior of the human body output by the trained human behavior recognition model.

[0006] In a third aspect, the embodiments of the present application provide a mobile terminal comprising a memory and a processor, wherein the memory is coupled to the processor, and the memory stores instructions, and when the instructions are executed by the processor, the processor executes the above method.

[0007] In a fourth aspect, the embodiments of the present application provide a computer-readable storage medium, wherein the computer-readable storage medium stores program codes, and the program codes can be invoked by a processor to execute the above method.

[0008] The method, device, mobile terminal and storage medium provided by the embodiments of the present application can collect motion data of a human body as first motion data, collect motion data of corresponding parts of the human body collected by a wearable device as second motion data, input the first motion data and the second motion data into a trained human behavior recognition model, and obtain a behavior of the human body output by the trained human behavior recognition model, so as to detect the motion data of the human body by the mobile terminal and the wearable device together, and identify the human behavior, thereby improving the accuracy and feasibility of human behavior recognition. BRIEF DESCRIPTION OF DRAWINGS

[0009] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative effort.

[0010] Figure 1 An application scenario schematic diagram of the method for identifying human behavior provided by the embodiments of the present application is shown.

[0011] Figure 2 A use scenario schematic diagram of the method for identifying human behavior provided by the embodiments of the present application is shown.

[0012] Figure 3 A flow schematic diagram of the method for identifying human behavior provided by an embodiment of the present application is shown.

[0013] Figure 4 A flow schematic diagram of the method for identifying human behavior provided by an embodiment of the present application is shown.

[0014] Figure 5 A flow schematic diagram of the method for identifying human behavior provided by an embodiment of the present application is shown.

[0015] Figure 6 A flow schematic diagram of the method for identifying human behavior provided by the embodiments of the present application is shown. Figure 5 A flow schematic diagram of step S310 of the method for identifying human behavior shown in the embodiments of the present application is shown.

[0016] Figure 7 A flow schematic diagram of the method for identifying human behavior provided by the embodiments of the present application is shown. Figure 5 A flow schematic diagram of step S320 of the method for identifying human behavior shown in the embodiments of the present application is shown.

[0017] Figure 8 A flow schematic diagram of the method for identifying human behavior provided by an embodiment of the present application is shown.

[0018] Figure 9 A flow schematic diagram of the method for identifying human behavior provided by the embodiments of the present application is shown. Figure 8A flowchart of step S420 of the method for recognizing human behavior shown;

[0019] Figure 10 A training schematic diagram of the CNN-LSTM model provided by the embodiments of the application is shown;

[0020] Figure 11 A module block diagram of the device for recognizing human behavior provided by the embodiments of the application is shown;

[0021] Figure 12 A block diagram of a mobile terminal for executing the method for recognizing human behavior according to the embodiments of the application is shown;

[0022] Figure 13 A storage unit for storing or carrying program code for implementing the method for recognizing human behavior according to the embodiments of the application is shown. DETAILED DESCRIPTION

[0023] In order to enable personnel in the art to better understand the schemes of the present application, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application.

[0024] Human behavior recognition, as an important detection means, can well improve the interaction and monitoring functions of mobile terminals. Human behavior recognition has been widely applied to current mainstream mobile terminals and wearable devices, such as the lifting wake-up of a smart phone, the motion type recognition of a smart watch, etc. In these devices, the common human behavior recognition technology mainly measures the acceleration, angular velocity and orientation change of a single device through the inertial measurement unit (IMU) carried by the single device, uses a corresponding data processing method and recognition model, and obtains different behaviors performed by a human body. With the continuous development of deep learning, deep neural networks have been proven to have higher accuracy in human behavior recognition and can recognize more complex behaviors, actions, states or scenes.

[0025] Current human behavior recognition models based on mobile terminals use data collected by a single position placed mobile terminal, and can only recognize some fixed human motion states or simple behavior patterns, and cannot recognize complex human behaviors and actions. In addition, although some experiments and technologies use sensors fixed on different parts of the human body to improve the types of human behaviors they can recognize, these complex sensors are limited by the daily behaviors and wearing requirements of the human body, and are not suitable for daily life, and the application scenarios are very limited. At the same time, the emergence of deep learning enables recognition models to obtain good recognition results using a large amount of training data, and therefore new requirements are put forward for the diversity and data volume of human behavior data.

[0026] To address the aforementioned problems, the inventors, through long-term research, discovered and proposed the human behavior recognition method, device, mobile terminal, and storage medium provided in the embodiments of this application. By jointly detecting human motion data through a mobile terminal and wearable devices for human behavior recognition, the accuracy and feasibility of human behavior recognition can be improved. The specific human behavior recognition method will be described in detail in subsequent embodiments.

[0027] The following will describe the application scenarios of the human behavior recognition method provided in the embodiments of this application.

[0028] Please see Figure 1 , Figure 1 A schematic diagram illustrating an application scenario for the human behavior recognition method provided in the embodiments of this application is shown. For example... Figure 1 As shown, this application scenario can include a mobile terminal 100 and a wearable device 200. The mobile terminal 100 is connected to the wearable device 200 to enable data interaction between them. For example, it can transmit motion data of corresponding body parts collected by the wearable device 200 to the mobile terminal 100. Alternatively, the mobile terminal 100 can connect to the wearable device 200 wirelessly, such as via Wi-Fi, Bluetooth, or ZigBee, etc., without limitation.

[0029] As one possible implementation, the number of mobile terminals 100 can be one, and the number of wearable devices 200 can be one or more, wherein the wearable devices can be a combination of one or more of wireless headphones, smartwatches, and smart sports shoes, without limitation.

[0030] Please see Figure 2 , Figure 2 This illustration shows a schematic diagram of a usage scenario for the human behavior recognition method provided in an embodiment of this application. For example... Figure 2 As shown, the wearable device 200 can be a smartwatch, which is worn on the user's hand and can be used to collect the user's hand movement data. The mobile terminal 100 (not shown) can be placed in the user's pocket and used to collect the user's overall movement data.

[0031] Please see Figure 3 , Figure 3A flowchart of a human behavior recognition method according to an embodiment of the present application is shown. The method is used to detect motion data of a human body by a mobile terminal and a wearable device, for recognizing human behavior, which can improve the accuracy and feasibility of human behavior recognition. In specific embodiments, the human behavior recognition method is applied to a human behavior recognition device 200 as shown in Figure 11 and a mobile terminal 100 configured with the human behavior recognition device 200. Figure 12 The specific process of the embodiment will be described below with the mobile terminal as an example. It should be understood that the mobile terminal applied in the embodiment can be a smartphone, a tablet computer, etc., which is not limited herein. In the embodiment, the mobile terminal is connected with a wearable device. The human behavior recognition method will be described in detail with reference to the flowchart shown in Figure 3 The human behavior recognition method can include the following steps:

[0032] Step S110: Collect motion data of a human body as first motion data, and obtain motion data of a corresponding part of the human body collected by the wearable device as second motion data.

[0033] In the embodiment, the mobile terminal can collect motion data of a human body and collect the collected motion data as first motion data. The wearable device can collect motion data of a corresponding part of the human body and transmit the collected motion data to the mobile terminal based on the connection with the mobile terminal. Accordingly, the mobile terminal can obtain the motion data of the corresponding part of the human body collected by the wearable device as second motion data.

[0034] In some embodiments, the mobile terminal can collect motion data of a human body in real time, collect motion data of a human body at a predetermined time interval, collect motion data of a human body at a predetermined time point, or collect motion data of a human body according to other predetermined rules, etc., which is not limited herein. In some embodiments, the wearable device can collect motion data of a corresponding part of a human body in real time, collect motion data of a corresponding part of a human body at a predetermined time interval, collect motion data of a corresponding part of a human body at a predetermined time point, or collect motion data of a corresponding part of a human body according to other predetermined rules, etc., which is not limited herein.

[0035] In some embodiments, the mobile terminal can be placed in a bag of the human body, such as a clothes bag, a trousers bag, a backpack, etc., or held in a hand of the human body, etc., for collecting the motion data of the whole human body. The wearable device can be worn on different parts of the human body for collecting the motion data of the corresponding parts of the human body. For example, when the wearable device is a wireless earphone, the wearable device can be worn on the ear of the human body for collecting the motion data of the head of the human body; when the wearable device is a smart watch, the wearable device can be worn on the hand of the human body for collecting the motion data of the hand of the human body; when the wearable device is a smart sports shoe, the wearable device can be worn on the foot of the human body for collecting the motion data of the foot of the human body. Of course, in some embodiments, the motion data collected by the wearable device can also include the motion data of the whole human body. For example, when the wearable device is a smart sports shoe, the motion data of the waist of the human body can also be collected in addition to the motion data of the foot of the human body. For another example, when the wearable device includes a smart watch, a smart earphone and a smart sports shoe at the same time, the wearable device can collect the motion data of the whole human body from the head to the foot at the same time.

[0036] It can be understood that, based on the above manner, the motion data of the whole human body can be collected by the mobile terminal, and the motion data of the local human body can be collected by the wearable device, so that the collected motion data can be more comprehensive.

[0037] Step S120: inputting the first motion data and the second motion data into a trained human behavior recognition model.

[0038] In some embodiments, after obtaining the first motion data and the second motion data, the mobile terminal can input the first motion data and the second motion data into a trained human behavior recognition model, wherein the trained human behavior recognition model is obtained through machine learning. Specifically, first, a training data set is collected, wherein the attributes or features of one type of data in the training data set are different from those of another type of data, and then the collected training data set is trained and modeled on a neural network according to a preset algorithm, so as to summarize the rules based on the training data set and obtain the trained human behavior recognition model. In this embodiment, the training data set can be, for example, a set of first training data, second training data and a training human body, the first training data can be obtained by collecting the motion data of the training human body by the mobile terminal, and the second training data can be obtained by collecting the motion data of the corresponding part of the training human body by the wearable device.

[0039] As a manner, the trained human behavior recognition model can be stored locally in the mobile terminal after being trained in advance. Based on this, the mobile terminal can directly call the trained human behavior recognition model locally after obtaining the first motion data and the second motion data, for example, can directly send an instruction to the human behavior recognition model to instruct the trained human behavior recognition model to read the first motion data and the second motion data in the target storage area, or the mobile terminal can directly input the first motion data and the second motion data into the trained human behavior recognition model stored locally, thereby effectively avoiding the influence of network factors to reduce the speed of inputting the first motion data and the second motion data into the trained human behavior recognition model, to improve the speed of the trained human behavior recognition model to obtain the first motion data and the second motion data, and to improve the user experience.

[0040] As another manner, the trained human behavior recognition model can also be stored in a server in communication connection with the mobile terminal after being trained in advance. Based on this, the mobile terminal can send an instruction to the trained human behavior recognition model stored in the server through the network after obtaining the first motion data and the second motion data, to instruct the trained human behavior recognition model to read the first motion data and the second motion data sent by the mobile terminal through the network, or the mobile terminal can send the first motion data and the second motion data to the trained human behavior recognition model stored in the server through the network, thereby reducing the occupation of the storage space of the mobile terminal by storing the trained human behavior recognition model in the server, and reducing the influence on the normal operation of the mobile terminal.

[0041] Step S130: obtaining the behavior of the human output by the trained human behavior recognition model.

[0042] In the embodiment, the trained human behavior recognition model can recognize the human behavior based on the input first motion data and second motion data and output the recognized human behavior, and accordingly, the mobile terminal can obtain the human behavior output by the trained human behavior recognition model.

[0043] In some embodiments, if the trained human behavior recognition model is stored locally in the mobile terminal, the mobile terminal directly obtains the human behavior output by the trained human behavior recognition model; if the trained human behavior recognition model is stored in the server, the mobile terminal can obtain the human behavior output by the trained human behavior recognition model from the server through the network. As a manner that can be implemented, the human behavior obtained by the trained human behavior recognition model can be output in the form of voice information, text information, picture information, etc., which is not limited herein.

[0044] Some relatively simple human behaviors can be recognized by the trained human behavior recognition model through the first motion data, for example, walking, running, standing still, going up and down stairs, and some transportation tools, such as riding a bicycle, taking a subway, taking a car, taking an airplane, and taking a high-speed rail. Some relatively fine human behaviors can be recognized by the trained human behavior recognition model through the first motion data and the second motion data, for example, whether the human body is in a standing state or a sitting state when the human body is still, whether the human body is driving a vehicle when the human body is taking a car, common daily behaviors such as eating, smoking, typing, sweeping, and nodding and shaking of specific parts, and some transitional human behaviors, such as from standing to sitting, from sitting to standing, from sitting to squatting, and from standing to squatting.

[0045] The method for recognizing human behaviors provided in an embodiment of the present application collects motion data of a human body as first motion data, and acquires motion data of corresponding parts of the human body collected by a wearable device as second motion data, inputs the first motion data and the second motion data into a trained human behavior recognition model, and acquires a behavior of the human body output by the trained human behavior recognition model, so that the motion data of the human body is detected by a mobile terminal and a wearable device together to recognize human behaviors, which can improve the accuracy and feasibility of human behavior recognition.

[0046] Referring to Figure 4 , Figure 4 A flowchart of a method for recognizing human behaviors provided in an embodiment of the present application is shown. The method is applied to the mobile terminal described above, and the mobile terminal is connected with a wearable device. In this embodiment, the mobile terminal includes a first inertial measurement unit, and the wearable device includes a second inertial measurement unit. The method for recognizing human behaviors will be described in detail below with reference to the flowchart shown in Figure 4 The method for recognizing human behaviors can specifically include the following steps:

[0047] Step S210: Collecting motion data of the human body by the first inertial measurement unit as first motion data.

[0048] In this embodiment, the mobile terminal includes a first inertial measurement unit, and the mobile terminal can collect motion data of the whole human body by the first inertial measurement unit as first motion data. The first inertial measurement unit can include an accelerometer, a gyroscope, and a magnetometer, and then the accelerometer, the gyroscope, and the magnetometer of the first inertial measurement unit can detect three-axis values of x-axis, y-axis, and z-axis, i.e., 9-dimensional data.

[0049] As a manner, the accelerometer, the gyroscope and the magnetometer in the first inertial measurement unit can be taken as three branches, the data collected by each branch of the three branches is subjected to a plurality of convolution layers, batch normalization, a pooling layer and a Dropout layer, feature extraction is performed through convolution to obtain spatial features of the data, and then the data is input into a 3-layer LSTM for time sequence feature extraction. After the above feature extraction is performed on each branch, the data features obtained are connected in parallel to obtain the first motion data.

[0050] In some embodiments, the number of the first inertial measurement units included in the mobile terminal can be one or more. When the number of the first inertial measurement units is one, the mobile terminal can collect the motion data of the human body as the first motion data through one first inertial measurement unit. When the number of the first inertial measurement units is more than one, the mobile terminal can collect the motion data of the human body as the first motion data through a plurality of first inertial measurement units.

[0051] Step S220: acquiring motion data of the corresponding part of the human body collected by the wearable device through the second inertial measurement unit as second motion data.

[0052] In the embodiment, the wearable device includes the second inertial measurement unit, and the wearable device can collect the motion data of the corresponding part of the human body through the second inertial measurement unit as the second motion data. The second inertial measurement unit can detect three-axis values of x-axis, y-axis and z-axis, i.e., three-dimensional data. For example, it is assumed that the wearable device includes two wireless earphones, a smart watch and two smart sports shoes, the two wireless earphones each include a second inertial measurement unit, the smart watch includes a second inertial measurement unit, and the two smart sports shoes each include a second inertial measurement unit. Therefore, the wearable device includes a total of five second inertial measurement units, and can obtain motion data of the corresponding part with a dimension of 15 as the second motion data.

[0053] As a manner, the accelerometer, the gyroscope and the magnetometer in the first inertial measurement unit can be taken as three branches, the data collected by each branch of the three branches is subjected to a plurality of convolution layers, batch normalization, a pooling layer and a Dropout layer, feature extraction is performed through convolution to obtain spatial features of the data, and then the data is input into a 3-layer LSTM for time sequence feature extraction. After the above feature extraction is performed on each branch, the data features obtained are connected in parallel to obtain the first motion data.

[0054] Step S230: inputting the first motion data and the second motion data into the trained human behavior recognition model.

[0055] Step S240: obtaining the behavior of the human body output by the trained human behavior recognition model.

[0056] For details of steps S230-S240, please refer to steps S120-S130, which will not be repeated here.

[0057] The embodiment of the present application provides a human behavior recognition method. The first inertial measurement unit is used to collect the motion data of the human body as the first motion data, the wearable device is used to collect the motion data of the corresponding part of the human body through the second inertial measurement unit as the second motion data, the first motion data and the second motion data are input into the trained human behavior recognition model, and the behavior of the human body output by the trained human behavior recognition model is obtained. Compared with the prior art, the embodiment of the present application has the advantages that Figure 3 The embodiment of the present application provides a human behavior recognition method. The first inertial measurement unit is used to collect the motion data of the human body as the first motion data, the wearable device is used to collect the motion data of the corresponding part of the human body through the second inertial measurement unit as the second motion data, the first motion data and the second motion data are input into the trained human behavior recognition model, and the behavior of the human body output by the trained human behavior recognition model is obtained. Compared with the prior art, the embodiment of the present application has the advantages that

[0058] For details of steps S230-S240, please refer to steps S120-S130, which will not be repeated here. Figure 5 Figure 5 The embodiment of the present application provides a human behavior recognition method. The first inertial measurement unit is used to collect the motion data of the human body as the first motion data, the wearable device is used to collect the motion data of the corresponding part of the human body through the second inertial measurement unit as the second motion data, the first motion data and the second motion data are input into the trained human behavior recognition model, and the behavior of the human body output by the trained human behavior recognition model is obtained. Compared with the prior art, the embodiment of the present application has the advantages that Figure 5 The embodiment of the present application provides a human behavior recognition method. The first inertial measurement unit is used to collect the motion data of the human body as the first motion data, the wearable device is used to collect the motion data of the corresponding part of the human body through the second inertial measurement unit as the second motion data, the first motion data and the second motion data are input into the trained human behavior recognition model, and the behavior of the human body output by the trained human behavior recognition model is obtained. Compared with the prior art, the embodiment of the present application has the advantages that

[0059] Step S310: collecting motion data of the human body in a preset time period as first motion data.

[0060] In some embodiments, the mobile terminal can be provided with a preset time period, which is used as a basis for judging the time period for collecting motion data by the mobile terminal. Therefore, in the embodiment, when the human behavior recognition needs to be performed, the mobile terminal can determine the preset time period set and collect the motion data of the human body in the preset time period as the first motion data. The preset time period may, for example, include 10 seconds, 15 seconds, etc., which is not limited herein.

[0061] In some embodiments, the mobile terminal can be provided with a preset time period and a first collection frequency, the preset time period being used as a basis for judging the time period for collecting motion data by the mobile terminal, and the first collection frequency being used as a basis for judging the frequency for collecting motion data by the mobile terminal. Therefore, in the embodiment, when the human behavior recognition needs to be performed, the mobile terminal can determine the preset time period and the first collection frequency set, and collect the motion data of the human body in the preset time period at the first collection frequency as the first motion data. The first collection frequency may, for example, be greater than or equal to 50 HZ.

[0062] ​Please refer to Figure 6 , Figure 6 The flowchart of step S310 of the method for recognizing human behavior of the present application is shown. Figure 5 In the present embodiment, the preset time period includes a start time period, an intermediate time period and an end time period. The following will be described in detail with respect to the flow shown in Figure 6 The method can specifically include the following steps:

[0063] Step S311: Collecting motion data of the human body in the preset time period as third motion data.

[0064] In some embodiments, the preset time period can include a start time period, an intermediate time period and an end time period in chronological order. Then, the motion data of the human body in the preset time period collected by the mobile terminal as the third motion data can include: the motion data of the human body in the start time period collected by the mobile terminal, the motion data of the human body in the intermediate time period collected by the mobile terminal and the motion data of the human body in the end time period collected by the mobile terminal as the third motion data.

[0065] The size among the start time period, the intermediate time period and the end time period can be automatically set by the mobile terminal or manually set by the user through the mobile terminal, which is not limited herein. In addition, the size relationship among the start time period, the intermediate time period and the end time period is not limited herein, i.e., the size of the start time period, the intermediate time period and the end time period can be the same or different, for example, start time period < end time period < intermediate time period, start time period < intermediate time period < end time period, intermediate time period < start time period < intermediate time period, etc.

[0066] As a way, the start time period can be 2 seconds and the end time period can be 2 seconds.

[0067] Step S312: deleting the third motion data collected in the start time period and the end time period and retaining the third motion data collected in the intermediate time period as the first motion data.

[0068] The motion data collected by the mobile terminal can include information of other interfering actions in the start time period and the end time period, therefore, in order to ensure the accuracy of the motion data participating in the recognition of the human behavior, the information possibly including other interfering actions can be deleted. That is, in the present embodiment, in order to ensure the accuracy of the third motion data participating in the recognition of the human behavior, the third motion data collected in the start time period and the end time period can be deleted and the third motion data collected in the intermediate time period can be retained as the first motion data participating in the recognition of the human behavior.

[0069] In some embodiments, the mobile terminal can perform data slicing on the third motion data based on a sliding window, delete the third motion data collected in the start time period and the end time period, and retain the third motion data collected in the middle time period as the first motion data. As an implementable manner, in order to extract the features of each behavior (motion) in the third motion data, the mobile terminal can sample (perform data slicing) the third motion data by using a sliding window. When the collection frequency of the mobile terminal is 50HZ, the mobile terminal can perform data slicing on the third motion data by using a sliding window with a size of 128, so that the obtained third motion data in the middle time period contains motion data in 2.56 seconds, which can cover most daily actions.

[0070] Step S320: acquiring motion data of the corresponding part of the human body collected by the wearable device in the preset time period as the second motion data.

[0071] In some embodiments, the mobile terminal can be provided with a preset time period, and send the preset time period to the wearable device based on the connection with the wearable device, the preset time period being used as a basis for judging the time period for collecting motion data by the wearable device. Therefore, in the present embodiment, when it is necessary to identify the behavior of the human body, the mobile terminal can determine the preset time period set by the mobile terminal, and send the preset time period to the wearable device. Accordingly, the wearable device collects motion data of the corresponding part of the human body in the preset time period as the second motion data.

[0072] In some embodiments, the mobile terminal can be provided with a preset time period and a second collection frequency, and send the preset time period and the second collection frequency to the wearable device based on the connection with the wearable device, the preset time period being used as a basis for judging the time period for collecting motion data by the wearable device, and the second collection frequency being used as a basis for judging the frequency for collecting motion data by the wearable device. Therefore, in the present embodiment, when it is necessary to identify the behavior of the human body, the mobile terminal can determine the preset time period and the second collection frequency set by the mobile terminal, and send the preset time period and the second collection frequency to the wearable device. Accordingly, the wearable device collects motion data of the corresponding part of the human body in the preset time period at the second collection frequency as the second motion data. The second collection frequency can be greater than or equal to 50HZ. The first collection frequency and the second collection frequency can be the same or different, which is not limited herein.

[0073] Please refer to Figure 7 , Figure 7 The present application is illustrated by Figure 5A flowchart of step S320 of the method for recognizing human body behavior is shown. In this embodiment, the preset time period includes a start time period, an intermediate time period, and an end time period. The following will be described in detail with respect to the flow shown, and the method can specifically include the following steps: Figure 7

[0074] Step S321: Obtain the motion data of the corresponding part of the human body collected by the wearable device in the preset time period as the fourth motion data.

[0075] In some embodiments, the preset time period can include a start time period, an intermediate time period, and an end time period in chronological order. Then, obtaining the motion data of the corresponding part of the human body collected by the wearable device in the preset time period as the fourth motion data can include: the motion data of the corresponding part of the human body collected by the wearable device in the start time period, the motion data of the corresponding part of the human body collected by the wearable device in the intermediate time period, and the motion data of the corresponding part of the human body collected by the wearable device in the end time period as the fourth motion data.

[0076] Step S322: Delete the fourth motion data collected in the start time period and the end time period, and retain the fourth motion data collected in the intermediate time period as the second motion data.

[0077] Among the motion data collected by the wearable device, the part in the start time period and the end time period can include information of other interfering actions, therefore, in order to ensure the accuracy of the motion data participating in the recognition of the human body behavior, the information that can include other interfering actions can be deleted. That is, in this embodiment, in order to ensure the accuracy of the fourth motion data participating in the recognition of the human body behavior, the fourth motion data collected in the start time period and the end time period can be deleted, and the fourth motion data collected in the intermediate time period can be retained as the second motion data participating in the recognition of the human body behavior.

[0078] In some embodiments, the mobile terminal can perform data cutting on the fourth motion data based on a sliding window, delete the fourth motion data collected in the start time period and the end time period, and retain the fourth motion data collected in the intermediate time period as the second motion data. As an implementable way, in order to extract the features of each behavior (motion) in the fourth motion data, a sliding window can be used to sample (data cutting) the fourth motion data, wherein when the collection frequency of the mobile terminal is 50HZ, a sliding window of 128 size can be used to perform data cutting on the fourth motion data, so that the obtained fourth motion data of the intermediate time period contains motion data within 2.56 seconds, which can cover most daily actions. ​

[0079] Step S330: inputting the first motion data and the second motion data into a trained human behavior recognition model.

[0080] Step S340: obtaining a behavior of the human output by the trained human behavior recognition model.

[0081] The specific description of steps S330-S340 can refer to steps S120-S130, which will not be repeated here.

[0082] The human behavior recognition method provided in an embodiment of the present application collects motion data of a human in a preset time period as first motion data, obtains motion data of corresponding parts of the human collected by a wearable device in the preset time period as second motion data, inputs the first motion data and the second motion data into a trained human behavior recognition model, and obtains a behavior of the human input by the trained human behavior recognition model. Compared with the human behavior recognition method shown in Figure 3 The human behavior recognition method provided in an embodiment of the present application collects motion data in a preset time period to prompt the accuracy of human recognition by increasing the amount of obtained motion data.

[0083] Please refer to Figure 8 , Figure 8 A flowchart of a human behavior recognition method provided in an embodiment of the present application is shown. The method is applied to the mobile terminal described above, and the mobile terminal is connected with a wearable device. The human behavior recognition method will be described in detail below with reference to the flowchart shown in Figure 8 The human behavior recognition method can specifically include the following steps:

[0084] Step S410: obtaining a training data set, wherein the training data set includes first training data, second training data, and a behavior of a training human, the first training data is obtained by a mobile terminal collecting motion data of the training human, and the second training data is obtained by a wearable device collecting motion data of corresponding parts of the training human.

[0085] In the embodiment, a training data set can be obtained. The training data set can be obtained based on a training human, the number of the training human is multiple, and the training human can include a minor human body, an adult human body, etc.

[0086] In some embodiments, the training data set includes first training data, second training data, and behavior of the training person. The first training data is obtained by collecting motion data of the training person by a mobile terminal, and the second training data is obtained by collecting motion data of the corresponding part of the training person by a wearable device. As an implementable way, for a certain training person, the motion data of the training person can be collected by a mobile terminal as first training data, the motion data of the corresponding part of the training person can be collected by a wearable device as second training data, and the behavior of the training person during the collection of the motion data can be obtained. At this time, the first training data of the training person, the second training data of the training person, and the behavior of the training person constitute a set of training data in the training data set.

[0087] Step S420: training the CNN-LSTM model by taking the first training data and the second training data as input data and the behavior of the training person as output data, to obtain a trained human behavior recognition model.

[0088] In some embodiments, after obtaining the training data set, the first training data and the second training data in the training data set can be taken as input data, the behavior of the training person can be taken as output data, and the CNN-LSTM (deep convolutional-long short-term memory network) model can be trained to obtain a trained human behavior recognition model.

[0089] For the CNN-LSTM model, the spatial features in the first training data and the second training data can be extracted by CNN, and the time sequence features of the first training data and the second training data can be extracted by LSTM, so that effective learning of a large number of motion protectors under complex motion can be realized, the motion state and behavior category of the human body can be monitored at any time, and various functions such as intelligent human-computer interaction, digital twin, augmented reality, etc. can be served.

[0090] In some embodiments, for the CNN-LSTM model, the first motion data and the second motion data are input through multiple convolution layers, batch normalization layers, pooling layers, and Dropout layers, feature extraction is performed through convolution to obtain spatial features of the data, and then the data is input into 3-layer LSTM for time sequence feature extraction. After each branch performs the above feature extraction, the obtained data features are connected in parallel, pass through a fully connected layer and a Softmax layer, and the maximum probability of the probability mapping can be output.

[0091] In addition, during the training of the CNN-LSTM model, the Adam optimizer and the cross-entropy loss function can be used, and the comprehensive evaluation method combining F1 value, accuracy, precision, and recall rate can be used.

[0092] The cross-entropy loss function formula is as follows:

[0093]

[0094] where y i represents the value of the real category, p i represents the predicted value of the model output, and there are N category possibilities in a single sample.

[0095] Please refer to Figure 9 , Figure 9 shows the steps S420 of the human behavior recognition method of the present application. In this embodiment, the CNN-LSTM model includes a data fusion module and a classifier, and the following will be described in detail with respect to the flow shown in Figure 8 The steps of the method can include the following steps: Figure 9

[0096] Step S421: input the first training data and the second training data into the data fusion module to obtain third training data obtained by the data fusion module performing data fusion on the first training data and the second training data.

[0097] In this embodiment, the CNN-LSTM model can include a feature extraction module, a feature fusion module, a first classifier, and a second classifier. The feature extraction module and the feature fusion module can be integrated into one module or divided into two modules, which is not limited here.

[0098] In some embodiments, after the mobile terminal obtains the first training data and the second training data, the mobile terminal can input the first training data and the second training data into the data fusion module of the CNN-LSTM model to perform data fusion on the first training data and the second training data through the data fusion module to obtain third training data. Accordingly, the mobile terminal can obtain the third training data obtained by the data fusion module performing data fusion on the first training data and the second training data.

[0099] ​As an implementable manner, when the data fusion module of the CNN-LSTM model receives the input first training data and second training data, the first training data and the second training data can be concatenated, and the whole concatenated data can be subjected to convolutional transformation using a plurality of convolutional neural layers to map the original first training data and the second training data to a high-dimensional data space, then subjected to time series modeling using a plurality of LSTM neural networks in the high-dimensional data space to capture potential correlations in time, and finally input into a fully connected network for dimension reduction to obtain third training data. The third training data obtained after the first training data and the second training data are processed by the data fusion module of the CNN-LSTM model can have a good mapping relationship with the behavior of the training human body, and thus can be used for the human behavior recognition task. Moreover, the cascade operation is introduced in the data input layer, the number of redundant convolution kernels is increased in the convolution layer to map the data to a high-dimensional space, and the global average pooling is introduced to make the model adaptive to the dimension and number of input data.

[0100] Step S422: training the classifier based on the third training data and the behavior of the training human body to obtain the trained human behavior recognition model.

[0101] In some embodiments, after obtaining the third training data, the classifier can be trained based on the third training data and the behavior of the training human body. The classifier can include a fine action classifier, so that the trained classifier can recognize the fine human behavior from the motion data collected by the mobile terminal and the wearable device, such as standing / standing, eating, typing, smoking, driving / riding, sweeping, nodding / shaking, etc. The classifier can also include a transition action classifier, so that the trained classifier can recognize the transition action of the human body from the motion data collected by the mobile terminal and the wearable device, such as from standing to sitting, from sitting to squatting, from standing to squatting, etc. The classifier can include a fine action classifier and a transition action classifier, so that the trained classifier can recognize the fine human behavior and the transition action of the human body from the motion data of the mobile terminal and the wearable device.

[0102] Please refer to Figure 10 , Figure 10A training schematic diagram of the CNN-LSTM model provided by the embodiment of the application is shown. The CNN-LSTM model can include two branches. The input of the mobile terminal branch is the first motion data collected by the mobile terminal. The first motion data is input to the motion state classifier through feature extraction, and the human motion state can be identified, such as human walking, running, standing, cycling, taking the subway, taking the car, going up / down the stairs, etc. The input of the multi-device fusion branch is the first motion data collected by the mobile terminal and the second motion data collected by the wearable device. The first motion data and the second motion data are input to the fine action classifier through data fusion for fine action classification, such as human standing / standing, eating, typing, smoking, driving / passenger, sweeping, nodding / shaking, etc. The transition action classifier can identify some transition actions, such as human standing to sitting, sitting to squatting, standing to squatting, etc.

[0103] Step S430: collecting motion data of the human body as first motion data, and obtaining motion data of a corresponding part of the human body collected by the wearable device as second motion data.

[0104] Step S440: inputting the first motion data and the second motion data into the trained human behavior recognition model.

[0105] Step S450: obtaining the behavior of the human body output by the trained human behavior recognition model.

[0106] The specific description of steps S430-S450 can be referred to steps S110-S130, which will not be repeated here.

[0107] The embodiment of the application provides a human behavior recognition method. Training data sets are obtained, wherein the training data sets include first training data, second training data, and the behavior of a training human body. The first training data is obtained by collecting motion data of the training human body by a mobile terminal. The second training data is obtained by collecting motion data of a corresponding part of the training human body by a wearable device. The first training data and the second training data are used as input data, and the behavior of the training human body is used as output data. A CNN-LSTM model is trained to obtain a trained human behavior recognition model. Motion data of a human body is collected as first motion data, and motion data of a corresponding part of the human body collected by a wearable device is obtained as second motion data. The first motion data and the second motion data are input into the trained human behavior recognition model, and the behavior of the human body output by the trained human behavior recognition model is obtained. Compared with the prior art, the embodiment of the application can identify the human behavior more accurately. Figure 3The method for recognizing human behavior also trains the CNN-LSTM model by using a training data set to obtain a trained human behavior recognition model, so that the behavior features in the extracted motion data are effectively improved, and the recognition accuracy of the trained human behavior recognition model is higher.

[0108] Please refer to Figure 11 , Figure 11 A module block diagram of a human behavior recognition device provided by an embodiment of the application is shown. The human behavior recognition device 300 is applied to the mobile terminal described above, and the mobile terminal is connected with a wearable device. In the following, the block diagram shown in the figure will be described, and the human behavior recognition device 300 includes a motion data acquisition module 310, a motion data input module 320, and a human behavior acquisition module 330, wherein: Figure 11

[0109] The motion data acquisition module 310 is configured to collect motion data of a human body as first motion data, and collect motion data of a corresponding part of the human body collected by the wearable device as second motion data.

[0110] Further, the mobile terminal includes a first inertial measurement unit, the wearable device includes a second inertial measurement unit, and the motion data acquisition module 310 includes a first motion data acquisition submodule and a second motion data acquisition submodule, wherein:

[0111] The first motion data acquisition submodule is configured to collect motion data of the human body by the first inertial measurement unit as the first motion data.

[0112] The second motion data acquisition submodule is configured to collect motion data of the corresponding part of the human body collected by the second inertial measurement unit of the wearable device as the second motion data.

[0113] Further, the motion data acquisition module 310 includes a third motion data acquisition submodule and a fourth motion data acquisition submodule, wherein:

[0114] The third motion data acquisition submodule is configured to collect motion data of the human body in a preset time period as the first motion data.

[0115] Further, the preset time period includes a start time period, an intermediate time period, and an end time period, and the third motion data acquisition submodule includes a first motion data acquisition unit and a second motion data acquisition unit, wherein:

[0116] The first motion data acquisition unit is configured to collect motion data of the human body in the preset time period as the third motion data. ​

[0117] The second motion data acquisition unit is configured to delete third motion data collected in the start time period and the end time period, and retain third motion data collected in the middle time period as the first motion data.

[0118] The fourth motion data acquisition sub-module is configured to acquire motion data of the corresponding part of the human body collected by the wearable device in the preset time period as second motion data.

[0119] Further, the preset time period includes a start time period, a middle time period, and an end time period. The fourth motion data acquisition sub-module includes a third motion data acquisition unit and a fourth motion data acquisition unit, wherein:

[0120] The third motion data acquisition unit is configured to acquire motion data of the corresponding part of the human body collected by the wearable device in the preset time period as fourth motion data.

[0121] The fourth motion data acquisition unit is configured to delete fourth motion data collected in the start time period and the end time period, and retain fourth motion data collected in the middle time period as the second motion data.

[0122] The motion data input module 320 is configured to input the first motion data and the second motion data into a trained human behavior recognition model.

[0123] The human behavior obtaining module 330 is configured to acquire a behavior of the human body output by the trained human behavior recognition model.

[0124] Further, the human behavior recognition device 300 further includes a training data set acquisition module and a model training module, wherein:

[0125] The training data set acquisition module is configured to acquire a training data set, wherein the training data set includes first training data, second training data, and a behavior of a training human body. The first training data is obtained by a mobile terminal collecting motion data of the training human body. The second training data is obtained by a wearable device collecting motion data of a corresponding part of the training human body.

[0126] The model training module is configured to train a CNN-LSTM model by taking the first training data and the second training data as input data and the behavior of the training human body as output data, to obtain a trained human behavior recognition model.

[0127] Further, the CNN-LSTM model comprises a data fusion module and a classifier, and the model training module comprises a data fusion submodule and a model training submodule, wherein:

[0128] The data fusion submodule is configured to input the first training data and the second training data into the data fusion module, and obtain third training data obtained by data fusion of the first training data and the second training data by the data fusion module.

[0129] The model training submodule is configured to train the classifier based on the third training data and the behavior of the training human body, and obtain the trained human body behavior recognition model.

[0130] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the above-described device and module can refer to the corresponding process in the foregoing method embodiments, which will not be described here.

[0131] In several embodiments provided in the present application, the coupling between the modules can be electrical, mechanical or other forms of coupling.

[0132] In addition, each functional module in each embodiment of the present application can be integrated in one processing module, or each module can exist physically independently, or two or more modules can be integrated in one module. The integrated module can be realized in the form of hardware or in the form of a software functional module.

[0133] Please refer to Figure 12 which shows a structural block diagram of a mobile terminal 100 provided in an embodiment of the present application. The mobile terminal 100 can be a smart phone, a tablet computer, an electronic book, etc. capable of running an application program. The mobile terminal 100 in the present application can comprise one or more of the following components: a processor 110, a memory 120, and one or more application programs, wherein the one or more application programs can be stored in the memory 120 and configured to be executed by the one or more processors 110, and the one or more programs are configured to execute the method as described in the foregoing method embodiments.

[0134] The processor 110 can include one or more processing cores. The processor 110 connects various parts within the mobile terminal 100 by various interfaces and lines, and performs various functions of the mobile terminal 100 and processes data by running or executing instructions, programs, code sets or instruction sets stored in the memory 120, and calling data stored in the memory 120. Alternatively, the processor 110 can be implemented in at least one of a hardware form of a digital signal processing (DSP), a field-programmable gate array (FPGA), a programmable logic array (PLA). The processor 110 can be integrated with a combination of one or more of a central processing unit (CPU), a graphics processing unit (GPU), and a modem. Among them, the CPU mainly processes an operating system, a user interface, and an application program; the GPU is responsible for rendering and drawing the content to be displayed; and the modem is used for processing wireless communication. It can be understood that the above-mentioned modem can also not be integrated into the processor 110, but can be implemented by a separate communication chip.

[0135] The memory 120 can include a random access memory (RAM) and can also include a read-only memory (ROM). The memory 120 can be used to store instructions, programs, codes, code sets or instruction sets. The memory 120 can include a program storage area and a data storage area, wherein the program storage area can store instructions for implementing an operating system, instructions for implementing at least one function (such as a touch function, a sound playing function, an image playing function, etc.), instructions for implementing each of the methods described below, etc. The data storage area can also store data created by the mobile terminal 100 in use (such as a phone book, audio and video data, chat record data, etc.).

[0136] Please refer to Figure 13 which shows a structural block diagram of a computer readable storage medium provided by the embodiments of the present application. The computer readable medium 400 stores program codes therein, and the program codes can be called and executed by a processor to perform the methods described in the above method embodiments.

[0137] The computer-readable storage medium 400 can be an electronic memory such as a flash memory, EEPROM (electrically erasable programmable read-only memory), EPROM, hard disk or ROM. Optionally, the computer-readable storage medium 400 comprises a non-transitory computer-readable medium. The computer-readable storage medium 400 has storage space for program code 410 to perform any of the method steps of the above-described methods. The program code can be read from or written to one or more computer program products. The program code 410 can be compressed, for example, in a suitable form.

[0138] To sum up, the method, device, mobile terminal and storage medium for recognizing human behavior provided by the embodiments of the present application collect motion data of a human body as first motion data, and acquire motion data of corresponding parts of the human body collected by a wearable device as second motion data, input the first motion data and the second motion data into a trained human behavior recognition model, and acquire the behavior of the human body output by the trained human behavior recognition model, so as to detect the motion data of the human body through the mobile terminal and the wearable device, and recognize the human behavior, which can improve the accuracy and feasibility of human behavior recognition.

[0139] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, but not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part of the technical features; and these modifications or replacements do not drive the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for recognizing human behavior, characterized in that, Applied to a mobile terminal, wherein the mobile terminal is connected to a wearable device, the method includes: A training dataset is obtained, wherein the training dataset includes first training data, second training data, and the behavior of the training human body. The first training data is obtained by a mobile terminal collecting motion data of the training human body, and the second training data is obtained by a wearable device collecting motion data of corresponding parts of the training human body. Using the first training data and the second training data as input data, and the behavior of the trained human as output data, the CNN-LSTM model is trained to obtain a trained human behavior recognition model. The process of training the CNN-LSTM model by using the first training data and the second training data as input data, and the behavior of the trained human as output data, includes: The first training data is input into the motion state classifier of the CNN-LSTM model to identify the motion state of the human body; the first training data and the second training data are fused and input into the fine action classifier of the CNN-LSTM model to identify fine actions; and the first training data and the second training data are fused and input into the transition action classifier of the CNN-LSTM model to identify transition actions. The motion data of the human body is collected as the first motion data, and the motion data of the corresponding parts of the human body collected by the wearable device is obtained as the second motion data. The first motion data and the second motion data are input into the trained human behavior recognition model; Obtain the human behavior output by the trained human behavior recognition model.

2. The method according to claim 1, characterized in that, The mobile terminal includes a first inertial measurement unit, and the wearable device includes a second inertial measurement unit. The process of collecting human motion data as first motion data and acquiring motion data of corresponding parts of the human body collected by the wearable device as second motion data includes: The motion data of the human body is collected by the first inertial measurement unit and used as the first motion data; The wearable device acquires motion data of the corresponding parts of the human body through the second inertial measurement unit, which is used as the second motion data.

3. The method according to claim 1, characterized in that, The process of collecting human motion data as first motion data and acquiring motion data of corresponding parts of the human body collected by the wearable device as second motion data includes: Collect the human body's motion data within a preset time period as the first motion data; The wearable device collects motion data of corresponding parts of the human body within the preset time period, which is used as second motion data.

4. The method according to claim 3, characterized in that, The preset time period includes a start time period, a middle time period, and an end time period. Collecting the human body's motion data within the preset time period, as the first motion data, includes: The motion data of the human body within the preset time period is collected as third motion data; Delete the third motion data collected within the start time period and the end time period, and retain the third motion data collected within the middle time period as the first motion data.

5. The method according to claim 4, characterized in that, The step of acquiring motion data of corresponding parts of the human body collected by the wearable device within the preset time period as second motion data includes: The wearable device collects motion data of the corresponding parts of the human body within the preset time period, and uses this as the fourth motion data. Delete the fourth motion data collected during the start and end time periods, and retain the fourth motion data collected during the middle time period as the second motion data.

6. The method according to any one of claims 1-5, characterized in that, The CNN-LSTM model includes a data fusion module and a classifier. The process of training the CNN-LSTM model using the first training data and the second training data as input data, and the human behavior as output data, to obtain a trained human behavior recognition model includes: The first training data and the second training data are input into the data fusion module to obtain the third training data obtained by the data fusion module through data fusion of the first training data and the second training data; The classifier is trained based on the third training data and the behavior of the training human to obtain the trained human behavior recognition model.

7. A human behavior recognition device, characterized in that, Applied to a mobile terminal, wherein the mobile terminal is connected to a wearable device, the device includes: The training dataset acquisition module is used to acquire a training dataset, wherein the training dataset includes first training data, second training data, and the behavior of the training human body. The first training data is obtained by a mobile terminal collecting motion data of the training human body, and the second training data is obtained by a wearable device collecting motion data of corresponding parts of the training human body. The model training module is used to train the CNN-LSTM model by taking the first training data and the second training data as input data and the behavior of the training human as output data, to obtain a trained human behavior recognition model. The process of training the CNN-LSTM model by using the first training data and the second training data as input data, and the behavior of the trained human as output data, includes: The first training data is input into the motion state classifier of the CNN-LSTM model to identify the motion state of the human body; the first training data and the second training data are fused and input into the fine action classifier of the CNN-LSTM model to identify fine actions; and the first training data and the second training data are fused and input into the transition action classifier of the CNN-LSTM model to identify transition actions. The motion data acquisition module is used to collect human motion data as first motion data, and to acquire motion data of corresponding parts of the human body collected by the wearable device as second motion data. The motion data input module is used to input the first motion data and the second motion data into the trained human behavior recognition model; The human behavior acquisition module is used to acquire the human behavior output by the trained human behavior recognition model.

8. A mobile terminal, characterized in that, The method includes a memory and a processor, the memory being coupled to the processor, the memory storing instructions, and the processor performing the method as described in any one of claims 1-6 when the instructions are executed by the processor.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium contains program code that can be invoked by a processor to execute the method as described in any one of claims 1-6.

Images