A state recognition method and apparatus

Abstract

The application discloses a state recognition method and device, and relates to the field of radar detection.The method comprises the following steps: obtaining target attributes corresponding to a user, wherein the target attributes comprise position information and speed information; determining characteristic data of the user change according to the position information and the speed information; and matching the characteristic data through a preset state threshold to determine the state of the user.The method can be applied to a non-contact environment, does not need a camera to collect information, and has the advantages of privacy protection, good comfort and the like.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of radar detection, in particular to a state recognition method and device. BACKGROUND

[0002] The state recognition of the detected target in the prior art usually adopts the following three technologies: one is a video-based state recognition technology, the other is a wearable sensor-based state recognition technology, and the third is an RF (Radio Frequency) signal-based state recognition technology.

[0003] Among them, the video-based state recognition technology uses a camera device, which has the risk of privacy leakage in some sensitive environments such as bedrooms and bathrooms; the wearable device-based state recognition technology requires users to actively wear related sensor devices, which affects the comfort of users, and the continuity of data collection also depends on the cooperation of users; the sensor is worn on different parts of the user, such as wrists or ankles, and the data obtained mainly focuses on the changes of the parts when a certain action is completed, so the recognizable actions are less and the accuracy is not very high; the RF signal-based state recognition technology has a resolution much worse than that of millimeter wave signals, and the positioning and tracking are not as fine as millimeter waves. Although the prior art has used millimeter waves to recognize the state of the user, it is generally less to recognize the corresponding state of the user and has low recognition accuracy. SUMMARY

[0004] In view of the above problems, the present application is proposed in order to provide a state recognition method and device which overcomes the above problems or at least partially solves the above problems.

[0005] According to a first aspect of the present application, a state recognition method is provided, the method comprising:

[0006] obtaining target attributes corresponding to a user, wherein the target attributes include position information and speed information;

[0007] determining feature data of changes of the user according to the position information and the speed information;

[0008] matching the feature data with a preset state threshold to determine the state of the user.

[0009] According to a second aspect of the present application, a state recognition device is provided, the device comprising:

[0010] a target point acquisition module configured to obtain target attributes corresponding to a user, wherein the target attributes include position information and speed information;

[0011] a data determining module configured to determine characteristic data of the user according to the position information and the speed information;

[0012] a state matching module configured to match the characteristic data with a preset state threshold to determine the state of the user.

[0013] In the scheme, the target attribute corresponding to the user is acquired, wherein the target attribute can include position information and speed information. According to the position information and the speed information, the height change, the horizontal displacement change and the speed change of the user can be determined to form the characteristic data corresponding to the state of the user. The characteristic data is matched with a preset state threshold to determine the specific state of the user. It can be applied in a non-contact environment without the need of a camera for information collection, and has the advantages of privacy protection and good comfort.

[0014] The above description is only a summary of the technical scheme of the application. In order to make the above and other purposes, features and advantages of the application more obvious and easy to understand, the following will specifically describe the embodiments of the application. BRIEF DESCRIPTION OF DRAWINGS

[0015] Various other advantages and benefits will become apparent to those of ordinary skill in the art upon reading the following detailed description of the preferred embodiments. The accompanying drawings are included to provide a description of the preferred embodiments and are not meant to limit the present application. Furthermore, the same reference numerals are used throughout the several drawings to designate the same or similar parts.

[0016] In the drawings:

[0017] Figure 1 is a step flow chart of a state recognition method provided by an embodiment of the application;

[0018] Figure 2 is a step flow chart of another state recognition method provided by an embodiment of the application;

[0019] Figure 3 is a step flow chart of a recognition of sitting and falling states provided by an embodiment of the application;

[0020] Figure 4 is a step flow chart of a recognition of walking and running states provided by an embodiment of the application;

[0021] Figure 5 is a step flow chart of a recognition of standing states provided by an embodiment of the application;

[0022] Figure 6 is a step flow chart of a state recognition method based on a neural network provided by an embodiment of the application;

[0023] Figure 7 is a flow chart of an execution step of a neural network provided by an embodiment of the present application;

[0024] Figure 8 is another flow chart of an execution step of a neural network provided by an embodiment of the present application;

[0025] Figure 9 is a flow chart of a training method of a neural network provided by an embodiment of the present application;

[0026] Figure 10 is another flow chart of a training method of a neural network provided by an embodiment of the present application;

[0027] Figure 11 is a flow chart of a state recognition method based on millimeter wave perception provided by an embodiment of the present application;

[0028] Figure 12 is another flow chart of a state recognition method based on millimeter wave perception provided by an embodiment of the present application;

[0029] Figure 13 is a block diagram of a state recognition device provided by an embodiment of the present application;

[0030] Figure 14 is a block diagram of a state recognition device based on a neural network provided by an embodiment of the present application;

[0031] Figure 15 is a block diagram of a training device of a neural network provided by an embodiment of the present application;

[0032] Figure 16 is a block diagram of a state recognition device based on millimeter wave perception provided by an embodiment of the present application. DETAILED DESCRIPTION

[0033] Exemplary embodiments of the present application will be described herein below with reference to the accompanying drawings. While exemplary embodiments of the present application are shown in the drawings, it is understood that the present application can be embodied in various forms and should not be limited by the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the application to those skilled in the art.

[0034] Referring to Figure 1 , a flow chart of a state recognition method provided by an embodiment of the present application is shown, which can include:

[0035] Step 101, obtaining a target attribute corresponding to a user, wherein the target attribute includes position information and speed information.

[0036] In the embodiment of the present application, when the radar collects data of users located in the target area, the human body will reflect millimeter waves at different parts, so each user corresponds to multiple target points, and the attributes of the target points can include position information, speed information, signal-to-noise ratio information, and the like.

[0037] In addition, according to the attributes of the multiple target points, the target attributes of the corresponding user can be determined. The target attributes can be understood as taking the user as a whole to represent the position and speed of the user and the like. In one example, the attributes of the multiple target points can be processed by a preset tracking algorithm to determine the corresponding target attributes. In another example, the target attributes can be determined by taking the mean, median, or the like of the attributes of the multiple target points.

[0038] Therefore, the target attributes can be obtained by taking the mean or median of the attributes of the multiple target points. According to the target attributes, the position and speed change of the user as a whole can be determined. Therefore, the target attributes can include position information and speed information and the like. According to the differences in the basic information such as position information and speed information of the user corresponding to different states, the different states can be divided into multiple state groups.

[0039] In one example, the state of the user can include at least one of the following: sitting, falling, standing, walking, and running. For example, sitting or falling can be determined as a group of state groups, walking or running as a group of state groups, and the like. Then, the state corresponding to the user can be accurately obtained by using a neural network or a recognition method that distinguishes the differences in the related information such as position information and speed information corresponding to the same state group.

[0040] In step 102, the feature data of the user change is determined according to the position information and the speed information.

[0041] In the embodiment of the present application, in order to be able to distinguish different states into multiple state groups according to the differences in the position information and the speed information between the different states corresponding to the user, the position information and the speed information corresponding to the user in consecutive multiple frames can be recorded.

[0042] For example, according to the position information, such as the Z-axis information obtained, the height value corresponding to the user can be determined, and the displacement of the user in the XY plane can be determined by the X-axis and Y-axis information obtained, and the speed value corresponding to the user can be determined according to the relationship between the displacement and time. Thus, the height value and the speed value corresponding to the user in consecutive multiple frames can be determined as feature data. The feature data can be used as basic information to determine the state groups corresponding to different states of the user.

[0043] Step 103, matching the feature data by a preset state threshold to determine the state of the user.

[0044] In the embodiment of the application, the state of each user can be determined according to the feature data obtained according to the preset identification rule, and output. In one example, the state threshold and the like can be determined according to the difference in the change of the feature data when the user is in different states. The state matching relationship between the state threshold and the feature data is established as the preset identification rule. In another example, the state matching relationship and the neural network can be combined together as the preset identification rule. The state corresponding to the user can include walking, running, sitting, standing and falling, and the like.

[0045] In summary, the state recognition method provided by the embodiment of the application obtains the target attribute corresponding to the user, wherein the target attribute can include position information and speed information. According to the position information and the speed information, the height change, the horizontal displacement change and the speed change of the user can be determined, and the feature data corresponding to each state of the user is formed, so that the specific state of the user is determined by matching the feature data by a preset state threshold. It can be applied to a non-contact environment, and does not need a camera to collect information, and has the advantages of protecting privacy and good comfort

[0046] Referring to Figure 2 , another state recognition method provided by the embodiment of the application is shown in a step flowchart, which can include:

[0047] Step 201, obtaining the target attribute corresponding to the user, wherein the target attribute includes position information and speed information.

[0048] In the embodiment of the application, when the radar collects data of the user located in the target area, the millimeter wave is reflected by different parts of the human body, so that each target point corresponding to each user has multiple target points, and the attributes of the target points can include position information, speed information and signal-to-noise ratio information.

[0049] In addition, the target attribute of the corresponding user can be determined according to the attributes of the multiple target points. The target attribute can be understood as taking the user as a whole to represent the position and speed of the user. Therefore, the target attribute can be obtained by taking the mean or median of the attributes of the multiple target points. According to the target attribute, the position and speed change of the user as a whole can be determined. Therefore, the target attribute can include position information and speed information. According to the difference in the basic information such as position information and speed information of the user in different states, different states can be divided into multiple state groups.

[0050] In an example, the state of the user can include at least one of the following: sitting, falling, standing, walking and running. For example, sitting or falling can be determined as a group of state groups, walking or running as a group of state groups, etc. Then, the state corresponding to the user is accurately obtained by using a neural network or a distinguishing manner that distinguishes the difference between the position information, speed information and other related information corresponding to the same state group.

[0051] In an optional embodiment, the feature data of the user is determined according to the position information and the speed information, which can include the following steps: step 202-step 203.

[0052] Step 202, the height and speed of the user in the continuous multiple frames are determined according to the position information and the speed information.

[0053] Step 203, the height and speed are taken as the feature data of the user.

[0054] In the embodiment of the application, in order to distinguish different states into a plurality of state groups according to the difference between the position information and the speed information corresponding to different states of the user, the position information and the speed information corresponding to the user in the continuous multiple frames can be recorded.

[0055] According to the position information, for example, the Z-axis information obtained, the height value corresponding to the user can be determined, the displacement of the user in the XY plane can be determined by the X-axis and Y-axis information obtained, and the speed value corresponding to the user can be determined according to the relationship between the displacement and the time. Thus, the height value and the speed value corresponding to the user in the continuous multiple frames can be determined as the feature data. The feature data can be used as the basic information to determine the state groups corresponding to different states of the user.

[0056] In an example, the state threshold value can include a height threshold value and a first speed threshold value. The state of the user is determined by matching the feature data with the preset state threshold value, which can include the following steps: step 204-step 208.

[0057] Step 204, it is determined whether the current height of the user is greater than the preset height threshold value.

[0058] In the case where the current height of the user is greater than the preset height threshold value, step 206 is performed; in the case where the current height of the user is not greater than the preset height threshold value, step 207 is performed.

[0059] In the embodiment of the present application, the height threshold can be understood as a height value for dividing the state group. For example, the height value of the user's state of walking, running and standing is obviously higher than that of the user's state of sitting and falling. Therefore, a height threshold can be set in advance according to the actual application scenario, and according to the height threshold, the user state can be divided into two state classes. The first state class includes walking, running and standing, and the second state class includes sitting and falling.

[0060] Therefore, in an example, whether the current height of the user is greater than the corresponding height threshold can be determined according to the obtained feature data. The mean value of the height values of the continuous multiple frames can be taken and matched with the height threshold. In the case that the current height of the user is greater than the preset height threshold, it can be determined that the current user state class is the first state class. In the case that the current height of the user is not greater than the preset height threshold, it can be determined that the current state class is the second state class.

[0061] Step 205, determining whether the speed of the user in the continuous multiple frames is less than a first speed threshold.

[0062] When the speed of the user in the continuous multiple frames is less than the first speed threshold, step 208 is performed, and when the speed of the user in the continuous multiple frames is not less than the first speed threshold, step 209 is performed.

[0063] Step 206, when the height of the user in the continuous multiple frames decreases, determining that the state of the user is a first state group, wherein the first state group includes sitting and falling.

[0064] Step 207, determining that the state of the user is a second state group, wherein the second state group includes standing.

[0065] Step 208, determining that the state of the user is a third state group, wherein the third state group includes walking and running.

[0066] In the embodiment of the present application, the first speed threshold can be understood as a speed value for dividing the corresponding state in the state class on the basis of dividing the user state into two state classes. For example, when the user is in a standing state, the speed value should be zero. There is a significant difference between the speed value when the user is in a walking or running state. Therefore, the first speed threshold can be set in advance, and according to the first speed threshold, the standing state can be distinguished from the second state class.

[0067] Therefore, in an example, it can be determined according to the acquired feature data whether the speed value of the user in the continuous multiple frames is less than the corresponding first speed threshold. Wherein, the average of the speed values of the continuous multiple frames is taken and matched with the first speed threshold. When the speed of the user in the continuous multiple frames is not less than the first speed threshold, it can be determined that the corresponding state of the user is walking or running. When the speed of the user in the continuous multiple frames is less than the first speed threshold, it can be determined that the corresponding state of the user is standing. In addition, in order to prevent the height corresponding to the user from gradually decreasing from the standing state to the sitting or falling state from above the height threshold to below the height threshold, in order to improve the accuracy of distinguishing the state groups, when the current height of the user is greater than the preset height threshold, the height change trend of the user in the continuous multiple frames can be determined, and when the height corresponding to the user is continuously decreased, it can be determined that the corresponding state of the user is sitting or falling. Thus, the sitting and falling states can be divided into a first state group, the standing state can be divided into a second state group, and the walking and running states can be divided into a third state group.

[0068] In an optional inventive embodiment, after determining the state group corresponding to the user, the specific state of the user can be recognized by a neural network.

[0069] Step 209, according to the target state group, the attributes of a plurality of target points corresponding to the user are acquired.

[0070] Step 210, the attributes of the plurality of target points are input into the neural network corresponding to the target state group for processing, and the state corresponding to the user is output.

[0071] In the embodiment of the application, when the radar collects data of the user located in the target area, the millimeter wave is reflected by different parts of the human body, so that each user has a plurality of target points corresponding thereto, and the attributes of the target points can include position information, speed information, and signal-to-noise ratio information.

[0072] In addition, the artificial neural network (Artificial Neural Networks, ANN), also known as neural network, refers to an algorithm mathematical model for distributed parallel information processing. Therefore, the attributes of the plurality of target points corresponding to the user can be simultaneously input into the neural network, so that the attributes of the target points can be analyzed in detail based on the numerous attribute characteristics of the target points, and thus the specific state of each user can be determined based on the target state group.

[0073] After determining the target state group corresponding to the user state, the attributes of the plurality of target points are input into the neural network corresponding to the target state group for processing. The number of neural networks is the same as the number of state groups with at least two or more states. For example, the first state group and the third state group each contain two states, so the neural network can include a neural network for distinguishing between sitting and falling, and a neural network for distinguishing between walking and running.

[0074] In an example, the position information is represented by three-dimensional coordinates such as X-axis, Y-axis, and Z-axis. Therefore, the attributes of the target points can be divided into coordinate X, coordinate Y, coordinate Z, speed V, and signal-to-noise ratio S. Five attribute features are counted, and the corresponding attribute features are input into the target neural network in parallel for the target state group, so as to extract the features of the plurality of target points corresponding to the user.

[0075] In addition, in order to extract time and space information at the same time, each attribute feature is input into the target neural network in the form of a three-dimensional matrix. For example, the input of the target neural network can be a P*t*1 matrix. Wherein P refers to the number of target points in each frame, and t refers to time, which can be selected according to the specific application scenario.

[0076] The neural network can extract relevant attribute features from the input three-dimensional information and perform flattening operation on the corresponding three-dimensional information. The flattening operation refers to flattening the corresponding three-dimensional matrix into a one-dimensional vector. Then, the neural network maps the probabilities corresponding to different states. Thus, the state corresponding to the maximum probability can be determined as the state of the user.

[0077] For example, the output of the neural network for distinguishing between sitting and falling can be a one-dimensional vector of 1*2, denoted as [R1, R2] T , wherein R1 represents the probability of the state being sitting, and R2 represents the probability of the state being falling.

[0078] The output of the neural network for distinguishing between walking and running can be a one-dimensional vector of 1*2, denoted as [N1, N2] T , wherein N1 represents the probability of the state being walking, and N2 represents the probability of the state being running. The softmax function maps the overall feature information to the range of (0, 1). For example, the one-dimensional vector output by the neural network for distinguishing between sitting and falling is [0.11, 0.89] T , so the one-dimensional vector can determine that the probability of the state being falling is the maximum value, and thus the state of the user is determined to be falling. For another example, the one-dimensional vector output by the neural network for distinguishing between walking and running is [0.73, 0.27] TThus, the one-dimensional vector can determine that the probability of the state of walking is the maximum value, and thus determine that the state corresponding to the user is walking.

[0079] In an optional embodiment, the state recognition process is avoided due to the movement of different body parts of the user, so that the state determined according to the target attribute of the user does not match the actual state of the user. Therefore, the state corresponding to the user can be determined from the corresponding target state group by combining the previous state of the user and the corresponding threshold, thereby improving the accuracy of the state recognition of the user. The previous state of the user refers to the state of the user output in the previous frame.

[0080] Specifically, after determining the target state group corresponding to the user, the specific state of the user can be recognized according to the difference between the previous state and the basic information such as the position information and the speed information corresponding to the same state group.

[0081] In the case where the target state group is the first state group, the state of the user is determined to be sitting or falling, and the method can further include sub-steps S31-S37.

[0082] In the case where the target state group is the first state group, the state of the user is determined to be sitting or falling, and the method can further include sub-steps S31-S37.

[0083] Specifically, according to the position information of the continuous multiple frames, the height change and the horizontal position change corresponding to the user can be determined, so that the height drop difference of the user in the continuous multiple frames can be determined according to the height change, and the horizontal displacement of the user in the continuous multiple frames can be determined according to the horizontal position change.

[0084] Sub-step S32, determining whether the height drop difference is greater than a preset difference threshold.

[0085] In the case where the height drop difference is greater than the preset difference threshold, sub-step S33 is performed; otherwise, sub-step S37 is performed.

[0086] Sub-step S33, determining whether the horizontal displacement is greater than a preset displacement threshold.

[0087] When the horizontal displacement is greater than the preset displacement threshold, sub-step S35 is performed.

[0088] When the horizontal displacement is not greater than the preset displacement threshold, the previous state and the current speed corresponding to the user are obtained, and sub-step S34 is performed.

[0089] Sub-step S34, determining whether the previous state corresponding to the user is falling.

[0090] In a case where the last state corresponding to the user is a state other than falling, sub-step S36 is performed.

[0091] In a case where the last state corresponding to the user is falling, sub-step S37 is performed.

[0092] Sub-step S35, determining that the current state of the user is falling.

[0093] Sub-step S36, determining that the current state of the user is sitting when the current speed is less than the first speed threshold.

[0094] Sub-step S37, determining that the current state of the user is the same as the last state.

[0095] In an embodiment of the present application, when the user remains in the same state, the height value corresponding to the user will fluctuate within a certain range due to the movement of a part of the body. Therefore, a difference threshold can be set in advance, and according to the difference threshold, it can be determined whether the height change corresponding to the user occurs when the user remains in the same state or when the user changes between different states.

[0096] In an example, the obtained height drop difference is matched with the preset difference threshold, and in a case where the height drop difference is not greater than the difference threshold, it is indicated that the user remains in the same state, and it is determined that the current state of the user is the same as the last state and output.

[0097] In a case where the height drop difference is greater than the difference threshold, it is indicated that the user has changed the state. At this time, the horizontal displacement can be used to determine whether the user has moved horizontally based on the change of the state. Specifically, in a case where the state of the user changes to sitting, the horizontal displacement corresponding to the user will fluctuate within a certain range due to the movement of a part of the body. In a case where the state of the user changes to falling, the horizontal displacement corresponding to the user is obviously different from the horizontal displacement corresponding to the sitting state. Therefore, a displacement threshold can be set in advance, and according to the displacement threshold, it can be determined whether the user has changed the horizontal position obviously based on the change of the state.

[0098] Therefore, in an example, the obtained horizontal displacement is matched with the preset displacement threshold, in a case where the horizontal displacement is greater than the preset displacement threshold, it is determined that the user has moved horizontally obviously, and it can be identified that the current state of the user is falling. In a case where the horizontal displacement is not greater than the preset displacement threshold, the last state corresponding to the user can be obtained. In a case where the last state is falling, it is indicated that the current state of the user is also falling and is the same as the last state; otherwise, the current speed of the user is obtained, and in a case where the current speed is less than the first speed threshold, it is determined that the current state of the user is sitting.

[0099] In an example, when the known target state group is the third state group, the method can further comprise: sub-step S41-sub-step S42.

[0100] Sub-step S41, determining whether the current speed of the user is greater than a preset second speed threshold.

[0101] When the current speed of the user is not greater than the preset second speed threshold, sub-step S42 is executed; when the current speed of the user is greater than the preset second speed threshold, sub-step S43 is executed.

[0102] In the embodiment of the application, the second speed threshold can be understood as a speed value for dividing a plurality of states in the third state group on the basis of the third state group. For example, there is a difference in speed when the user is in a walking or running state. Therefore, by pre-setting the second speed threshold, the walking or running state is preliminarily distinguished according to the second speed threshold. The second speed threshold is greater than the first speed threshold.

[0103] Therefore, in an example, the current speed of the user obtained is matched with the second speed threshold, when the current speed of the user is not greater than the second speed threshold, the current state of the user is predicted to be walking; when the current speed of the user is greater than the second speed threshold, the current state of the user is predicted to be running.

[0104] Sub-step S42, determining whether the last state of the user is any one of the first state group. If yes, sub-step S44 is executed; otherwise, sub-step S46 is executed.

[0105] Sub-step S43, determining whether the last state of the user is any one of the first state group. If yes, sub-step S45 is executed; otherwise, sub-step S47 is executed.

[0106] Sub-step S44, determining whether the frame number of the user maintaining the walking state is greater than a preset count threshold. If yes, sub-step S46 is executed; otherwise, sub-step S48 is executed.

[0107] Sub-step S45, determining whether the frame number of the user maintaining the running state is greater than a preset count threshold. If yes, sub-step S47 is executed; otherwise, sub-step S49 is executed.

[0108] Sub-step S46, determining that the current state of the user is walking.

[0109] Sub-step S47, determining that the current state of the user is running.

[0110] Sub-step S48, counting the frame number of the user maintaining the walking state, and determining that the current state of the user is the same as the last state.

[0111] Sub-step S49, count the frame number of the user maintaining the state of running, and determine that the current state of the user is the same as the previous state.

[0112] In the embodiment of the application, before determining that the current state of the user is walking or running, the possibility of measurement error caused by the movement of the corresponding body part of the user needs to be considered. For example, when the previous state of the user is any state in the first state group, there may be a condition that the current speed is greater than or less than the second speed threshold, but the duration of this condition is very short. Therefore, after matching the current speed of the user according to the second speed threshold, the previous state of the user is obtained to pre-set a count threshold to exclude this condition, thereby improving the accuracy of state recognition.

[0113] Therefore, if the previous state of the user is any state in the first state group, the frame number of the user continuously maintaining the state of walking or running is determined. Wherein, the matching of the current speed of the user and the second speed threshold is completed once for each target attribute obtained. For example, when the current speed of the user is continuously detected to be greater than the second speed threshold, the frame number of the state of running is matched according to the preset count threshold. When the current speed of the user is continuously detected to be not greater than the second speed threshold, the frame number of the state of walking is matched according to the preset count threshold.

[0114] Specifically, when the frame number of maintaining the state of walking or running is not greater than the count threshold, it is determined that the current state of the user is the same as the previous state. When the frame number of the user maintaining the state of walking is greater than the preset count threshold, it is determined that the current state of the user is walking. When the frame number of the user maintaining the state of running is greater than the count threshold, it is determined that the current state of the user is running.

[0115] Correspondingly, when the previous state of the user is not any state in the first state group, when the current speed of the user is not greater than the second speed threshold, it is determined that the current state of the user is walking. When the current speed of the user is greater than the second speed threshold, it is determined that the current state of the user is running. And the state corresponding to the user can be output.

[0116] In an example, when it is determined that the current state of the user is standing, the method can further include sub-steps S51-S54.

[0117] Sub-step S51, in the case that the previous state of the user corresponds to any state in the first state group, the height rising difference value and the rising trend of the user in consecutive frames are determined according to the position information.

[0118] Sub-step S52, determining whether the height rising difference value is greater than a preset difference threshold and the user's rising trend is continuous rising, if yes, executing sub-step S53; otherwise, executing sub-step S54.

[0119] Sub-step S53, determining that the user's current state is standing.

[0120] Sub-step S54, determining that the user's current state is the same as the previous state.

[0121] In the embodiment of the present application, the height values of the corresponding states in the first state group and the second state group are different, so that it can be determined whether the previous state of the user is one of the states in the first state group to predict the state of the user. In one example, in the case that the previous state of the user is sitting or falling, the height values of the user in continuous frames can be obtained through the position information corresponding to the target attribute, so as to determine the height rising difference value and the rising trend of the user.

[0122] If the height rising difference value of the user is greater than the height difference threshold and the height of the user in continuous frames is in a continuous rising trend, it indicates that the state of the user has been converted, so that it can be determined that the current state of the user is standing and the state is output. Otherwise, the height change of the user is understood as normal fluctuation within a certain range when the user maintains the same state, so that it is determined that the current state of the user is the same as the previous state.

[0123] In another example, in the case that the previous state of the user is not one of the states in the first state group, the current height of the user can be obtained through the position information corresponding to the target attribute, and if the current height of the user is greater than the height threshold, it can be determined that the current state of the user is standing and the state is output.

[0124] In summary, the other state recognition method provided by the embodiment of the present application obtains the target attribute corresponding to the user, wherein the target attribute can include position information and speed information. According to the position information and the speed information, the height change, the horizontal displacement change and the speed change of the user can be determined to form the feature data corresponding to the states of the user, so that the specific state of the user is determined by matching the feature data with the preset state threshold. It can be applied in a non-contact environment without the need of a camera for information collection, and has the advantages of privacy protection and good comfort.

[0125] Referring to Figure 6 , a step flowchart of a state recognition method based on a neural network provided by an embodiment of the present application is shown, which can include:

[0126] Step 601, obtaining the attributes of a plurality of target points corresponding to a user.

[0127] The attribute of the target point includes position information, speed information and signal-to-noise ratio information.

[0128] In step 602, the attributes of the plurality of target points are input into the neural network for processing, and the state corresponding to the user is output.

[0129] In the embodiment of the application, when the radar collects data of the user located in the target area, the millimeter wave is reflected by different parts of the human body, so that each user corresponds to multiple target points, and the attributes of the target points can include position information, speed information and signal-to-noise ratio information.

[0130] In addition, the artificial neural network (ANN), also known as neural network, refers to an algorithmic mathematical model for distributed parallel information processing. Therefore, the attributes of the plurality of target points corresponding to the user can be simultaneously input into the preset neural network, so that the attributes of the target points can be analyzed in detail based on the various attribute characteristics of the target points, thereby accurately determining the state corresponding to each user. The state corresponding to the user can include walking, running, jumping, lying, squatting, sitting, standing and falling, etc.

[0131] In an example, the position information is represented by three-dimensional coordinates such as X-axis, Y-axis and Z-axis. Therefore, the attributes of the target points can be divided into coordinate X, coordinate Y, coordinate Z, speed V and signal-to-noise ratio S, and the five attribute characteristics are counted and input into the neural network in parallel, so as to extract the attribute characteristics of the plurality of target points corresponding to the user.

[0132] In addition, in order to simultaneously extract time and space information, each attribute characteristic is input into the neural network in the form of a three-dimensional matrix, for example, the input of the neural network can be a P*t*1 matrix. Wherein P refers to the number of target points in each frame, and t refers to time, which can be selected according to the specific application scenario.

[0133] The neural network can extract relevant attribute characteristics from the input three-dimensional information, and perform flattening operation on the corresponding three-dimensional information. The flattening operation refers to flattening the corresponding three-dimensional matrix into a one-dimensional vector. Then the probability corresponding to different states is mapped by the neural network. Thus, the state corresponding to the maximum probability can be determined as the state of the user.

[0134] Specifically, the neural network can include a convolutional layer and a fully connected layer, and the neural network performs the following processing steps:

[0135] Sub-step S71, the attributes of the plurality of target points are convoluted through the convolution layer to obtain corresponding overall feature information.

[0136] In the embodiment of the application, the respective attribute features corresponding to the plurality of target points are input into the convolution layer, and the attribute features obtained are subjected to convolution operation according to a preset convolution kernel, so that the corresponding attribute features can be subjected to feature extraction, as the overall feature information corresponding to the user, wherein the overall feature information can be understood as related information obtained by convolution operation after concatenation of the plurality of attribute features after convolution extraction.

[0137] Sub-step S72, the overall feature information is input into the fully connected layer for classification to obtain the state corresponding to the user.

[0138] In the embodiment of the application, the fully connected layer can be understood as a simple multi-classification sub-network, and the neural network subjects the overall feature information in the form of a three-dimensional matrix expression to flattening operation, which refers to flattening the corresponding three-dimensional matrix into a one-dimensional vector. Then the one-dimensional vector is input into the fully connected layer to map the overall feature information into probabilities corresponding to different states, which are output from the fully connected layer. Therefore, the state corresponding to the maximum probability can be obtained, and the state is determined as the state corresponding to the user.

[0139] For example, the output of the fully connected layer can be a one-dimensional vector of 1*8, denoted as [Z1, Z2, Z3, Z4, Z5, Z6, Z7, Z8] T , wherein Z1 represents the probability of the state being walking, Z2 represents the probability of the state being running, Z3 represents the probability of the state being jumping, Z4 represents the probability of the state being lying, Z5 represents the probability of the state being squatting, Z6 represents the probability of the state being sitting, Z7 represents the probability of the state being standing, and Z8 represents the probability of the state being falling. The overall feature information can be mapped between (0, 1) according to a normalized exponential function, for example, a softmax function, for example, the output one-dimensional vector is [0.11, 0.14, 0, 0, 0.05, 0.1, 0.6, 0] T . From the one-dimensional vector, it can be determined that the probability of the state being standing is the maximum value, and therefore the state corresponding to the user is determined to be standing.

[0140] In an alternative embodiment of the application, since the data is subjected to multiple convolution stacking for feature extraction in the neural network, the accuracy of the final classification is improved to a certain extent. Therefore, a plurality of convolution layers can also be set to perform deep feature extraction on the attribute features of the target points.

[0141] In an example, the convolution layer can further include a first convolution layer, a second convolution layer, and a third convolution layer, and the step of convolving the attributes of the plurality of target points through the convolution layer to obtain corresponding overall feature information can include the following steps: sub-step S81 to sub-step S83.

[0142] In sub-step S81, the first convolution layer respectively convolves the attributes of the target points to determine a plurality of low-dimensional attribute features.

[0143] In the embodiment of the present application, the attribute features corresponding to the plurality of target points are respectively input into the first convolution layer, and a low-dimensional attribute feature is determined by performing convolution operation on the obtained attribute features according to a preset first convolution kernel.

[0144] In an optional embodiment of the present application, the low-dimensional attribute feature is subjected to a max-pooling process.

[0145] In the embodiment of the present application, max-pooling refers to a pooling operation that reduces the amount of data by means of maximum value. Therefore, the max-pooling operation can be performed on the low-dimensional attribute feature to reduce the parameters of the neural network without affecting the result of state recognition, thereby reducing the calculation cost and space occupation.

[0146] In sub-step S82, the second convolution layer is used to convolve the low-dimensional attribute feature to determine a plurality of high-dimensional attribute features.

[0147] In the embodiment of the present application, the second convolution layer performs convolution operation on the input low-dimensional attribute feature based on a preset second convolution kernel to determine a high-dimensional attribute feature. The second convolution layer can include a plurality of sub-convolution layers. Sub-step S82 can include the following steps:

[0148] The sub-convolution layer convolves the low-dimensional attribute feature to obtain an intermediate attribute feature.

[0149] The intermediate attribute feature and the low-dimensional attribute feature are subjected to a residual operation to determine a plurality of high-dimensional attribute features.

[0150] In the embodiment of the present application, since the data is stacked and extracted features in the neural network, to a certain extent, it is beneficial to improve the accuracy of the final classification. Therefore, the relevant feature information can also be extracted by setting multiple sub-convolution layers, and at the same time, with the superposition of network layers, the neural network will degenerate, and when the neural network degenerates, the network with shallow structure has better training effect than the network with deep structure. Therefore, the residual unit can be added to the multiple sub-convolution layer structure, and the residual operation is performed on the corresponding feature information in the feature information extraction process. The residual operation can be understood as adding the input of a sub-convolution layer to the output of the sub-convolution layer as the input of the next convolution layer. Then the next step is processed, so as to solve the corresponding neural network degeneration.

[0151] In an example, the second convolutional layer can include a first sub-convolutional layer and a second sub-convolutional layer. The first sub-convolutional layer extracts corresponding feature information after performing convolution operation on the input low-dimensional attribute feature. The second sub-convolutional layer performs convolution operation on the feature information extracted by the first sub-convolutional layer to determine intermediate attribute features. At this time, the intermediate attribute features and the low-dimensional attribute features are performed once residual operation to obtain high-dimensional attribute features. Through the attribute features of different target points, multiple high-dimensional attribute features can be extracted.

[0152] In another example, the first sub-convolutional layer, the second sub-convolutional layer and the residual module can be combined into one residual module, and the number of residual modules can be selected according to the actual application scene. For example, two residual modules in series are configured in the second convolutional layer.

[0153] In sub-step S83, the multiple high-dimensional attribute features are connected and convolution processing is performed by the third convolutional layer to determine the overall feature information of the user.

[0154] In the embodiment of the present application, the connection means that the high-dimensional attribute features corresponding to each attribute feature are connected in series. For example, multiple three-dimensional matrices are stacked to form a three-dimensional matrix. Therefore, the third convolutional layer can perform convolution operation on the input high-dimensional attribute features based on the preset third convolutional kernel to extract the overall feature information corresponding to the user.

[0155] The low-dimensional attribute features, the high-dimensional attribute features and the overall feature information mentioned above are represented in the form of three-dimensional matrix data. The size and number of the first convolutional kernel, the second convolutional kernel and the third convolutional kernel can be selected according to actual conditions. For example, the size can be selected as 3*3, 5*5 and 7*7.

[0156] In an optional embodiment of the present application, the overall feature information is subjected to average pooling processing.

[0157] In the embodiment of the present application, the average pooling refers to a kind of pooling operation, which reduces the data amount by taking the average value. Therefore, the corresponding parameters of the neural network can be reduced by performing the average pooling operation on the low-dimensional attribute features, so as to reduce the calculation cost and the space occupation, reduce the overfitting, and improve the fault tolerance of the neural network without affecting the result of state recognition.

[0158] The sub-step S84 is inputting the overall feature information into the full connection layer for classification to obtain the state corresponding to the user.

[0159] In the embodiment of the present application, the full connection layer can be understood as a simple multi-classification sub-network. The neural network performs a flattening operation on the overall feature information in the form of a three-dimensional matrix according to the input overall feature information. The flattening operation refers to flattening the corresponding three-dimensional matrix into a one-dimensional vector. Then, the one-dimensional vector is input into the full connection layer to map the overall feature information into probabilities corresponding to different states, which are output from the full connection layer. Therefore, the state corresponding to the maximum probability can be obtained, and the state is determined as the state corresponding to the user.

[0160] For example, the output of the full connection layer can be a one-dimensional vector of 1*8, denoted as [Z1, Z2, Z3, Z4, Z5, Z6, Z7, Z8] T , wherein Z1 represents the probability of the state being walking, Z2 represents the probability of the state being running, Z3 represents the probability of the state being jumping, Z4 represents the probability of the state being lying, Z5 represents the probability of the state being squatting, Z6 represents the probability of the state being sitting, Z7 represents the probability of the state being standing, and Z8 represents the probability of the state being falling. The overall feature information can be mapped to (0, 1) according to a normalized exponential function, for example, a softmax function. For example, the output one-dimensional vector is [0.11, 0.14, 0, 0, 0.05, 0.1, 0.6, 0] T . From the one-dimensional vector, it can be determined that the probability of the state being standing is the maximum value, and therefore the state corresponding to the user is determined to be standing.

[0161] An alternative embodiment of the present application, the method can further comprise:

[0162] After performing the convolution processing operation, the feature information obtained by the convolution processing is normalized.

[0163] An alternative embodiment of the present application, the method can further comprise:

[0164] After performing the convolution processing operation, the feature information obtained by the convolution processing is activated by a predetermined activation function.

[0165] In the embodiment of the present application, after performing the convolution processing operation each time, the feature information obtained by convolution can also be normalized and / or activated. Specifically, the normalization processing refers to mapping the relevant attribute features to the range of 0-1, and the activation operation refers to adding a preset activation function in the neural network, so that the nonlinear factor is introduced into the neural network to solve the problems that cannot be solved by the linear model. The normalization processing and / or the activation operation can make the neural network converge faster and improve the performance of the neural network.

[0166] In summary, the state recognition method based on the neural network provided by the embodiment of the present application can obtain a plurality of target point attributes corresponding to the state of a user, wherein the attributes of the target point can include position information, speed information and signal-to-noise ratio information. The neural network can split the target point attributes into a plurality of attribute features for parallel analysis, so as to accurately determine a plurality of states corresponding to the user. The method can be applied to a non-contact environment, and does not need a camera to collect information, and has the advantages of protecting privacy and good comfort.

[0167] Reference Figure 9 The training method of the neural network provided by the embodiment of the present application can include the following steps:

[0168] In step 901, the historical target point attributes of the state of a user are collected, and the state of the user includes at least one of the following: walking, running, jumping, lying, squatting, sitting, standing and falling.

[0169] The attributes of the target point include position information, speed information and signal-to-noise ratio information.

[0170] In the embodiment of the present application, the state of the user can include walking, running, jumping, lying, squatting, sitting, standing and falling. Therefore, one of the states can be selected as a target state. Then, the historical reflection points of the user in the target region in the target state are collected by the millimeter wave radar, and the historical target point attributes corresponding to the target state are determined after tracking the historical reflection points.

[0171] In step 902, the historical target point attributes are input into the neural network for processing, and the corresponding state is output.

[0172] In step 903, the parameters of the neural network are adjusted according to the target state and the state.

[0173] In the embodiment of the present application, a plurality of historical target point attributes are input into the neural network, so that the neural network performs the following steps:

[0174] The attributes of the plurality of target points are convolved by the convolution layer to obtain corresponding overall feature information.

[0175] inputting the whole feature information into the full connection layer for classification to obtain the state corresponding to the user.

[0176] Thus, the state corresponding to the historical target point attribute is determined by the above. The state is compared with the target state to determine the loss function corresponding to the neural network, such as using a cross-entropy function as the loss function. Therefore, according to the loss function, the parameters of the neural network can be adjusted by using an optimizer, wherein the optimizer can be an Adam (adaptive moment estimation) function or the like. When the matching degree of the state determined by the historical target point attribute and the target state reaches a matching threshold, such as 98%, it can be determined that the neural network training is completed.

[0177] Referring to Figure 10 , another method for training a neural network is shown, which can include:

[0178] obtaining the historical target point attribute corresponding to the target state can include the following steps: step 1001-step 1002.

[0179] Step 1001, the state corresponding to the user is divided into at least one state group, and the group includes at least two states.

[0180] Step 1002, the historical target point attribute corresponding to the target state of the at least one state group is collected, and the historical target point attribute corresponding to the at least one state group is determined.

[0181] The attributes of the target point include position information, speed information, and signal-to-noise ratio information.

[0182] In the embodiment of the application, the state corresponding to the user can include sitting, falling, standing, walking, and running. After determining the basic state group of the above user state, at least two states in each group are distinguished. For example, sitting and falling in the first state group, and walking and running in the second state group are distinguished. Therefore, any one state in each state group can be selected as the target state in turn. Then the historical reflection point of the user in the target area in the target state is collected by the millimeter wave radar, and the historical target point attribute corresponding to each state group is determined after tracking the historical reflection point.

[0183] Step 1003, inputting the historical target point attribute into the neural network for processing to output the corresponding state.

[0184] Step 1004, adjusting the parameters of the neural network according to the target state and the state.

[0185] In the embodiment of the present application, a plurality of historical target point attributes are input into the neural network, so that the neural network performs the following steps:

[0186] The attributes of the plurality of target points are convoluted through the convolution layer to obtain corresponding overall feature information.

[0187] The overall feature information is input into the fully connected layer for classification to obtain the corresponding state of the user.

[0188] Thus, the state corresponding to the historical target point attribute is determined by the above. The state is compared with the target state to determine the loss function corresponding to the neural network, such as using a cross-entropy function as the loss function. Therefore, according to the loss function, the parameters of the neural network can be adjusted by using an optimizer, wherein the optimizer can be an Adam function. When the matching degree between the state determined by the historical target point attribute and the target state reaches a matching threshold, such as 98%, it can be determined that the neural network training is completed.

[0189] Referring to Figure 11 , a step flowchart of a state recognition method based on millimeter wave perception provided by an embodiment of the present application is shown, which can include:

[0190] Step 1101, acquiring a plurality of reflection points corresponding to at least one user through a millimeter wave radar.

[0191] In the embodiment of the present application, millimeter wave, as a valuable perception technology, can be used to detect targets and provide distance, speed and angle information of the corresponding targets. At the same time, millimeter wave can provide sub-millimeter accuracy and can penetrate certain specific materials, such as plastic, clothing, etc., and is not easily affected by environmental conditions such as rain, fog, dust and snow.

[0192] Specifically, after determining the target area detected by the millimeter wave, the user located in the target area is detected by the millimeter wave radar. The millimeter wave radar can use FMCW (Frequency Modulated Continuous Wave, frequency modulated continuous wave) millimeter wave radar. Since different parts of the human body will reflect the signals emitted by the millimeter wave radar, the millimeter wave radar will perform signal processing after receiving the corresponding plurality of reflection signals to obtain the related attributes of the plurality of reflection points, for example, the attributes of the reflection points can include position information, speed information, signal-to-noise ratio information, etc.

[0193] In the case that the number of users located in the target area is uncertain, a plurality of reflection points in each frame are obtained by millimeter wave radar collection and processing, which are reflection point sets corresponding to a plurality of users, and therefore the plurality of collected reflection points need to be further processed to determine the reflection points corresponding to each user.

[0194] Step 1102, tracking the plurality of reflection points to determine the plurality of target points corresponding to each user.

[0195] In the embodiment of the application, all the reflection points obtained in each frame can be processed according to the preset tracking rule, so as to screen out the plurality of reflection points corresponding to each user, and determine the reflection points as the target points corresponding to the user. The attributes of the target points also include position information, speed information, signal-to-noise ratio information, etc. Therefore, the plurality of target points as the key reflection points representing the state of the user can accurately identify the state of each user through the attributes of the plurality of target points corresponding to the user.

[0196] Step 1103, outputting the state of each user according to the attributes of the plurality of target points.

[0197] In the embodiment of the application, the state of each user can be determined and output according to the attributes of the plurality of target points obtained in the continuous plurality of frames according to the preset identification rule. For example, a neural network can be used as the preset identification rule, and for another example, a state threshold value can be determined according to the attributes of the target points, a state matching relationship between the state threshold value and the attributes of the target points is established, the state matching relationship is used as the preset identification rule, and for another example, a neural network and a state matching relationship can be combined together as the preset identification rule. The state corresponding to the user can include walking, running, jumping, lying, squatting, sitting, standing and falling, etc.

[0198] In summary, the state recognition method based on millimeter wave perception provided by the embodiment of the application collects a plurality of reflection points of at least one user in a target area by a millimeter wave radar, processes and tracks the plurality of reflection points, screens out a plurality of reflection points corresponding to each user, and takes the reflection points as effective target points, accurately determines the state corresponding to each user according to the attributes of the plurality of target points. It can be applied in a non-contact environment, does not need a camera to collect information, and has the advantages of protecting privacy and good comfort.

[0199] Reference Figure 12 Another step flowchart of the state recognition method based on millimeter wave perception provided by the embodiment of the application is shown, which can include:

[0200] Step 1201, collecting a plurality of reflection points corresponding to at least one user by a millimeter wave radar.

[0201] In the embodiment of the present application, after determining the target area of millimeter wave detection, the user located in the target area is detected by the millimeter wave radar. The millimeter wave radar can adopt FMCW (Frequency Modulated Continuous Wave) millimeter wave radar. Since different parts of the human body will reflect the signal emitted by the millimeter wave radar, after the millimeter wave radar receives the corresponding multiple reflection signals, signal processing is performed to obtain the related attributes of multiple reflection points, for example, the attributes of the reflection points can include position information, speed information, signal-to-noise ratio information, etc.

[0202] In the case where the number of users located in the target area is not determined, the multiple reflection points in each frame obtained by the millimeter wave radar are the reflection point set corresponding to multiple users, therefore, the multiple reflection points collected need to be further processed to determine the reflection point corresponding to each user.

[0203] Tracking the multiple reflection points to determine the multiple target points corresponding to each user, including the following steps: step 1202-step 1203.

[0204] Step 1202, denoising processing is performed on the multiple reflection points.

[0205] Step 1203, correlation analysis is performed on the multiple reflection points after denoising processing to determine the multiple target points corresponding to each user.

[0206] In the embodiment of the present application, all the reflection points obtained in each frame can be processed according to the preset tracking rule, so as to screen the multiple reflection points corresponding to each user.

[0207] In an example, generating the preset tracking rule can include the following steps:

[0208] Based on the state recognition corresponding to the target area set by the user in advance, all the reflection points in each frame are screened. For example, if it is detected that the reflection point is located outside the target area, the reflection point is marked as a noise point, and the noise point is removed, thereby completing the denoising processing of the reflection point.

[0209] The multiple reflection points after denoising processing are clustered. The clustering can be understood as segmenting all the reflection points located in the target area according to the distance between the multiple reflection points associated with a single user, forming a reflection point set equal to the number of users in the target area, and calculating the centroid position of the corresponding reflection point set through each reflection point set after clustering.

[0210] According to the positions of the respective centroids, the frame of the set of reflection points is matched to the identity corresponding to the user clustered in the previous frame, and if there is no user identity in the previous frame that meets the matching condition, a new user identity is established based on the set of reflection points. Thus, the accuracy of the reflection point data obtained by continuously tracking the corresponding user in the target area is ensured.

[0211] Therefore, the reflection points in the set of reflection points corresponding to the user identity are determined as target points corresponding to the user, wherein the attributes of the target points also include position information, speed information, signal-to-noise ratio information, etc. The plurality of target points can be key reflection points representing the state of the user, and the state of each user can be accurately identified through the attributes of the plurality of target points corresponding to the user.

[0212] According to the attributes of the plurality of target points, the state of each user is output, including the following step: step 1204.

[0213] Step 1204, the attributes of the plurality of target points are input into a neural network for processing, and the state corresponding to each user is output.

[0214] In the embodiment of the application, when the radar collects data of the user located in the target area, the millimeter wave is reflected by different parts of the human body. Therefore, each user has a plurality of target points, and the attributes of the target points can include position information, speed information, and signal-to-noise ratio information.

[0215] In addition, artificial neural networks (ANN), also known as neural networks, refer to an algorithmic mathematical model for distributed parallel information processing. Therefore, the attributes of the plurality of target points corresponding to the user can be simultaneously input into a predetermined neural network, so that the attributes of the target points can be analyzed in detail based on the numerous attribute characteristics of the target points, thereby accurately determining the state corresponding to each user. The state corresponding to the user can include walking, running, jumping, lying, squatting, sitting, standing, and falling, etc.

[0216] In an example, the position information is represented by three-dimensional coordinates such as X-axis, Y-axis, and Z-axis. Therefore, the attributes of the target points can be divided into coordinate X, coordinate Y, coordinate Z, speed V, and signal-to-noise ratio S, and the corresponding attribute characteristics are input into the neural network in parallel, so as to extract the attribute characteristics of the plurality of target points corresponding to the user.

[0217] In addition, in order to extract time and space information at the same time, each attribute feature is input into the neural network in the form of a three-dimensional matrix, for example, the input of the neural network can be a P*t*1 matrix. Wherein P refers to the number of target points in each frame, t refers to time, and the above parameters can be selected according to the specific application scenario.

[0218] The neural network can extract relevant attribute features from the input three-dimensional information, and perform flattening operation on the corresponding three-dimensional information. Wherein, the flattening operation refers to flattening the corresponding three-dimensional matrix into a one-dimensional vector. Then the neural network maps the probability corresponding to different states. Thus, the state corresponding to the maximum probability can be determined as the state of the user.

[0219] In an alternative embodiment, the method can further comprise:

[0220] Monitoring the state of the user, and forming the activity track corresponding to the user according to the state;

[0221] Outputting prompt information to the user according to the activity track.

[0222] In the embodiment, the state of the user can be monitored for a long time, and the state change of the corresponding user is recorded in time sequence, and the state change and the position change of the user are determined as the activity track of the user in the corresponding period. Specifically, the reminding time length of the activity track can be set in advance, such as 3h, 4h, etc., for example, when it is monitored that the user maintains the same activity track for more than the reminding time length, prompt information will be output to the user, for example, when the user maintains the sitting state for a long time and exceeds the reminding time length, the voice will send the prompt message of sedentary reminder, other health guidance suggestions, etc. to the user. The method can be applied to the scene of analyzing the home activity habits of family members.

[0223] In an alternative embodiment, the method is applied to a home control system, and the home control system comprises at least one of the following: a lighting system, an air conditioning system. The method further comprises:

[0224] Adjusting the home control system according to the activity track corresponding to the user.

[0225] In the embodiment of the present application, the method can also be applied to a home control system, wherein the home control system can include but is not limited to a lighting system, an air conditioning system, etc. Therefore, the home control system can be adjusted according to the motion trajectory of the user in different time periods. In an example, the lighting system and / or the air conditioning system can be turned on after it is monitored that the user enters a target area or gets up at night. The lighting system and / or the air conditioning system can be automatically turned off after it is monitored that the user sleeps or leaves the target area. In another example, the light intensity of the corresponding light in the lighting system and the air conditioning gear of the corresponding air conditioning system can also be adjusted according to the corresponding number of users and activity trajectory. Thus, the method can be widely applied to the field of smart home control and energy saving, which can provide a convenient and comfortable living environment for the user, prevent power waste, and achieve the effect of energy saving and environmental protection.

[0226] In an optional embodiment of the present application, the method can further include:

[0227] According to the activity trajectory, the health information of the user is determined and stored.

[0228] In the embodiment of the present application, the activity trajectory of the user is monitored and analyzed, and the health information representing the health state of the user can be determined. In an example, the activity trajectory of the user in the night period can be obtained, the number and time of getting up of the user are determined through the activity trajectory, and the sleep quality of the user is analyzed according to the number and time of getting up, for example, is divided into multiple levels such as excellent, good, and poor. The level information representing the sleep quality can be stored as health information. The method can be applied to the environment of health state monitoring of the user, and can interact with the intelligent medical system to make the intelligent medical system obtain the health information of the user in a period of time, and provide objective reference data for doctors in diagnosis and treatment.

[0229] In an optional embodiment of the present application, the method can further include:

[0230] According to the activity trajectory, the rescue condition of the user is determined.

[0231] In the case that the user needs rescue, the rescue information is output.

[0232] In the embodiment of the present application, the activity track of a user is monitored, and the activity track is analyzed to determine whether the user is in a situation requiring rescue in different corresponding states. In an example, it is analyzed from the activity track that the state of the user is falling, and no state change is monitored in a period of time. It is determined that the user needs to be rescued, and rescue information is automatically sent to the family or caregiver of the user to request help. In addition, especially in the application scenario of activity track monitoring of an elderly person living alone, rescue information can be immediately sent to the family or caregiver of the user in the case that the state of the user is falling, so that the user can receive timely rescue at the first time.

[0233] To sum up, another state recognition method based on millimeter wave perception provided by the embodiment of the present application is provided. A plurality of reflection points of at least one user in a target region are collected by a millimeter wave radar, and the plurality of reflection points are tracked and processed. A plurality of reflection points corresponding to each user are screened out and used as effective target points. According to the attributes of the plurality of target points, the state corresponding to each user is accurately determined. It can be applied in a non-contact environment, does not need a camera to collect information, and has the advantages of protecting privacy and good comfort.

[0234] Reference Figure 13 A state recognition device provided by the embodiment of the present application is shown. The device can include:

[0235] A target point acquisition module 1301 is configured to acquire target attributes corresponding to a user, wherein the target attributes include position information and speed information.

[0236] A data determination module 1302 is configured to determine characteristic data of the user according to the position information and the speed information.

[0237] A state matching module 1303 is configured to match the characteristic data by a preset state threshold to determine the state of the user.

[0238] An optional embodiment of the present application, the data determination module can include:

[0239] A first information determination unit is configured to determine the height and speed of the user in a plurality of continuous frames according to the position information and the speed information.

[0240] A characteristic data generation unit is configured to take the height and speed as the characteristic data of the user.

[0241] An optional embodiment of the present application, the state threshold includes a height threshold and a first speed threshold. The state matching module can include:

[0242] The first state unit is configured to determine a state of the user as a first state group when the height of the user in the continuous frames decreases, if the current height of the user is not greater than a preset height threshold, wherein the first state group comprises sitting and falling.

[0243] The second state unit is configured to determine the state of the user as a second state group when the speed of the user in the continuous frames is less than a first speed threshold, if the current height of the user is greater than the preset height threshold, wherein the second state group comprises standing.

[0244] The third state unit is configured to determine the state of the user as a third state group when the speed of the user in the continuous frames is not less than the first speed threshold, wherein the third state group comprises walking and running.

[0245] An optional embodiment of the application, the apparatus can further comprise:

[0246] The first information determination module is configured to determine a height difference and a horizontal displacement of the user in the continuous frames according to the position information.

[0247] When the horizontal displacement is greater than a preset displacement threshold, the current state of the user is determined as falling, if the height difference is greater than a preset difference threshold.

[0248] When the horizontal displacement is not greater than the preset displacement threshold, a previous state and a current speed of the user are monitored.

[0249] When the previous state of the user is a state other than falling, the current state of the user is determined as sitting, if the current speed is less than the first speed threshold.

[0250] An optional embodiment of the application, the apparatus can further comprise:

[0251] The second information determination module is configured to determine a height difference of the user in the continuous frames according to the position information, if the previous state of the user is any state in the first state group.

[0252] When the height difference is greater than a preset difference threshold, and the height of the user in the continuous frames is continuously increasing, the current state of the user is determined as standing.

[0253] An optional embodiment of the application, the apparatus can further comprise:

[0254] The third information determination module is configured to determine the current state of the user as running, if the current speed of the user is greater than a preset second speed threshold.

[0255] When the current speed of the user is not greater than a preset second speed threshold, it is determined that the current state of the user is walking.

[0256] An optional inventive embodiment, the third information determination module can include:

[0257] A previous state acquisition unit is configured to, when the current speed of the user is greater than a preset second speed threshold, acquire a previous state of the user.

[0258] A first state frame number matching unit is configured to, when the previous state is any one state in a first state group, match a frame number of the state of running according to a preset count threshold.

[0259] When the frame number of the state of running maintained by the user is greater than the preset count threshold, it is determined that the current state of the user is running.

[0260] An optional inventive embodiment, the third information determination module can further include:

[0261] A previous state acquisition unit is configured to, when the current speed of the user is not greater than a preset second speed threshold, acquire a previous state of the user.

[0262] A second state frame number matching unit is configured to, when the previous state is any one state in a first state group, match a frame number of the state of walking according to a preset count threshold.

[0263] When the frame number of the state of walking maintained by the user is greater than the preset count threshold, it is determined that the current state of the user is walking.

[0264] In summary, the state recognition device provided by the embodiment of the application acquires target attributes of a user, wherein the target attributes can include position information and speed information. According to the position information and the speed information, information such as height change, horizontal displacement change and speed change of the user can be determined, feature data corresponding to each state of the user is formed, and the feature data is matched by a preset state threshold to determine the specific state of the user. The device can be applied in a non-contact environment, does not need a camera to collect information, and has the advantages of protecting privacy and good comfort.

[0265] Reference Figure 14 is shown, which shows a state recognition device based on a neural network provided by an embodiment of the application. The device can include:

[0266] An attribute acquisition module 1401 is configured to acquire attributes of a plurality of target points of a user, wherein the attributes of the target points include position information, speed information and signal-to-noise ratio information.

[0267] The state determination module 1402 is configured to input the attributes of the plurality of target points into the neural network for processing, and output a state corresponding to the user.

[0268] The neural network includes a convolutional layer and a fully connected layer, and can include the following modules:

[0269] The overall feature extraction module is configured to perform convolutional processing on the attributes of the plurality of target points through the convolutional layer to obtain corresponding overall feature information.

[0270] The feature classification module is configured to input the overall feature information into the fully connected layer for classification to obtain a state corresponding to the user.

[0271] In an optional embodiment, the convolutional layer includes a first convolutional layer, a second convolutional layer, and a third convolutional layer. The overall feature extraction module can include:

[0272] The first feature determination unit is configured to perform convolutional processing on the attributes of the target points through the first convolutional layer to determine a plurality of low-dimensional attribute features.

[0273] The second feature determination unit is configured to perform convolutional processing on the low-dimensional attribute features through the second convolutional layer to determine a plurality of high-dimensional attribute features.

[0274] The overall feature extraction unit is configured to connect the plurality of high-dimensional attribute features and perform convolutional processing through the third convolutional layer to determine the overall feature information of the user.

[0275] In an optional embodiment, the apparatus can further include:

[0276] The max-pooling module is configured to perform max-pooling processing on the low-dimensional attribute features.

[0277] In an optional embodiment, the apparatus can further include:

[0278] The average-pooling module is configured to perform average-pooling processing on the overall feature information.

[0279] In an optional embodiment, the second convolutional layer includes a plurality of sub-convolutional layers. The second feature determination unit can include:

[0280] The intermediate feature determination sub-unit is configured to perform convolutional processing on the low-dimensional attribute features through the sub-convolutional layer to obtain intermediate attribute features.

[0281] The second feature extraction sub-unit is configured to perform a residual operation on the intermediate attribute features and the low-dimensional attribute features to determine a plurality of high-dimensional attribute features.

[0282] An optional embodiment of the application, the device can further comprise:

[0283] After performing the convolution processing operation, the feature information obtained by the convolution processing is normalized.

[0284] An optional embodiment of the application, the device can further comprise: after performing the convolution processing operation, the feature information obtained by the convolution processing is activated by a preset activation function.

[0285] In summary, the state recognition method based on the neural network provided by the embodiment of the application, by obtaining a plurality of target point attributes corresponding to the user state, wherein the attributes of the target point can include position information, speed information and signal-to-noise ratio information. The neural network splits the target point attributes into a plurality of attribute features for parallel analysis, so as to accurately determine the multiple states corresponding to the user. It can be applied to a non-contact environment, without the need for a camera to collect information, and has the advantages of protecting privacy and good comfort.

[0286] Referring to Figure 15 , a neural network training device provided by an embodiment of the application is shown, which can include:

[0287] The point cloud sequence acquisition module 1501 is configured to obtain historical target point attributes corresponding to a target state, wherein the attributes of the target point include position information, speed information and signal-to-noise ratio information.

[0288] The state generation module 1502 is configured to input the historical target point attributes into a neural network for processing and output a corresponding state.

[0289] The parameter adjustment module 1503 is configured to adjust parameters of the neural network according to the target state and the state.

[0290] An optional embodiment of the application, the point cloud sequence acquisition module can include:

[0291] The first attribute acquisition unit is configured to acquire historical target point attributes corresponding to a state of a user, wherein the state of the user includes at least one of the following: walking, running, jumping, lying, squatting, sitting, standing and falling.

[0292] An optional embodiment of the application, the point cloud sequence acquisition module can further include:

[0293] The state grouping unit is configured to divide the state of the user into at least one state group, wherein the group includes at least two states.

[0294] The second attribute acquisition unit is configured to acquire historical target point attributes of the target state corresponding to the at least one state group, and determine the historical target point attributes corresponding to the at least one state group.

[0295] Referring to Figure 16 The application embodiment provides a state recognition device based on millimeter wave perception, which can include:

[0296] The data acquisition module 1601 is configured to acquire a plurality of reflection points corresponding to at least one user by using a radar.

[0297] The data tracking module 1602 is configured to track the plurality of reflection points, and determine a plurality of target points corresponding to each user.

[0298] The state output module 1603 is configured to output the state of each user according to the attributes of the plurality of target points.

[0299] An optional application embodiment, the device can further include:

[0300] The trajectory monitoring module is configured to monitor the state of the user, and form an activity trajectory corresponding to the user according to the state.

[0301] According to the activity trajectory, the prompt information is output to the user.

[0302] An optional application embodiment, the device is applied to a home control system, and the home control system includes at least one of the following: a lighting system, an air conditioning system. The device can further include:

[0303] The adjustment module is configured to adjust the home control system according to the activity trajectory corresponding to the user.

[0304] An optional application embodiment, the device can further include:

[0305] The information storage module is configured to determine the health information corresponding to the user according to the activity trajectory, and store the health information.

[0306] An optional application embodiment, the device can further include:

[0307] The rescue confirmation module is configured to judge the rescue situation of the user according to the activity trajectory.

[0308] In the case where it is determined that the user needs rescue, rescue information is output.

[0309] An optional application embodiment, the data tracking module can include:

[0310] A de-noising unit is configured to de-noise the plurality of reflection points.

[0311] An association analysis unit is configured to perform association analysis on the plurality of de-noised reflection points to determine the state of each user corresponding to the plurality of target points.

[0312] In an optional embodiment, the attribute of the target point includes position information, speed information, and signal-to-noise ratio information. The state output module can be further configured to:

[0313] input the attribute of the plurality of target points into a neural network for processing, and output the state of each user corresponding to the plurality of target points.

[0314] In an optional embodiment, the neural network includes a convolutional layer and a fully connected layer, and the neural network can include the following modules:

[0315] An overall feature extraction module is configured to perform convolutional processing on the attribute of the plurality of target points through the convolutional layer to obtain corresponding overall feature information.

[0316] A feature classification module is configured to input the overall feature information into the fully connected layer for classification to obtain the state of each user.

[0317] In an optional embodiment, the attribute of the target point includes position information and speed information. The device can further include:

[0318] A data determination module is configured to determine characteristic data of the user according to the position information and the speed information.

[0319] A state matching module is configured to match the characteristic data with a preset state threshold to determine the state of the user.

[0320] In an optional embodiment, the state threshold includes a height threshold and a first speed threshold, and the characteristic data includes the height and the speed of the user in a plurality of continuous frames.

[0321] The state matching module can include:

[0322] A first state unit is configured to, in a case where the current height of the user is not greater than the preset height threshold and the height of the user in the plurality of continuous frames decreases, determine the state of the user as a first state group, wherein the first state group includes sitting and falling.

[0323] A second state unit is configured to, in a case where the current height of the user is greater than the preset height threshold and the speed of the user in the plurality of continuous frames is less than a first speed threshold, determine the state of the user as a second state group, wherein the second state group includes standing.

[0324] a third state unit, configured to determine the state of the user as a third state group when the speed of the user in consecutive frames is not less than a first speed threshold, wherein the third state group comprises walking and running.

[0325] An optional embodiment, the attribute of the target point further comprises: signal-to-noise ratio information, the device can further comprise:

[0326] A target point attribute acquisition module is configured to acquire attributes of a plurality of target points corresponding to the user according to the target state group.

[0327] The attributes of the plurality of target points are input into a neural network corresponding to the target state group for processing, and the state corresponding to the user is output.

[0328] In summary, the embodiment of the present application provides a state recognition device based on millimeter wave perception, which collects a plurality of reflection points of at least one user in a target area through a millimeter wave radar, and processes the plurality of reflection points, filters a plurality of reflection points corresponding to each user as effective target points, and accurately determines the state corresponding to each user according to the attributes of the plurality of target points. It can be applied in a non-contact environment, without the need for a camera to collect information, and has the advantages of protecting privacy and good comfort.

[0329] Each embodiment in the specification is described in a progressive manner, and each embodiment focuses on the difference from other embodiments. The same or similar parts between the embodiments can be referred to each other.

[0330] As can be readily perceived by those skilled in the art, any combination of the above embodiments is feasible, so any combination of the above embodiments is an embodiment of the present application. However, due to the limited space, the specification will not be described in detail here.

[0331] In the specification provided herein, a large number of specific details are described. However, it can be understood that the embodiments of the present application can be practiced without these specific details. In some examples, well-known methods, structures and techniques are not shown in detail in order not to obscure the understanding of the specification.

[0332] Similarly, it is to be understood that the embodiments of the present application can be alternately grouped together in a single embodiment, figure, or description thereof for the purpose of brevity and clarity. However, the disclosure is not to be interpreted that the claimed application requires more features than those expressly recited in each claim. Rather, as is reflected in the claims themselves, the inventive aspect lies in fewer than all features of a single disclosed embodiment. Accordingly, the claims as follows are hereby expressly incorporated into this detailed description, with each claim by itself precision as a separate embodiment of the application.

[0333] Those skilled in the art will appreciate that the modules in the apparatuses in the embodiments can be adapted and placed in one or more apparatuses other than the embodiments. The modules or units or components in the embodiments can be combined into one module or unit or component, and further can be divided into multiple sub-modules or sub-units or sub-components. Except that at least some of such features and / or processes or units are mutually exclusive, all combinations of all features disclosed in this specification (including the accompanying claims, abstract and drawings) and all processes or units of any methods or apparatuses disclosed thus can be taken, in any combination. Unless explicitly stated, each feature disclosed in this specification (including the accompanying claims, abstract and drawings) can be replaced by alternative features that serve the same, equivalent or similar purpose.

[0334] An electronic device, comprising:

[0335] one or more processors;

[0336] a memory;

[0337] one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs configured to perform the methods described in the above embodiments.

[0338] A computer readable storage medium storing a computer program for use in conjunction with an electronic device, the computer program executable by a processor to perform the methods described in the above embodiments.

[0339] Those skilled in the art will appreciate that embodiments of the present application can be readily used as a method, apparatus, or computer program product. Accordingly, embodiments of the present application can take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, embodiments of the present application can take the form of a computer program product on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage devices, and the like) embodying computer program instructions.

[0340] Embodiments of the present application are described herein with reference to the Figure 1 one or more functions specified in a flow or multiple flows and / or block Figure 1 means for performing the function specified by the block or blocks.

[0341] These computer program instructions can also be stored in a computer- readable memory that can direct a computer or other programmable data processing Figure 1 one or more functions specified in a flow or multiple flows and / or block Figure 1 means for performing the function specified by the block or blocks.

[0342] These computer program instructions can also be loaded onto a computer or other programmable data processing devices to cause a series of operational steps to be performed on the computer or other programmable devices to produce a computer implemented process such that the instructions which execute on the computer or other programmable devices provide steps for implementing the functions specified in the flow or multiple flows and / or blocks Figure 1 one or more functions specified in a flow or multiple flows and / or block Figure 1 means for performing the function specified by the block or blocks.

[0343] Although preferred embodiments of the present application have been described, those skilled in the art will recognize that additional modifications and changes can be made thereto without departing from the scope of the present application. Accordingly, the appended claims are intended to cover all such modifications and changes as fall within the scope of the present application.

[0344] Finally, it is to be understood that the phraseology or terminology such as "first" and "second" etc. used herein is merely intended to differentiate one entity or operation from another entity or operation, without necessarily requiring or implying any such actual relationship or order between such entities or operations. Moreover, the terms "comprising", "including", or any other closure, are intended to cover the non-exclusive inclusion such that a process, method, article, or apparatus that comprises a list of elements does not include those elements alone but can include other elements not expressly listed or even include elements inherent in such process, method, article, or apparatus. Without more limitations, the element defined by the statement "comprising a" does not exclude the presence of additional identical elements in the process, method, article, or apparatus including the element.

[0345] The above describes in detail the state recognition method and the state recognition device provided by the present application, and the principles and implementation manners of the present application are described by using specific examples. The above description of the embodiments is only used to help understand the method of the present application and its core idea. Meanwhile, for those skilled in the art, the specific implementation manners and application ranges will be changed according to the idea of the present application. In summary, the content of the specification should not be understood as a limitation of the present application.

Claims

1. A state recognition method, characterized in that, The method includes: Obtain the target attributes corresponding to the user, wherein the target attributes include: location information and speed information; Based on the location information and speed information, the characteristic data of the user's changes are determined, and the characteristic data includes height and speed; The user's status is determined by matching the feature data with a preset status threshold. The state thresholds include: a height threshold and a first speed threshold; The step of matching the feature data with a preset state threshold to determine the user's state includes: If the user's current height is not greater than a preset height threshold, when the user's height decreases in multiple consecutive frames, the user's state is determined to be a first state group, wherein the first state group includes sitting and falling. If the user's current height is greater than a preset height threshold, and the user's speed is less than a first speed threshold in multiple consecutive frames, the user's state is determined to be a second state group, wherein the second state group includes standing. When the user's speed is not less than a first speed threshold in multiple consecutive frames, the user's state is determined to be a third state group, wherein the third state group includes walking and running; Based on the location information, determine the user's height drop difference and horizontal displacement in multiple consecutive frames; If the height difference is greater than a preset difference threshold, and the horizontal displacement is greater than a preset displacement threshold, the user's current state is determined to be a fall. When the horizontal displacement is not greater than a preset displacement threshold, monitor the user's previous state and current speed. If the user's previous state was any state other than falling, and the current speed is less than the first speed threshold, then the user's current state is determined to be sitting. Based on the target state group corresponding to the user, obtain the attributes of multiple target points corresponding to the user; The attributes of the multiple target points are processed through the first convolutional layer of the neural network to determine multiple low-dimensional attribute features; The low-dimensional attribute features are processed through the second convolutional layer of the neural network to determine multiple high-dimensional attribute features; The multiple high-dimensional attribute features are connected and processed through the third convolutional layer of the neural network to determine the overall feature information of the user. The overall feature information is input into the fully connected layer of the neural network for classification to obtain the user's corresponding state.

2. The method according to claim 1, characterized in that, The step of determining the user's changing characteristic data based on the location information and speed information includes: Based on the location and speed information, the user's height and speed in multiple consecutive frames are determined; The height and speed are used as characteristic data of the user's changes.

3. The method according to claim 1, characterized in that, The method further includes: If the user's previous state is any one of the states in the first state group, the height increase difference of the user in consecutive frames is determined based on the location information. When the height difference is greater than a preset difference threshold, and the user's height is continuously increasing in multiple consecutive frames, the user's current state is determined to be standing.

4. The method according to claim 1, characterized in that, The method further includes: When the user's current speed is greater than a preset second speed threshold, the user's current state is determined to be running; When the user's current speed is not greater than a preset second speed threshold, the user's current state is determined to be walking.

5. The method according to claim 4, characterized in that, The step of determining the user's current state as running when the user's current speed is greater than a preset second speed threshold includes: When the user's current speed is greater than a preset second speed threshold, obtain the user's previous state; If the previous state is any one of the states in the first state group, the number of frames in the running state is matched according to a preset counting threshold. When the number of frames in which the user maintains the running state is greater than a preset counting threshold, the user's current state is determined to be running.

6. The method according to claim 4, characterized in that, The step of determining the user's current state as walking when the user's current speed is not greater than a preset second speed threshold includes: When the user's current speed is not greater than a preset second speed threshold, obtain the user's previous state; If the previous state is any one of the states in the first state group, the number of frames in the state of "walking" is matched according to a preset counting threshold. When the number of frames in which the user maintains the state of walking is greater than a preset counting threshold, the user's current state is determined to be walking.

7. A state recognition device, characterized in that, The device includes: The target point acquisition module is used to acquire the target attributes corresponding to the user, wherein the target attributes include: location information and speed information; The data determination module is used to determine the user's changing characteristic data based on the location information and speed information, wherein the characteristic data includes height and speed; The state matching module is used to match the feature data with a preset state threshold to determine the user's state; The state matching module includes: A first state unit is configured to determine the user's state as a first state group when the user's current height is not greater than a preset height threshold and the user's height decreases in consecutive frames, wherein the first state group includes sitting and falling. The second state unit is used to determine the user's state as a second state group when the user's current height is greater than a preset height threshold and the user's speed is less than a first speed threshold in multiple consecutive frames, wherein the second state group includes standing. The third state unit is used to determine the user's state as a third state group when the user's speed is not less than a first speed threshold in multiple consecutive frames, wherein the third state group includes walking and running. The first information determination module is used to determine the user's height drop difference and horizontal displacement in consecutive frames based on the location information. A module for determining that the user's current state is a fall when the difference in height descent is greater than a preset difference threshold and the horizontal displacement is greater than a preset displacement threshold. A module for monitoring the user's previous state and current speed when the horizontal displacement is not greater than a preset displacement threshold. A module for determining that the user's current state is sitting when the current speed is less than the first speed threshold, provided that the user's previous state was any state other than falling. A module for obtaining the attributes of multiple target points corresponding to the user based on the target state grouping corresponding to the user; A module for processing the attributes of the multiple target points through the first convolutional layer of a neural network to determine multiple low-dimensional attribute features; A module for processing the low-dimensional attribute features through the second convolutional layer of the neural network to determine multiple high-dimensional attribute features; This module is used to connect multiple high-dimensional attribute features, process them through the third convolutional layer of the neural network, and determine the overall feature information of the user. A module for inputting the overall feature information into the fully connected layer of the neural network for classification to obtain the user's corresponding state.

8. An electronic device, comprising: One or more processors; Memory; One or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs being configured to perform the method of any one of claims 1-6.

9. A computer-readable storage medium storing a computer program for use in conjunction with an electronic device, said computer program being executable by a processor to perform the method of any one of claims 1-6.

Images