Human activity recognition model training method and recognition method for human activities containing confusing activities

Abstract

The application provides a human activity recognition model training method and recognition method containing confusing activities, the training method comprises the following steps: training a neural network model based on a plurality of point cloud frame sequences carrying human activity classification labels, so as to train the neural network model into a human activity recognition model used for outputting classification results according to the point cloud frame sequences carrying human activity classification labels; wherein the point cloud frame sequences carrying human activity classification labels are obtained based on initial point cloud frame sequences; the neural network model is obtained in advance according to a pre-trained human activity understanding and reconstruction model, the human activity understanding and reconstruction model comprises sequentially connected understanding modules and reconstruction modules; the neural network model comprises sequentially connected understanding modules after knowledge transfer and a classifier. The application can accurately recognize various human activities under the premise of non-contact and non-invasion of privacy by using the human dynamics associated with the point cloud sequence learned in the pre-training stage.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of human activity recognition and machine learning, and particularly relates to a human activity recognition model training method and a recognition method. BACKGROUND

[0002] Human activity recognition refers to a technology that provides us with in-depth understanding and prediction of human activities by analyzing and interpreting the behavior patterns of individuals or groups in real time. It has shown its indispensable value in multiple fields. For example, in the field of security monitoring, activity perception can timely identify abnormal behaviors of workers to avoid safety accidents; in the field of health care, activity perception can provide an additional protection layer for the elderly or individuals with special needs, and can predict the occurrence of diseases through long-term behavior prediction; in addition, in the field of smart home, activity perception can automatically adjust the environmental settings of the home, such as light, temperature and security system, according to the behavior habits of the occupants, thereby improving the comfort and convenience of living.

[0003] At present, the mainstream activity recognition methods mainly include two categories: a wearable device-based scheme and an optical device-based scheme. However, these two schemes have the defects of low comfort and privacy leakage, which hinders their widespread use in daily life. Specifically, the wearable device-based scheme requires the subject to wear one or more small devices, and analyzes the activity state of the person based on the acceleration, angle and other information measured by the built-in inertial measurement unit (IMU). However, this scheme has poor comfort and needs to be charged regularly, and people's discomfort limits its widespread use. The optical device-based scheme usually analyzes human activity based on images. However, optical sensors involve privacy issues in daily scenes and are limited by lighting conditions, and cannot be widely applied in daily scenes, especially in home environments.

[0004] In recent years, emerging wireless sensing technologies such as millimeter wave sensors have emerged, which have the potential for continuous sensing of human activities in daily scenes. Since human activities will cause corresponding disturbances to wireless signals, people have begun to explore human activity recognition schemes based on wireless sensing technology. However, existing wireless sensing-based schemes do not consider the diversity of human activities in daily scenes, and most daily scenes do not belong to ideal environments. Activity recognition in daily scenes is easily confused, especially for activities that only produce movement of part of the body, so it cannot be truly applied in daily scenes. SUMMARY

[0005] In view of this, the embodiments of the present application provide a human activity recognition model training method and a recognition method to eliminate or improve one or more defects in the prior art.

[0006] The first aspect of the application provides a human activity recognition model training method, which comprises the following steps:

[0007] training a first neural network model based on a plurality of point cloud frame sequences carrying human activity classification labels, so as to train the first neural network model into a human activity recognition model for outputting human activity classification results according to the point cloud frame sequences carrying human activity classification labels;

[0008] wherein the point cloud frame sequences carrying human activity classification labels are obtained based on initial point cloud frame sequences;

[0009] The first neural network model is obtained in advance according to a human activity understanding and reconstruction model, the human activity understanding and reconstruction model comprises a human activity understanding module and a human activity reconstruction module, the human activity understanding module is used for extracting associated human dynamics features in the initial point cloud frame sequences, and the human activity reconstruction module is used for outputting a reconstructed point cloud frame sequence according to the extracted human dynamics features; the first neural network model comprises a human activity understanding module after knowledge transfer and a human activity classifier, the human activity understanding module after knowledge transfer is used for extracting human dynamics features in the input point cloud frame sequence, and the human activity classifier is used for outputting human activity classification results according to the extracted human dynamics features.

[0010] In some embodiments of the application, the human activity understanding and reconstruction model is pre-trained by the following steps:

[0011] pre-training a second neural network model according to a plurality of initial point cloud frame sequences by minimizing a loss function based on optimal transportation and feature learning, so as to train the second neural network model into a human activity understanding and reconstruction model for outputting a reconstructed point cloud frame sequence according to the initial point cloud frame sequences;

[0012] wherein the initial point cloud frame sequences are obtained based on millimeter wave signals generated for human activities; the second neural network model and the human activity understanding and reconstruction model have the same network structure.

[0013] In some embodiments of the application, the human activity understanding module in the second neural network model and the human activity understanding and reconstruction model comprises a point cloud feature extractor, a mask module and a human activity understanding device;

[0014] The point cloud feature extractor is used for extracting high-dimensional features of the initial point cloud frame sequence, and outputting a high-dimensional feature point cloud frame sequence;

[0015] The mask module is used for randomly masking the output high-dimensional feature frame sequence according to a proportion, and outputting a high-dimensional feature point cloud frame sequence containing a mask feature frame.

[0016] The human activity understander is used to extract human dynamic features associated with unmasked high-dimensional feature point cloud frames from the output high-dimensional feature point cloud frame sequence containing masked feature frames, and outputs a high-dimensional human dynamic feature frame sequence containing masked feature frames.

[0017] The human activity reconstruction module includes a human activity feature reconstructor and a point cloud generator;

[0018] The human activity feature reconstructor is used to recover the output high-dimensional human dynamic feature frame sequence containing mask feature frames, and outputs a reconstructed high-dimensional feature point cloud frame sequence.

[0019] The point cloud generator is used to reconstruct the output high-dimensional feature point cloud frame sequence and generate a corresponding reconstructed point cloud frame sequence.

[0020] In some embodiments of the present invention, the loss function based on optimal transportation and feature learning includes a loss function based on optimal transportation and a loss function based on feature learning. The loss function based on optimal transportation is used to optimize the distance between corresponding points in corresponding point cloud frames in the generated reconstructed point cloud frame sequence and the corresponding initial point cloud frame sequence, and to optimize the point cloud distribution of the corresponding reconstructed point cloud frame sequence according to the initial point cloud frame sequence. The loss function based on feature learning is used to optimize the feature difference between the output reconstructed high-dimensional feature point cloud frame sequence and the corresponding high-dimensional feature point cloud frame sequence.

[0021] In some embodiments of the present invention, the knowledge-transferred human activity understanding module includes a point cloud feature extractor and a human activity understander. The point cloud feature extractor is used to extract high-dimensional features from the input point cloud frame sequence and output a corresponding high-dimensional feature point cloud frame sequence. The human activity understander is used to extract human dynamic features from the output high-dimensional feature point cloud frame sequence and output a corresponding high-dimensional human dynamic feature frame sequence.

[0022] The human activity classifier is used to identify the output high-dimensional human dynamics feature frame sequence and output the corresponding human activity classification result.

[0023] In some embodiments of the present invention, the human dynamics characteristics include continuity characteristics within the same human activity and continuity characteristics between different human activities.

[0024] A second aspect of the present invention provides a method for human activity recognition, the method comprising the following steps:

[0025] Obtain the initial point cloud frame sequence of the target generated based on millimeter-wave signals for human activity;

[0026] input the target initial point cloud frame sequence into a human activity recognition model, so that the human activity recognition model outputs a human activity classification result corresponding to the target initial point cloud frame sequence, wherein the human activity recognition model is obtained by pre-training according to the human activity recognition model training method in the first aspect.

[0027] A third aspect of the present application provides an electronic device, comprising: a computer device, the computer device comprising a processor and a memory, the memory storing computer instructions, and the processor being configured to execute the computer instructions stored in the memory, so that the device implements the steps of the human activity recognition model training method in the first aspect or the steps of the human activity recognition method in the second aspect.

[0028] A fourth aspect of the present application provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the steps of the human activity recognition model training method in the first aspect or the steps of the human activity recognition method in the second aspect.

[0029] A fifth aspect of the present application provides a computer program product, comprising computer instructions, which are executed by a processor to implement the steps of the human activity recognition model training method in the first aspect or the steps of the human activity recognition method in the second aspect.

[0030] The human activity recognition model training method and the human activity recognition method of the present application generate a millimeter wave point cloud by utilizing the disturbance of human activity on a millimeter wave signal, especially for the case that the millimeter wave point cloud is missing when only part of the body (local part of the body) moves, and based on a self-supervised neural network algorithm, the neural network understands the association between various activities of the human body and the millimeter wave point cloud in the pre-training task, learns the associated human dynamics in the human activity, so as to realize accurate recognition of various types of activities in the fine-tuning task.

[0031] Further, the method can realize accurate recognition of easily confused human activities under various intensity activity conditions in daily scenarios under the premise of non-contact and non-invasion of privacy by using millimeter wave wireless signals.

[0032] Additional advantages, objects, and features of the application will be set forth in part by the description that follows, and will become apparent to those skilled in the art upon examination of the following or can be learned by practice of the application. The objects and other advantages of the application can be realized and attained by the structure particularly pointed out in the written description and claims hereof as well as the appended drawings.

[0033] Those skilled in the art will understand that the objects and advantages of the application can be realized and attained by means of the application especially pointed out in the written description and claims hereinafter. The application will be more fully understood and appreciated by referring to the following detailed description taken in conjunction with the accompanying drawings of which: BRIEF DESCRIPTION OF DRAWINGS

[0034] The accompanying drawings, which are included to provide a further understanding of the application and are incorporated in and constitute a part of this application, illustrate embodiments of the application and together with the description serve to explain the principles of the application.

[0035] Figure 1 A flowchart of a human activity recognition model training method in an embodiment of the application;

[0036] Figure 2 A process diagram of a model pre-training and fine-tuning stage in an embodiment of the application;

[0037] Figure 3 A representation content diagram of human dynamics features in an embodiment of the application;

[0038] Figure 4 A structure diagram of a second neural network model in an embodiment of the application;

[0039] Figure 5 A structure diagram of a point cloud feature extractor in an embodiment of the application;

[0040] Figure 6 A diagram of a loss function based on optimal transport and feature learning in an embodiment of the application;

[0041] Figure 7 A structure diagram of a first neural network model in an embodiment of the application;

[0042] Figure 8 A flowchart of a human activity recognition method in an embodiment of the application. DETAILED DESCRIPTION

[0043] To make the objects, technical solutions and advantages of the application clearer, further detailed description will be given to the application in conjunction with the embodiments and drawings. Herein, the illustrative embodiments of the application and the description thereof are used to explain the application but not to limit the application.

[0044] It should be noted that, in order to avoid the application being obscured by unnecessary details, only the structures and / or processing steps closely related to the solutions according to the application are shown in the drawings, and other details not closely related to the application are omitted.

[0045] It should be emphasized that the term "comprises / comprising" when used in this specification is taken to specify the presence of stated features, elements, steps or components but does not preclude the presence or addition of one or more other features, elements, steps, components, or groups thereof.

[0046] It should also be noted herein that the term "connected" is used herein to refer both direct and indirect connections, unless otherwise specifically stated.

[0047] Embodiments of the present application will be described below with reference to the accompanying drawings. In the drawings, like reference numerals represent the same or similar components, or the same or similar steps.

[0048] Human activities in daily scenarios mainly include two types: "local part activity" (only part of the body moves) and "whole body activity" (all parts of the body move). The recognition confusion of human activities mostly occurs in the "local part activity" state, and people spend more than 80% of their time in this state in daily scenarios. Through quantitative observation, it is found that in the "whole body activity" state, point clouds are almost distributed in each part of the human body; while in the "local part activity" state, point clouds are only distributed on the active part of the human body. In the millimeter wave field of view, both describe the overall motion state of the human body, but the point cloud in the "local part activity" state is obviously not the case, so it is easy to cause confusion in the recognition of activities in this state, which is specifically manifested in that two different activities produce point cloud sequences with consistent trends, for example, shaking legs while sitting is easily confused with falling, and thus is incorrectly recognized as falling. The key reason for the above problem is that the current research algorithm is only limited to learning the human point cloud itself, and ignores the connection between the point cloud generated by the millimeter wave signal for human activity and the human activity in daily scenarios, that is, the related human dynamics knowledge in the point cloud.

[0049] Therefore, the embodiments of the present application provide a human activity recognition model training method and a recognition method. Figure 1 The flowchart of a human activity recognition model training method in an embodiment of the present application is shown in FIG. 1. As shown in the figure, the method comprises the following steps: Figure 1

[0050] Step S110, training a first neural network model based on a plurality of point cloud frame sequences carrying human activity classification labels, so as to train the first neural network model into a human activity recognition model for outputting human activity classification results according to the point cloud frame sequences carrying human activity classification labels;

[0051] Among them, the point cloud frame sequence carrying human activity classification labels is obtained based on an initial point cloud frame sequence;

[0052] ​The first neural network model is pre-obtained based on a human activity understanding and reconstruction model, which includes a human activity understanding module and a human activity reconstruction module. The human activity understanding module is used to extract human dynamic features associated with the initial point cloud frame sequence, and the human activity reconstruction module is used to output a reconstructed point cloud frame sequence based on the extracted human dynamic features. The first neural network model also includes a knowledge-transferred human activity understanding module and a human activity classifier. The knowledge-transferred human activity understanding module is used to extract human dynamic features from the input point cloud frame sequence, and the human activity classifier is used to output a human activity classification result based on the extracted human dynamic features.

[0053] This method employs a self-supervised learning approach, such as... Figure 2 As shown, the human activity understanding and reconstruction model first learns the correlation and relationship between various activities performed by people in daily scenes and their corresponding point clouds through pre-training. This enables the model to understand various human activities and human dynamics in daily scenes. Then, in the fine-tuning stage, the pre-trained human activity understanding and reconstruction model undergoes knowledge transfer and fine-tuning to pre-obtain the first neural network model. The fine-tuning involves removing the human activity reconstruction module from the model, retaining only the human activity understanding module, and performing knowledge transfer on this module. A human activity classifier is then added to the knowledge-transferred human activity understanding module, aiming to utilize the human dynamics learned in the pre-training stage to accurately classify daily activities. The specific operation of knowledge transfer is freezing the weight parameters. A small number of labeled point cloud frame sequences representing the continuity of local and whole-body human activities are input into the first neural network model for iterative training, enabling the trained human activity recognition model to output accurate human activity classification results.

[0054] Therefore, before training the fine-tuned model, it is necessary to obtain a small number of initial point cloud frame sequences and label each of the obtained initial point cloud frame sequences with their respective labels. These labels are used to indicate the type of human activity to which the initial point cloud frame sequence belongs, i.e., human activity classification labels. They can be part or all of the following local and whole-body activity types: various hand activities such as waving and stretching forward when sitting, which are easily confused with walking and standing; various hand activities such as standing, which are easily confused with walking; various leg activities such as shaking and stretching legs when sitting, which are easily confused with falling; and activity types such as walking, standing, sitting, and falling.

[0055] In some embodiments, the human activity understanding and reconstruction model is pre-trained through the following steps:

[0056] Step S010, pre-training a second neural network model according to a plurality of initial point cloud frame sequences by minimizing a loss function based on optimal transport and feature learning, so as to train the second neural network model into a human activity understanding and reconstruction model for outputting a reconstructed point cloud frame sequence according to the initial point cloud frame sequence;

[0057] The initial point cloud frame sequence is generated based on a millimeter wave signal for a human activity; the second neural network model and the human activity understanding and reconstruction model have the same network structure.

[0058] When the initial point cloud frame sequence used for training the model is obtained, a frequency-modulated continuous wave (FMCW) radar of various frequency bands can be used to emit FMCW millimeter wave signals to continuously perceive human activities, and then receive and process the changed reflection signals generated due to the human activities, and generate an initial point cloud frame sequence for the human activities based on the changed reflection signals. The point cloud in the sequence frame describes the coordinates of the most active part of the human activity. In the process of generating the initial point cloud frame sequence, a millimeter wave single-frame point cloud is first generated using a signal processing algorithm, and then a continuous single-frame point cloud generated for a human activity forms an initial point cloud frame sequence. The specific process of the signal processing algorithm is as follows: ① distance Fourier transform is performed on the millimeter wave signal to obtain a distance map; ② static filtering algorithm is used to filter the static information in the environment on the distance map; ③ Capon beamforming algorithm is used to process the azimuth angle direction virtual antenna data in the filtered distance map to generate a distance-azimuth angle matrix; ④ constant false alarm rate algorithm is used to remove noise in the distance-azimuth angle matrix to obtain distance and azimuth angle information of the target; ⑤ Capon beamforming algorithm and Doppler Fourier transform are used to process the data of the target distance information in the elevation angle virtual antenna direction of the azimuth angle information to obtain the elevation angle and radial velocity information of the target, respectively. Finally, a set of points with position and velocity information distributed on the surface of the human body is obtained.

[0059] In fact, in the process of pre-training and training the fine-tuned model, the point cloud frames contained in the point cloud frame sequence data iteratively trained in the input model can be continuous frames or discontinuous frames.

[0060] In the pre-training process, the unlabeled initial point cloud frame sequence is input into the second neural network model, so that the second neural network model learns the association and connection between different types of human activities in daily scenarios and millimeter wave point clouds, that is, understands different types of human activities and human dynamics in daily scenarios. The model understands and learns human dynamics mainly by analyzing the "continuity within the same activity" and "continuity between different activities" of daily activities by the second neural network model. For example, Figure 3As shown in the content of the background art, the human dynamics features include continuity features within the same activity of the human body and continuity features between different activities of the human body. Specifically, (1) for the continuity of the same activity: when the human body is performing an activity, the representation of the activity is usually composed of a series of continuous point cloud frames. For the same activity, whether it is a short-term or long-term activity, the action in the later frame is relatively easy to predict according to the action in the previous frame, for example, in Figure 3 In the background art, the "hand waving" action in the i+1 frame can be directly inferred from the i frame, and the "arm lifting to the head" action in the i+9 frame can also be predicted, although the interval is long, the same activity is continuous in the spatial and temporal dimensions; (2) for the continuity of different activities: when people continuously transform activities, continuity between different activities can still be observed. For example, in Figure 3 When people transform from "sitting" in the j frame to "standing" in the j+1 frame, the coordinates of the body parts will gradually change, rather than suddenly transform, or instantaneously complete the transformation. It can also be said that the human body cannot instantaneously transform from "sitting" in the j frame to "standing and waving hands" in the j+2 frame, and there should be a process of getting up from sitting to standing, i.e., including "standing" in the j+1 frame, between the two. This means that the switching between different actions of the human body is gradual and also has continuity. In addition, the activity transformation transition of the human body and the trajectory passed during the activity will follow the habits formed by people over a long period of time, for example, people usually sit down to rest after standing and exercising for a period of time, for example Figure 3 A series of changes from the j+1 frame to the j+31 frame are shown.

[0061] In some embodiments, as shown in Figure 4 The second neural network model and the human activity understanding and reconstruction model in the human activity understanding module include a point cloud feature extractor, a mask module, and a human activity understanding module.

[0062] The point cloud feature extractor is configured to extract high-dimensional features of the initial point cloud frame sequence, and correspondingly output a high-dimensional feature point cloud frame sequence.

[0063] The mask module is configured to randomly mask the output high-dimensional feature frame sequence according to a proportion, and correspondingly output a high-dimensional feature point cloud frame sequence containing mask feature frames.

[0064] The human activity understanding module is configured to extract the associated human dynamics features in the high-dimensional feature point cloud frames that are not masked from the output high-dimensional feature point cloud frame sequence containing mask feature frames, and correspondingly output a high-dimensional human dynamics feature frame sequence containing mask feature frames.

[0065] The human activity reconstruction module includes a human activity feature reconstructor and a point cloud generator.

[0066] The human activity feature reconstructor is configured to recover the output high-dimensional human dynamics feature frame sequence containing the mask feature frame, and correspondingly output a reconstructed high-dimensional feature point cloud frame sequence.

[0067] The point cloud generator is configured to perform point cloud reconstruction on the output reconstructed high-dimensional feature point cloud frame sequence, and correspondingly output a reconstructed point cloud frame sequence.

[0068] As shown in Figure 5 As shown in FIG. 4, the point cloud feature extractor includes three layers of convolutional networks, a feature compressor based on a self-attention mechanism, and three layers of fully connected networks connected in sequence, each convolutional network includes a one-dimensional convolution and a ReLU activation function, and each fully connected network includes a fully connected (FC) layer and a ReLU activation function. Figure 2 As shown in FIG. 4, the mask module can randomly mask the high-dimensional feature point cloud frame sequence output by the point cloud feature extractor at a proportion of 70%, that is, replace a number of continuous or discontinuous meaningful high-dimensional feature point cloud frames in the sequence with meaningless Gaussian noise, and output a corresponding high-dimensional feature point cloud frame sequence containing the mask feature frame. As shown in Figure 4 As shown in FIG. 5, the human activity understanding device and the human activity feature reconstructor respectively use transformer encoder and transformer decoder structures, and the point cloud generator is composed of one layer of FC layer. The output high-dimensional feature point cloud frame sequence containing the mask feature frame is input into the human activity understanding device, and the self-attention mechanism in the human activity understanding device enables the human activity understanding device to learn the continuity between different activities and the continuity within the same activity from the high-dimensional feature point cloud frames not masked by the mask, and extract the continuity between different activities and the continuity within the same activity. The human activity feature reconstructor recovers the feature frames masked by the mask according to the extracted continuity between different activities and the continuity within the same activity, and further obtains a corresponding reconstructed high-dimensional feature point cloud frame sequence.

[0069] In some embodiments, the loss function based on optimal transport and feature learning includes a loss function based on optimal transport and a loss function based on feature learning. The loss function based on optimal transport is used to optimize the distance between corresponding points in the corresponding point cloud frames in the generated reconstructed point cloud frame sequence and the corresponding initial point cloud frame sequence, and to optimize the point cloud distribution of the corresponding reconstructed point cloud frame sequence according to the initial point cloud frame sequence. The loss function based on feature learning is used to optimize the feature difference between the output reconstructed high-dimensional feature point cloud frame sequence and the corresponding high-dimensional feature point cloud frame sequence.

[0070] In the pre-training stage, a self-defined loss function based on optimal transport and feature learning is used to make the training task of the second neural network optimally fit, which can avoid the center clustering error that may be caused by the conventional method, improve the similarity between the reconstructed point cloud frame sequence and the initial point cloud frame sequence, and thus solve the problem that the training task of the second neural network falls into a local optimal solution (the conventional method only considers the average distance between each point in the point cloud, without considering the overall shape and topological structure of the point cloud, thus causing the recovered point cloud to have a center clustering problem). The specific idea of the loss function based on optimal transport and feature learning defined in the embodiment is as shown in Figure 6 The loss function based on optimal transport is defined as L D , the basic idea of which is to compare not only the distance between points in two sets of point clouds, but also the point cloud distribution of the two sets of point clouds. L D contains two parts: (i) distance constraint (L Distance ), which uses a cost function to minimize the cost of point cloud transport, i.e., the distance between corresponding points in the initial point cloud and the corresponding reconstructed point cloud, as shown by the black dashed line in Figure 6 . Specifically, the Euclidean distance is used as the cost function of the point cloud; (ii) distribution constraint (L Distribution ), which defines an entropy that produces an unreasonable distribution to punish the transport of the point cloud that is incorrectly clustered to the center (as shown by the red circle in Figure 6 ) during the reduction of the cost of point cloud transport, as shown by the red dashed line in Figure 6 . To achieve optimality, the entropy function corrects the red circle to the green circle pointed to by the red arrow. Specifically, the Kullback-Leibler (KL) divergence is used to measure the difference between the actual relationship of the point cloud and the ideal independent combination (i.e., the dependency of each point in the point cloud). By optimizing both, the distance between the two sets of point clouds is minimized, and the distributions are the same. L D is defined as shown in the following formula:

[0071]

[0072] where i represents the serial number of the point cloud frame sequence (the initial point cloud frame sequence or the reconstructed point cloud frame sequence corresponding to the initial point cloud frame sequence), i = 1, 2, …, n, n represents the total number of the point cloud frame sequence; ∈ is a hyperparameter (empirically valued at 0.01) for balancing L Distance and L Distribution ; c is the cost function, which is defined as c(p j , gp j ) = ‖p j -gpj ‖ 2 , i.e. the Euclidean distance between each point in each point cloud frame of the initial point cloud frame sequence and the corresponding reconstructed point cloud frame sequence, p j represents the initial point cloud in the jth initial point cloud frame of the initial point cloud frame sequence, gp j represents the reconstructed point cloud in the jth reconstructed point cloud frame of the corresponding reconstructed point cloud frame sequence of the initial point cloud frame sequence, j represents the serial number of each point cloud frame in the point cloud frame sequence; π is the transportation plan between all points in all point cloud frames in the initial point cloud frame sequence and the corresponding reconstructed point cloud frame sequence; μ(p j ) and v(gp j ) are the probability distributions of the initial point cloud p j in the jth initial point cloud frame and the reconstructed point cloud gp j in the corresponding jth reconstructed point cloud frame, respectively, A i represents the ith initial point cloud frame sequence, GA i represents the reconstructed point cloud frame sequence corresponding to the ith initial point cloud frame sequence A i , represents the tensor product operation.

[0073] Another part is the loss function L S based on feature learning, the basic principle of which is to minimize the difference between the high-dimensional features of the initial point cloud and the corresponding generated reconstructed point cloud. Since the high-dimensional features usually contain the geometric properties of the point cloud, such as density, shape and curvature, etc., the core design concept is to minimize the difference between the high-dimensional features generated by the point cloud feature extractor after processing the initial point cloud and the high-dimensional features of the corresponding reconstructed point cloud which has not yet been processed by the point cloud generator , and the specific method is vector dot product, as shown in the following formula:

[0074]

[0075] where τ is a hyperparameter used to adjust the similarity distribution of the high-dimensional features (empirically assigned as 0.1), which not only enhances the geometric correlation between the high-dimensional features of the two groups of point clouds, but also makes the point cloud generator and the point cloud feature extractor mutually coupled, thereby accelerating the fitting of pre-training.

[0076] Finally, the loss function based on optimal transportation and feature learning in the pre-training process is defined as shown in the following formula:

[0077] L = L D + λL S

[0078] where λ is an adjustable hyperparameter used to control L D and LS a specific gravity (empirically assigned as 0.1 to achieve optimal results).

[0079] In some embodiments, as shown in Figure 7 the knowledge transferred human body activity understanding module includes a point cloud feature extractor and a human body activity understandinger, the point cloud feature extractor is used to extract high-dimensional features of the input point cloud frame sequence, and a high-dimensional feature point cloud frame sequence is output; the human body activity understandinger is used to extract human body dynamics features in the output high-dimensional feature point cloud frame sequence, and a high-dimensional human body dynamics feature frame sequence is output.

[0080] The human body activity classifier is used to identify the output high-dimensional human body dynamics feature frame sequence, and a human body activity classification result is output.

[0081] As shown in Figure 2 In the fine-tuning stage of the present embodiment, further, only the point cloud feature extractor and the human body activity understandinger of the human body activity understanding module in the human body activity understanding and reconstruction model are retained, the mask module in the human body activity understanding module is removed, and the human body activity understanding module is weight frozen, to obtain a weight frozen and fine-tuned human body activity understanding module, and then the output of the weight frozen and fine-tuned human body activity understanding module is connected to the human body activity classifier, to obtain a second neural network model. In other embodiments, the mask module can also be retained. As shown in Figure 7 The human body activity classifier includes three layers of residual blocks, an FC layer and a Softmax activation function connected in sequence.

[0082] Figure 8 is a flowchart of a human body activity recognition method in an embodiment of the present application. As shown in Figure 8 The present application also provides a human body activity recognition method, which includes the following steps:

[0083] Step S810, obtaining a target initial point cloud frame sequence generated based on a millimeter wave signal for human body activity;

[0084] Step S820, inputting the target initial point cloud frame sequence into a human body activity recognition model, so that the human body activity recognition model outputs a human body activity classification result corresponding to the target initial point cloud frame sequence, wherein the human body activity recognition model is obtained by pre-training according to the human body activity recognition model training method in any one of the preceding embodiments.

[0085] In this embodiment, the human activity recognition model trained by the human activity recognition model training method can accurately recognize various human activities in daily scenarios and output human activity classification results. The human activity classification results include various hand activity types such as waving, stretching, etc. that are easily confused with walking and standing, various hand activity types that are easily confused with walking when standing, various leg activity types such as shaking and stretching legs that are easily confused with falling when sitting, and activity types such as walking, standing, sitting, and falling. Since the human dynamics associated in various human activities are learned in the transfer pre-training stage, various human activities in daily scenarios can be accurately recognized and classified, and the human activity recognition model can greatly avoid the confusion of human activities, especially the activities of local parts of the human body. It can achieve an accuracy of more than 90% and has high recognition accuracy. It can also be applied to various tasks such as skeleton prediction and health analysis.

[0086] The human activity recognition model training method and the human activity recognition method of the embodiment of the present application generate a millimeter wave point cloud by using the disturbance of human activity on the millimeter wave signal, especially for the case of missing millimeter wave point cloud when only part of the body (local part of the body) moves. Based on the self-supervised neural network algorithm, the neural network understands the association between various activities of the human body in daily scenarios and the millimeter wave point cloud in the pre-training task, learns the human dynamics associated in human activities, and thus can accurately recognize various activities under various intensity conditions in daily activities in the fine-tuning task. The present application breaks the disadvantage of easy confusion in the recognition of human daily activities by conventional wireless sensing methods, and is suitable for long-term monitoring of continuous activities in free human activity scenarios. In addition, since the millimeter wave radar only generates coarse-grained point cloud data of the human body, which describes the coordinates of the most active parts of the human body, rather than visual images generated by cameras, the privacy invasion of the present application is much smaller. At the same time, the millimeter wave wireless signal is not sensitive to light conditions and can work even in the dark.

[0087] In summary, the method provides a new non-contact and privacy-invasive human daily activity recognition scheme based on millimeter waves, which can accurately recognize human daily activities in a human-unaware manner all day long and continuously, and can robustly achieve high-accuracy activity recognition in various intensity activity states of the human body, has high applicability, and can be widely applied in the fields of smart home control, health monitoring, and fall detection.

[0088] Corresponding to the above method, the present application also provides an electronic device, which includes a computer device, the computer device includes a processor and a memory, the memory stores computer instructions, and the processor is used to execute the computer instructions stored in the memory. When the computer instructions are executed by the processor, the device realizes the steps of the foregoing method.

[0089] The embodiment of the present application further provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the steps of the foregoing method. The computer readable storage medium can be a tangible storage medium, such as a random access memory (RAM), a memory, a read only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a floppy disk, a hard disk, a removable memory disk, a CD-ROM, or any other form of storage medium known in the art.

[0090] The embodiment of the present application further provides a computer program product, which comprises computer instructions, and the computer instructions are executed by a processor to implement the steps of the foregoing method.

[0091] Those skilled in the art should understand that each example component, system and method described in connection with the embodiments disclosed herein can be implemented in hardware, software or a combination of both. The actual implementation depends on the specific application and design constraints imposed on the overall system. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application. When implemented in hardware, it can be, for example, an electronic circuit, an application specific integrated circuit (ASIC), appropriate firmware, a plug-in, a functional card, etc. When implemented in software, the elements of the present application are program or code segments used to perform the required tasks. The program or code segments can be stored in a machine readable medium or transmitted through a data signal carried in a carrier wave in a transmission medium or communication link.

[0092] It should be noted that the present application is not limited to the specific configurations and processes described above and shown in the drawings. For the sake of brevity, detailed descriptions of well-known methods are omitted. In the above embodiments, several specific steps are described and shown as examples. However, the method processes of the present application are not limited to the specific steps described and shown, and those skilled in the art can make various changes, modifications and additions, or change the order of the steps, after understanding the spirit of the present application.

[0093] In the present application, the features described and / or illustrated for one embodiment can be used in the same way or in a similar way in one or more other embodiments, and / or in combination with or instead of features of other embodiments.

[0094] The above merely illustrates the preferred embodiments of the present application, and is not used to limit the present application. The embodiments of the present application can be variously changed and modified by those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall fall within the scope of protection of the present application.

Claims

1. A method for training a human activity recognition model, characterized in that, The method comprises: training a first neural network model based on a plurality of point cloud frame sequences carrying human activity classification labels, so as to train the first neural network model into a human activity recognition model for outputting human activity classification results according to the point cloud frame sequences carrying human activity classification labels; wherein the point cloud frame sequences carrying human activity classification labels are obtained based on initial point cloud frame sequences, and the initial point cloud frame sequences are obtained based on millimeter wave signals generated for human activities; the first neural network model is obtained in advance according to a human activity understanding and reconstruction model, the human activity understanding and reconstruction model comprises a human activity understanding module and a human activity reconstruction module, the human activity understanding module is used to extract associated human dynamics features in the initial point cloud frame sequences according to a pre-learned association relationship between human activities and point cloud frame sequences, and the human activity reconstruction module is used to output a reconstructed point cloud frame sequence according to the extracted human dynamics features; the first neural network model comprises a human activity understanding module after knowledge transfer and a human activity classifier, the human activity understanding module after knowledge transfer is used to extract human dynamics features in an input point cloud frame sequence, and the human activity classifier is used to output a human activity classification result according to the extracted human dynamics features.

2. The method of claim 1, wherein, The human activity understanding and reconstruction model is pre-trained through the following steps: pre-training a second neural network model by minimizing a loss function based on optimal transportation and feature learning according to a plurality of initial point cloud frame sequences, so as to train the second neural network model into a human activity understanding and reconstruction model for outputting a reconstructed point cloud frame sequence according to the initial point cloud frame sequences.

3. The method of claim 2, wherein, The second neural network model and the human activity understanding module in the human activity understanding and reconstruction model comprise a point cloud feature extractor, a mask module and a human activity understanding device; the point cloud feature extractor is used to extract high-dimensional features of the initial point cloud frame sequences and output a high-dimensional feature point cloud frame sequence; the mask module is used to randomly mask the output high-dimensional feature frame sequence according to a proportion and output a high-dimensional feature point cloud frame sequence containing a mask feature frame; the human activity understanding device is used to extract associated human dynamics features from the output high-dimensional feature point cloud frame sequence containing a mask feature frame, and output a high-dimensional human dynamics feature frame sequence containing a mask feature frame; the human activity reconstruction module comprises a human activity feature reconstructor and a point cloud generator; the human activity feature reconstructor is used to restore the output high-dimensional human dynamics feature frame sequence containing a mask feature frame and output a reconstructed high-dimensional feature point cloud frame sequence; the point cloud generator is used to reconstruct the output reconstructed high-dimensional feature point cloud frame sequence and generate a reconstructed point cloud frame sequence.

4. The method of claim 3, wherein, The loss function based on optimal transport and feature learning includes a loss function based on optimal transport and a loss function based on feature learning. The loss function based on optimal transport is used to optimize the distance between corresponding points in the generated reconstructed point cloud frame sequence and the corresponding initial point cloud frame sequence, and to optimize the point cloud distribution of the corresponding reconstructed point cloud frame sequence according to the initial point cloud frame sequence. The loss function based on feature learning is used to optimize the feature difference between the output reconstructed high-dimensional feature point cloud frame sequence and the corresponding high-dimensional feature point cloud frame sequence.

5. The method of claim 1, wherein, The human activity understanding module after knowledge transfer includes a point cloud feature extractor and a human activity understanding device. The point cloud feature extractor is used to extract high-dimensional features of the input point cloud frame sequence, and the corresponding output high-dimensional feature point cloud frame sequence. The human activity understanding device is used to extract human dynamics features in the output high-dimensional feature point cloud frame sequence, and the corresponding output high-dimensional human dynamics feature frame sequence. The human activity classifier is used to identify the output high-dimensional human dynamics feature frame sequence, and the corresponding output human activity classification result.

6. The method according to any one of claims 1 to 5, characterized in that, The human dynamics features include continuity features within the same human activity and continuity features between different human activities.

7. A human activity recognition method characterized by comprising: The method includes: obtaining a target initial point cloud frame sequence generated based on a millimeter wave signal for human activity generation; inputting the target initial point cloud frame sequence into a human activity recognition model to make the human activity recognition model output a human activity classification result corresponding to the target initial point cloud frame sequence, wherein the human activity recognition model is pre-trained by the human activity recognition model training method according to any one of claims 1 to 6.

8. An electronic device comprising a processor, a memory, and computer instructions stored on the memory, wherein the computer instructions, when executed by the processor, cause the electronic device to: The processor is used to execute the computer instructions, and when the computer instructions are executed, the device implements the steps of the method according to any one of claims 1 to 7.

9. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the method according to any one of claims 1 to 7.

10. A computer program product comprising computer instructions, characterized in that, The computer instructions are executed by the processor to implement the steps of the method according to any one of claims 1 to 7. The computer instructions are executed by the processor to implement the steps of the method according to any one of claims 1 to 7.

Images