Training method and recognition method for human body activity recognition model containing easily confused activities

Abstract

The invention provides a training method and a recognition method for a human body activity recognition model containing easily-confused activities, and the training method comprises the steps: training a neural network model based on a plurality of point cloud frame sequences carrying human body activity classification labels, the neural network model is trained into a human body activity recognition model used for correspondingly outputting a classification result according to a point cloud frame sequence carrying a human body activity classification label; wherein the point cloud frame sequence carrying the human body activity classification label is obtained based on the initial point cloud frame sequence; the neural network model is obtained in advance according to a human body activity understanding and reconstruction model obtained through pre-training, and the human body activity understanding and reconstruction model comprises an understanding module and a reconstruction module which are connected in sequence; the neural network model comprises a knowledge-migrated understanding module and a classifier which are connected in sequence. According to the invention, on the premise of non-contact and no privacy invasion, various human body activities can be accurately identified by using the human body dynamics associated with the point cloud sequence learned in the pre-training stage.

Description

Technical Field

[0001] The present invention relates to the technical field of human activity recognition and machine learning, and in particular to a human activity recognition model training method and a recognition method. Background Art

[0002] Human activity recognition refers to the technology that provides us with a deep understanding and prediction of human activities by analyzing and interpreting the behavior patterns of individuals or groups in real time. It has shown its indispensable value in many fields. For example, in the field of security monitoring, activity sensing can timely identify abnormal behavior of workers to avoid safety accidents; in health monitoring, activity sensing can provide an additional layer of protection for the elderly or individuals with special needs, and can predict the occurrence of diseases through long-term behavior; in addition, in the field of smart homes, activity sensing can automatically adjust the home environment settings, such as lighting, temperature and security systems, according to the behavioral habits of residents, thereby improving living comfort and convenience.

[0003] At present, the mainstream activity recognition methods are mainly divided into two categories: solutions based on wearable devices and solutions based on optical devices. However, these two types of solutions have the defects of low comfort and privacy leakage, which hinder their widespread use in daily life. Specifically, the solution based on wearable devices requires the subject to wear one or more small devices, and analyzes the person's activity status based on the acceleration, angle and other information measured by the built-in inertial measurement unit (IMU). However, this solution is not comfortable and requires regular charging. People's discomfort limits its widespread use. Solutions based on optical devices usually analyze human activities based on images. However, optical sensors involve privacy issues in daily scenes and are limited by lighting conditions. They cannot be widely used in daily scenes (especially in home environments).

[0004] In recent years, emerging wireless sensing technologies such as millimeter wave sensors have emerged, which have the potential to continuously sense human activities in daily scenes. Since human activities will produce corresponding disturbances to wireless signals, people have begun to explore human activity recognition solutions based on wireless sensing technology. However, existing wireless sensing-based solutions do not take into account the diversity of human activities in daily scenes, and most daily scenes are not ideal environments. Activity recognition in daily scenes is easily confused, especially for activities that only produce movement of a part of the body, so it cannot be truly applied in daily scenes. Summary of the invention

[0005] In view of this, an embodiment of the present invention provides a human activity recognition model training method and a recognition method to eliminate or improve one or more defects existing in the prior art.

[0006] A first aspect of the present invention provides a method for training a human activity recognition model, the method comprising the following steps:

[0007] Training a first neural network model based on a plurality of point cloud frame sequences carrying human activity classification labels, so as to train the first neural network model into a human activity recognition model for outputting human activity classification results corresponding to the point cloud frame sequences carrying human activity classification labels;

[0008] Wherein, the point cloud frame sequence carrying the human activity classification label is obtained based on the initial point cloud frame sequence;

[0009] The first neural network model is obtained in advance based on a human activity understanding and reconstruction model, which includes a human activity understanding module and a human activity reconstruction module. The human activity understanding module is used to extract human dynamics features associated with an initial point cloud frame sequence, and the human activity reconstruction module is used to output a reconstructed point cloud frame sequence according to the extracted human dynamics features. The first neural network model includes a human activity understanding module after knowledge transfer and a human activity classifier. The human activity understanding module after knowledge transfer is used to extract human dynamics features from an input point cloud frame sequence, and the human activity classifier is used to output a human activity classification result according to the extracted human dynamics features.

[0010] In some embodiments of the present invention, the human activity understanding and reconstruction model is pre-trained by the following steps:

[0011] Pre-training a second neural network model according to a plurality of initial point cloud frame sequences by minimizing a loss function based on optimal transport and feature learning, so as to train the second neural network model as a human activity understanding and reconstruction model for reconstructing a point cloud frame sequence according to the corresponding output of the initial point cloud frame sequence;

[0012] Among them, the initial point cloud frame sequence is generated based on millimeter wave signals for human activities; the second neural network model and the human activity understanding and reconstruction model have the same network structure.

[0013] In some embodiments of the present invention, the human activity understanding module in the second neural network model and the human activity understanding and reconstruction model includes a point cloud feature extractor, a mask module and a human activity understander;

[0014] The point cloud feature extractor is used to extract high-dimensional features of the initial point cloud frame sequence, and correspondingly output a high-dimensional feature point cloud frame sequence;

[0015] The mask module is used to randomly mask the output high-dimensional feature frame sequence according to the ratio, and correspondingly output a high-dimensional feature point cloud frame sequence containing the mask feature frame;

[0016] The human activity understander is used to extract the human body dynamics features associated in the unmasked high-dimensional feature point cloud frames from the output high-dimensional feature point cloud frame sequence containing the mask feature frames, and correspondingly output the high-dimensional human body dynamics feature frame sequence containing the mask feature frames;

[0017] The human activity reconstruction module includes a human activity feature reconstructor and a point cloud generator;

[0018] The human activity feature reconstructor is used to restore the output high-dimensional human dynamics feature frame sequence containing the mask feature frame, and correspondingly output the reconstructed high-dimensional feature point cloud frame sequence;

[0019] The point cloud generator is used to perform point cloud reconstruction on the output reconstructed high-dimensional feature point cloud frame sequence, and correspondingly generate a reconstructed point cloud frame sequence.

[0020] In some embodiments of the present invention, the loss function based on optimal transportation and feature learning includes a loss function based on optimal transportation and a loss function based on feature learning. The loss function based on optimal transportation is used to optimize the distance between corresponding points in the corresponding point cloud frames in the generated reconstructed point cloud frame sequence and the corresponding initial point cloud frame sequence, and to optimize the point cloud distribution of the corresponding reconstructed point cloud frame sequence according to the initial point cloud frame sequence. The loss function based on feature learning is used to optimize the feature difference between the output reconstructed high-dimensional feature point cloud frame sequence and the corresponding high-dimensional feature point cloud frame sequence.

[0021] In some embodiments of the present invention, the human activity understanding module after knowledge transfer includes a point cloud feature extractor and a human activity understander, wherein the point cloud feature extractor is used to extract high-dimensional features of an input point cloud frame sequence, and correspondingly outputs a high-dimensional feature point cloud frame sequence; the human activity understander is used to extract human dynamic features in the output high-dimensional feature point cloud frame sequence, and correspondingly outputs a high-dimensional human dynamic feature frame sequence;

[0022] The human activity classifier is used to identify the output high-dimensional human dynamics feature frame sequence and output a human activity classification result accordingly.

[0023] In some embodiments of the present invention, the human body dynamics characteristics include continuity characteristics within the same human body activity and continuity characteristics between different human body activities.

[0024] A second aspect of the present invention provides a method for identifying human activities, the method comprising the following steps:

[0025] Obtain a target initial point cloud frame sequence generated based on millimeter wave signals for human activities;

[0026] The target initial point cloud frame sequence is input into a human activity recognition model so that the human activity recognition model outputs a human activity classification result corresponding to the target initial point cloud frame sequence, wherein the human activity recognition model is pre-trained by the human activity recognition model training method as described in the first aspect above.

[0027] The third aspect of the present invention provides an electronic device, which includes: a computer device, the computer device includes a processor and a memory, the memory stores computer instructions, the processor is used to execute the computer instructions stored in the memory, when the computer instructions are executed by the processor, the device implements the steps of the human activity recognition model training method as described in the first aspect above, or implements the steps of the human activity recognition method as described in the second aspect above.

[0028] The fourth aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the human activity recognition model training method as described in the first aspect above, or implements the steps of the human activity recognition method as described in the second aspect above.

[0029] The fifth aspect of the present invention provides a computer program product, comprising computer instructions, which, when executed by a processor, implement the steps of the human activity recognition model training method as described in the first aspect, or implement the steps of the human activity recognition method as described in the second aspect.

[0030] The human activity recognition model training method and recognition method of the present invention generate millimeter wave point clouds by utilizing the disturbance of millimeter wave signals caused by human activities, especially for the situation where the millimeter wave point clouds are missing when only a part of the body (local part of the body) moves. Based on the self-supervised neural network algorithm, in the pre-training task, the neural network understands the relationship between various human activities in daily scenes and millimeter wave point clouds, and learns the human body dynamics associated with human activities, so that accurate recognition of various types of activities can be achieved in the fine-tuning task.

[0031] Furthermore, the method can use millimeter wave wireless signals to accurately identify easily confusing human activities under various intensities of activity conditions in daily scenarios without contact and without violating privacy.

[0032] Additional advantages, purposes, and features of the present invention will be described in part in the following description, and will become apparent to those skilled in the art after studying the following, or may be learned from the practice of the present invention. The purposes and other advantages of the present invention may be achieved and obtained by the structures specifically indicated in the specification and the accompanying drawings.

[0033] Those skilled in the art will appreciate that the objectives and advantages that can be achieved with the present invention are not limited to the above specific description, and the above and other objectives that can be achieved by the present invention will be more clearly understood from the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] The drawings described herein are used to provide a further understanding of the present invention, constitute a part of this application, and do not constitute a limitation of the present invention.

[0035] Figure 1 A schematic diagram of a flow chart of a human activity recognition model training method according to an embodiment of the present invention;

[0036] Figure 2 A schematic diagram of the process of model pre-training and fine-tuning stages in one embodiment of the present invention;

[0037] Figure 3 Schematic diagram of the representation content of human body dynamic characteristics in one embodiment of the present invention;

[0038] Figure 4 is a schematic diagram of the structure of a second neural network model in one embodiment of the present invention;

[0039] Figure 5 is a schematic diagram of the structure of a point cloud feature extractor in one embodiment of the present invention;

[0040] Figure 6 is a schematic diagram of a loss function based on optimal transportation and feature learning in one embodiment of the present invention;

[0041] Figure 7 is a schematic diagram of the structure of a first neural network model in one embodiment of the present invention;

[0042] Figure 8 The figure is a flow chart of a method for identifying human activities in one embodiment of the present invention. DETAILED DESCRIPTION

[0043] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments and the accompanying drawings. Here, the illustrative embodiments of the present invention and their descriptions are used to explain the present invention, but are not intended to limit the present invention.

[0044] It should also be noted that, in order to avoid obscuring the present invention due to unnecessary details, only structures and / or processing steps closely related to the solutions according to the present invention are shown in the accompanying drawings, while other details that are not closely related to the present invention are omitted.

[0045] It should be emphasized that the term “include / comprises” when used herein refers to the presence of features, elements, steps or components, but does not exclude the presence or addition of one or more other features, elements, steps or components.

[0046] It should also be noted that, unless otherwise specified, the term “connection” herein may refer not only to a direct connection but also to an indirect connection involving an intermediate.

[0047] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. In the accompanying drawings, the same reference numerals represent the same or similar components, or the same or similar steps.

[0048] Human activities in daily scenes are mainly divided into two categories: "local activities" (only part of the body moves) and "whole body activities" (all parts of the body move). Most of the confusion in the recognition of human activities occurs when the human body is in the "local activity" state, and in daily scenes, people are in this state for more than 80% of the time. Through quantitative observation, it is found that in the "whole body activity" state, the point cloud is distributed almost all parts of the human body; while in the "local activity" state, the point cloud is only distributed on the active parts of the human body. In the millimeter wave field of view, both describe the overall movement state of the person, but the point cloud in the "local activity" state is obviously not the case. Therefore, it is very easy to confuse the recognition between activities in this state, which is specifically manifested in that two different activities produce a point cloud sequence with the same trend. For example, shaking legs while sitting is easily confused as falling, and is thus mistakenly identified as falling. The key reason for the above problems is that the current research algorithm is limited to learning the human point cloud itself, but ignores the connection between the point cloud generated by the millimeter wave signal for human activities in daily scenes and human activities, that is, the human dynamics knowledge associated in the point cloud.

[0049] Therefore, an embodiment of the present invention provides a human activity recognition model training method and a recognition method. Figure 1 FIG. 1 is a flow chart of a method for training a human activity recognition model according to an embodiment of the present invention. Figure 1 As shown, the method comprises the following steps:

[0050] Step S110, training a first neural network model based on a plurality of point cloud frame sequences carrying human activity classification labels, so as to train the first neural network model into a human activity recognition model for outputting human activity classification results corresponding to the point cloud frame sequences carrying human activity classification labels;

[0051] Wherein, the point cloud frame sequence carrying the human activity classification label is obtained based on the initial point cloud frame sequence;

[0052] The first neural network model is obtained in advance based on a human activity understanding and reconstruction model, which includes a human activity understanding module and a human activity reconstruction module. The human activity understanding module is used to extract human dynamics features associated with an initial point cloud frame sequence, and the human activity reconstruction module is used to output a reconstructed point cloud frame sequence according to the extracted human dynamics features. The first neural network model includes a human activity understanding module after knowledge transfer and a human activity classifier. The human activity understanding module after knowledge transfer is used to extract human dynamics features from an input point cloud frame sequence, and the human activity classifier is used to output a human activity classification result according to the extracted human dynamics features.

[0053] This method adopts self-supervised learning, such as Figure 2 As shown, first, through pre-training, the human activity understanding and reconstruction model understands and learns the association and connection between various activities performed by people in daily scenes and the corresponding point clouds, so that the human activity understanding and reconstruction model understands various human activities and human dynamics in daily scenes. Then, in the fine-tuning stage, the first neural network model is obtained in advance by performing knowledge transfer and fine-tuning on the human activity understanding and reconstruction model obtained by pre-training. The specific operation of fine-tuning is to remove the human activity reconstruction module in the human activity understanding and reconstruction model, retain only the human activity understanding module and perform knowledge transfer on the module, and add a human activity classifier on the basis of the human activity understanding module after knowledge transfer, the purpose of which is to use the human dynamics learned in the pre-training stage to complete the accurate classification of daily activities; the specific operation of knowledge transfer is weight parameter freezing. A small number of labeled point cloud frame sequences representing the continuity of local part activities and whole body activities of the human body are input into the first neural network model for iterative training, so that the trained human activity recognition model can output accurate human activity classification results.

[0054] Therefore, before training the fine-tuned model, it is necessary to obtain a small number of initial point cloud frame sequences and mark the obtained small number of initial point cloud frame sequences with respective labels. The label is used to indicate the human activity type to which the initial point cloud frame sequence belongs, that is, the human activity classification label, which may be part or all of the following local and whole-body activity types: various hand activity types such as waving and reaching forward when sitting that are easily confused with walking and standing, various hand activity types when standing that are easily confused with walking, various leg activity types such as shaking legs and stretching legs when sitting that are easily confused with falling, as well as activity types such as walking, standing, sitting and falling.

[0055] In some embodiments, the human activity understanding and reconstruction model is pre-trained by the following steps:

[0056] Step S010, pre-training a second neural network model according to a plurality of initial point cloud frame sequences by minimizing a loss function based on optimal transport and feature learning, so as to train the second neural network model as a human activity understanding and reconstruction model for reconstructing a point cloud frame sequence according to the corresponding output of the initial point cloud frame sequence;

[0057] Among them, the initial point cloud frame sequence is generated based on millimeter wave signals for human activities; the second neural network model and the human activity understanding and reconstruction model have the same network structure.

[0058] When obtaining the initial point cloud frame sequence for training the model, this embodiment can use frequency modulated continuous wave (FMCW) radars of various frequency bands to transmit FMCW millimeter wave signals to continuously sense human activities, and then receive and process the changing reflection signals generated by human activities, and generate the initial point cloud frame sequence for the human activities based on the changing reflection signals. The point cloud in the sequence frame describes the coordinates of the most active parts of the human body during activities. In the process of generating the initial point cloud frame sequence, a signal processing algorithm is first used to generate a millimeter wave single-frame point cloud, and then the continuous single-frame point clouds generated for human activities form an initial point cloud frame sequence. The specific process of the signal processing algorithm is as follows: ① Perform a distance Fourier transform on the millimeter wave signal to obtain a distance map; ② Use a static filtering algorithm to filter the static information in the environment on the distance map; ③ Use the Capon beamforming algorithm to process the azimuth direction virtual antenna data in the filtered distance map to generate a distance-azimuth matrix; ④ Use a constant false alarm rate algorithm to remove noise in the distance-azimuth matrix to obtain the distance and azimuth information of the target; ⑤ Use the Capon beamforming algorithm and Doppler Fourier transform to process the data of the pitch angle virtual antenna direction in the azimuth information that has detected the target distance information, and obtain the pitch angle and radial velocity information of the target respectively. Finally, a set of points with position and velocity information distributed on the surface of the human body is obtained.

[0059] In fact, during the pre-training and training of the fine-tuned model, the point cloud frames contained in the point cloud frame sequence data input into the model for iterative training may be continuous frames or discontinuous frames.

[0060] During the pre-training process, the unlabeled initial point cloud frame sequence is input into the second neural network model, so that the second neural network model learns the association and connection between different types of human activities and millimeter wave point clouds in daily scenes, that is, understands different types of human activities and human dynamics in daily scenes. The second neural network model mainly analyzes the "continuity within the same activity" and "continuity between different activities" of daily activities to enable the model to understand and learn human dynamics. Figure 3As shown in the representation, the human body dynamics features include the continuity features within the same human activity and the continuity features between different human activities. Specifically, (1) For the continuity of the same activity: When a human body is performing an activity, the representation of the activity is usually composed of a series of continuous point cloud frames. For the same activity, whether it is a short-term or long-term activity, the action in the later frame is relatively easy to predict based on the action in the previous frame. For example, Figure 3 In the example above, the “waving” action in the i+1th frame can be directly inferred from the ith frame, and the “raising the arm to the head” action in the i+9th frame can also be predicted. Although the interval is long, the same activity is continuous in both space and time dimensions; (2) For the continuity of different activities: when people continuously change activities, the continuity between different activities can still be observed. For example, in Figure 3 In the figure, when people switch from "sitting" in the jth frame to "standing" in the j+1th frame, the coordinates of various parts of the body will change gradually, rather than suddenly or instantly. It can also be said that the human body cannot instantly switch from "sitting" in the jth frame to "standing and waving" in the j+2th frame. There should be a process of getting up from sitting to standing, which includes the "standing" in the j+1th frame. This means that the switching between different actions of the human body is gradual and also continuous. In addition, the transition of human activity changes and the trajectory taken during the activity will follow the habits formed by people over a long period of time. For example, people usually sit down to rest after standing and exercising for a period of time. Figure 3 Shown is a series of changes from the j+1th frame to the j+31th frame.

[0061] In some embodiments, Figure 4 As shown, the human activity understanding module in the second neural network model and the human activity understanding and reconstruction model includes a point cloud feature extractor, a mask module and a human activity understander;

[0062] The point cloud feature extractor is used to extract high-dimensional features of the initial point cloud frame sequence, and correspondingly output a high-dimensional feature point cloud frame sequence;

[0063] The mask module is used to randomly mask the output high-dimensional feature frame sequence according to the ratio, and correspondingly output a high-dimensional feature point cloud frame sequence containing the mask feature frame;

[0064] The human activity understander is used to extract the human body dynamics features associated in the unmasked high-dimensional feature point cloud frames from the output high-dimensional feature point cloud frame sequence containing the mask feature frames, and correspondingly output the high-dimensional human body dynamics feature frame sequence containing the mask feature frames;

[0065] The human activity reconstruction module includes a human activity feature reconstructor and a point cloud generator;

[0066] The human activity feature reconstructor is used to restore the output high-dimensional human dynamics feature frame sequence containing the mask feature frame, and correspondingly output the reconstructed high-dimensional feature point cloud frame sequence;

[0067] The point cloud generator is used to perform point cloud reconstruction on the output reconstructed high-dimensional feature point cloud frame sequence, and correspondingly generate a reconstructed point cloud frame sequence.

[0068] For example, Figure 5 As shown in , the point cloud feature extractor includes three layers of convolutional networks connected in sequence, a feature compressor based on the self-attention mechanism, and three layers of fully connected networks. Each convolutional network contains a one-dimensional convolution and a ReLU activation function, and each fully connected network contains a fully connected (FC) layer and a ReLU activation function. Figure 2 As shown in FIG4 , the mask module can randomly mask the high-dimensional feature point cloud frame sequence output by the point cloud feature extractor at a ratio of 70%, that is, replace a number of continuous or discontinuous meaningful high-dimensional feature point cloud frames in the sequence with meaningless Gaussian noise, and output the corresponding high-dimensional feature point cloud frame sequence containing masked feature frames. Figure 4 As shown in the figure, the human activity understander and the human activity feature reconstructor use transformer encoder and transformer decoder structures respectively, and the point cloud generator consists of one FC layer. The output high-dimensional feature point cloud frame sequence containing the mask feature frame is input into the human activity understander. Through the self-attention mechanism in the human activity understander, the human activity understander learns the continuity between different activities and the continuity process within the same activity reflected in the unmasked high-dimensional feature point cloud frame with a global vision, and extracts the continuity features between different activities and the continuity features within the same activity. The human activity feature reconstructor restores the masked feature frame based on the extracted continuity features between different activities and the continuity features within the same activity, and then obtains the corresponding reconstructed high-dimensional feature point cloud frame sequence.

[0069] In some embodiments, the loss function based on optimal transportation and feature learning includes a loss function based on optimal transportation and a loss function based on feature learning. The loss function based on optimal transportation is used to optimize the distance between corresponding points in the corresponding point cloud frames in the generated reconstructed point cloud frame sequence and the corresponding initial point cloud frame sequence, and to optimize the point cloud distribution of the corresponding reconstructed point cloud frame sequence according to the initial point cloud frame sequence. The loss function based on feature learning is used to optimize the feature difference between the output reconstructed high-dimensional feature point cloud frame sequence and the corresponding high-dimensional feature point cloud frame sequence.

[0070] In the pre-training stage, a customized loss function based on optimal transport and feature learning is used to make the training task of the second neural network fit optimally. This loss function can avoid the center clustering errors that may be caused by conventional methods, improve the similarity between the reconstructed point cloud frame sequence and the initial point cloud frame sequence, and thus make the training task of the second neural network fall into the problem of local optimal solution (conventional methods only consider the average distance between each point in the point cloud, without considering the overall shape and topological structure of the point cloud, which leads to the problem of center clustering of the restored point cloud). The specific idea of ​​the loss function based on optimal transport and feature learning defined in this embodiment is as follows: Figure 6 As shown (each red circle, green circle and blue circle in the figure represents each point in the point cloud, where the red circle and green circle represent the reconstructed point cloud and the blue circle represents the initial point cloud), the loss function based on optimal transportation is defined as L D The basic idea is to compare not only the distances between points in two sets of point clouds, but also the point cloud distribution of the two sets of point clouds. D It consists of the following two parts: (i) distance constraint (L Distance ), use the cost function to minimize the cost of point cloud transmission, that is, the distance between the corresponding points in the initial point cloud and the corresponding reconstructed point cloud, such as Figure 6 Specifically, the Euclidean distance is used as the cost function of the point cloud; (ii) the distribution constraint (L Distribution ), in the process of reducing the transmission cost of point cloud, an entropy that produces unreasonable distribution is defined to punish the point cloud that is incorrectly oriented to the center (such as Figure 6 The transport of aggregates is shown in red circles in Figure 6 As shown by the red dotted line in . To achieve the optimal, the entropy function corrects the red circle to the green circle pointed by the red arrow. Specifically, the Kullback-Leibler (KL) divergence is used to measure the difference between the actual relationship of the point cloud and the ideal independent combination (i.e., the dependency of each point in the point cloud). By optimizing both, the distance between the two groups of point clouds is minimized and the distribution is the same. L D The definition of is shown in the following formula:

[0071]

[0072] Where i represents the sequence number of the point cloud frame sequence (the initial point cloud frame sequence or the reconstructed point cloud frame sequence corresponding to the initial point cloud frame sequence), i = 1, 2, ..., n, n represents the total number of point cloud frame sequences; ∈ is used to balance L Distance and L Distribution is a hyperparameter (empirically assigned to 0.01); c is the cost function, defined as c(p j ,gp j )=‖p j -gpj ‖ 2 , that is, the Euclidean distance between each point in each point cloud frame of the initial point cloud frame sequence and the corresponding reconstructed point cloud frame sequence, p j represents the initial point cloud in the jth initial point cloud frame of the initial point cloud frame sequence, gp j represents the reconstructed point cloud in the jth reconstructed point cloud frame of the reconstructed point cloud frame sequence corresponding to the initial point cloud frame sequence, j represents the sequence number of each point cloud frame in the point cloud frame sequence; π is the transportation plan between all points in all point cloud frames in the initial point cloud frame sequence and the corresponding reconstructed point cloud frame sequence; μ(p j ) and v(gp j ) are the initial point clouds p in the jth initial point cloud frame. j and the corresponding reconstructed point cloud gp in the jth reconstructed point cloud frame j The probability distribution of A i represents the i-th initial point cloud frame sequence, GA i Represents the i-th initial point cloud frame sequence A i The corresponding reconstructed point cloud frame sequence, Represents a tensor product operation.

[0073] The other part is the loss function L based on feature learning S The basic principle is to minimize the difference between the high-dimensional features of the initial point cloud and the corresponding reconstructed point cloud. Since high-dimensional features usually contain geometric properties of point clouds, such as density, shape, and curvature, the core design concept is to minimize the high-dimensional features generated by the point cloud feature extractor after processing the initial point cloud. and the high-dimensional features of the reconstructed point cloud that have not yet been processed by the point cloud generator The specific method is the vector dot product, as shown in the following formula:

[0074]

[0075] Among them, τ is a hyperparameter used to adjust the similarity distribution of high-dimensional features (empirically assigned a value of 0.1), which can not only enhance the geometric correlation between the high-dimensional features of the two sets of point clouds, but also make the point cloud generator and the point cloud feature extractor mutually coupled, thereby accelerating the fitting of pre-training.

[0076] Finally, the loss function based on optimal transport and feature learning during pre-training is defined as follows:

[0077] L=L D +λL S

[0078] Among them, λ is an adjustable hyperparameter used to control L D and LS The proportion (empirically assigned a value of 0.1 to achieve the best effect).

[0079] In some embodiments, Figure 7 As shown, the human activity understanding module after the knowledge transfer includes a point cloud feature extractor and a human activity understander, wherein the point cloud feature extractor is used to extract high-dimensional features of the input point cloud frame sequence, and correspondingly outputs a high-dimensional feature point cloud frame sequence; the human activity understander is used to extract human dynamics features in the output high-dimensional feature point cloud frame sequence, and correspondingly outputs a high-dimensional human dynamics feature frame sequence;

[0080] The human activity classifier is used to identify the output high-dimensional human dynamics feature frame sequence and output a human activity classification result accordingly.

[0081] like Figure 2 As shown, in the fine-tuning stage of this embodiment, further, only the point cloud feature extractor and the human activity understander of the human activity understanding module in the human activity understanding and reconstruction model are retained, the mask module in the human activity understanding module is removed, and the weight of the human activity understanding module is frozen to obtain the weight-frozen and fine-tuned human activity understanding module, and then the output of the weight-frozen and fine-tuned human activity understanding module is connected to the human activity classifier to obtain the second neural network model. In other embodiments, the mask module can also be retained. Figure 7 As shown in Figure 1, the human activity classifier contains three layers of residual blocks, FC layers and Softmax activation functions connected in sequence.

[0082] Figure 8 FIG. 1 is a flow chart of a method for identifying human activity in one embodiment of the present invention. Figure 8 As shown, the embodiment of the present invention also provides a method for identifying human activities, the method comprising the following steps:

[0083] Step S810, obtaining a target initial point cloud frame sequence generated based on a millimeter wave signal for human activity;

[0084] Step S820, inputting the target initial point cloud frame sequence into a human activity recognition model, so that the human activity recognition model outputs a human activity classification result corresponding to the target initial point cloud frame sequence, wherein the human activity recognition model is pre-trained by the human activity recognition model training method described in any one of the aforementioned embodiments.

[0085] In this embodiment, the human activity recognition model trained by the human activity recognition model training method can accurately recognize various human activities in daily scenes and output human activity classification results. The human activity classification results include various types of hand activities such as waving and reaching forward when sitting, which are easily confused with walking and standing, various types of hand activities when standing, which are easily confused with walking, various types of leg activities such as shaking legs and stretching legs when sitting, which are easily confused with falling, and activity types such as walking, standing, sitting and falling. Due to the use of the human body dynamics associated with various human activities learned in the transfer pre-training stage, various human activities in daily scenes can be accurately recognized and classified. The human activity recognition model can greatly avoid the confusion and recognition of human activities, especially the activities of local parts of the human body; and can achieve an accuracy rate of more than 90%, with high recognition accuracy; it can also be applied to various tasks such as skeleton prediction and behavioral health analysis.

[0086] The human activity recognition model training method and human activity recognition method of the embodiment of the present invention generate millimeter wave point clouds by utilizing the disturbance of human activities on millimeter wave signals, especially for the situation where the millimeter wave point clouds are missing when only a part of the body (local body part) generates movement, and the self-supervised neural network algorithm enables the neural network to understand the association between various human activities in daily scenes and millimeter wave point clouds in the pre-training task, and learn the human body dynamics associated with human activities, so that accurate recognition of various activities under various daily activity conditions of various intensities can be achieved in the fine-tuning task. The present invention breaks the disadvantage that the conventional wireless sensing method is easily confused in the recognition of daily human activities, and is suitable for long-term monitoring of continuous activities in free human activity scenes. In addition, since the millimeter wave radar only generates coarse-grained point cloud data of the human body, the point cloud data describes the coordinates of the most active parts of the human body, rather than the visual images generated by the camera, so the privacy intrusion of the present invention is much less. At the same time, the millimeter wave wireless signal is not sensitive to light conditions and can work even in the dark.

[0087] In summary, this method provides a new millimeter-wave-based non-contact, privacy-intrusive human daily activity recognition solution, which can accurately identify people's daily activities continuously around the clock without the human body being aware of it, and can robustly achieve high-accuracy activity recognition under various intensity activities of the human body. It has high applicability and can be widely used in smart home control, health monitoring, fall detection and other fields.

[0088] Corresponding to the above method, the present invention also provides an electronic device, which includes a computer device, the computer device includes a processor and a memory, the memory stores computer instructions, the processor is used to execute the computer instructions stored in the memory, and when the computer instructions are executed by the processor, the device implements the steps of the above method.

[0089] The embodiment of the present invention also provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, the steps of the aforementioned method are implemented. The computer-readable storage medium can be a tangible storage medium, such as a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a floppy disk, a hard disk, a removable storage disk, a CD-ROM, or any other form of storage medium known in the technical field.

[0090] An embodiment of the present invention further provides a computer program product, comprising computer instructions, which implement the steps of the aforementioned method when executed by a processor.

[0091] It should be understood by those skilled in the art that the exemplary components, systems and methods described in conjunction with the embodiments disclosed herein can be implemented in hardware, software or a combination of the two. Whether it is performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention. When implemented in hardware, it can be, for example, an electronic circuit, an application specific integrated circuit (ASIC), appropriate firmware, a plug-in, a function card, etc. When implemented in software, the elements of the present invention are programs or code segments used to perform the required tasks. The program or code segment can be stored in a machine-readable medium, or transmitted on a transmission medium or a communication link via a data signal carried in a carrier.

[0092] It should be clear that the present invention is not limited to the specific configuration and processing described above and shown in the figures. For the sake of simplicity, a detailed description of the known method is omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present invention is not limited to the specific steps described and shown, and those skilled in the art can make various changes, modifications and additions, or change the order between the steps after understanding the spirit of the present invention.

[0093] In the present invention, features described and / or illustrated for one embodiment may be used in the same or similar manner in one or more other embodiments, and / or combined with or replace features of other embodiments.

[0094] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the embodiments of the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A human activity recognition model training method, characterized in that: The method comprises: Training a first neural network model based on a plurality of point cloud frame sequences carrying human activity classification labels, so as to train the first neural network model into a human activity recognition model for outputting human activity classification results corresponding to the point cloud frame sequences carrying human activity classification labels; Wherein, the point cloud frame sequence carrying the human activity classification label is obtained based on the initial point cloud frame sequence; The first neural network model is obtained in advance based on a human activity understanding and reconstruction model, which includes a human activity understanding module and a human activity reconstruction module. The human activity understanding module is used to extract human dynamics features associated with an initial point cloud frame sequence, and the human activity reconstruction module is used to output a reconstructed point cloud frame sequence according to the extracted human dynamics features. The first neural network model includes a human activity understanding module after knowledge transfer and a human activity classifier. The human activity understanding module after knowledge transfer is used to extract human dynamics features from an input point cloud frame sequence, and the human activity classifier is used to output a human activity classification result according to the extracted human dynamics features.

2. The method according to claim 1, characterized in that The human activity understanding and reconstruction model is pre-trained by the following steps: Pre-training a second neural network model according to a plurality of initial point cloud frame sequences by minimizing a loss function based on optimal transport and feature learning, so as to train the second neural network model as a human activity understanding and reconstruction model for reconstructing a point cloud frame sequence according to the corresponding output of the initial point cloud frame sequence; The initial point cloud frame sequence is generated based on millimeter wave signals for human activities.

3. The method according to claim 2, characterized in that The human activity understanding module in the second neural network model and the human activity understanding and reconstruction model includes a point cloud feature extractor, a mask module and a human activity understander; The point cloud feature extractor is used to extract high-dimensional features of the initial point cloud frame sequence, and correspondingly output a high-dimensional feature point cloud frame sequence; The mask module is used to randomly mask the output high-dimensional feature frame sequence according to the ratio, and correspondingly output a high-dimensional feature point cloud frame sequence containing the mask feature frame; The human activity understander is used to extract the human body dynamics features associated in the unmasked high-dimensional feature point cloud frames from the output high-dimensional feature point cloud frame sequence containing the mask feature frames, and correspondingly output the high-dimensional human body dynamics feature frame sequence containing the mask feature frames; The human activity reconstruction module includes a human activity feature reconstructor and a point cloud generator; The human activity feature reconstructor is used to restore the output high-dimensional human dynamics feature frame sequence containing the mask feature frame, and correspondingly output the reconstructed high-dimensional feature point cloud frame sequence; The point cloud generator is used to perform point cloud reconstruction on the output reconstructed high-dimensional feature point cloud frame sequence, and correspondingly generate a reconstructed point cloud frame sequence.

4. The method according to claim 3, characterized in that The loss function based on optimal transportation and feature learning includes a loss function based on optimal transportation and a loss function based on feature learning. The loss function based on optimal transportation is used to optimize the distance between corresponding points in the corresponding point cloud frames in the generated reconstructed point cloud frame sequence and the corresponding initial point cloud frame sequence, and to optimize the point cloud distribution of the corresponding reconstructed point cloud frame sequence according to the initial point cloud frame sequence. The loss function based on feature learning is used to optimize the feature difference between the output reconstructed high-dimensional feature point cloud frame sequence and the corresponding high-dimensional feature point cloud frame sequence.

5. The method according to claim 1, characterized in that The human activity understanding module after knowledge transfer includes a point cloud feature extractor and a human activity understander. The point cloud feature extractor is used to extract high-dimensional features of an input point cloud frame sequence, and correspondingly outputs a high-dimensional feature point cloud frame sequence; the human activity understander is used to extract human dynamics features in the output high-dimensional feature point cloud frame sequence, and correspondingly outputs a high-dimensional human dynamics feature frame sequence; The human activity classifier is used to identify the output high-dimensional human dynamics feature frame sequence and output a human activity classification result accordingly.

6. The method according to any one of claims 1 to 5, characterized in that The human body dynamics characteristics include continuity characteristics within the same human body activity and continuity characteristics between different human body activities.

7. A method for identifying human activity, characterized in that: The method comprises: Obtain a target initial point cloud frame sequence generated based on millimeter wave signals for human activities; The target initial point cloud frame sequence is input into a human activity recognition model so that the human activity recognition model outputs a human activity classification result corresponding to the target initial point cloud frame sequence, wherein the human activity recognition model is pre-trained by the human activity recognition model training method as described in any one of claims 1 to 6.

8. An electronic device comprising a processor, a memory and computer instructions stored in the memory, characterized in that: The processor is used to execute the computer instructions, and when the computer instructions are executed, the device implements the steps of the method according to any one of claims 1 to 7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer program product comprising computer instructions, characterized in that When the computer instructions are executed by a processor, the steps of the method described in any one of claims 1 to 7 are implemented.

Images